Risk-grading unmanned aerial vehicle railway foreign matter invasion open detection method

Abstract

The invention provides an unmanned aerial vehicle railway foreign matter invasion open detection method based on risk grading. According to the method, the risk level is determined according to the transverse distance between an invading foreign matter and a track core area, smoothing processing is carried out on a track area segmentation mask through a boundary-limited MCP method, and parameter-configurable unmanned aerial vehicle railway image multi-level risk area division is achieved. The invention further provides a railway foreign matter intrusion detection method architecture FOCS-MID under the open-closed set fusion condition, on the basis of intrusion detection of conventional foreign matters under the closed set condition, secondary detection under open set setting is carried out on conventional and unconventional foreign matters at the same time, and the risk perception capability of railway line foreign matters is comprehensively improved.

Description

Technical Field

[0001] The invention relates to a railway foreign body intrusion detection method for unmanned aerial vehicle railway inspection, and belongs to the field of rail transit operation safety and security. Background Art

[0002] In recent years, drone technology has developed rapidly and has been widely used in inspections in industries such as railways, electricity, and photovoltaics. Drones can perform a variety of tasks such as aerial reconnaissance, logistics distribution, and inspection monitoring. Edge computing, as a supplement to cloud computing, deploys data processing and analysis capabilities at the edge of the network, greatly reducing latency and improving response speed. The combination of drone technology and edge computing enables drones to process complex multi-source data on the railway site in real time and make decisions autonomously, thus playing a key role in real-time railway inspection and monitoring. Summary of the invention

[0003] Foreign objects within and around the railway line may pose a serious threat to the safe operation of trains, and may even cause train derailment and vehicle damage, or even endanger the lives of passengers. Early detection and early warning of intrusive foreign objects are the key to ensuring the safe operation of the railway system. UAVs have good maneuverability, diverse viewing angles, and a wide field of view, and can play an important role in railway foreign object intrusion detection. The rapid and normalized railway foreign object intrusion detection based on drones has excellent application value in the real-time monitoring and control of foreign object risks on railway lines. However, the traditional railway foreign object intrusion detection method does not distinguish the risks brought by intrusive foreign objects appearing in different locations, but simply regards all foreign objects or potential foreign objects as risk factors at the same risk level; at the same time, the categories of foreign objects in real-world open railway scenes are diverse and uncertain, and it is impossible to fully define the categories of all intrusive foreign objects in advance, so it is impossible to effectively deal with intrusive foreign objects of undefined categories.

[0004] In response to the above problems, the present invention proposes a multi-risk level area division method for UAV railway inspection images based on boundary-restricted MCP. This method determines the risk level of the intruding foreign object according to the lateral distance between the intruding foreign object and the core area of ​​the track, and smoothes the track area segmentation mask through the boundary-restricted MCP method, thus realizing parameter-configurable multi-level risk area division of UAV railway images. The present invention further proposes a railway foreign object intrusion detection method architecture FOCS-MID under open-closed set fusion conditions. On the basis of the intrusion detection of conventional foreign objects under closed set conditions, the conventional and unconventional foreign objects are simultaneously subjected to secondary detection under open set settings, thereby comprehensively improving the risk perception capability of foreign objects on railway lines.

[0005] The present invention proposes a new multi-risk level region division method for UAV railway inspection images based on boundary-constrained MCP, and a railway foreign object intrusion open detection method architecture FOCS-MID based on the risk region division. The overall operation architecture is as follows Figure 1 It specifically includes the following two parts, which are described below in conjunction with the accompanying drawings:

[0006] (1) Method for dividing multi-level risk areas of MCP with limited boundaries

[0007] Aiming at the fact that the risks caused by foreign objects in different areas of UAV railway inspection images are different, a minimum circumscribed parallelogram (MCP) risk area division method with restricted boundaries is proposed. The multi-level risk area division with smooth boundaries is achieved based on the track area segmentation results, and the image is divided into warning area, early warning area and attention area.

[0008] (2) Foreign Object Intrusion Open Detection Method Architecture FOCS-MID

[0009] Aiming at the problem that foreign objects in real open-world railway scenarios are diverse and uncertain, and it is impossible to fully define all types of foreign objects in advance, a multi-risk intrusion detection method formulation under fusion of open-closed sets (FOCS-MID) is proposed. It realizes foreign object intrusion detection with multi-level risk area matching, and expands the capability of railway foreign object intrusion detection to open scenarios.

[0010] The method is implemented in the following way;

[0011] The drone-viewed railway images are divided into three levels of risk areas according to the distance from the core area of ​​the track: core warning area, adjacent warning area and peripheral attention area.

[0012] The core warning area should only focus on the maximum outer boundary area where the train runs. The maximum connected domain of the segmentation mask of the core railway area predicted by the deep segmentation network is expressed as Where j represents the number of the largest connected domain in the image; the largest connected domain in the core area of ​​a single track parsed by the network Draw at equal intervals vertically or horizontally The connected domain is sampled by a horizontal or vertical line, corresponding to the two cases where the railway lines in the image are distributed horizontally and vertically; the railway lines are divided vertically and horizontal sampling lines are drawn at equal intervals along the vertical direction; for each horizontal sampling line, The midpoint of the two points farthest apart that coincide with the horizontal line is represented by (x i ,y i ), and the distance between these two points is d i ; In this way, the center line of the core area of ​​the track is evenly sampled longitudinally points, which are represented here as Then use this Performing simple linear regression on the points will yield the equation for the center line of the railway area. The center point of the points is represented by (x o ,y o ),in:

[0013]

[0014] At the same time, the sample Maximum horizontal distance Here its maximum value is defined as the pixel width of the track core area, denoted as d:

[0015]

[0016] This The equation of the center line l of the orbital core area obtained by performing simple linear regression on the two-dimensional plane is expressed as:

[0017] f(x,y)=ax+by+c=0 (3)

[0018] The equation of the right boundary line l1 of the orbital core area in the two-dimensional plane is obtained by translating the line d / 2 along the positive direction of the x-axis, which is expressed as:

[0019] f1(x,y)=a(xd / 2)+by+c=0 (4)

[0020] The equation of the left boundary line l2 of the orbital core region on the two-dimensional plane is obtained by translating the line l along the negative direction of the x-axis by a distance of d / 2, which is expressed as:

[0021] f2(x,y)=a(x+d / 2)+by+c=0 (5)

[0022] Here we first give a method to represent the half-plane area on the same side and the opposite side of a point outside a straight line. In a two-dimensional plane, there is a straight line l, whose analytical equation is f(x,y)=ax+by+c=0. There is a point P outside the straight line l, whose coordinate value is (xo ,y o ), satisfying ax o +by o +c≠0; Line l divides the plane into two half-plane regions A and B; suppose that region A and point P are on different sides of line l, and region B and point P are on the same side of line l, then the half-plane of region A and the half-plane of region B are expressed by the following inequalities respectively:

[0023]

[0024] Here, neither half plane contains its dividing line l, where “·” represents a product operation, which is the same in the following text; sgn(·) is a sign function, which is specifically defined as:

[0025]

[0026] The orbital core warning area range is determined by the following plane constraints:

[0027]

[0028] where w and h are the pixel width and height of the image; thus, the maximum connected domain of the core area of ​​the segmented track Further regularized into a region given by formula (8) In most cases, the region is a parallelogram. The two boundary lines together with the upper and lower boundaries of the image form a railway region with smooth boundaries. The minimum circumscribed parallelogram MCP with restricted boundaries;

[0029] Considering that the width of the adjacent warning area may not be completely consistent in different railway sections, the horizontal width of the adjacent warning area on both sides of the railway line is defined as λd, and λ is uniformly set to 1, that is, the adjacent warning area is set to have the same width as the core area; therefore, the boundary lines l1 and l2 of the track core warning area are shifted to the right and left by λd respectively, and the two boundary lines l3 and l4 of the railway adjacent warning area are obtained;

[0030] The equations of the straight line l3 and straight line l4 on both sides of the line near the warning area boundary are respectively obtained as follows:

[0031] f3(x,y)=a(xd / 2-λd)+by+c=0 (9)

[0032] f4(x,y)=a(x+d / 2+λd)+by+c=0 (10)

[0033] The core area of ​​the track The railway approach warning area on the right is represented by the following constraints on the two-dimensional plane:

[0034]

[0035] The core area of ​​the track The railway approach warning area on the left is represented by the following constraints on the two-dimensional plane:

[0036]

[0037] Therefore, the neighboring areas on both sides of the railway line It is expressed as:

[0038]

[0039] The set of all pixels in the entire image range is defined as Peripheral Areas of Concern Calculated by the following formula:

[0040]

[0041] The above gives a method for multi-level risk area division when dealing with a single track line;

[0042] UAV image risk area division: First, it is necessary to solve the multi-level division results corresponding to each track line, and two track lines are used for illustration; the area division result corresponding to track line k is set as The final image multi-level risk area is divided into:

[0043]

[0044] where ∪ k (·) represents the union operation of multiple sets, ∩ represents the intersection operation of two sets, Express The complement operation of

[0045] In fact, from the perspective of image pixels, risk area segmentation is also a process of assigning three different attributes to each pixel in the image; therefore, setting the image The regional attribute of the pixel point p in the image is t, where the values ​​of t are 0, 1 and 2, corresponding to the three attributes of the core warning area, the adjacent warning area and the peripheral attention area respectively; assuming that there are n track lines in the image, the regional attributes of the pixel point p in the regional division corresponding to each track line are t p,1 ,t p,2 ,…,t p,n , then the risk area division of the entire image can be completed by determining the final area attribute of pixel point p, and the formula equivalent to formula (15) is given as follows:

[0046]

[0047] It is divided into three stages: Stage 1 mainly completes the multi-level risk area division; Stage 2 conducts conventional foreign body intrusion detection under closed set conditions; Stage 3 conducts conventional and unconventional foreign body intrusion detection under open set conditions. The following stages are explained;

[0048] In the first stage, the unmanned aerial vehicle remote sensing railway image to be processed is first input into the semantic segmentation model encoder for image feature extraction, and the extracted features are input into the model decoder to predict and generate pixel-level high-level semantic features. After completing the track area segmentation, the track area segmentation mask is obtained.

[0049]

[0050] Iterate over track region masks Each connected domain in the and get the largest area, get the largest connected domain

[0051] Then, the track area is smoothed using the risk area method of boundary-constrained MCP to obtain the final track core area; then the adjacent area boundary division with configurable horizontal width is completed, and finally the risk area division mask map at the pixel level of the whole image is obtained.

[0052] In the second stage, the remote sensing railway image to be inspected by the UAV is input into the target detector backbone network for image multi-scale feature extraction, and detection heads of different scales are defined on the image multi-scale feature map to complete the bounding box position regression and predefined category prediction of foreign objects of different scales, and finally generate relatively dense bounding box predictions. There are M targets, and Represents the mth target foreign body detected by the closed set detection model:

[0053]

[0054] In the third stage, in addition to inputting the image into the detector backbone network to generate the multi-scale feature map of the image, the user is also required to input the conventional foreign object semantic vocabulary and the unconventional foreign object semantic vocabulary into the pre-trained text encoder to generate text word embedding, and perform multi-level cross-modal feature fusion of visual-linguistic features, then match the text semantics with the position area, and jointly generate the dense target foreign object bounding box prediction with the multi-scale bounding box obtained by regression. There are N targets, and the prediction is performed with Indicates the nth target foreign body detected by the closed set detection model:

[0055]

[0056] After completing the above three stages, the dense prediction results of the open and closed set foreign object detection bounding box are fused and processed together with the non-maximum suppression NMS operation to eliminate redundant prediction targets. It is assumed that there are Q foreign objects in the end, and obj q Indicates the nth target foreign body after non-maximum suppression:

[0057]

[0058] Finally, the target position attribute matching of the foreign object is performed on the obtained pixel-level risk area attribute mask map, and the foreign object intrusion detection work of multi-level risk matching is finally completed; for each foreign object target obj q , its center pixel point Loc(obj q )=(x q ,y q ) and pixel-level risk area division mask To match:

[0059]

[0060] Among them, Risk(obj q ) indicates the target foreign body obj q The risk level of the center is divided into three levels: high, medium and low according to the different areas where the center is located.

[0061] The advantages and positive effects of the present invention are:

[0062] (1) A boundary-constrained MCP multi-level risk zone division method is designed, which can achieve smooth boundary multi-level risk zone division of UAV railway inspection images, thereby achieving differentiated risk zone matching of railway intrusion foreign objects and realizing better railway line foreign object risk monitoring and control;

[0063] (2) An open detection method architecture for railway foreign object intrusion is designed. When faced with the problem of diverse and uncertain foreign object categories in real open-world railway scenarios, this method can not only effectively detect conventional predefined categories of foreign objects, but also utilize the zero-sample detection capability of large-scale pre-training models to deal with unconventional and undefined categories of foreign objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is the overall framework of open detection of foreign body intrusion by drones on railways

[0065] Figure 2 Schematic diagram of multi-level risk area division of UAV railway inspection images

[0066] Figure 3It is a method of expressing the area on the same side and opposite side of a point outside a straight line.

[0067] Figure 4 The core warning area of ​​the train track is determined

[0068] Figure 5 The scope of the train running track approaching warning area is determined

[0069] Figure 6 It is a multi-level risk area division method for multi-track lines

[0070] Figure 7 Detailed explanation of the FOCS-MID multi-risk level railway foreign body intrusion open detection method architecture

[0071] Figure 8 The experimental results of multi-level risk area division on FWA dataset

[0072] Fig. 9 The experimental results of multi-level risk area division on the MWB dataset

[0073] Fig.10 This is the partial result of foreign body target detection under closed set conditions

[0074] Fig.11 It is the partial result of foreign body target detection under open and closed set conditions

[0075] Fig.12 This is an example of visualizing multi-level railway intrusion detection results. DETAILED DESCRIPTION

[0076] The present invention will be further described in detail below with reference to the accompanying drawings and examples.

[0077] The present invention proposes a new multi-risk level region division method based on boundary-constrained MCP for UAV railway inspection images and a railway foreign object intrusion open detection method architecture FOCS-MID. The details are as follows:

[0078] (1) Multi-risk level regional division method

[0079] It is generally believed that the closer to the core area of ​​the track where the train is running, the higher the risk of intrusion of foreign objects to the train operation. Therefore, based on this idea, the multi-level risk area division of the drone's overhead railway image is considered to be mainly divided into three levels: core warning area, adjacent warning area and peripheral attention area, as shown in detail. Figure 2 shown.

[0080] Generally speaking, the core warning area should only focus on the maximum outer boundary area through which the train runs. Therefore, for the two different railway area images of single-track railway and double-track railway, the division of warning area, early warning area and attention area should be distinguished in theory. The division methods of risk areas for single-track and double-track railways are given in Figure 2 Shown in the left and right pictures.

[0081] Next, we introduce a multi-level risk area division method based on a deep semantic segmentation network. The track area parsed by the deep network usually has relatively fuzzy and rough boundaries, which is not conducive to the implementation of risk area division based on distance measurement. Therefore, the boundary of the final track core area needs to be smoothed first.

[0082] First, the maximum connected domain of the railway area segmentation mask initially predicted by the deep segmentation network is expressed as Where j represents the number (or index) of the largest connected domain in the image. Draw at equal intervals vertically or horizontally The connected domain is sampled by a horizontal or vertical line, corresponding to the two cases where the railway lines in the image are distributed horizontally or vertically. Take the railway lines as an example, where the railway lines are distributed vertically and horizontal sampling lines are drawn at equal intervals along the vertical direction. For each horizontal sampling line, The midpoint of the two points farthest apart that coincide with the horizontal line is represented by (x i ,y i ), and the distance between these two points is d i In this way, the center line of the track core area is uniformly sampled longitudinally. points, which are represented here as Then use this Perform simple linear regression on the points to get the equation of the railway area center line. The center point of the points is represented by (x o ,y o ),in:

[0083]

[0084] At the same time, the sampled Maximum horizontal distance Here its maximum value is defined as the pixel width of the track core area, denoted as d:

[0085]

[0086] This The equation of the center line l of the orbital core area obtained by performing simple linear regression on the two-dimensional plane is expressed as:

[0087] f(x,y)=ax+by+c=0 (3)

[0088] Furthermore, the straight line is translated along the positive direction of the x-axis by a distance of d / 2 to obtain the equation of the right boundary straight line l1 of the track core area on the two-dimensional plane, which is expressed as:

[0089] f1(x,y)=a(xd / 2)+by+c=0 (4)

[0090] Similarly, the equation of the left boundary line l2 of the orbital core region in the two-dimensional plane is obtained by translating the line l along the negative direction of the x-axis by a distance d / 2, which is expressed as:

[0091] f2(x,y)=a(x+d / 2)+by+c=0 (5)

[0092] Here we first give a method to represent the half-plane area on the same side and the opposite side of a point outside a straight line. Figure 3 As shown, there is a straight line l in a two-dimensional plane, and its analytical equation is f(x,y)=ax+by+c=0. There is a point P outside the straight line l, and its coordinate value is (x o ,y o ), satisfying ax o +by o +c≠0. Line l divides the plane into two half-plane regions, A and B. Assuming that region A and point P are on different sides of line l, and region B and point P are on the same side of line l, then the half-plane of region A and the half-plane of region B are expressed by the following inequalities:

[0093]

[0094] Here, neither half-plane contains its dividing line l, where “·” represents a product operation, and the same is true in the following text, where sgn(·) is a sign function, specifically defined as:

[0095]

[0096] Therefore, the scope of the orbital core warning area is determined by the following plane constraints, such as Figure 4 As shown:

[0097]

[0098] where w and h are the pixel width and height of the image. Thus, the maximum connected domain of the core area of ​​the segmented track Further regularized into a region given by formula (8) In most cases, the region is a parallelogram. The two boundary lines together with the upper and lower boundaries of the image form a railway region with smooth boundaries. The minimum circumscribed parallelogram (MCP) with restricted boundaries.

[0099] Taking into account that the width of the adjacent warning area may not be completely consistent in different railway sections, the horizontal width of the adjacent warning area on both sides of the railway line is defined as λd, where λ is an adjustable parameter used to indicate the ratio of the width of the adjacent warning area to the width of the track core area. In this patent, λ is uniformly set to 1, that is, the adjacent warning area and the core area are set to have the same width. Therefore, the boundary lines l1 and l2 of the track core warning area are translated to the right and left by a length of λd respectively, to obtain the two boundary lines l3 and l4 of the railway adjacent warning area. Similarly, the equations of the boundary lines l3 and l4 of the adjacent warning areas on both sides of the line are obtained as follows:

[0100] f3(x,y)=a(xd / 2-λd)+by+c=0 (9)

[0101] f4(x,y)=a(x+d / 2+λd)+by+c=0 (10)

[0102] Similarly, the core area of ​​the track The railway approach warning area on the right is represented by the following constraints on the two-dimensional plane:

[0103]

[0104] Similarly, the orbital core region The railway approach warning area on the left is represented by the following constraints on the two-dimensional plane:

[0105]

[0106] Therefore, the neighboring areas on both sides of the railway line It is expressed as:

[0107]

[0108] The set of all pixels in the entire image range is defined as Peripheral Areas of Concern It is calculated by the following formula:

[0109]

[0110] The above gives a method for multi-level risk area division when processing a single track line. Further, a method for risk area division of drone images with two or more track lines is given. First, it is necessary to solve the multi-level division results corresponding to each track line. Here, two track lines are used as an example. Assume that the area division result corresponding to track line k is The final image multi-level risk area is divided into:

[0111]

[0112] where ∪ k (·) represents the union operation of multiple sets, ∩ represents the intersection operation of two sets, Express The complement operation of . Figure 6 As shown in the figure, the multi-level risk area division process of double-track railway is given.

[0113] In fact, from the perspective of image pixels, risk area segmentation is also a process of assigning three different attributes to each pixel in the image. The regional attribute of the pixel point p in the image is t, where the values ​​of t are 0, 1 and 2, corresponding to the three attributes of the core warning area, the adjacent warning area and the peripheral attention area respectively. Assume that there are n rail lines in the image, and the regional attributes of the pixel point p in the regional division corresponding to each rail line are t p,1 ,t p,2 ,…,t p,n , then the risk area division of the entire image can be completed by determining the final area attribute of pixel point p, and the formula equivalent to formula (15) can be given as follows:

[0114]

[0115] (2) Foreign body intrusion open detection method architecture

[0116] This study constructs a risk-differentiated foreign body intrusion detection architecture FOCS-MID based on the fusion of open and closed sets. Figure 7 As shown in the figure, it is mainly divided into three stages: (1) Stage 1 mainly completes the multi-level risk area division; (2) Stage 2 conducts conventional category foreign body intrusion detection under closed set conditions; (3) Stage 3 conducts conventional and unconventional category foreign body intrusion detection under open set conditions. The following is an explanation of the stages.

[0117] In the first stage, the unmanned aerial vehicle remote sensing railway image to be processed is first input into the semantic segmentation model encoder for image feature extraction, and the extracted features are input into the model decoder to predict and generate pixel-level high-level semantic features. After completing the track area segmentation, the track area segmentation mask is obtained.

[0118]

[0119] Iterate over track region masks Each connected domain in the and get the largest area, get the largest connected domain Then, the track area is smoothed using the risk area method of boundary-constrained MCP to obtain the final track core area; then the adjacent area boundary division with configurable horizontal width is completed, and finally the risk area division mask map at the pixel level of the whole image is obtained.

[0120]

[0121] In the second stage, the UAV remote sensing railway image to be inspected is input into the target detector backbone network for image multi-scale feature extraction, and detection heads of different scales are defined on the image multi-scale feature map to complete the bounding box position regression and predefined category prediction of foreign objects of different scales, and finally generate relatively dense bounding box predictions. Assuming there are M targets, Represents the mth target foreign body detected by the closed set detection model:

[0122]

[0123] In stage three, in addition to inputting the image into the detector backbone network to generate the multi-scale feature map of the image, the user is also required to input the conventional foreign body semantic vocabulary (or category) and the unconventional foreign body semantic vocabulary (or category) into the pre-trained text encoder (generally a large-scale text-image pre-trained text encoder such as CLIP) to generate text word embedding, and perform multi-level cross-modal feature fusion on the visual-linguistic features, then match the text semantics with the position area, and jointly generate the dense target foreign body bounding box prediction with the multi-scale bounding box obtained by regression. Assuming there are N targets, use Indicates the nth target foreign body detected by the closed set detection model:

[0124]

[0125] After completing the above three stages, the dense prediction results of the open and closed set foreign object detection bounding box are fused and processed together with the non-maximum suppression (NMS) operation to eliminate redundant prediction targets. Assuming that there are Q foreign objects in the end, use obj q Indicates the nth target foreign body after non-maximum suppression:

[0126]

[0127] Finally, the target location attribute matching of the foreign object is performed on the obtained pixel-level risk area attribute mask map, and the foreign object intrusion detection work of multi-level risk matching is finally completed. q , its center pixel point Loc(obj q )=(x q ,y q ) and pixel-level risk area division mask To match:

[0128]

[0129] Among them, Risk(obj q ) indicates the target foreign body obj q The risk level of the center is divided into three levels: high, medium and low according to the different areas where the center is located.

[0130] In this example, the track line scene data was collected by drones in two different railway sections in Maanshan, and the corresponding railway / track area segmentation and foreign object detection datasets under closed conditions were constructed based on the data. The specific situation is shown in Table 1. The datasets composed of data collected by fixed-wing and multi-rotor drones are called FWA and MRB respectively. In FWA, the segmentation dataset contains a total of 1602 images, of which 1442 are used for training and 160 are used for testing. The dataset has two classes, representing railway areas and non-railway areas respectively; the closed-set detection dataset contains 1123 images, of which 1010 are used for training and 113 are used for testing. The dataset only involves one category, namely the pedestrian (person) category. Due to the high flight altitude and high flight speed of fixed-wing drones, affected by the weather environment, the collected images are of poor quality and limited clarity. At the same time, given that the current open target detection model is still limited in capability, the FWA dataset is not used for open-set foreign object intrusion recognition. In MRB, the segmentation dataset contains a total of 1255 images, of which 1004 are used for training and 251 are used for testing. The dataset has two classes, representing the track area and non-track area respectively; the closed set detection dataset contains 770 images, of which 693 are used for training and 77 are used for testing. The dataset also involves only one category, namely the pedestrian (person) category. At the same time, the images in this dataset are used for foreign object recognition in open scenes.

[0131] Table 1 Statistics of datasets constructed from images collected by drones

[0132]

[0133] Among them, due to the constraints of various existing conditions, the samples of foreign body intrusion image data that can be collected are relatively small, and there are very few foreign body categories in the collected images. Therefore, it is considered to set the conventional foreign body category as a single category in closed set detection, that is, {pedestrian: person}, and in open scene foreign body detection, set the unconventional foreign body category outside the conventional category provided to the model as {vehicle: vehicle, stone: stone}.

[0134] · Experimental results of multi-risk level regional division

[0135] Experiments were conducted on two segmentation datasets, FWA and MRB, to complete the multi-level risk area division of rail lines. The results are as follows: Figure 8 and Fig. 9 As shown. In the FWA dataset, the entire railway area is segmented as the core warning area. The experiment shows that the segmentation accuracy mIoU index value of the railway area for the trained deep segmentation network reaches 0.962. As can be seen from the figure, when the railway area presents different distribution angles and lateral widths, the proposed multi-level risk area division method based on boundary-restricted MCP can achieve efficient and accurate risk area division. When some images in the model segmentation results have rough edges due to occlusion by trees or houses, the proposed risk area division method can effectively deal with it. According to the distribution characteristics of the railway line image from the perspective of the drone, the through-type railway core area division from image boundary to boundary can be realized, and the pixel-level risk level determination of the entire image can be further completed. The division method of using the entire railway area as the core warning area of ​​the drone image in this experiment can be used in the foreign body intrusion detection task with the railway limit as the boundary.

[0136] At the same time, in the MRB dataset, the track area is segmented as the core warning area. The experimental results show that the segmentation accuracy of the track area is 0.974. More examples of images with rough edges or truncated sections due to occlusion by trees or vehicles are given, such as Fig. 9 The experimental results also give examples of region division of the track at various angles and different lateral widths. It can be seen that the region division results of the whole image show good robustness.

[0137] · Experimental results of foreign body intrusion detection based on open and closed set fusion

[0138] This paper conducts foreign object intrusion detection under closed conditions on the constructed FWA closed foreign object (single-category foreign object pedestrian) dataset, and secondly conducts foreign object intrusion detection (mainly pedestrians, vehicles, stones, etc.) in open scenes on the MRB dataset.

[0139] (1) Closed foreign body intrusion detection

[0140] This section gives the experimental results of potential pedestrian foreign object detection in the whole image using two object detection networks. The relevant quantitative indicators are shown in Table 2, which gives the AP related indicator values ​​on two closed foreign object intrusion detection datasets, FWA and MWB. 50 , R 50 and mAP 50 The index values ​​on the MRB dataset are as high as 0.949, 0.895 and 0.983, respectively, and the mAP value is 0.567. This fully demonstrates the feasibility of closed-set detection in the current dataset, and the high prediction accuracy also shows the necessity of closed-set detection for common foreign objects.

[0141] At the same time, network 2 was further used to test and evaluate on two data sets, and the results are shown in Table 2. It can be seen that the AP-related index values ​​of the model under the closed-set foreign body detection setting on the two data sets are relatively high, and the overall index values ​​of the network 1 model have been improved to a certain extent, but the accuracy improvement is not significant, which also shows that the current general target detection model can actually achieve good performance in closed-set foreign body intrusion detection, and the performance difference between different algorithms is limited.

[0142] exist Fig.10 Some closed-set foreign object detection image examples are given in Figure 1. It can be seen that many small-target pedestrians, including those riding electric bikes or motorcycles, can be detected in most cases, but there are still some cases of false detection or missed detection in a small number of images, namely FP and FN cases. As shown in the red box example in the figure, the FN case shows that pedestrians appearing in the railway core warning area cannot be detected, while the FP case mistakenly detects other unidentified objects as pedestrian foreign object targets.

[0143] Table 2 Foreign body intrusion detection AP index values

[0144]

[0145] (2) Open-collection foreign body intrusion detection

[0146] We further conducted foreign body intrusion detection under closed set and open set conditions on the MRB dataset, and provided some recognition results. Fig.11As shown in the second and fifth columns of the figure, it can be seen from all the examples that the foreign object detection model trained under closed set conditions can only detect and identify foreign object targets of the predefined category, namely <pedestrian: person>, and all have high prediction confidence and low false alarm rate. However, it will basically not respond to other categories outside the predefined categories, such as motorcycles and cars, and therefore it is bound to be unable to adapt to the requirements of foreign object recognition in real-world open scenes. At the same time, the model pre-trained on a large-scale dataset and not fine-tuned on any drone image dataset was also used to perform zero-sample open set detection. In addition to the closed set predefined category <pedestrian: person>, the provided foreign object category also adds <car: car, motorcycle: motorbike> to evaluate the zero-sample generalization ability of foreign object detection in open scenes. The relevant image examples are shown in the third and sixth columns of the figure. As can be seen from the examples, not only pedestrians in the image can be identified (as shown in some images in the third and sixth columns), but also cars and motorcycles on the road next to the railway can be identified (as shown in the first and second rows, and the third and sixth columns). This also shows that the open detector trained on a large-scale visual-language dataset has a strong zero-shot generalization ability and can maintain considerable recognition ability when specifying other categories outside the closed set predefined categories. However, it can also be seen that pedestrians on the railway in some images cannot be fully detected (as shown in the images of row 3, column 3, row 1, 2, and column 6). This also shows that relying solely on open-set detectors that have not been fine-tuned and trained on relevant datasets from the perspective of drones cannot achieve high confidence and high-precision detection of conventional foreign objects, but can serve as an effective supplement to foreign object detection under closed-set conditions. From the above results, it can be seen that

[0147] When the foreign object recognition results under closed set and open set conditions are fused and further NMS operations are performed, richer foreign object perception and recognition results of railway scenes from the perspective of drones will be obtained. Among them, closed set and open set detection require different confidence thresholds due to their different scene settings.

[0148] (3) Foreign body intrusion detection based on risk area matching

[0149] After completing the detection of potential foreign bodies by opening and closing set fusion on the whole image, the multi-level risk area division and all potential foreign bodies can be matched by spatial position and risk attribute results, thus completing the railway foreign body intrusion detection by risk area matching. Fig.12 As shown in the figure, a possible representation of foreign body detection results matching the risk area is given. The detected potential foreign bodies are represented by solid circles of different colors to assess the risk of foreign body intrusion in railway scenes. Red, yellow and green represent high, medium and low risks respectively. For the sake of image clarity, the peripheral areas of concern are not drawn with obvious colors.

[0150] In summary, it can be seen that the method of the present invention can efficiently realize foreign body intrusion detection in UAV railway inspection images, and expand the detection capability from predefined categories under closed set conditions to open scenes. The present invention provides a universal real-world open scene foreign body intrusion detection mode for UAV-based railway line inspection, which has broad application prospects for UAV-based railway inspection. This architecture can not only be used for foreign body intrusion detection in railway open scenes, but also in other open scenes. The core feature of this type of open scene is that the predefined target categories are relatively concentrated, but there are other uncertain target categories that cannot be fully defined in advance. The method of the present invention can effectively realize the effective detection of predefined categories and precision recognition, and strong generalization of undefined categories and zero samples.

Claims

1. A risk-graded UAV railway foreign body intrusion detection method, characterized in that It includes the following two parts: (1) Multi-level risk area division method In view of the fact that the risks caused by foreign objects in different areas of UAV railway inspection images are different, a risk area division method with a minimum circumscribed parallelogram with limited boundaries is proposed. Based on the track area segmentation results, a multi-level risk area division with smooth boundaries is achieved, and the image is divided into warning area, early warning area and attention area. (2) Foreign body intrusion open detection method architecture In view of the diverse and uncertain types of foreign objects in real open-world railway scenarios, a multi-risk level foreign object intrusion detection method architecture based on open and closed set fusion was proposed, which realized foreign object intrusion detection with multi-level risk area matching, and expanded the capability of railway foreign object intrusion detection to open scenarios.

2. A multi-level risk area division method according to claim 1, characterized in that The method is implemented in the following way; The drone-viewed railway images are divided into three levels of risk areas according to the distance from the core area of ​​the track: core warning area, adjacent warning area and peripheral attention area. The core warning area should only focus on the maximum outer boundary area where the train runs. The maximum connected domain of the segmentation mask of the core railway area predicted by the deep segmentation network is expressed as Where j represents the number of the largest connected domain in the image; the largest connected domain in the core area of ​​a single track parsed by the network Draw at equal intervals vertically or horizontally The connected domain is sampled by a horizontal or vertical line, corresponding to the two cases where the railway lines in the image are distributed horizontally and vertically; the railway lines are divided vertically and horizontal sampling lines are drawn at equal intervals along the vertical direction; for each horizontal sampling line, The midpoint of the two points farthest apart that coincide with the horizontal line is represented by (x i ,y i ), and the distance between these two points is d i ; In this way, the center line of the core area of ​​the track is evenly sampled longitudinally. points, which are represented here as Then use this Performing simple linear regression on the points will yield the equation for the center line of the railway area. The center point of the points is represented by (x o ,y o ),in: At the same time, the sample Maximum horizontal distance Here its maximum value is defined as the pixel width of the track core area, denoted as d: This The equation of the center line l of the orbital core area obtained by performing simple linear regression on the two-dimensional plane is expressed as: f(x,y)=ax+by+c=0 (3) The equation of the right boundary line l1 of the orbital core area in the two-dimensional plane is obtained by translating the line d / 2 along the positive direction of the x-axis, which is expressed as: f1(x,y)=a(xd / 2)+by+c=0 (4) The equation of the left boundary line l2 of the orbital core region on the two-dimensional plane is obtained by translating the line l along the negative direction of the x-axis by a distance of d / 2, which is expressed as: f2(x,y)=a(x+d / 2)+by+c=0 (5) Here we first give a method to represent the half-plane area on the same side and the opposite side of a point outside a straight line. In a two-dimensional plane, there is a straight line l, whose analytical equation is f(x,y)=ax+by+c=0. There is a point P outside the straight line l, whose coordinate value is (x o ,y o ), satisfying ax o +by o +c≠0; Line l divides the plane into two half-plane regions A and B; suppose that region A and point P are on different sides of line l, and region B and point P are on the same side of line l, then the half-plane of region A and the half-plane of region B are expressed by the following inequalities respectively: Here, neither half-plane contains its dividing line l, where "·" represents a product operation, which is the same in the following text; sgn(·) is a sign function, which is specifically defined as: The orbital core warning area range is determined by the following plane constraints: where w and h are the pixel width and height of the image; thus, the maximum connected domain of the core area of ​​the segmented track Further regularized into a region given by formula (8) In most cases, the region is a parallelogram. The two boundary lines together with the upper and lower boundaries of the image form a railway region with smooth boundaries. The minimum circumscribed parallelogram MCP with restricted boundaries; Considering that the width of the adjacent warning area may not be completely consistent in different railway sections, the horizontal width of the adjacent warning area on both sides of the railway line is defined as λd, and λ is uniformly set to 1, that is, the adjacent warning area is set to have the same width as the core area; therefore, the boundary lines l1 and l2 of the track core warning area are shifted to the right and left by λd respectively, and the two boundary lines l3 and l4 of the railway adjacent warning area are obtained; The equations of the straight line l3 and straight line l4 on both sides of the line near the warning area boundary are respectively obtained as follows: f3(x,y)=a(xd / 2-λd)+by+c=0 (9) f4(x,y)=a(x+d / 2+λd)+by+c=0 (10) The core area of ​​the track The railway approach warning area on the right is represented by the following constraints on the two-dimensional plane: The core area of ​​the track The railway approach warning area on the left is represented by the following constraints on the two-dimensional plane: Therefore, the neighboring areas on both sides of the railway line It is expressed as: The set of all pixels in the entire image range is defined as Peripheral Areas of Concern Calculated by the following formula: The above gives a method for multi-level risk area division when dealing with a single track line; UAV image risk area division: First, it is necessary to solve the multi-level division results corresponding to each track line, and two track lines are used for illustration; the area division result corresponding to track line k is set as The final image multi-level risk area is divided into: where ∪ k (·) represents the union operation of multiple sets, ∩ represents the intersection operation of two sets, Express The complement operation of In fact, from the perspective of image pixels, risk area segmentation is also a process of assigning three different attributes to each pixel in the image; therefore, setting the image The regional attribute of the pixel point p in the image is t, where the values ​​of t are 0, 1 and 2, corresponding to the three attributes of the core warning area, the adjacent warning area and the peripheral attention area respectively; assuming that there are n track lines in the image, the regional attributes of the pixel point p in the regional division corresponding to each track line are t p,1 ,t p,2 ,…,t p,n , then the risk area division of the entire image can be completed by determining the final area attribute of pixel point p, and the formula equivalent to formula (15) is given as follows:

3. According to claim 1, a railway foreign body intrusion detection method based on drone inspection images is characterized in that It is divided into three stages: Stage 1 mainly completes the multi-level risk area division; Stage 2 conducts conventional foreign body intrusion detection under closed set conditions; Stage 3 conducts conventional and unconventional foreign body intrusion detection under open set conditions. The following stages are explained; In the first stage, the unmanned aerial vehicle remote sensing railway image to be processed is first input into the semantic segmentation model encoder for image feature extraction, and the extracted features are input into the model decoder to predict and generate pixel-level high-level semantic features. After completing the track area segmentation, the track area segmentation mask is obtained. Iterate over track region masks Each connected domain in the and get the largest area, get the largest connected domain Then, the track area is smoothed using the risk area method of boundary-constrained MCP to obtain the final track core area; then the adjacent area boundary division with configurable horizontal width is completed, and finally the risk area division mask map at the pixel level of the whole image is obtained. In the second stage, the remote sensing railway image to be inspected by the UAV is input into the target detector backbone network for image multi-scale feature extraction, and detection heads of different scales are defined on the image multi-scale feature map to complete the bounding box position regression and predefined category prediction of foreign objects of different scales, and finally generate relatively dense bounding box predictions. There are M targets, and Represents the mth target foreign body detected by the closed set detection model: In the third stage, in addition to inputting the image into the detector backbone network to generate the multi-scale feature map of the image, the user is also required to input the conventional foreign object semantic vocabulary and the unconventional foreign object semantic vocabulary into the pre-trained text encoder to generate text word embedding, and perform multi-level cross-modal feature fusion of visual-linguistic features, then match the text semantics with the position area, and jointly generate the dense target foreign object bounding box prediction with the multi-scale bounding box obtained by regression. There are N targets, and the prediction is performed with Indicates the nth target foreign body detected by the closed set detection model: After completing the above three stages, the dense prediction results of the open and closed set foreign object detection bounding box are fused and processed together with the non-maximum suppression NMS operation to eliminate redundant prediction targets. It is assumed that there are Q foreign objects in the end, and obj q Indicates the nth target foreign body after non-maximum suppression: Finally, the target position attribute matching of the foreign object is performed on the obtained pixel-level risk area attribute mask map, and the foreign object intrusion detection work of multi-level risk matching is finally completed; for each foreign object target obj q , its center pixel point Loc(obj q )=(x q ,y q ) and pixel-level risk area division mask To match: Among them, Risk(obj q ) indicates the target foreign body obj q The risk level of the center is divided into three levels: high, medium and low according to the different areas where the center is located.

Images