Railway environmental hidden danger identification and risk level evaluation method based on unmanned aerial vehicle image

By introducing large convolution kernel and feature aggregation modules in the railway environmental hazard identification model, the problem of inaccurate risk level assessment in the existing technology is solved, accurate area calculation and risk assessment of railway environmental hazards are realized, and railway operations are improved Security.

CN119027837AActive Publication Date: 2024-11-26CHINA STATE RAILWAY GRP CO LTD +1

Patent Information

Application Number
CN202411086896.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-07-05
Filing Date
2024-08-08
Publication Date
2024-11-26
Estimated Expiration
2044-08-08

AI Technical Summary

Technical Problem

The existing deep learning-based railway environmental hazard identification methods cannot effectively estimate the area of ​​environmental hazards and are susceptible to complex environments, especially when the track area in the drone image is not completely captured, resulting in inaccurate risk level assessment.

Method used

Using a railway environmental hazard identification model based on a large convolution kernel, the risk distance between environmental hazard targets and track edges and lightweight float are calculated by introducing cross-stage local connection modules and feature aggregation and allocation of neck structures, extracting and fusion of multi-scale features. The actual area of ​​an object or contaminating object, and then evaluate the risk level of the railway environment.

Benefits of technology

The model processing speed and accuracy are improved, the area of ​​environmental risk targets is accurately calculated, the accuracy of risk assessment is enhanced, and the safety of railway operations is ensured.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027837A_ABST
    Figure CN119027837A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of railway environment and image processing, and discloses a railway environment hidden danger recognition and risk grade evaluation method based on an unmanned aerial vehicle image, comprising the following steps: collecting an unmanned aerial vehicle railway environment image; introducing a cross-stage local connection module fused with a large convolution kernel receptive field dynamic adjustment module into a large convolution kernel backbone network, constructing a railway environment hidden danger recognition model based on a large convolution kernel, inputting an unmanned aerial vehicle railway environment image for training, and obtaining the trained railway environment hidden danger recognition model based on the large convolution kernel; the environment hazard target is used for predicting the unmanned aerial vehicle environment image; according to the predicted environmental hazard target and the track mask, calculating the risk distance between the environmental hazard and the track edge and the actual area of the light floating object and the pollution object, and according to the calculated risk distance and the actual area, evaluating the railway environmental risk level; according to the method, the railway environment risk identification precision and speed are improved, and the risk level evaluation precision is also improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of railway environment and image processing, and in particular to a method for identifying railway environmental hidden dangers and risk level assessment based on unmanned aerial vehicle images. Background Art

[0002] In recent years, computer vision-based automatic detection technology has received extensive attention and has developed into two main directions in detecting potential environmental hazards and various facilities along railways: (1) Image processing. These methods, such as oriented gradient histogram feature extraction, scale-invariant feature transform, Hough transform, and local Weber-like contrast enhancement, have shown great potential in detecting environmental hazards along high-speed railways and other railway equipment. However, image processing-based methods analyze objects with predefined features, and idealized feature processing cannot meet the needs of railway detection. (2) Deep learning. Compared with image processing, convolutional neural networks used for end-to-end model training are less sensitive to environmental changes and can directly process complex features of original images, thus becoming a research hotspot for automatic detection of environmental hazards. At present, existing deep learning-based railway environmental hazard identification methods use standard track gauge as the actual reference and can estimate the risk distance, but cannot estimate the area of ​​environmental hazards, which may result in a large area of ​​light floating objects or pollutants being judged as lower risk by a single factor (risk distance). In addition, the assessment model is easily affected by complex environments, especially when the track area in the drone image is incompletely captured, which may directly lead to the inability to estimate the risk distance. Secondly, YOLARC uses a simple deep residual network and feature pyramid network structure to extract and fuse feature information, which may hinder the network from mining key discriminative features and cannot effectively transmit information. In addition, in terms of evaluation, YOLARC relies heavily on the track integrity in the image as a true map to estimate the risk distance, and when faced with the irregular edges of lightweight floating objects (such as plastic tarpaulins) or contaminated objects (such as garbage), it is difficult to assess the area of ​​these risk objects. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method for railway environmental hazard identification and risk level assessment based on drone images, which is used to solve the problems of low accuracy in environmental risk target identification and inaccurate risk level assessment.

[0004] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0005] The railway environmental hidden danger identification and risk level assessment method based on UAV images includes the following steps:

[0006] S1, using a drone to fly along the track to collect drone railway environment images;

[0007] S2, introduce the cross-stage local connection module integrated with the large convolution kernel receptive field dynamic adjustment module into the large convolution kernel backbone network, and build a railway environmental hazard identification model based on large convolution kernels;

[0008] S3, inputting the UAV railway environment image into the railway environmental hazard recognition model based on the large convolution kernel for training, and obtaining a trained railway environmental hazard recognition model based on the large convolution kernel, which is used to predict the environmental hazard target of the UAV environment image;

[0009] S4, reacquire the UAV railway environment image and input it into the trained railway environmental hazard identification model based on large convolution kernel to identify environmental hazard targets, and obtain the predicted environmental hazard targets and track masks;

[0010] S5. Calculate the risk distance between the environmental hazard target and the orbital edge and the actual area of ​​the light floating objects or pollutants in the environmental hazard target based on the predicted environmental hazard target and the orbital mask;

[0011] S6. Assess the risk level of the railway environment based on the risk distance between the environmental hazard target and the track edge and the actual area of ​​light floating objects or pollutants in the environmental hazard target.

[0012] Furthermore, the railway environmental hazard identification model based on large convolution kernels includes a large convolution kernel backbone network module, a feature aggregation and allocation neck structure module, and a prototype network plus prediction head module connected in sequence;

[0013] The large convolution kernel backbone network module includes a cross-stage local connection module integrated with a large convolution kernel receptive field dynamic adjustment module;

[0014] The cross-stage local connection module integrating the large convolution kernel receptive field dynamic adjustment module includes a first convolution module, a second convolution module and a residual connection module;

[0015] The residual connection module includes a large convolution kernel receptive field dynamic adjustment module;

[0016] The large convolution kernel receptive field dynamic adjustment module includes a large convolution kernel sequence decoupling module and a spatial selection attention mechanism module;

[0017] The large convolution kernel sequence decoupling module includes the first large convolution kernel and the second large convolution kernel;

[0018] The feature aggregation and allocation neck structure module includes a low-dimensional fusion module and a high-dimensional fusion module.

[0019] Furthermore, step S3 specifically includes:

[0020] S31, inputting the UAV railway environment image into the large convolution kernel backbone network module of the railway environment hidden danger identification model based on the large convolution kernel to perform multi-scale feature extraction, and obtaining feature maps of different scales;

[0021] S32, inputting feature maps of different scales into a feature aggregation and distribution neck structure module to fuse multi-scale features to obtain a multi-scale fused feature map;

[0022] S33. Input the multi-scale fusion feature map into the prototype network plus the prediction head module to perform position detection, classification and segmentation of the target of interest, and finally obtain a trained railway environmental hazard recognition model based on a large convolution kernel, which is used to predict environmental hazard targets in drone environmental images.

[0023] 4. The method for identifying railway environmental hazards and assessing risk levels based on drone images according to claim 3 is characterized in that step S31 specifically comprises:

[0024] S311, inputting the UAV railway environment image into the first convolution module of the large convolution kernel backbone network module for convolution processing and then splitting it along the channel to obtain a first branch feature map and a second branch feature map;

[0025] S312, inputting the first branch feature map into the residual connection module to perform a residual connection operation, thereby obtaining a first branch feature map of the residual connection module;

[0026] S313, splicing the first branch feature map and the second branch feature map of the residual connection module through the channel and inputting them into the second convolution module for convolution operation to obtain feature maps of different scales.

[0027] Further, step S312 specifically includes:

[0028] S3121, the first branch feature map is sequentially input into the first large convolution kernel and the second large convolution kernel of the large convolution kernel sequence decoupling module for feature extraction to obtain features of different receptive fields, namely:

[0029]

[0030] Among them, x0 and x both represent the first branch feature map, x i represents the characteristics of the receptive field of the i-th largest convolution kernel, represents the i-th largest convolution kernel, x i-1 Represents the characteristics of the receptive field of the i-1th large convolution kernel;

[0031] S3122, the features of different receptive fields are mixed by channels to obtain intermediate features, namely:

[0032]

[0033] Among them, M i represents the intermediate features of the i-th largest convolution kernel, represents the 1×1 convolution layer of the i-th largest convolution kernel;

[0034] S3123, concatenating the intermediate features to obtain a feature map of the large convolution kernel sequence decoupling module;

[0035] S3124, input the feature map of the large convolution kernel sequence decoupling module into the spatial selection attention mechanism module, and use average feature extraction and maximum feature extraction to obtain the spatial attention map of the channel dimension, that is:

[0036] SAM=[AFE(M),MFE(M)]

[0037] Among them, SAM represents the spatial attention map of the channel dimension, AFE represents the average feature extraction, MFE represents the maximum feature extraction, and M represents the feature map of the large convolution kernel sequence decoupling module;

[0038] S3125. Convert the spatial attention map of the channel dimension into several spatial attention maps, and use the sigmoid activation function to obtain the activated spatial selection mask, that is:

[0039] S = sigmoid(F 2→n (SAM))=[S1,…,S i ,…,S n ]

[0040] Where S represents the spatial selection mask of activation, and F 2→n (SAM) means that the spatial attention map of the channel dimension is converted into n spatial attention maps using a 1×1 convolutional layer, S1 represents the first spatial selection mask, S i represents the i-th spatial selection mask, S n represents the nth spatial selection mask;

[0041] S3126. The intermediate features are weighted according to the activated spatial selection mask and fused through the convolution layer to obtain the final attention feature, namely:

[0042]

[0043] in, represents the final attention feature, F 1×1 represents a 1×1 convolutional layer;

[0044] S3127, multiply the first branch feature map by the final attention feature to obtain the first branch feature map of the residual connection module, that is:

[0045]

[0046] Where y represents the first branch feature map of the residual connection module.

[0047] Further, step S32 specifically includes:

[0048] First, the feature maps of different scales are input into the feature aggregation and distribution neck structure module for convolution processing. Through the feature aggregation mechanism, the local features of different scales are aligned and spliced. Then, convolution processing is performed on the basis of the spliced ​​multi-scale features to obtain local features.

[0049] Secondly, the local features are aligned with the global features through the feature allocation mechanism to obtain the global features;

[0050] Then, the local features are input into the low-dimensional fusion module and fused with the high-dimensional fusion module, and then the convolution operation is performed to obtain the embedded local features. At the same time, the global features are divided into two parts. One part is processed by the sigmoid activation function after the convolution operation to obtain the global activation feature, and the other part is directly convolved to obtain the global feature to be fused.

[0051] Finally, the global activation feature is upsampled or downsampled and then multiplied with the embedded local feature, and added to the upsampled or downsampled global feature to be fused to obtain a multi-scale fused feature map.

[0052] Furthermore, the calculation formula of the multi-scale fusion feature map is:

[0053]

[0054] Among them, F fused represents the multi-scale fusion feature map, R represents the global feature size reshaping, that is, upsampling or downsampling, x act represents the global activation feature, x g_emb represents the global features to be fused, x l_emb represents the embedding of local features, Conv represents the convolution operation, and x g represents the global feature, x l Represents local features.

[0055] Furthermore, step S5 specifically includes:

[0056] S51. According to the predicted environmental hazard target and the track mask, the track edge is refined by using the convex hull fitting and contour fitting of image morphology, and the pixel-level distance between the track edge closest to the environmental hazard target and the environmental hazard target is calculated, that is:

[0057]

[0058] Among them, dpixel represents the pixel-level distance between the edge of the track closest to the environmental hazard target and the environmental hazard target, x1 and y1 represent the horizontal and vertical coordinates of the upper corner point of the edge of the track closest to the environmental hazard target, x2 and y2 represent the horizontal and vertical coordinates of the lower corner point of the edge of the track closest to the environmental hazard target, respectively. peh ,y peh The horizontal and vertical coordinates of the center point of the bounding box representing the predicted environmental hazard target;

[0059] S52. Calculate the key parameters per pixel required for mapping the UAV railway environment image to the real world, namely:

[0060]

[0061] Among them, δ pixel The key parameters required for mapping the UAV railway environment image to the real world are expressed in pixels, tan represents the tangent function, DFoV represents the diagonal field of view of the camera, f represents the focal length of the camera, W represents the width of the resolution of the UAV railway environment image, and H represents the height of the resolution of the UAV railway environment image;

[0062] S53, calculate the length of the unit pixel of the UAV railway environment image mapped to the real world, that is:

[0063]

[0064] Among them, Δ d H represents the length of the unit pixel of the UAV railway environment image mapped to the real world. UAV Indicates the flight altitude;

[0065] S54, calculate the area of ​​the UAV railway environment image mapped to the real world by unit pixel, that is:

[0066] Δ a =Δ d 2

[0067] Among them, Δ a Indicates the area of ​​the UAV railway environment image mapped from unit pixels to the real world;

[0068] S55. Calculate the risk distance between the environmental hazard target and the edge of the track and the actual area of ​​the light floating objects or pollutants in the environmental hazard target based on the pixel-level distance between the edge of the track closest to the environmental hazard target and the environmental hazard target and the length of the drone image unit pixel mapped to the real world, that is:

[0069]

[0070] Among them, d represents the risk distance between the environmental hazard target and the track edge, and a ′ represents the actual area of the light floating objects or polluted objects in the environmental hazard target, and n pixel represents the number of pixels in the predicted mask of the light floating objects or polluted objects in the environmental hazard target.

[0071] Furthermore, step S6 specifically includes:

[0072] According to the risk distance between the environmental hazard target and the track edge and the actual area of the light floating objects or polluted objects in the environmental hazard target, the environmental hazard target is divided into three risk levels, specifically:

[0073] If the environmental hazard target is identified as a building construction target containing temporary buildings, then:

[0074] When 0m < d ≤ 20m, the risk level is high;

[0075] When 20m < d ≤ 500m, the risk level is medium;

[0076] When 500m < d, the risk level is low;

[0077] If the environmental hazard target is identified as a plastic film target or a garbage accumulation target, then:

[0078] When 0m < d ≤ 20m, the risk level is high;

[0079] When 20m < d ≤ 500m and a ′ ≥ 10m 2 , the risk level is high;

[0080] When 20m < d ≤ 500m and a ′ < 10m 2 , the risk level is medium;

[0081] When 500m < d, the risk level is low.

[0082] The present invention has the following beneficial effects:

[0083] 1. The method for identifying railway environmental hidden dangers and evaluating risk levels based on UAV images proposed by the present invention improves the model processing speed and accuracy. At the same time, key UAV parameters are introduced to calculate the actual area of light floating objects or polluted objects in the environmental risk target, improving the calculation accuracy and risk assessment accuracy;

[0084] 2. The model is used to effectively extract useful environmental risk information from UAV railway environmental images, which helps railway maintenance and ensures railway operation safety. Description of the Drawings

[0085] Figure 1 This is a flow chart of the method for identifying railway environmental hazards and assessing risk levels based on drone images proposed in the present invention;

[0086] Figure 2 This is a schematic diagram of the principle of the railway environmental hazard identification model based on large convolution kernel;

[0087] Figure 3 This is a schematic diagram of the structure of the cross-stage local connection module that integrates the large convolution kernel receptive field dynamic adjustment module;

[0088] Figure 4 This is a schematic diagram of the structure of the large convolution kernel receptive field dynamic adjustment module;

[0089] Figure 5 Schematic diagram of the principle of feature aggregation and allocation of neck structure module;

[0090] Figure 6 Schematic diagram of the aggregation, allocation and injection process of low-dimensional features and high-dimensional features of the feature aggregation and allocation neck structure module;

[0091] Figure 7 Schematic diagram of the calculation principle of the risk distance between the environmental hazard target and the track edge and the actual area of ​​the light floating objects or pollutants in the environmental hazard target. DETAILED DESCRIPTION

[0092] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0093] like Figure 1 As shown, the railway environmental hidden danger identification and risk level assessment method based on UAV images includes the following steps S1-S6:

[0094] S1. Use a drone to fly along the track to collect drone railway environment images.

[0095] In this embodiment, the flight parameters of the drone are set and the drone is flown along the track to collect railway environment images during the flight of the drone, so that the obtained railway environment images can be used in the subsequent model training process.

[0096] S2. Introduce the cross-stage local connection module that integrates the large convolution kernel receptive field dynamic adjustment module into the large convolution kernel backbone network to build a railway environmental hazard identification model based on large convolution kernels.

[0097] Specifically, the railway environmental hazard identification model based on large convolution kernels includes a large convolution kernel backbone network module, a feature aggregation and allocation neck structure module, and a prototype network plus prediction head module connected in sequence; the large convolution kernel backbone network module includes a cross-stage local connection module integrated with a large convolution kernel receptive field dynamic adjustment module; the cross-stage local connection module integrated with the large convolution kernel receptive field dynamic adjustment module includes a first convolution module, a second convolution module, and a residual connection module; the residual connection module includes a large convolution kernel receptive field dynamic adjustment module; the large convolution kernel receptive field dynamic adjustment module includes a large convolution kernel sequence decoupling module and a spatial selection attention mechanism module; the large convolution kernel sequence decoupling module includes a first large convolution kernel and a second large convolution kernel; the feature aggregation and allocation neck structure module includes a low-dimensional fusion module and a high-dimensional fusion module.

[0098] like Figure 2 As shown, Figure 2 The railway environmental hazard identification model based on large convolution kernels includes a large convolution kernel backbone network module, a feature aggregation and allocation neck structure module, and a prototype network (ProtoNet) plus a prediction head module. The large convolution kernel backbone network is used for multi-scale feature extraction of UAV railway environment images, the feature aggregation and allocation neck structure is used for multi-scale feature fusion, the prototype network (ProtoNet) and the prototype network in the prediction head module are used to generate multiple feature prototype maps, and the prediction head is responsible for the location detection of risk environment targets in UAV railway environment images, target category classification, and generation and feature prototype Figure 1 The corresponding coefficients are obtained by using the coefficients and the prototype feature map for matrix calculation to obtain the target segmentation map.

[0099] The large convolution kernel backbone network includes a cross-stage local connection module integrated with a large convolution kernel receptive field dynamic adjustment module. The cross-stage local connection module integrated with a large convolution kernel receptive field dynamic adjustment module is used to reduce the redundant calculation of gradients during the back propagation process, thereby reducing the amount of model calculation while ensuring a high recognition rate. Figure 3 As shown, Figure 3 The schematic diagram of the structure of the cross-stage local connection module is integrated with the large convolution kernel receptive field dynamic adjustment module, including the first convolution module, the second convolution module and the residual connection module. The first convolution module and the second convolution module are used for convolution operation respectively, and the residual connection module is used for residual connection operation, so as to avoid gradient disappearance or gradient explosion; the residual connection module includes the large convolution kernel receptive field dynamic adjustment module; and the specific structure of the large convolution kernel receptive field dynamic adjustment module is as follows Figure 4As shown, it includes a large convolution kernel sequence decoupling module and a spatial selection attention mechanism module; the large convolution kernel sequence decoupling module includes a first large convolution kernel (large convolution kernel 1) and a second large convolution kernel (large convolution kernel 2), the first large convolution kernel and the second large convolution kernel are used for feature extraction, respectively, to obtain features of different receptive fields; the spatial selection attention mechanism module is used to establish a spatial selection attention mechanism, that is, to splice the features of multiple different receptive fields in the channel dimension, and use average feature extraction and maximum feature extraction and activation function processing to obtain a spatial attention map, and the spatial attention map is applied to the initial features of multiple different receptive fields to adapt to the UAV remote sensing targets with significant scale changes.

[0100] Secondly, the network parameter configuration of the large convolution kernel backbone network is shown in Table 1:

[0101] Table 1 Network parameter configuration table of large convolution kernel backbone network

[0102]

[0103] In Table 1, Block is a cross-stage local connection module that integrates the large convolution kernel receptive field dynamic adjustment module. The large convolution kernel backbone network also includes a CBS module. The CBS module is a module composed of convolution, batch normalization and SiLu activation function. Its function is to perform convolution operations, batch normalization operations and SiLu activation function operations on the input feature map, i.e., the UAV railway environment image. The (K, D) sequence parameters represent the convolution kernel size and expansion rate size of the first large convolution kernel (large convolution kernel 1) and the second large convolution kernel (large convolution kernel 2) respectively in the large convolution kernel sequence decoupling module.

[0104] The principle of feature aggregation and allocation of neck structure module is as follows Figure 5 As shown in the figure, the feature aggregation and allocation neck structure module performs convolution processing on the basis of extracting multi-scale features {B2, B3, B4, B5} from the large convolution kernel backbone network module. Through the feature aggregation mechanism, the local features of different scales are preferentially aligned in scale, and then convolution processing is performed on the basis of splicing multi-scale features to fully fuse the semantic information between local features, thereby obtaining the fused global features; the global features are adjusted through the feature allocation mechanism to align the local features to be fused. This module can not only mine the potential key features of the UAV railway environment image, but also enrich and strengthen the limited semantic information. Among them, {B2, B3, B4, B5} are extracted from the outputs of layers 3, 5, 7 and 9 in the large convolution kernel backbone network, and B2, B3, B4, B5 are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image size, respectively. At the same time, the aggregation, allocation and injection process of low-dimensional features and high-dimensional features of the feature aggregation and allocation neck structure module are shown in the figure. Figure 6 As shown:

[0105] First, {B2, B3, B4, B5} is used as the input of the low-dimensional feature aggregation mechanism. In the low-dimensional feature aggregation mechanism, B5 is downsampled and {B2, B3} is upsampled so that the feature size of {B2, B3, B5} is aligned with B4. Then, the aligned feature maps are concatenated to obtain the low-dimensional aggregate feature F align In the low-dimensional feature allocation mechanism, low-dimensional aggregate features F are processed by Block align , in order to exchange enough information to generate global features. Subsequently, the global features are divided into two global features (x g_B3 and x g_B4 ) for the subsequent local feature information injection. In order to achieve more efficient information exchange between global and local information, attention operation is also used in the injection process, specifically: local features are calculated through a 1×1 convolution layer (Conv) to obtain embedded local features, and two parallel branches with different convolutions are used for global features to generate global activation features processed by the Sigmoid function and global features to be fused, and the global activation features will be used to calculate the matrix multiplication of the corresponding positions with the embedded local features; the result of the global activation features and the embedded local features is matrix added with the global features to be fused to obtain the fused features. Then, in order to align the size between local features and global features, upsampling or downsampling is used in the global activation features and the global features to be fused. Similarly, the aggregation, distribution and injection process of high-dimensional features is the same as that of low-dimensional features. The difference is that the high-level information processing fuses the features {C3, C4, C5} from the low-level to obtain the high-dimensional aggregate feature F ′ align , and the high-dimensional aggregation feature F ′ align Split along the channel dimension to get x g_C4 With x g_C5 , where C5=B5, C3 is the sum of B3 and x g_B3 Information injection, C4 is B4 and x g_B4 information injection; at the same time, {C3, C4} are downsampled to adjust the size of the feature map to align with C5; in addition, in the high-dimensional feature injection process, Figure 5 It can also be seen that the final high-dimensional fusion features P4 and P5 can be respectively obtained through {x g_C4 ,C4} and {x g_C5 ,C5} is injected, and P3=C3.

[0106] In addition, in order to enhance the local cross-layer information flow, a low-dimensional fusion module and a high-dimensional fusion module are introduced in the feature aggregation and distribution neck structure module, that is, the local features are fused with adjacent features before feature injection, and low-dimensional fusion or high-dimensional fusion modules are used for feature injection. Figure 6 Medium and low dimensional fusion and high dimensional fusion parts, parameters Refers to the local features of the current dimension, Refers to the local features of one dimension lower than the current dimension, Refers to local features with a dimension higher than the current dimension, such as Figure 5 During the B4 injection phase, For B4, For B3, It is B5.

[0107] S3. Input the UAV railway environment image into the railway environmental hazard identification model based on large convolution kernel for training, and obtain the trained railway environmental hazard identification model based on large convolution kernel, which is used to predict the environmental hazard targets of the UAV environment image.

[0108] Specifically, step S3 specifically includes S31-S33:

[0109] S31. Input the UAV railway environment image into the large convolution kernel backbone network module of the railway environment hazard identification model based on large convolution kernels to perform multi-scale feature extraction to obtain feature maps of different scales.

[0110] Specifically, step S31 includes S311-S313:

[0111] S311, input the UAV railway environment image into the first convolution module of the large convolution kernel backbone network module for convolution processing, and then split it along the channel to obtain the first branch feature map and the second branch feature map.

[0112] In this embodiment, the first convolution module is a convolution module composed of a convolution kernel, a batch normalization operation and a SiLu activation function.

[0113] S312. Input the first branch feature map into the residual connection module to perform a residual connection operation to obtain the first branch feature map of the residual connection module.

[0114] Specifically, step S312 specifically includes S3121-S3127:

[0115] S3121, the first branch feature map is sequentially input into the first large convolution kernel and the second large convolution kernel of the large convolution kernel sequence decoupling module for feature extraction to obtain features of different receptive fields, namely:

[0116]

[0117] Among them, x0 and x both represent the first branch feature map, x i represents the characteristics of the receptive field of the i-th largest convolution kernel, represents the i-th largest convolution kernel, x i-1 Represents the characteristics of the receptive field of the i-1th large convolution kernel.

[0118] S3122, the features of different receptive fields are mixed by channels to obtain intermediate features, namely:

[0119]

[0120] Among them, M i represents the intermediate features of the i-th largest convolution kernel, Represents a 1×1 convolutional layer with the i-th largest convolution kernel.

[0121] S3123. Concatenate the intermediate features to obtain the feature map of the large convolution kernel sequence decoupling module.

[0122] S3124, input the feature map of the large convolution kernel sequence decoupling module into the spatial selection attention mechanism module, and use average feature extraction and maximum feature extraction to obtain the spatial attention map of the channel dimension, that is:

[0123] SAM=[AFE(M),MFE(M)]

[0124] Among them, SAM represents the spatial attention map of the channel dimension, AFE represents average feature extraction, MFE represents maximum feature extraction, and M represents the feature map of the large convolution kernel sequence decoupling module.

[0125] S3125. Convert the spatial attention map of the channel dimension into several spatial attention maps, and use the sigmoid activation function to obtain the activated spatial selection mask, that is:

[0126] S = sigmoid(F 2→n (SAM))=[S1,…,S i ,…,S n ]

[0127] Where S represents the spatial selection mask of activation, F 2→n (SAM) means that the spatial attention map of the channel dimension is converted into n spatial attention maps using a 1×1 convolutional layer, S1 represents the first spatial selection mask, S i represents the i-th spatial selection mask, S n Represents the nth spatial selection mask.

[0128] S3126. The intermediate features are weighted according to the activated spatial selection mask and fused through the convolutional layer to obtain the final attention feature, namely:

[0129]

[0130] in, represents the final attention feature, F 1×1 Represents a 1×1 convolutional layer.

[0131] S3127, multiply the first branch feature map by the final attention feature to obtain the first branch feature map of the residual connection module, that is:

[0132]

[0133] Where y represents the first branch feature map of the residual connection module.

[0134] S313, splicing the first branch feature map and the second branch feature map of the residual connection module through the channel and inputting them into the second convolution module for convolution operation to obtain feature maps of different scales.

[0135] In this embodiment, the second convolution module is a convolution module composed of a convolution kernel, a batch normalization operation and a SiLu activation function.

[0136] S32. Input feature maps of different scales into the feature aggregation and distribution neck structure module to fuse multi-scale features and obtain a multi-scale fused feature map.

[0137] Specifically, step S32 includes:

[0138] Firstly, the feature maps of different scales are input into the feature aggregation and distribution neck structure module for convolution processing. Through the feature aggregation mechanism, the local features of different scales are aligned and spliced. Then, convolution processing is performed on the basis of the spliced ​​multi-scale features to obtain the local features.

[0139] Secondly, the local features are aligned with the global features through the feature allocation mechanism to obtain the global features.

[0140] Then, the local features are input into the low-dimensional fusion module and fused with the high-dimensional fusion module, and then the convolution operation is performed to obtain the embedded local features. At the same time, the global features are divided into two parts. One part is processed by the sigmoid activation function after the convolution operation to obtain the global activation feature, and the other part is directly convolved to obtain the global feature to be fused.

[0141] Finally, the global activation feature is upsampled or downsampled and then multiplied with the embedded local feature, and added to the upsampled or downsampled global feature to be fused to obtain a multi-scale fused feature map.

[0142] Specifically, the calculation formula of the multi-scale fusion feature map is:

[0143]

[0144] Among them, F fused represents the multi-scale fusion feature map, R represents the global feature size reshaping, x act represents the global activation feature, x a_emb represents the global features to be fused, x l_emb represents the embedding of local features, Conv represents the convolution operation, and x g represents the global feature, x l Represents local features.

[0145] S33. Input the multi-scale fusion feature map into the prototype network plus the prediction head module to perform position detection, classification and segmentation of the target of interest, and finally obtain a trained railway environmental hazard recognition model based on a large convolution kernel, which is used to predict environmental hazard targets in drone environmental images.

[0146] S4. Reacquire the UAV railway environment image and input it into the trained railway environmental hazard identification model based on large convolution kernel to identify environmental hazard targets, and obtain the predicted environmental hazard targets and track masks.

[0147] In this embodiment, the UAV railway environment image is input into the trained railway environment hazard identification model based on a large convolution kernel, which can predict the position of various targets in the image, that is, the coordinate information, the category, that is, the category to which they belong, and the polygonal mask formed by the edge of the target, that is, the coordinate information of the segmentation point of the target.

[0148] S5. Based on the predicted environmental hazard target and the orbital mask, calculate the risk distance between the environmental hazard target and the orbital edge and the actual area of ​​the light floating objects or pollutants in the environmental hazard target.

[0149] like Figure 7 As shown, Figure 7 This is a schematic diagram of the calculation principle of the risk distance between the environmental hazard target and the track edge and the actual area of ​​the light floating objects or pollutants in the environmental hazard target. Figure 7It includes the results of track semantic segmentation extraction, segmentation refinement processing and external polygon fitting process, where hidden dangers 1-3 refer to the predicted environmental hazard targets respectively, the white area is the predicted track mask, and the distance and area estimation from pixel level to reality level is performed according to the external polygon, specifically: based on the pixel-level segmentation results of the railway track area, the predicted environmental hazard target (i.e., the predicted bounding box) and the track mask are obtained, and the track mask is subjected to image morphological processing, circumscribed polygon fitting and other image processing techniques to eliminate the edge features of the track edge that affect the distance assessment, such as jagged and uneven edge, to obtain a smoother track edge; that is, the convex hull fitting and contour fitting of image morphology are used to refine the track edge, so as to extract the minimum boundary polygon of the track area, specifically in this embodiment, the minimum circumscribed quadrilateral of the track area is extracted, and the reason is that the track area is usually a quadrilateral, so the track edge can be fitted using the minimum circumscribed quadrilateral. After obtaining the track edge, the risk distance between the environmental hazard target and the track edge and the actual area of ​​the light floating objects or pollutants in the environmental hazard target are calculated, so as to perform risk level assessment in subsequent steps.

[0150] Specifically, step S5 specifically includes S51-S55:

[0151] S51. According to the predicted environmental hazard target and the track mask, the track edge is refined by using the convex hull fitting and contour fitting of image morphology, and the pixel-level distance between the track edge closest to the environmental hazard target and the environmental hazard target is calculated, that is:

[0152]

[0153] Among them, d pixel represents the pixel-level distance between the edge of the track closest to the environmental hazard target and the environmental hazard target, x1 and y1 represent the horizontal and vertical coordinates of the upper corner point of the edge of the track closest to the environmental hazard target, x2 and y2 represent the horizontal and vertical coordinates of the lower corner point of the edge of the track closest to the environmental hazard target, respectively. peh ,y peh Represents the horizontal and vertical coordinates of the center point of the bounding box of the predicted environmental hazard target.

[0154] S52. Calculate the key parameters per pixel required for mapping the UAV railway environment image to the real world, namely:

[0155]

[0156] Among them, δ pixelThe key parameter unit pixel required for mapping the UAV railway environment image to the real world is represented, tan represents the tangent function, DFoV represents the diagonal field of view of the camera, f represents the focal length of the camera, W represents the width of the resolution of the UAV railway environment image, and H represents the height of the resolution of the UAV railway environment image.

[0157] S53, calculate the length of the unit pixel of the UAV railway environment image mapped to the real world, that is:

[0158]

[0159] Among them, Δ d H represents the length of the unit pixel of the UAV railway environment image mapped to the real world. UAV Indicates the flight altitude.

[0160] S54, calculate the area of ​​the UAV railway environment image mapped to the real world by unit pixel, that is:

[0161] Δ a =Δ d 2

[0162] Among them, Δ a It represents the area of ​​the real world mapped from the unit pixel of the UAV railway environment image.

[0163] S55. Calculate the risk distance between the environmental hazard target and the edge of the track and the actual area of ​​the light floating objects or pollutants in the environmental hazard target based on the pixel-level distance between the edge of the track closest to the environmental hazard target and the environmental hazard target and the length of the drone image unit pixel mapped to the real world, that is:

[0164]

[0165] Where d represents the risk distance between the environmental hazard target and the track edge, a ′ Indicates the actual area of ​​light floating objects or pollutants in the environmental hazard target, n pixel The number of pixels in the predicted mask that represent lightweight floating objects or contaminants in the environmental hazard target.

[0166] S6. Assess the risk level of the railway environment based on the risk distance between the environmental hazard target and the track edge and the actual area of ​​light floating objects or pollutants in the environmental hazard target.

[0167] Specifically, step S6 includes:

[0168] According to the risk distance between the environmental hazard target and the edge of the track and the actual area of ​​light floating objects or pollutants in the environmental hazard target, the environmental hazard target is divided into three risk levels, specifically:

[0169] If the environmental hazard target is identified as a building construction target including temporary buildings, then there is:

[0170] When 0m < d ≤ 20m, the risk level is high.

[0171] When 20m < d ≤ 500m, the risk level is medium.

[0172] When 500m < d, the risk level is low.

[0173] If the environmental hazard target is identified as a plastic film target or a garbage accumulation target, then there is:

[0174] When 0m < d ≤ 20m, the risk level is high.

[0175] When 20m < d ≤ 500m and a ′ ≥ 10m 2 , the risk level is high.

[0176] When 20m < d ≤ 500m and a ′ < 10m 2 , the risk level is medium.

[0177] When 500m < d, the risk level is low.

[0178] The method for identifying potential hazards and assessing risk levels in railway environments based on UAV images proposed by the present invention first uses a UAV to fly along the track to collect UAV railway environment images. Subsequently, a large convolutional kernel backbone network is used to extract multi-scale feature maps of the UAV railway environment images. On this basis, a feature aggregation and distribution neck structure is used to fully fuse the information of the feature maps to enhance the limited semantic information of the UAV railway environment images. Subsequently, under the processing of the prototype network and the prediction head, the position detection, classification, and segmentation of the target of interest in the UAV railway environment images are realized; finally, a risk level assessment is carried out on the detected potential hazards, that is, environmental hazard targets. This method can achieve a processing speed of 45.0fps, an average absolute accuracy of 95.8% for the bounding boxes of target predictions, and an average absolute accuracy of 91.7% for the bounding box polygon masks of target predictions in all detection categories. At the same time, key UAV parameters are utilized and integrated into the calculation of the risk distance between the environmental hazard target and the track edge, as well as the actual area of the light floating objects or polluted objects in the environmental hazard target, for automatic risk assessment, which can effectively convert the UAV railway environment images into useful information to assist railway maintenance, thereby ensuring the safety of railway operations.

[0179] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0180] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A railway environmental hazard identification and risk level assessment method based on drone images, characterized in that: The following steps are involved: S1, using a drone to fly along the track to collect drone railway environment images; S2, introduce the cross-stage local connection module integrated with the large convolution kernel receptive field dynamic adjustment module into the large convolution kernel backbone network, and build a railway environmental hazard identification model based on large convolution kernels; S3, inputting the UAV railway environment image into the railway environmental hazard recognition model based on the large convolution kernel for training, and obtaining a trained railway environmental hazard recognition model based on the large convolution kernel, which is used to predict the environmental hazard target of the UAV environment image; S4, reacquire the UAV railway environment image and input it into the trained railway environmental hazard identification model based on large convolution kernel to identify environmental hazard targets, and obtain the predicted environmental hazard targets and track masks; S5. Calculate the risk distance between the environmental hazard target and the orbital edge and the actual area of ​​the light floating objects or pollutants in the environmental hazard target based on the predicted environmental hazard target and the orbital mask; S6. Assess the risk level of the railway environment based on the risk distance between the environmental hazard target and the track edge and the actual area of ​​light floating objects or pollutants in the environmental hazard target.

2. The method for identifying railway environmental hazards and assessing risk levels based on drone images according to claim 1 is characterized in that: The railway environmental hazard identification model based on large convolution kernels includes a large convolution kernel backbone network module, a feature aggregation and allocation neck structure module, and a prototype network plus prediction head module connected in sequence; The large convolution kernel backbone network module includes a cross-stage local connection module integrated with a large convolution kernel receptive field dynamic adjustment module; The cross-stage local connection module integrating the large convolution kernel receptive field dynamic adjustment module includes a first convolution module, a second convolution module and a residual connection module; The residual connection module includes a large convolution kernel receptive field dynamic adjustment module; The large convolution kernel receptive field dynamic adjustment module includes a large convolution kernel sequence decoupling module and a spatial selection attention mechanism module; The large convolution kernel sequence decoupling module includes the first large convolution kernel and the second large convolution kernel; The feature aggregation and allocation neck structure module includes a low-dimensional fusion module and a high-dimensional fusion module.

3. The method for identifying railway environmental hazards and assessing risk levels based on drone images according to claim 2 is characterized in that: Step S3 specifically includes: S31, inputting the UAV railway environment image into the large convolution kernel backbone network module of the railway environment hidden danger identification model based on the large convolution kernel to perform multi-scale feature extraction, and obtaining feature maps of different scales; S32, inputting feature maps of different scales into a feature aggregation and distribution neck structure module to fuse multi-scale features to obtain a multi-scale fused feature map; S33. Input the multi-scale fusion feature map into the prototype network plus the prediction head module to perform position detection, classification and segmentation of the target of interest, and finally obtain a trained railway environmental hazard recognition model based on a large convolution kernel, which is used to predict environmental hazard targets in drone environmental images.

4. The method for identifying railway environmental hazards and assessing risk levels based on drone images according to claim 3 is characterized in that: Step S31 specifically includes: S311, inputting the UAV railway environment image into the first convolution module of the large convolution kernel backbone network module for convolution processing and then splitting it along the channel to obtain a first branch feature map and a second branch feature map; S312, inputting the first branch feature map into the residual connection module to perform a residual connection operation, thereby obtaining a first branch feature map of the residual connection module; S313, splicing the first branch feature map and the second branch feature map of the residual connection module through the channel and inputting them into the second convolution module for convolution operation to obtain feature maps of different scales.

5. The method for identifying railway environmental hazards and assessing risk levels based on drone images according to claim 4 is characterized in that: Step S312 specifically includes: S3121, the first branch feature map is sequentially input into the first large convolution kernel and the second large convolution kernel of the large convolution kernel sequence decoupling module for feature extraction to obtain features of different receptive fields, namely: Among them, x0 and x both represent the first branch feature map, x i represents the characteristics of the receptive field of the i-th largest convolution kernel, represents the i-th largest convolution kernel, x i-1 Represents the characteristics of the receptive field of the i-1th large convolution kernel; S3122, the features of different receptive fields are mixed by channels to obtain intermediate features, namely: Among them, M i represents the intermediate features of the i-th largest convolution kernel, represents the 1×1 convolution layer of the i-th largest convolution kernel; S3123, concatenating the intermediate features to obtain a feature map of the large convolution kernel sequence decoupling module; S3124, input the feature map of the large convolution kernel sequence decoupling module into the spatial selection attention mechanism module, and use average feature extraction and maximum feature extraction to obtain the spatial attention map of the channel dimension, that is: SAM=[AFE(M),MFE(M)] Among them, SAM represents the spatial attention map of the channel dimension, AFE represents the average feature extraction, MFE represents the maximum feature extraction, and M represents the feature map of the large convolution kernel sequence decoupling module; S3125. Convert the spatial attention map of the channel dimension into several spatial attention maps, and use the sigmoid activation function to obtain the activated spatial selection mask, that is: S=sigmoid(F 2→n (SAM00=[S1,…,S i ,…,S n ] Where S represents the spatial selection mask of activation, F 2→n (SAM) means that the spatial attention map of the channel dimension is converted into n spatial attention maps using a 1×1 convolutional layer, S1 represents the first spatial selection mask, S i represents the i-th spatial selection mask, S n represents the nth spatial selection mask; S3126. The intermediate features are weighted according to the activated spatial selection mask and fused through the convolution layer to obtain the final attention feature, namely: in, represents the final attention feature, F 1×1 represents a 1×1 convolutional layer; S3127, multiply the first branch feature map by the final attention feature to obtain the first branch feature map of the residual connection module, that is: Where y represents the first branch feature map of the residual connection module.

6. The method for identifying railway environmental hazards and assessing risk levels based on drone images according to claim 5 is characterized in that: Step S32 specifically includes: First, the feature maps of different scales are input into the feature aggregation and distribution neck structure module for convolution processing. Through the feature aggregation mechanism, the local features of different scales are aligned and spliced. Then, convolution processing is performed on the basis of the spliced ​​multi-scale features to obtain local features. Secondly, the local features are aligned with the global features through the feature allocation mechanism to obtain the global features; Then, the local features are input into the low-dimensional fusion module and fused with the high-dimensional fusion module, and then the convolution operation is performed to obtain the embedded local features. At the same time, the global features are divided into two parts. One part is processed by the sigmoid activation function after the convolution operation to obtain the global activation feature, and the other part is directly convolved to obtain the global feature to be fused. Finally, the global activation feature is upsampled or downsampled and then multiplied with the embedded local feature, and added to the upsampled or downsampled global feature to be fused to obtain a multi-scale fused feature map.

7. The method for identifying railway environmental hazards and assessing risk levels based on drone images according to claim 6 is characterized in that: The calculation formula of the multi-scale fusion feature map is: Among them, F fused represents the multi-scale fusion feature map, R represents the global feature size reshaping, that is, upsampling or downsampling, x act represents the global activation feature, x g_emb represents the global features to be fused, x l_emb represents the embedding of local features, Conv represents the convolution operation, and x g represents the global feature, x l Represents local features.

8. The method for identifying railway environmental hazards and assessing risk levels based on drone images according to claim 7 is characterized in that: Step S5 specifically includes: S51. According to the predicted environmental hazard target and the track mask, the track edge is refined by using the convex hull fitting and contour fitting of image morphology, and the pixel-level distance between the track edge closest to the environmental hazard target and the environmental hazard target is calculated, that is: Among them, d pixel represents the pixel-level distance between the edge of the track closest to the environmental hazard target and the environmental hazard target, x1 and y1 represent the horizontal and vertical coordinates of the upper corner point of the edge of the track closest to the environmental hazard target, x2 and y2 represent the horizontal and vertical coordinates of the lower corner point of the edge of the track closest to the environmental hazard target, respectively. peh ,y peh The horizontal and vertical coordinates of the center point of the bounding box representing the predicted environmental hazard target; S52. Calculate the key parameters per pixel required for mapping the UAV railway environment image to the real world, namely: Among them, δ pixel The key parameters required for mapping the UAV railway environment image to the real world are expressed in pixels, tan represents the tangent function, DFoV represents the diagonal field of view of the camera, f represents the focal length of the camera, W represents the width of the resolution of the UAV railway environment image, and H represents the height of the resolution of the UAV railway environment image; S53. Calculate the length of a single pixel in the UAV railway environment image mapped to the real world, i.e.: Among them, Δ d H represents the length of the unit pixel of the UAV railway environment image mapped to the real world. UAV Indicates the flight altitude; S54. Calculate the area of a single pixel in the UAV railway environment image mapped to the real world, i.e.: D a =D d 2 Among them, Δ a Indicates the area of ​​the UAV railway environment image mapped from unit pixels to the real world; S55. According to the pixel-level distance between the track edge closest to the environmental hazard target and the environmental hazard target and the length of a single pixel in the UAV image mapped to the real world, calculate the risk distance between the environmental hazard target and the track edge and the actual area of the light floating objects or polluted objects in the environmental hazard target, i.e.: Where d represents the risk distance between the environmental hazard target and the track edge, a ′ Indicates the actual area of ​​light floating objects or pollutants in the environmental hazard target, n pixel The number of pixels in the predicted mask that represent lightweight floating objects or contaminants in the environmental hazard target.

9. The method for identifying railway environmental hazards and assessing risk levels based on drone images according to claim 8 is characterized in that: Step S6 specifically includes: According to the risk distance between the environmental hazard target and the track edge and the actual area of the light floating objects or polluted objects in the environmental hazard target, classify the environmental hazard target into three risk levels, specifically: If the environmental hazard target is identified as a building target containing temporary buildings, then: When 0m < d ≤ 20m, the risk level is high; When 20m < d ≤ 500m, the risk level is medium; When 500m < d, the risk level is low; If the environmental hazard target is identified as a plastic film target or a garbage accumulation target, then: When 0m < d ≤ 20m, the risk level is high; When 20m < d ≤ 500m and a ′ ≥ 10m 2 , the risk level is high; When 20m < d ≤ 500m and a ′ <10m 2 , the risk level is medium; When 500m < d, the risk level is low.

Citation Information

Patent Citations

  • Method, device and equipment for evaluating hidden dangers of hanging objects outside transformer substation

    CN116523895A

  • Autonomous train distance measurement optimization method and system based on parallel elements

    CN118154697A

  • Method for detecting image target in smart home environment

    WO2021244079A1

  • Dangerous object information management server capable of estimating actual size of dangerous object on road in conjunction with information collection terminal mounted to vehicle, and method for operating same

    WO2023128113A1

Cited By

  • Power transmission channel tree obstacle hidden danger detection model training method and detection method

    CN120032182A