Railway environment hidden danger identification and risk grade evaluation method based on unmanned aerial vehicle image

Abstract

The present application relates to the field of railway environment and image processing technology, and discloses a railway environment hidden danger identification and risk level evaluation method based on unmanned aerial vehicle image, comprising the following steps: collecting unmanned aerial vehicle railway environment image; introducing a cross-stage local connection module of a fusion large convolution kernel receptive field dynamic adjustment module into a large convolution kernel backbone network, constructing a railway environment hidden danger identification model based on a large convolution kernel, and inputting the unmanned aerial vehicle railway environment image for training to obtain a trained railway environment hidden danger identification model based on a large convolution kernel, which is used for predicting the environment hazard target of the unmanned aerial vehicle environment image; according to the predicted environment hazard target and track mask, the risk distance between the environment hazard and the track edge and the actual area of light floating object and pollutant object are calculated, and the railway environment risk level is evaluated according to the calculated risk distance and actual area; the method improves the railway environment risk identification precision and speed, and also improves the risk level evaluation precision.

Description

Technical Field

[0001] This invention relates to the field of railway environment and image processing technology, specifically to a method for identifying railway environmental hazards and assessing risk levels based on UAV images. Background Technology

[0002] In recent years, computer vision-based automatic detection technology has received widespread attention and has developed into two main directions in detecting potential environmental hazards and various facilities along railway lines: (1) Image processing. These methods, such as directional gradient histogram feature extraction, scale-invariant feature transformation, Hough transform, and local Weber-like contrast enhancement, have shown great potential in detecting environmental hazards along high-speed rail and other railway equipment. However, image processing-based methods analyze objects with predefined features, and idealized feature processing cannot meet the needs of railway detection. (2) Deep learning. Compared with image processing, convolutional neural networks used for end-to-end model training are less sensitive to environmental changes and can directly process the complex features of the original image, thus becoming a research hotspot for automatic detection of environmental hazards. Currently, existing deep learning-based railway environmental hazard identification methods use standard track gauge as a practical reference to estimate the risk distance, but cannot estimate the area of ​​environmental hazards. This may lead to large areas of lightweight floating objects or pollutants being judged as low risk by a single factor (risk distance). In addition, the evaluation model is easily affected by complex environments, especially when the track area in the UAV image is not fully captured, which may directly lead to the inability to estimate the risk distance. Secondly, YOLARC uses a simple deep residual network and feature pyramid network structure to extract and fuse feature information, which may hinder the network from mining key discriminative features and effectively transmit information. Furthermore, in terms of evaluation, YOLARC relies heavily on track integrity in the image as a true mapping to estimate risk distance, and it is difficult to assess the area of ​​these risk objects when faced with irregular edges of lightweight floating objects (such as plastic tarpaulins) or contaminants (such as garbage). Summary of the Invention

[0003] To address the aforementioned shortcomings in existing technologies, this invention provides a method for identifying railway environmental hazards and assessing risk levels based on UAV images, which solves the problems of low accuracy in identifying environmental risk targets and inaccurate risk level assessment.

[0004] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0005] A method for identifying potential hazards and assessing risk levels in railway environments based on UAV imagery includes the following steps:

[0006] S1. Use drones to fly along the track and collect images of the railway environment.

[0007] S2. Introduce the cross-stage local connection module that integrates the dynamic adjustment module of the receptive field of the large convolutional kernel into the backbone network of the large convolutional kernel to construct a railway environmental hazard identification model based on the large convolutional kernel.

[0008] S3. Input the UAV railway environment image into the railway environment hazard identification model based on large convolution kernel for training, and obtain the trained railway environment hazard identification model based on large convolution kernel, which is used to predict environmental hazard targets in UAV environment images.

[0009] S4. Reacquire the UAV railway environment image and input it into the trained railway environment hazard identification model based on large convolution kernel to identify environmental hazard targets and obtain the predicted environmental hazard targets and track mask.

[0010] S5. Based on the predicted environmental hazard targets and the track mask, calculate the risk distance between the environmental hazard targets and the track edge, as well as the actual area of ​​lightweight floating objects or pollutants in the environmental hazard targets.

[0011] S6. Assess the risk level of the railway environment based on the risk distance between the environmental hazard target and the edge of the track, as well as the actual area of ​​lightweight floating objects or pollutants in the environmental hazard target.

[0012] Furthermore, the railway environmental hazard identification model based on large convolutional kernels includes a large convolutional kernel backbone network module, a feature aggregation and allocation neck structure module, and a prototype network plus prediction head module connected in sequence.

[0013] The large convolutional kernel backbone network module includes a cross-stage local connectivity module that integrates the large convolutional kernel receptive field dynamic adjustment module;

[0014] The cross-stage local connectivity module that integrates the large convolutional kernel receptive field dynamic adjustment module includes a first convolutional module, a second convolutional module, and a residual connectivity module.

[0015] The residual connection module includes a module for dynamically adjusting the receptive field of large convolutional kernels;

[0016] The module for dynamic adjustment of the receptive field of large convolutional kernels includes a module for decoupling large convolutional kernel sequences and a module for spatial selection attention mechanism.

[0017] The large convolutional kernel sequence decoupling module includes a first large convolutional kernel and a second large convolutional kernel;

[0018] The feature aggregation and allocation neck structure module includes a low-dimensional fusion module and a high-dimensional fusion module.

[0019] Furthermore, step S3 specifically includes:

[0020] S31. Input the UAV railway environment image into the large convolutional kernel backbone network module of the railway environment hazard identification model based on large convolutional kernel for multi-scale feature extraction to obtain feature maps of different scales;

[0021] S32. Input feature maps of different scales into the feature aggregation and allocation neck structure module to fuse multi-scale features and obtain multi-scale fused feature maps;

[0022] S33. Input the multi-scale fused feature map into the prototype network and prediction head module to perform location detection, classification and segmentation of the target of interest, and finally obtain the trained railway environmental hazard identification model based on large convolutional kernel, which is used to predict environmental hazard targets in UAV environmental images.

[0023] 4. The method for identifying railway environmental hazards and assessing risk levels based on UAV images according to claim 3, characterized in that step S31 specifically includes:

[0024] S311. Input the UAV railway environment image into the first convolutional module of the large convolutional kernel backbone network module for convolution processing, and then split it along the channel to obtain the first branch feature map and the second branch feature map.

[0025] S312. Input the first branch feature map into the residual connection module to perform residual connection operation, and obtain the first branch feature map of the residual connection module;

[0026] S313. The first branch feature map and the second branch feature map of the residual connection module are concatenated through channels and then input into the second convolution module for convolution operation to obtain feature maps of different scales.

[0027] Furthermore, step S312 specifically includes:

[0028] S3121. The first branch feature map is sequentially input into the first and second large convolutional kernels of the large convolutional kernel sequence decoupling module for feature extraction, resulting in features of different receptive fields, i.e.:

[0029]

[0030] Where x0 and x both represent the first branch feature map, x i The features representing the receptive field of the i-th largest convolutional kernel. Let x represent the i-th largest convolution kernel. i-1 The features representing the receptive field of the (i-1)th largest convolutional kernel;

[0031] S3122. Channel mixing of features from different receptive fields yields intermediate features, namely:

[0032]

[0033] Among them, M i This represents the intermediate features of the i-th largest convolutional kernel. This represents a 1×1 convolutional layer with the i-th largest convolutional kernel;

[0034] S3123. Concatenate the intermediate features to obtain the feature map of the large convolutional kernel sequence decoupling module;

[0035] S3124. Input the feature map of the large convolutional kernel sequence decoupling module into the spatial selection attention mechanism module, and use average feature extraction and maximum feature extraction to obtain the channel-dimensional spatial attention map, that is:

[0036] SAM = [AFE(M), MFE(M)]

[0037] Wherein, SAM represents the spatial attention map of the channel dimension, AFE represents the average feature extraction, MFE represents the maximum feature extraction, and M represents the feature map of the large convolutional kernel sequence decoupling module.

[0038] S3125. Convert the channel-dimensional spatial attention map into several spatial attention maps, and use the sigmoid activation function to obtain the spatial selection mask for activation, i.e.:

[0039] S = sigmoid(F) 2→n (SAM))=[S1,…,S i ,…,S n ]

[0040] Where S represents the activated spatial selection mask, and F 2→n (SAM) indicates that a channel-dimensional spatial attention map is transformed into n spatial attention maps using a 1×1 convolutional layer, where S1 represents the first spatial selection mask, and S... i S represents the spatial selection mask for the i-th space. n This represents the nth spatial selection mask;

[0041] S3126. The intermediate features are weighted according to the activation spatial selection mask and fused through a convolutional layer to obtain the final attention features, i.e.:

[0042]

[0043] in, F represents the final attention feature. 1×1 This represents a 1×1 convolutional layer;

[0044] S3127. Multiply the first branch feature map by the final attention feature to obtain the first branch feature map of the residual connection module, i.e.:

[0045]

[0046] Where y represents the first branch feature map of the residual connection module.

[0047] Furthermore, step S32 specifically includes:

[0048] First, feature maps of different scales are input into the feature aggregation and allocation neck structure module for convolution processing. Through the feature aggregation mechanism, local features of different scales are scale-aligned and then concatenated. Then, convolution processing is performed on the concatenated multi-scale features to obtain local features.

[0049] Secondly, the local features are aligned with the global features after feature adjustment through a feature allocation mechanism to obtain the global features;

[0050] Then, the local features are input into the low-dimensional fusion module and the high-dimensional fusion module for fusion, and then convolution is performed to obtain embedded local features. At the same time, the global features are divided into two parts. One part is processed by the sigmoid activation function after the convolution operation to obtain global activation features, and the other part is directly convolved to obtain the global features to be fused.

[0051] Finally, the global activation features are upsampled or downsampled and multiplied with the embedded local features, and then added to the global features to be fused after upsampling or downsampling to obtain a multi-scale fused feature map.

[0052] Furthermore, the calculation formula for the multi-scale fused feature map is as follows:

[0053]

[0054] Among them, F fused This represents a multi-scale fused feature map, where R represents global feature size reshaping, i.e., upsampling or downsampling, and x... act x represents the global activation feature. g_emb x represents the global features to be fused. l_emb Indicates embedding local features, Conv represents the convolution operation, and x represents the local feature embedding. g Represents global features, x l Indicates local features.

[0055] Furthermore, step S5 specifically includes:

[0056] S51. Based on the predicted environmental hazard targets and the orbital mask, refine the orbital edges using convex hull fitting and contour fitting in image morphology, and calculate the pixel-level distance between the orbital edge closest to the environmental hazard target and the environmental hazard target, i.e.:

[0057]

[0058] Where, dpixel This represents the pixel-level distance between the orbital edge closest to the environmental hazard target and the target itself. x1 and y1 represent the x and y coordinates of the upper corner of the orbital edge closest to the target, respectively, while x2 and y2 represent the x and y coordinates of the lower corner of the orbital edge closest to the target, respectively. peh y peh The x and y coordinates of the center point of the bounding box representing the predicted environmental hazard target;

[0059] S52. The key parameter unit pixel needed to calculate the mapping from UAV railway environment image to the real world is:

[0060]

[0061] Where, δ pixel The key parameters required for mapping UAV railway environment images to the real world are represented in pixels, tan represents the tangent function, DFoV represents the camera's diagonal field of view, f represents the camera's focal length, W represents the width of the UAV railway environment image resolution, and H represents the height of the UAV railway environment image resolution.

[0062] S53. Calculate the length of a unit pixel in the UAV railway environment image mapped to the real world, i.e.:

[0063]

[0064] Where, Δ d H represents the length of a unit pixel in a UAV railway environment image mapped to the real world. UAV Indicates flight altitude;

[0065] S54. Calculate the area of ​​the real world mapped from a unit pixel of the UAV railway environment image, i.e.:

[0066] Δ a =Δ d 2

[0067] Where, Δ a This represents the area of ​​the real world mapped from a single pixel in a drone image of the railway environment.

[0068] S55. Based on the pixel-level distance between the orbital edge closest to the environmental hazard target and the environmental hazard target, and the length mapped from a unit pixel of the UAV image to the real world, calculate the risk distance between the environmental hazard target and the orbital edge, as well as the actual area of ​​lightweight floating objects or pollutants within the environmental hazard target, i.e.:

[0069]

[0070] wherein, d represents the risk distance between the environmental hazard target and the track edge, a ′ represents the actual area of light floating objects or polluting objects in the environmental hazard target, n pixel represents the number of pixels in the prediction mask of light floating objects or polluting objects in the environmental hazard target.

[0071] Further, step S6 specifically includes:

[0072] According to the risk distance between the environmental hazard target and the track edge and the actual area of light floating objects or polluting objects in the environmental hazard target, the environmental hazard targets are divided into three risk levels, specifically:

[0073] If the environmental hazard target is identified as a housing construction target including temporary buildings, the following holds:

[0074] When 0m < d ≤ 20m, the risk level is high;

[0075] When 20m < d ≤ 500m, the risk level is medium;

[0076] When 500m < d, the risk level is low;

[0077] If the environmental hazard target is identified as a plastic film target or a garbage accumulation target, the following holds:

[0078] When 0m < d ≤ 20m, the risk level is high;

[0079] When 20m < d ≤ 500m and a ′ ≥ 10m 2 , the risk level is high;

[0080] When 20m < d ≤ 500m and a ′ < 10m 2 , the risk level is medium;

[0081] When 500m < d, the risk level is low.

[0082] The present invention has the following beneficial effects:

[0083] 1. The method for identifying railway environmental hidden dangers and assessing risk levels based on drone images proposed by the present invention improves the processing speed and precision of the model. Meanwhile, key drone parameters are introduced to calculate the actual area of light floating objects or polluting objects in environmental hazard targets, which improves the calculation precision and risk assessment precision;

[0084] 2. The model effectively extracts useful environmental risk information from drone railway environment images, thereby facilitating railway maintenance and ensuring railway operation safety. Description of Drawings

[0085] Figure 1 This is a flowchart illustrating the railway environmental hazard identification and risk level assessment method based on UAV images proposed in this invention.

[0086] Figure 2 This is a schematic diagram illustrating the principle of a railway environmental hazard identification model based on large convolutional kernels.

[0087] Figure 3 A schematic diagram of the cross-stage local connectivity module for integrating the dynamic adjustment module of the receptive field of large convolutional kernels;

[0088] Figure 4 A schematic diagram of the structure of the module for dynamically adjusting the receptive field of a large convolutional kernel;

[0089] Figure 5 This is a schematic diagram illustrating the principle of feature aggregation and allocation of the neck structure module;

[0090] Figure 6 A schematic diagram illustrating the aggregation, allocation, and injection process of low-dimensional and high-dimensional features of the neck structure module for feature aggregation and allocation;

[0091] Figure 7 This is a schematic diagram illustrating the calculation principle of the risk distance between the environmental hazard target and the edge of the track, as well as the actual area of ​​lightweight floating objects or pollutants within the environmental hazard target. Detailed Implementation

[0092] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0093] like Figure 1 As shown, the method for identifying railway environmental hazards and assessing risk levels based on UAV images includes the following steps S1-S6:

[0094] S1. Use drones to fly along the track and collect images of the railway environment.

[0095] In this embodiment, by setting the flight parameters of the UAV, the UAV is used to fly along the track to collect railway environment images during the UAV flight, so that the obtained railway environment images can be used in the subsequent model training process.

[0096] S2. Introduce the cross-stage local connection module that integrates the dynamic adjustment module of the receptive field of the large convolutional kernel into the backbone network of the large convolutional kernel to construct a railway environmental hazard identification model based on the large convolutional kernel.

[0097] Specifically, the railway environmental hazard identification model based on large convolutional kernels includes a large convolutional kernel backbone network module, a feature aggregation and allocation neck structure module, and a prototype network plus prediction head module connected in sequence. The large convolutional kernel backbone network module includes a cross-stage local connection module that integrates the large convolutional kernel receptive field dynamic adjustment module. The cross-stage local connection module that integrates the large convolutional kernel receptive field dynamic adjustment module includes a first convolutional module, a second convolutional module, and a residual connection module. The residual connection module includes a large convolutional kernel receptive field dynamic adjustment module. The large convolutional kernel receptive field dynamic adjustment module includes a large convolutional kernel sequence decoupling module and a spatial selection attention mechanism module. The large convolutional kernel sequence decoupling module includes a first large convolutional kernel and a second large convolutional kernel. The feature aggregation and allocation neck structure module includes a low-dimensional fusion module and a high-dimensional fusion module.

[0098] like Figure 2 As shown, Figure 2 The railway environmental hazard identification model based on large convolutional kernels includes a large convolutional kernel backbone network module, a feature aggregation and allocation neck structure module, and a prototype network (ProtoNet) plus a prediction head module. The large convolutional kernel backbone network is used for multi-scale feature extraction from UAV railway environmental images. The feature aggregation and allocation neck structure is used for multi-scale feature fusion. The prototype network (ProtoNet) in the prediction head module is used to generate multiple feature prototype maps. The prediction head is responsible for the location detection of hazardous environmental targets in UAV railway environmental images, target category classification, and the generation of feature prototypes. Figure 1 A corresponding coefficient is obtained, and the target segmentation map is obtained by matrix calculation using the coefficient and the prototype feature map.

[0099] The large convolutional kernel backbone network includes a cross-stage local connectivity module that integrates a large convolutional kernel receptive field dynamic adjustment module. This cross-stage local connectivity module is used to reduce redundant gradient computation during backpropagation, thereby reducing the model's computational cost while maintaining a high recognition rate. Specifically, as follows... Figure 3 As shown, Figure 3 This is a schematic diagram of the cross-stage local connectivity module that integrates the large convolutional kernel receptive field dynamic adjustment module. It includes a first convolutional module, a second convolutional module, and a residual connectivity module. The first and second convolutional modules are used for convolution operations, and the residual connectivity module is used for residual connection operations to avoid gradient vanishing or gradient exploding. The residual connectivity module includes the large convolutional kernel receptive field dynamic adjustment module. The specific structure of the large convolutional kernel receptive field dynamic adjustment module is shown below. Figure 4As shown, it includes a large convolutional kernel sequence decoupling module and a spatial selection attention mechanism module. The large convolutional kernel sequence decoupling module includes a first large convolutional kernel (large convolutional kernel 1) and a second large convolutional kernel (large convolutional kernel 2). The first and second large convolutional kernels are used for feature extraction to obtain features of different receptive fields. The spatial selection attention mechanism module is used to establish a spatial selection attention mechanism, which involves concatenating the features of multiple different receptive fields along the channel dimension and using average feature extraction, maximum feature extraction, and activation function processing to obtain a spatial attention map. By applying the spatial attention map to the features of multiple initial different receptive fields, it can adapt to UAV remote sensing targets with significant scale changes.

[0100] Secondly, the network parameter configuration of the large convolutional kernel backbone network is shown in Table 1:

[0101] Table 1. Network parameter configuration table for the large convolutional kernel backbone network.

[0102]

[0103] In Table 1, Block represents the cross-stage local connection module that integrates the dynamic adjustment module of the receptive field of the large convolutional kernel. The backbone network of the large convolutional kernel also includes the CBS module, which is composed of convolution, batch normalization, and SiLu activation function. Its function is to perform convolution, batch normalization, and SiLu activation function operations on the input feature map, i.e., the UAV railway environment image. The (K, D) sequence parameters represent the kernel size and dilation rate of the first large convolutional kernel (large convolutional kernel 1) and the second large convolutional kernel (large convolutional kernel 2) in the large convolutional kernel sequence decoupling module, respectively.

[0104] The principle of feature aggregation and allocation of neck structure modules is as follows: Figure 5 As shown, the feature aggregation and allocation neck structure module performs convolution processing on the multi-scale features {B2, B3, B4, B5} extracted by the large convolutional kernel backbone network module. Through the feature aggregation mechanism, local features of different scales are first scale-aligned. Then, convolution processing is performed on the concatenated multi-scale features to fully fuse the semantic information between local features, thereby obtaining the fused global features. The feature allocation mechanism adjusts the global features to align with the local features to be fused. This module not only mines potential key features from UAV railway environment images but also enriches and strengthens limited semantic information. {B2, B3, B4, B5} are extracted from the outputs of layers 3, 5, 7, and 9 in the large convolutional kernel backbone network, with B2, B3, B4, and B5 being 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image size, respectively. Simultaneously, the aggregation, allocation, and injection process of low-dimensional and high-dimensional features in the feature aggregation and allocation neck structure module is as follows: Figure 6 As shown:

[0105] First, {B2, B3, B4, B5} are used as input to the low-dimensional feature aggregation mechanism. In this mechanism, B5 is downsampled, and {B2, B3} are upsampled, aligning the feature sizes of {B2, B3, B5} with those of B4. Then, the aligned feature maps are concatenated to obtain the low-dimensional aggregated feature F. align In the low-dimensional feature allocation mechanism, low-dimensional aggregated features F are processed through Block. align This allows for sufficient information exchange, thereby generating global features. Subsequently, the global features are divided into two global features (x... g_B3 and x g_B4 This is used for subsequent local feature information injection. To achieve more efficient information exchange between global and local information, attention operations are also employed during the injection process. Specifically, local features are calculated through a 1×1 convolutional layer (Conv) to obtain embedded local features. Two parallel branches with different convolutions are used for global features to generate global activation features processed by the Sigmoid function and global features to be fused. The global activation features are then used to perform matrix multiplication with the embedded local features at corresponding positions. The result of the interaction between the global activation features and the embedded local features is then matrix-added with the global features to be fused to obtain the fused feature. Then, to align the sizes between the local and global features, upsampling or downsampling is used in the global activation features and the global features to be fused. Similarly, the aggregation, distribution, and injection process of high-dimensional features is the same as that of low-dimensional features, except that high-level information processing fuses the features {C3, C4, C5} from the low-level features to obtain the high-dimensional aggregated feature F. ′ align And aggregate the high-dimensional features F ′ align x is obtained by splitting along the channel dimension. g_C4 With x g_C5 Where C5 = B5, and C3 is the sum of B3 and x g_B3 Information injection, C4 is B4 and x g_B4 Information injection; simultaneously, downsampling is performed in {C3,C4} to adjust the size of the feature map to align with C5; furthermore, during the high-dimensional feature injection process, from Figure 5 It can also be seen that the final high-dimensional fusion features P4 and P5 can be obtained respectively through {x g_C4 C4} and {x g_C5 The information is obtained by injecting C5, and P3 = C3.

[0106] Furthermore, to enhance local cross-layer information flow, low-dimensional fusion and high-dimensional fusion modules were introduced into the feature aggregation and allocation neck structure module. Specifically, local features are fused with adjacent features before feature injection, and feature injection is performed using either low-dimensional or high-dimensional fusion modules. Figure 6 The parameters for low- and mid-dimensional fusion and high-dimensional fusion are as follows. Refers to local features in the current dimension. This refers to local features that are one dimension lower than the current dimension. This refers to a local feature that is one dimension higher than the current dimension, such as in... Figure 5 During the B4 injection phase, It is B4. It is B3. It is B5.

[0107] S3. Input the UAV railway environment image into the railway environment hazard identification model based on large convolution kernel for training, and obtain the trained railway environment hazard identification model based on large convolution kernel, which is used to predict environmental hazard targets in UAV environment images.

[0108] Specifically, step S3 includes S31-S33:

[0109] S31. Input the UAV railway environment image into the large convolutional kernel backbone network module of the railway environment hazard identification model based on large convolutional kernels for multi-scale feature extraction to obtain feature maps of different scales.

[0110] Specifically, step S31 includes S311-S313:

[0111] S311. The UAV railway environment image is input into the first convolution module of the large convolution kernel backbone network module for convolution processing, and then split along the channel to obtain the first branch feature map and the second branch feature map.

[0112] In this embodiment, the first convolutional module is a convolutional module composed of a convolutional kernel, a batch normalization operation, and a SiLu activation function.

[0113] S312. Input the first branch feature map into the residual connection module to perform residual connection operation, and obtain the first branch feature map of the residual connection module.

[0114] Specifically, step S312 includes S3121-S3127:

[0115] S3121. The first branch feature map is sequentially input into the first and second large convolutional kernels of the large convolutional kernel sequence decoupling module for feature extraction, resulting in features of different receptive fields, i.e.:

[0116]

[0117] Where x0 and x both represent the first branch feature map, x i The features representing the receptive field of the i-th largest convolutional kernel. Let x represent the i-th largest convolution kernel. i-1 This represents the features of the receptive field of the (i-1)th largest convolutional kernel.

[0118] S3122. Channel mixing of features from different receptive fields yields intermediate features, namely:

[0119]

[0120] Among them, M i This represents the intermediate features of the i-th largest convolutional kernel. This represents a 1×1 convolutional layer with the i-th largest convolutional kernel.

[0121] S3123. Concatenate the intermediate features to obtain the feature map of the large convolutional kernel sequence decoupling module.

[0122] S3124. Input the feature map of the large convolutional kernel sequence decoupling module into the spatial selection attention mechanism module, and use average feature extraction and maximum feature extraction to obtain the channel-dimensional spatial attention map, that is:

[0123] SAM = [AFE(M), MFE(M)]

[0124] Where SAM represents the spatial attention map of the channel dimension, AFE represents average feature extraction, MFE represents maximum feature extraction, and M represents the feature map of the large convolutional kernel sequence decoupling module.

[0125] S3125. Convert the channel-dimensional spatial attention map into several spatial attention maps, and use the sigmoid activation function to obtain the spatial selection mask for activation, i.e.:

[0126] S = sigmoid(F) 2→n (SAM))=[S1,…,S i ,…,S n ]

[0127] Where S represents the activated spatial selection mask, and F 2→n (SAM) indicates that a channel-dimensional spatial attention map is transformed into n spatial attention maps using a 1×1 convolutional layer, where S1 represents the first spatial selection mask, and S... i S represents the spatial selection mask for the i-th space. n This indicates the nth spatial selection mask.

[0128] S3126. The intermediate features are weighted according to the activation spatial selection mask and fused through a convolutional layer to obtain the final attention features, i.e.:

[0129]

[0130] in, F represents the final attention feature. 1×1 This represents a 1×1 convolutional layer.

[0131] S3127. Multiply the first branch feature map by the final attention feature to obtain the first branch feature map of the residual connection module, i.e.:

[0132]

[0133] Where y represents the first branch feature map of the residual connection module.

[0134] S313. The first branch feature map and the second branch feature map of the residual connection module are concatenated through channels and then input into the second convolution module for convolution operation to obtain feature maps of different scales.

[0135] In this embodiment, the second convolution module is a convolution module composed of a convolution kernel, a batch normalization operation, and a SiLu activation function.

[0136] S32. Input feature maps of different scales into the feature aggregation and allocation neck structure module to fuse multi-scale features and obtain multi-scale fused feature maps.

[0137] Specifically, step S32 includes:

[0138] First, feature maps of different scales are input into the feature aggregation and allocation neck structure module for convolution processing. Through the feature aggregation mechanism, local features of different scales are scale-aligned and then concatenated. Then, convolution processing is performed on the concatenated multi-scale features to obtain local features.

[0139] Secondly, the local features are aligned with the global features after feature adjustment through a feature allocation mechanism to obtain the global features.

[0140] Then, the local features are input into the low-dimensional fusion module and the high-dimensional fusion module for fusion, and then a convolution operation is performed to obtain the embedded local features. At the same time, the global features are divided into two parts. One part is processed by the sigmoid activation function after the convolution operation to obtain the global activation features, and the other part is directly convolved to obtain the global features to be fused.

[0141] Finally, the global activation features are upsampled or downsampled and multiplied with the embedded local features, and then added to the global features to be fused after upsampling or downsampling to obtain a multi-scale fused feature map.

[0142] Specifically, the formula for calculating the multi-scale fused feature map is as follows:

[0143]

[0144] Among them, F fused Represents a multi-scale fused feature map, R represents global feature size reshaping, and x act x represents the global activation feature. a_emb x represents the global features to be fused. l_emb Indicates embedding local features, Conv represents the convolution operation, and x represents the local feature embedding. g Represents global features, x l Indicates local features.

[0145] S33. Input the multi-scale fused feature map into the prototype network and prediction head module to perform location detection, classification and segmentation of the target of interest, and finally obtain the trained railway environmental hazard identification model based on large convolutional kernel, which is used to predict environmental hazard targets in UAV environmental images.

[0146] S4. Reacquire the UAV railway environment image and input it into the trained railway environment hazard identification model based on large convolution kernel to identify environmental hazard targets and obtain the predicted environmental hazard targets and track mask.

[0147] In this embodiment, the railway environment image of the UAV is input into the pre-trained railway environment hazard identification model based on large convolution kernel. The model can predict the position (coordinate information), category (class) of various targets in the image, and the polygonal mask formed by the target edges (the coordinate information of the target segmentation points).

[0148] S5. Based on the predicted environmental hazard targets and the track mask, calculate the risk distance between the environmental hazard targets and the track edge, as well as the actual area of ​​lightweight floating objects or pollutants within the environmental hazard targets.

[0149] like Figure 7 As shown, Figure 7 This diagram illustrates the calculation principle for the risk distance between environmental hazard targets and the edge of the track, as well as the actual area of ​​lightweight floating objects or pollutants within the environmental hazard targets. Figure 7The process includes track semantic segmentation extraction, segmentation refinement, and external polygon fitting. Hazard 1-Hazard 3 refer to predicted environmental hazards, and the white area represents the predicted track mask. Pixel-level to real-world distance and area estimations are performed based on the external polygons. Specifically: Based on the pixel-level segmentation results of the railway track area, the predicted environmental hazard targets (i.e., the predicted bounding boxes) and the track mask are obtained. Image processing techniques such as image morphology processing and circumscribed polygon fitting are used to eliminate jagged edges and unevenness that affect distance assessment, resulting in smoother track edges. This involves refining the track edges using convex hull fitting and contour fitting in image morphology, thereby extracting the minimum boundary polygon of the track area. Specifically, in this embodiment, the minimum circumscribed quadrilateral of the track area is extracted because track areas are typically quadrilaterals, and the minimum circumscribed quadrilateral can be used to fit the track edges. After obtaining the track edges, the risk distance between the environmental hazard targets and the track edges, as well as the actual area of ​​lightweight floating objects or pollutants within the environmental hazard targets, are calculated to perform risk level assessments in subsequent steps.

[0150] Specifically, step S5 includes S51-S55:

[0151] S51. Based on the predicted environmental hazard targets and the orbital mask, refine the orbital edges using convex hull fitting and contour fitting in image morphology, and calculate the pixel-level distance between the orbital edge closest to the environmental hazard target and the environmental hazard target, i.e.:

[0152]

[0153] Where, d pixel This represents the pixel-level distance between the orbital edge closest to the environmental hazard target and the target itself. x1 and y1 represent the x and y coordinates of the upper corner of the orbital edge closest to the target, respectively, while x2 and y2 represent the x and y coordinates of the lower corner of the orbital edge closest to the target, respectively. peh y peh The x and y coordinates represent the center point of the bounding box of the predicted environmental hazard target.

[0154] S52. The key parameter unit pixel needed to calculate the mapping from UAV railway environment image to the real world is:

[0155]

[0156] Where, δ pixelThe key parameters required for mapping UAV railway environment images to the real world are represented in pixels, tan represents the tangent function, DFoV represents the camera's diagonal field of view, f represents the camera's focal length, W represents the width of the UAV railway environment image resolution, and H represents the height of the UAV railway environment image resolution.

[0157] S53. Calculate the length of a unit pixel in the UAV railway environment image mapped to the real world, i.e.:

[0158]

[0159] Where, Δ d H represents the length of a unit pixel in a UAV railway environment image mapped to the real world. UAV Indicates flight altitude.

[0160] S54. Calculate the area of ​​the real world mapped from a unit pixel of the UAV railway environment image, i.e.:

[0161] Δ a =Δ d 2

[0162] Where, Δ a This represents the area of ​​the real world mapped from a single pixel in a drone image of the railway environment.

[0163] S55. Based on the pixel-level distance between the orbital edge closest to the environmental hazard target and the environmental hazard target, and the length mapped from a unit pixel of the UAV image to the real world, calculate the risk distance between the environmental hazard target and the orbital edge, as well as the actual area of ​​lightweight floating objects or pollutants within the environmental hazard target, i.e.:

[0164]

[0165] Where d represents the risk distance between the environmental hazard target and the edge of the orbit, a ′ n represents the actual area of ​​lightweight floating objects or pollutants within an environmental hazard target. pixel This represents the number of pixels in the prediction mask for lightweight floating objects or pollutants within the environmental hazard target.

[0166] S6. Assess the risk level of the railway environment based on the risk distance between the environmental hazard target and the edge of the track, as well as the actual area of ​​lightweight floating objects or pollutants in the environmental hazard target.

[0167] Specifically, step S6 includes:

[0168] Based on the risk distance between the environmental hazard target and the edge of the track, and the actual area of ​​lightweight floating objects or pollutants within the environmental hazard target, the environmental hazard targets are divided into three risk levels, as follows:

[0169] If the environmental hazard target is identified as a house construction target including temporary buildings, then:

[0170] When 0m < d ≤ 20m, the risk level is high.

[0171] When 20m < d ≤ 500m, the risk level is medium.

[0172] When 500m < d, the risk level is low.

[0173] If the environmental hazard target is identified as a plastic film target or a garbage accumulation target, then:

[0174] When 0m < d ≤ 20m, the risk level is high.

[0175] When 20m < d ≤ 500m and a ′ ≥ 10m 2 , the risk level is high.

[0176] When 20m < d ≤ 500m and a ′ <10m 2 , the risk level is medium.

[0177] When 500m < d, the risk level is low.

[0178] In the railway environmental hidden danger identification and risk level assessment method based on UAV images proposed by the present invention, the UAV is first used to fly along the track to collect UAV railway environment images, then a large convolution kernel backbone network is used to extract multi-scale feature maps of the UAV railway environment images. On this basis, a feature aggregation and distribution neck structure is used to fully fuse the information of the feature maps to enhance the limited semantic information of the UAV railway environment images. Then, under the processing of the prototype network and the prediction head, position detection, classification and segmentation of targets of interest in UAV railway environment images are realized; finally, risk level assessment is carried out on the detected hidden dangers, that is, environmental hazard targets. The method can achieve a processing speed of 45.0 fps, a mean absolute precision of bounding boxes for target prediction of 95.8% and a mean absolute precision of bounding box polygonal masks for target prediction of 91.7% across all detection categories. At the same time, key UAV parameters are utilized and integrated into the calculation of the risk distance between environmental hazard targets and track edges and the actual area of light floating objects or polluting objects among environmental hazard targets, which is used for automatic risk assessment. It can effectively convert UAV railway environment images into useful information to assist railway maintenance, thereby ensuring railway operation safety.

[0179] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0180] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A method for identifying railway environmental hazards and assessing risk levels based on UAV imagery, characterized in that, Includes the following steps: S1. Use drones to fly along the track and collect images of the railway environment. S2. Introduce the cross-stage local connection module that integrates the dynamic adjustment module of the receptive field of the large convolutional kernel into the backbone network of the large convolutional kernel to construct a railway environmental hazard identification model based on the large convolutional kernel; the cross-stage local connection module that integrates the dynamic adjustment module of the receptive field of the large convolutional kernel includes a first convolutional module, a second convolutional module and a residual connection module; S3. Input the UAV railway environment image into the railway environment hazard identification model based on large convolution kernel for training, and obtain the trained railway environment hazard identification model based on large convolution kernel, which is used to predict environmental hazard targets in UAV environment images. S4. Reacquire the UAV railway environment image and input it into the trained railway environment hazard identification model based on large convolution kernel to identify environmental hazard targets and obtain the predicted environmental hazard targets and track mask. S5. Based on the predicted environmental hazard targets and the orbital mask, calculate the risk distance between the environmental hazard targets and the orbital edge, as well as the actual area of ​​lightweight floating objects or pollutants within the environmental hazard targets. Specifically: S51. Based on the predicted environmental hazard targets and the orbital mask, refine the orbital edges using convex hull fitting and contour fitting in image morphology, and calculate the pixel-level distance between the orbital edge closest to the environmental hazard target and the environmental hazard target, i.e.: in, This represents the pixel-level distance between the orbital edge closest to the environmental hazard target and the target itself. , The x and y coordinates represent the upper corner point of the orbital edge closest to the environmental hazard target. , Let x and y represent the x and y coordinates of the lower corner point of the orbital edge closest to the environmental hazard target, respectively. , The x and y coordinates of the center point of the bounding box representing the predicted environmental hazard target; S52. The key parameter unit pixel needed to calculate the mapping from UAV railway environment image to the real world is: in, The key parameter unit, pixels, represents the mapping required from UAV images of railway environments to the real world. Represents the tangent function. Indicates the camera's diagonal field of view. Indicates the camera's focal length. Width representing the resolution of the UAV railway environment image. The height indicating the resolution of the UAV railway environment image; S53. Calculate the length of a unit pixel in the UAV railway environment image mapped to the real world, i.e.: in, This represents the length of a single pixel in a UAV image of a railway environment mapped to the real world. Indicates flight altitude; S54. Calculate the area of ​​the real world mapped from a unit pixel of the UAV railway environment image, i.e.: in, This represents the area of ​​the real world mapped from a single pixel in a UAV image of the railway environment. S55. Based on the pixel-level distance between the orbital edge closest to the environmental hazard target and the environmental hazard target, and the length mapped from a unit pixel of the UAV image to the real world, calculate the risk distance between the environmental hazard target and the orbital edge, as well as the actual area of ​​lightweight floating objects or pollutants within the environmental hazard target, i.e.: in, Indicates the risk distance between the environmentally hazardous target and the edge of the orbit. This indicates the actual area of ​​lightweight floating objects or pollutants within the environmental hazard target. This represents the number of pixels in the prediction mask for lightweight floating objects or pollutants within the environmental hazard target. S6. Assess the risk level of the railway environment based on the risk distance between the environmental hazard target and the edge of the track, as well as the actual area of ​​lightweight floating objects or pollutants in the environmental hazard target.

2. The method for identifying railway environmental hazards and assessing risk levels based on UAV imagery according to claim 1, characterized in that, The railway environmental hazard identification model based on large convolutional kernels includes a large convolutional kernel backbone network module, a feature aggregation and allocation neck structure module, and a prototype network plus prediction head module connected in sequence. The large convolutional kernel backbone network module includes a cross-stage local connectivity module that integrates the large convolutional kernel receptive field dynamic adjustment module; The residual connection module includes a module for dynamically adjusting the receptive field of large convolutional kernels; The module for dynamic adjustment of the receptive field of large convolutional kernels includes a module for decoupling large convolutional kernel sequences and a module for spatial selection attention mechanism. The large convolutional kernel sequence decoupling module includes a first large convolutional kernel and a second large convolutional kernel; The feature aggregation and allocation neck structure module includes a low-dimensional fusion module and a high-dimensional fusion module.

3. The method for identifying railway environmental hazards and assessing risk levels based on UAV imagery according to claim 2, characterized in that, Step S3 specifically includes: S31. Input the UAV railway environment image into the large convolutional kernel backbone network module of the railway environment hazard identification model based on large convolutional kernel for multi-scale feature extraction to obtain feature maps of different scales; S32. Input feature maps of different scales into the feature aggregation and allocation neck structure module to fuse multi-scale features and obtain multi-scale fused feature maps; S33. Input the multi-scale fused feature map into the prototype network and prediction head module to perform location detection, classification and segmentation of the target of interest, and finally obtain the trained railway environmental hazard identification model based on large convolutional kernel, which is used to predict environmental hazard targets in UAV environmental images.

4. The method for identifying railway environmental hazards and assessing risk levels based on UAV imagery according to claim 3, characterized in that, Step S31 specifically includes: S311. Input the UAV railway environment image into the first convolutional module of the large convolutional kernel backbone network module for convolution processing, and then split it along the channel to obtain the first branch feature map and the second branch feature map. S312. Input the first branch feature map into the residual connection module to perform residual connection operation, and obtain the first branch feature map of the residual connection module; S313. The first branch feature map and the second branch feature map of the residual connection module are concatenated through channels and then input into the second convolution module for convolution operation to obtain feature maps of different scales.

5. The method for identifying railway environmental hazards and assessing risk levels based on UAV images according to claim 4, characterized in that, Step S312 specifically includes: S3121. The first branch feature map is sequentially input into the first and second large convolutional kernels of the large convolutional kernel sequence decoupling module for feature extraction, resulting in features of different receptive fields, i.e.: in, , Both represent the feature maps of the first branch. Indicates the first Features of the receptive field of large convolutional kernels Indicates the first Large convolution kernel, Indicates the first Features of the receptive field of large convolutional kernels; S3122. Channel mixing of features from different receptive fields yields intermediate features, namely: in, Indicates the first Intermediate features of large convolutional kernels Indicates the first Large convolution kernel Convolutional layer; S3123. Concatenate the intermediate features to obtain the feature map of the large convolutional kernel sequence decoupling module; S3124. Input the feature map of the large convolutional kernel sequence decoupling module into the spatial selection attention mechanism module, and use average feature extraction and maximum feature extraction to obtain the channel-dimensional spatial attention map, that is: in, Spatial attention map representing the channel dimension. Indicates average feature extraction. This indicates the extraction of the maximum feature. Feature map representing the decoupling module of large convolutional kernel sequence; S3125. Convert the channel-dimensional spatial attention map into several spatial attention maps, and use the sigmoid activation function to obtain the spatial selection mask for activation, i.e.: in, Indicates the active spatial selection mask. Indicates the use of Convolutional layers transform the spatial attention map of the channel dimension into... A spatial attention map, Indicates the first Select a mask for the space. Indicates the first Select a mask for the space. Indicates the first Select a mask for a given space; S3126. The intermediate features are weighted according to the activation spatial selection mask and fused through a convolutional layer to obtain the final attention features, i.e.: in, This represents the final attention feature. express Convolutional layer; S3127. Multiply the first branch feature map by the final attention feature to obtain the first branch feature map of the residual connection module, i.e.: in, This represents the feature map of the first branch of the residual connection module.

6. The method for identifying railway environmental hazards and assessing risk levels based on UAV images according to claim 5, characterized in that, Step S32 specifically includes: First, feature maps of different scales are input into the feature aggregation and allocation neck structure module for convolution processing. Through the feature aggregation mechanism, local features of different scales are scale-aligned and then concatenated. Then, convolution processing is performed on the concatenated multi-scale features to obtain local features. Secondly, the local features are aligned with the global features after feature adjustment through a feature allocation mechanism to obtain the global features; Then, the local features are input into the low-dimensional fusion module and the high-dimensional fusion module for fusion, and then convolution is performed to obtain embedded local features. At the same time, the global features are divided into two parts. One part is processed by the sigmoid activation function after the convolution operation to obtain global activation features, and the other part is directly convolved to obtain the global features to be fused. Finally, the global activation features are upsampled or downsampled and multiplied with the embedded local features, and then added to the global features to be fused after upsampling or downsampling to obtain a multi-scale fused feature map.

7. The method for identifying railway environmental hazards and assessing risk levels based on UAV imagery according to claim 6, characterized in that, The formula for calculating the multi-scale fused feature map is: in, This represents a multi-scale fused feature map. This indicates global feature size reshaping, i.e., upsampling or downsampling. Represents global activation features. This represents the global features to be merged. Indicates embedded local features, This represents the convolution operation. Represents global features. Indicates local features.

8. The method for identifying railway environmental hazards and assessing risk levels based on UAV imagery according to claim 7, characterized in that, Step S6 specifically includes: Based on the risk distance between the environmental hazard target and the edge of the track, and the actual area of ​​lightweight floating objects or pollutants within the environmental hazard target, the environmental hazard targets are divided into three risk levels, as follows: If the environmental hazard target is identified as a building-related target that includes temporary structures, then: when At that time, the risk level was high. when At that time, the risk level was medium. when At that time, the risk level was low; If the environmental hazard target is identified as a plastic film target or a garbage dump target, then: when At that time, the risk level was high. when Sometimes The risk level is high. when Sometimes The risk level is medium. when At that time, the risk level was low.

Images