Method and device for identifying potential safety hazards around railway

Abstract

The invention provides a railway periphery potential safety hazard identification method and device, and the method comprises the steps: introducing a double-branch feature extraction network, extracting rail line features and railway periphery texture, edge and other low-level features based on a multi-convolution layer and an average pooling layer of a shallow branch on one hand, and introducing a multi-residual module-based jump connection structure of a deep branch on the other hand, the method pays attention to extraction of semantic features of railway infrastructures and surroundings; weighted fusion or direct splicing is performed on the features of the shallow branches and the deep branches, so that the requirement on a deep structure can be reduced on the basis of ensuring the recognition precision, and the dependence on computing resources is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of potential safety hazards identification, and in particular to a method and device for identifying potential safety hazards around railways. Background Art

[0002] At present, the need to identify safety hazards around railways mainly relies on video surveillance and manual or sensor-based monitoring. Introducing neural networks to process and analyze safety monitoring data around railways has significant advantages, including fast speed, accurate recognition, and the ability to process data from multiple regions and multiple sections at the same time. In order to achieve better recognition results, existing technologies need to use deep-structured neural networks to extract complex semantic features. However, there is a contradiction between the complexity of the network structure and the demand for computing resources. The training of deep network structures requires a huge amount of computation, a long training time, and higher requirements for hardware equipment, which limits its application scenarios. Faced with the increasingly complex forms and content of railway detection data, the introduction of neural networks for high-precision, fast, and low-resource safety hazard detection is an urgent problem that needs to be solved. Summary of the Invention

[0003] In view of this, an embodiment of the present invention provides a method and device for identifying safety hazards around railways to eliminate or improve one or more defects in the existing technology, and solve the problem that the existing technology cannot take into account both accuracy and low computing power requirements when introducing an application to identify safety hazards around railways via the network.

[0004] One aspect of the present invention provides a method for identifying potential safety hazards around railways, the method comprising the following steps:

[0005] Acquire a remote sensing image of the railway to be inspected, and preprocess the remote sensing image, wherein the preprocessing includes radiation correction, geometric correction, and image cropping within a set range along the railway;

[0006] The pre-processed railway remote sensing image is input into a pre-trained railway surrounding safety hazard identification model, and a preliminary identification result of the safety hazard location and type is output. The preliminary identification result is filtered based on non-maximum suppression to obtain a target recognition result; wherein, the safety hazard identification model includes two parts: a shallow branch and a deep branch. The shallow branch includes multiple continuous convolutional layers and average pooling layers, and the deep branch includes multiple continuous stages, each stage is composed of multiple residual block layers, and the output of the average pooling layer is connected to the input of the last stage.

[0007] In some embodiments, the radiation correction in the pre-processing includes:

[0008] The original digital quantized value is converted into an apparent reflectivity with physical meaning based on sensor radiation calibration;

[0009] Atmospheric correction is performed based on dark pixel subtraction to eliminate the effects of atmospheric scattering, absorption, and aerosols to obtain the true reflectance of the surface;

[0010] Perform a terrain correction to remove shadows and radiation differences caused by terrain relief.

[0011] In some embodiments, the geometric correction in the preprocessing includes:

[0012] Obtain a reference image and use the radiometrically corrected remote sensing image as the target influence, and use image registration tools to generate homonymous control points;

[0013] Low-order polynomial fitting of geometric distortion is used to correct for flat terrain areas, and digital elevation models are used to correct for undulating terrain areas.

[0014] Use bilinear interpolation or cubic convolution to resample the image.

[0015] In some embodiments, cropping an image within a set range along a railway line includes:

[0016] Pre-dividing the railway line into supervisory areas according to preset railway safety management standards, and establishing boundary vectors for the supervisory areas based on a geographic information system;

[0017] A buffer zone is set for the boundary vector of the regulatory area according to a preset distance, and the buffer zone range is used as a mask to crop the pre-processed railway remote sensing image to retain the target area.

[0018] In some embodiments, the pre-training step of the railway surrounding safety hazard identification model includes:

[0019] Obtaining a training sample set, which includes multiple samples, each sample includes a sample railway remote sensing image, and marking the real location and real classification of the safety hazard in the sample railway remote sensing image as a label;

[0020] Preprocessing the sample railway remote sensing image, wherein the preprocessing includes radiation correction, geometric correction, and image cropping within a set range along the railway;

[0021] An initial neural network is obtained, comprising a shallow branch and a deep branch, wherein the shallow branch comprises a plurality of consecutive convolutional layers and an average pooling layer, and the deep branch comprises a plurality of consecutive stages, each stage being composed of a plurality of residual blocks, and the output of the average pooling layer is connected to the input of the last stage; the sample railway remote sensing image is respectively imported into the shallow branch and the deep branch, and then weighted feature fusion or direct splicing is performed, and multiple candidate safety hazard location frames and their corresponding classification probabilities are output, and then the predicted location and predicted classification of the safety hazard are output based on non-maximum suppression screening;

[0022] The preprocessed training sample set is used to train the initial neural network, the cross entropy loss function of the predicted classification and the true classification is used as the classification loss, the mean square error loss between the predicted position and the true position is used as the positioning loss, and the classification loss and the positioning loss are combined to update the parameters of the initial neural network to obtain the railway periphery safety hazard identification model.

[0023] In some embodiments, the first convolution layer in the shallow branch uses a convolution kernel of size 3×9, and subsequent convolution layers use a combination of convolution kernels of size 3×3 or 3×5, and the size of the average pooling layer is 2×2;

[0024] The deep branch includes at least 4 stages, and the structure of a single residual block in each stage includes three consecutive convolutional layers. The input data of the residual block is jump-connected to the output, and the sizes of the convolution kernels are 1×1, 3×3 and 1×1 respectively.

[0025] In some embodiments, the classification loss is calculated as:

[0026]

[0027] Among them, y i represents the true classification, p i represents the predicted classification;

[0028] The calculation formula of the positioning loss is:

[0029]

[0030] in, and represents the abscissa and ordinate of the real position, and representing the abscissa and ordinate of the predicted position;

[0031] The formula for calculating the joint loss is:

[0032] L=L ce +λL mse ;

[0033] Where λ is the balance coefficient.

[0034] In some embodiments, an Adam optimizer is used to adaptively adjust the learning rate of each parameter. The Adam optimizer calculates the learning rate by:

[0035] Calculate the first-order moment estimate, the calculation formula is:

[0036] m t =β1m t-1 +(1-β1)g t ;

[0037] Calculate the second-order moment estimate, the calculation formula is:

[0038]

[0039] Then the learning rate calculation formula is:

[0040]

[0041] Among them, g t is the current gradient, β1 and β2 are the exponential decay rates of the moment estimates, θ is the learning rate, and ε is a small constant that prevents the denominator from being zero.

[0042] On the other hand, the present invention also provides a device for identifying safety hazards in the periphery of railways, comprising a processor, a memory, and a computer program / instruction stored in the memory, wherein the processor is used to execute the computer program / instruction, and when the computer program / instruction is executed, the device implements the steps of the above method.

[0043] On the other hand, the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of the above method when executed by a processor.

[0044] The beneficial effects of the present invention are at least:

[0045] The method and device for identifying safety hazards in the railway periphery described in the present invention introduce a dual-branch feature extraction network. On the one hand, multiple convolutional layers and average pooling layers based on shallow branches extract rail line features and low-level features such as texture and edges around the railway. On the other hand, a skip connection structure based on multiple residual modules is introduced into deep branches, focusing on the extraction of semantic features of railway infrastructure and the surrounding environment. By weighted fusion or direct splicing of the features of shallow branches and deep branches, the demand for deep structures can be reduced while ensuring recognition accuracy, reducing dependence on computing resources.

[0046] Furthermore, according to the characteristics of remote sensing data in different railway scenarios, the sample railway remote sensing images are respectively imported into the shallow branch and the deep branch and then weighted feature fusion or direct splicing is performed, and different proportions are set to effectively focus on reliable features between features at each level.

[0047] Additional advantages, objects, and features of the present invention will be set forth in part in the following description and will become apparent to those skilled in the art upon examination of the following or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained by the structures particularly pointed out in the description and drawings.

[0048] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present invention are not limited to the above specific descriptions, and the above and other purposes that can be achieved by the present invention will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention. In the drawings:

[0050] Figure 1 This is a schematic structural diagram of a railway perimeter safety hazard identification model in a railway perimeter safety hazard identification method according to an embodiment of the present invention.

[0051] Figure 2 for Figure 1 Schematic diagram of the structure of the residual block in

[15] . DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0053] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, the accompanying drawings only show structures and / or processing steps closely related to the solutions according to the present invention, while other details that are not closely related to the present invention are omitted.

[0054] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0055] This application uses railway remote sensing data to detect safety hazards around railways by introducing deep neural networks. In order to ensure the accuracy of identifying safety hazards while reducing computing power requirements, this application introduces a dual-branch feature extraction network. On the one hand, shallow branches are used to extract low-level features such as railway track line features, textures and edges around the railway. On the other hand, multiple residual modules of deep branches focus on the extraction of semantic features of railway infrastructure and surrounding environment. Through weighted fusion, it can pay attention to the differences in features at different levels, reducing dependence on computing resources while ensuring recognition accuracy.

[0056] Specifically, the present invention provides a method for identifying safety hazards around railways, which includes the following steps S101-S102:

[0057] Step S101: obtaining a remote sensing image of the railway to be detected, and preprocessing the remote sensing image. The preprocessing includes radiation correction, geometric correction, and image cropping within a set range along the railway.

[0058] Step S102: Input the pre-processed railway remote sensing image into the pre-trained railway surrounding safety hazard identification model, and output the preliminary identification results of the safety hazard location and type, and filter the preliminary identification results based on non-maximum suppression to obtain the target identification results; wherein, Figure 1 As shown in the figure, the safety hazard identification model consists of two parts: a shallow branch and a deep branch. The shallow branch includes multiple consecutive convolutional layers and average pooling layers, and the deep branch includes multiple consecutive stages. Each stage is composed of multiple residual block layers, and the output of the average pooling layer is connected to the input of the last stage.

[0059] In step S101, railway remote sensing images include optical remote sensing images and synthetic aperture radar (SAR) images. Optical remote sensing images provide rich spectral information and can be used to identify elements such as buildings and vegetation along the railway. SAR images, on the other hand, have all-weather and all-day imaging capabilities and can penetrate clouds and vegetation to capture microwave scattering characteristics of the earth's surface.

[0060] During the data collection process, ensuring data accuracy is crucial, including selecting appropriate sensors and platforms, as well as determining appropriate imaging times and frequencies. Secondly, the image resolution should be set according to pre-defined standards to meet the requirements of railway safety monitoring. Furthermore, image coverage must be considered to ensure comprehensive coverage of the railway area.

[0061] The acquired railway remote sensing images need to be preprocessed to eliminate data defects.

[0062] In some embodiments, the radiation correction in the pre-processing includes steps S201 to S203:

[0063] Step S201: converting the original digital quantized value into an apparent reflectivity with physical meaning based on sensor radiation calibration.

[0064] Step S202: Perform atmospheric correction based on dark pixel subtraction to eliminate atmospheric scattering, absorption and aerosol effects, and obtain the true reflectance of the surface.

[0065] Step S203: Perform terrain correction to eliminate shadows and radiation differences caused by terrain undulations.

[0066] In step S201, radiometric calibration converts the sensor's digital quantization value (DN value) into a physically meaningful apparent reflectance. Radiometric calibration eliminates radiometric errors caused by sensor response characteristics, ensuring the accuracy and consistency of image data. Common radiometric calibration methods include using standard lamps and integrating spheres.

[0067] In step S202, atmospheric correction is performed to eliminate the effects of atmospheric scattering, absorption, and aerosols on the image, thereby obtaining the true surface reflectance. Dark pixel subtraction, assuming that there are pixels with extremely low reflectance (such as shadowed areas or dark water bodies), eliminates atmospheric effects by subtracting the minimum value of these pixels. This method is simple and easy to use, but it may overlook the effects of multiple scattering and terrain differences.

[0068] In step S203, terrain correction is performed to eliminate shadows and radiation differences caused by terrain undulations. By calculating factors such as slope, aspect, and solar altitude, the image's radiometric measurements are converted to a reference plane (e.g., a horizontal plane) to correct for radiation distortion caused by terrain. Common terrain correction methods include physical models (e.g., the Sandmeier model) and semi-physical, semi-empirical models (e.g., the C-correction model).

[0069] In some embodiments, the geometric correction in the pre-processing includes steps S301 to S303:

[0070] Step S301: Acquire a reference image and use the radiometrically corrected remote sensing image as a target influence, and use an image registration tool to generate control points with the same name.

[0071] Step S302: using a low-order polynomial to fit the geometric distortion to correct the flat terrain area, and combining the digital elevation model to correct the undulating terrain area.

[0072] Step S303: resampling the image using bilinear interpolation or cubic convolution interpolation.

[0073] In step S301, the remote sensing image to be calibrated is registered with the reference image using imaging tools such as AROSICS, SimpleITK, or Elastix to generate control points with the same name. These control points should be located at easily identifiable feature points, such as road intersections and building corners, to ensure accurate measurement of their actual coordinates and image coordinates.

[0074] In step S302, low-order polynomial fitting is suitable for correction in flat terrain areas. By fitting the relationship between the input image and the output image, the optimal transformation parameters are obtained to eliminate geometric distortion.

[0075] The use of digital elevation models (DEM) to correct terrain undulations can more accurately correct geometric deformations caused by terrain.

[0076] In step S303 , bilinear interpolation or cubic convolution interpolation is used for image resampling to ensure the quality of the corrected image.

[0077] Bilinear interpolation determines the value of a new pixel by calculating a linear combination of four adjacent pixels. It is suitable for scenes that require smooth transitions.

[0078] Cubic convolution interpolation uses cubic polynomial fitting to calculate the value of the new pixel, which can provide smoother interpolation results and is suitable for correction with high precision requirements.

[0079] In some embodiments, the image cropping of a set range along the railway includes steps S401 to S402:

[0080] Step S401: pre-dividing the supervision area along the railway according to the preset railway safety management standard, and establishing boundary vectors for the supervision area based on the geographic information system.

[0081] Step S402: a buffer zone is set for the boundary vector of the supervision area according to a preset distance, and the buffer zone range is used as a mask to crop the pre-processed railway remote sensing image to retain the target area.

[0082] In step S401, the railway is pre-divided into supervisory areas according to railway safety management standards. These areas are usually based on railway safety management requirements to ensure coverage of key areas such as railway tracks, bridges, tunnels, etc.

[0083] Establish boundary vectors: Using Geographic Information System (GIS) technology, establish boundary vectors for each regulatory area. Boundary vectors can be drawn manually using GIS software or extracted from existing geographic data.

[0084] In step S402, a buffer zone is set for the boundary vector of the regulatory area according to a preset distance. The width of the buffer zone can be adjusted according to the needs of railway safety management, and is usually used to expand the scope of the regulatory area to cover areas with potential safety hazards. The buffer zone range is used as a mask to crop the image. The mask can be a vector file or raster file generated in GIS software. The mask is used to crop the pre-processed railway remote sensing image to retain the target area. Cropping can be completed using GIS software or remote sensing image processing tools (such as ENVI) to ensure that the cropped image only contains the regulatory area and its buffer zone.

[0085] In some embodiments, the pre-training step of the railway periphery safety hazard identification model includes steps S501 to S504:

[0086] Step S501: obtaining a training sample set, which includes multiple samples, each sample includes a sample railway remote sensing image, and marking the real location and real classification of the safety hazard in the sample railway remote sensing image as a label.

[0087] Step S502: Preprocessing the sample railway remote sensing image. The preprocessing includes radiation correction, geometric correction, and image cropping within a set range along the railway.

[0088] Step S503: Obtain an initial neural network, which includes two parts: a shallow branch and a deep branch. The shallow branch includes multiple consecutive convolutional layers and average pooling layers, and the deep branch includes multiple consecutive stages. Each stage is composed of multiple residual blocks, and the output of the average pooling layer is connected to the input of the last stage. Sample railway remote sensing images are imported into the shallow branch and the deep branch respectively, and weighted feature fusion or direct splicing is performed. After outputting multiple candidate safety hazard location frames and their corresponding classification probabilities, the predicted location of the safety hazard and its predicted classification are output based on non-maximum suppression screening.

[0089] Step S504: The preprocessed training sample set is used to train the initial neural network, the cross entropy loss function of the predicted classification and the true classification is used as the classification loss, and the mean square error loss between the predicted position and the true position is used as the positioning loss. The classification loss and positioning loss are combined to update the parameters of the initial neural network to obtain a railway periphery safety hazard identification model.

[0090] In step S501, the training samples can be required to cover railway scenes under different weather, lighting, and terrain conditions to ensure diversity. The sample railway remote sensing images can be in the form of optical remote sensing images and synthetic aperture radar (SAR) images. Furthermore, in step S502, the preprocessing of the sample railway remote sensing images can refer to steps S201-S203, steps S301-S303, and steps S401-S402.

[0091] In step S503, the dual-branch neural network optimizes computing power consumption by dividing the work between shallow and deep branches, taking into account both the local details and global semantic information of safety hazards in railway remote sensing imagery. The shallow branch adopts a lightweight design, using only a few convolutional layers and average pooling layers to quickly extract low-level features such as texture and edges of small targets around the track (such as scattered foreign objects and missing equipment parts), avoiding complex calculations. The deep branch progressively learns high-level semantic features such as track structure deformation and large obstacles through staged stacking of residual blocks, using residual connections to alleviate the gradient vanishing problem and ensure efficient training of the deep network. The two features are complemented through dynamic weighted fusion or channel splicing strategies. For example, an attention mechanism is introduced at the end of the deep branch to automatically filter key information of shallow details and deep semantics, suppressing redundant feature calculations. After fusion, 1×1 convolution is used to compress the channel dimension to reduce the computational load of the subsequent detection head. In addition, the dual-branch parallel processing architecture fully utilizes the GPU multi-stream computing capabilities, combining low-precision quantization of shallow branches (such as FP16) with hardware-aware compilation optimization of deep branches (such as TensorRT kernel acceleration). While reducing the amount of floating-point operations by approximately 15%, it improves the accuracy of small target detection by more than 3% through multi-scale feature collaboration, ultimately achieving a balance between real-time performance (such as 30+FPS) and accuracy (AP exceeding 72%) in model deployment, adapting to the dual requirements of low-computing power devices and robustness in complex environments in railway inspection scenarios.

[0092] During this process, sample railway remote sensing images are imported into the shallow and deep branches, followed by weighted feature fusion or direct stitching. The method can be selected based on the image quality of the scene. Feature stitching works best when features at different levels in the railway remote sensing data are clear, independent, and lack significant mutual interference. For example, in high-resolution remote sensing images acquired under clear weather, the linear features of the railway tracks and the semantic features of surrounding large buildings are clearly defined in the image, and feature stitching can effectively integrate these features. If the data is noisy, blurry, or features at different levels overlap, weighted fusion can adjust weights to highlight more reliable features. For example, in remote sensing images of railway lines acquired under heavy fog, some low-level features may become blurred. In this case, the weight α of the shallow branch can be appropriately reduced, while the weight β of the deep branch can be increased. This leverages the deep branch's ability to understand the overall semantics to compensate for the shortcomings of low-level features, thereby improving the model's accuracy in identifying safety hazards.

[0093] In some embodiments, the first convolutional layer in the shallow branch uses a 3×9 convolution kernel, and subsequent convolutional layers use a combination of 3×3 or 3×5 convolution kernels. The average pooling layer has a size of 2×2. The first convolutional layer uses a 3×9 convolution kernel to more effectively capture the characteristics of the railway track lines. Subsequent 3×3 or 3×5 convolution kernels extract low-level features such as texture and edges around the railway, such as the edge of the railway embankment and the texture of vegetation along the railway. The average pooling layer is used to reduce the resolution of the feature map and reduce the amount of computation.

[0094] Deep branching includes at least 4 stages, such as Figure 2 As shown in the figure, the structure of a single residual block in each stage consists of three consecutive convolutional layers. The input data of the residual block is jump-connected to the output, and the sizes of the convolution kernels are 1×1, 3×3, and 1×1, respectively. This can effectively extract high-level semantic features related to safety hazards in the railway environment.

[0095] In step S504, the classification loss is calculated as:

[0096]

[0097] Among them, y i represents the true classification, p i Represents the predicted classification;

[0098] The calculation formula for positioning loss is:

[0099]

[0100] in, and The horizontal and vertical coordinates represent the real position, and The horizontal and vertical coordinates representing the predicted position;

[0101] The formula for calculating the joint loss is:

[0102] L=L ce +λL nse ;

[0103] Where λ is the balance coefficient.

[0104] In some embodiments, the Adam optimizer is used to adaptively adjust the learning rate of each parameter. The Adam optimizer calculates the learning rate by:

[0105] Calculate the first-order moment estimate, the calculation formula is:

[0106] m t =β1m t-1 +(1-β1)g t ;

[0107] Calculate the second-order moment estimate, the calculation formula is:

[0108]

[0109] Then the learning rate calculation formula is:

[0110]

[0111] Among them, g t is the current gradient, β1 and β2 are the exponential decay rates of the moment estimates, θ is the learning rate, and ε is a small constant that prevents the denominator from being zero.

[0112] On the other hand, the present invention also provides a device for identifying safety hazards in the periphery of railways, comprising a processor, a memory, and a computer program / instruction stored in the memory, wherein the processor is used to execute the computer program / instruction, and when the computer program / instruction is executed, the device implements the steps of the above method.

[0113] On the other hand, the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of the above method when executed by a processor.

[0114] The present invention will be described below in conjunction with a specific embodiment:

[0115] This embodiment provides a dual-branch feature extraction network model. In specific scenarios surrounding railway environments, different branching strategies are selected for algorithm optimization based on different task characteristics, model complexity, and data specificity. This avoids inherent model defects and achieves high-quality identification of typical hidden dangers in railway environments. Specifically, the following steps are included:

[0116] 1. Data Collection and Preprocessing

[0117] Data Collection: Collect high-resolution remote sensing imagery, covering different time periods and seasons within the study area, to comprehensively reflect the surrounding environmental conditions. Also, collect geographic information data related to safety hazards, such as topography and land use types, to assist in subsequent analysis.

[0118] Data preprocessing: For railway remote sensing images, radiometric and geometric corrections are applied, and image cropping techniques are used to accurately extract the railway and surrounding areas within a certain range. For example, cropping is performed with the railway track as the center and extending 500 meters to either side. This removes irrelevant background information, reduces data processing, and preserves areas around the railway that may pose safety risks.

[0119] 2. Construct a dual-branch feature extraction network model

[0120] 2.1 Design shallow branch and deep branch structure:

[0121] (1) Shallow branch design: A convolutional neural network (CNN) structure of a LeNet5 variant is used to enhance the sensitivity to the linear features of the railway and the surrounding small facilities in the convolution layer design. The first convolution layer uses a slender convolution kernel (such as 3×9) to more effectively capture the characteristics of the railway track lines; the subsequent convolution layers use a combination of convolution kernels of different sizes (such as 3×3, 5×5) to extract low-level features such as texture and edges around the railway, such as the edge of the railway embankment and the texture of vegetation along the railway; then a 2×2 average pooling layer is used to reduce the resolution of the feature map and reduce the amount of calculation.

[0122] (2) Deep branch design: The ResNet50 structure is selected, which contains multiple residual modules. Each residual module consists of multiple convolutional layers, and the gradient vanishing problem during deep network training is solved through jump connections. In the residual module, the focus is on extracting the semantic features of railway infrastructure and the surrounding environment. In each residual module, the input feature map undergoes channel compression through 1×1 convolution, then extracts features through 3×3 convolution, and finally restores the number of channels through 1×1 convolution and adds it to the input feature map. The deep branch ResNet50 has a total of 4 stages, each of which contains multiple residual modules, which can effectively extract high-level semantic features related to safety hazards in the railway surrounding environment.

[0123] 2.2 Fusion strategy design:

[0124] (1) Feature splicing: After the last pooling layer of the shallow branch and the third stage of the deep branch, the feature maps of the two are spliced ​​according to the channel dimension. For railway scenarios, the low-level feature maps of railway tracks, small facilities, etc. output by the shallow branch are spliced ​​with the semantic feature maps of the railway surrounding environment output by the deep branch, and then connected to the fully connected layer or other convolutional layers for further fusion processing. For example, the output dimension of the shallow branch is [batch_size, C1, H, W], and the output dimension of the deep branch is [batch_size, C2, H, W]. The dimension after splicing is [batch_size, C1+C2, H, W], so that the model can comprehensively use high-level and low-level features to judge safety hazards around the railway.

[0125] (2) Weighted fusion: Different weights are assigned to the features of the two branches according to different task requirements. Let the shallow branch feature be F1 and the deep branch feature be F2, with weights α and β (α + β = 1), respectively. The fused feature F = α * F1 + β F2. The weights α and β can be optimized through experiments on the validation set, for example, using a grid search algorithm to try different combinations of α and β values ​​and select the combination that minimizes the validation set loss function as the final weight.

[0126] 2.3 Fusion strategy selection principles:

[0127] When deciding whether to choose feature concatenation or weighted fusion, consider the following three key aspects:

[0128] (1) Task characteristics

[0129] Feature splicing: This approach is applicable to situations where low-level features (such as track lines and small facility outlines) and high-level semantic features (such as surrounding building types and geological conditions) are equally important to the final judgment in railway safety hazard extraction tasks, and there is no obvious redundancy between the two. For example, when identifying whether there are newly constructed simple buildings along the railway that affect railway operation safety, shallow branches are needed to capture low-level features such as the building's edges to determine its location and general shape, while deep branches are needed to provide semantic information about the surrounding environment, such as whether the area is planned for railway use. The combination of the two can quickly and accurately determine hidden dangers, and this is when feature splicing can be fully utilized.

[0130] Weighted fusion: If different types of railway safety hazards have significantly different reliance on features at different levels, weighted fusion is more appropriate. For example, in detecting small hazards such as loose track fasteners, which primarily rely on detailed features extracted by shallow branches, increasing the weight α of the shallow branches allows the model to focus more on these details when fusing features, thereby enhancing its detection capabilities. For tasks such as determining the risk of large-scale landslides around railways, which require the integration of complex semantic information such as geology and topography, increasing the weight β of the deep branches allows the model to more effectively utilize deep semantic features for judgment.

[0131] (2) Model complexity and computing resources

[0132] Feature splicing: The operation is relatively simple and straightforward. After splicing, the feature dimensions increase to C1+C2. This increases the computational effort of subsequent network layers, but the calculation process is relatively intuitive. Feature splicing is a good choice when computing resources are sufficient and you want the model to fully utilize features from all levels without making subjective judgments about feature importance. For example, when conducting a large-scale railway safety hazard survey in a computing environment equipped with high-end GPUs, feature splicing can be used to allow the model to autonomously learn the patterns of feature combinations at different levels.

[0133] Weighted fusion: Although a weight optimization step (such as a grid search algorithm) is added, the feature dimensions after fusion are consistent with those of the original branches, and the increase in computational effort is relatively controllable. Weighted fusion is more advantageous when computing resources are limited and it is necessary to optimize the model's detection performance for different types of hidden dangers by adjusting weights without significantly increasing the computational effort. For example, when running a railway safety monitoring model on edge computing devices, weighted fusion can be used to reduce the computational burden. This method balances the utilization of features at different levels by properly setting weights.

[0134] (3) Data characteristics

[0135] Feature stitching works best when different layers of features in railway remote sensing data are clear, independent, and free of significant interference. For example, in high-resolution remote sensing imagery acquired under clear skies, the linear features of railway tracks and the semantic features of surrounding large buildings are clearly defined within the image. Feature stitching can effectively integrate these features.

[0136] Weighted fusion: If data is noisy, fuzzy, or features at different levels overlap, weighted fusion can adjust weights to highlight more reliable features. For example, in remote sensing images of a railway line acquired in foggy weather, some low-level features may become blurred. In this case, the weights α of shallow branches can be appropriately reduced, while the weights β of deep branches can be increased. The deep branches' ability to understand the overall semantics can compensate for the shortcomings of low-level features and improve the model's accuracy in identifying safety hazards.

[0137] 3. Model Training

[0138] 3.1 Sample Annotation: Manually annotate pre-processed remote sensing images, marking the locations and types of various safety hazards, such as color-coated steel tiles, greenhouses, ground film, and dust screens, to construct a training sample set. Accuracy and consistency must be ensured during annotation to provide reliable supervisory information for model training.

[0139] 3.2 Select loss function and optimizer:

[0140] (1) Loss function: Since the extraction of safety hazards around railways involves classification (determining the type of hazard) and positioning (determining the location of the hazard), a multi-task loss function is used. The classification part uses the cross entropy loss function. Let the true label be y and the predicted probability distribution be p. The cross entropy loss is:

[0141]

[0142] The positioning part uses the mean square error loss function, assuming the true position coordinates are:

[0143] (x gt ,y gt )

[0144] The predicted position coordinates are:

[0145] (x pred ,y pred )

[0146] Mean squared error loss:

[0147]

[0148] Total loss function:

[0149] L=L ce +λL mse

[0150] Where λ is the balance coefficient, and its value is determined through experiments.

[0151] (2) Optimizer: The Adam optimizer is used, which adaptively adjusts the learning rate of each parameter. When calculating the gradient, the Adam optimizer calculates the first-order moment estimate (mean) of the gradient separately:

[0152] m t =β1m t-1 +(1-β1)g t

[0153] And the second moment estimate (unnormalized variance):

[0154]

[0155] And use these estimates to dynamically adjust the learning rate of each parameter and update the parameters according to the update formula.

[0156]

[0157] Where gt is the current gradient, β1 and β2 are the exponential decay rates of the moment estimates, usually set to 0.9 and 0.999, θ is the learning rate, and ε is a small constant that prevents the denominator from being zero.

[0158] 3.3 Set Training Parameters: Determine parameters such as the number of training epochs and batch size. Adjust these parameters experimentally to balance model training time and performance, preventing overfitting and underfitting. In complex environments along railway lines, appropriately increase the number of training epochs to improve the model's adaptability to different scenarios. During training, regularly verify the model's performance on railway-related validation sets, observing changes in metrics such as loss function value and accuracy, and adjust the training strategy accordingly.

[0159] 4. Model Evaluation and Optimization

[0160] 4.1 Evaluation Metric Selection: Use multiple evaluation metrics to comprehensively evaluate model performance, such as accuracy, recall, and F1 score. Recall is particularly important for security risk extraction tasks, ensuring that as many actual security risks as possible are detected.

[0161] Accuracy:

[0162]

[0163] Where TP is the true positive example (the number of samples correctly predicted as positive), FP is the false positive example (the number of samples incorrectly predicted as positive), FN is the false negative example (the number of samples incorrectly predicted as negative), and TN is the true negative example (the number of samples correctly predicted as negative).

[0164] Recall:

[0165]

[0166] F1 value:

[0167]

[0168] in

[0169]

[0170] 4.3 Model Optimization: Based on the evaluation results, analyze the model's performance in extracting different types of safety hazards. If the model performs poorly for certain types of hazards, try adjusting the network structure, such as increasing or decreasing the number of network layers or adjusting the convolution kernel size. Alternatively, enhance the training data, such as by rotating, scaling, or adding noise, to improve the model's generalization capabilities.

[0171] (1) Network structure adjustment: If the model performs poorly on small-target safety hazard detection, add convolutional layers to the shallow branches to improve the ability to extract detailed features. If the model is inaccurate in identifying hazards in complex scenarios, add residual modules to the deep branches to deepen the network's understanding of complex semantics. After adjustment, retrain the model and evaluate its performance on the validation set.

[0172] (2) Data augmentation: For railway scenes, the OpenCV library is used to perform data augmentation operations. For example, railway remote sensing images are randomly rotated (the rotation angle range is set to [5,5] degrees to simulate different shooting angles), randomly scaled (the scaling ratio range is [0.9,1.1] to consider images with different resolutions), and Gaussian noise is added (the noise mean is set to 0 and the standard deviation is set to 0.01 to simulate the noise interference in actual shooting). By increasing the diversity of training data through data augmentation, the model can learn more generalized characteristics of safety hazards around railways.

[0173] 5. Safety hazard extraction

[0174] 5.1 Loading the trained model: Load the evaluated and optimized dual-branch feature extraction network model into the railway surrounding environment processing system.

[0175] 5.2 Input the remote sensing image to be tested: The remote sensing image along the railway to be tested is preprocessed in the same way as the training data, including radiation correction, geometric correction, and image cropping (cropping a specific range centered on the railway), and then input into the model.

[0176] 5.3 Output results and post-processing:

[0177] (1) Model output: The model outputs the predicted location and type of safety hazards around railways, representing the probability of each pixel belonging to a different type of railway safety hazard in the form of a probability distribution. For example, the output is a three-dimensional array with dimensions [H, W, num_classes], where H and W are the image height and width, and num_classes is the number of railway safety hazard types, such as color-coated steel tiles, greenhouses, ground film, and dust screens.

[0178] (2) Post-processing: The non-maximum suppression (NMS) algorithm is used to remove small areas of false detection in the railway scene. For railway safety hazards, an appropriate confidence threshold (such as 0.6) is set, and prediction results below the threshold are discarded. For the remaining prediction boxes, their overlap is calculated. When the overlap is greater than the set threshold (such as 0.4), the prediction boxes with high confidence are retained and the overlapping low-confidence boxes are removed. The expansion function in the OpenCV library (the kernel size is set to 3×3) is used to perform an expansion operation on the railway hazard detection results to connect adjacent small target areas (such as adjacent small foreign objects on the railway track); then the erosion function (the kernel size is also 3×3) is used to remove isolated small areas caused by noise. Finally, accurate results of railway surrounding environment safety hazard extraction are obtained and displayed in a visual way on the railway remote sensing image, such as marking different types of hazards with different colors, providing an intuitive basis for the decision-making of railway operation management departments.

[0179] Corresponding to the above method, the present invention also provides an apparatus / system, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the apparatus / system implements the steps of the method described above.

[0180] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the art.

[0181] In summary, the method and device for identifying safety hazards in the vicinity of railways described in the present invention introduce a dual-branch feature extraction network. On the one hand, multiple convolutional layers and average pooling layers based on shallow branches extract rail line features and low-level features such as texture and edges around the railway. On the other hand, a jump connection structure based on multiple residual modules is introduced into deep branches, focusing on the extraction of semantic features of railway infrastructure and surrounding environment. By weighted fusion or direct splicing of the features of shallow branches and deep branches, the demand for deep structures can be reduced while ensuring recognition accuracy, and the dependence on computing resources can be reduced.

[0182] Furthermore, according to the characteristics of remote sensing data in different railway scenarios, the sample railway remote sensing images are respectively imported into the shallow branch and the deep branch and then weighted feature fusion or direct splicing is performed, and different proportions are set to effectively focus on reliable features between features at each level.

[0183] It should be understood by those skilled in the art that the various exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0184] It should be understood that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted. In the above embodiments, several specific steps are described and illustrated as examples. However, the method of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art may make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present invention.

[0185] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0186] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations to the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for identifying potential safety hazards around railways, characterized in that: The method comprises the following steps: Acquire a remote sensing image of the railway to be inspected, and preprocess the remote sensing image, wherein the preprocessing includes radiation correction, geometric correction, and image cropping within a set range along the railway; The pre-processed railway remote sensing image is input into a pre-trained railway surrounding safety hazard identification model, and a preliminary identification result of the safety hazard location and type is output. The preliminary identification result is filtered based on non-maximum suppression to obtain a target recognition result; wherein, the safety hazard identification model includes two parts: a shallow branch and a deep branch. The shallow branch includes multiple continuous convolutional layers and average pooling layers, and the deep branch includes multiple continuous stages, each stage is composed of multiple residual block layers, and the output of the average pooling layer is connected to the input of the last stage.

2. The method for identifying potential safety hazards around railways according to claim 1, characterized in that: The radiation correction in the preprocessing includes: The original digital quantized value is converted into an apparent reflectivity with physical meaning based on sensor radiation calibration; Atmospheric correction is performed based on dark pixel subtraction to eliminate the effects of atmospheric scattering, absorption, and aerosols to obtain the true reflectance of the surface; Perform a terrain correction to remove shadows and radiation differences caused by terrain relief.

3. The method for identifying potential safety hazards around railways according to claim 2, characterized in that: The geometric correction in the preprocessing includes: Obtain a reference image and use the radiometrically corrected remote sensing image as the target influence, and use image registration tools to generate homonymous control points; Low-order polynomial fitting of geometric distortion is used to correct for flat terrain areas, and digital elevation models are used to correct for undulating terrain areas. Use bilinear interpolation or cubic convolution to resample the image.

4. The method for identifying potential safety hazards around railways according to claim 3, characterized in that: Image cropping for a set range along the railway line, including: Pre-dividing the railway line into supervisory areas according to preset railway safety management standards, and establishing boundary vectors for the supervisory areas based on a geographic information system; A buffer zone is set for the boundary vector of the regulatory area according to a preset distance, and the buffer zone range is used as a mask to crop the pre-processed railway remote sensing image to retain the target area.

5. The method for identifying potential safety hazards around railways according to claim 1, characterized in that: The pre-training steps of the railway surrounding safety hazard identification model include: Obtaining a training sample set, which includes multiple samples, each sample includes a sample railway remote sensing image, and marking the real location and real classification of the safety hazard in the sample railway remote sensing image as a label; Preprocessing the sample railway remote sensing image, wherein the preprocessing includes radiation correction, geometric correction, and image cropping within a set range along the railway; An initial neural network is obtained, comprising a shallow branch and a deep branch, wherein the shallow branch comprises a plurality of consecutive convolutional layers and an average pooling layer, and the deep branch comprises a plurality of consecutive stages, each stage being composed of a plurality of residual blocks, and the output of the average pooling layer is connected to the input of the last stage; the sample railway remote sensing image is respectively imported into the shallow branch and the deep branch, and then weighted feature fusion or direct splicing is performed, and multiple candidate safety hazard location frames and their corresponding classification probabilities are output, and then the predicted location and predicted classification of the safety hazard are output based on non-maximum suppression screening; The preprocessed training sample set is used to train the initial neural network, the cross entropy loss function of the predicted classification and the true classification is used as the classification loss, the mean square error loss between the predicted position and the true position is used as the positioning loss, and the classification loss and the positioning loss are combined to update the parameters of the initial neural network to obtain the railway periphery safety hazard identification model.

6. The method for identifying potential safety hazards around railways according to claim 5, characterized in that: The first convolution layer in the shallow branch uses a convolution kernel of size 3×9, and the subsequent convolution layers use a combination of convolution kernels of size 3×3 or 3×5. The size of the average pooling layer is 2×2. The deep branch includes at least 4 stages, and the structure of the residual block in each stage includes three consecutive convolutional layers, and the sizes of the convolution kernels are 1×1, 3×3 and 1×1 respectively.

7. The method for identifying potential safety hazards around railways according to claim 6, characterized in that: The calculation formula of the classification loss is: Among them, y i represents the true classification, p i represents the predicted classification; The calculation formula of the positioning loss is: in, and represents the abscissa and ordinate of the real position, and representing the abscissa and ordinate of the predicted position; The formula for calculating the joint loss is: L=L ce +λL mse ; Where λ is the balance coefficient.

8. The method for identifying potential safety hazards around railways according to claim 7, characterized in that: The Adam optimizer is used to adaptively adjust the learning rate of each parameter. The Adam optimizer includes: Calculate the first-order moment estimate, the calculation formula is: m t =β1m t-1 +(1-β1)g t ; Calculate the second-order moment estimate, the calculation formula is: Then the learning rate calculation formula is: Among them, g t is the current gradient, β1 and β2 are the exponential decay rates of the moment estimates, θ is the learning rate, and ε is a small constant that prevents the denominator from being zero.

9. A device for identifying safety hazards around railways, comprising a processor, a memory, and a computer program / instruction stored in the memory, characterized in that: The processor is configured to execute the computer program / instructions. When the computer program / instructions are executed, the device implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images