Railway environment unknown risk identification method for unmanned aerial vehicle remote sensing image

Abstract

The invention provides a railway environment unknown risk identification method for unmanned aerial vehicle remote sensing images, and the method designs a target and multi-classification decoupling training strategy based on CNN, and improves the image foreground and background distinguishing capability of a model. An unmanned aerial vehicle remote sensing image set along a railway and an existing unknown risk image and other scene unmanned aerial vehicle image set are used as training data of a model, and while an in-distribution data set is normally trained, feature sets with dynamic elements are constructed for in-distribution target features and out-of-distribution target features respectively; and performing multivariate Gaussian distribution parameter estimation on the feature set, and performing similarity-based feature resampling on the basis of parameter estimation to obtain the OoD features obeying the distribution of the feature set, so that the resampled OoD features have higher generalization. And finally realizing OoD detection in combination with an energy score function. And the generalization of unknown risk identification is improved while the high target detection rate of potential hidden dangers is ensured.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle control information technology, and in particular to a method for identifying unknown risks in railway environment using unmanned aerial vehicle remote sensing images. Background Art

[0002] As of September 2024, China's railway operating mileage reached 160,000 kilometers, including 49,000 kilometers of high-speed railways. In recent years, with the rapid expansion of the high-speed railway network, potential environmental hazards around the tracks have posed a serious threat to the reliability and safety of railway operations. Common potential hazards that frequently occur along the railway include, but are not limited to, temporary buildings, plastic films, and garbage. These hidden dangers are like time bombs that are likely to invade the railway track area under the influence of strong winds. Typical examples of foreign body intrusion, such as the roof of a temporary building invading the track area, the contact wire of the overhead contact network with plastic films hanging, and the collision of a moving train with a color steel tile, may cause serious accidents and even catastrophic derailment of the vehicle. In addition, risks such as mudslides, falling rocks, and human / animal intrusions, although they occur less frequently, are very likely to cause major accidents with vehicle destruction and death and huge economic losses. Therefore, it is necessary to conduct frequent inspections along the railway to minimize safety hazards.

[0003] Traditional railway environmental inspections mainly rely on personnel walking near the railway line to perform inspections. This method is labor-intensive, inefficient, and heavily dependent on the subjective judgment of inspectors. In particular, current visual inspections are severely limited by terrain and can hardly meet the inspection needs of large-scale lines and remote areas. Fortunately, railway infrastructure inspections based on drone photography have recently received widespread attention (e.g., tracks, sound barriers, and contact lines). Naturally, the use of drone images to provide a large field of view and fast cruising speed has great potential to make up for the shortcomings of current manual environmental inspections. Despite this, manually browsing a large number of drone images is still time-consuming and laborious. Therefore, there is an urgent need to develop a visual system to automatically detect common hidden dangers and unknown risks with extremely low occurrence frequencies in drone railway environmental images.

[0004] Thanks to the rapid development of computer vision automatic detection technology, some recent studies have explored the feasibility of intelligent inspection of railway environment based on drones combined with visual methods. However, the common hidden danger identification methods of drone remote sensing images of railway environment based on end-to-end convolutional neural networks are limited by the significant difference in the scale of hidden danger targets and limited semantic information; in addition, the identification of unknown risks along the railway is limited by the sparsity and random uncertainty of abnormal samples. As a result, the accuracy of visual automatic detection of environmental risks along the railway is greatly reduced. Moreover, the existing unknown anomaly recognition methods mainly focus on anomaly detection outside the known data distribution. While improving the performance of unknown anomaly recognition, they cannot guarantee a high-precision recognition rate for known data distribution, which makes such methods unable to be directly applied to the risk and hidden danger identification of the railway environment.

[0005] In summary, this patent focuses on achieving high-precision identification of common railway environmental hazards and improving the generalization of unknown risk target identification. The results of this patent can improve the detection efficiency of known / unknown risks in the railway environment, help improve the safety level of the railway operating environment, ensure the safe operation and smooth operation of the railway, and lay the foundation for all-weather and large-scale environmental status perception along the line.

[0006] Prior art

[0007] (1) Current status of research on risk identification of railway UAV remote sensing images

[0008] With the remarkable progress of CNN, many efforts have been made to use drone images for automatic identification or health monitoring of railway infrastructure and its environment. For example, Tong et al. [9] proposed a semantic segmentation convolutional network for railway scene parsing, which adopted a fully decoupled residual block to meet the high real-time requirements under limited resources. Subsequently, a novel dual-branch architecture ARTNet

[14] was designed to solve the problem of full-angle detection of railway tracks. Unfortunately, these two methods only focus on the identification of track areas and track lines from the perspective of drones and cannot achieve anomaly detection. In terms of structural health monitoring, Mu et al.

[15] developed a shallow attention network combined with an adaptive cropping strategy for the detection of bolts, steel surfaces, and railings in railway bridges. Cui et al. [4] proposed a skip-YOLO detection network for detecting various defects on the surface of high-speed railway sound barriers. Chen et al. [6] constructed an integrated framework integrating detection and segmentation branches for anomaly detection of tracks and other track components. Although the existing automatic detection methods based on UAV remote sensing images have achieved good results, it is still difficult to accurately detect environmental hazards around railways when semantic information is limited and the target scales vary significantly. In particular, it is impossible to complete the risk level assessment of potential hazards along the line. Fortunately, YOLARC[8], as a pioneering work, solved the risk assessment problem, but the model still has shortcomings: YOLARC uses a simple ResNet and FPN structure to extract and fuse feature information, which may hinder the network from mining key discriminant features and cannot effectively transmit information.

[0009] (2) Current status of research on unknown anomaly identification

[0010] At present, the research on unknown anomaly recognition is mainly divided into unsupervised anomaly detection, out-of-distribution (OoD) anomaly recognition, and open world object detection (OWOD). Unsupervised anomaly detection [16, 17, 18, 19] mainly adopts the architecture of encoder and decoder to learn the latent feature space distribution of normal data to identify the abnormal area of ​​test data samples. However, this method is extremely sensitive to data and requires single sample background information. It is particularly suitable for anomaly detection of industrial parts

[20] , but cannot realize the recognition of unknown anomalies in the environmental image data along the railway with complex background information and multiple disturbances. In terms of out-of-distribution anomaly recognition, Liu et al.

[21] proposed out-of-distribution anomaly recognition based on energy score loss, and proved that energy score has better out-of-distribution anomaly recognition ability than traditional Softmax function; Ming et al.

[22] added external abnormal data in the model training stage based on energy score loss to better induce the model to reduce the fuzziness of the boundary between the in-distribution (iD) and the out-of-distribution, thereby improving the model OoD recognition performance. Unfortunately, these methods focus on image-level classification tasks and cannot achieve instance-level object detection. Therefore, the latest research [12,13] refines the energy score-based model from the image level to the instance level and achieves instance-level OoD object detection. Since these methods only use iD data for model training and use simulated OoD features to realize anomaly recognition in test data, the model OoD recognition performance is insufficient and there is still room for improvement. In terms of unknown anomaly recognition based on OWOD, most studies [11,23,24,25] use incremental learning strategies to gradually improve the model's ability to recognize unknown anomalies through multi-stage model training. Among them, studies [23,24] cannot achieve a high unknown anomaly recognition index (~10% recall rate). These methods mistakenly classify most unknown anomaly targets as background, resulting in low recall rate. Therefore, OW-DETR

[25] uses a pseudo-label mechanism to assign the part with high feature activation extracted by the backbone as the unknown target label; PROB

[11] focuses on improving the model's ability to distinguish between foreground and background to improve the recognition of unknown targets. Unfortunately, while this type of method uses incremental learning to gradually improve the recognition performance of unknown anomalies, it cannot guarantee the stability of the recognition performance of known category targets.

[0011] The prior art has the following problems:

[0012] (1) Convolutional Neural Network (CNN), which is based on computer vision theory, can achieve high-precision detection of potential hazards in railway environments after a large amount of sample training and feature learning. However, when faced with unknown risks with sparse samples and significant feature differences between different risks, CNN cannot achieve accurate target detection.

[0013] (2) The CNN method can realize target detection of a limited number of known categories. However, the number of unknown risk categories can be infinite. Faced with infinite categories of unknown risks, conventional CNN-based target detection methods are powerless.

[0014] (3) CNN-based out-of-distribution object detection and open-world object detection methods can realize the recognition and detection of unknown categories, but when faced with the task of detecting known and unknown categories, it is impossible to guarantee that both can achieve accurate detection effects at the same time. Summary of the invention

[0015] In order to solve the above technical problems, the present invention designs a targeted and multi-classification decoupling training strategy based on CNN, improves the model's ability to distinguish image foreground and background, and uses the drone remote sensing image set of the railway environment and the existing unknown risk images and drone image sets of other scenes as the training data of the model. While training the data set within the distribution normally, a feature set with dynamic elements is constructed for the target features within the distribution and the target features outside the distribution, and the multivariate Gaussian distribution parameters of the feature set are estimated. Based on the parameter estimation, feature resampling based on similarity is performed to obtain OoD features that obey the distribution of the feature set, so that the resampled OoD features are more generalized. Finally, OoD detection is realized by combining the energy scoring function. While ensuring a high target detection rate for potential hidden dangers, the generalization of unknown risk identification is improved. The present invention specifically adopts the following technical solutions:

[0016] A method for identifying unknown risks in railway environment using UAV remote sensing images, the method comprising the following steps:

[0017] Step 1: First, data annotation is performed to produce a set of drone remote sensing images of the railway environment, unknown risk images, and drone images of other scenes;

[0018] Step 2: Based on the two prepared data sets, the network model is sent to training to learn the common hidden dangers identification in the UAV remote sensing image set of the railway environment, the unknown risk images and the unknown risks identification in the UAV image set of other scenes, and the model weights are saved after the training.

[0019] Step 3: Load the trained model weights and UAV remote sensing images of the railway environment, perform location detection and category recognition of common hidden dangers and unknown risks in the images, mark them in the original images, and output the result map.

[0020] Preferably, the step 1 specifically includes the following steps:

[0021] Firstly, the common hidden dangers data of the UAV remote sensing image set along the railway are annotated and the common hidden dangers are marked into corresponding categories;

[0022] For unknown risk images and other scene drone image sets, various targets that appear are uniformly annotated as a label independent of common hidden danger categories;

[0023] The acquired UAV remote sensing image sets of the railway environment, unknown risk images and other scene UAV image sets and their corresponding labels are input into the convolutional neural network for training.

[0024] Preferably, step 2 specifically includes the following steps:

[0025] (2.1) Model training step 1: Obtain complete UAV remote sensing image features of the railway environment, unknown risk images, and other scene UAV image features through feature extraction and feature fusion;

[0026] (2.2) Model training step 2: The complete image features are processed by convolution to obtain the coordinates of the bounding box that may be the target in the image, and the target loss function 1 and the target bounding box loss function 1 are calculated. The target loss function 1 is a binary classification loss function based on cross entropy, which realizes the binary classification loss value calculation of the image predicted background and the real background. The target bounding box loss function 1 is a loss function based on the intersection of the predicted and real bounding boxes, which realizes the intersection of the predicted box and the real box.

[0027] (2.3) Model training step 3: Combine the target bounding box coordinates in model training step 2 to extract the target features of the corresponding part on the complete image features. The target features in the UAV remote sensing image features of the railway environment are regarded as in-distribution target features, and the target features in the UAV image features of unknown risk images and other scenes are regarded as out-distribution target features.

[0028] (2.4) Model training step 4: Combine the in-distribution and out-of-distribution target features obtained in model training step 3 with the corresponding bounding box coordinates obtained in model training step 2, and calculate the target loss function 2 and the target bounding box loss function 2 through convolution processing. The target loss function 2 and the target bounding box loss function 2 are consistent with the target loss function 1 and the target bounding box loss function 1 in form, respectively;

[0029] (2.5) Model training step 5: Set the model training iteration number i1 in advance. This step is executed when the current training iteration number is greater than i1. The target features in the distribution obtained in the model training step 3 are combined with the corresponding bounding box coordinates obtained in the model training step 2, and the multi-classification loss function is calculated through convolution processing; the multi-classification loss function is a multi-classification loss function based on cross entropy, which realizes the calculation of loss values ​​of specific multiple categories of predicted and real common hidden dangers;

[0030] (2.6) Model training step 6: The target features within the distribution are stored in a feature set according to the specific common hidden danger categories. If the number of target features in the set exceeds n, the features that entered the set first are removed before being stored. The target features outside the distribution are stored in a feature set. If the number of elements in the set exceeds n, the features that entered the set first are removed before being stored. The set is expressed as the formula:

[0031]

[0032] in, is a storage set of the kth common hidden dangers of the target features within the distribution, Seq OoD is a storage collection of target features within the distribution. is the target feature of the nth distribution of the kth common hidden danger stored in the corresponding set, K is the total number of common hidden danger categories, It is the nth out-of-distribution target feature of the unknown risk stored in the corresponding set; the set will be used as a sample set for subsequent parameter estimation of multivariate Gaussian distribution;

[0033] (2.7) Model training step 7: Set the model training iteration number i2 in advance. This step is executed when the training iteration number is greater than i2, i2 ≥ i1. Multivariate Gaussian distribution parameter estimation is performed on the current K+1 feature sets respectively, and OoD features are obtained by sampling according to the similarity sampling strategy; the target features within the distribution are regarded as iD features; labels are assigned to iD features and OoD features respectively;

[0034] The multivariate Gaussian distribution parameter estimation process includes estimating the mean μ and variance σ 2 , the specific calculation process is as follows, for the K distribution target feature sets Get the mean of each feature set respectively As the mean parameter of the multivariate Gaussian distribution of each category, calculate the covariance of all target features in the K feature set Similarly, the target feature set Seq outside the distribution OoD Get its mean μ OoD As the mean parameter of the multivariate Gaussian distribution, Seq OoD Calculate the covariance of all target features in

[0035] The feature resampling strategy based on similarity is described as follows: The multivariate Gaussian distribution parameters of the target feature set under the distribution of K common hidden danger categories are obtained, and the corresponding probability distribution can be expressed as:

[0036]

[0037] Among them, h(x iD ,b iD ) are common hidden danger target features in the complete UAV remote sensing images of the railway environment;

[0038] The obtained multivariate Gaussian distribution of the out-of-distribution target feature set can be expressed as follows for the corresponding probability distribution:

[0039]

[0040] h(x OoD ,b OoD ) is the complete unknown risk image and the unknown risk target features in other scene drone images;

[0041] On the basis of the probability distribution, resampling is performed in the K+1 target feature sets respectively. The resampling process uses inverse Cholesky decomposition to decompose their respective covariances, namely:

[0042]

[0043] Among them, L iD and L OoD is a lower triangular matrix that generates a sample vector from an independent standard normal distribution and z OoD , get the sample:

[0044]

[0045] s OoD =μ OoD +L OoD z OoD

[0046] For K+1 samplings, the number of samples is set to a larger value m each time to ensure that the number of samples is large enough to better cover the distribution space of the feature set to be obeyed;

[0047] (2.8) Model training step 8: This step is executed when the current training iteration number is greater than i2, i2 ≥ i1; the iD feature and the OoD feature are convolved by the classification branch in the model prediction head, and the energy score is calculated on the processed result. After performing full connection processing on the two parts of the feature, the binary loss function is calculated in combination with the labels assigned in step (2.8), where the binary loss function is a binary classification loss based on the cross entropy loss;

[0048] The energy score calculation process is as follows:

[0049]

[0050] Among them, s max is a single sample s as the maximum value in the matrix, conv k (·) represents the mapping output result of the classification branch in the model prediction head for the kth common hidden danger category;

[0051] (2.9) Model training step 9: Target loss function 1, target bounding box loss function 1, target loss function 2, target bounding box loss function 2, multi-classification loss function, and binary loss function are weighted to obtain the total loss calculation, and the model gradient is back-propagated and the model gradient is cleared based on the current total loss;

[0052] (10) Model training step 10: Set the model training iteration number i3 in advance. When the current training iteration number is greater than i3, (i3>i2≥i1), the model training stops and the model weights are saved. Otherwise, the model training steps 1 to 9 are iteratively executed.

[0053] Preferably, step 3 specifically includes the following steps:

[0054] (3.1) Model inference step 1: read the saved model weights, load the image, perform feature extraction and feature fusion to obtain complete image features;

[0055] (3.2) Model reasoning step 2: The complete image features are processed through convolution to obtain the possible target bounding box coordinates and target probability in the image, and the non-maximum suppression algorithm is used to delete redundant bounding boxes based on the target probability and possible bounding box coordinates;

[0056] (3.3) Model reasoning step 3: Combine the predicted bounding box coordinates obtained in model reasoning step 2 to extract the target features of the corresponding part on the complete image features;

[0057] (3.4) Model reasoning step 4: Combine the target features obtained in model reasoning step 3 with the corresponding bounding box coordinates obtained in model reasoning step 2, and obtain the category probabilities and prediction box coordinates including common hidden danger classifications through convolution processing. Based on the category probabilities and prediction box coordinates, perform a category-independent non-maximum suppression algorithm to delete redundant bounding boxes;

[0058] (3.5) Model reasoning step 5: After the target features obtained in model reasoning step 3 are processed by the model prediction head classification branch, the energy score calculation and full connection processing are used to determine whether the target is an out-of-distribution target; if so, the target category is directly determined as OoD; if not, the target category is determined to be the category with the highest probability among the common hidden danger categories;

[0059] (3.6) Model reasoning step 6: Mark the final target category and prediction box coordinates in the original image. If it is a common hidden danger category, additionally mark the corresponding category probability; output the model's prediction map.

[0060] The present invention has the following beneficial effects:

[0061] (1) The model can ensure that it has a high recognition accuracy for common hidden dangers in UAV remote sensing images along the railway. The average precision (AP) can reach 95.7% when IoU ≥ 50%. Figure 2 shown.

[0062] (2) At the same time, it can achieve good recognition results for unknown risks in the image, and there is basically no situation where common hidden dangers and unknown risks are misclassified (such as Figure 3 and Figure 4 ), and existing technologies will, to a certain extent, identify common hidden dangers as unknown risks (such as Figure 4 As shown in the figure, unknown risks may be mistakenly classified as common hidden dangers (such as Figure 5 shown). BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 Basic flow chart of the present invention.

[0064] Figure 2 The performance results of the present invention on average precision (AP) and average recall (recall).

[0065] Figure 3 The present invention is an effect diagram of identifying common hidden dangers.

[0066] Figure 4 The effect diagram of the present invention on identifying unknown risks.

[0067] Figure 5 The existing technology misclassifies common hidden dangers into unknown risk maps.

[0068] Figure 6 The existing technology misclassifies unknown risks into common hidden danger maps. DETAILED DESCRIPTION

[0069] Terminology explanation:

[0070] (1) Potential hazards: refers to hazards that frequently occur in the environment around the railway, including but not limited to garbage accumulation, plastic film, color steel tiles, plastic greenhouses, etc. These hazards are called potential hazards because they will not affect the operation of the railway under natural conditions, but may cause potential hazards to invade the railway limits under severe conditions (such as strong winds).

[0071] (2) Unknown risks: refers to risks that rarely occur in the environment surrounding the railway, including but not limited to mudslides, falling rocks, human or animal intrusion, etc., which, if they occur, will inevitably threaten the safe operation of the railway.

[0072] (3) Target recognition: refers to the target detection method based on convolutional neural network, which only considers which parts of the image are targets and which are not, without considering what category the target belongs to, that is, only binary recognition of the foreground and background of the image is performed.

[0073] (4) Known / in-distribution target features: After the image is extracted and fused by the Convolution Neural Network (CNN), the features of the known / in-distribution targets in the complete image area refer to the features of common hidden danger targets.

[0074] (5) Unknown / out-of-distribution target features: After the image is extracted and fused by CNN, the features of unknown / out-of-distribution targets in the complete image area refer to the features of unknown risk targets.

[0075] The present invention designs a targeted and multi-classification decoupling training strategy based on CNN to improve the model's ability to distinguish between image foreground and background, and avoid the unknown risk targets that cannot be identified due to the misclassification of unknown risks as background. According to the OoDOD and OWOD methods based on CNN, potential hidden dangers with abundant samples and frequent occurrences in the environment along the railway are regarded as known categories or data within the distribution, including garbage accumulation, plastic film, color steel tiles, asbestos tiles, iron houses, and masonry houses; unknown risks that rarely appear in the environment along the railway are regarded as unknown categories or data outside the distribution, including mudslides, falling rocks, animal invasions, etc. Here, unknown categories or data outside the distribution are not divided into specific categories in model recognition, and are uniformly identified as OoD. The present invention uses the drone remote sensing image set of the railway environment (containing abundant potential hidden danger samples, regarded as the in-distribution data set) and the existing unknown risk images and other scene drone image sets (regarded as the out-distribution data set) as the training data of the model. While the normal training (specific category detection of various potential hidden dangers) is performed on the in-distribution data set, a feature set with dynamic elements is constructed for the in-distribution target features and the out-distribution target features respectively, and the multivariate Gaussian distribution parameter estimation is performed on the feature set. Based on the parameter estimation, the feature resampling based on similarity is performed to obtain the OoD feature that obeys the distribution of the feature set, so that the resampled OoD feature is more generalized, instead of directly regarding the out-distribution target feature as the OoD feature. Finally, the energy scoring function is combined to realize OoD detection. While ensuring a high target detection rate for potential hidden dangers (known categories / in-distribution data), the generalization of the recognition of unknown risks (unknown categories / out-distribution data) is improved.

[0076] The specific steps of the method of the present invention are as follows:

[0077] Step 1: First, data annotation is performed to produce a set of drone remote sensing images of the railway environment, unknown risk images, and drone images of other scenes;

[0078] Step 2: Based on the two prepared data sets, the network model is sent to training to learn the recognition of common hidden dangers (data within the distribution) in the remote sensing image set of UAVs along the railway environment, and the recognition of unknown risks (data outside the distribution) in the UAV image set of other scenes. After the training, the model weights are saved.

[0079] Step 3: Load the trained model weights and UAV remote sensing images of the railway environment, perform location detection and category recognition of common hidden dangers and unknown risks in the images, mark them in the original images, and output the result map.

[0080] The purpose of the present invention is to ensure that the model has a high recognition rate for common hidden dangers appearing in UAV images of railway environments, while being able to significantly improve the recognition ability and generalization of unknown risks. In order to fully mine the characteristic information of unknown risk samples under sparse and limited unknown risk samples, the present invention decides to add a set of unknown risk images and other scene drone images while producing a set of railway environment drone remote sensing images. The two data sets are jointly sent to the model training to ensure that the unknown risk images and other scene drone images do not contain common hidden danger targets in the railway environment drone remote sensing images.

[0081] The step 1 is specifically as follows:

[0082] Firstly, the common hidden dangers data of the UAV remote sensing image set along the railway are annotated and the common hidden dangers are marked into corresponding categories;

[0083] For unknown risk images and other scene drone image sets (not containing common hidden danger targets), all the targets that appear (regardless of category subdivision) are uniformly labeled as a label independent of the common hidden danger category;

[0084] The acquired UAV remote sensing image sets of the railway environment, unknown risk images and other scene UAV image sets and their corresponding labels are input into the convolutional neural network for training.

[0085] Step 2 is the model training phase step, specifically:

[0086] (1) Model training step 1: Obtain complete UAV remote sensing image features of the railway environment, unknown risk images, and other scene UAV image features through feature extraction and feature fusion;

[0087] (2) Model training step 2: The complete image features are processed by convolution to obtain the coordinates of the bounding box that may be the target in the image, and the target loss function 1 and the target bounding box loss function 1 are calculated. The target loss function 1 is a binary classification loss function based on cross entropy, which realizes the binary classification loss value calculation of the image predicted background and the real background, and the target bounding box loss function 1 is a loss function based on the intersection over union (IoU) loss of the predicted and real bounding boxes, which realizes the loss value calculation of the intersection over union (IoU) loss of the predicted box and the real box;

[0088] (3) Model training step 3: Combine the target bounding box coordinates in model training step 2 to extract the target features of the corresponding part on the complete image features. The target features in the UAV remote sensing image features of the railway environment are regarded as in-distribution target features, and the target features in the UAV image features of unknown risk images and other scenes are regarded as out-distribution target features;

[0089] (4) Model training step 4: The in-distribution and out-of-distribution target features obtained in model training step 3 are combined with the corresponding bounding box coordinates obtained in model training step 2, and the target loss function 2 and the target bounding box loss function 2 are calculated through convolution processing. The target loss function 2 and the target bounding box loss function 2 are consistent with the target loss function 1 and the target bounding box loss function 1 in form, respectively;

[0090] (5) Model training step 5: Set the model training iteration number i1 in advance. This step is executed when the current training iteration number is greater than i1. The target features in the distribution obtained in the model training step 3 are combined with the corresponding bounding box coordinates obtained in the model training step 2, and the multi-classification loss function is calculated through convolution processing. The multi-classification loss function is a multi-classification loss function based on cross entropy, which realizes the calculation of loss values ​​of specific multiple categories of predicted and real common hidden dangers.

[0091] The multi-classification loss function is executed when the number of model training iterations exceeds the set value i1. The multi-classification loss function is not calculated in all model training processes before the number of iterations is less than i1. In other words, the model only has a target loss value in the classification task. This is the target-oriented and multi-classification decoupling training strategy involved in the present invention. The model is allowed to perform a large amount of target-oriented learning first, and then multi-classification losses are added to induce the model to learn specific common hidden danger categories after obtaining excellent foreground / background recognition performance. This is because the unknown risk features are sparse and random. In the absence of good foreground / background recognition capabilities, allowing the model to learn multiple categories of common hidden dangers can easily cause the model to fall into a dilemma, that is, it is very easy to misclassify unknown risks as backgrounds, resulting in the final model being unable to identify unknown risk targets. Therefore, the present invention designs this target classification and multi-classification decoupling training strategy to avoid the model from falling into the above dilemma.

[0092] (6) Model training step 6: The target features within the distribution are stored in a feature set according to the specific common hidden danger categories. If the number of target features in the set exceeds n, the features that entered the set first are removed before being stored. The target features outside the distribution are stored in a feature set. If the number of elements in the set exceeds n, the features that entered the set first are removed before being stored. The set can be expressed as the formula:

[0093]

[0094] in, is a storage set of the kth common hidden dangers of the target features within the distribution, Seq OoD is a storage collection of target features within the distribution. is the target feature of the nth distribution of the kth common hidden danger stored in the corresponding set, K is the total number of common hidden danger categories, It is the nth out-of-distribution target feature of the unknown risk stored in the corresponding set. The set will be used as a sample set for subsequent parameter estimation of multivariate Gaussian distribution. In order to better utilize the local sample set to reflect the feature distribution of the real image set, the present invention decides to use the above-mentioned dynamic set update mechanism.

[0095] (7) Model training step 7: Set the model training iteration number i2 in advance. This step is executed when the training iteration number is greater than i2, i2 ≥ i1. Multivariate Gaussian distribution parameter estimation is performed on the current K+1 feature sets respectively, and OoD features are obtained by sampling according to the similarity sampling strategy; the target features within the distribution are regarded as iD features. Labels are assigned to iD features and OoD features respectively.

[0096] The multivariate Gaussian distribution parameter estimation process includes estimating the mean μ and variance σ 2 The specific calculation process is as follows: for the K target feature sets in the distribution Get the mean of each feature set respectively As the mean parameter of the multivariate Gaussian distribution of each category, calculate the covariance of all target features in the K feature set Similarly, the target feature set Seq outside the distribution OoD Get its mean μ OoD As the mean parameter of the multivariate Gaussian distribution, Seq OoD Calculate the covariance of all target features in

[0097] The similarity-based feature resampling strategy is described as follows. As mentioned above, the multivariate Gaussian distribution parameters of the target feature set in the distribution of K common hidden danger categories are obtained, and the corresponding probability distribution can be expressed as:

[0098]

[0099] Among them, h(x iD ,b iD ) are common hidden danger target features (in-distribution target features) in the complete UAV remote sensing images of the railway environment.

[0100] The obtained multivariate Gaussian distribution of the out-of-distribution target feature set can be expressed as follows for the corresponding probability distribution:

[0101]

[0102] h(x OoD ,b ooD ) are the unknown risk target features (out-of-distribution target features) in the complete unknown risk image and other scene drone images.

[0103] On the basis of the probability distribution, resampling is performed in the K+1 target feature sets respectively. The resampling process uses inverse Cholesky decomposition to decompose their respective covariances, namely:

[0104]

[0105] Among them, L iD and L OoD is a lower triangular matrix that generates a sample vector from an independent standard normal distribution and z OoD , get the sample:

[0106]

[0107] s OoD =μ OoD +L OoD z OoD

[0108] For the K+1 sampling times, the sampling number is set to a larger value m each time to ensure that the number of samples is large enough to better cover the distribution space of the feature set.

[0109] In order to improve the model's recognition ability and generalization for out-of-distribution targets, out-of-distribution target features cannot be used as OoD samples. This is because unknown risks (out-of-distribution targets) are random and uncertain, and existing sparse samples cannot be directly used as OoD samples. Instead, the probability distribution of their feature space should be mined, and resampling technology should be used to obtain new OoD samples in the feature space based on the probability distribution after parameter estimation. Only in this way can the generalization performance of the model for OoD recognition be improved.

[0110] For the samples obtained by sampling the target features within the distribution under the kth common hidden danger category (There are m samples), perform logarithmic calculation to obtain the corresponding probability density value. Since it comes from the probability distribution of the target feature within the distribution, the lowest probability density value, that is, the sample corresponding to the lowest similarity value, is selected as one of the OoD samples; for the sample s obtained by sampling the target feature outside the distribution under unknown risk ooD (There are m samples), perform logarithmic calculation to obtain the corresponding probability density value. Since it comes from the probability distribution of the target feature outside the distribution, the first t highest probability density values, that is, the samples corresponding to the high similarity values, are selected as OoD samples. Finally, K+t OoD samples are obtained.

[0111] In addition to obtaining OoD samples as described in the previous paragraph, threshold methods can also be used to obtain OoD samples. That is, for samples obtained by sampling the target features within the distribution of the kth common hidden danger category, (There are m samples), perform logarithmic calculation to obtain the corresponding probability density value, and select the sample corresponding to the probability threshold lower than ∈; for the sample s obtained by sampling the out-of-distribution target feature under unknown risk OoD (There are m samples), perform logarithmic calculation to obtain the corresponding probability density value, and select the sample corresponding to the probability threshold higher than (1-∈). ∈ is a value between 0 and 1, which is small enough.

[0112] For iD samples, there is no need to use similarity like in the acquisition of OoD samples. iD samples come from target features within the distribution. The common hidden danger features under the same category are relatively stable and the feature differences are not significant. Therefore, the target features within the distribution are directly used as iD samples to contrast with OoD samples. At this time, n*K iD samples are obtained.

[0113] (8) Model training step 8: This step is executed when the current training iteration number is greater than i2, i2 ≥ i1. The iD feature and the OoD feature are convolved by the classification branch in the model prediction head, and the energy score is calculated on the processed result. After performing full connection processing on the two parts of the feature, the binary loss function is calculated in combination with the labels assigned in step 7, where the binary loss function is a binary classification loss based on the cross entropy loss.

[0114] The above energy score calculation process is as follows:

[0115]

[0116] Among them, s max is a single sample s as the maximum value in the matrix, conv k (·) represents the mapping output result of the classification branch in the model prediction head for the kth common hidden danger category.

[0117] (9) Model training step 9: Target loss function 1, target bounding box loss function 1, target loss function 2, target bounding box loss function 2, multi-classification loss function (if any), and binary loss function (if any) are weighted to obtain the total loss calculation, and the model gradient is back-propagated and the model gradient is cleared based on the current total loss.

[0118] (10) Model training step 10: Set the model training iteration number i3 in advance. When the current training iteration number is greater than i3, (i3>i2≥i1), the model training stops and the model weights are saved. Otherwise, the model training steps 1 to 9 are iteratively executed.

[0119] Step 3 is the model reasoning stage step, specifically:

[0120] (1) Model inference step 1: read the saved model weights, load the image, perform feature extraction and feature fusion to obtain complete image features;

[0121] (2) Model reasoning step 2: The complete image features are processed through convolution to obtain the possible target bounding box coordinates and target probability in the image, and the non-maximum suppression algorithm is used to delete redundant bounding boxes based on the target probability and possible bounding box coordinates;

[0122] (3) Model reasoning step 3: Combine the predicted bounding box coordinates obtained in model reasoning step 2 to extract the target features of the corresponding part on the complete image features;

[0123] (4) Model reasoning step 4: Combine the target features obtained in model reasoning step 3 with the corresponding bounding box coordinates obtained in model reasoning step 2, and obtain the category probabilities and prediction box coordinates including common hidden danger classifications through convolution processing. Based on the category probabilities and prediction box coordinates, perform a category-independent non-maximum suppression algorithm to delete redundant bounding boxes;

[0124] (5) Model reasoning step 5: After the target features obtained in model reasoning step 3 are processed by the model prediction head classification branch, the energy score calculation and full connection processing are used to determine whether the target is an out-of-distribution target. If so, the target category is directly determined as OoD; if not, the target category is determined to be the category with the highest probability among the common hidden danger categories;

[0125] (6) Model reasoning step 6: The final target category and prediction box coordinates are marked in the original image. If it is a common hidden danger category, the corresponding category probability is additionally marked. Output the model's prediction map.

Claims

1. A method for identifying unknown risks in railway environment using UAV remote sensing images, characterized in that: The method comprises the following steps: Step 1: First, data annotation is performed to produce a set of drone remote sensing images of the railway environment, unknown risk images, and drone images of other scenes; Step 2: Based on the two prepared data sets, the network model is sent to training to learn the common hidden dangers identification in the UAV remote sensing image set of the railway environment, the unknown risk images and the unknown risks identification in the UAV image set of other scenes, and the model weights are saved after the training. Step 3: Load the trained model weights and UAV remote sensing images of the railway environment, perform location detection and category recognition of common hidden dangers and unknown risks in the images, mark them in the original images, and output the result map.

2. The method for identifying unknown risks in the railway environment using UAV remote sensing images as claimed in claim 1, characterized in that: The step 1 specifically includes the following steps: Firstly, the common hidden dangers data of the UAV remote sensing image set along the railway are annotated and the common hidden dangers are marked into corresponding categories; For unknown risk images and other scene drone image sets, various targets that appear are uniformly annotated as a label independent of common hidden danger categories; The acquired UAV remote sensing image sets of the railway environment, unknown risk images and other scene UAV image sets and their corresponding labels are input into the convolutional neural network for training.

3. The method for identifying unknown risks in the railway environment using UAV remote sensing images as claimed in claim 1, characterized in that: The step 2 specifically includes the following steps: (2.1) Model training step 1: Obtain complete UAV remote sensing image features of the railway environment, unknown risk images, and other scene UAV image features through feature extraction and feature fusion; (2.2) Model training step 2: The complete image features are processed by convolution to obtain the coordinates of the bounding box that may be the target in the image, and the target loss function 1 and the target bounding box loss function 1 are calculated. The target loss function 1 is a binary classification loss function based on cross entropy, which realizes the binary classification loss value calculation of the image predicted background and the real background. The target bounding box loss function 1 is a loss function based on the intersection of the predicted and real bounding boxes, which realizes the intersection of the predicted box and the real box. (2.3) Model training step 3: Combine the target bounding box coordinates in model training step 2 to extract the target features of the corresponding part on the complete image features. The target features in the UAV remote sensing image features of the railway environment are regarded as in-distribution target features, and the target features in the UAV image features of unknown risk images and other scenes are regarded as out-distribution target features. (2.4) Model training step 4: Combine the in-distribution and out-of-distribution target features obtained in model training step 3 with the corresponding bounding box coordinates obtained in model training step 2, and calculate the target loss function 2 and the target bounding box loss function 2 through convolution processing. The target loss function 2 and the target bounding box loss function 2 are consistent with the target loss function 1 and the target bounding box loss function 1 in form, respectively; (2.5) Model training step 5: Set the model training iteration number i1 in advance. This step is executed when the current training iteration number is greater than i1. The target features in the distribution obtained in the model training step 3 are combined with the corresponding bounding box coordinates obtained in the model training step 2, and the multi-classification loss function is calculated through convolution processing; the multi-classification loss function is a multi-classification loss function based on cross entropy, which realizes the calculation of loss values ​​of specific multiple categories of predicted and real common hidden dangers; (2.6) Model training step 6: The target features within the distribution are stored in a feature set according to the specific common hidden danger categories. If the number of target features in the set exceeds n, the features that entered the set first are removed before being stored. The target features outside the distribution are stored in a feature set. If the number of elements in the set exceeds n, the features that entered the set first are removed before being stored. The set is expressed as the formula: in, is a storage set of the kth common hidden dangers of the target features within the distribution, Seq OoD is a storage collection of target features within the distribution. is the target feature of the nth distribution of the kth common hidden danger stored in the corresponding set, K is the total number of common hidden danger categories, It is the nth out-of-distribution target feature of the unknown risk stored in the corresponding set; the set will be used as a sample set for subsequent parameter estimation of multivariate Gaussian distribution; (2.7) Model training step 7: Set the model training iteration number i2 in advance. This step is executed when the training iteration number is greater than i2, i2 ≥ i1. Multivariate Gaussian distribution parameter estimation is performed on the current K+1 feature sets respectively, and OoD features are obtained by sampling according to the similarity sampling strategy; the target features within the distribution are regarded as iD features; labels are assigned to iD features and OoD features respectively; The multivariate Gaussian distribution parameter estimation process includes estimating the mean μ and variance σ 2 , the specific calculation process is as follows, for the K distribution target feature sets Get the mean of each feature set respectively As the mean parameter of the multivariate Gaussian distribution of each category, calculate the covariance of all target features in the K feature set Similarly, the target feature set Seq is distributed outside OoD Get its mean μ OoD As the mean parameter of the multivariate Gaussian distribution, Seq OoD Calculate the covariance of all target features in The feature resampling strategy based on similarity is described as follows: The multivariate Gaussian distribution parameters of the target feature set under the distribution of K common hidden danger categories are obtained, and the corresponding probability distribution can be expressed as: Among them, h(x iD ,b iD ) are common hidden danger target features in the complete UAV remote sensing images of the railway environment; The obtained multivariate Gaussian distribution of the out-of-distribution target feature set can be expressed as follows for the corresponding probability distribution: h(x ooD ,b OoD ) is the complete unknown risk image and the unknown risk target features in other scene drone images; On the basis of the probability distribution, resampling is performed in the K+1 target feature sets respectively. The resampling process uses inverse Cholesky decomposition to decompose their respective covariances, namely: Among them, L iD and L OoD is a lower triangular matrix that generates a sample vector from an independent standard normal distribution and z OoD , get the sample: s OoD =μ OoD +L OoD z OoD For K+1 samplings, the number of samples is set to a larger value m each time to ensure that the number of samples is large enough to better cover the distribution space of the feature set to be obeyed; (2.8) Model training step 8: This step is executed when the current training iteration number is greater than i2, i2 ≥ i1; the iD feature and the OoD feature are convolved by the classification branch in the model prediction head, and the energy score is calculated on the processed result. After performing full connection processing on the two parts of the feature, the binary loss function is calculated in combination with the labels assigned in step (2.8), where the binary loss function is a binary classification loss based on the cross entropy loss; The energy score calculation process is as follows: Among them, s max is a single sample s as the maximum value in the matrix, conv k (·) represents the mapping output result of the classification branch in the model prediction head for the kth common hidden danger category; (2.9) Model training step 9: Target loss function 1, target bounding box loss function 1, target loss function 2, target bounding box loss function 2, multi-classification loss function, and binary loss function are weighted to obtain the total loss calculation, and the model gradient is back-propagated and the model gradient is cleared based on the current total loss; (10) Model training step 10: Set the model training iteration number i3 in advance. When the current training iteration number is greater than i3, (i3>i2≥i1), the model training stops and the model weights are saved. Otherwise, the model training steps 1 to 9 are iteratively executed.

4. The method for identifying unknown risks in the railway environment using UAV remote sensing images as claimed in claim 1, characterized in that: The step 3 specifically includes the following steps: (3.1) Model inference step 1: read the saved model weights, load the image, perform feature extraction and feature fusion to obtain complete image features; (3.2) Model reasoning step 2: The complete image features are processed through convolution to obtain the possible target bounding box coordinates and target probability in the image, and the non-maximum suppression algorithm is used to delete redundant bounding boxes based on the target probability and possible bounding box coordinates; (3.3) Model reasoning step 3: Combine the predicted bounding box coordinates obtained in model reasoning step 2 to extract the target features of the corresponding part on the complete image features; (3.4) Model reasoning step 4: Combine the target features obtained in model reasoning step 3 with the corresponding bounding box coordinates obtained in model reasoning step 2, and obtain the category probabilities and prediction box coordinates including common hidden danger classifications through convolution processing. Based on the category probabilities and prediction box coordinates, perform a category-independent non-maximum suppression algorithm to delete redundant bounding boxes; (3.5) Model reasoning step 5: After the target features obtained in model reasoning step 3 are processed by the model prediction head classification branch, the energy score calculation and full connection processing are used to determine whether the target is an out-of-distribution target; if so, the target category is directly determined as OoD; if not, the target category is determined to be the category with the highest probability among the common hidden danger categories; (3.6) Model reasoning step 6: Mark the final target category and prediction box coordinates in the original image. If it is a common hidden danger category, additionally mark the corresponding category probability; output the model's prediction map.

Images