Edge-deployed semi-supervised anomaly detection method and system for railway track foreign object

Abstract

Disclosed in the present invention are an edge-deployed semi-supervised anomaly detection method and system for a railway track foreign object. The method comprises the following steps: an edge device encoding and decoding a video stream captured by a camera to obtain an image frame sequence, and performing frame extraction; and using a semantic segmentation model to perform image segmentation on a certain image frame obtained by means of frame extraction, to obtain a railway track region segmentation image. The use of a single image as input may generate an expert model result having a high weight value; however, the determination based on a single image is not stable, multiple consecutive images of the task scene need to be inputted, the frequency of each expert model obtaining the highest weight is computed, and the expert model corresponding to the highest frequency is the final solution. The present invention supports scene-adaptive foreign object detection algorithm automatic selection, and a user can perform selection on the basis of prior knowledge, or selection may be performed by a scene-adaptive automatic algorithm selection method; the user only needs to provide a batch of image data of the current scene, and the optimal algorithm selection can be evaluated.

Description

A semi-supervised anomaly detection method and system for foreign objects on railway tracks deployed at the edge. Technical Field

[0001] This invention relates to the field of foreign object detection on railway tracks, and in particular to a semi-supervised anomaly detection method and system for foreign objects on railway tracks deployed at the edge. Background Technology

[0002] As a crucial transportation infrastructure, railways are susceptible to accidents and damage from foreign objects (people, animals, mudslides, or tools) on the tracks. Therefore, rapid detection and handling of these objects is essential. Traditional methods for identifying foreign objects require installing multiple cameras along the tracks, lack analytical capabilities, necessitate manual inspection, are costly, and carry a high risk of omissions.

[0003] In recent years, with the development of artificial intelligence technology, more and more intelligent analysis methods have been applied to railway track foreign object detection. The problems with existing methods are: current algorithms are mainly trained in a supervised manner, requiring a large amount of manual annotation, resulting in long development cycles and poor adaptability to different scenarios, making them unable to cope with unexpected situations; at the same time, the algorithms are deployed on the server side, and considering the overhead and latency of data transmission, the real-time performance and stability of the detection will be greatly reduced.

[0004] Currently, the main methods used for foreign object detection in the railway field are: (1) using adversarial generative networks to compare the differences between the input image and the reconstructed image to determine foreign objects; (2) using target detection and segmentation methods to detect foreign objects; and (3) using traditional visual anomaly detection methods to detect the continuity of the railway track, abnormal edges, variance and correlation, and optical flow field anomalies.

[0005] For example, technical solution one (patent application number CN202110785139.1, patent application title "A Semi-Supervised Foreign Object Detection Method for Railway Track Images") uses a high-definition camera to obtain two-dimensional images of railway tracks and converts them into grayscale images; it employs a GAN-based anomaly detection method, inputting only normal railway track images during model training; during testing, the model can reconstruct images of normal samples but cannot reconstruct images of abnormal samples; during inference, a threshold is set, and if the difference between the reconstructed image and the original image exceeds the set threshold, the image is judged as abnormal. This invention uses a semi-supervised approach, requiring no abnormal sample images during model training, and can accurately detect foreign object images on railway tracks during the inference stage.

[0006] The disadvantages of technical solution 1 are: (1) the adversarial generative network used is difficult to train and the training cost is high; (2) the detection area is not limited and global detection will increase the false alarm recall rate.

[0007] Technical Solution 2 (patent application number CN202211323105.1, patent application title "A method and system for identifying foreign objects on railway tracks based on a monocular thermal infrared camera") uses YOLOv5 for initial localization of foreign objects, and then uses different models for semantic segmentation for different target categories. The original image set is automatically labeled according to the segmentation results to train the improved semantic segmentation network proposed in this patent. The temperature information of the thermal image is added to the calculation weight of the improved semantic segmentation network to complete the segmentation of the original image and obtain the foreign object segmentation map.

[0008] The disadvantages of technical solution 2 are: (1) It adopts supervised learning methods such as target detection and segmentation, which requires data collection and labeling for specific scenarios, resulting in high labor costs; (2) It adopts the transformer structure, which is not easy to port to the edge.

[0009] Technical Solution 3 (patent application number CN202211053889.0, patent application title "A Method for Detecting Foreign Objects on Railway Tracks") aims to solve the technical problems existing in the current technology using video surveillance. This method for detecting foreign objects on railway tracks includes: reading the monitoring area, detecting railway track continuity, abnormal edges, variance and correlation, and optical flow field anomalies; fusing the detected data to obtain estimates of the size and distance of the railway track obstacles.

[0010] The disadvantages of technical solution 3 are: (1) Using traditional visual foreign object detection methods to detect railway obstacles has limited generalization ability and low scene adaptability; (2) The number of foreign object detection targets is limited, only supporting pedestrian detection, and retraining is required to add categories.

[0011] In summary, the shortcomings of the existing technologies described above are as follows:

[0012] (1) The algorithm schemes used are either difficult to train, have high labeling costs, or are supervised training with poor generalization and cannot cope with unknown foreign objects, such as GAN, target detection algorithms and traditional foreign object detection methods.

[0013] (2) The railway track scene is quite complex, and a single algorithm is difficult to meet the needs of different scene changes;

[0014] (3) The algorithm is not lightweight and is not suitable for deployment on edge devices;

[0015] (4) There is a lot of human involvement, the degree of automation is not high, and there is a lack of unified multi-algorithm support and simple deployment methods. Summary of the Invention

[0016] The purpose of this invention is to overcome the shortcomings and deficiencies of existing technologies and provide a semi-supervised anomaly detection method for railway tracks deployed at the edge. This method supports three semi-supervised anomaly detection algorithms, which can be automatically selected according to scene changes to meet the needs of detecting foreign objects of different sizes. All algorithms support the detection of unknown foreign objects and can handle various unexpected situations. The algorithm development process only requires positive sample images, without additional annotation, and can complete the entire process from model training, quantization, and porting deployment in one stop, achieving a high degree of automation and significantly shortening the development cycle. The algorithm is deployed on edge devices, ensuring the real-time performance and stability of the detection system. Finally, this invention also supports edge training of some algorithms, allowing for model updates at the edge based on scene changes.

[0017] Another objective of this invention is to provide a semi-supervised anomaly detection system for foreign objects on railway tracks deployed at the edge.

[0018] The objective of this invention is achieved through the following technical solution:

[0019] A semi-supervised anomaly detection method for foreign objects on railway tracks, deployed at the edge, includes the following steps:

[0020] The edge device encodes and decodes the video stream captured by the camera to obtain an image frame sequence, and sets a frame interval threshold to perform frame extraction processing on the image frame sequence;

[0021] For a frame of image obtained by frame extraction, a semantic segmentation model is used to perform image segmentation to obtain a segmented image of the railway track area.

[0022] For a given frame of image, the previously obtained railway track region segmentation image is input into the hybrid expert system. The threshold network of the hybrid expert system assigns weights to several expert models, and the expert model with the highest assigned weight in the current frame is recorded, with its corresponding frequency incremented by 1. Similarly, through the statistics of multiple consecutive frames of images, the statistical frequency of each expert model is obtained. The semi-supervised anomaly detection model for railway foreign objects corresponding to the expert model with the highest frequency is used as the detection model for the current railway track region segmentation image and completes the detection of railway foreign objects.

[0023] The output formula of the hybrid expert system is as follows:

[0024] Where x is the input image, y is the output foreign object probability density map; G(x) i E represents the probability output of the threshold network for the i-th expert model. i For the i-th expert model, when G(x) i When the value is 0, the corresponding expert model will not be activated.

[0025] Theoretically, G(x) iThe larger the value, the greater the probability that the i-th expert model will be selected, and the output of that expert model will become the final output.

[0026] The loss function L of the threshold network is as follows:

[0027] Where c represents an image sample, d is the output of the i-th expert model; c It is the ideal output, that is, the truth value. The network learns to make it so. The target value will be gradually approached; It represents the weights assigned by the threshold network to the i-th expert model, equivalent to G(x). i By reducing the loss function L, the weights of the threshold network parameters are trained.

[0028] The semi-supervised rail foreign object detection model corresponding to the expert model includes a rail foreign object detection method based on the automatic training and deployment of a standardized flow model. The training process of the standardized flow model adopts an early stopping strategy based on the statistical variance of the training data: when the mean square error of the output feature Z of the standardized flow model is not greater than a preset threshold, it is judged that the detection model is about to collapse and the training of the detection model is stopped; and the detection model with the closest time point to before the training was stopped is taken as the final result.

[0029] The total training data of the standardized flow model is divided into N batches for mean square error calculation. The mean square error Z of the output features of the total training data after passing through the standardized flow model is then calculated. std The process is as follows:

[0030] Among them, S i S represents the overall variance of the first i batches of data. B M represents the local variance of the current i-th batch. B M represents the local mean of the current i-th batch. i B represents the overall mean of the first i batches of data; j and B i Let represent the images of the j-th and i-th batches, respectively.

[0031] The rail foreign object detection method based on the automatic training and deployment of the standardized flow model determines whether a rail foreign object has been generated by outputting the probability density value of feature z from the standardized flow model. If the probability density value is greater than a preset threshold, the rail foreign object is considered to have been generated.

[0032] The calculation method for the probability map h of foreign objects on the rails at all pixel locations, based on the output feature Z of the standardized flow model, is as follows:

[0033] Where Up represents the size at which each downsampled feature map is upsampled to the original image, and the final probability value of the foreign object on the rail is the average of the results at the three scales; z1, z2, and z3 represent output feature maps of different sizes;

[0034] After obtaining the probability map h of foreign objects on the rails, post-processing is required to obtain the segmentation and detection results: the probability map of foreign objects on the rails is segmented by setting a threshold thres to obtain a segmentation map. In the segmentation map, the area greater than the set threshold thres is the area of ​​foreign objects on the rails, otherwise it is a normal area; the minimum envelope of each segmentation map is used as the envelope result of the detection of foreign objects on the rails; if there is a region of foreign objects on the rails, it is determined that there are foreign objects on the rails.

[0035] The set threshold thres is an adaptively set threshold, calculated in the following way:

[0036] Here, the threshold `thres` is set as a probability value for foreign objects on the rails, representing the boundary between normal features and foreign object features on the rails. mean Z is the mean point of feature z on the training data. std Let Z be the mean squared error of feature Z, and Z0 be the feature point used to calculate the segmentation threshold; P(Z0) is the sum of the mean squared errors of features Z and Z0. mean +2Z std The function P(*) is equivalent to the function P(*) for calculating the probability density of *.

[0037] The set threshold threshold thres is used to process the edge region of the product image, and a higher threshold (thres+0.1) is used to process the non-edge region of the product.

[0038] The standardized flow model uses FastFlow.

[0039] The automatic training and deployment method for detecting foreign objects on railway tracks based on a standardized flow model uses fastflow as the standardized flow model, a standard normal distribution as the posterior estimate of fastflow, and a ResNet18 network pre-trained on the ImageNet classification dataset as the feature extractor.

[0040] The semi-supervised rail foreign object detection model corresponding to the expert model includes a rail foreign object detection method based on multi-layer feature perception using a variational autoencoder, comprising the following steps:

[0041] S1. Self-supervised training of the autoencoder is carried out by distillation training. A deep convolutional neural network is used as the teacher model of the autoencoder. Distillation training of multi-scale features is carried out during the distillation training stage.

[0042] S2. Use image overlay to synthesize abnormal railway track image samples;

[0043] S3. The variational autoencoder is trained by distillation. The generated normal rail images and abnormal rail images from step S2 are used as inputs to the variational autoencoder. The autoencoder trained in step S1 is used as the teacher model of the variational autoencoder to perform multi-scale feature-supervised training of the decoding module of the variational autoencoder, so that the decoding module of the variational autoencoder is aligned with the decoding module of the autoencoder in the feature space of multiple scales, and different semantic information is reconstructed.

[0044] S4. Perform multi-feature perception image anomaly detection using trained autoencoders and variational autoencoders.

[0045] Step S4 is as follows:

[0046] 1) Statistical analysis of the mean and variance of image features at different scales on the test set for autoencoders and variational autoencoders;

[0047] 2) Input the preprocessed image to be tested, and the autoencoder and variational autoencoder output three image features and reconstructed images of the same size, respectively;

[0048] 3) Use the mean and variance normalized by the statistical results from step 1) to normalize the image features output by the autoencoder and variational autoencoder.

[0049] 4) Calculate the mean square error between the three dimensions of the autoencoder and variational autoencoder image features, which is called the feature reconstruction error. Accumulate the feature reconstruction error according to the feature channel dimension, and perform bilinear interpolation upsampling operation to obtain the reconstructed feature error with the same width and height as the input image to be tested.

[0050] 5) Calculate the mean square error between the image reconstructed by the autoencoder and the image generated by the variational autoencoder. This is called the image reconstruction error. Accumulate the image reconstruction error along the channel dimension to obtain the image reconstruction error with the same dimension as the feature reconstruction error.

[0051] 6) Sum the feature reconstruction error and image reconstruction error of the three dimensions, and perform min-max standardization to map the error values ​​to the [0,1] interval. Areas with error values ​​greater than the error threshold are judged as abnormal areas of the railway image, otherwise they are normal areas of the railway image.

[0052] The training process of the autoencoder is as follows:

[0053] Prepare large-scale open-source image data for self-supervised distillation training;

[0054] Building an autoencoder model: The encoding module of the autoencoder consists of 4 downsampling network layers, and the decoding module of the autoencoder consists of 4 upsampling network layers. Each downsampling network layer reduces the width and height of the input image features by half, and each upsampling network layer doubles the width and height of the input image features.

[0055] During the training phase, image preprocessing is performed. First, the input image is scaled to 256×256. Then, image horizontal flip enhancement, HSV color gamut dithering enhancement, and small-angle image rotation enhancement are performed according to a preset ratio.

[0056] After preprocessing, the images are input into the autoencoder and the teacher model of the autoencoder respectively to obtain the corresponding reconstructed images;

[0057] The mean square error between the 32×32, 64×64, and 128×128 image features output by the autoencoder's decoding module and the corresponding scale image features of the autoencoder's teacher model is calculated, and the mean square error between the reconstructed image and the input image is calculated for use in autoencoder model training.

[0058] The loss function of the autoencoder model is backpropagated to optimize the autoencoder model parameters, resulting in a pre-trained autoencoder.

[0059] The loss function for training the autoencoder model is calculated using the following formula:

[0060] in, This indicates the autoencoder decoding module w i ×h i Image features of size, Teacher model w i ×h i Image features of size, Indicates w i ×h i Image features at different sizes and The mean square error between them; I r This represents the image reconstructed by the autoencoder, where I represents the input image. r -I) 2 Image I r The mean square error of I.

[0061] The teacher model of the autoencoder is one of ResNet50, VGG19, or HRNet.

[0062] Step S2 is as follows:

[0063] Prepare the COCO instance segmentation dataset, use the object instances in the data as pseudo-foreign objects on the railway tracks, and extract the mask image of each instance; the COCO dataset includes common objects, animals and pedestrians in daily life, which are consistent with the foreign objects that may appear on the railway tracks.

[0064] Acquire normal image data of railway tracks and filter out special image samples of trains passing by;

[0065] Based on the instance mask image, the instance image and the railway track area are superimposed and synthesized according to a preset ratio. The synthesis formula is as follows:

[0066] Among them, I m Represents a COCO image, I n I represents a normal image sample of railway tracks. a M represents the synthesized railway track pseudo-anomaly image. a Represents an instance of an object in image I m The mask image, It is the mask image M a The inverse of the result, β is the superposition coefficient for image synthesis.

[0067] The training process of the variational autoencoder is as follows:

[0068] Prepare normal railway track images and synthetic pseudo-anomaly railway track images, and divide them into training and test sets;

[0069] A variational autoencoder model is constructed, which is similar in structure to the autoencoder. On the basis of the autoencoder, one downsampling network layer and one upsampling network layer are added to encode the input image into a vector distribution in the latent space, and then reconstruct to generate a new image.

[0070] In the image preprocessing stage, the input image is enhanced by HSV color gamut dithering and small-angle horizontal rotation.

[0071] After preprocessing, the normal image of the railway track and the synthesized pseudo-anomaly image of the railway track are input into the variational autoencoder to generate normal image samples. At the same time, the normal image is input into the autoencoder to obtain the reconstructed image.

[0072] The mean square error (MSE) between the image features of the variational autoencoder decoding module at sizes of 32×32, 64×64, and 128×128 and the corresponding scale image features of the autoencoder is calculated. The MSE between the image generated by the variational autoencoder and the normal image is also calculated, yielding the feature reconstruction and image generation loss functions L. g :

[0073] In the formula, This indicates the variational autoencoder decoding module w. i ×hi Image features of size, This indicates the autoencoder decoding module w i ×h i Image features of size, Indicates w i ×h i Image features at different sizes and The mean square error between them; I n I represents the normal images input from the variational autoencoder model and the autoencoder model, respectively. g This represents a normal image generated by a variational autoencoder model, (I g -I n ) 2 Image I g and I n The mean square error between them;

[0074] Based on the loss function of the variational autoencoder, the model parameters of the variational autoencoder are optimized so that the variational autoencoder can generate normal images from both abnormal and normal input images.

[0075] The loss function L of the variational autoencoder VAE The definition is as follows:

[0076] In the formula, L' g L″ represents the loss function for feature reconstruction and image generation from a normal input image. g Let λ1, λ2, and λ3 represent the feature reconstruction and image generation loss functions of the input pseudo-anomaly image, respectively, and let λ1, λ2, and λ3 represent the weight coefficients of different losses. The KL divergence loss is calculated for the normal distribution and standard normal distribution of the input data.

[0077] The Calculated using the following formula:

[0078] Where N is the dimension of the latent space vector, μ n and σ n Let μ and σ represent the mean and variance of the nth dimension of the latent space vector, respectively.

[0079] The semi-supervised rail foreign object detection model corresponding to the expert model includes a rail foreign object detection method based on deep feature comparison, comprising the following steps:

[0080] Training phase:

[0081] Normal samples are obtained and overlapped into blocks, and then abnormal rail samples are generated.

[0082] Anomaly samples and normal samples of railway tracks are combined into a training sample pair for preprocessing. The preprocessed data is then processed by an encoding module and a decoding module to extract, fuse, and decode features, thus completing the training.

[0083] Testing phase:

[0084] First, fix the detection scene, and obtain the image of the next frame of the current scene of the railway track without foreign objects as the normal sample of the railway track. The sample of the railway track to be detected is the real-time lens scene image.

[0085] Then, the normal rail sample and the rail sample to be inspected are used as a test sample pair for preprocessing.

[0086] The trained encoding and decoding modules are used to output the corresponding prediction results. Finally, the decoded results are logically judged to provide real-time alarms for abnormal track conditions.

[0087] During the training phase, the method for generating images of foreign objects in the abnormal rail samples is as follows:

[0088] (1) Specify the image of the foreign object generation area;

[0089] (2) Simulate unknown rail debris of different sizes and colors that may appear in a single scene:

[0090] Using publicly available segmentation task datasets, foreground images of different object categories are obtained;

[0091] The foreground image is scaled, rotated, and its color space transformed at different ratios according to a preset probability to simulate foreign objects of different sizes and colors, thus obtaining an image of foreign objects on the railway track.

[0092] During the training phase, the method for generating images of foreign objects in the abnormal rail samples is as follows:

[0093] A. Image of the specified foreign object generation area (I) s ;

[0094] B. Simulate foreign objects on railway tracks in most scenarios using a few railway track scene images:

[0095] a. Image I of the foreign object generation area s The brightness, saturation, and hue of the image are randomly adjusted. The adjusted image is then divided into a proportionally proportional grid. The grid is then rearranged and recombined to obtain a noise image (I) simulating structural anomalies. n ;

[0096] b. Using the size of the region of interest or the size of the railway track area, randomly generate a two-dimensional Burmester noise P of the same region size, and set a threshold to perform binarization to obtain the processing result M;

[0097] c. Obtain the noisy foreground image I' of the structural anomaly using the following formula. n By simulating foreign objects that may appear in different railway track scenarios, images of foreign objects on the railway tracks are obtained: I' n =α(M⊙I n )+(1-α)(M⊙I s );

[0098] Here, α is the introduced transparency coefficient, which aims to better balance the image I of the foreign object generation region. s and noisy image I n The fusion process makes the merged image closer to the real anomaly.

[0099] During the training phase, the foreign object image I in the abnormal rail sample a After generation, an abnormal region image I' is generated, and the corresponding formula is:

[0100] Among them, I s Generate an image of the foreign object region. The result M is obtained by inverting the processing result M; using the size of the region of interest or the size of the railway track area, a two-dimensional Burmester noise P of the same area size is randomly generated, and a threshold is set for binarization to obtain the processing result M;

[0101] Replace the foreign matter generation region image I of the normal sample with the abnormal region image I'. s That is, splicing the sample to the region of interest or the rail region of the corresponding normal sample to obtain the final abnormal rail sample.

[0102] The encoding module uses two feature extraction networks with equal weights to form a Siamese network, which processes the preprocessed abnormal and normal samples, and outputs features at three different levels:

[0103] Feature map f of normal samples 1_normal (N×64×56×96), f 2_normal (N×128×28×48) and f 3_normal (N×256×14×24);

[0104] and the feature map f of the abnormal samples 1_anomaly (N×64×56×96), f 2_anomaly (N×128×28×48) and f 3_anomaly (N×256×14×24);

[0105] Wherein, the first dimension N represents the number of samples input to the network in one iteration of training, the second dimension represents the number of channels, and the third and fourth dimensions represent the height and width of the feature map;

[0106] Then, the feature maps corresponding to the normal and abnormal samples at three different levels are concatenated along the channel dimension:

[0107] Where i = 1, 2, 3, corresponding to the three different layers in the feature extraction network. This represents a tensor channel concatenation operation, and the final result is denoted as f. 1_concat (N×128×56×96), f 2_concat (N×256×28×48) and f 3_concat (N×512×14×24);

[0108] Finally, the differences between the feature maps of normal and abnormal samples at three different levels are calculated separately, as follows: f i_diff =MSE(f i_normal ,f i_anomaly );

[0109] Where i = 1, 2, 3, corresponding to the three different layers in the feature extraction network, and MSE(·) represents the sum of the mean square errors at each position of the two input tensors, denoted as f. 1_diff (N×64×56×96), f 2_diff (N×128×28×48) and f 3_diff (N×256×14×24).

[0110] The encoding module processes the feature f obtained by channel splicing through a multi-scale fusion network. 1_concat f 2_concat and f 3_concat Perform feature fusion:

[0111] First, a 3×3 convolution with half the number of channels and a 3×3 convolution with the same number of channels are used to pair f. 1_concat f 2_concat and f 3_concat Features are extracted, and then a 1×1 convolution is used to increase the non-linearity of the network to obtain f. 1_conv (N×64×56×96), f 2_conv (N×128×28×48) and f 3_conv (N×256×14×24);

[0112] Let f 3_msff =f 3_conv , feature map f 3_msff After doubling the upsampling size to N×256×28×48, a 3×3 convolution operation with half the number of channels is performed to obtain the feature map f. _temp (N×128×28×48), then f_temp with f 2_conv f is obtained by adding elements one by one 2_msff (N×128×28×48);

[0113] Similarly, for f 2_msff Perform 3×3 convolution operations with double upsampling and halved channel count respectively, then combine with f 1_conv f is obtained by adding elements one by one 1_msff (N×64×56×96);

[0114] Finally, the multi-scale fused feature f is output. 3_msff (N×256×14×24), f 2_msff (N×128×28×48) and f 1_msff (N×64×56×96).

[0115] The decoding module consists of multiple upsampling convolution operations combined with repeated 3×3 convolution operations, executed sequentially from deep features, and finally using a 3×3 convolution operation to reduce the channel dimension to 2 dimensions.

[0116] The training process of the decoding module is as follows:

[0117] The output f of the encoding module 1_encode f 2_encode and f 3_encode As the input to the decoding module, let f 3_decode =f 3_encode , for f 3_decode After doubling the upsampling of (N×256×14×24) to a size of N×256×28×48, a 3×3 convolution operation is performed to reduce the channel dimension and obtain the feature map f. _tepm1 (N×128×28×48), finally f _temp1 with f 2_encode f is obtained by concatenation along the channel dimension. 2_decode (N×256×28×48);

[0118] Similarly, for f 2_decode After doubling the upsampling, a 3×3 convolution operation is performed to obtain the feature map f. _tepm2 (N×64×56×96), then the feature map f _tepm2 with f 1_encode f is obtained by concatenation along the channel dimension. 1_decode (N×128×56×96);

[0119] To further reduce the computational cost of the model, we continue to work on f. 1_decode After performing a 2x upsampling, a 3×3 convolution operation is performed to reduce the channel dimension and obtain the feature map f. 0_decoder(N×32×112×192);

[0120] f 0_decoder After performing a 2x upsampling, a 3×3 convolution operation that maintains the same channel dimension is applied to obtain a feature map f with the same size as the input sample, which is 384×224. _up2mask (N×32×224×384);

[0121] Finally, a 3×3 convolution operation is used to apply f. _up2mask Extract features and reduce the channel dimension to 2D to obtain f _mask (N×2×224×384), where the first dimension of the channel dimension is the foreground output and the second dimension is the background output.

[0122] Another objective of this invention is achieved through the following technical solution:

[0123] A semi-supervised anomaly detection system for railway tracks deployed at the edge includes a semantic segmentation model and a hybrid expert system. The semantic segmentation model is used to segment a frame of an image obtained by frame extraction to obtain a segmented image of the railway track region. The hybrid expert system includes a threshold network and several expert models. The output of the threshold network determines which expert model is activated. The semi-supervised anomaly detection model for railway tracks corresponding to this expert model serves as the detection model for the current segmented image of the railway track region and completes the detection of railway tracks.

[0124] Meanwhile, this invention provides:

[0125] A server includes a processor and a memory, the memory storing at least one program that is loaded and executed by the processor to implement the aforementioned semi-supervised anomaly detection method for foreign objects on railway tracks deployed at the edge.

[0126] A computer-readable storage medium storing at least one program, which is loaded and executed by a processor to implement the above-described semi-supervised anomaly detection method for foreign objects on railway tracks deployed at the edge.

[0127] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0128] 1. This invention constructs a method for detecting foreign objects on railway tracks. By autonomously acquiring images or manually providing a batch of actual on-site images, the model training, model quantization, and porting deployment process can be completed in one stop. It has a high degree of automation and a short development cycle. Moreover, it can effectively filter out interference items during the inference stage and runs on edge devices, with real-time performance and stability.

[0129] 2. This invention applies a semi-supervised anomaly detection algorithm deployed at multiple edge endpoints to the field of railway foreign object detection. It can cover the detection of foreign objects of varying sizes, improving the algorithm system's adaptability to complex railway conditions. All algorithms are trained semi-supervised or self-supervised, enhancing the detection capability of unknown foreign objects. Furthermore, it eliminates the need for manual annotation, reducing the algorithm development cycle. Edge endpoint deployment improves the algorithm's real-time response and system stability. Existing technical methods employ algorithms (such as GANs, object detection algorithms, and traditional foreign object detection methods) that are either difficult to train, have high annotation costs, or require supervised training, resulting in poor generalization and an inability to handle unknown foreign objects.

[0130] 3. This invention supports scene-adaptive automatic selection of railway track object detection algorithms. Employing multiple railway track anomaly detection algorithms, this invention allows users to select based on prior knowledge, while also providing a scene-adaptive automatic algorithm selection method. Utilizing the matching probabilities of various algorithms output by the threshold network in MoE, users only need to provide a batch of image data for the current scene to evaluate and determine the optimal algorithm.

[0131] 4. This invention applies a rail foreign object detection method based on standardized flow model for automatic training and deployment to the field of rail foreign object detection. It only requires semi-supervised training using positive sample rail images without anomalies, without the need for manual annotation. The algorithm has strong generalization ability, and small changes in the field of view (rotation or translation) will not affect the detection effect.

[0132] 5. This invention applies a rail foreign object detection method based on variational autoencoder multi-layer feature perception to the field of rail foreign object detection. It uses positive sample rail images without anomalies and synthetic negative sample image data for supervised training, which eliminates the need for manual annotation and enables precise foreign object detection.

[0133] 6. This invention applies a rail foreign object detection method based on deep feature comparison to the field of rail foreign object detection. It uses positive sample rail images without anomalies and synthetic negative sample image data for supervised training, which does not require manual annotation and can support transfer to multiple different scenarios. It has good detection capability for some common large targets (such as people, animals, etc.). Attached Figure Description

[0134] Figure 1 is a flowchart of the semi-supervised anomaly detection method for foreign objects on railway tracks deployed at the edge end according to the present invention.

[0135] Figure 2 is a schematic diagram of the railway track area definition.

[0136] Figure 3 is a schematic diagram of the foreign object detection effect.

[0137] Figure 4 is a schematic diagram of the overall network structure of the hybrid expert system.

[0138] Figure 5 is a flowchart of a rail foreign object detection method based on the automatic training and deployment of a standardized flow model.

[0139] Figure 6 shows the workflow of the FastFlow model.

[0140] Figure 7 is a schematic diagram of the adaptive threshold.

[0141] Figure 8 is a schematic diagram of the training process of the autoencoder described in this invention.

[0142] Figure 9 is a schematic diagram of the training process of the variational autoencoder described in this invention.

[0143] Figure 10 is a schematic diagram of anomaly detection in the rail foreign object detection method based on multi-layer feature perception of variational autoencoder described in this invention.

[0144] Figure 11 is an overall flowchart of the rail foreign object detection method based on depth feature comparison according to the present invention.

[0145] Figure 12 is a flowchart of the training phase of the rail foreign object detection method based on depth feature comparison described in this invention.

[0146] Figure 13 is a flowchart of the testing phase of the rail foreign object detection method based on depth feature comparison described in this invention.

[0147] Figure 14 is a flowchart of the encoding module described in this invention.

[0148] Figure 15 is a flowchart of the abnormal alarm module of the present invention. Detailed Implementation

[0149] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0150] As shown in Figure 1, the present invention first uses MoE to select three algorithm models, and after determining the expert model, it trains and deploys the model; finally, it uses the model to carry out the foreign object detection process.

[0151] As shown in Figures 1 to 4, the present invention provides a semi-supervised anomaly detection method for foreign objects on railway tracks deployed at the edge. The specific technical solution implementation process includes the following steps: (1) Overview and implementation process of the foreign object detection algorithm model; (2) Automatic selection of scene-adaptive algorithm; (3) Construction of the foreign object detection algorithm system; (4) Edge model training and update.

[0152] I. Overview and Implementation Process of Foreign Object Detection Algorithm Model for Railway Rails

[0153] 1.1 Algorithm Model Overview

[0154] The proposed method for detecting foreign objects on railway tracks includes a semantic segmentation model and three optional semi-supervised anomaly detection models.

[0155] The semantic segmentation model used is STDC. Given an image, this model can segment a specified region. The algorithm employs a lightweight feature extraction module, stdc, for segmentation tasks. This module geometrically reduces the number of channels and uses smaller convolutional kernels to effectively reduce the number of parameters. It also connects all blocks to increase the receptive field and multi-scale features, making it suitable for porting to edge devices.

[0156] The semi-supervised anomaly detection methods employed include a rail foreign object detection method based on automatic training and deployment of a standardized flow model, a rail foreign object detection method based on multi-layer feature perception using a variational autoencoder, and a rail foreign object detection method based on deep feature comparison.

[0157] 1.2 Comparison of Three Anomaly Detection Algorithms

[0158] Three different anomaly detection algorithms were introduced because actual testing revealed that different algorithms have different focuses, and a single algorithm cannot meet the needs of railway track condition scenarios with rich content and varied perspectives. The characteristics of the three anomaly detection methods are as follows:

[0159] (1) The rail foreign object detection method based on the automatic training and deployment of the standardized flow model has strong generalization ability. Small changes in the field of view (rotation or translation) will not affect the detection effect. Moreover, the model structure is lightweight and can be trained by the CPU of the edge device. The shortcoming is that the fine detection capability is insufficient.

[0160] (2) The rail foreign object detection method based on variational autoencoder multi-layer feature perception focuses on texture information and is not sensitive to color changes. It can detect small objects and targets whose colors are close to the background (such as stones and tools). The shortcoming is that it is sensitive to scene changes and the field of view must remain unchanged.

[0161] (3) The rail foreign object detection method based on deep feature comparison can support multiple different scenes. It has good detection capability for some common large targets (such as people, animals, etc.), but it is easy to miss some targets whose colors are similar to the background or whose targets are too small. The smallest detectable target size is 50*70 pixels.

[0162] The above conclusions are empirical results obtained under the current test data. The specific application should be based on the actual test situation and the solution should be selected accordingly.

[0163] 1.3 Algorithm Model Training and Deployment Process

[0164] For the semantic segmentation model STDC, manual annotation of the railway track regions is required in the on-site railway images. Each railway track region is defined as the area encompassing the two rails and the sleepers. Considering the standardized construction practices within the railway track regions, this semantic segmentation model can adapt to segmenting railway track regions in various scenarios after one round of training. During the porting and deployment process, to improve the model's efficiency at the edge, the last two layers of the network (deconvolution and ArgMax filtering) were separated from the model and executed separately as post-processing on the CPU. Finally, the entire post-processing was optimized for parallel processing using compilation options, effectively reducing the overall time consumption of the edge segmentation model.

[0165] For the rail foreign object detection method based on the standardized flow model for automatic training and deployment, only semi-supervised training with positive sample rail images without anomalies is required. Considering that the data obtained in actual field is basically free of anomalies, manual classification and labeling are largely unnecessary. To resist camera shake or small-range changes in field of view (rotation or translation), this invention employs several data augmentation methods to enhance the diversity of the training data. These methods include random rotation, cropping, and changes in brightness and contrast. The FastFlow model network is relatively lightweight and can be ported to the edges without further processing, exhibiting a certain level of efficiency.

[0166] For the railway foreign object detection method based on variational autoencoder multi-layer feature perception, this invention, in addition to using normal samples for semi-supervised training, also employs an image synthesis strategy to simulate abnormal samples and introduces abnormal samples during training to complete self-supervised learning, allowing the model to learn what constitutes a normal pattern. This is mainly achieved by adding noise to normal samples. This invention uses 80 target classes from the COCO dataset as noise. The image synthesis strategy mainly consists of three steps: First, randomly selecting a target from the 80 COCO classes as a noise image, and merging the noise image with its corresponding segmentation mask image to obtain a noisy foreground image. Second, performing random filtering on the noisy foreground image (including random resizing of the foreground target size, random cropping, random radiometric transformation, random noise addition, random brightness and contrast changes, and random color space conversion). Third, randomly superimposing the noisy foreground image onto the original image to obtain a simulated abnormal image. To port and deploy the railway foreign object detection method based on variational autoencoder multi-layer feature perception, it is necessary to replace multiple attention modules in the post-processing with convolutional operators to improve edge efficiency.

[0167] For the rail foreign object detection method based on deep feature comparison, the training mode is the same as that of the rail foreign object detection method based on variational autoencoder multi-layer feature perception. During the porting and deployment process, this invention removes the attention head in the network and reduces the number of normal sample features in the memory storage module to one, effectively improving the edge running efficiency.

[0168] 1.4 Implementation Process of Foreign Object Detection on Railway Tracks

[0169] The edge device encodes and decodes the video stream captured by the camera to obtain an image frame sequence, and performs frame extraction on the frame sequence by setting a frame interval threshold f. For a given frame obtained by frame extraction, the semantic segmentation model STDC is used for image segmentation to obtain a segmented image of the railway track area. Simultaneously, one of the three anomaly detection algorithms described in Section 1.1 is used to detect foreign objects in the image, and a probability threshold is set to obtain a segmented image of the foreign object. The segmented image of the railway track area and the segmented image of the foreign object are fused, and foreign objects outside the railway track area are filtered out to obtain a foreground foreign object image. Finally, area filtering and score threshold filtering are applied to the foreground foreign object in the foreground foreign object image to obtain the final foreign object detection result.

[0170] Based on the foreign object detection results of multiple frames, if a foreign object is detected by n consecutive frames of images, an alarm will be reported.

[0171] More specifically, the processing flow for each of the three anomaly detection algorithms is as follows:

[0172] The first type: a method for detecting foreign objects on railway tracks based on the automatic training and deployment of a standardized flow model.

[0173] As shown in Figure 5, the method for detecting foreign objects on railway tracks based on the automatic training and deployment of a standardized flow model mainly includes four parts: product image preprocessing, early stopping strategy in automated model training, calculation of segmentation threshold in automated model deployment, and post-processing of model output results.

[0174] The specific technical solution implementation process is as follows:

[0175] (1) Image enhancement preprocessing

[0176] As shown in Figure 6, this invention is a method for detecting foreign objects on railway tracks using Fastflow as a standardized flow model. To estimate the distribution of normal samples, this invention uses the standard normal distribution N(0,1) as the posterior estimate for Fastflow and a ResNet18 network pre-trained on the ImageNet classification dataset as the feature extractor. Specifically, the product image is input to the ResNet18 with a size of 288*320, and the last layer features of the first three network blocks are used as the feature input for the subsequent flow model. These three feature layers X = {x1, x2, x3} have different downsampling rates (4, 8, 16) and are respectively mapped through eight radiative transformation modules to obtain features Z = {z1, z2, z3} of the same scale. This invention uses the probability density value of the output feature Z from the standardized flow model to determine whether there are foreign objects on the railway tracks, thereby achieving automated detection of foreign objects.

[0177] Compared to traditional methods for detecting foreign objects on railway tracks, this invention is lightweight and easy to train and deploy at the edge. This makes it ideal for automated training and deployment on edge devices. Furthermore, since the collected samples without foreign objects on railway tracks have high similarity, this invention also employs several data augmentation methods to enhance the diversity of the training data. These methods include random rotation, cropping, and adjustments to brightness and contrast.

[0178] This invention proposes a lightweight foreign object detection method for railway tracks based on Fastflow and ResNet18, which is fast, accurate, and reliable. It can help manufacturers automate foreign object detection in the product manufacturing process, thereby improving product quality and production efficiency.

[0179] (2) Statistical early termination of training

[0180] In edge computing environments, automating the training process is crucial, but problems encountered during training can lead to model crashes. These issues are often related to factors such as the loss function, data distribution, and training strategy. In many cases, model training is affected by the loss function because its optimization objective often doesn't align with the actual data distribution.

[0181] This invention reveals a problem in Fastflow's automated training process: after a certain training time, the transformed feature Z tends to 0, indicating that Z does not satisfy the posterior distribution (Gaussian distribution), signifying model training failure. This invention creatively discovers that the model failure is primarily caused by the maximum likelihood loss function. The algorithm essentially requires learning the distribution of X, but maximum likelihood loss only guarantees that the input feature X has the highest probability value under the probability distribution learned by the streaming module; this does not mean that the learned probability distribution is consistent with the distribution of X. To solve the model failure problem, this invention employs an early stopping strategy. For datasets with test data and test labels, training can be continuously performed on the test data, and the model with the best test results can be selected. However, for edge-based automated training, there is no test data, let alone labeled test data.

[0182] Early stopping is a training strategy that determines when to stop training based on several metrics. These metrics might include the rate of change of training loss, the rate of change of validation loss, the magnitude of gradients, and so on. When test data and test labels are known, they can be used to evaluate model performance and select the best model. However, in automated training on edge devices, test data and test labels are often unavailable, making the selection of the optimal model more difficult.

[0183] To address this issue, this invention proposes an early stopping strategy based on the statistical variance of training data. The core idea of ​​this strategy is to use the mean squared error of feature Z to determine whether the model is about to collapse. When the mean squared error of the feature falls below a certain threshold, model training is stopped, and the final model obtained is taken as the result. This method avoids model collapse and does not require the support of test data and test labels. Therefore, this method is particularly suitable for automated training scenarios on edge devices, without the need for algorithm engineers.

[0184] In actual training, the amount of training data is large, making it impossible to load it all into memory at once for mean squared error calculation, especially on edge devices where computing resources are insufficient. This invention divides the data into N batches for mean squared error calculation, where the i-th batch contains B... i Frame images, calculate the mean squared error Z of the overall training data. std The process is as follows:

[0185] Among them, S i S represents the overall variance of the first i batches of data. B M represents the local variance of the current i-th batch. B M represents the local mean of the current i-th batch.i B represents the overall mean of the first i batches of data; j and B i Let represent the images of the j-th and i-th batches, respectively.

[0186] (3) Statistically-based adaptive threshold

[0187] After training, Fastflow obtains the feature Z. This invention calculates the probability map h of foreign objects on the rails at each pixel location according to the following formula:

[0188] Where Up represents the size of upsampling each downsampled feature map to the original image, and the final probability value of foreign objects on the rail is the average of the results of the three scales; Output feature maps of different sizes;

[0189] After obtaining the probability map h of foreign objects on the rails, post-processing is required to obtain the segmentation and detection results: the probability map of foreign objects on the rails is segmented by setting a threshold thres to obtain a segmentation map. In the segmentation map, the area greater than the set threshold thres is the area of ​​foreign objects on the rails, otherwise it is a normal area; the minimum envelope of each segmentation map is used as the envelope result of the detection of foreign objects on the rails; if there is a region of foreign objects on the rails, it is determined that there are foreign objects on the rails.

[0190] After obtaining the probability map of foreign objects on the railway tracks, post-processing is required to obtain segmentation and detection results. A common approach is to set a threshold, but the threshold setting is related to the specific dataset. A threshold that is too low will lead to false positives, while a threshold that is too high will lead to false negatives. Therefore, a suitable threshold has a significant impact on the final result of the algorithm. In the process of automated deployment at the edge, without the involvement of algorithm engineers, the algorithm needs to automatically select a threshold suitable for the dataset.

[0191] To address this problem, this invention proposes a method for adaptively setting thresholds based on statistical results from training data. Specifically, this method is based on the trained features Z, which follow a standard normal distribution in the posterior distribution (as shown in Figure 7). The characteristic of a standard normal distribution is that most features cluster around the mean of 0 and are considered normal samples, while only a few features deviate from the mean and are considered foreign objects on the rails.

[0192] Therefore, this invention proposes a formula to adaptively set the threshold, the specific formula being as follows:

[0193] Here, the threshold `thres` is set as a probability value for foreign objects on the rails, representing the boundary between normal features and foreign object features on the rails. mean Let z be the mean point of feature z in the training data. stdLet Z be the mean squared error of feature Z, and Z0 be the feature point used to calculate the segmentation threshold; P(Z0) is the sum of the mean squared errors of features Z and Z0. mean +2Z std The function P(*) is equivalent to the function P(*) for calculating the probability density of *.

[0194] (4) Dual threshold post-processing

[0195] Based on actual test results, it was found that when two foreign objects on the railway tracks are relatively close, the product portion between the foreign objects exhibits a higher probability value, leading to a larger segmentation area. To address this issue, this invention proposes a novel method that employs a dual-threshold approach for post-processing to obtain more refined segmentation results.

[0196] Specifically, this method first converts the original RGB image to a grayscale image and then uses OpenCV operators to obtain the edge image. When processing edges, a low threshold `thres` is used, while a high threshold (`thres+0.1`) is applied to non-edge areas to obtain more accurate results. This is mainly because foreign objects on railway tracks often have irregular shapes, easily generating dense edge lines, while product surfaces are relatively regular with sparser edge lines. Furthermore, the probability value of a foreign object at the center of a railway track is higher than the threshold, while the probability value of a foreign object on the product between two adjacent railway track objects is often near the threshold. Using a high threshold effectively suppresses these product areas near the threshold. This dual-threshold method effectively handles the mutual influence between railway track objects, avoiding misjudgments caused by adjacent railway track objects.

[0197] In addition to the segmentation map, we also need to obtain the bounding box location of the foreign object on the rail. We use the minimum bounding box of each segmentation map as the bounding box result for the foreign object detection on the rail.

[0198] The second method is a rail foreign object detection method based on multi-layer feature perception using a variational autoencoder.

[0199] The railway track foreign object detection method based on variational autoencoder and multi-layer feature perception can be defined as a binary classification image semantic segmentation method to segment abnormal regions from the input image. This method consists of two models: an autoencoder and a variational autoencoder. The implementation process includes autoencoder distillation training, pseudo-abnormal image data synthesis, variational autoencoder distillation training, and multi-feature perception image anomaly detection. To better demonstrate the effectiveness of this method, this example illustrates the railway track foreign object detection task. The specific technical solution is as follows:

[0200] 1. Autoencoder distillation training

[0201] To achieve a lightweight network structure with good reconstruction capabilities, the autoencoder employs a distillation training method for self-supervised training. A pre-trained model with a large number of parameters is used as the teacher model, which can be a deep convolutional neural network such as ResNet50, VGG19, or HRNet. The teacher model's parameters are frozen during the training phase, and no parameter updates are performed. Simultaneously, to ensure that the features extracted by the decoding module at different network layers possess multi-level semantic information—that is, features extracted by shallow networks describe the image's texture information, while features extracted by deep networks express the image's semantic information—multi-scale feature distillation training is performed during the distillation training phase. The autoencoder training framework is shown in Figure 8 and consists of the following steps:

[0202] 1) Prepare large-scale open-source image data for self-supervised distillation training. In this example, ImageNet, which has rich categories and diverse backgrounds, is used as training data.

[0203] 2) Build an autoencoder model. The encoding module consists of 4 downsampling network layers, and the decoding module consists of 4 upsampling network layers. Each downsampling network layer reduces the width and height of the input image features by half, and each upsampling network layer doubles the width and height of the input image features.

[0204] 3) Image preprocessing is performed during the training phase. First, the input image is scaled to 256×256, and then image horizontal flip enhancement, HSV color gamut dithering enhancement, and small-angle image rotation enhancement are performed at a certain ratio.

[0205] 4) After preprocessing, the images are input into the autoencoder and the teacher model respectively to obtain the corresponding reconstructed images;

[0206] 5) Calculate the mean square error (MSE) between the image features of the autoencoder decoding module at sizes of 32×32, 64×64, and 128×128 and the corresponding image features of the teacher model. Also calculate the MSE between the reconstructed image and the input image. The loss function for training the autoencoder model is calculated as follows:

[0207] in, This indicates the autoencoder decoding module w i ×h i Image features of size, Teacher model w i ×h i Image features of size, Indicates w i ×h i Image features at different sizes and The mean square error between them; Ir This represents the image reconstructed by the autoencoder, where I represents the input image. r -I) 2 Image I r The mean square error of I.

[0208] 6) Based on the above loss function, backpropagation is performed to optimize the autoencoder model parameters to obtain the pre-trained model.

[0209] Since the autoencoder is used in this invention to reconstruct semantic features and input images at multiple levels, and is independent of the image data for anomaly detection, the pre-trained autoencoder can be reused.

[0210] 2. Pseudo-anomaly image data synthesis

[0211] In practical applications, it is rare to obtain a sufficient number of real anomaly images for supervised training. Synthesis techniques are often used to generate pseudo-anomaly images to augment the training data. However, careful processing is necessary to ensure that the generated pseudo-anomaly images accurately reflect the actual scene. The railway track foreign object detection task is no exception; it is difficult to obtain anomaly image samples from real-world scenes. Therefore, this invention employs an image overlay method to synthesize railway track anomaly image samples, which is carried out in the following steps:

[0212] 1) Prepare the COCO instance segmentation dataset, use the object instances in the data as pseudo-foreign objects on the railway tracks, and extract the mask image of each instance; the COCO dataset includes common items, animals and pedestrians in daily life, which are consistent with the foreign objects that may appear on the railway tracks.

[0213] 2) Acquire normal image data of railway tracks and filter out special image samples of trains passing by;

[0214] 3) Based on the instance mask image, the instance image is superimposed and synthesized with the railway track area at a certain ratio. The synthesis formula is as follows:

[0215] In the formula, I m Represents a COCO image, I n Indicates a normal image sample, I a M represents the synthesized pseudo-anomaly image. a Represents an instance of an object in image I m The mask image, It is the mask image M a The inverse of the result, β is the superposition coefficient for image synthesis.

[0216] 3. Variational autoencoder distillation training

[0217] The variational autoencoder (VAE) is used to generate normal images and is trained using the distillation method. The autoencoder trained in the first stage is used as the teacher model for multi-scale feature-supervised training of the decoding module. This aligns the VAE's decoding module with the autoencoder's decoding module across multiple feature spaces, reconstructing different semantic information. The VAE distillation training framework is shown in Figure 9 and consists of the following steps:

[0218] 1) Prepare normal images of railway tracks and synthetic pseudo-anomaly images, and divide them into training and test sets;

[0219] 2) Construct a variational autoencoder model, which is similar in structure to the autoencoder. On the basis of the autoencoder, add one downsampling network layer and one upsampling network layer to encode the input image into a vector distribution in the latent space, and then reconstruct to generate a new image.

[0220] 3) In the image preprocessing stage, the input image is enhanced by HSV color gamut dithering and small-angle horizontal rotation.

[0221] 4) After image preprocessing, normal images and pseudo-abnormal images are input into variational autoencoders to generate normal image samples, and normal images are input into autoencoders to obtain reconstructed images.

[0222] 5) Calculate the mean square error between the image features of the variational autoencoder decoding module at sizes of 32×32, 64×64, and 128×128 and the image features of the corresponding scale of the autoencoder, and calculate the mean square error between the image generated by the variational autoencoder and the normal image to obtain the feature reconstruction and image generation loss functions:

[0223] In the formula, This indicates the variational autoencoder decoding module w. i ×h i Image features of size, This indicates the autoencoder decoding module w i ×h i Image features of size, Indicates w i ×h i Image features at different sizes and The mean square error between them; I n I represents the normal images input from the variational autoencoder model and the autoencoder model, respectively. g This represents a normal image generated by a variational autoencoder model, (I g -I n ) 2 Image I g and I n The mean square error between them.

[0224] To enable the variational autoencoder model to generate images, the posterior distribution of the latent space is constrained to follow a standard normal distribution, and the KL divergence between the normal distribution and the standard normal distribution of the input data is increased. The loss is defined by the following formula:

[0225] In the formula, N is the dimension of the latent space vector, and μ n and σ n Let Ln represent the mean and variance of the nth dimension of the latent space vector, and μ and σ represent the mean and variance of the latent space vector, respectively. Therefore, the loss function Ln of the variational autoencoder... VAE The definition is as follows:

[0226] In the formula, L' g L″ represents the loss function for feature reconstruction and image generation from a normal input image. g L' represents the loss function for feature reconstruction and image generation of the input pseudo-anomaly image. g and L″ g The calculation formula and L g Consistent, λ1, λ2, and λ3 represent the weighting coefficients for different losses; The KL divergence loss is calculated for the normal distribution and standard normal distribution of the input data.

[0227] 6) Based on the above loss function, optimize the model parameters so that the variational autoencoder can generate normal images from both abnormal and normal input images.

[0228] 4. Multi-feature perception image anomaly detection

[0229] After obtaining the pre-trained autoencoder and variational autoencoder, multi-feature perception image anomaly detection is performed, as shown in Figure 10, and is carried out in the following steps:

[0230] 1) Statistical analysis of the mean and variance of image features at different scales on the test set for autoencoders and variational autoencoders;

[0231] 2) Input the preprocessed image to be tested, the autoencoder outputs image features and reconstructed images of size 32×32, 64×64, and 128×128, and the variational autoencoder outputs image features and generated images of size 32×32, 64×64, and 128×128.

[0232] 3) Use the mean and variance normalized by the statistical results from step 1) to normalize the image features output by the autoencoder and variational autoencoder.

[0233] 4) Calculate the mean square error between the three dimensions of the autoencoder and variational autoencoder image features, which is called the feature reconstruction error. Accumulate the feature reconstruction error according to the feature channel dimension, and perform bilinear interpolation upsampling operation to obtain the reconstructed feature error with the same width and height as the input image.

[0234] 5) Calculate the mean square error between the image reconstructed by the autoencoder and the image generated by the variational autoencoder. This is called the image reconstruction error. Accumulate the image reconstruction error along the channel dimension to obtain the image reconstruction error with the same dimension as the feature reconstruction error.

[0235] 6) Sum the feature reconstruction error and image reconstruction error of the three dimensions, and perform min-max normalization to map the error values ​​to the [0,1] interval;

[0236] 7) Set an error threshold ε. Areas with error values ​​greater than ε are judged as abnormal areas of the image, otherwise they are normal areas of the image.

[0237] The third method: a rail foreign object detection method based on depth feature comparison.

[0238] The foreign object detection method for railway tracks based on deep feature comparison mainly utilizes the differences in multi-level features between normal and abnormal samples in the same scene to locate foreign objects. The overall process of the railway track foreign object detection method based on deep feature comparison is shown in Figure 11. The corresponding railway track anomaly detection system mainly includes a sample input module, a data preprocessing module, an encoding module, a decoding module, and an anomaly alarm module. Due to the diversity and complexity of railway track scenes, it is difficult to extract relatively universal normal sample features for comparison with abnormal sample features. Therefore, this invention uses a single normal sample in the scene under test to compare with an abnormal sample, and updates the normal sample through a certain strategy to ensure the effectiveness of its detection.

[0239] The overall process is divided into two phases: training and testing, as shown in Figures 12 and 13. During the training phase, anomalous samples are generated to simulate foreign objects in different scenarios using two anomalous synthesis strategies. During the testing phase, the anomalous samples are simply replaced with the samples to be tested. The encoding and decoding modules are common components; their parameters are updated during training, but remain fixed for network inference during testing. Finally, an additional anomalous alarm module is included in the testing phase to handle the logical judgment of normal sample updates and anomalous alarms.

[0240] During the training phase, a normal sample is first input and divided into four equal parts with overlapping regions. These parts are then processed by the railroad object generation module to generate abnormal samples for the corresponding scene. These abnormal samples are then combined with the normal sample to form a training sample pair, which undergoes data preprocessing operations such as scaling, color mode conversion, standardization, and normalization. Finally, the encoder and decoder extract, fuse, and decode features from the preprocessed data.

[0241] Unlike the training phase, the testing phase does not require the participation of the foreign object generation module and adds a foreign object alarm module. First, the detection scene is fixed, and the next frame of that scene without foreign objects is acquired as a normal sample, while the sample to be detected is a real-time image of the scene. Then, the normal sample and the sample to be detected are used as a test sample pair, undergoing the same data preprocessing operations such as scaling, color mode conversion, standardization, and normalization. The trained decoder and encoder output the corresponding prediction results, and finally, the anomaly alarm module performs logical judgment on the decoded results, issuing real-time alarms for abnormal situations and continuously updating the normal samples in that scene, which can reduce false positives to some extent.

[0242] The specific steps for each module in both phases are as follows:

[0243] Step 1: Obtaining Input Samples

[0244] The method proposed in this invention locates foreign objects based on the differences in features between normal and abnormal samples in the same scene. Therefore, the model requires one normal sample and one abnormal sample as input. Since the definition and acquisition of abnormal samples differ between the training and testing phases, the abnormal samples in the testing phase will be collectively referred to as test samples in the following invention.

[0245] During the training phase, normal samples are primarily acquired using fixed lenses in different scenes, with a resolution of 1920×1080. Considering that some foreign object targets are relatively small, to better improve the recall rate for detecting small foreign objects, this invention adopts a block-based strategy for normal samples. Specifically, the original normal samples are divided into four equal parts with overlapping regions, with an overlap rate of 20%. These four equally sized image blocks serve as the input normal samples. Abnormal samples are primarily obtained through the railway track foreign object generation module proposed in this invention, which simulates the generation of abnormal samples.

[0246] During the testing phase, since the detection scene is relatively fixed, this invention selects a frame of image without foreign objects under a fixed shot of the scene to be detected as a normal sample. It should be noted that, unlike the training phase, the normal sample input in the testing phase is of its original size and has not been segmented. The model must provide a normal sample image of the scene for its first run. The test samples are real-time video image frames of the corresponding scene.

[0247] Step 2: Generation of Foreign Objects on the Rails

[0248] This invention achieves self-supervised learning by introducing anomalous samples during the training phase. It proposes a method for generating simulated randomly occurring unknown objects on railway tracks. The specific operation is as follows:

[0249] (1) Specify the region where foreign objects are generated. Given a normal sample (i.e., a railway track image without foreign objects), manually specify the region of interest (ROI) on the normal sample where foreign objects are generated, or obtain the specified railway track region through a railway track contour extraction algorithm to obtain the region image I where foreign objects are generated;

[0250] (2) Generation of Foreign Objects. Due to the diversity of railway scene and the uncertainty of foreign objects, coupled with the difficulty in collecting training data for some railway scene scenarios with foreign objects, previous work has typically generated simulated abnormal railway scene samples from normal railway scene samples. Common generation methods include image generation networks, copy and paste, and image fusion, but these methods have poor transferability in complex railway scene and generally poor anti-interference ability. For example, in the same scene, the algorithm performance may differ significantly between daytime and nighttime. Based on this, this invention proposes two foreign object simulation strategies to generate railway foreign objects.

[0251] Method 1: Simulate unknown rail debris of different sizes and colors that may appear in a single scene.

[0252] Considering the diversity and uncertainty of foreign objects on railway tracks, this invention mainly utilizes segmented foreground images of common objects to simulate common railway track foreign objects. The specific operation is as follows:

[0253] a. Use publicly available segmentation task datasets to obtain foreground images of different object categories;

[0254] b. The foreground image is scaled, rotated, and its color space transformed at different probabilities to simulate foreign objects of different sizes and colors.

[0255] Method 2: Simulate foreign objects on railway tracks in most scenarios using a few railway track scene images.

[0256] Method 1 generates anomalous samples only in a single or limited number of scenarios, which has certain limitations. This method can only identify different foreign objects in a specific single or a few scenarios. To achieve the same detection effect in more untrained and difficult-to-collect data scenarios related to railway tracks, this invention proposes Method 2 combined with Method 1 to adapt to more complex untrained railway track scenarios. The specific operation of the method is as follows:

[0257] a. Image I of the foreign object generation region obtained in step (1) s The brightness, saturation, and hue of the image are randomly adjusted. The adjusted image is then divided into a proportionally proportional grid. The grid is then rearranged and recombined to obtain a noise image (I) simulating structural anomalies. n ;

[0258] b. Using the region of interest size or railway track area size obtained in step (1), randomly generate a two-dimensional Perlin noise P of the same region size, and set a threshold to perform binarization to obtain M;

[0259] c. Obtain the noisy foreground image I' of the structural anomaly using the following formula. n This simulation aims to identify foreign objects that may appear in different railway track scenarios, where α is an introduced transparency coefficient designed to better balance the image I of the foreign object generation area. s and noisy image I n The fusion makes the fused image I' n A more realistic anomaly. n =α(M⊙I n )+(1-α)(M⊙I s );

[0260] (3) Let I be the foreign object image generated by the two strategies in step (2). a Obtained by reversing M Image I of the region where foreign matter is generated s Element-wise multiplication and then superimposed onto the foreign object image I a Generate the final abnormal region image I'. Refer to the following formula for specific operations:

[0261] (4) Finally, replace the foreign matter generation region image I of the normal sample in step (1) with the abnormal region image I'. s The abnormal samples are then spliced ​​to the corresponding regions of interest or railway tracks of the normal samples to obtain the final abnormal samples.

[0262] (5) Simulate the generation of foreign objects on the railway track by selecting either method one or method two with a certain probability.

[0263] Step 3: Data Preprocessing

[0264] (1) To reduce the computational load of model training and testing and improve detection speed, the input data of this invention is proportionally scaled and padded to a resolution of 384×224. Specifically, the original sample width is denoted as w, the height as h, and the scaling target width is denoted as w1, the height as h1. Calculation and The ratio of the two values ​​is used, and the minimum value is taken as the scaling factor r. r is then used to calculate the dimension and size to be filled. If the height is the dimension to be filled, the fill size is (h1-r×h); otherwise, if the width is the dimension to be filled, the fill size is (w1-r×w).

[0265] (2) The color mode of the input sample after scaling is converted to RGB format. Then the image data is normalized and converted into tensor format supported by the model training code framework. This invention uses PyTorch framework tensors.

[0266] (3) The tensor data from step (2) is standardized. The mean of each channel is (0.485, 0.456, 0.406) and the variance is (0.229, 0.224, 0.225).

[0267] Step 4: Encoding Module Training

[0268] The encoding module consists of three networks: a feature extraction network, a feature multi-scale fusion network, and a spatial attention mechanism. To better extract and compare features from normal and abnormal samples, this invention uses two feature extraction networks with equal weights to form a Siamese network. The feature extraction network uses a convolutional neural network trained on the public ImageNet dataset; this invention uses ResNet18. The overall data flow of the encoding module is shown in Figure 14, and the specific training process is as follows:

[0269] (1) Freeze the parameters of the pre-trained feature extraction network and input pre-processed normal and abnormal samples. To retain more detailed information and ensure that the extracted information is not too abstract and biased towards ImageNet data, this invention selects three different levels of features from ResNet18 as outputs: f1 (N×64×56×96), f2 (N×128×28×48), and f3 (N×256×14×24), where the first dimension N represents the number of samples input to the network in one iteration of training, the second dimension represents the number of channels, and the third and fourth dimensions represent the height and width of the feature map. Then, the following operations are performed:

[0270] a. Input normal and abnormal samples into ResNet18 respectively to obtain feature maps of three different levels for the corresponding samples, denoted as f. 1_normal (N×64×56×96), f 2_normal (N×128×28×48) and f 3_normal (N×256×14×24) represents the feature map of normal samples, f 1_anomaly (N×64×56×96), f 2_anomaly (N×128×28×48) and f 3_anomaly (N×256×14×24) is the feature map of the abnormal sample.

[0271] b. According to the following formula, the feature maps of normal samples and abnormal samples corresponding to three different levels are concatenated along the channel dimension:

[0272] Where i = 1, 2, 3, corresponding to the three different layers in ResNet18. This represents a tensor channel concatenation operation, and the final result is denoted as f. 1_concat (N×128×56×96), f 2_concat (N×256×28×48) and f 3_concat (N×512×14×24).

[0273] c. Calculate the differences between the feature maps of normal and abnormal samples at three different levels, as follows: f i_diff =MSE(f i_normal ,f i_anomaly );

[0274] Where i = 1, 2, 3, corresponding to the three different layers in ResNet18, and MSE(·) in this invention represents the sum of the mean square errors at each position of the two input tensors, the result of which is denoted as f. 1_diff (N×64×56×96), f 2_diff (N×128×28×48) and f 3_diff (N×256×14×24).

[0275] (2) To avoid feature redundancy and inefficiency caused by subsequent multi-layer feature calculations, this invention modifies the feature f obtained by channel splicing in step (1). 1_concat f 2_concat and f 3_concat The data is then fed into a multi-scale fusion network for feature fusion. Specific steps include:

[0276] a. First, use a 3×3 convolution with half the number of channels and a 3×3 convolution with the same number of channels to pair f. 1_concat f 2_concat and f 3_concat Features are extracted, and then a 1×1 convolution is used to increase the non-linearity of the network to obtain f. 1_conv (N×64×56×96), f 2_conv (N×128×28×48) and f 3_conv (N×256×14×24).

[0277] b. Remember f 3_msff =f 3_conv , feature map f 3_msff After doubling the upsampling size to N×256×28×48, a 3×3 convolution operation with half the number of channels is performed to obtain the feature map f. _temp (N×128×28×48), then f _temp with f 2_convf is obtained by adding elements one by one 2_msff (N×128×28×48).

[0278] c. Similarly, for f 2_msff Perform 3×3 convolution operations with double upsampling and halved channel count respectively, then combine with f 1_conv f is obtained by adding elements one by one 1_msff (N×64×56×96).

[0279] d. Finally, output the multi-scale fused features f 3_msff (N×256×14×24), f 2_msff (N×128×28×48) and f 1_msff (N×64×56×96).

[0280] (3) Subsequently, in order to further improve the algorithm’s attention to the difference information in abnormal regions, this invention proposes a spatial attention mechanism network.

[0281] a. First, the differences f between the three levels of features of normal samples and abnormal samples obtained in step (1) are analyzed. 1_diff f 2_diff and f 3_diff Channel average pooling is performed separately, outputting feature maps of three sizes, denoted as f. 1_AvgPool (N×1×56×96), f 2_AvgPool (N×1×28×48) and f 3_AvgPool (N×1×14×24).

[0282] b. Subsequent note f 3_SA =f 3_AvgPool After upsampling m3 to a size of (N×1×28×48), and then combining it with f... 2_AvgPool Element-wise multiplication yields f 2_SA (N×1×28×48).

[0283] c. Similarly, f 2_SA After doubling the upsampling to a size of N×1×56×96, and then using f 1_AvgPool Element-wise multiplication yields f 1_SA (N×1×56×96).

[0284] d. f 1_SA f 2_SA and f 3_SA As the output of the spatial attention mechanism network.

[0285] (4) Finally, the features f after multi-scale fusion in step (2) are combined. 1_msff f 2_msff and f 3_msff After spatial attention calculation in step (3), f is obtained.1_SA f 2_SA and f 3_SA Perform element-wise multiplication accordingly, referring to the following formula: f i_encode =f i_msff ⊙f i_SA ;

[0286] Where i = 1, 2, 3, and ⊙ represents the element-wise multiplication of the two input tensors at corresponding positions, with the result denoted as f. 1_encode (N×64×56×96), f 2_encode (N×128×28×48) and f 3_encode Output (N×256×14×24).

[0287] Step 5: Decoding Module Training

[0288] The decoding module mainly consists of multiple upsampling convolution operations combined with repeated 3×3 convolution operations, executed sequentially from deep features (the smaller the feature map size, the deeper the layer), and finally using a 3×3 convolution operation to reduce the channel dimension to 2 dimensions. The specific operations are as follows:

[0289] (1) The output f of the encoding module 1_encode f 2_encode and f 3_encode As the input to the decoding module, let f 3_decode =f 3_encode , for f 3_decode After doubling the upsampling of (N×256×14×24) to a size of N×256×28×48, a 3×3 convolution operation is performed to reduce the channel dimension and obtain the feature map f. _tepm1 (N×128×28×48), finally f _temp1 with f 2_encode f is obtained by concatenation along the channel dimension. 2_decode (N×256×28×48);

[0290] (2) Similarly, for f 2_decode After doubling the upsampling, a 3×3 convolution operation is performed to obtain the feature map f. _tepm2 (N×64×56×96), then the feature map f _tepm2 with f 1_encode f is obtained by concatenation along the channel dimension. 1_decode (N×128×56×96);

[0291] (3) In order to further reduce the computational cost of the model, we continue to refine f. 1_decode After performing a 2x upsampling, a 3×3 convolution operation is performed to reduce the channel dimension and obtain the feature map f. 0_decoder (N×32×112×192).

[0292] (4) f 0_decoder After performing a 2x upsampling, a 3×3 convolution operation that maintains the same channel dimension is applied to obtain a feature map f of the same size as the input sample (384×224). _up2mask (N×32×224×384).

[0293] (5) Finally, a 3×3 convolution operation is used to apply f _up2mask Extract features and reduce the channel dimension to 2D to obtain f _mask (N×2×224×384), where the first dimension of the channel dimension is the foreground output and the second dimension is the background output.

[0294] Step 6: Anomaly Alarm

[0295] This module executes only during the testing phase and primarily performs logical judgments and operations on the output of the trained decoding module. Referring to Figure 15, the specific operations are as follows:

[0296] (1) Let the first-dimensional foreground channel of the feature map f_mask output by the decoding module be C1, and the second-dimensional background channel be C2. In order to enhance the output of the target foreground and weaken the influence of the background, this invention filters out some interfering pixels by setting a threshold s. The specific operation is as follows:

[0297] a. In C1, pixels with values ​​greater than s are set to 1, and in C2, pixels with values ​​less than s are set to 0;

[0298] b. By comparing the pixel values ​​of C1 and C2 pixel by pixel, if C1 is greater than C2, set it to 1, otherwise set it to 0, and save the comparison result as the final mask image of the abnormal region (mask (384×224)).

[0299] (2) In actual scenario testing, if there are abnormalities in more than 20 consecutive input video frames, an abnormality alarm will be triggered. If there are no abnormalities in more than 50 consecutive frames, the last frame without abnormalities will be used to update and replace the input normal sample.

[0300] II. Implementation of Automatic Algorithm Selection for Scene Adaptation

[0301] This invention provides three anomaly detection algorithms, which are described in Section 1.2 regarding their characteristics, and selection experience is also provided. Considering the varying knowledge bases of users, this invention also provides a scenario-adaptive automatic algorithm selection scheme, the main principle of which employs a threshold network in a Mixture-of-Experts (MoE) system for evaluation.

[0302] The overall network structure of the hybrid expert system is shown in Figure 4. This system consists of multiple expert models, each responsible for different task scenarios. A probabilistic gating network is also designed; the output of the gating network determines which expert is selected for the current input scene image. The network output formula is as follows:

[0303] Where x is the input image, y is the output foreign object probability density map; G(x) i E represents the probability output of the threshold network for the i-th expert model. i For the i-th expert model, when G(x) i When the value is 0, the corresponding expert model will not be activated.

[0304] Based on the MoE concept, this invention uses the output weights of the threshold network as the algorithm selection index. The corresponding experts are a rail foreign object detection method based on automatic training and deployment of a standardized flow model, a rail foreign object detection method based on multi-layer feature perception of a variational autoencoder, and a rail foreign object detection method based on deep feature comparison.

[0305] Assuming the experts mentioned above have already been trained independently, it is now necessary to train a threshold network for the current scene data. The training loss function is designed as follows:

[0306] Where c represents an image sample, d is the output of the i-th expert model; c It is the ideal output, that is, the truth value. The network learns to make it so. The target value will be gradually approached; It represents the weights assigned by the threshold network to the i-th expert model, equivalent to G(x). i By reducing the loss function L, the weights of the threshold network parameters are trained.

[0307] Based on the above principles, the scene-adaptive algorithm automatically selects a solution as follows:

[0308] The first step involves using positive sample data from actual scenarios provided by users, combined with 80 target classes from the COCO dataset as foreground noise, and employing the image synthesis strategy mentioned in Section 1.3 to synthesize an automatically selected training dataset. Two key differences are: (1) The size of the noise targets is divided into three categories: large, medium, and small, maintaining a balance in the number of targets in each category. (2) STDC is used to segment the positive sample data into railway tracks, limiting the synthesized area of ​​the noise targets on the positive samples to the railway tracks only. The synthesis process records the noise segmentation area information, yielding the ground truth label T0 of the training dataset.

[0309] The second step is to train three different anomaly detection algorithms to obtain the expert model.

[0310] The third step is to freeze the three expert models and train the threshold network.

[0311] The fourth step is to input multiple images of the current task scene into the threshold network and record the weight values ​​assigned to each expert each time. The experts with higher weight values ​​and higher frequency of occurrence are selected as the final solutions.

[0312] Using a single image as input will produce expert model results with high weight values. However, the judgment of a single image is not stable. Multiple images of the task scene need to be input consecutively. The expert model with the highest frequency corresponding to each expert model is counted, and the expert model with the highest frequency is the final solution. That is: for a given frame image, the previously obtained railway track region segmentation image is input into the hybrid expert system. The threshold network of the hybrid expert system assigns weights to several expert models, and the expert model with the highest assigned weight in the current frame is recorded, with its corresponding frequency incremented by 1. Similarly, by statistically analyzing multiple consecutive frames, the statistical frequency of each expert model is obtained. The semi-supervised anomaly detection model for railway tracks corresponding to the expert model with the highest frequency, or the semi-supervised anomaly detection model for railway tracks selected by the user based on prior knowledge, is used as the detection model for the current railway track region segmentation image to complete the railway track object detection.

[0313] III. Construction of a Foreign Object Detection Algorithm System for Railway Tracks

[0314] Combining the three algorithm models and the automatic selection scheme mentioned above, a foreign object detection algorithm system for railway tracks is constructed. This system has the following characteristics:

[0315] (1) The system's input data is mainly on-site positive sample image data, which can be collected automatically or manually;

[0316] (2) The system adopts a semi-supervised training method. The input positive sample data does not need to be labeled. At the same time, negative sample data is obtained through synthesis. Finally, the positive and negative sample data are used to achieve supervised training (the fastflow scheme only uses positive sample data, while the other two use both positive and negative samples).

[0317] (3) Once the three scheme models have been trained, users can import images to evaluate the effects and decide which scheme to use; at the same time, the algorithm can be automatically selected and evaluated using the trained MOE system.

[0318] (4) The system integrates a toolchain for porting edge device platforms, which can automatically deploy the model quantization porting after the model training is completed.

[0319] (5) During the system's inference phase, it can automatically filter out interference and foreign objects outside the rail area, retaining only those within the rail area. Moreover, it operates on edge devices, ensuring real-time performance and stability.

[0320] Overall, the training process of the algorithm system of this invention is basically unmanned, the data level is semi-supervised training, no manual annotation is required, the training process uses default parameters, and only a batch of actual field data is needed. The algorithm system of this invention can complete the entire process from model training, model quantization and porting deployment in one stop; at the same time, it effectively filters out foreign interference items in the inference stage.

[0321] IV. Edge Model Training and Update

[0322] The training and porting process described in Section 1.3 is performed on the server, and the model is finally deployed to the edge. This is the mainstream approach. Furthermore, if the algorithm experiences a high number of false alarms during actual use and requires an update, the training and porting process must be repeated on the server. To further improve the efficiency of algorithm development and deployment, this invention also ports the model training process to edge devices. Executing the training executable file for the railway foreign object detection method based on a standardized flow model on the edge triggers the model update function. First, a batch of training images of the current scene is collected. Then, the model is trained using the edge CPU during nighttime hours, and finally, the original model is updated and replaced.

[0323] The railway anomaly detection algorithms used in this invention are selectable. Combining the characteristics of each algorithm, it can meet the needs of most railway foreign object detection scenarios. The scenario-adaptive foreign object detection algorithm automatic selection method provides a better combination of algorithm and scenario. Moreover, the algorithm itself is semi-supervised, which can adapt to unknown foreign object detection and greatly improve the development efficiency of the algorithm. Furthermore, the deployment at the edge improves the real-time response of the algorithm and the stability of the system.

[0324] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

A method for rail foreign object semi-supervised anomaly detection at edge end deployment, characterized in that, The method comprises the following steps: The edge device encodes and decodes the video stream captured by the camera to obtain an image frame sequence, and sets a frame interval threshold to perform frame extraction processing on the image frame sequence; For a certain frame image obtained by frame extraction, an image segmentation is performed on the frame image by using a semantic segmentation model to obtain a rail region segmentation image; For the frame image, the rail region segmentation image obtained in the foregoing is input into a hybrid expert system, a threshold network of the hybrid expert system allocates weights to a plurality of expert models, records the expert model with the highest allocated weight of the current frame, and adds 1 to the corresponding frequency; similarly, through the statistics of a plurality of continuous frame images, the statistical frequencies of the expert models are obtained, and the expert model corresponding to the highest frequency is taken as a detection model of the current rail region segmentation image to complete rail foreign matter detection. The rail foreign object semi-supervised anomaly detection method of edge deployment according to claim 1, wherein, The output of the hybrid expert system is formulated as follows: where x is the input picture, y is the output foreign matter probability density map; G(x) i is the probability output of the threshold network for the i-th expert model, E i is the i-th expert model, when G(x) i = 0, the corresponding expert model will not be activated. The rail foreign object semi-supervised anomaly detection method of edge deployment according to claim 1, wherein, The loss function L of the threshold network is as follows: wherein c denotes one image sample, output for the i-th expert model; d c is the ideal output, i.e. the true value, the network learns to make approaching the target value step by step; is the weight assigned to the ith expert model by the gating network, equivalent to G(x) i The gating network parameters weights are trained by reducing the loss function L. The rail foreign object semi-supervised anomaly detection method of edge deployment according to claim 1, characterized in that, The semi-supervised rail foreign matter detection model corresponding to the expert model comprises a rail foreign matter detection method based on automatic training and deployment of a standardized flow model, and a training process of the standardized flow model adopts an early stopping strategy based on statistical variance of training data: when a mean square deviation of output features Z of the standardized flow model is not greater than a preset threshold, it is judged that the detection model is about to collapse, and the training of the detection model is stopped; and the detection model closest to the time point before stopping the training is taken as a final result. The rail foreign object semi-supervised anomaly detection method of claim 4, wherein, The overall training data of the standardized flow model is divided into N batches for mean square error calculation, and the mean square error Z of the output features of the overall training data after the standardized flow model is calculated std The process is as follows: where S i represents the overall variance of the first i batches of data, S B represents the local variance of the current i-th batch, M B represents the local mean of the current i-th batch, M i represents the overall mean of the first i batches of data; B j and B i represent the images of the j-th and i-th batches, respectively. The rail foreign object semi-supervised anomaly detection method of claim 4, wherein, The rail foreign matter detection method based on automatic training and deployment of the standardized flow model judges whether rail foreign matter is generated by using a probability density value of the output features Z of the standardized flow model, and if the probability density value is greater than a preset threshold, it is considered that rail foreign matter is generated. The rail foreign object semi-supervised anomaly detection method of claim 4, wherein, The calculation of the track foreign object probability map h for all pixel positions by the standardized flow model output feature Z is calculated as follows: wherein Up denotes up-sampling each down-sampled feature map to the size of the original map, and the final probability value of the rail foreign matter is the mean value of the results of the three scales; The output feature maps of different sizes are represented as After obtaining the rail foreign matter probability map h, post-processing is required to obtain a segmentation result and a detection result: the rail foreign matter probability map is segmented by setting a threshold thres to obtain a segmentation map, in the segmentation map, a region greater than the set threshold thres is a rail foreign matter region, and otherwise, it is a normal region; a minimum envelope box of each segmentation map is taken as an envelope box result of rail foreign matter detection; and it is determined that rail foreign matter exists if the rail foreign matter region exists. The rail foreign object semi-supervised anomaly detection method of claim 7, wherein, The set threshold value thres is an adaptive threshold value, which is calculated in the following manner: Wherein, the threshold value thres is a rail foreign matter probability value, indicating the demarcation line between normal features and rail foreign matter features, Z mean is the mean point of feature Z on the training data, Z std is the mean square error of feature Z, Z0 is a feature point used to calculate the segmentation threshold; P(Z0) is equivalent to P(Z mean + 2Z std ), and the P(*) function is a probability density of * calculation. The rail foreign object semi-supervised anomaly detection method of claim 7, wherein, The set threshold thres is used for processing a product image edge region, and a high threshold (thres+0.1) is used for processing a product non-edge region. The rail foreign object semi-supervised anomaly detection method of claim 4, wherein, The standardized flow model adopts fastflow. The rail foreign object semi-supervised anomaly detection method of claim 4, wherein, The rail foreign matter detection model adopts fastflow as the standardized flow model, adopts a standard normal distribution as a posterior estimation of fastflow, and uses a ResNet18 network pre-trained on an ImageNet classification data set as a feature extractor. The rail foreign object semi-supervised anomaly detection method of edge deployment according to claim 1, wherein, The semi-supervised rail foreign matter detection model corresponding to the expert model comprises a rail foreign matter detection method based on multi-layer feature perception of a variational autoencoder, comprising the following steps: S1, performing self-supervised training on the autoencoder in a distillation training manner, using a deep convolutional neural network as a teacher model of the autoencoder, and performing multi-scale feature distillation training in a distillation training stage; S2, synthesizing rail abnormal image samples by using an image superposition method; S3, the variational autoencoder adopts distillation training, and the generated normal track image and the track abnormal image sample in step S2 are jointly used as the input of the variational autoencoder; the trained autoencoder in step S1 is used as the teacher model of the variational autoencoder, and the multi-scale feature supervision training of the decoding module of the variational autoencoder is performed, so that the decoding module of the variational autoencoder and the decoding module of the autoencoder are aligned in the feature space of multiple scales, and different semantic information is reconstructed; S4, the image anomaly detection of multi-feature perception is performed through the trained autoencoder and variational autoencoder. The rail foreign object semi-supervised anomaly detection method of claim 12, wherein, The step S4 is specifically as follows: 1) statistics of the mean and variance of the image features of different scales of the autoencoder and the variational autoencoder on the test set; 2) input the preprocessed image to be tested, and the autoencoder and the variational autoencoder output three image features and reconstructed images of the same size respectively; 3) normalize the image features output by the autoencoder and the variational autoencoder using the mean and variance statistics in step 1); 4) calculate the mean square error value between the three size image features of the autoencoder and the variational autoencoder, called feature reconstruction error, accumulate the feature reconstruction error according to the feature channel dimension, and perform bilinear interpolation upsampling operation to obtain the reconstruction feature error with the same width and height as the input image to be tested; 5) calculate the mean square error value between the reconstructed image of the autoencoder and the generated image of the variational autoencoder, called image reconstruction error, accumulate the reconstruction image error according to the channel dimension, and obtain the image reconstruction error with the same dimension as the feature reconstruction error; 6) sum the feature reconstruction error and the image reconstruction error of three sizes, and perform min-max standardization processing to map the error value to the [0, 1] interval, and the area where the above error value is greater than the error threshold is judged as the abnormal area of the track image, otherwise it is the normal area of the track image. The rail foreign object semi-supervised anomaly detection method of claim 12, wherein, The training process of the autoencoder is as follows: Prepare large-scale open source image data for self-supervised distillation training; Build an autoencoder model: the encoding module of the autoencoder is composed of four down-sampling network layers, and the decoding module of the autoencoder is composed of four up-sampling network layers, each down-sampling network layer reduces the width and height of the input image feature by half, and each up-sampling network layer expands the width and height of the input image feature by one time; In the training stage, the image is preprocessed, first scaled to 256x256 size, then enhanced by image horizontal flip, HSV color domain jitter and small angle image rotation according to the preset ratio; After the image is preprocessed, it is input into the autoencoder and the teacher model of the autoencoder respectively to obtain the corresponding reconstructed image; Calculate the mean square error between the 32x32, 64x64 and 128x128 size image features output by the decoding module of the autoencoder and the corresponding scale image features of the teacher model of the autoencoder, and calculate the mean square error between the reconstructed image and the input image, which is used for autoencoder model training; The loss function of the autoencoder model training is optimized by back propagation to optimize the parameters of the autoencoder model, and a pre-trained autoencoder is obtained. The rail foreign object semi-supervised anomaly detection method of claim 12, wherein, The loss function formula of the self-encoder model training is as follows: wherein, representing the encoder-decoder module w i x h i image features of a size, representing the teacher model w i x h i image features of a size, w i x h i image features of a size and between the mean square error; I r denotes the autoencoder reconstructed image, I denotes the input image, (I r -I) 2 denotes the image I r and the mean square error of I. The rail foreign object semi-supervised anomaly detection method of claim 12, wherein, The teacher model of the autoencoder is one of ResNet50, VGG19 and HRNet. The rail foreign object semi-supervised anomaly detection method of claim 12, wherein, The step S2 is specifically as follows: Prepare the COCO instance segmentation dataset, use the object instances in the data as pseudo foreign objects on the rail, and extract the mask map of each instance; the COCO dataset includes common objects, animals and pedestrians in life, which conforms to the possible foreign objects on the rail; Obtain the normal image data of the rail, and filter the special image samples of the train passing through; Based on the instance mask map, the instance image is superimposed and synthesized with the rail area at a preset ratio, and the synthesis formula is as follows: where I m denotes a COCO image, I n denotes a normal track image sample, I a denotes a synthetic track pseudo-anomalous image, M a denotes a mask image of object instances in image I m ​ is a mask map M a the negated result of the mask map M, and β is an overlay coefficient for image blending. The rail foreign object semi-supervised anomaly detection method of claim 12, wherein, The training process of the variational autoencoder is as follows: Prepare the normal image of the rail and the synthesized pseudo abnormal image of the rail, and divide the training set and the test set; Build a variational autoencoder model, which has a similar structure to the autoencoder, adds one down-sampling network layer and one up-sampling network layer based on the autoencoder, encodes the input image into a vector distribution in the hidden space, and then reconstructs a new image; In the image preprocessing stage, the input image is subjected to HSV color domain jitter enhancement and small-angle horizontal rotation enhancement; After the image is preprocessed, the normal image of the rail and the synthesized pseudo abnormal image of the rail are input into the variational autoencoder to generate normal image samples, and the normal image is input into the autoencoder to obtain a reconstructed image. The variational autoencoder decoding module 32x32, 64x64, 128x128 size image feature and the mean square error between the image feature corresponding to the scale of the autoencoder is calculated, and the mean square error between the image generated by the variational autoencoder and the normal image is calculated, and the feature reconstruction and image generation loss function L is obtained g : In the formulae, representing a variational autoencoder decoding module w i x h i image features of a size, representing the encoder-decoder module w i x h i image features of a size, w i x h i image features of a size and the mean squared error between I n denote normal images input to the variational autoencoder model and the autoencoder model, respectively, I g denote normal images generated by the variational autoencoder model, (I g -I n ) 2 denote the image I g and I n the mean squared error between Based on the loss function of the variational autoencoder, the model parameters of the variational autoencoder are optimized, so that the variational autoencoder has the ability to generate normal images from both abnormal images and normal images. The rail foreign object semi-supervised anomaly detection method of claim 12, wherein, The loss function L of the variational autoencoder VAE is defined as follows: In the formula, L' g represents the feature reconstruction and image generation loss function of input normal images, L" g represents the feature reconstruction and image generation loss function of input pseudo abnormal images, λ1, λ2 and λ3 respectively represent the weight coefficients of different losses, The KL divergence loss of the normal distribution of the input data and the standard normal distribution. The rail foreign object semi-supervised anomaly detection method of claim 19, wherein, The The following equation is used to calculate: where N is the dimension of the hidden space vector, μ n and σ n denote the mean and variance of the n-th dimension of the hidden space vector, respectively, and μ and σ denote the mean and variance of the hidden space vector, respectively. The rail foreign object semi-supervised anomaly detection method of edge deployment according to claim 1, wherein, The semi-supervised rail foreign object detection model corresponding to the expert model includes a rail foreign object detection method based on deep feature comparison, comprising the following steps: Training stage: Obtain normal samples and perform overlapping block processing, and then generate rail abnormal samples; The rail abnormal sample and the rail normal sample form a training sample pair for preprocessing, and the preprocessed data is subjected to feature extraction, fusion and decoding through the encoding module and the decoding module to complete the training; Test stage: First, fix the detection scene, obtain an image without foreign objects in the next frame of the current scene of the rail as a rail normal sample, and the rail to be detected sample is the real-time camera scene image; Then, the rail normal sample and the rail to be detected sample are preprocessed as a test sample pair; Use the trained encoding module and decoding module to output the corresponding prediction results, and finally perform logical judgment on the decoded results to realize real-time rail abnormality alarm. The rail foreign object semi-supervised anomaly detection method of claim 21, wherein, In the training stage, the rail abnormal sample image generation method is as follows: (1) specify the foreign object generation area image; (2) simulate different sizes and colors of unknown rail foreign objects that may appear in a single scene: Use the public segmentation task dataset to obtain the foreground image of different categories of objects; Scale, rotate and color space transform the foreground image by different proportions according to the preset probability to simulate foreign objects of different sizes and colors, and obtain the rail foreign object image. The rail foreign object semi-supervised anomaly detection method of claim 21, wherein, In the training stage, the rail abnormal sample image generation method is as follows: A, a foreign matter generation region specifying image I s ; B, use a few rail scene images to simulate rail foreign objects in most scenes: a, generates an image I of the foreign matter generation region s randomly adjusts the brightness, saturation, and hue of the image I, divides the adjusted image into a grid of equal proportions, and rearranges the grid to obtain a noise image I that simulates a structural anomaly n ; b, use the size of the region of interest or the size of the rail region to randomly generate a two-dimensional Bernoulli noise P of the same size, and set a threshold to obtain a binary processing result M; c. The structural anomaly noise foreground image I' is obtained by the following formula n The rail foreign object image is obtained by simulating the foreign objects that may appear in different rail scenes. I' n =α(M⊙I n )+(1-α)(M⊙I s ); Wherein, α is a transparent coefficient introduced, the purpose is to better balance the foreign matter generated area image I s and noise image I n fusion, make the image after fusion closer to the real anomaly. The rail foreign object semi-supervised anomaly detection method of claim 21, wherein, In the training phase, the foreign matter image I in the rail anomaly sample a After generation, the abnormal area image I' is generated again, and the corresponding formula is: wherein I s generates a foreign matter area image, The processing result M is obtained by reversing; a two-dimensional Perlin noise P of the same region size is randomly generated using the region size of interest or the track region size, and a threshold is set to obtain a binary processing result M; replacing the foreign matter generated region image I of the normal sample with the abnormal region image I' s , i.e. splicing to the region of interest or the track region of the corresponding normal sample, to obtain the final track abnormal sample. The rail foreign object semi-supervised anomaly detection method of claim 21, wherein, The encoding module adopts two feature extraction networks with the same weight to form a twin network, processes the preprocessed abnormal samples and normal samples, and outputs three different levels of features; Feature map f of normal sample 1_normal (N x 64 x 56 x 96), f 2_normal (N x 128 x 28 x 48), and f 3_normal (N x 256 x 14 x 24); and the feature map f of the abnormal sample 1_anomaly (N x 64 x 56 x 96), f 2_anomaly (N x 128 x 28 x 48), and f 3_anomaly (N x 256 x 14 x 24); Wherein, the first dimension N represents the number of samples input into the network in an iterative training, the second dimension represents the number of channels, and the third and fourth dimensions represent the height and width of the feature map; Then, the normal sample and the abnormal sample corresponding to the three different levels of feature maps are spliced in the channel dimension: where i = 1, 2, 3, corresponding to three different levels in the middle of the feature extraction network, represents a tensor channel concatenation operation, and the final result is denoted as f 1_concat (N x 128 x 56 x 96), f 2_concat (N x 256 x 28 x 48), and f 3_concat (N x 512 x 14 x 24); Finally, the differences between the three different levels of feature maps corresponding to the normal samples and the abnormal samples are calculated respectively, and the operation is as follows: f i_diff = MSE(f i_normal ,f i_anomaly ); where i = 1, 2, 3, corresponding to three different levels in the middle of the feature extraction network, MSE(·) represents the sum of the mean square error of each position of two input tensors, and the result is recorded as f 1_diff (N x 64 x 56 x 96), f 2_diff (N x 128 x 28 x 48), and f 3_diff (N x 256 x 14 x 24). The rail foreign object semi-supervised anomaly detection method of claim 21, wherein, The encoding module performs feature fusion on the features f 1_concat , f 2_concat , and f 3_concat obtained by channel splicing through a multi-scale fusion network: First, f 1_concat , f 2_concat and f 3_concat are extracted by using a 3x3 convolution with half of the number of channels and a 3x3 convolution with the same number of channels, followed by a 1x1 convolution to increase the nonlinearity of the network, resulting in f 1_conv (N x 64 x 56 x 96), f 2_conv (N x 128 x 28 x 48) and f 3_conv (N x 256 x 14 x 24); Note f 3_msff = f 3_conv , the feature map f 3_msff is doubled up-sampled to N x 256 x 28 x 48 size, and then a 3 x 3 convolution operation with half the number of channels is connected to obtain the feature map f _temp (N x 128 x 28 x 48), and then f _temp is element-wise added to f 2_conv to obtain f 2_msff (N x 128 x 28 x 48); Similarly for f 2_msff After two times up-sampling and 3x3 convolution operation with half channel number, f 1_conv Element-wise addition to get f 1_msff (N x 64 x 56 x 96); Finally output the multi-scale fused feature f 3_msff (N x 256 x 14 x 24), f 2_msff (N x 128 x 28 x 48), and f 1_msff (N x 64 x 56 x 96). The rail foreign object semi-supervised anomaly detection method of claim 21, wherein, The decoding module is repeatedly composed of multiple up-sampling convolution operations combined with 3*3 convolution operations, and is executed from deep features, and finally a 3*3 convolution operation is used to reduce the channel dimension to 2 dimensions. The rail foreign object semi-supervised anomaly detection method of claim 21, wherein, The training process of the decoding module is as follows: The output f of the encoding module 1_encode f 2_encode and f 3_encode As the input to the decoding module, let f 3_decode =f 3_encode , for f 3_decode After doubling the upsampling of (N×256×14×24) to a size of N×256×28×48, a 3×3 convolution operation is performed to reduce the channel dimension and obtain the feature map f. _tepm1 (N×128×28×48), finally f _temp1 with f 2_encode f is obtained by concatenation along the channel dimension. 2_decode (N×256×28×48); Similarly, for f 2_decode A 3x3 convolution operation is connected after the doubling of the sampling to obtain the feature map f _tepm2 (Nx64x56x96), and the feature map f _tepm2 is concatenated with f 1_encode In the channel dimension, f 1_decode (Nx128x56x96) is obtained. To further reduce the computational cost of the model, f 1_decode is connected with a 3x3 convolution operation to reduce the channel dimension to get the feature map f 0_decoder (Nx32x112x192); f 0_decoder Continuing with the doubling of the up-sampling followed by a 3x3 convolution operation that keeps the channel dimension constant results in a feature map f of the same size as the input sample, 384x224 _up2mask (N x 32 x 224 x 384); Finally, a 3x3 convolution operation is applied to f _up2mask The features are extracted and the channel dimension is reduced to 2 dimensions to obtain f _mask (N x 2 x 224 x 384), where the first dimension of the channel dimension is the foreground output and the second dimension is the background output. A rail foreign object semi-supervised anomaly detection system for edge deployment, characterized in that, The program is loaded and executed by the processor to realize the edge deployment of the track foreign matter semi-supervised anomaly detection method in any one of claims 1 to 28; including a semantic segmentation model and a hybrid expert system; wherein the semantic segmentation model is used for image segmentation on a certain frame image obtained by frame extraction to obtain a track region segmentation image; the hybrid expert system includes a threshold network and a plurality of expert models, the output of the threshold network determines which expert model is activated, and the track foreign matter semi-supervised anomaly detection model corresponding to the expert model is used as the detection model of the current track region segmentation image and completes the track foreign matter detection. A server comprising a processor and a memory, characterized in that The memory stores at least one program, and the program is loaded and executed by the processor to realize the edge deployment of the track foreign matter semi-supervised anomaly detection method in any one of claims 1 to 28. A computer readable storage medium, the storage medium having stored therein at least one segment of a program, characterized in that, The program is loaded and executed by the processor to realize the edge deployment of the track foreign matter semi-supervised anomaly detection method in any one of claims 1 to 28.

Images