A semi-supervised anomaly detection method and system for rail foreign matter deployed at the edge

Through the semi-supervised anomaly detection method deployed at the edge end, combined with hybrid expert systems and other technologies, the problems of high training difficulty, high labeling cost and poor generalization in the existing technology are solved, and efficient and real-time railway foreign matter detection is achieved to adapt to Complex scenes and unknown foreign object detection.

CN118552800BActive Publication Date: 2024-11-19GUANGZHOU EMBEDDED MASCH TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411027178.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-19
Estimated Expiration
2044-07-30

AI Technical Summary

Technical Problem

The existing railway track foreign object detection technology has difficulty in training, high labeling cost, poor generalization, and inability to deal with unknown foreign objects. It is also deployed on the server side to cause poor detection real-time and stability.

Method used

Adopt the semi-supervised anomaly detection method deployed by edge end, and through hybrid expert systems and standardized flow models, multi-layer feature perception of variational autoencoders, deep feature comparison and other technologies, automated detection and model updates are realized, and unknown foreign object detection is supported, reducing artificial Label requirements to improve scenario adaptability and real-timeness.

Benefits of technology

It realizes highly automated railway track foreign matter detection, shortens the development cycle, improves the real-time and stability of detection, supports the detection of foreign matters of different sizes, adapts to complex railway track scenarios, and reduces the cost of algorithm development and deployment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118552800B_ABST
    Figure CN118552800B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and system for semi-supervised anomaly detection of foreign bodies on rails deployed at the edge, the method comprising the following steps: the edge device encodes and decodes the video stream captured by the camera to obtain an image frame sequence, and performs frame extraction processing; for a certain frame image obtained by frame extraction, a semantic segmentation model is used to perform image segmentation to obtain a rail area segmentation image; a single image as input will produce an expert model result with a high weight value, but the judgment of a single image is not stable, and multiple images of the task scene need to be continuously input, and the frequency of the highest weight corresponding to each expert model is counted, and the expert model corresponding to the highest frequency is the final solution. The present invention supports the automatic selection of scene-adaptive foreign body detection algorithms, and users can select based on prior knowledge, or by the scene-adaptive automatic algorithm selection method, users only need to provide a batch of image data of the current scene to evaluate the optimal algorithm selection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of rail foreign body detection, and in particular to a rail foreign body semi-supervised anomaly detection method and system deployed at an edge end. Background Art

[0002] Railways are one of the most important transportation infrastructures. The presence of foreign objects (people, animals, debris flow or tools) on the rails may cause train accidents and damage to railway equipment. Therefore, rapid detection and handling of foreign objects on the rails is crucial. Traditional methods of identifying foreign objects require the installation of multiple cameras along the rails, which lack analysis functions, require manual inspection, are costly, and have a high risk of omission.

[0003] In recent years, with the development of artificial intelligence technology, more and more intelligent analysis methods have been applied to foreign body detection on rails. The problems with existing methods are: the current algorithms are mainly based on supervised training, which requires a lot of manual annotation, has a long development cycle and poor scene adaptability, and cannot cope with emergencies; at the same time, the algorithms are deployed on the server side, which greatly reduces the real-time and stability of detection considering the overhead and delay of data transmission.

[0004] The main methods currently used for foreign object detection in the railway field are: (1) using a generative adversarial network to compare the differences between the input image and the reconstructed image to determine foreign objects; (2) using target detection and segmentation methods for foreign object detection; (3) using traditional visual anomaly detection methods to detect rail continuity, abnormal edges, variance and correlation, and optical flow field anomalies.

[0005] For example, Technical Solution 1 (patent application number CN202110785139.1, patent application name "A semi-supervised foreign body detection method for rail images") uses a high-definition camera to obtain a two-dimensional image of the railway track and converts it into a grayscale image; adopts a GAN-based anomaly detection method, and only inputs normal rail images in the model training phase; in the test phase, the model can reconstruct the image of the normal sample, but cannot reconstruct the abnormal sample image; in the inference phase, a threshold is set, and when the difference between the reconstructed image and the original image is greater than the set threshold, the image is judged to be abnormal. The present invention adopts a semi-supervised approach, does not require abnormal sample images during model training, and can accurately detect rail foreign body images in the inference phase.

[0006] The disadvantages of technical solution 1 are: (1) the generative adversarial network used is difficult to train and has high training costs; (2) the detection area is not limited, and global detection will increase the false alarm recall rate.

[0007] Technical solution 2 (patent application number CN202211323105.1, patent application name "A method and system for identifying foreign objects on rails based on a monocular thermal infrared camera") uses yolov5 to perform preliminary positioning of foreign objects, and then uses different models for semantic segmentation for different target categories. The original image set is automatically labeled according to the segmentation results for training the improved semantic segmentation network proposed in this patent. The temperature information carried by the thermal image is added to the calculation weight of the improved semantic segmentation network, the original image is segmented, and a foreign object segmentation map is obtained.

[0008] The disadvantages of technical solution 2 are: (1) It uses supervised learning methods such as target detection and segmentation, which requires data collection and labeling for specific scenarios, and has high labor costs; (2) It uses a transformer structure, which is not easy to transplant to the edge.

[0009] Technical Solution 3 (patent application number CN202211053889.0, patent application name "A method for detecting foreign objects on rails") aims to solve the technical problems existing in the existing technology of video surveillance. The method for detecting foreign objects on rails includes rail obstacle detection: reading the monitoring area, detecting rail continuity, abnormal edges, variance and correlation, and optical flow field abnormal points, and fusing the detected data to obtain rail obstacle size and distance estimation.

[0010] The disadvantages of technical solution three are: (1) the traditional visual foreign object detection method is used to detect rail obstacles, but the generalization ability is limited and the scene adaptability is not high; (2) the number of foreign object detection targets is limited, and only pedestrian detection is supported. Retraining is required to add categories.

[0011] Summarizing the above technical solutions, the shortcomings of the prior art are summarized as follows:

[0012] (1) The algorithm solutions used are either difficult to train, or have high labeling costs, or have supervised training but poor generalization and cannot handle unknown foreign objects, such as GAN, target detection algorithms, and traditional foreign object detection methods;

[0013] (2) Railway scenes are relatively complex, and a single algorithm is difficult to meet the needs of different scene changes;

[0014] (3) The algorithm is not lightweight and is not suitable for deployment on edge devices;

[0015] (4) There is a high degree of manual involvement, the degree of automation is not high, and there is a lack of unified multi-algorithm support and simple deployment methods. Summary of the invention

[0016] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a semi-supervised anomaly detection method for foreign objects on rails deployed at the edge. The method supports three semi-supervised anomaly detection algorithms, which can be automatically selected according to scene changes to meet the needs of foreign object detection of different sizes, and all support unknown foreign object detection to deal with different emergencies. The algorithm development process only needs to provide positive sample images without additional annotation, and the entire process from model training, quantification and transplantation deployment can be completed in one stop, with a high degree of automation and greatly shortening the development cycle. The algorithm is deployed on the edge device to ensure the real-time and stability of the detection system. Finally, the present invention also supports edge training of some algorithms, and the model can be updated at the edge according to scene changes.

[0017] Another object of the present invention is to provide a semi-supervised anomaly detection system for foreign objects on rails deployed at an edge end.

[0018] The purpose of the present invention is achieved through the following technical solutions:

[0019] A semi-supervised anomaly detection method for rail foreign bodies deployed at the edge comprises the following steps:

[0020] The edge device encodes and decodes the video stream captured by the camera to obtain an image frame sequence, and sets a frame interval threshold to extract frames from the image frame sequence;

[0021] For a certain frame image obtained by frame extraction, the semantic segmentation model is used to perform image segmentation to obtain a segmented image of the rail area;

[0022] For a certain frame of image, the previously obtained rail area segmentation image is input into the hybrid expert system. The threshold network of the hybrid expert system assigns weights to several expert models, records the expert model with the highest weight assigned to the current frame, and adds 1 to the corresponding frequency. Similarly, through the statistics of multiple consecutive frames of images, the statistical frequency of each expert model is obtained, and the semi-supervised anomaly detection model for foreign objects on the rail corresponding to the expert model with the highest frequency is used as the detection model for the current rail area segmentation image and completes the detection of foreign objects on the rail.

[0023] The output formula of the hybrid expert system is as follows:

[0024] ;

[0025] in, For input image, To output the foreign matter probability density map; For the threshold network The probability output of the expert model is is the i-th expert model, when , the corresponding expert model will not be activated.

[0026] In theory, The larger the The greater the probability that an expert model will be selected, the output of the expert model will become the final output.

[0027] The loss function of the threshold network is as follows:

[0028] ;

[0029] Among them, c represents an image sample, For the The output of the expert model; is the ideal output, that is, the true value. The network learns so that The target value will be gradually approached; is the threshold network assigned to the The weight of an expert model is equivalent to ; By reducing the loss function , train the threshold network parameter weights.

[0030] The semi-supervised rail foreign body detection model corresponding to the expert model includes a rail foreign body detection method based on automatic training and deployment of a standardized flow model. The training process of the standardized flow model adopts an early stopping strategy based on the statistical variance of training data: when the standardized flow model outputs a feature When the mean square error is not greater than the preset threshold, it is determined that the detection model is about to crash and the training of the detection model is stopped; and the detection model with the time point closest to the time before the training is stopped is taken as the final result.

[0031] The overall training data of the standardized flow model is divided into The mean square error is calculated for each batch, and the mean square error of the output features of the overall training data after passing through the standardized flow model is calculated. The process is as follows:

[0032] ;

[0033] ;

[0034] ;

[0035] in, Before The overall variance of the batch data, Indicates the current The local variance of the batch, Indicates the current The local mean of the batches, Before The overall mean of the batch data; and Respectively represent the jth and The number of images in a batch.

[0036] The rail foreign body detection method based on automatic training and deployment of the standardized flow model outputs features through the standardized flow model The probability density value of is used to determine whether a foreign object has appeared on the rail. If it is greater than a preset threshold, it is considered that a foreign object has appeared on the rail.

[0037] Output features through the standardized flow model Calculate the probability map of foreign objects on the rails at all pixel locations The calculation method is as follows:

[0038] ;

[0039] in, It means that each downsampled feature map is upsampled to the size of the original image, and the final rail foreign body probability value is the average of the results of the three scales; , , Represents output feature maps of different sizes;

[0040] Get the probability map of foreign matter on the rails After that, post-processing is required to obtain the segmentation results and detection results: by setting the threshold To segment the foreign body probability map of the rail, we get a segmentation map. In the segmentation map, the probability of foreign bodies exceeding the set threshold The area with a foreign object on the rail is the foreign object area on the rail, otherwise it is a normal area; the minimum envelope box of each segmentation image is used as the envelope box result of the foreign object detection on the rail; if there is a foreign object area on the rail, it is determined that there is a foreign object on the rail.

[0041] The set threshold To adaptively set the threshold, it is calculated as follows:

[0042] ;

[0043] Among them, setting the threshold is a probability value of foreign matter on the rail, indicating the dividing line between normal features and foreign matter features on the rail. Features for training data The mean point of Features The mean square error, is the feature point used to calculate the segmentation threshold; and equivalence, The function is to calculate the probability density of *.

[0044] The set threshold Used to process the edge area of ​​the product image and use a high threshold in the non-edge area of ​​the product to be processed.

[0045] The standardized flow model adopts fastflow.

[0046] The rail foreign object detection method based on automatic training and deployment of a standardized flow model adopts fastflow as the standardized flow model, adopts the standard normal distribution as the posterior estimation of fastflow, and uses the ResNet18 network pre-trained on the ImageNet classification data set as the feature extractor.

[0047] The semi-supervised rail foreign body detection model corresponding to the expert model includes a rail foreign body detection method based on multi-layer feature perception of variational autoencoder, including the following steps:

[0048] S1. Use distillation training to perform self-supervisory training on the autoencoder. Use a deep convolutional neural network as the teacher model of the autoencoder. Perform distillation training of multi-scale features in the distillation training stage.

[0049] S2, synthesizing rail abnormality image samples by using image superposition method;

[0050] S3, the variational autoencoder adopts distillation training, and the generated normal rail image and the rail abnormal image sample of step S2 are used as the input of the variational autoencoder; the autoencoder trained in step S1 is used as the teacher model of the variational autoencoder, and multi-scale feature supervision training of the decoding module of the variational autoencoder is performed, so that the decoding module of the variational autoencoder is aligned with the decoding module of the autoencoder in feature spaces of multiple scales, and different semantic information is reconstructed;

[0051] S4. Multi-feature-aware image anomaly detection through trained autoencoders and variational autoencoders.

[0052] The S4 is specifically as follows:

[0053] 1) Statistical autoencoder and variational autoencoder on the test set of different scale image features mean and variance;

[0054] 2) Input the preprocessed image to be tested, and the autoencoder and variational autoencoder output three image features and reconstructed images of the same size respectively;

[0055] 3) Use the mean and variance of the statistics in step 1) to normalize the image features output by the autoencoder and variational autoencoder;

[0056] 4) Calculate the mean square error between the three-size image features of the autoencoder and the variational autoencoder, called the feature reconstruction error, accumulate the feature reconstruction errors according to the feature channel dimension, and perform bilinear interpolation upsampling operations to obtain the reconstructed feature error with the same width and height as the input image to be tested;

[0057] 5) Calculate the mean square error between the image reconstructed by the autoencoder and the image generated by the variational autoencoder, called the image reconstruction error, and accumulate the reconstructed image errors by channel dimension to obtain an image reconstruction error of the same dimension as the feature reconstruction error;

[0058] 6) Sum the feature reconstruction error and image reconstruction error of the three sizes, perform min-max normalization, and map the error value to In the interval, the area where the error value is greater than the error threshold is judged as an abnormal area of ​​the rail image, otherwise it is a normal area of ​​the rail image.

[0059] The training process of the autoencoder is as follows:

[0060] Prepare large-scale open-source image data for self-supervised distillation training;

[0061] Build an autoencoder model: The encoding module of the autoencoder consists of 4 downsampling network layers, and the decoding module of the autoencoder consists of 4 upsampling network layers. Each downsampling network layer reduces the width and height of the input image features by half, and each upsampling network layer doubles the width and height of the input image features.

[0062] Image preprocessing is performed during the training phase. First, the input image is scaled to 256×256, and then image horizontal flip enhancement, HSV color gamut dithering enhancement, and small-angle image rotation enhancement are performed according to a preset ratio.

[0063] After the image is preprocessed, it is input into the autoencoder and the teacher model of the autoencoder to obtain the corresponding reconstructed image;

[0064] Calculate the mean square error between the image features of 32×32, 64×64, and 128×128 sizes output by the decoding module of the autoencoder and the image features of the corresponding scales of the teacher model of the autoencoder, and calculate the mean square error between the reconstructed image and the input image for autoencoder model training;

[0065] The loss function of the autoencoder model training is back-propagated to optimize the autoencoder model parameters to obtain a pre-trained autoencoder.

[0066] The loss function calculation formula for the autoencoder model training is as follows:

[0067] ;

[0068] in, Represents the autoencoder decoding module Image features of size, Represents the teacher model Image features of size, express Image features under size and The mean square error between represents the image reconstructed by the autoencoder, represents the input image, Representing images and The mean square error.

[0069] The teacher model of the autoencoder is one of ResNet50, VGG19, and HRNet.

[0070] The S2 is specifically as follows:

[0071] Prepare the COCO instance segmentation dataset, use the object instances in the data as pseudo foreign objects on the rails, and extract the mask map of each instance; the COCO dataset includes common objects, animals, and pedestrians in life, which are consistent with the foreign objects that may appear on the rails;

[0072] Get normal rail image data and filter special image samples where trains pass by;

[0073] Based on the instance mask map, the instance image and the rail area are superimposed and synthesized according to a preset ratio. The synthesis formula is as follows:

[0074] ;

[0075] in, represents a COCO image, represents a normal image sample of a railway track, represents the synthetic rail pseudo-anomaly image, Indicates that an object instance is in the image The mask map of It is a mask image The negation result of is the stacking coefficient of image synthesis.

[0076] The training process of the variational autoencoder is as follows:

[0077] Prepare normal rail images and synthetic pseudo-abnormal rail images, and divide them into training sets and test sets;

[0078] Build a variational autoencoder model, which has a similar structure to the autoencoder. On the basis of the autoencoder, it adds a downsampling network layer and an upsampling network layer to encode the input image into a vector distribution in the latent space, and then reconstruct it to generate a new image.

[0079] In the image preprocessing stage, the input image is enhanced by HSV color domain dithering and small-angle horizontal rotation;

[0080] After the image is preprocessed, the normal rail image and the synthesized pseudo-abnormal rail image are input into the variational autoencoder to generate normal image samples, and the normal image is input into the autoencoder to obtain the reconstructed image;

[0081] Calculate the mean square error between the image features of the variational autoencoder decoding module with sizes of 32×32, 64×64, and 128×128 and the image features of the corresponding scale of the autoencoder, and calculate the mean square error between the image generated by the variational autoencoder and the normal image to obtain the feature reconstruction and image generation loss function :

[0082] ;

[0083] In the formula, Represents the variational autoencoder decoding module Image features of size, Represents the autoencoder decoding module Image features of size, express Image features under size and The mean square error between represents the normal images input into the variational autoencoder model and the autoencoder model respectively, represents a normal image generated by the variational autoencoder model, Representing images and The mean square error between

[0084] Based on the loss function of the variational autoencoder, the model parameters of the variational autoencoder are optimized so that the variational autoencoder has the ability to generate normal images when both abnormal images and normal images are input.

[0085] The loss function of the variational autoencoder is The definition is as follows:

[0086] ;

[0087] In the formula, represents the feature reconstruction and image generation loss function of the input normal image, represents the feature reconstruction and image generation loss function of the input pseudo-anomaly image, , and Represent the weight coefficients of different losses respectively, is the KL divergence loss between the normal distribution of the input data and the standard normal distribution.

[0088] Said Calculated by the following formula:

[0089] ;

[0090] in, is the dimension of the latent space vector, and They represent the latent space vector The mean and variance of the dimensions, and They represent the mean and variance of the latent space vector respectively.

[0091] The semi-supervised rail foreign body detection model corresponding to the expert model includes a rail foreign body detection method based on deep feature comparison, comprising the following steps:

[0092] Training phase:

[0093] Obtain normal samples and perform overlapping block processing, and then generate rail abnormal samples;

[0094] The abnormal rail samples and normal rail samples form a training sample pair for preprocessing, and the encoding module and decoding module extract, fuse and decode the features of the preprocessed data to complete the training;

[0095] Testing phase:

[0096] First, fix the detection scene, obtain the next frame of the current rail scene without foreign objects as the normal rail sample, and the rail to be inspected sample is the real-time lens scene image;

[0097] Then, the normal rail sample and the rail sample to be inspected are treated as a test sample pair and preprocessed;

[0098] The trained encoding module and decoding module are used to output the corresponding prediction results. Finally, logical judgment is made on the decoded results to issue real-time rail abnormality alarms.

[0099] During the training phase, the method for generating the foreign body image in the rail abnormality sample is as follows:

[0100] (1) Specify the image of the foreign body generation area;

[0101] (2) Simulate unknown rail foreign objects of different sizes and colors that may appear in a single scene:

[0102] Use public segmentation task datasets to obtain foreground images of objects of different categories;

[0103] The resolution of the foreground image is scaled, rotated and the color space transformed at different ratios according to the preset probability to simulate foreign objects of different sizes and colors and obtain the foreign object image on the rail.

[0104] During the training phase, the method for generating the foreign body image in the rail abnormality sample is as follows:

[0105] A. Specify the image of the foreign body generation area ;

[0106] B. Use a few rail scene images to simulate rail foreign bodies in most scenarios:

[0107] a. Generate regional image of foreign matter The brightness, saturation and hue of the image are randomly adjusted, and then the adjusted image is divided into grids of equal proportions, and then the grids are rearranged to obtain a noise image simulating structural abnormalities. ;

[0108] b. Using the size of the region of interest or the size of the rail area, randomly generate a two-dimensional Perlin noise of the same size , and set the threshold to perform binarization processing to obtain the processing result ;

[0109] c. Obtain the noise foreground image with structural abnormality through the following formula , simulate the foreign objects that may appear in different rail scenes, and thus obtain the rail foreign object image:

[0110] ;

[0111] in, The transparency factor introduced is to better balance the image of the foreign body generation area and noisy image The fusion of the two images makes the fused image closer to the real anomaly.

[0112] During the training phase, the foreign body images in the rail abnormality samples After generation, generate the abnormal area image , the corresponding formula is:

[0113] ;

[0114] in, Generates an area image for the foreign object, To process the results Invert to get; use the size of the region of interest or the size of the rail area to randomly generate a two-dimensional Perlin noise of the same area size , and set the threshold to perform binarization processing to obtain the processing result ;

[0115] The abnormal area image Image of the foreign matter generation area replacing the normal sample , that is, spliced ​​to the region of interest or rail area corresponding to the normal sample to obtain the final rail abnormal sample.

[0116] The encoding module uses two feature extraction networks with the same weight to form a twin network, processes the preprocessed abnormal samples and normal samples, and outputs three different levels of features:

[0117] Feature map of normal samples , and ;

[0118] And the characteristic map of abnormal samples , and ;

[0119] Among them, the first dimension N represents the number of samples input to the network in one iterative training, the second dimension represents the number of channels, and the third and fourth dimensions represent the height and width of the feature map;

[0120] Then, the feature maps of three different levels corresponding to normal samples and abnormal samples are spliced ​​according to the channel dimension:

[0121] ;

[0122] Among them, i=1,2,3, corresponding to the three different levels in the middle of the feature extraction network, It represents the tensor channel concatenation operation, and the final result is recorded as , and ;

[0123] Finally, the differences between the three different levels of feature maps corresponding to normal samples and abnormal samples are calculated respectively. The operation is as follows:

[0124] ;

[0125] Among them, i=1,2,3, corresponding to the three different levels in the middle of the feature extraction network, It represents the mean square error of each position of the two input tensors, and the result is recorded as , and .

[0126] The encoding module uses a multi-scale fusion network to obtain the feature f obtained by channel splicing. 1_concat 、f 2_concat and f 3_concat Perform feature fusion:

[0127] First, a channel number is halved. Convolution and a constant number of channels Convolution on f 1_concat 、f 2_concat and f 3_concat Extract features and then use a Convolution increases the nonlinearity of the network, and we get , and ;

[0128] Remember f 3_msff = f 3_conv , the feature map f 3_msff Upsampled twice After the size is connected, the number of channels is halved Convolution operation obtains feature map , then f _temp With f 2_conv Add element by element to get ;

[0129] Similarly for f 2_msff Double upsampling and halving the number of channels After the convolution operation, 1_conv Add element by element to get ;

[0130] Finally, the multi-scale fusion features are output , and .

[0131] The decoding module is composed of multiple upsampling convolution operations combined The convolution operation is repeated, starting from the deep features, and finally using a The convolution operation reduces the channel dimension to 2 dimensions.

[0132] The training process of the decoding module is as follows:

[0133] The output f of the encoding module 1_encode 、f 2_encode and f 3_encode As the input of the decoding module, let f 3_decode =f 3_encode ,right Upsample twice to Size followed by one The convolution operation reduces the channel dimension to obtain the feature map Finally, f _temp1 With f 2_encode Concatenate in the channel dimension ;

[0134] Similarly, for f 2_decode Connect one after double upsampling Convolution operation obtains feature map , and then the feature map f _tepm2 With f 1_encode Concatenate in the channel dimension ;

[0135] In order to further reduce the amount of calculation of the model, we continue to 1_decode After upsampling twice, connect a The convolution operation reduces the channel dimension to obtain the feature map ;

[0136] f 0_decoder Continue to upsample twice and connect a channel dimension unchanged The convolution operation obtains the same size as the input sample Feature map ;

[0137] Finally, use a The convolution operation is performed on f _up2mask Extract features and reduce the channel dimension to 2 dimensions to obtain , where the first dimension of the channel dimension is the foreground output and the second dimension is the background output.

[0138] Another object of the present invention is achieved by the following technical solutions:

[0139] A rail foreign body semi-supervised anomaly detection system deployed at the edge includes a semantic segmentation model and a hybrid expert system; wherein the semantic segmentation model is used to perform image segmentation on a frame image obtained by frame extraction to obtain a rail area segmentation image; the hybrid expert system includes a threshold network and several expert models, the output of the threshold network determines which expert model is activated, and the rail foreign body semi-supervised anomaly detection model corresponding to the expert model is used as the detection model of the current rail area segmentation image and completes the rail foreign body detection.

[0140] At the same time, the present invention provides:

[0141] A server comprises a processor and a memory, wherein the memory stores at least one program, and the program is loaded and executed by the processor to implement the semi-supervised anomaly detection method for foreign objects on a rail deployed at the edge.

[0142] A computer-readable storage medium stores at least one program, which is loaded and executed by a processor to implement the above-mentioned semi-supervised anomaly detection method for foreign objects on rails deployed at the edge.

[0143] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0144] 1. The present invention constructs a rail foreign body detection method. By autonomously collecting pictures or manually providing a batch of actual pictures on site, the model training, model quantization and transplantation deployment processes can be completed in one stop. It has a high degree of automation and a short development cycle. Moreover, it can effectively filter interference items in the inference stage and run on edge devices with real-time and stability.

[0145] 2. The present invention applies multiple edge-deployed semi-supervised anomaly detection algorithms to the field of foreign body detection on rails, which can cover the detection of foreign bodies of different sizes and improve the adaptability of the algorithm system to complex road conditions in railway scenes; the algorithms are all semi-supervised or self-supervised training, which improves the detection ability of unknown foreign bodies, and does not require manual labeling, reducing the algorithm development cycle; the edge deployment improves the real-time response of the algorithm and the stability of the system. The algorithm solutions adopted by the existing technical methods (such as GAN, target detection algorithms and traditional foreign body detection methods) are either difficult to train, or have high labeling costs, or have supervised training, and the generalization is not strong, and they cannot deal with unknown foreign bodies.

[0146] 3. The present invention supports scene-adaptive automatic selection of rail foreign body detection algorithms. Using multiple rail anomaly detection algorithms, on the one hand, users can evaluate prior knowledge to make selections, and on the other hand, the present invention also provides a scene-adaptive automatic algorithm selection method, using the matching probability of each algorithm output by the threshold network in MoE. Users only need to provide a batch of image data of the current scene to evaluate the optimal algorithm selection.

[0147] 4. The present invention applies the rail foreign object detection method based on automatic training and deployment of the standardized flow model to the field of rail foreign object detection. It only needs to use positive sample rail images without abnormalities for semi-supervised training, and no manual labeling is required. The algorithm has strong generalization ability, and small-scale field of view changes (rotation or translation) will not affect the detection effect.

[0148] 5. The present invention applies the rail foreign object detection method based on multi-layer feature perception of variational autoencoders to the field of rail foreign object detection, and uses positive sample rail images without abnormalities and synthetic negative sample image data for supervised training. No manual labeling is required, and precise foreign object detection can be achieved.

[0149] 6. The present invention applies the rail foreign object detection method based on deep feature comparison to the field of rail foreign object detection, and uses positive sample rail images without abnormalities and synthetic negative sample image data for supervised training. It does not require manual labeling and can support the migration of multiple different scenes. It has good abnormal object detection capabilities for some common large targets (such as people, animals, etc.). BRIEF DESCRIPTION OF THE DRAWINGS

[0150] Figure 1 The present invention provides a flow chart of the semi-supervised anomaly detection method for foreign objects on rails deployed at the edge.

[0151] Figure 2 Define a schematic diagram for the rail area.

[0152] Figure 3 Schematic diagram of foreign body detection effect.

[0153] Figure 4 Schematic diagram of the overall network structure of the hybrid expert system.

[0154] Figure 5 The flowchart of the rail foreign object detection method based on automatic training and deployment of the standardized flow model.

[0155] Figure 6 This is the workflow diagram of the FastFlow model.

[0156] Figure 7 Schematic diagram of adaptive threshold.

[0157] Figure 8 Schematic diagram of the training process of the autoencoder described in the present invention.

[0158] Fig. 9 Schematic diagram of the training process of the variational autoencoder described in the present invention.

[0159] Fig.10 Schematic diagram of anomaly detection of the rail foreign object detection method based on multi-layer feature perception of variational autoencoder described in the present invention.

[0160] Fig.11 This is an overall flow chart of the rail foreign body detection method based on depth feature comparison described in the present invention.

[0161] Fig.12 This is a flow chart of the training phase of the rail foreign body detection method based on deep feature comparison described in the present invention.

[0162] Fig.13 This is a flow chart of the testing phase of the rail foreign body detection method based on depth feature comparison described in the present invention.

[0163] Fig.14 This is a workflow diagram of the encoding module described in the present invention.

[0164] Fig.15 The following is a flowchart of the abnormal alarm module of the present invention. DETAILED DESCRIPTION

[0165] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0166] like Figure 1 ,The present invention first uses MoE to select three algorithm models, and after determining the expert model, the model is trained and deployed; ,finally, the model is used to perform foreign object detection ,processing flow.

[0167] like Figures 1 to 4 The present invention provides a semi-supervised anomaly detection method for foreign objects on rails deployed at the edge. The specific technical solution implementation process includes the following steps: (1) an overview of the rail foreign object detection algorithm model and implementation process; (2) automatic selection and implementation of scene-adaptive algorithm; (3) construction of the rail foreign object detection algorithm system; (4) edge model training and update.

[0168] 1. Overview of the Rail Foreign Object Detection Algorithm Model and Implementation Process

[0169] 1.1 Overview of Algorithm Model

[0170] The rail foreign body detection method proposed in the present invention includes a semantic segmentation model and three optional semi-supervised anomaly detection models.

[0171] The semantic segmentation model used is STDC. When a picture is input, the model can segment the specified area. The algorithm designs a lightweight feature extraction module stdc for the segmentation task. This module reduces the number of channels by geometric multiples, uses a smaller convolution kernel to effectively reduce the number of parameters, and connects all blocks to increase the receptive field and multi-scale features, which is suitable for porting edge devices.

[0172] The semi-supervised anomaly detection methods used include a rail foreign object detection method based on automatic training and deployment of a standardized flow model, a rail foreign object detection method based on multi-layer feature perception of a variational autoencoder, and a rail foreign object detection method based on deep feature comparison.

[0173] 1.2 Comparison of three anomaly detection algorithms

[0174] The reason why three different anomaly detection algorithms are introduced is that actual tests have found that different algorithms have different focuses, and a single algorithm cannot meet the needs of railway track scenes with rich content and changing perspectives. The characteristics of the three anomaly detection methods are as follows:

[0175] (1) The rail foreign object detection method based on automatic training and deployment of the standardized flow model has strong generalization ability. Small changes in the field of view (rotation or translation) will not affect the detection effect. In addition, the model structure is lightweight and can be trained through the CPU of the edge device. The only shortcoming is that the fine detection capability is insufficient.

[0176] (2) The rail foreign object detection method based on multi-layer feature perception of variational autoencoders focuses on texture information and is insensitive to color changes. It can detect small objects and targets whose colors are close to the background (such as stones and tools). Its disadvantage is that it is sensitive to scene changes and the field of view must remain unchanged.

[0177] (3) The rail foreign body detection method based on deep feature comparison can support the migration of multiple different scenes. It has good abnormal object detection capabilities for some common large targets (such as people, animals, etc.), but it is easy to miss some targets whose colors are close to the background or are too small. The current minimum detectable target size is 50*70

[0178] pixel.

[0179] The above conclusions are all empirical results obtained under the current test data. The specific use should be based on the actual test situation to select the solution.

[0180] 1.3 Algorithm model training and deployment process

[0181] For the semantic segmentation model STDC, it is necessary to manually annotate the rail area of ​​the rail site image. The rail area is defined as the area enveloping two rails and wooden sleepers. Considering the unified construction standards within the rail area, the semantic segmentation model can adapt to the segmentation of rail areas in various scenarios after one round of training. During the transplantation and deployment process, in order to improve the efficiency of the model on the edge, the last two layers of network operators (deconvolution and ArgMax filtering) are separated from the model and executed separately as post-processing on the CPU. Finally, the entire post-processing is uniformly optimized using compilation options in parallel, which effectively reduces the overall time consumption of the edge segmentation model.

[0182] For the rail foreign body detection method based on the automatic training and deployment of the standardized flow model, only the positive sample rail images without abnormalities need to be used for semi-supervised training. Considering that the data actually obtained on site basically have no abnormalities, manual classification and labeling are basically not required. In order to resist lens jitter or small-scale field of view changes (rotation or translation), the present invention adopts some data enhancement methods to improve the diversity of training data. These methods include random rotation, cropping, and changing brightness and contrast. The fastflow model network is relatively lightweight and can be transplanted without other processing. It has a certain efficiency after the edge.

[0183] For the rail foreign body detection method based on multi-layer feature perception of variational autoencoder, in addition to using normal samples for semi-supervised training, the present invention also uses an image synthesis strategy to simulate abnormal samples, and introduces abnormal samples during the training process to complete self-supervised learning, so that the model can learn what is the normal mode. The synthesis purpose is mainly achieved by adding noise to normal samples. The present invention uses 80 types of targets in the coco data set as noise. The image synthesis strategy is mainly divided into three steps: the first step is to randomly use a target in the coco80 class as a noise image, and merge the noise image and its corresponding segmentation mask image to obtain a noise foreground image. The second step is to perform random filtering on the noise foreground image (including random resizing operation of the foreground target size, random cropping operation, random radiation transformation operation, random noise addition, random brightness and contrast changes, and random color space conversion). The third step is to randomly superimpose the noise foreground image on the original image to obtain a simulated abnormal image. To transplant and deploy the rail foreign body detection method based on multi-layer feature perception of variational autoencoder, it is necessary to replace multiple attention heads (attension modules) of post-processing with convolution operators to improve edge efficiency.

[0184] For the rail foreign body detection method based on deep feature comparison, the training mode is consistent with the rail foreign body detection method based on variational autoencoder multi-layer feature perception. During the transplantation and deployment process, the present invention removes the attention head attension in the network and reduces the number of normal sample features in the memory storage module to one, effectively improving the edge operation efficiency.

[0185] 1.4 Rail Foreign Object Detection Implementation Process

[0186] The edge device obtains an image frame sequence by encoding and decoding the video stream captured by the camera, and sets the frame interval threshold f to extract frames from the frame sequence. For a frame image obtained by extraction, the semantic segmentation model STDC is used to perform image segmentation to obtain a railroad area segmentation image. At the same time, one of the three anomaly detection algorithms described in Section 1.1 is used to detect foreign objects in the image, and a probability threshold is set to obtain a foreign object segmentation image. The railroad area segmentation image and the foreign object segmentation image are fused, and foreign objects outside the railroad area are filtered to obtain a foreground foreign object image. Finally, the foreign object foreground in the foreground foreign object image is filtered by area and score threshold to obtain the final foreign object detection result.

[0187] According to the multi-frame foreign body detection results, if a foreign body is detected by continuous n frames of images, an alarm is reported.

[0188] More specifically, for the three anomaly detection algorithms, their respective processing flows are as follows:

[0189] The first one is a railroad foreign object detection method based on automatic training and deployment of a standardized flow model.

[0190] like Figure 5 The rail foreign body detection method based on the automatic training and deployment of the standardized flow model mainly includes four parts: product image preprocessing, early stopping strategy in model automatic training, calculation of segmentation threshold in model automatic deployment and post-processing of model output results.

[0191] The specific technical solution implementation process is as follows:

[0192] (1) Image enhancement pre-processing

[0193] like Figure 6 As shown, the present invention is a method for detecting foreign bodies on rails using fastflow as a standardized flow model. In order to estimate the distribution of normal samples, the present invention uses a standard normal distribution As the posterior estimation of fastflow, the ResNet18 network pre-trained on the ImageNet classification dataset is used as the feature extractor. Specifically, the size of the product image input ResNet18 is 288*320, and the last layer features of the first three network blocks are output as the feature input of the subsequent flow model. These three layers of features They have different downsampling rates (4, 8, 16) and are mapped to probability distributions through 8 radiation transformation modules to obtain features of the same scale. The present invention uses the standardized flow model to output features The probability density value of is used to determine whether there is foreign matter on the rail, thereby realizing the automatic detection of foreign matter on the rail.

[0194] Compared with the traditional rail foreign body detection method, the present invention adopts the characteristics of light weight and convenient edge training and deployment. This makes the present invention very suitable for automated training and deployment on edge devices. In addition, due to the high similarity between the collected rail foreign body-free samples, the present invention also adopts some data enhancement methods to improve the diversity of training data. These methods include random rotation, cropping, brightness and contrast transformation, etc.

[0195] This paper proposes a lightweight rail foreign body detection method based on FastFlow and ResNet18, which is fast, accurate and reliable. It can help manufacturers realize automated rail foreign body detection in the product production process, thereby improving product quality and production efficiency.

[0196] (2) Statistical-based early stopping of training

[0197] In edge computing environments, automating the training process is an important task, but problems that arise during training may cause the model to crash. These problems are often related to factors such as loss function, data distribution, and training strategy. In many cases, model training is affected by the loss function because the optimization objective of the loss function often does not conform to the actual data distribution.

[0198] The present invention has found through research that there is a problem in the automated training process of FastFlow, that is, after a certain period of training, the converted features tends to 0, which means The posterior distribution (Gaussian distribution) is not satisfied, that is, the model training collapses. The inventor creatively discovered that the main reason for the model collapse is the maximum likelihood loss function. The essential requirement of the algorithm is to learn distribution, but the maximum likelihood loss can only ensure that the input features The probability value is the largest under the probability distribution learned by the flow module, but this does not mean that the learned probability distribution is the same as The distribution of is consistent. In order to solve the problem of model collapse, the solution of the present invention is to adopt an early stopping strategy. For a data set with test data and test labels, you can continuously test on the test data during the training process and take the model result with the best test result. However, for edge automatic training, there is no test data, let alone labeled test data.

[0199] The early stopping strategy is a training strategy that can decide when to stop training based on some indicators during the training process. These indicators may include the rate of change of training loss, the rate of change of validation loss, the size of the gradient, and so on. When the test data and test labels are known, the test data can be used to evaluate the performance of the model and select the best model. However, in the process of automated training of edge devices, test data and test labels are usually not available, which makes it more difficult to select the best model.

[0200] To solve this problem, the present invention proposes an early stopping strategy based on the statistical variance of training data. The core idea of ​​this strategy is to use the feature The mean square error of the feature is used to determine whether the model is about to crash. When the mean square error of the feature drops below a certain threshold, the model training is stopped and the final model is used as the result. This method can avoid the problem of model crash and does not require the support of test data and test labels. Therefore, this method is particularly suitable for automated training scenarios on edge devices without the involvement of algorithm personnel.

[0201] In the actual training process, there are many training data, which cannot be loaded into the memory at one time to calculate the mean square error, especially on the edge device side, where computing resources are insufficient. The mean square error is calculated for batches, where Batches contain Frame image, calculate the mean square error of the overall training data The process is as follows:

[0202] ;

[0203] ;

[0204] ;

[0205] in, Before The overall variance of the batch data, Indicates the current The local variance of the batch, Indicates the current The local mean of the batches, Before The overall mean of the batch data; and Respectively represent the jth and The number of images in a batch.

[0206] (3) Statistical-based adaptive threshold

[0207] After training, fastflow obtains the features The present invention calculates the rail foreign body probability map at each pixel position according to the following formula: :

[0208] ;

[0209] in, It means that each downsampled feature map is upsampled to the size of the original image, and the final rail foreign body probability value is the average of the results of the three scales; , , Represents output feature maps of different sizes;

[0210] Get the probability map of foreign matter on the rails After that, post-processing is required to obtain the segmentation results and detection results: by setting the threshold To segment the foreign body probability map of the rail, we get a segmentation map. In the segmentation map, the probability of foreign bodies exceeding the set threshold The area with a foreign object on the rail is the foreign object area on the rail, otherwise it is a normal area; the minimum envelope box of each segmentation image is used as the envelope box result of the foreign object detection on the rail; if there is a foreign object area on the rail, it is determined that there is a foreign object on the rail.

[0211] After obtaining the probability map of foreign objects on the rails, post-processing is required to obtain the segmentation results and detection results. The common way is to set a threshold, but the setting of the threshold is related to the specific data set. If the threshold is too low, it will lead to false detection, and if the threshold is too high, it will lead to missed detection. Therefore, a suitable threshold has a great impact on the final result of the algorithm. In the process of automated deployment at the edge, there is no participation of algorithm personnel, which requires the algorithm to automatically select a threshold suitable for the data set.

[0212] To address this problem, the present invention proposes a method for adaptively setting thresholds based on the statistical results of training data. Specifically, the method is based on the trained feature , these features follow the characteristics of the standard normal distribution in the posterior distribution (such as Figure 7 The characteristic of the standard normal distribution is that most of the features are clustered around the mean of 0 and are considered normal samples, while only a few features deviate from the mean and are considered foreign objects on the rails.

[0213] Therefore, the present invention proposes a formula to adaptively set the threshold value, and the specific formula is as follows:

[0214] ;

[0215] Among them, setting the threshold is a probability value of foreign matter on the rail, indicating the dividing line between normal features and foreign matter features on the rail. Features for training data The mean point of Features The mean square error, is the feature point used to calculate the segmentation threshold; and equivalence, The function is used to calculate the probability density of *.

[0216] (4) Double threshold post-processing

[0217] According to the actual test results, it is found that when two foreign objects on the rails are close to each other, the probability value of the product part between the foreign objects on the rails will be higher, which will lead to a larger segmentation area. In order to solve this problem, the present invention proposes a new method, which uses a double threshold method for post-processing to obtain a more refined segmentation result.

[0218] Specifically, this method first converts the original RGB image into a grayscale image and uses OpenCV operators to obtain edge images. When processing edges, a low threshold is used. , while in the non-edge part, a high threshold is used To obtain more accurate results. This is mainly based on the fact that the location of foreign objects on the rails is often irregular in shape, which easily produces dense edge lines. The surface of the product is relatively regular, with sparse edge lines. The probability value of foreign objects on the rails at the center of the foreign objects is higher than the threshold, while the probability value of foreign objects on the products between two adjacent foreign objects on the rails is often near the threshold. Using a high threshold will effectively suppress these product areas near the threshold. This dual threshold method can effectively handle the mutual influence between foreign objects on the rails and avoid misjudgment caused by foreign objects on adjacent rails.

[0219] In addition to the segmentation map, we also need to obtain the envelope box position of the foreign object on the rail. We use the minimum envelope box of each segmentation map as the envelope box result of foreign object detection on the rail.

[0220] The second method is to detect foreign objects on railroad tracks based on multi-layer feature perception of variational autoencoders.

[0221] The rail foreign body detection method based on multi-layer feature perception of variational autoencoder can be defined as a binary classification image semantic segmentation method to segment abnormal areas from the input image. This method consists of two models: autoencoder and variational autoencoder. The implementation process includes autoencoder distillation training, pseudo-abnormal image data synthesis, variational autoencoder distillation training, and multi-feature perception image anomaly detection. In order to better demonstrate the effectiveness of this method, this example is explained in conjunction with the rail foreign body detection task. The specific technical solution is as follows:

[0222] 1. Autoencoder distillation training

[0223] In order to make the autoencoder have a lightweight network structure and good reconstruction ability, distillation training is used for self-supervised training, and a pre-trained model with a large number of parameters is used as the teacher model. The teacher model can be a deep convolutional neural network such as ResNet50, VGG19 and HRNet. The teacher model parameters are frozen during the training phase and no parameter update is performed. At the same time, in order to make the features extracted by the decoding module at different network layers have multiple levels of semantic information, that is, the features extracted by the shallow network describe the texture information of the image, and the features extracted by the deep network express the semantic information of the image, multi-scale feature distillation training is performed during the distillation training phase. The autoencoder training framework is as follows: Figure 8 As shown, it is divided into the following steps:

[0224] 1) Prepare large-scale open-source image data for self-supervised distillation training. This example uses ImageNet, which has rich categories and diverse backgrounds, as training data;

[0225] 2) Build an autoencoder model. The encoding module consists of 4 downsampling network layers, and the decoding module consists of 4 upsampling network layers. Each downsampling network layer reduces the width and height of the input image features by half, and each upsampling network layer doubles the width and height of the input image features.

[0226] 3) Image preprocessing is performed during the training phase. First, the input image is scaled to 256×256, and then image horizontal flip enhancement, HSV color gamut jitter enhancement, and small-angle image rotation enhancement are performed at a certain ratio;

[0227] 4) After the image is preprocessed, it is input into the autoencoder and teacher model respectively to obtain the corresponding reconstructed image;

[0228] 5) Calculate the mean square error (MSE) between the image features of the autoencoder decoding module with sizes of 32×32, 64×64, and 128×128 and the image features of the teacher model with the corresponding scales, and calculate the mean square error between the reconstructed image and the input image. The loss function calculation formula for the autoencoder model training is as follows:

[0229] ;

[0230] in, Represents the autoencoder decoding module Image features of size, Represents the teacher model Image features of size, express Image features under size and The mean square error between represents the image reconstructed by the autoencoder, represents the input image, Representing images and The mean square error.

[0231] 6) Based on the above loss function, back propagation is performed to optimize the autoencoder model parameters to obtain a pre-trained model.

[0232] Since the autoencoder is used in the present invention to reconstruct multiple levels of semantic features and input images, and is independent of the image data for anomaly detection, the pre-trained autoencoder can be repeatedly applied.

[0233] 2. Pseudo-abnormal image data synthesis

[0234] In practical applications, it is rare to obtain a sufficient number of real abnormal images for supervised training, and synthetic techniques are often used to generate pseudo abnormal images to expand training data. However, it is also necessary to be careful to ensure that the generated pseudo abnormal images are consistent with the actual scene. The task of foreign body detection on rails is no exception. It is difficult to obtain abnormal image samples in real scenes. To this end, the present invention uses an image superposition method to synthesize rail abnormal image samples, which is divided into the following steps:

[0235] 1) Prepare the COCO instance segmentation dataset, use the object instances in the data as pseudo foreign objects on the rails, and extract the mask map of each instance; the COCO dataset includes common objects, animals, and pedestrians in life, which are consistent with the foreign objects that may appear on the rails;

[0236] 2) Obtain normal image data of the rails and filter out special image samples of trains passing by;

[0237] 3) Based on the instance mask map, the instance image and the rail area are superimposed and synthesized at a certain ratio. The synthesis formula is as follows:

[0238] ;

[0239] In the formula, represents a COCO image, represents a normal image sample, represents the synthetic pseudo-abnormal image, Indicates that an object instance is in the image The mask map of It is a mask image The negation result of is the stacking coefficient of image synthesis.

[0240] 3. Variational Autoencoder Distillation Training

[0241] The variational autoencoder is used to generate normal images and is also trained using distillation. The autoencoder trained in the first stage is used as the teacher model to perform multi-scale feature supervision training on the decoding module, so that the decoding module of the variational autoencoder is aligned with the decoding module of the autoencoder in feature spaces of multiple scales and different semantic information is reconstructed. The variational autoencoder distillation training framework is as follows: Fig. 9 As shown, it is divided into the following steps:

[0242] 1) Prepare normal rail images and synthetic pseudo-abnormal images, and divide them into training sets and test sets;

[0243] 2) Build a variational autoencoder model, which has a similar structure to the autoencoder. On the basis of the autoencoder, a downsampling network layer and an upsampling network layer are added to encode the input image into a vector distribution in the latent space, and then reconstruct it to generate a new image;

[0244] 3) In the image preprocessing stage, the input image is enhanced by HSV color domain dithering and small-angle horizontal rotation;

[0245] 4) After the image is preprocessed, the normal image and the pseudo-abnormal image are input into the variational autoencoder to generate normal image samples, and the normal image is input into the autoencoder to obtain the reconstructed image;

[0246] 5) Calculate the mean square error between the image features of the variational autoencoder decoding module with sizes of 32×32, 64×64, and 128×128 and the image features of the corresponding scales of the autoencoder, and calculate the mean square error between the image generated by the variational autoencoder and the normal image to obtain the feature reconstruction and image generation loss function:

[0247] ;

[0248] In the formula, Represents the variational autoencoder decoding module Image features of size, Represents the autoencoder decoding module Image features of size, express Image features under size and The mean square error between represents the normal images input into the variational autoencoder model and the autoencoder model respectively, represents a normal image generated by the variational autoencoder model, Representing images and The mean square error between .

[0249] In order to make the variational autoencoder model have the ability to generate images, the posterior distribution of the latent space is constrained to obey the standard normal distribution, and the KL divergence between the normal distribution of the input data and the standard normal distribution is increased. The loss is defined as follows:

[0250] ;

[0251] In the formula, is the dimension of the latent space vector, and They represent the mean and variance of the nth dimension of the latent space vector, and Represent the mean and variance of the latent space vector respectively. To this end, the loss function of the variational autoencoder is The definition is as follows:

[0252] ;

[0253] In the formula, represents the feature reconstruction and image generation loss function of the input normal image, represents the feature reconstruction and image generation loss function of the input pseudo-anomaly image, and The calculation formula and Consistent, , and Represent the weight coefficients of different losses respectively; is the KL divergence loss between the normal distribution of the input data and the standard normal distribution.

[0254] 6) Based on the above loss function, optimize the model parameters so that the variational autoencoder has the ability to generate normal images when both abnormal images and normal images are input.

[0255] 4. Multi-feature-aware image anomaly detection

[0256] After obtaining the pre-trained autoencoder and variational autoencoder, multi-feature-aware image anomaly detection is performed, which is implemented as follows: Fig.10 As shown, it is divided into the following steps:

[0257] 1) Statistical autoencoder and variational autoencoder on the test set of different scale image features mean and variance;

[0258] 2) Input the preprocessed image to be tested, the autoencoder outputs image features and reconstructed images of sizes 32×32, 64×64, and 128×128, and the variational autoencoder outputs image features and generated images of sizes 32×32, 64×64, and 128×128;

[0259] 3) Use the mean and variance of the statistics in step 1) to normalize the image features output by the autoencoder and variational autoencoder;

[0260] 4) Calculate the mean square error between the three-size image features of the autoencoder and the variational autoencoder, called the feature reconstruction error, accumulate the feature reconstruction errors according to the feature channel dimension, and perform bilinear interpolation upsampling operations to obtain the reconstructed feature error with the same width and height as the input image;

[0261] 5) Calculate the mean square error between the image reconstructed by the autoencoder and the image generated by the variational autoencoder, called the image reconstruction error, and accumulate the reconstructed image errors by channel dimension to obtain an image reconstruction error of the same dimension as the feature reconstruction error;

[0262] 6) Sum the feature reconstruction error and image reconstruction error of the three sizes, perform min-max normalization, and map the error value to On the interval;

[0263] 7) Set the error threshold , the above error value is greater than The area is judged as the abnormal image area, otherwise it is the normal image area.

[0264] The third method: rail foreign body detection method based on deep feature comparison.

[0265] The rail foreign body detection method based on deep feature comparison mainly uses the difference between the multi-level features of normal samples and abnormal samples in the same scene to locate foreign bodies. The overall process of the rail foreign body detection method based on deep feature comparison is as follows: Fig.11 As shown, the corresponding rail anomaly detection system mainly includes a sample input module, a data preprocessing module, an encoding module, a decoding module and an anomaly alarm module. Due to the diversity and complexity of rail scenes, it is difficult to extract relatively common normal sample features to compare with abnormal sample features. Therefore, the present invention uses a single normal sample in the scene to be tested to compare with the abnormal sample, and updates the normal sample through a certain strategy to ensure the effectiveness of its detection.

[0266] The overall process is divided into two stages: training and testing. For details, refer to Fig.12 and Fig.13 In the training phase, abnormal samples are mainly simulated by two abnormal synthesis strategies in different scenarios. In the test phase, the abnormal samples are simply replaced with the samples to be tested. The encoding module and the decoding module are common parts. The parameters of the module network will be updated in the training phase, and the parameters will be fixed in the test phase for network reasoning. Finally, in the test phase, there will be an additional abnormal alarm module to update normal samples and make logical judgments on abnormal alarms.

[0267] In the training phase, a normal sample is first input, and then divided into four equal parts with overlapping areas. The blocks are passed through the rail foreign body generation module to generate abnormal samples in the corresponding scene. Then, a training sample is formed with the normal sample to perform data preprocessing operations such as scaling, color mode conversion, standardization and normalization. Finally, the encoder and decoder are used to extract, fuse and decode the features of the preprocessed data.

[0268] Unlike the training phase, the test phase does not require the participation of the foreign body generation module, and a foreign body alarm module is added. First, the detection scene is fixed, and the image of the next frame without foreign bodies in the scene is obtained as a normal sample, and the sample to be tested is a real-time lens scene image. Then, the normal sample and the sample to be tested are used as a test sample for the same data preprocessing operations such as scaling, color mode conversion, standardization and normalization. The trained decoder and encoder are used to output the corresponding prediction results. Finally, the abnormal alarm module performs logical judgment on the decoded results, and abnormal situation alarms are performed in real time. The normal samples in the scene are continuously updated, which can reduce misjudgment to a certain extent.

[0269] The specific operation steps of each module in the two stages are as follows:

[0270] Step 1: Obtaining input samples

[0271] The method proposed in the present invention locates foreign objects based on the difference between the features of normal samples and abnormal samples in the same scene, so the input of the model requires a normal sample and an abnormal sample. Since there are certain differences in the definition and acquisition of abnormal samples in the training phase and the test phase, the present invention hereinafter refers to the abnormal samples in the test phase as test samples.

[0272] During the training phase, normal samples are mainly obtained through fixed lenses in different scenes with a resolution of Considering that some foreign objects are relatively small in size, in order to better improve the recall rate of small foreign object detection, the present invention adopts a block strategy on normal samples, that is, the original normal sample is divided into four equal parts with overlapping areas, and the overlap rate is 20%. The four image blocks of equal size are the input normal samples. The acquisition of abnormal samples is mainly generated by simulating normal samples through the rail foreign object generation module proposed by the present invention.

[0273] In the test phase, due to the relative fixedness of the detection scene, the present invention selects a frame of image without foreign objects under the fixed lens of the scene to be detected as a normal sample. It should be noted that, unlike the training phase, the normal sample input in the test phase is of original size and is not processed in blocks. The model must provide a frame of normal sample image in the scene for the first run. The test sample is a real-time video image frame in the corresponding scene.

[0274] Step 2: Rail Foreign Object Generation

[0275] The present invention completes self-supervised learning by introducing abnormal samples in the training stage, and proposes a method for simulating the generation of unknown foreign objects on rails that appear randomly. The specific operation is as follows:

[0276] (1) Specifying the area where foreign objects are generated. Given a normal sample, i.e., a rail image without foreign objects, artificially specify the area of ​​interest where foreign objects are generated on the normal sample or obtain the specified rail area through the rail contour extraction algorithm to obtain the area image where foreign objects are generated. ;

[0277] (2) Generation of foreign objects. Due to the diversity of rail scenes and the uncertainty of foreign objects, and the difficulty in collecting training data for some scenes with foreign objects on the rails, previous work usually generates simulated abnormal rail scene samples from normal rail scene samples. Common generation methods include image generation networks, copy and paste, and image fusion, but these methods have poor migration capabilities on complex rail scenes and general anti-interference capabilities. For example, in the same scene, the performance difference between the algorithms during the day and at night may be large. Based on this, the present invention proposes two foreign object simulation strategies to generate rail foreign objects.

[0278] Method 1: Simulate unknown rail foreign objects of different sizes and colors that may appear in a single scene

[0279] Considering the diversity and uncertainty of foreign objects on the rails, the present invention mainly uses the segmentation foreground image of common objects to simulate common foreign objects on the rails. The specific operation is as follows:

[0280] a. Use the public segmentation task dataset to obtain foreground images of objects of different categories;

[0281] b. Scaling, rotating, and color space transformation are performed on the foreground image at different resolutions with a certain probability to simulate foreign objects of different sizes and colors.

[0282] Method 2: Using a few rail scene images to simulate rail foreign bodies in most scenarios

[0283] Method 1 only generates abnormal samples in a single scene or a small number of scenes, which has certain limitations. This method can only identify different foreign objects in a specific single scene or a small number of scenes. In order to achieve the same detection effect on more untrained and difficult to collect data rail scenes, the present invention proposes method 2 combined with method 1 to adapt to more untrained complex rail scenes. The specific operation of the method is as follows:

[0284] a. Image of the foreign body generation area obtained in step (1) The brightness, saturation and hue of the image are randomly adjusted, and then the adjusted image is divided into grids of equal proportions, and then the grids are rearranged to obtain a noise image simulating structural abnormalities. ;

[0285] b. Using the size of the region of interest or the size of the rail area obtained in step (1), randomly generate a two-dimensional Perlin noise of the same size , and set the threshold to perform binarization processing to obtain ;

[0286] c. The noise foreground image with structural abnormality is obtained by the following formula , simulating possible foreign objects in different rail scenarios, including The transparency factor introduced is to better balance the image of the foreign body generation area and noisy image The fusion of Closer to the real anomaly.

[0287] ;

[0288] (3) The foreign body images generated by the two strategies in step (2) are , obtained by inverting M After the foreign body generated area image Multiply element by element and then add it to the foreign body image , generate the final abnormal area image , refer to the following formula for specific operations:

[0289] ;

[0290] (4) Finally, the abnormal area image Replace the foreign matter generation area image of the normal sample in step (1) , and spliced ​​to the region of interest or rail area corresponding to the normal sample to obtain the final abnormal sample.

[0291] (5) Select method 1 or method 2 with a certain probability to simulate the generation of foreign matter on the rails.

[0292] Step 3: Data Preprocessing

[0293] (1) Read the two input samples required by the model. In order to reduce the computational complexity of model training and testing and improve the detection speed, the input data of the present invention is scaled proportionally in resolution and filled to a resolution of Size. Specifically, let the original sample width be , Gao Wei , the target width is scaled to , Gao Wei .calculate and The minimum value of the two is taken as the scaling ratio . use Calculate the dimensions and size to be filled. If , then the dimension to be filled is high, and the filling size is Otherwise, the dimension to be filled is width, and the filling size is ;

[0294] (2) Convert the color mode of the scaled input sample to RGB format, then normalize the image data and convert it into a tensor format supported by the model training code framework. The present invention uses the Pytorch framework tensor;

[0295] (3) The tensor data in step (2) is standardized, and the mean of each channel is (0.485, 0.456, 0.406) and the variance is (0.229, 0.224, 0.225).

[0296] Step 4: Encoding module training

[0297] The encoding module consists of three networks: feature extraction network, feature multi-scale fusion combined with spatial attention mechanism. In order to better extract the features of normal samples and abnormal samples for comparison, the present invention uses two feature extraction networks with the same weights to form a twin network. The feature extraction network uses a convolutional neural network trained on the public dataset ImageNet, and the present invention uses ResNet18. The overall encoding module data flow is as follows Fig.14 As shown, the specific training process is:

[0298] (1) Freeze the parameters of the pre-trained feature extraction network and input the pre-processed normal samples and abnormal samples. In order to retain more detailed information and ensure that the extracted information is not too abstract and biased towards ImageNet data, the present invention selects three different levels of features in the middle of ResNet18 as output, which are: , and , where the first dimension The first dimension represents the number of samples input to the network in one iteration of training. The second dimension represents the number of channels. The third and fourth dimensions represent the height and width of the feature map. Then perform the following operations:

[0299] a. Input normal samples and abnormal samples into ResNet18 respectively, and obtain three different levels of feature maps of the corresponding samples. , and is the feature map of normal samples, , and It is the feature map of abnormal samples.

[0300] b. According to the following formula, the feature maps of three different levels corresponding to normal samples and abnormal samples are spliced ​​according to the channel dimension:

[0301] ;

[0302] Among them, i=1, 2, 3, corresponding to the three different levels in the middle of ResNet18, It represents the tensor channel concatenation operation, and the final result is recorded as , and .

[0303] c. Calculate the differences between the three different levels of feature maps corresponding to normal samples and abnormal samples respectively. The operation is as follows:

[0304] ;

[0305] Among them, i=1, 2, 3, corresponding to the three different levels in the middle of ResNet18, In the present invention, it represents the mean square error of each position of the two input tensors, and the result is recorded as , and .

[0306] (2) In order to avoid feature redundancy and low efficiency caused by subsequent multi-layer feature calculation, the present invention concatenates the features f obtained by channel in step (1) 1_concat 、f 2_concat and f 3_concat Send it to the multi-scale fusion network for feature fusion. Specific steps:

[0307] a. First, use a method that reduces the number of channels by half. Convolution and a constant number of channels Convolution on f 1_concat 、f 2_concat and f 3_concat Extract features and then use a Convolution increases the nonlinearity of the network, and we get , and .

[0308] b. Remember f 3_msff = f 3_conv , the feature map f 3_msff Upsampled twice After the size is connected, the number of channels is halved Convolution operation obtains feature map , then f _temp With f 2_conv Add element by element to get .

[0309] c. Similarly for f 2_msff Double upsampling and halving the number of channels After the convolution operation, 1_conv Add element by element to get .

[0310] d. Finally, output the multi-scale fused features , and .

[0311] (3) Subsequently, in order to further improve the algorithm's attention to the difference information of abnormal regions, the present invention proposes a spatial attention mechanism network.

[0312] a. First, the difference between the three-level features of the normal sample and the abnormal sample obtained in step (1) is 1_diff 、f 2_diff and f 3_diff Channel average pooling is performed separately to output feature maps of three sizes, which are recorded as , and .

[0313] b. Then remember f 3_SA =f 3_AvgPool , upsample m3 twice to Size and f 2_AvgPool Multiply element by element to get .

[0314] c. Similarly, f 2_SA Upsampled twice Size and f 1_AvgPool Multiply element by element to get .

[0315] d. 1_SA 、f 2_SA and f 3_SA As the output of the spatial attention mechanism network.

[0316] (4) Finally, the multi-scale fusion feature f 1_msff 、f 2_msff and f 3_msff After spatial attention calculation in step (3), we get f 1_SA 、f2_SA and f 3_SA Perform element-by-element multiplication respectively. Refer to the following formula for the operation:

[0317] ;

[0318] where i=1, 2, 3, Represents the multiplication of the corresponding position elements of the two input tensors, and the result is recorded as , and And output.

[0319] Step 5: Decoding module training

[0320] The decoding module is mainly composed of multiple upsampling convolution operations. The convolution operation is repeated, starting from the deep features (the smaller the feature map size, the deeper the level), and finally using a The convolution operation reduces the channel dimension to 2 dimensions. The specific operation is as follows:

[0321] (1) The output f of the encoding module 1_encode 、f 2_encode and f 3_encode As the input of the decoding module, let f 3_decode =f 3_encode ,right Upsample twice to Size followed by one The convolution operation reduces the channel dimension to obtain the feature map Finally, f _temp1 With f 2_encode Concatenate in the channel dimension ;

[0322] (2) Similarly, for f 2_decode Connect one after double upsampling Convolution operation obtains feature map , and then the feature map f _tepm2 With f 1_encode Concatenate in the channel dimension ;

[0323] (3) In order to further reduce the amount of calculation of the model, we continue to 1_decode After upsampling twice, connect a The convolution operation reduces the channel dimension to obtain the feature map .

[0324] (4) f 0_decoder Continue to upsample twice and connect a channel dimension unchanged The convolution operation obtains the same size as the input sample Feature map .

[0325] (5) Finally, use a The convolution operation is performed on f _up2mask Extract features and reduce the channel dimension to 2 dimensions to obtain , where the first dimension of the channel dimension is the foreground output and the second dimension is the background output.

[0326] Step 6: Abnormal alarm

[0327] This module is only executed in the testing phase, and mainly performs certain logical judgments and operations on the output of the trained decoding module. Fig.15 , the specific operations are as follows:

[0328] (1) The first dimension foreground channel of the feature map f_mask finally output by the decoding module is recorded as C1, and the second dimension background channel is recorded as C2. In order to enhance the output of the target foreground and weaken the influence of the background, the present invention sets the threshold value s to filter out some interfering pixels. The specific operation is as follows:

[0329] a. The pixels in C1 with values ​​greater than s are set to 1, and the pixels in C2 with values ​​less than s are set to 0;

[0330] b. Compare the pixel values ​​of C1 and C2 pixel by pixel. If C1 is greater than C2, it is set to 1, otherwise it is set to 0. The comparison result is saved as the mask of the final abnormal area. .

[0331] (2) In the actual scene test, if there are abnormalities in more than 20 consecutive input video frames, an abnormality alarm is triggered. If there are no abnormalities in more than 50 consecutive frames, the last frame without abnormalities is used to update and replace the input normal sample.

[0332] 2. Automatic Selection and Implementation of Scene Adaptive Algorithms

[0333] The present invention provides three anomaly detection algorithms, and describes the characteristics of each algorithm according to Section 1.2, and also provides selection experience. Considering the different knowledge structures of users, the present invention also provides a scenario-adaptive algorithm automatic selection scheme, the main principle of which adopts the Mixture-of-Expert System

[0334] The threshold network in Experts (MoE) is evaluated.

[0335] The overall network structure diagram of the hybrid expert system is shown in the attached figure. Figure 4As shown in the figure, the system consists of multiple expert models, responsible for different task scenarios; a threshold network (Gating Network) based on a probabilistic model is also designed, which combines the output of the threshold network to decide which expert to select for the current input scene image. The network output formula is as follows:

[0336] ;

[0337] in, For input image, To output the foreign matter probability density map; For the threshold network The probability output of the expert model is is the i-th expert model, when , the corresponding expert model will not be activated.

[0338] According to the idea of ​​MoE, the output weight value of the threshold network is used as the algorithm selection index in the present invention. The corresponding experts are respectively a rail foreign object detection method based on automatic training and deployment of a standardized flow model, a rail foreign object detection method based on multi-layer feature perception of a variational autoencoder, and a rail foreign object detection method based on deep feature comparison.

[0339] Assuming that the above experts have been trained independently, it is now necessary to train a threshold network for the current scene data. The training loss function is designed as:

[0340] ;

[0341] Among them, c represents an image sample, For the The output of the expert model; is the ideal output, that is, the true value. The network learns so that The target value will be gradually approached; is the threshold network assigned to the The weight of an expert model is equivalent to ; By reducing the loss function , train the threshold network parameter weights.

[0342] Combining the above principles, the automatic selection scheme of the scene adaptive algorithm is implemented as follows:

[0343] In the first step, we use the actual scene positive sample data provided by the user, combined with the 80 types of targets in the coco data as foreground noise, and use the image synthesis strategy mentioned in Section 1.3 to synthesize the automatic selection scheme training data set. There are two differences: (1) The size of the noise target is divided into three categories of large, medium and small, and the number of each category is controlled to maintain a balance. (2) STDC is used to segment the positive sample data into railroad areas, limiting the synthesis area of ​​the noise target on the positive sample to only within the railroad area. The synthesis process records the noise segmentation area information and obtains the true value label T0 of the training data set;

[0344] The second step is to train three different anomaly detection algorithms to obtain the expert model;

[0345] The third step is to freeze the three expert models and train the threshold network;

[0346] The fourth step is to pass multiple pictures of the current task scene to the threshold network, record the value of the weight assigned to each expert each time, and the expert with a higher weight and a higher frequency of occurrence is the final solution.

[0347] A single image as input will produce an expert model result with a high weight value, but the judgment of a single image is not stable, and multiple images of the task scene need to be continuously input, and the frequency of the highest weight corresponding to each expert model is counted. The expert model with the highest frequency is the final solution. That is: for this frame of image, the previously obtained rail area segmentation image is input into the hybrid expert system, and the threshold network of the hybrid expert system assigns weights to several expert models, and the expert model with the highest weight assigned to the current frame is recorded, and the corresponding frequency is increased by 1; similarly, through the statistics of multiple consecutive frames of images, the statistical frequency of each expert model is obtained, and the semi-supervised anomaly detection model for foreign objects on the rail corresponding to the expert model with the highest frequency or the semi-supervised anomaly detection model for foreign objects on the rail selected by the user based on prior knowledge is used as the detection model for the current rail area segmentation image to complete the foreign object detection on the rail.

[0348] 3. Construction of Rail Foreign Object Detection Algorithm System

[0349] Combining the above three algorithm models and automatic algorithm selection scheme, a rail foreign body detection algorithm system is constructed. The system has the following characteristics:

[0350] (1) The input data of the system is mainly the positive sample image data of the scene, which can be collected autonomously or manually;

[0351] (2) The system uses a semi-supervised training method. The positive sample data does not need to be labeled. At the same time, the negative sample data is obtained by synthesis. Finally, the positive and negative sample data are used to implement supervised training (the fastflow solution only uses positive sample data, and the other two use both positive and negative samples);

[0352] (3) After the three scheme models are trained, users can import pictures to evaluate the results and decide which scheme to adopt. At the same time, they can also automatically select algorithms and use the trained MOE system for evaluation.

[0353] (4) The system integrates the migration tool chain of the edge device platform, which can automatically deploy the model quantization migration after the model training is completed.

[0354] (5) During the system operation reasoning phase, it can automatically filter out foreign objects that interfere with detection outside the rail area and only retain foreign objects within the rail area. Moreover, it runs on edge devices and has real-time and stability.

[0355] In general, the algorithm system training process of the present invention has basically no human participation, the data level is semi-supervised training, no manual labeling is required, the training process uses default parameters, and only needs to provide a batch of actual field data. The algorithm system of the present invention can complete the entire process from model training, model quantization and transplantation deployment in one stop; at the same time, it can effectively filter out foreign interference items in the reasoning stage.

[0356] 4. Edge Model Training Update

[0357] The training and transplantation process described in Section 1.3 is carried out in the server, and the model is finally deployed to the edge. This is also the current mainstream method. In addition, if the algorithm has many false positives during actual use and needs to be updated, the training and transplantation process must be repeated on the server. In order to further improve the efficiency of algorithm development and deployment, the present invention also transplants the model training process to the edge device. The training executable file of the rail foreign body detection method that is automatically trained and deployed based on the standardized flow model is executed on the edge, which triggers the model update function. First, a batch of training images of the current scene are collected, and then the model is trained using the edge CPU during the evening period, and finally the original model is updated and replaced.

[0358] The rail anomaly detection algorithm used in the present invention is optional. Combined with the characteristics of each algorithm, it can meet the needs of most rail foreign object detection scenarios. The provided scene-adaptive foreign object detection algorithm automatic selection method better combines the algorithm and the scene; and the algorithm itself is semi-supervised, which can adapt to unknown foreign object detection and greatly improve the algorithm development efficiency; and the deployment at the edge end improves the algorithm's real-time response and system stability.

[0359] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A semi-supervised anomaly detection method for foreign objects on rails deployed at the edge. It is characterized in that The following steps are involved: The edge device encodes and decodes the video stream captured by the camera to obtain an image frame sequence, and sets a frame interval threshold to extract frames from the image frame sequence; For a certain frame image obtained by frame extraction, the semantic segmentation model is used to perform image segmentation to obtain a segmented image of the rail area; For this frame of image, the previously obtained rail area segmentation image is input into the hybrid expert system. The threshold network of the hybrid expert system assigns weights to several expert models, records the expert model with the highest weight assigned to the current frame, and adds 1 to the corresponding frequency. Similarly, the statistical frequency of each expert model is obtained through statistics of multiple consecutive frames of images. The semi-supervised anomaly detection model for foreign objects on the rail corresponding to the expert model with the highest frequency or the semi-supervised anomaly detection model for foreign objects on the rail selected by the user based on prior knowledge is used as the detection model for the current rail area segmentation image to complete the foreign object detection on the rail. The semi-supervised rail foreign body detection model corresponding to the expert model includes a rail foreign body detection method based on automatic training and deployment of a standardized flow model, which divides the overall training data of the standardized flow model into The mean square error is calculated for each batch, and the mean square error of the output features of the overall training data after passing through the standardized flow model is calculated. The process is as follows: ; ; ; in, Before The overall variance of the batch data, Indicates the current The local variance of the batch, Indicates the current The local mean of the batches, Before The overall mean of the batch data; and Respectively represent and The number of images in a batch.

2. According to the edge-deployed semi-supervised anomaly detection method for rail foreign bodies as described in claim 1, It is characterized in that The output formula of the hybrid expert system is as follows: ; in, For input image, To output the foreign matter probability density map; For the threshold network The probability output of the expert model is is the i-th expert model, when , the corresponding expert model will not be activated.

3. According to the edge-deployed semi-supervised anomaly detection method for rail foreign bodies as described in claim 1, It is characterized in that The loss function of the threshold network is as follows: ; Among them, c represents an image sample, For the The output of the expert model; is the ideal output, that is, the true value. The network learns so that The target value will be gradually approached; is the threshold network assigned to the The weight of an expert model is equivalent to , For the threshold network The probability output of the expert model; by reducing the loss function , train the threshold network parameter weights.

4. According to the edge-deployed semi-supervised anomaly detection method for rail foreign bodies as described in claim 1, It is characterized in that The training process of the standardized flow model adopts an early stopping strategy based on the statistical variance of the training data: when the mean square error of the output features of the standardized flow model is not greater than a preset threshold, it is judged that the detection model is about to crash, and the training of the detection model is stopped; The detection model with the time point closest to the time before stopping training is taken as the final result.

5. According to the edge-deployed semi-supervised anomaly detection method for rail foreign bodies as described in claim 1, It is characterized in that The rail foreign body detection method based on automatic training and deployment of the standardized flow model outputs features through the standardized flow model The probability density value of is used to determine whether a foreign object has appeared on the rail. If it is greater than a preset threshold, it is considered that a foreign object has appeared on the rail.

6. According to the edge-deployed semi-supervised anomaly detection method for rail foreign bodies as described in claim 1, It is characterized in that Output features through the standardized flow model Calculate the probability map of foreign objects on the rails at all pixel locations The calculation method is as follows: ; in, It means that each downsampled feature map is upsampled to the size of the original image, and the final rail foreign body probability value is the average of the results of the three scales; , , Represents output feature maps of different sizes; Get the probability map of foreign matter on the rails After that, post-processing is required to obtain the segmentation results and detection results: by setting the threshold To segment the foreign body probability map of the rail, we get a segmentation map. In the segmentation map, the probability of foreign bodies exceeding the set threshold The area with a foreign object on the rail is the foreign object area on the rail, otherwise it is a normal area; the minimum envelope box of each segmentation image is used as the envelope box result of the foreign object detection on the rail; if there is a foreign object area on the rail, it is determined that there is a foreign object on the rail.

7. According to the edge-deployed semi-supervised anomaly detection method for rail foreign bodies as claimed in claim 6, It is characterized in that The set threshold To adaptively set the threshold, it is calculated as follows: ; Among them, setting the threshold is a probability value of foreign matter on the rail, indicating the dividing line between normal features and foreign matter features on the rail. Features for training data The mean point of Features The mean square error, is the feature point used to calculate the segmentation threshold; and equivalence, The function is to calculate the probability density of *.

8. According to the edge-deployed semi-supervised anomaly detection method for foreign objects on rails as claimed in claim 6, It is characterized in that The set threshold Used to process the edge area of ​​the product image and use a high threshold in the non-edge area of ​​the product to be processed.

9. According to the edge-deployed semi-supervised anomaly detection method for rail foreign bodies as claimed in claim 1, It is characterized in that The standardized flow model adopts fastflow.

10. According to claim 1, the edge-deployed semi-supervised anomaly detection method for foreign objects on rails, It is characterized in that Its rail foreign object detection model uses fastflow as the standardized flow model, standard normal distribution as the posterior estimate of fastflow, and uses the ResNet18 network pre-trained on the ImageNet classification dataset as the feature extractor.

11. According to claim 1, the edge-deployed semi-supervised anomaly detection method for foreign objects on rails, It is characterized in that The semi-supervised rail foreign body detection model corresponding to the expert model includes a rail foreign body detection method based on multi-layer feature perception of variational autoencoder, including the following steps: S1. Use distillation training to perform self-supervisory training on the autoencoder. Use a deep convolutional neural network as the teacher model of the autoencoder. Perform distillation training of multi-scale features in the distillation training stage. The training process of the autoencoder is as follows: Prepare large-scale open-source image data for self-supervised distillation training; Build an autoencoder model: The encoding module of the autoencoder consists of 4 downsampling network layers, and the decoding module of the autoencoder consists of 4 upsampling network layers. Each downsampling network layer reduces the width and height of the input image features by half, and each upsampling network layer doubles the width and height of the input image features. Image preprocessing is performed during the training phase. First, the input image is scaled to 256×256, and then image horizontal flip enhancement, HSV color gamut dithering enhancement, and small-angle image rotation enhancement are performed according to a preset ratio. After the image is preprocessed, it is input into the autoencoder and the teacher model of the autoencoder to obtain the corresponding reconstructed image; Calculate the mean square error between the image features of 32×32, 64×64, and 128×128 sizes output by the decoding module of the autoencoder and the image features of the corresponding scales of the teacher model of the autoencoder, and calculate the mean square error between the reconstructed image and the input image for autoencoder model training; The loss function of the autoencoder model training is back-propagated to optimize the autoencoder model parameters to obtain a pre-trained autoencoder; S2, synthesizing rail abnormality image samples by using image superposition method; S3, the variational autoencoder adopts distillation training, and the generated normal rail image and the rail abnormal image sample of step S2 are used as the input of the variational autoencoder; the autoencoder trained in step S1 is used as the teacher model of the variational autoencoder, and multi-scale feature supervision training of the decoding module of the variational autoencoder is performed, so that the decoding module of the variational autoencoder is aligned with the decoding module of the autoencoder in feature spaces of multiple scales, and different semantic information is reconstructed; The training process of the variational autoencoder is as follows: Prepare normal rail images and synthetic pseudo-abnormal rail images, and divide them into training sets and test sets; Build a variational autoencoder model, which has a similar structure to the autoencoder. On the basis of the autoencoder, it adds a downsampling network layer and an upsampling network layer to encode the input image into a vector distribution in the latent space, and then reconstruct it to generate a new image. In the image preprocessing stage, the input image is enhanced by HSV color domain dithering and small-angle horizontal rotation; After the image is preprocessed, the normal rail image and the synthesized pseudo-abnormal rail image are input into the variational autoencoder to generate normal image samples, and the normal image is input into the autoencoder to obtain the reconstructed image; Calculate the mean square error between the image features of the variational autoencoder decoding module with sizes of 32×32, 64×64, and 128×128 and the image features of the corresponding scale of the autoencoder, and calculate the mean square error between the image generated by the variational autoencoder and the normal image to obtain the feature reconstruction and image generation loss function : ; In the formula, Represents the variational autoencoder decoding module Image features of size, Represents the autoencoder decoding module Image features of size, express Image features under size and The mean square error between represents the normal images input into the variational autoencoder model and the autoencoder model respectively, represents a normal image generated by the variational autoencoder model, Representing images and The mean square error between Based on the loss function of the variational autoencoder, the model parameters of the variational autoencoder are optimized so that the variational autoencoder has the ability to generate normal images when both abnormal images and normal images are input; S4. Multi-feature-aware image anomaly detection through trained autoencoders and variational autoencoders.

12. The method for semi-supervised anomaly detection of foreign objects on rails deployed at the edge according to claim 11, It is characterized in that The S4 is specifically as follows: 1) Statistical autoencoder and variational autoencoder on the test set of different scale image features mean and variance; 2) Input the preprocessed image to be tested, and the autoencoder and variational autoencoder output three image features and reconstructed images of the same size respectively; 3) Use the mean and variance of the statistics in step 1) to normalize the image features output by the autoencoder and variational autoencoder; 4) Calculate the mean square error between the three-size image features of the autoencoder and the variational autoencoder, called the feature reconstruction error, accumulate the feature reconstruction errors according to the feature channel dimension, and perform bilinear interpolation upsampling operations to obtain the reconstructed feature error with the same width and height as the input image to be tested; 5) Calculate the mean square error between the image reconstructed by the autoencoder and the image generated by the variational autoencoder, called the image reconstruction error, and accumulate the reconstructed image errors by channel dimension to obtain an image reconstruction error of the same dimension as the feature reconstruction error; 6) Sum the feature reconstruction error and image reconstruction error of the three sizes, perform min-max normalization, and map the error value to In the interval, the area where the error value is greater than the error threshold is judged as an abnormal area of ​​the rail image, otherwise it is a normal area of ​​the rail image.

13. The method for semi-supervised anomaly detection of foreign objects on rails deployed at the edge according to claim 11, It is characterized in that The loss function calculation formula for the autoencoder model training is as follows: ; in, Represents the autoencoder decoding module Image features of size, Represents the teacher model Image features of size, express Image features under size and The mean square error between represents the image reconstructed by the autoencoder, represents the input image, Representing images and The mean square error.

14. The method for semi-supervised anomaly detection of foreign objects on rails deployed at the edge according to claim 11, It is characterized in that The teacher model of the autoencoder is one of ResNet50, VGG19, and HRNet.

15. The method for semi-supervised anomaly detection of foreign objects on rails deployed at the edge according to claim 11, It is characterized in that The S2 is specifically as follows: Prepare the COCO instance segmentation dataset, use the object instances in the data as pseudo foreign objects on the rails, and extract the mask map of each instance; the COCO dataset includes common objects, animals, and pedestrians in life, which are consistent with the foreign objects that may appear on the rails; Get normal rail image data and filter special image samples where trains pass by; Based on the instance mask map, the instance image and the rail area are superimposed and synthesized according to a preset ratio. The synthesis formula is as follows: ; in, represents a COCO image, represents a normal image sample of a railway track, represents the synthetic rail pseudo-anomaly image, Indicates that an object instance is in the image The mask map of It is a mask image The negation result of is the stacking coefficient of image synthesis.

16. The method for semi-supervised anomaly detection of foreign objects on rails deployed at the edge according to claim 11, It is characterized in that The loss function of the variational autoencoder is The definition is as follows: ; In the formula, represents the feature reconstruction and image generation loss function of the input normal image, represents the feature reconstruction and image generation loss function of the input pseudo-anomaly image, , and Represent the weight coefficients of different losses respectively, is the KL divergence loss between the normal distribution of the input data and the standard normal distribution.

17. The method for semi-supervised anomaly detection of foreign objects on rails deployed at the edge according to claim 16, It is characterized in that Said Calculated by the following formula: ; in, is the dimension of the latent space vector, and They represent the latent space vector The mean and variance of the dimensions, and They represent the mean and variance of the latent space vector respectively.

18. According to claim 1, the edge-deployed semi-supervised anomaly detection method for foreign objects on rails, It is characterized in that The semi-supervised rail foreign body detection model corresponding to the expert model includes a rail foreign body detection method based on deep feature comparison, comprising the following steps: Training phase: Obtain normal samples and perform overlapping block processing, and then generate rail abnormal samples; The abnormal rail samples and normal rail samples form a training sample pair for preprocessing, and the encoding module and decoding module extract, fuse and decode the features of the preprocessed data to complete the training; Testing phase: First, fix the detection scene, obtain the next frame of the current rail scene without foreign objects as the normal rail sample, and the rail to be inspected sample is the real-time lens scene image; Then, the normal rail sample and the rail sample to be inspected are treated as a test sample pair and preprocessed; The trained encoding module and decoding module are used to output the corresponding prediction results. Finally, logical judgment is made on the decoded results to issue real-time alarms for abnormal rail conditions.

19. The method for semi-supervised anomaly detection of foreign objects on rails deployed at the edge according to claim 18, It is characterized in that During the training phase, the method for generating the foreign body image in the rail abnormality sample is as follows: (1) Specify the image of the foreign body generation area; (2) Simulate unknown rail foreign objects of different sizes and colors that may appear in a single scene: Use public segmentation task datasets to obtain foreground images of objects of different categories; The resolution of the foreground image is scaled, rotated and the color space transformed at different ratios according to the preset probability to simulate foreign objects of different sizes and colors and obtain the foreign object image on the rail.

20. The method for semi-supervised anomaly detection of foreign objects on rails deployed at the edge according to claim 18, It is characterized in that During the training phase, the method for generating the foreign body image in the rail abnormality sample is as follows: A. Specify the image of the foreign body generation area ; B. Use a few rail scene images to simulate rail foreign bodies in most scenarios: a. Generate regional image of foreign matter The brightness, saturation and hue of the image are randomly adjusted, and then the adjusted image is divided into grids of equal proportions, and then the grids are rearranged to obtain a noise image simulating structural abnormalities. ; b. Using the size of the region of interest or the size of the rail area, randomly generate a two-dimensional Perlin noise of the same size , and set the threshold to perform binarization processing to obtain the processing result ; c. Obtain the noise foreground image with structural abnormality through the following formula , simulate the foreign objects that may appear in different rail scenes, and thus obtain the rail foreign object image: ; in, The transparency factor introduced is to better balance the image of the foreign body generation area and noisy image The fusion of the two images makes the fused image closer to the real anomaly.

21. The method for detecting rail foreign bodies in a semi-supervised manner at the edge according to claim 18, It is characterized in that During the training phase, the foreign body images in the rail abnormality samples After generation, generate the abnormal area image , the corresponding formula is: ; in, Generates an area image for the foreign object, To process the results Invert to get; use the size of the region of interest or the size of the rail area to randomly generate a two-dimensional Perlin noise of the same area size , and set the threshold to perform binarization processing to obtain the processing result ; The abnormal area image Image of the foreign matter generation area replacing the normal sample , that is, spliced ​​to the region of interest or rail area corresponding to the normal sample to obtain the final rail abnormal sample.

22. The method for detecting rail foreign bodies in a semi-supervised manner at the edge according to claim 18, It is characterized in that The encoding module uses two feature extraction networks with the same weight to form a twin network, processes the preprocessed abnormal samples and normal samples, and outputs three different levels of features: Feature map of normal samples , and ; And the characteristic map of abnormal samples , and ; Among them, the first dimension N represents the number of samples input to the network in one iterative training, the second dimension represents the number of channels, and the third and fourth dimensions represent the height and width of the feature map; Then, the feature maps of three different levels corresponding to normal samples and abnormal samples are spliced ​​according to the channel dimension: ; Among them, i=1,2,3, corresponding to the three different levels in the middle of the feature extraction network, It represents the tensor channel concatenation operation, and the final result is recorded as , and ; Finally, the differences between the three different levels of feature maps corresponding to normal samples and abnormal samples are calculated respectively. The operation is as follows: ; Among them, i=1,2,3, corresponding to the three different levels in the middle of the feature extraction network, It represents the mean square error of each position of the two input tensors, and the result is recorded as , and .

23. The method for semi-supervised anomaly detection of foreign objects on rails deployed at the edge according to claim 22, It is characterized in that The encoding module obtains the feature f obtained by channel splicing through a multi-scale fusion network. 1_concat 、f 2_concat and f 3_concat Perform feature fusion: First, a channel number is halved. Convolution and a constant number of channels Convolution on f 1_concat 、f 2_concat and f 3_concat Extract features and then use a Convolution increases the nonlinearity of the network, and we get , and ; Remember f 3_msff = f 3_conv , the feature map f 3_msff Upsampled twice After the size is connected, the number of channels is halved Convolution operation obtains feature map , then f _temp With f 2_conv Add element by element to get ; Similarly for f 2_msff Upsampling by two times and halving the number of channels After the convolution operation, 1_conv Add element by element to get ; Finally, the multi-scale fusion features are output , and .

24. The method for semi-supervised anomaly detection of foreign objects on rails deployed at the edge according to claim 18, It is characterized in that The decoding module is composed of multiple upsampling convolution operations combined The convolution operation is repeated, starting from the deep features, and finally using a The convolution operation reduces the channel dimension to 2 dimensions.

25. The method for semi-supervised anomaly detection of foreign objects on rails deployed at the edge according to claim 18, It is characterized in that The training process of the decoding module is as follows: The output f of the encoding module 1_encode 、f 2_encode and f 3_encode As the input of the decoding module, let f 3_decode =f 3_encode ,right Upsample twice to Size followed by one The convolution operation reduces the channel dimension to obtain the feature map Finally, f _temp1 With f 2_encode Concatenate in the channel dimension ; Similarly, for f 2_decode Connect one after double upsampling Convolution operation obtains feature map , and then the feature map f _tepm2 With f 1_encode Concatenate in the channel dimension ; In order to further reduce the amount of calculation of the model, we continue to 1_decode After upsampling twice, connect a The convolution operation reduces the channel dimension to obtain the feature map ; f 0_decoder Continue to upsample twice and connect a channel dimension unchanged The convolution operation obtains the same size as the input sample Feature map ; Finally, use a The convolution operation is performed on f _up2mask Extract features and reduce the channel dimension to 2 dimensions to obtain , where the first dimension of the channel dimension is the foreground output and the second dimension is the background output.

26. A semi-supervised anomaly detection system for foreign objects on rails deployed at the edge. It is characterized in that A method for semi-supervised anomaly detection of foreign objects on rails deployed at the edge as described in any one of claims 1 to 25; comprising a semantic segmentation model and a hybrid expert system; wherein the semantic segmentation model is used to perform image segmentation on a frame image obtained by frame extraction to obtain a rail area segmentation image; the hybrid expert system comprises a threshold network and several expert models, the output of the threshold network determines which expert model is activated, and the semi-supervised anomaly detection model for foreign objects on rails corresponding to the expert model is used as the detection model for the current rail area segmentation image and completes the detection of foreign objects on rails.

27. A server comprising a processor and a memory, It is characterized in that At least one program is stored in the memory, and the program is loaded and executed by the processor to implement the semi-supervised anomaly detection method for foreign objects on rails deployed at the edge as described in any one of claims 1 to 25.

28. A computer-readable storage medium, wherein at least one program is stored in the storage medium. It is characterized in that The program is loaded and executed by the processor to implement the semi-supervised anomaly detection method for foreign objects on rails deployed at the edge as described in any one of claims 1 to 25.