A railway foreign matter intruding detection method, system, device and medium

Abstract

The application discloses a railway foreign matter intrusion detection method, system, device and medium, and relates to the field of railway foreign matter detection.The method comprises the following steps: acquiring a first frame key frame railway image of a current scene as a template image and an Nth frame railway image as a detection image; inputting the template image and the detection image of the current scene into a foreign matter intrusion prediction model to obtain a foreign matter feature map of the current scene, so as to determine whether there is a difference target between the detection image and the template image of the current scene; if there is a difference target, mobile objects in the difference target of the detection image of the current scene are filtered out by using a mobile object filtering model, and the real foreign matter in the detection image of the current scene is obtained.The application can improve the reliability of railway foreign matter intrusion detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of railway foreign matter detection, and in particular to a railway foreign matter intrusion detection method, system, device and medium. BACKGROUND

[0002] Foreign matter intrusion is the most serious threat to the external environment of railway operation. The traditional foreign matter detection is completed by manual work. Since the occurrence of foreign matter intrusion events is sudden, unpredictable and irregular, when facing sudden foreign matter intrusion (such as rockfall and landslide), if timely warning cannot be given, serious consequences are likely to occur.

[0003] Image recognition is one of the main research directions in the field of artificial intelligence. The traditional image detection method often adopts the difference method, that is, real-time comparison of the difference between the snapshot and the template image. However, this method has single technical means, is easily affected by light and camera angle, has poor reliability, and is difficult to adapt to complex outdoor scenes. SUMMARY

[0004] Therefore, the embodiments of the present application provide a railway foreign matter intrusion detection method, system, device and medium to improve the reliability of railway foreign matter intrusion detection.

[0005] To achieve the above object, the embodiments of the present application provide the following solutions.

[0006] A railway foreign matter intrusion detection method comprises:

[0007] Obtaining a template image, a to-be-detected image and a to-be-measured image set of a current scene; the template image is a first frame key frame railway image; the to-be-detected image is an Nth frame railway image; the to-be-measured image set comprises an (N+1)th frame railway image to an (N+m)th frame railway image; wherein N>1 and m>1;

[0008] Inputting the template image of the current scene and the to-be-detected image of the current scene into a foreign matter intrusion prediction model to obtain a foreign matter feature map of the current scene; the foreign matter feature map is used to determine whether there is a difference target between the to-be-detected image and the template image; the foreign matter intrusion prediction model is constructed based on a twin network and a UNet segmentation network;

[0009] If the current scene to be detected map and the template map of the current scene exist difference target, the position of the difference target is determined, the target position is obtained, and the moving object filtering model is used to determine whether the target position of each frame image in the current scene to be detected map set exists difference target with the template map of the current scene. If the number of image frames with difference target from the Nth frame railway image to the N+mth frame railway image in the current scene is greater than a set value, it is determined that there is foreign matter in the difference target of the current scene to be detected map, and the area where the position of the difference target in the current scene from the Nth frame railway image to the N+mth frame railway image changes is determined as a moving object. The moving object in the difference target of the current scene to be detected map is filtered out to obtain the real foreign matter in the current scene to be detected map.

[0010] Optionally, the moving object filtering model is used to determine whether the target position of each frame image in the current scene to be detected map set exists difference target with the template map of the current scene, and specifically includes:

[0011] The moving object filtering model uses the IOU matching method to determine whether the target position of each frame image in the current scene to be detected map set exists difference target with the template map of the current scene.

[0012] Optionally, the determination method of the foreign matter intrusion limit prediction model specifically includes:

[0013] Obtain training data; the training data includes: template maps, to-be-detected maps and corresponding label data of different training scenes; the label data includes whether the template map and the to-be-detected map of the training scene exist difference target;

[0014] A fusion network model of a fusion twin network and a UNet segmentation network is constructed; the fusion network model includes: a first encoder, a second encoder, a feature fusion module and a decoder; the first encoder and the second encoder have the same structure and are connected with the feature fusion module; the feature fusion module is connected with the decoder;

[0015] The template maps of different training scenes in the training data are taken as the input of the first encoder, the to-be-detected maps of different training scenes in the training data are taken as the input of the second encoder, the fusion network model is trained with the loss function minimum as the target, and the trained fusion network model is determined as the foreign matter intrusion limit prediction model.

[0016] Optionally, the first encoder includes: a first image division layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer and a fifth convolutional layer connected in sequence;

[0017] The first image division layer is configured to traverse the input template image in a sliding window manner to obtain a plurality of small block template images; the first convolutional layer is configured to perform convolutional operation on each of the small block template images to obtain a first feature map of each of the small block template images; the second convolutional layer is configured to perform convolutional and down-sampling pooling operation on the first feature map to obtain a second feature map; the third convolutional layer is configured to perform convolutional and down-sampling pooling operation on the second feature map to obtain a third feature map; the fourth convolutional layer is configured to perform convolutional and down-sampling pooling operation on the third feature map to obtain a fourth feature map; and the fifth convolutional layer is configured to perform convolutional and down-sampling pooling operation on the fourth feature map to obtain a fifth feature map.

[0018] The second encoder comprises a second image division layer, a sixth convolutional layer, a seventh convolutional layer, an eighth convolutional layer, a ninth convolutional layer and a tenth convolutional layer.

[0019] The second image division layer is configured to traverse the input to-be-detected image in a sliding window manner to obtain a plurality of small block to-be-detected images; the sixth convolutional layer is configured to perform convolutional operation on each of the small block to-be-detected images to obtain a sixth feature map of each of the small block to-be-detected images; the seventh convolutional layer is configured to perform convolutional and down-sampling pooling operation on the sixth feature map to obtain a seventh feature map; the eighth convolutional layer is configured to perform convolutional and down-sampling pooling operation on the seventh feature map to obtain an eighth feature map; the ninth convolutional layer is configured to perform convolutional and down-sampling pooling operation on the eighth feature map to obtain a ninth feature map; and the tenth convolutional layer is configured to perform convolutional and down-sampling pooling operation on the ninth feature map to obtain a tenth feature map.

[0020] The feature fusion module comprises a concatenation layer and a spatial pyramid pooling layer connected in sequence; the concatenation layer is configured to concatenate the fifth feature map and the tenth feature map to obtain a concatenated feature; and the spatial pyramid pooling layer is configured to perform spatial pyramid pooling operation on the concatenated feature to obtain a fused feature.

[0021] The decoder comprises a first up-sampling layer, a second up-sampling layer, a third up-sampling layer, a fourth up-sampling layer and a fifth up-sampling layer connected in sequence; the first up-sampling layer is connected with the spatial pyramid pooling layer; an input of the second up-sampling layer is connected with outputs of the fourth convolutional layer and the ninth convolutional layer; an input of the third up-sampling layer is connected with outputs of the third convolutional layer and the eighth convolutional layer; an input of the fourth up-sampling layer is connected with outputs of the second convolutional layer and the seventh convolutional layer; and an input of the fifth up-sampling layer is connected with outputs of the first convolutional layer and the sixth convolutional layer.

[0022] The first up-sampling layer is configured to perform an up-sampling operation on the fused feature to obtain a first decoding feature; the second up-sampling layer is configured to splice the first decoding feature, the fourth feature map and the ninth feature map to obtain a second decoding feature; the third up-sampling layer is configured to splice the second decoding feature, the third feature map and the eighth feature map to obtain a third decoding feature; the fourth up-sampling layer is configured to splice the third decoding feature, the second feature map and the seventh feature map to obtain a fourth decoding feature; and the fifth up-sampling layer is configured to splice the fourth decoding feature, the first feature map and the sixth feature map to obtain a foreign matter feature map.

[0023] Optionally, the template image of the current scene and the to-be-inspected image of the current scene are input into the foreign matter intrusion prediction model to obtain a foreign matter feature map of the current scene, and the method specifically comprises the following steps.

[0024] The template image of the current scene and the to-be-inspected image of the current scene are respectively subjected to image enhancement processing to obtain a template enhanced image of the current scene and a to-be-inspected enhanced image of the current scene.

[0025] The template enhanced image of the current scene and the to-be-inspected enhanced image of the current scene are input into the foreign matter intrusion prediction model to obtain a foreign matter feature map of the current scene.

[0026] Optionally, the template image of the current scene and the to-be-inspected image of the current scene are respectively subjected to image enhancement processing to obtain a template enhanced image of the current scene and a to-be-inspected enhanced image of the current scene, and the method specifically comprises the following steps.

[0027] The template image of the current scene is sequentially subjected to image correction and image noise processing to obtain a template enhanced image of the current scene.

[0028] The to-be-inspected image of the current scene is sequentially subjected to image correction and image noise processing to obtain a to-be-inspected enhanced image of the current scene.

[0029] Optionally, m=5; and the set value is 4.

[0030] The application further provides a railway foreign matter intrusion detection system, comprising:

[0031] An image acquisition module is configured to acquire a template image, a to-be-inspected image and a to-be-measured image set of a current scene; the template image is a first frame of key frame railway image; the to-be-inspected image is an Nth frame of railway image; and the to-be-measured image set comprises an N+1th frame of railway image to an N+mth frame of railway image; wherein N>1 and m>1.

[0032] The foreign matter invasion prediction module is configured to input a template image of a current scene and a to-be-detected image of the current scene into a foreign matter invasion prediction model to obtain a foreign matter feature map of the current scene, wherein the foreign matter feature map is used to determine whether there is a difference target between the to-be-detected image and the template image, and the foreign matter invasion prediction model is constructed based on a twin network and a UNet segmentation network.

[0033] The filtering module is configured to determine a position of the difference target if there is a difference target between the to-be-detected image of the current scene and the template image of the current scene, obtain the target position, and determine whether there is a difference target between the target position of each frame of image in a to-be-detected image set of the current scene and the template image of the current scene by using a moving object filtering model, and if the number of image frames with a difference target in the Nth railway image to the N+mth railway image of the current scene is greater than a set value, it is determined that there is a foreign matter in the difference target of the to-be-detected image of the current scene, and the region with a changed position in the difference target of the Nth railway image to the N+mth railway image of the current scene is determined as a moving object, and the moving object in the difference target of the to-be-detected image of the current scene is filtered to obtain the real foreign matter in the to-be-detected image of the current scene.

[0034] The present application also provides an electronic device comprising a memory and a processor, wherein the memory is configured to store a computer program, and the processor is configured to execute the computer program to enable the electronic device to perform the railway foreign matter invasion detection method.

[0035] The present application also provides a computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the railway foreign matter invasion detection method.

[0036] According to the embodiments of the present application, the following technical effects are achieved:

[0037] The foreign matter invasion prediction model is constructed based on the twin network and the UNet segmentation network, the recognition of the foreign matter feature map is realized, and it is determined whether there is a difference target between the to-be-detected image and the template image of the target scene; the moving object filtering model is used to filter the moving object in the difference target of the to-be-detected image of the target scene, the real foreign matter in the to-be-detected image of the target scene is obtained, the newly added foreign matter can be quickly and accurately recognized, the reliability of the railway foreign matter invasion detection can be improved, more specific, scientific and efficient monitoring means can be provided for railway construction, operation and maintenance, disaster prevention and mitigation, and the like, the railway operation safety level can be improved, and the present application has great practical application value. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0039] Figure 1 The flowchart of the railway foreign matter intrusion detection method provided by the embodiment of the present application;

[0040] Figure 2 The structural diagram of the fusion network model provided by the embodiment of the present application;

[0041] Figure 3 The railway foreign matter intrusion prediction result diagram provided by the embodiment of the present application;

[0042] Figure 4 The structural diagram of the railway foreign matter intrusion detection system provided by the embodiment of the present application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.

[0044] In recent years, deep learning technology has made great progress in the field of artificial intelligence. In the railway foreign matter intrusion inspection work, image recognition can train the model according to the image data collected on site, so as to extract key information from the image data to identify foreign matters and provide support for railway operation work.

[0045] The purpose of the present application is to provide a railway foreign matter intrusion detection method, system, device and medium, which realizes railway foreign matter intrusion detection based on a twin network and a Unet segmentation network, and improves the reliability of railway foreign matter intrusion detection.

[0046] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] Embodiment one

[0048] Referring to Figure 1 The railway foreign matter intrusion detection method of the embodiment comprises:

[0049] Step 101: Obtain a template image, a to-be-inspected image and a to-be-measured image set of a current scene.

[0050] The template image is a first frame key frame railway image; the to-be-inspected image is an Nth frame railway image; the to-be-measured image set comprises an N+1th frame railway image to an N+mth frame railway image; wherein N>1 and m>1. For example, m=5.

[0051] Step 102: input the template image of the current scene and the to-be-inspected image of the current scene into the foreign matter intrusion prediction model to obtain a foreign matter feature map of the current scene; the foreign matter feature map is used to determine whether there is a difference target between the to-be-inspected image and the template image; the foreign matter intrusion prediction model is constructed based on a twin network and a UNet segmentation network.

[0052] Step 102, specifically comprising:

[0053] (1) performing image enhancement processing on the template image of the current scene and the to-be-inspected image of the current scene respectively to obtain a template enhanced image of the current scene and a to-be-inspected enhanced image of the current scene. Specifically:

[0054] performing image correction and image noise processing on the template image of the current scene in sequence to obtain the template enhanced image of the current scene; performing image correction and image noise processing on the to-be-inspected image of the current scene in sequence to obtain the to-be-inspected enhanced image of the current scene.

[0055] (2) inputting the template enhanced image of the current scene and the to-be-inspected enhanced image of the current scene into the foreign matter intrusion prediction model to obtain a foreign matter feature map of the current scene.

[0056] Wherein, the determination method of the foreign matter intrusion prediction model specifically comprises:

[0057] ① obtaining training data; the training data includes: template images, to-be-inspected images and corresponding label data of different training scenes; the label data includes whether there is a difference target between the template image and the to-be-inspected image of the training scene.

[0058] ② constructing a fusion network model fusing a twin network and a UNet segmentation network; the fusion network model includes: a first encoder, a second encoder, a feature fusion module and a decoder; the first encoder and the second encoder have the same structure and are connected with the feature fusion module; the feature fusion module is connected with the decoder.

[0059] The structure of the fusion network model will be specifically introduced below. Figure 2

[0060] See Figure 2 ​The first encoder comprises a first image division layer (not shown in the figure), a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer and a fifth convolutional layer connected in sequence. The first image division layer is configured to traverse the input template image in a sliding window manner to obtain a plurality of small block template images; the first convolutional layer is configured to perform convolutional operation on each small block template image to obtain a first feature map F1 of each small block template image; the second convolutional layer is configured to perform convolutional and down-sampling pooling operation on the first feature map F1 to obtain a second feature map F2; the third convolutional layer is configured to perform convolutional and down-sampling pooling operation on the second feature map F2 to obtain a third feature map F3; the fourth convolutional layer is configured to perform convolutional and down-sampling pooling operation on the third feature map F3 to obtain a fourth feature map F4; and the fifth convolutional layer is configured to perform convolutional and down-sampling pooling operation on the fourth feature map F4 to obtain a fifth feature map F5.

[0061] The second encoder comprises a second image division layer, a sixth convolutional layer, a seventh convolutional layer, an eighth convolutional layer, a ninth convolutional layer and a tenth convolutional layer. The second image division layer is configured to traverse the input detection image in a sliding window manner to obtain a plurality of small block detection images; the sixth convolutional layer is configured to perform convolutional operation on each small block detection image to obtain a sixth feature map F1' of each small block detection image; the seventh convolutional layer is configured to perform convolutional and down-sampling pooling operation on the sixth feature map F1' to obtain a seventh feature map F2'; the eighth convolutional layer is configured to perform convolutional and down-sampling pooling operation on the seventh feature map F2' to obtain an eighth feature map F3'; the ninth convolutional layer is configured to perform convolutional and down-sampling pooling operation on the eighth feature map F3' to obtain a ninth feature map F4'; and the tenth convolutional layer is configured to perform convolutional and down-sampling pooling operation on the ninth feature map F4' to obtain a tenth feature map F5'.

[0062] The feature fusion module comprises a splicing layer and a spatial pyramid pooling layer connected in sequence; the splicing layer is configured to splice the fifth feature map F5 and the tenth feature map F5' to obtain a spliced feature; and the spatial pyramid pooling layer is configured to perform spatial pyramid pooling operation on the spliced feature to obtain a fused feature.

[0063] The decoder comprises a first upsampling layer, a second upsampling layer, a third upsampling layer, a fourth upsampling layer and a fifth upsampling layer connected in sequence; the first upsampling layer is connected with the spatial pyramid pooling layer; an input of the second upsampling layer is connected with outputs of the fourth convolutional layer and the ninth convolutional layer; an input of the third upsampling layer is connected with outputs of the third convolutional layer and the eighth convolutional layer; an input of the fourth upsampling layer is connected with outputs of the second convolutional layer and the seventh convolutional layer; and an input of the fifth upsampling layer is connected with outputs of the first convolutional layer and the sixth convolutional layer.

[0064] The first upsampling layer is configured to perform an upsampling operation on the fusion feature to obtain a first decoding feature P1; the second upsampling layer is configured to splice the first decoding feature P1, the fourth feature map F4 and the ninth feature map F4' to obtain a second decoding feature P2; the third upsampling layer is configured to splice the second decoding feature P2, the third feature map F3 and the eighth feature map F3' to obtain a third decoding feature P3; the fourth upsampling layer is configured to splice the third decoding feature P3, the second feature map F2 and the seventh feature map F2' to obtain a fourth decoding feature P4; and the fifth upsampling layer is configured to splice the fourth decoding feature P4, the first feature map F1 and the sixth feature map F1' to obtain a foreign matter feature map P5.

[0065] ③Taking the template maps of different training scenes in the training data as inputs of the first encoder and taking the detection maps of different training scenes in the training data as inputs of the second encoder, the fusion network model is trained with the minimum loss function as the target, and the trained fusion network model is determined as the foreign matter invasion prediction model.

[0066] Step 103: If there is a difference target between the detection map of the current scene and the template map of the current scene, the position of the difference target is determined to obtain the target position, and a moving object filtering model is used to determine whether there is a difference target between the target position of each frame of image in the detection map set of the current scene and the template map of the current scene.

[0067] Specifically, the moving object filtering model uses an IOU matching method to determine whether there is a difference target between the target position of each frame of image in the detection map set of the current scene and the template map of the current scene.

[0068] Step 104: If the number of image frames of the difference target existing in the Nth frame railway image to the N+mth frame railway image of the current scene is greater than the set value, it is determined that the difference target of the current scene contains foreign matter, and the region of the difference target in which the position changes from the Nth frame railway image to the N+mth frame railway image of the current scene is determined as the moving object. The moving object in the difference target of the current scene is filtered out, and the real foreign matter in the current scene is obtained. For example, the set value can be 4.

[0069] A specific example in practical application is provided below to further illustrate the railway foreign matter intrusion detection method of the above embodiment. The implementation process of the example is as follows: acquiring a key frame image to establish a template image, and acquiring a real-time image as a to-be-inspected image; image enhancement; extracting features of the template image and the to-be-inspected image; segmenting object contours of the template image and the to-be-inspected image; establishing a foreign matter intrusion prediction model, comparing differences between the template image and the to-be-inspected image; filtering moving objects; and predicting the difference objects. Each step is specifically introduced below.

[0070] S1, acquiring a key frame image to establish a template image, and acquiring a real-time image as a to-be-inspected image: acquiring camera video stream data, extracting a first frame key frame image as a template image, and continuously capturing real-time images as to-be-inspected images thereafter.

[0071] Specifically, the first frame key frame image is acquired as a template image by pulling a network camera video stream. The video stream can support RTSP protocol, RTMP protocol, support H.264, H.265 encoding format.

[0072] A fixed key frame interval time in the video stream is set to capture frame images in the real-time video stream as the to-be-inspected images at regular intervals. The to-be-inspected image capturing time is later than the template image, and the time stamp is used to distinguish the order.

[0073] S2, image enhancement: including image correction, noise processing and other enhancement processing, so that the image can truly reflect the shape of the object and keep consistent in the coordinate position.

[0074] Image correction processing: due to the erection angle of the camera, the image distortion and distortion caused by the sensor itself, which will interfere with the identification target, so image correction processing is needed. Geometric correction is adopted. According to the reason for distortion, the spatial position change relationship is used to realize the pixel coordinate transformation between the distorted image and the reference image by using the correction model. The correction is performed by using the calculation formula and the obtained auxiliary parameters.

[0075] Noise processing: The image is often affected by the imaging device and external environmental noise during digitization and transmission, resulting in interference in imaging. The tilt error of the image and the projection error caused by the terrain fluctuation are eliminated. The spatial domain method is used to directly operate data on the original image, the gray value of the pixel is processed, the bilateral filter is used, the spatial proximity of each pixel point to the center point is calculated, the weight value is optimized, and the weight value is optimized as the product of the weight value of the spatial proximity calculation and the weight value of the pixel value similarity calculation. The optimized weight value is convolved with the image, so as to achieve the effect of edge-preserving denoising.

[0076] S3: Extracting the foreign matter feature map of the template image and the to-be-inspected image. First, the processed template image and the to-be-inspected image obtained in step S2 are taken as the input end of the fusion network model, and a plurality of regions are obtained by traversing the template image and the to-be-inspected image through the sliding window method. Still referring to Figure 2 , a 3*3 convolution operation is performed on each region to obtain an initial feature map F1 and F1'. Then, based on the feature extraction idea of UNet, F1 and F1' are respectively subjected to four times of convolution and down-sampling pooling, and finally the output feature F5 of the template image and the output feature F5' of the to-be-inspected image are obtained. All channels of the output features F5 and F5' are spliced and subjected to spatial pyramid pooling to obtain P1, so as to expand the receptive field of the feature map. Finally, based on the feature fusion idea of UNet, P1 is up-sampled and subjected to feature channel splicing with F4, F4' to obtain P2; P2 is up-sampled and subjected to feature channel splicing with F3, F3' to obtain P3…, and so on. The foreign matter feature map P5 of the template image and the to-be-inspected image can be obtained.

[0077] The original image is traversed from left to right and from top to bottom by the sliding window method with a size of 100*100, and a complete image is divided into several parts according to the set mode and rule. The set mode refers to the step length and window size of the sliding window, and the window size is set to 100*100 and the step length is set to 50*50.

[0078] It should be noted that each convolution is accompanied by an activation function layer and Batch Normalization. The nonlinear activation function can improve the nonlinear learning ability of the network. Batch Normalization can speed up the convergence speed of model training, make the model training process more stable, and avoid gradient explosion or gradient disappearance.

[0079] S4: Training the fusion network model to establish a foreign matter invasion limit prediction model and making a prediction. The contrast loss is used as the loss function of the network for training, and the specific process is as follows:

[0080] The feature similarity distance D between the template image X1 of the training scene and the to-be-inspected image X2 of the training scene is considered W, P represents a feature dimension, Y is label data of whether two samples (namely X1 and X2) match (if Y = 1, it represents that the two samples are similar or match, that is, there is no difference target between the template image and the to-be-detected image; if Y = 0, it represents that the two samples are not similar or match, that is, there is a difference target between the template image and the to-be-detected image). Assuming that M is a set threshold, N is the number of samples, and W represents the network weight during training, the contrast loss L(W, (Y, X1, X2)) can be defined as:

[0081]

[0082] wherein the feature similarity distance D W Using the cosine similarity distance, it is represented as follows:

[0083]

[0084] It should be noted that in the formula, represents the feature matrix of the template image X1, represents the feature matrix of the to-be-detected image X2. The core idea is to measure the cosine similarity of the feature matrix, and to learn the similarity of the two inputs by reducing the loss. Finally, the trained network weight and network structure are obtained as the foreign matter invasion prediction model.

[0085] In the prediction process, only the to-be-detected image and the template image of the target scene need to be input into the foreign matter invasion prediction model, and the foreign matter segmentation image can be obtained. If there is a foreign matter in the to-be-detected image but not in the template image, it is considered as a foreign matter, and the foreign matter is labeled. The label is not limited to foreign matter, but also includes two types of trains and pedestrians. When trains and pedestrians appear, the system will filter them by default. The segmentation result includes the coordinate information, pixel information and area information of all object contours. It should be noted that the object is not distinguished according to the type of the object, but according to the obtained object contour and the image region framed by the object contour in the to-be-detected image to distinguish the object.

[0086] S5, moving object filtering model: a classification model is obtained by classifying and training the obtained moving target, and the moving object is identified for the difference target predicted by S4.

[0087] Based on the to-be-detected image with foreign matter obtained in step S4, the features of the to-be-detected image are extracted for comparison for 6 consecutive images, and the foreign matter of each to-be-detected image is tracked. If there is foreign matter information in 4 or more to-be-detected images, it is identified as a foreign matter; if it is less than 3, it is not identified as a foreign matter.

[0088] After identifying the foreign matter, the position information and pixel information of the target are compared and tracked, and the target whose position changes obviously is identified as a moving object and is filtered.

[0089] The system stores the detection results of the previous 6 frames in a result queue. When the first frame detects a foreign object, it checks whether the same position has a foreign object in the previous 5 frames. If 4 frames have a foreign object in the 6 frames, it is considered that there is a foreign object, otherwise it is considered that it is a moving object, such as a flying insect or a domestic cow or sheep, and the camera flash can also be suppressed. The method used to determine whether it is at the same position is IOU matching. When the IOU values of the foreign objects in the current and previous two frames are greater than a preset threshold, it is considered to be the same object.

[0090] S6, railway foreign object intrusion prediction: based on the classification results obtained in step S5, the moving objects are filtered, thereby realizing the prediction of the foreign object intrusion.

[0091] Based on the railway foreign object intrusion prediction model obtained by the above steps, the real-time camera video stream is accessed, the key frames are extracted and compared and analyzed in real time, and the prediction of the foreign object intrusion of the detected railway section is realized. Figure 3 The results of the railway foreign object intrusion prediction are shown, wherein, Figure 3 part (a) is a template image, Figure 3 part (b) is a to-be-detected image, Figure 3 part (c) is a prediction result image.

[0092] In this embodiment, the acquired real-time image of the railway section is subjected to image enhancement processing such as image correction and noise processing, the template image and the real-time to-be-detected image are extracted by using the sliding window technology, the target is segmented according to the characteristics, the foreign object intrusion prediction model is established according to the characteristics of the template image, the model compares the characteristics of the to-be-detected image, the difference target is obtained, and the moving object is filtered, thereby realizing the railway foreign object intrusion prediction.

[0093] This embodiment has the following advantages:

[0094] (1) The railway foreign object intrusion prediction can be realized without pre-collecting all foreign object types to train the recognition type ability, and the newly added foreign object can be quickly and accurately recognized, and the iteration frequency is high.

[0095] (2) The application cost is low, the coverage is wide, the reliability is high, and the system can provide more specific, scientific and efficient monitoring means for railway construction, operation and maintenance, disaster prevention and reduction, etc., can improve the safety level of railway operation, and has great practical application value.

[0096] Embodiment two

[0097] In order to perform the method corresponding to the above-mentioned embodiment one to realize the corresponding functions and technical effects, a railway foreign object intrusion detection system is provided below.

[0098] Referring to Figure 4 , the system comprises:

[0099] The image acquisition module 201 is configured to acquire a template image of a current scene, a to-be-inspected image and a set of to-be-tested images; the template image is a first frame of key frame railway image; the to-be-inspected image is an Nth frame of railway image; the set of to-be-tested images comprises an (N+1)th frame of railway image to an (N+m)th frame of railway image; wherein N>1 and m>1.

[0100] The foreign matter intrusion prediction module 202 is configured to input the template image of the current scene and the to-be-inspected image of the current scene into a foreign matter intrusion prediction model to obtain a foreign matter feature map of the current scene; the foreign matter feature map is used to determine whether there is a difference target between the to-be-inspected image and the template image; the foreign matter intrusion prediction model is constructed based on a twin network and a UNet segmentation network.

[0101] The filtering module 203 is configured to determine the position of the difference target if there is a difference target between the to-be-inspected image of the current scene and the template image of the current scene, to obtain the target position, and to determine whether there is a difference target between the target position of each frame of image in the set of to-be-tested images of the current scene and the template image of the current scene by using a moving object filtering model; if the number of image frames with a difference target in the Nth frame of railway image to the (N+m)th frame of railway image of the current scene is greater than a set value, it is determined that there is a foreign matter in the difference target of the to-be-inspected image of the current scene, and the region with a changed position in the difference target of the Nth frame of railway image to the (N+m)th frame of railway image of the current scene is determined as a moving object; the moving object in the difference target of the to-be-inspected image of the current scene is filtered out to obtain the real foreign matter in the to-be-inspected image of the current scene.

[0102] Embodiment three

[0103] The embodiment provides an electronic device, comprising a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to make the electronic device execute the railway foreign matter intrusion detection method of the embodiment one.

[0104] Optionally, the electronic device can be a server.

[0105] In addition, the embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the railway foreign matter intrusion detection method of the embodiment one.

[0106] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same and similar parts of each embodiment can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the related parts can be referred to the method part.

[0107] The principles and implementation manners of the present application are described by using specific examples in the present application, and the above examples are only used to help understand the method of the present application and its core idea; meanwhile, for the general technical personnel in the art, the specific implementation manners and application ranges will be changed according to the idea of the present application. In conclusion, the content of the present specification should not be understood as the limitation of the present application.

Claims

1. A method for detecting foreign object intrusion in railway lines, characterized in that, include: Obtain the template image, the image to be inspected, and the image to be tested for the current scene; The template image is the first keyframe railway image; The image to be inspected is the Nth frame of a railway image; The set of images to be tested includes: railway images from frame N+1 to frame N+m; where N>1, m>1; The template image and the image to be inspected in the current scene are input into the foreign object intrusion prediction model to obtain the foreign object feature map of the current scene; the foreign object feature map is used to determine whether there is a different target between the image to be inspected and the template image; the foreign object intrusion prediction model is built based on Siamese network and UNet segmentation network; If there are discrepancies between the image to be inspected in the current scene and the template image of the current scene, the location of the discrepancies is determined, and the target location is obtained. A moving object filtering model is used to determine whether there are discrepancies between the target location of each frame in the image set to be inspected in the current scene and the template image of the current scene. If the number of frames with discrepancies between the Nth and N+mth railway images in the current scene is greater than a set value, it is determined that there are foreign objects in the discrepancies of the image to be inspected in the current scene. The regions in the discrepancies between the Nth and N+mth railway images in the current scene whose positions change are identified as moving objects. The moving objects in the discrepancies of the image to be inspected in the current scene are filtered out to obtain the real foreign objects in the image to be inspected in the current scene.

2. The railway foreign object intrusion detection method according to claim 1, characterized in that, A moving object filtering model is used to determine whether there are differences between the target location in each frame of the test image set of the current scene and the template image of the current scene. Specifically, this includes: The moving object filtering model uses IOU matching to determine whether there is a difference between the target position of each frame in the test image set of the current scene and the template image of the current scene.

3. The railway foreign object intrusion detection method according to claim 1, characterized in that, The method for determining the foreign object intrusion limit prediction model specifically includes: Acquire training data; the training data includes: template images, images to be inspected, and corresponding label data for different training scenarios; the label data includes whether there are differences between the template images and images to be inspected in the training scenarios. A fusion network model integrating a Siamese network and a UNet segmentation network is constructed. The fusion network model includes a first encoder, a second encoder, a feature fusion module, and a decoder. The first encoder and the second encoder have the same structure and are both connected to the feature fusion module. The feature fusion module is connected to the decoder. Template images of different training scenarios in the training data are used as input to the first encoder, and images of the objects to be inspected in different training scenarios in the training data are used as input to the second encoder. The fusion network model is trained with the goal of minimizing the loss function, and the trained fusion network model is determined as the foreign object intrusion prediction model.

4. The railway foreign object intrusion detection method according to claim 3, characterized in that, The first encoder includes: a first image segmentation layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer connected in sequence; The first image segmentation layer is used to traverse the input template image using a sliding window to obtain multiple small template images; the first convolutional layer is used to perform convolution operations on each of the small template images to obtain a first feature map of each small template image; the second convolutional layer is used to perform convolution and downsampling pooling operations on the first feature map to obtain a second feature map; the third convolutional layer is used to perform convolution and downsampling pooling operations on the second feature map to obtain a third feature map; the fourth convolutional layer is used to perform convolution and downsampling pooling operations on the third feature map to obtain a fourth feature map; and the fifth convolutional layer is used to perform convolution and downsampling pooling operations on the fourth feature map to obtain a fifth feature map. The second encoder includes: a second image segmentation layer, a sixth convolutional layer, a seventh convolutional layer, an eighth convolutional layer, a ninth convolutional layer, and a tenth convolutional layer; The second image segmentation layer is used to traverse the input image to be inspected using a sliding window method to obtain multiple small blocks of the image to be inspected; the sixth convolutional layer is used to perform a convolution operation on each of the small blocks of the image to be inspected to obtain a sixth feature map for each of the small blocks of the image to be inspected; the seventh convolutional layer is used to perform convolution and downsampling pooling operations on the sixth feature map to obtain a seventh feature map; the eighth convolutional layer is used to perform convolution and downsampling pooling operations on the seventh feature map to obtain an eighth feature map; the ninth convolutional layer is used to perform convolution and downsampling pooling operations on the eighth feature map to obtain a ninth feature map; the tenth convolutional layer is used to perform convolution and downsampling pooling operations on the ninth feature map to obtain a tenth feature map; The feature fusion module includes: a stitching layer and a spatial pyramid pooling layer connected in sequence; the stitching layer is used to stitch the fifth feature map and the tenth feature map to obtain stitched features; the spatial pyramid pooling layer is used to perform spatial pyramid pooling operation on the stitched features to obtain fused features; The decoder includes: a first upsampling layer, a second upsampling layer, a third upsampling layer, a fourth upsampling layer, and a fifth upsampling layer connected in sequence; the first upsampling layer is connected to the spatial pyramid pooling layer; the input of the second upsampling layer is connected to the outputs of the fourth convolutional layer and the ninth convolutional layer; the input of the third upsampling layer is connected to the outputs of the third convolutional layer and the eighth convolutional layer; the input of the fourth upsampling layer is connected to the outputs of the second convolutional layer and the seventh convolutional layer; and the input of the fifth upsampling layer is connected to the outputs of the first convolutional layer and the sixth convolutional layer. The first upsampling layer is used to upsample the fused features to obtain a first decoded feature; the second upsampling layer is used to concatenate the first decoded feature, the fourth feature map, and the ninth feature map to obtain a second decoded feature; the third upsampling layer is used to concatenate the second decoded feature, the third feature map, and the eighth feature map to obtain a third decoded feature; the fourth upsampling layer is used to concatenate the third decoded feature, the second feature map, and the seventh feature map to obtain a fourth decoded feature; and the fifth upsampling layer is used to concatenate the fourth decoded feature, the first feature map, and the sixth feature map to obtain a foreign object feature map.

5. The railway foreign object intrusion detection method according to claim 1, characterized in that, The template image and the image to be inspected in the current scene are input into the foreign object intrusion prediction model to obtain the foreign object feature map of the current scene, specifically including: Image enhancement processing is performed on the template image and the image to be inspected in the current scene to obtain the template enhanced image and the image to be inspected in the current scene. The template enhancement map and the inspection enhancement map of the current scene are input into the foreign object intrusion prediction model to obtain the foreign object feature map of the current scene.

6. The railway foreign object intrusion detection method according to claim 5, characterized in that, Image enhancement processing is performed on the template image and the image to be inspected in the current scene, respectively, to obtain the enhanced template image and the enhanced image to be inspected in the current scene. Specifically, this includes: The template image of the current scene is sequentially subjected to image correction and image noise processing to obtain the template enhancement image of the current scene; The image to be inspected in the current scene is subjected to image correction and image noise processing in sequence to obtain the enhanced image to be inspected in the current scene.

7. The railway foreign object intrusion detection method according to claim 1, characterized in that, m = 5; the set value is 4.

8. A railway foreign object intrusion detection system, characterized in that, include: The image acquisition module is used to acquire a set of template images, images to be inspected, and images to be tested for the current scene; The template image is the first keyframe railway image; The image to be inspected is the Nth frame of a railway image; The set of images to be tested includes: railway images from frame N+1 to frame N+m; where N>1, m>1; The foreign object intrusion prediction module is used to input the template image of the current scene and the image to be inspected in the current scene into the foreign object intrusion prediction model to obtain the foreign object feature map of the current scene; the foreign object feature map is used to determine whether there is a different target between the image to be inspected and the template image; the foreign object intrusion prediction model is built based on Siamese network and UNet segmentation network; The filtering module is used to determine the location of the discrepancy target when there is a discrepancy between the target image to be inspected and the template image of the current scene. The module obtains the target location and uses a moving object filtering model to determine whether there is a discrepancy target between the target location of each frame in the target image set of the current scene and the template image of the current scene. If the number of frames with discrepancies between the Nth and N+mth railway images of the current scene is greater than a set value, it is determined that there is a foreign object in the discrepancy target of the target image to be inspected in the current scene. The module also identifies the areas where the position of the discrepancy target changes from the Nth to N+mth railway images of the current scene as moving objects and filters out the moving objects in the discrepancy target of the target image to be inspected in the current scene to obtain the real foreign objects in the target image to be inspected in the current scene.

9. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the railway foreign object intrusion detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the railway foreign object intrusion detection method as described in any one of claims 1 to 7.

Images