Railway foreign object intrusion detection method and system, device and medium

Abstract

The present invention relates to the field of railway foreign object detection. Disclosed are a railway foreign object intrusion detection method and system, a device and a medium. The method comprises: acquiring a first key railway image frame of the current scene as a template image, and an N-th railway image frame thereof as an image under test; inputting the template image and image under test of the current scene into a foreign object intrusion prediction model to obtain a foreign object feature map of the current scene, so as to determine whether there is a different object between the image under test and template image of the current scene; and, if there is a different object, using a moving object filtering model to filter out a moving object from the different object in the image under test of the current scene, so as to obtain a real foreign object in the image under test of the current scene. The present invention can improve the reliability of railway foreign object intrusion detection.

Description

Railway foreign body intrusion detection method, system, equipment and medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on January 2, 2024, with application number 2024100031658 and invention name “A method, system, equipment and medium for detecting foreign body intrusion in railways”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to the field of railway foreign object detection, and in particular to a railway foreign object intrusion detection method, system, equipment and medium. Background Art

[0003] Foreign object intrusion is the most serious threat to the external environment facing railway operations. Traditional foreign object detection is done manually. Since foreign object intrusion incidents are sudden, unpredictable and irregular, if timely warnings are not given when facing sudden foreign object intrusions (such as falling rocks and landslides), serious consequences can easily occur.

[0004] Image recognition is one of the main research directions in the field of artificial intelligence. Traditional image detection methods often use the differential method, that is, comparing the differences between the captured image and the template image in real time. However, this method has a single technical means, is easily affected by light and camera angle, has poor reliability, and is difficult to adapt to complex outdoor scenes.

[0005] Summary of the Invention

[0006] Based on this, embodiments of the present invention provide a railway foreign object intrusion detection method, system, device and medium to improve the reliability of railway foreign object intrusion detection.

[0007] To achieve the above objectives, the present invention provides the following solutions:

[0008] A railway foreign body intrusion detection method, comprising:

[0009] Obtain a template image, a to-be-tested image, and a set of to-be-tested images of the current scene; the template image is the first keyframe railway image; the to-be-tested image is the Nth frame railway image; the set of to-be-tested images includes: the N+1th frame railway image to the N+mth frame railway image; wherein N>1, m>1;

[0010] Input the template image of the current scene and the image to be inspected of the current scene into the foreign object intrusion prediction model to obtain the foreign object feature map of the current scene; the foreign object feature map is used to determine whether there are different targets between the image to be inspected and the template image; the foreign object intrusion prediction model is constructed based on the twin network and the UNet segmentation network;

[0011] If there is a difference target between the image to be inspected of the current scene and the template image of the current scene, the position of the difference target is determined to obtain the target position, and a moving object filtering model is used to determine whether there is a difference target between the target position of each frame image in the image to be inspected set of the current scene and the template image of the current scene. If the number of image frames with difference targets from the Nth frame railway image to the N+mth frame railway image of the current scene is greater than a set value, it is determined that there is a foreign object in the difference target of the image to be inspected of the current scene, and the area where the position changes in the difference target from the Nth frame railway image to the N+mth frame railway image of the current scene is determined as a moving object. The moving object in the difference target of the image to be inspected of the current scene is filtered out to obtain the real foreign object in the image to be inspected of the current scene.

[0012] Optionally, a moving object filtering model is used to determine whether there is a target difference between the target position of each frame image in the set of images to be tested of the current scene and the template image of the current scene, specifically including:

[0013] The moving object filtering model uses the IOU matching method to determine whether there is a different target between the target position of each frame image in the test image set of the current scene and the template image of the current scene.

[0014] Optionally, the method for determining the foreign object intrusion prediction model specifically includes:

[0015] Acquire training data; the training data includes: template images of different training scenes, images to be inspected, and corresponding label data; the label data includes whether there are different targets between the template images of the training scenes and the images to be inspected;

[0016] Constructing a fusion network model that integrates the twin network and the UNet segmentation network; the fusion network model includes: a first encoder, a second encoder, a feature fusion module, and a decoder; the first encoder and the second encoder have the same structure and are both connected to the feature fusion module; the feature fusion module is connected to the decoder;

[0017] The template images of different training scenes in the training data are used as the input of the first encoder, and the images to be inspected of different training scenes in the training data are used as the input of the second encoder. The fusion network model is trained with the goal of minimizing the loss function, and the trained fusion network model is determined as the foreign object intrusion prediction model.

[0018] Optionally, the first encoder includes: a first image division layer, a first convolution layer, a second convolution layer, a third convolution layer, a fourth convolution layer and a fifth convolution layer connected in sequence;

[0019] The first image segmentation layer is used to traverse the input template image in a sliding window manner to obtain multiple small template images; the first convolution layer is used to perform a convolution operation on each of the small template images to obtain a first feature map of each of the small template images; the second convolution layer is used to perform a convolution and downsampling pooling operation on the first feature map to obtain a second feature map; the third convolution layer is used to perform a convolution and downsampling pooling operation on the second feature map to obtain a third feature map; the fourth convolution layer is used to perform a convolution and downsampling pooling operation on the third feature map to obtain a fourth feature map; the fifth convolution layer is used to perform a convolution and downsampling pooling operation on the fourth feature map to obtain a fifth feature map;

[0020] The second encoder includes: a second image division layer, a sixth convolution layer, a seventh convolution layer, an eighth convolution layer, a ninth convolution layer and a tenth convolution layer;

[0021] The second image segmentation layer is used to traverse the input image to be inspected in a sliding window manner to obtain a plurality of small-block images to be inspected; the sixth convolutional layer is used to perform a convolution operation on each of the small-block images to be inspected to obtain a sixth feature map of each of the small-block images to be inspected; the seventh convolutional layer is used to perform a convolution and downsampling pooling operation on the sixth feature map to obtain a seventh feature map; the eighth convolutional layer is used to perform a convolution and downsampling pooling operation on the seventh feature map to obtain an eighth feature map; the ninth convolutional layer is used to perform a convolution and downsampling pooling operation on the eighth feature map to obtain a ninth feature map; the tenth convolutional layer is used to perform a convolution and downsampling pooling operation on the ninth feature map to obtain a tenth feature map;

[0022] The feature fusion module includes: a splicing layer and a spatial pyramid pooling layer connected in sequence; the splicing layer is used to splice the fifth feature map and the tenth feature map to obtain a spliced ​​feature; the spatial pyramid pooling layer is used to perform a spatial pyramid pooling operation on the spliced ​​feature to obtain a fused feature;

[0023] The decoder includes: a first upsampling layer, a second upsampling layer, a third upsampling layer, a fourth upsampling layer, and a fifth upsampling layer connected in sequence; the first upsampling layer is connected to the spatial pyramid pooling layer; the input of the second upsampling layer is connected to the output of the fourth convolutional layer and the ninth convolutional layer; the input of the third upsampling layer is connected to the output of the third convolutional layer and the eighth convolutional layer; the input of the fourth upsampling layer is connected to the output of the second convolutional layer and the seventh convolutional layer; the input of the fifth upsampling layer is connected to the output of the first convolutional layer and the sixth convolutional layer;

[0024] The first upsampling layer is used to perform an upsampling operation on the fusion feature to obtain a first decoding feature; the second upsampling layer is used to splice the first decoding feature, the fourth feature map and the ninth feature map to obtain a second decoding feature; the third upsampling layer is used to splice the second decoding feature, the third feature map and the eighth feature map to obtain a third decoding feature; the fourth upsampling layer is used to splice the third decoding feature, the second feature map and the seventh feature map to obtain a fourth decoding feature; the fifth upsampling layer is used to splice the fourth decoding feature, the first feature map and the sixth feature map to obtain a foreign body feature map.

[0025] Optionally, the template image of the current scene and the image to be inspected of the current scene are input into the foreign object intrusion prediction model to obtain the foreign object feature image of the current scene, specifically including:

[0026] Performing image enhancement processing on the template image of the current scene and the image to be inspected of the current scene respectively to obtain the template enhanced image of the current scene and the enhanced image to be inspected of the current scene;

[0027] The template enhancement image of the current scene and the to-be-detected enhancement image of the current scene are input into the foreign object intrusion prediction model to obtain the foreign object feature image of the current scene.

[0028] Optionally, performing image enhancement processing on the template image of the current scene and the image to be inspected of the current scene respectively to obtain the template enhanced image of the current scene and the enhanced image to be inspected of the current scene specifically includes:

[0029] Perform image correction and image noise processing on the template image of the current scene in sequence to obtain a template enhanced image of the current scene;

[0030] The image to be inspected of the current scene is subjected to image correction and image noise processing in sequence to obtain an enhanced image to be inspected of the current scene.

[0031] Optionally, m=5; the set value is 4.

[0032] The present invention also provides a railway foreign object intrusion detection system, comprising:

[0033] An image acquisition module is configured to acquire a template image, a to-be-tested image, and a set of to-be-tested images of a current scene; the template image is the first keyframe railway image; the to-be-tested image is the Nth frame railway image; the set of to-be-tested images includes: the N+1th frame railway image to the N+mth frame railway image; wherein N>1, m>1;

[0034] The foreign object intrusion prediction module is used to input the template image of the current scene and the image to be inspected of the current scene into the foreign object intrusion prediction model to obtain the foreign object feature map of the current scene; the foreign object feature map is used to determine whether there are different targets between the image to be inspected and the template image; the foreign object intrusion prediction model is constructed based on the twin network and the UNet segmentation network;

[0035] The filtering module is used to determine the position of the difference target when there is a difference target between the image to be inspected in the current scene and the template image of the current scene, obtain the target position, and use the moving object filtering model to determine whether there is a difference target between the target position of each frame image in the set of images to be inspected in the current scene and the template image of the current scene. If the number of image frames with difference targets from the Nth frame railway image to the N+mth frame railway image of the current scene is greater than a set value, it is determined that there is a foreign object in the difference target of the image to be inspected in the current scene, and the area with a changed position in the difference target from the Nth frame railway image to the N+mth frame railway image of the current scene is determined as a moving object. The moving object in the difference target of the image to be inspected in the current scene is filtered out to obtain the real foreign object in the image to be inspected in the current scene.

[0036] The present invention also provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the above-mentioned railway foreign object intrusion detection method.

[0037] The present invention also provides a computer-readable storage medium storing a computer program, which implements the above-mentioned railway foreign object intrusion detection method when executed by a processor.

[0038] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0039] The embodiment of the present invention constructs a foreign object intrusion prediction model based on the twin network and the Unet segmentation network to realize the recognition of the foreign object feature map and determine whether there are different targets between the target scene's to-be-inspected map and the template map; a moving object filtering model is used to filter out the moving objects in the different targets of the target scene's to-be-inspected map to obtain the real foreign objects in the target scene's to-be-inspected map, which can quickly and accurately identify new foreign objects, improve the reliability of railway foreign object intrusion detection, and provide more specific, scientific and efficient monitoring means for railway construction, operation and maintenance, disaster prevention and mitigation, etc., which can improve the safety level of railway operations and has great practical application value.

[0040] Figures in the specification

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] FIG1 is a flow chart of a railway foreign object intrusion detection method according to an embodiment of the present invention;

[0043] FIG2 is a structural diagram of a fusion network model provided by an embodiment of the present invention;

[0044] FIG3 is a diagram showing railway foreign body intrusion prediction results according to an embodiment of the present invention;

[0045] FIG4 is a structural diagram of a railway foreign object intrusion detection system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0047] In recent years, deep learning technology has made great progress in the field of artificial intelligence. In railway foreign object intrusion inspection work, image recognition can be used to train models based on image data collected on-site, thereby extracting key information from the image data to identify foreign objects and provide support for railway operations.

[0048] The purpose of the present invention is to provide a railway foreign object intrusion detection method, system, equipment and medium, which realize railway foreign object intrusion detection based on twin network and Unet segmentation network, thereby improving the reliability of railway foreign object intrusion detection.

[0049] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] Example 1

[0051] 1 , the railway foreign object intrusion detection method of this embodiment includes:

[0052] Step 101: Obtain the template image, the image to be inspected, and the image set to be tested of the current scene.

[0053] The template image is the first key frame railway image; the image to be tested is the Nth frame railway image; the set of images to be tested includes: the N+1th frame railway image to the N+mth frame railway image, where N>1, m>1. For example, m=5.

[0054] Step 102: Input the template image of the current scene and the image to be inspected of the current scene into the foreign object intrusion prediction model to obtain the foreign object feature map of the current scene; the foreign object feature map is used to determine whether there is a difference target between the image to be inspected and the template image; the foreign object intrusion prediction model is constructed based on the twin network and the UNet segmentation network.

[0055] Step 102 specifically includes:

[0056] (1) Perform image enhancement processing on the template image of the current scene and the image to be inspected of the current scene respectively to obtain the template enhanced image of the current scene and the enhanced image to be inspected of the current scene. Specifically:

[0057] The template image of the current scene is subjected to image correction and image noise processing in sequence to obtain the template enhanced image of the current scene; the image to be inspected of the current scene is subjected to image correction and image noise processing in sequence to obtain the enhanced image to be inspected of the current scene.

[0058] (2) The template enhancement image of the current scene and the to-be-detected enhancement image of the current scene are input into the foreign object intrusion prediction model to obtain the foreign object feature image of the current scene.

[0059] The method for determining the foreign body intrusion prediction model specifically includes:

[0060] ① Obtain training data; the training data includes: template images of different training scenes, images to be inspected and corresponding label data; the label data includes whether there are different targets between the template images of the training scenes and the images to be inspected.

[0061] ② Construct a fusion network model that integrates the twin network and the UNet segmentation network; the fusion network model includes: a first encoder, a second encoder, a feature fusion module and a decoder; the first encoder and the second encoder have the same structure and are both connected to the feature fusion module; the feature fusion module is connected to the decoder.

[0062] The structure of the fusion network model is described in detail below with reference to FIG2 .

[0063] Referring to Figure 2, the first encoder includes: a first image division layer (not shown in the figure), a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer connected in sequence. The first image division layer is used to traverse the input template image in a sliding window manner to obtain multiple small-block template images; the first convolutional layer is used to perform a convolution operation on each of the small-block template images to obtain a first feature map F1 for each of the small-block template images; the second convolutional layer is used to perform convolution and downsampling pooling operations on the first feature map F1 to obtain a second feature map F2; the third convolutional layer is used to perform convolution and downsampling pooling operations on the second feature map F2 to obtain a third feature map F3; the fourth convolutional layer is used to perform convolution and downsampling pooling operations on the third feature map F3 to obtain a fourth feature map F4; the fifth convolutional layer is used to perform convolution and downsampling pooling operations on the fourth feature map F4 to obtain a fifth feature map F5.

[0064] The second encoder includes: a second image division layer, a sixth convolutional layer, a seventh convolutional layer, an eighth convolutional layer, a ninth convolutional layer, and a tenth convolutional layer. The second image division layer is used to traverse the input image to be inspected in a sliding window manner to obtain multiple small-block images to be inspected; the sixth convolutional layer is used to perform a convolution operation on each of the small-block images to be inspected to obtain a sixth feature map F1′ of each of the small-block images to be inspected; the seventh convolutional layer is used to perform a convolution and downsampling pooling operation on the sixth feature map F1′ to obtain a seventh feature map F2′; the eighth convolutional layer is used to perform a convolution and downsampling pooling operation on the seventh feature map F2′ to obtain an eighth feature map F3′; the ninth convolutional layer is used to perform a convolution and downsampling pooling operation on the eighth feature map F3′ to obtain a ninth feature map F4′; the tenth convolutional layer is used to perform a convolution and downsampling pooling operation on the ninth feature map F4′ to obtain a tenth feature map F5′.

[0065] The feature fusion module includes: a splicing layer and a spatial pyramid pooling layer connected in sequence; the splicing layer is used to splice the fifth feature map F5 and the tenth feature map F5′ to obtain a spliced ​​feature; the spatial pyramid pooling layer is used to perform a spatial pyramid pooling operation on the spliced ​​feature to obtain a fused feature.

[0066] The decoder includes: a first upsampling layer, a second upsampling layer, a third upsampling layer, a fourth upsampling layer, and a fifth upsampling layer connected in sequence; the first upsampling layer is connected to the spatial pyramid pooling layer; the input of the second upsampling layer is connected to the output of the fourth convolutional layer and the ninth convolutional layer; the input of the third upsampling layer is connected to the output of the third convolutional layer and the eighth convolutional layer; the input of the fourth upsampling layer is connected to the output of the second convolutional layer and the seventh convolutional layer; and the input of the fifth upsampling layer is connected to the output of the first convolutional layer and the sixth convolutional layer.

[0067] The first upsampling layer is used to perform an upsampling operation on the fusion feature to obtain a first decoding feature P1; the second upsampling layer is used to splice the first decoding feature P1, the fourth feature map F4 and the ninth feature map F4′ to obtain a second decoding feature P2; the third upsampling layer is used to splice the second decoding feature P2, the third feature map F3 and the eighth feature map F3′ to obtain a third decoding feature P3; the fourth upsampling layer is used to splice the third decoding feature P3, the second feature map F2 and the seventh feature map F2′ to obtain a fourth decoding feature P4; the fifth upsampling layer is used to splice the fourth decoding feature P4, the first feature map F1 and the sixth feature map F1′ to obtain a foreign body feature map P5.

[0068] ③ Use the template images of different training scenes in the training data as the input of the first encoder, and use the images to be inspected of different training scenes in the training data as the input of the second encoder, train the fusion network model with the goal of minimizing the loss function, and determine the trained fusion network model as the foreign object intrusion prediction model.

[0069] Step 103: If there is a difference target between the image to be tested of the current scene and the template image of the current scene, the position of the difference target is determined, the target position is obtained, and a moving object filtering model is used to determine whether there is a difference target between the target position of each frame image in the image set to be tested of the current scene and the template image of the current scene.

[0070] Specifically, the moving object filtering model uses an IOU matching method to determine whether there is a different target between the target position of each frame image in the set of images to be tested of the current scene and the template image of the current scene.

[0071] Step 104: If the number of image frames containing a difference target between the Nth and N+mth railway images of the current scene is greater than a set value, a foreign object is determined to be present in the difference target of the image to be inspected in the current scene. Regions of the difference target whose position has changed between the Nth and N+mth railway images of the current scene are identified as moving objects. Moving objects are then filtered out of the difference target of the image to be inspected in the current scene, thereby obtaining the true foreign object in the image to be inspected in the current scene. For example, the set value may be 4.

[0072] The following is a specific example from a practical application to further illustrate the railway foreign object intrusion detection method described above. The implementation process of this example is as follows: acquiring keyframe images to establish a template image, acquiring a real-time image as the image to be inspected; image enhancement; extracting features from the template image and the image to be inspected; segmenting object contours in the template image and the image to be inspected; establishing a foreign object intrusion prediction model, comparing the differences between the template image and the image to be inspected; filtering moving objects; and predicting discrepant objects. Each step is described in detail below.

[0073] S1. Obtain key frame images to establish a template image, and obtain real-time images as images to be inspected: obtain camera video stream data, extract the first key frame image as a template image, and then continue to capture real-time images as images to be inspected.

[0074] Specifically, by pulling the video stream of the network camera, the first key frame image is obtained as the template image. The video stream can support the RTSP protocol, RTMP protocol, and support H.264 and H.265 encoding formats.

[0075] A fixed interval time for capturing key frames in the video stream is set, and frame images in the real-time video stream are captured regularly as the images to be inspected. The images to be inspected are captured later than the template images, and the order is distinguished by timestamps.

[0076] S2. Image enhancement: This includes image correction, noise processing and other enhancement processes, so that the image can truly reflect the shape of the object and remain consistent in coordinate position.

[0077] Image correction: Image distortion caused by the camera's mounting angle and the sensor's own imaging can interfere with target recognition, necessitating image correction. Geometric correction is employed, based on the cause of the distortion and the relationship between spatial position changes. A correction model is used to transform pixel coordinates between the distorted image and the reference image. Correction is performed using calculation formulas and obtained auxiliary parameters.

[0078] Noise Processing: During the digitization and transmission process, images are often affected by noise from the imaging device and the external environment, leading to interference in the image. This eliminates image tilt errors and projection errors caused by terrain undulations. Spatial domain methods are used to perform data operations directly on the original image, processing the pixel grayscale values. Using bilateral filtering, the weights for calculating the spatial proximity of each pixel to the center point are optimized. These weights are optimized to the product of the weights for the spatial proximity calculation and the weights for the pixel value similarity calculation. These optimized weights are then convolved with the image to achieve edge-preserving denoising.

[0079] S3: Extract the foreign body feature maps of the template image and the image to be inspected. First, the processed template image and the image to be inspected obtained in step S2 are used as the input of the fusion network model. The template image and the image to be inspected are traversed through the sliding window method to obtain multiple regions. Still referring to Figure 2, a 3*3 convolution operation is performed on each region to obtain the initial feature maps F1 and F1′. Then, based on the feature extraction concept of UNet, F1 and F1′ are respectively subjected to four convolutions and downsampling pooling, and finally the output feature F5 of the template image and the output feature F5′ of the image to be inspected are obtained. All channels of the output features F5 and F5′ are spliced ​​and spatial pyramid pooling is performed to obtain P1 to expand the receptive field of the feature map. Finally, based on the feature fusion concept of UNet, P1 is upsampled and spliced ​​with F4 and F4′ for feature channels to obtain P2; P2 is upsampled and spliced ​​with F3 and F3′ for feature channels to obtain P3... and so on, the foreign body feature map P5 of the template image and the image to be inspected can be obtained.

[0080] A sliding window method is used on the original image, traversing from left to right and from top to bottom in a 100×100 size window, dividing the entire image into several parts according to a set method and rules. The setting method refers to the step size and window size of the sliding window. Here, the window size is set to 100×100 and the step size is set to 50×50.

[0081] It's important to note that each convolution is accompanied by an activation function layer and batch normalization. Nonlinear activation functions can improve the network's nonlinear learning capabilities. Batch normalization can accelerate convergence during model training, making the training process more stable and preventing exploding or vanishing gradients.

[0082] S4: Train the fusion network model to establish a foreign body intrusion prediction model and make predictions. Use contrast loss as the loss function of the network for training, specifically:

[0083] Consider the feature similarity distance D between the template image X1 of the training scene and the image to be tested X2 of the training scene W, P represents the feature dimension, and Y is the label data indicating whether the two samples (i.e., X1 and X2) match (if Y = 1, it means the two samples are similar or matched, i.e., there is no difference between the template image and the image to be tested; if Y = 0, it means the two samples are not similar or matched, i.e., there is a difference between the template image and the image to be tested). Assuming M is the set threshold, N is the number of samples, and W is the network weight during training, the contrast loss L(W, (Y, X1, X2)) can be defined as:

[0084] Among them, the feature similarity distance D W Using cosine similarity distance, it is expressed as follows:

[0085] It should be noted that the Represents the feature matrix of template graph X1, Represents the feature matrix of the image to be inspected, X2. The core idea is to perform a cosine similarity metric on the feature matrix, learning the similarity between the two inputs by reducing the loss. Ultimately, the trained network weights and structure are used as the foreign object intrusion prediction model.

[0086] During the prediction process, it is only necessary to input the target scene's image to be inspected and the template image into the foreign object intrusion prediction model to obtain the foreign object segmentation map. If the foreign object result map exists in the image to be inspected but not in the template map, it is considered a foreign object and the foreign object is labeled. Its labels are not limited to foreign objects, but also include trains and pedestrians. When trains and pedestrians appear, the system will filter them out by default. The segmentation results include the coordinate information, pixel information and area information of all object contours. It should be noted that the objects are not distinguished according to the type of object, but are distinguished according to the obtained object contour and the image area framed by the object contour in the image to be inspected.

[0087] S5, moving object filtering model: Classify and train the acquired moving targets to obtain a classification model, and perform moving object recognition on the difference targets predicted by S4.

[0088] Based on the image to be inspected with foreign objects obtained in step S4, 6 consecutive images to be inspected with extracted features are compared, and the foreign objects in each image to be inspected are tracked. If there are 4 or more frames of the image to be inspected with foreign object information, it is identified as a foreign object; if there are less than 3 frames, it is not identified as a foreign object.

[0089] After being identified as a foreign object, the target's position information and pixel information are compared and tracked. Targets with obvious position changes are identified as moving objects and filtered.

[0090] The system stores the detection results of the past six frames in a result queue. When a foreign object is detected in the first frame, it checks the next five frames to see if a foreign object appears in the same location. If a foreign object appears in four of the six frames, it is considered a foreign object. Otherwise, it is considered a moving object, such as a fleeting flying insect or domesticated cattle or sheep. This also helps suppress camera flicker. The method used to determine whether a foreign object is in the same location is IOU matching. If the IOU value of the foreign object in the previous and next frames is greater than a preset threshold, it is considered the same object.

[0091] S6. Prediction of railway foreign body intrusion: Based on the classification result obtained in step S5, the moving objects are filtered to achieve foreign body prediction.

[0092] Based on the railway foreign object intrusion prediction model developed in the above steps, by real-time access to the camera video stream and real-time interception of key frames for comparison and analysis, the intrusion prediction of foreign objects in the inspected railway section is achieved. Figure 3 shows the results of the railway foreign object intrusion prediction, where part (a) of Figure 3 is the template image, part (b) of Figure 3 is the image to be inspected, and part (c) of Figure 3 is the predicted result image.

[0093] This embodiment uses the acquired real-time images of railway sections, undergoes image enhancement processing such as image correction and noise processing, and uses sliding window technology to extract the image features of the template image and the real-time image to be inspected. The target is segmented based on the features, and a foreign object intrusion prediction model is established based on the template image features. The model compares the features of the image to be inspected, obtains the difference targets, and filters the moving objects, thereby realizing the railway foreign object intrusion prediction.

[0094] This embodiment has the following advantages:

[0095] (1) It can predict the intrusion of foreign objects on railways without the need to collect all types of foreign objects in advance to train the ability to identify them. It can quickly and accurately identify new foreign objects with a high iteration frequency.

[0096] (2) The application cost is low, the coverage is wide, and the reliability is high. It can provide more specific, scientific and efficient monitoring means for railway construction, operation and maintenance, disaster prevention and mitigation, etc., and can improve the safety level of railway operations. It has great practical application value.

[0097] Example 2

[0098] In order to execute the method corresponding to the above-mentioned embodiment 1 and achieve the corresponding functions and technical effects, a railway foreign object intrusion detection system is provided below.

[0099] Referring to FIG4 , the system includes:

[0100] The image acquisition module 201 is used to acquire a template image, a to-be-tested image, and a set of to-be-tested images for the current scene; the template image is the first keyframe railway image; the to-be-tested image is the Nth frame railway image; the set of to-be-tested images includes: the N+1th frame railway image to the N+mth frame railway image; where N>1, m>1.

[0101] The foreign object intrusion prediction module 202 is used to input the template image of the current scene and the image to be inspected of the current scene into the foreign object intrusion prediction model to obtain the foreign object feature map of the current scene; the foreign object feature map is used to determine whether there is a difference target between the image to be inspected and the template image; the foreign object intrusion prediction model is constructed based on the twin network and the UNet segmentation network.

[0102] The filtering module 203 is used to determine the position of the difference target when a difference target exists between the image to be inspected in the current scene and the template image of the current scene, obtain the target position, and use the moving object filtering model to determine whether the target position of each frame image in the set of images to be inspected in the current scene and the template image of the current scene have a difference target. If the number of image frames with difference targets from the Nth frame railway image to the N+mth frame railway image of the current scene is greater than a set value, it is determined that there is a foreign object in the difference target of the image to be inspected in the current scene, and the area with a changed position in the difference target from the Nth frame railway image to the N+mth frame railway image of the current scene is determined as a moving object. The moving object in the difference target of the image to be inspected in the current scene is filtered out to obtain the true foreign object in the image to be inspected in the current scene.

[0103] Example 3

[0104] This embodiment provides an electronic device, including a memory and a processor. The memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the railway foreign object intrusion detection method of embodiment 1.

[0105] Optionally, the above-mentioned electronic device may be a server.

[0106] In addition, an embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the railway foreign object intrusion detection method of embodiment 1 is implemented.

[0107] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0108] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A method for detecting intrusion of foreign objects into railway lines, characterized in that, Including: Obtain the template image, the image to be inspected, and the set of images to be measured of the current scene; The template image is the first-frame key-frame railway image; The image to be inspected is the Nth-frame railway image; The set of images to be measured includes: the (N + 1)th-frame railway image to the (N + m)th-frame railway image; where N > 1 and m > 1; Input the template image of the current scene and the image to be inspected of the current scene into the foreign object intrusion prediction model to obtain the foreign object feature map of the current scene; the foreign object feature map is used to determine whether there is a difference target between the image to be inspected and the template image; the foreign object intrusion prediction model is constructed based on the siamese network and the UNet segmentation network; When there is a difference target between the image to be inspected and the template image of the current scene, determine the position of the difference target to obtain the target position, and use the moving object filtering model to determine whether there is a difference target between the target position of each frame of image in the set of images to be measured of the current scene and the template image of the current scene. If the number of image frames with difference targets in the (N)th-frame railway image to the (N + m)th-frame railway image of the current scene is greater than the set value, determine that there is a foreign object in the difference target of the image to be inspected of the current scene, and determine the area where the position changes in the difference target of the (N)th-frame railway image to the (N + m)th-frame railway image of the current scene as the moving object, and filter out the moving object in the difference target of the image to be inspected of the current scene to obtain the real foreign object in the image to be inspected of the current scene.

2. The railway foreign object intrusion detection method according to claim 1, characterized in that, Using the moving object filtering model to determine whether there is a difference target between the target position of each frame of image in the set of images to be measured of the current scene and the template image of the current scene, specifically including: The moving object filtering model uses the IOU matching method to determine whether there is a difference target between the target position of each frame of image in the set of images to be measured of the current scene and the template image of the current scene.

3. The railway foreign object intrusion detection method according to claim 1, characterized in that, The determination method of the foreign object intrusion prediction model specifically includes: Obtain training data; the training data includes: template images, images to be inspected, and corresponding label data of different training scenes; the label data includes whether there is a difference target between the template image and the image to be inspected of the training scene; Construct a fusion network model integrating the siamese network and the UNet segmentation network; the fusion network model includes: a first encoder, a second encoder, a feature fusion module, and a decoder; the structures of the first encoder and the second encoder are the same and are both connected to the feature fusion module; The feature fusion module is connected to the decoder; Use the template images of different training scenes in the training data as the input of the first encoder, use the images to be inspected of different training scenes in the training data as the input of the second encoder, train the fusion network model with the goal of minimizing the loss function, and determine the trained fusion network model as the foreign object intrusion prediction model.

4. The railway foreign object intrusion detection method according to claim 3, characterized in that, The first encoder includes: a first image division layer, a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer connected in sequence; The first image partitioning layer is used to traverse the input template image in a sliding window manner to obtain a plurality of small template images; the first convolutional layer is used to perform a convolutional operation on each of the small template images to obtain a first feature map of each of the small template images; the second convolutional layer is used to perform a convolutional and downsampling pooling operation on the first feature map to obtain a second feature map; the third convolutional layer is used to perform a convolutional and downsampling pooling operation on the second feature map to obtain a third feature map; the fourth convolutional layer is used to perform a convolutional and downsampling pooling operation on the third feature map to obtain a fourth feature map; the fifth convolutional layer is used to perform a convolutional and downsampling pooling operation on the fourth feature map to obtain a fifth feature map; The second encoder includes: a second image partitioning layer, a sixth convolutional layer, a seventh convolutional layer, an eighth convolutional layer, a ninth convolutional layer, and a tenth convolutional layer; The second image partitioning layer is used to traverse the input image to be inspected in a sliding window manner to obtain a plurality of small images to be inspected; the sixth convolutional layer is used to perform a convolutional operation on each of the small images to be inspected to obtain a sixth feature map of each of the small images to be inspected; the seventh convolutional layer is used to perform a convolutional and downsampling pooling operation on the sixth feature map to obtain a seventh feature map; the eighth convolutional layer is used to perform a convolutional and downsampling pooling operation on the seventh feature map to obtain an eighth feature map; the ninth convolutional layer is used to perform a convolutional and downsampling pooling operation on the eighth feature map to obtain a ninth feature map; the tenth convolutional layer is used to perform a convolutional and downsampling pooling operation on the ninth feature map to obtain a tenth feature map; The feature fusion module includes: a splicing layer and a spatial pyramid pooling layer connected in sequence; the splicing layer is used to splice the fifth feature map and the tenth feature map to obtain a spliced feature; the spatial pyramid pooling layer is used to perform a spatial pyramid pooling operation on the spliced feature to obtain a fused feature; The decoder includes: a first upsampling layer, a second upsampling layer, a third upsampling layer, a fourth upsampling layer, and a fifth upsampling layer connected in sequence; the first upsampling layer is connected to the spatial pyramid pooling layer; the input of the second upsampling layer is connected to the output of the fourth convolutional layer and the output of the ninth convolutional layer; the input of the third upsampling layer is connected to the output of the third convolutional layer and the output of the eighth convolutional layer; the input of the fourth upsampling layer is connected to the output of the second convolutional layer and the output of the seventh convolutional layer; the input of the fifth upsampling layer is connected to the output of the first convolutional layer and the output of the sixth convolutional layer; The first upsampling layer is used to perform an upsampling operation on the fused feature to obtain a first decoded feature; the second upsampling layer is used to splice the first decoded feature, the fourth feature map, and the ninth feature map to obtain a second decoded feature; the third upsampling layer is used to splice the second decoded feature, the third feature map, and the eighth feature map to obtain a third decoded feature; the fourth upsampling layer is used to splice the third decoded feature, the second feature map, and the seventh feature map to obtain a fourth decoded feature; the fifth upsampling layer is used to splice the fourth decoded feature, the first feature map, and the sixth feature map to obtain a foreign object feature map.

5. The railway foreign object intrusion detection method according to claim 1, wherein Inputting the template image of the current scene and the image to be inspected in the current scene into the foreign object intrusion prediction model to obtain the foreign object feature map of the current scene specifically includes: Performing image enhancement processing on the template image of the current scene and the image to be inspected in the current scene respectively to obtain the template enhanced image of the current scene and the to-be-inspected enhanced image of the current scene; Inputting the template enhanced image of the current scene and the to-be-inspected enhanced image of the current scene into the foreign object intrusion prediction model to obtain the foreign object feature map of the current scene.

6. The railway foreign object intrusion detection method according to claim 5, wherein Performing image enhancement processing on the template image of the current scene and the image to be inspected in the current scene respectively to obtain the template enhanced image of the current scene and the to-be-inspected enhanced image of the current scene specifically includes: Performing image correction and image noise processing on the template image of the current scene in sequence to obtain the template enhanced image of the current scene; Performing image correction and image noise processing on the to-be-inspected image of the current scene in sequence to obtain the to-be-inspected enhanced image of the current scene.

7. The railway foreign object intrusion detection method according to claim 1, characterized in that m = 5; the set value is 4.

8. A railway foreign object intrusion detection system, characterized in that Including: An image acquisition module for acquiring the template image, the image to be inspected, and the set of images to be measured in the current scene; The template image is the first key-frame railway image; The image to be inspected is the Nth frame railway image; The set of images to be measured includes: the (N + 1)th frame railway image to the (N + m)th frame railway image; where N > 1 and m > 1; A foreign object intrusion prediction module for inputting the template image of the current scene and the image to be inspected in the current scene into the foreign object intrusion prediction model to obtain the foreign object feature map of the current scene; the foreign object feature map is used to determine whether there is a difference target between the image to be inspected and the template image; the foreign object intrusion prediction model is constructed based on a siamese network and a UNet segmentation network; A filtering module for determining the position of the difference target if there is a difference target between the image to be inspected in the current scene and the template image of the current scene to obtain the target position, and using a moving object filtering model to determine whether there is a difference target between the target position of each frame of image in the set of images to be measured in the current scene and the template image of the current scene. If the number of image frames with difference targets in the (N)th frame railway image to the (N + m)th frame railway image in the current scene is greater than the set value, it is determined that there is a foreign object in the difference target of the image to be inspected in the current scene, and the area where the position of the difference target in the (N)th frame railway image to the (N + m)th frame railway image in the current scene changes is determined as a moving object, and the moving object in the difference target of the image to be inspected in the current scene is filtered out to obtain the real foreign object in the image to be inspected in the current scene.

9. An electronic device, characterized in that, It includes a memory and a processor. The memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the railway foreign object intrusion detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, and when the computer program is executed by a processor, it realizes the railway foreign object intrusion detection method according to any one of claims 1 to 7.

Images