SAM-based track foreign matter intrusion detection method, system and equipment and medium

Abstract

The invention discloses an SAM-based track foreign matter intrusion detection method, system and device and a medium. The method comprises the following steps: acquiring video stream data shot by a track field camera and corresponding alarm area contour coordinates; decoding the video stream data into an image frame set, segmenting each frame of image through an optimized SAM model to obtain a mask image, and filtering a mask of a non-alarm area according to the contour coordinates of the alarm area; performing overlapping degree analysis on two adjacent frames of mask images, and filtering inherent semantic information of the track scene; performing similarity analysis on the filtered masks, and screening validity information; and extracting hidden danger information, determining a foreign matter invasion event through multi-frame sequence reliability analysis, and outputting an alarm. The segmentation model based on the optimized SAM carries out automatic segmentation in a point collection mode, the problem that foreign matter invasion categories cannot be exhaustive is solved, the capturing capacity of foreign matter invasion events is improved, the generalization capacity of the model is further improved, and therefore the line operation safety of the track is guaranteed.

Description

Technical Field

[0001] This application relates to the technical field of rail transit safety monitoring, and particularly to a method, system, device and medium for detecting rail foreign object intrusion based on SAM. Background Art

[0002] With the rapid development of rail transit, rail safety monitoring has become an important link to ensure the safe operation of rail transit. Rail foreign object intrusion detection is an important part of the railway safety guarantee system, mainly used to detect foreign objects on railway tracks to prevent foreign objects from damaging trains and railway facilities.

[0003] Currently, traditional rail intrusion detection mainly relies on manual monitoring and object detection algorithms based on deep learning, which have problems such as high false alarm rates, poor real-time performance, and high data dependence. And it is difficult to enumerate all foreign object intrusion events in the rail scenario. Existing closed-set object detection algorithms require a large amount of data for training to obtain a detection algorithm for a specific category. Open-set object detection algorithms cannot meet the performance requirements due to their complex model structures and large parameter magnitudes. Therefore, developing an efficient and accurate rail foreign object intrusion detection algorithm is of great significance for improving rail transit safety. Summary of the Invention

[0004] This application provides a method, system, device and medium for detecting rail foreign object intrusion based on SAM to solve the above problems.

[0005] On the one hand, this application provides a method for detecting rail foreign object intrusion based on SAM, and the method includes the following steps:

[0006] Step S1: Obtain the video stream data captured by the on-site camera of the rail and the corresponding contour coordinates of the warning area;

[0007] Step S2: Decode the video stream data into a set of image frames, segment each frame of the image through an optimized SAM model to obtain a mask image, and filter the masks of non-warning areas according to the contour coordinates of the warning area;

[0008] Step S3: Analyze the overlap degree of adjacent two-frame mask images to filter out the inherent semantic information of the rail scenario;

[0009] Step S4: Analyze the similarity of the filtered masks to screen out valid information;

[0010] Step S5: Extract potential hazard information from the valid information, confirm the foreign object intrusion event through multi-frame sequence reliability analysis, and output an alarm.

[0011] In an implementation manner of this application, the step S2 specifically includes the following steps:

[0012] Step S21: Construct a training sample set in the form of point prompts using orbital scene data;

[0013] Step S22: The image encoder adopts the Tiny-Vit structure, and replaces some Transformer modules with CNN convolutional blocks;

[0014] Step S23: Introduce a local attention mechanism to replace the global self-attention mechanism;

[0015] Step S24: Reduce the number of attention nesting times and the number of channels in the decoding stage to compress model parameters;

[0016] Step S25: Freeze the image encoder and only fine-tune the mask decoder to complete model transfer training;

[0017] Step S26: Generate prompt points in the form of uniform sampling across the entire image and input them into the model to output a segmentation mask.

[0018] In an implementation manner of the present application, during the model fine-tuning in step S25, an orbital scene data set is adopted, and during the training process, the parameters of the image encoder are frozen, and only the parameters of the mask decoder are updated.

[0019] In an implementation manner of the present application, the overlap analysis in step S3 includes:

[0020] Step S31: Extract the external contour coordinates of two adjacent frame mask images, and calculate the intersection over union (IOU) of the masks at the same position;

[0021] Step S32: If the IOU is greater than the first threshold, it is determined as the same mask and filtered; if the IOU is less than the first threshold, it is retained as a suspected hazard or a mask in the adjacent position.

[0022] In an implementation manner of the present application, the similarity analysis in step S4 includes:

[0023] Step S41: For the suspected hazard mask, extract the local area at the same position and measure it using the structural similarity index (SSIM);

[0024] Step S42: If the SSIM is less than the second threshold, it is determined as a suspected hazard; if the SSIM is greater than the second threshold, it is determined as interference and filtered.

[0025] In an implementation manner of the present application, the multi-frame sequence reliability analysis in step S5 includes:

[0026] Step S51: Extract the suspected hazard information of three consecutive frames;

[0027] Step S52: Generate the minimum bounding rectangle for the hidden danger contour and calculate the intersection-over-union ratio of the rectangular regions in adjacent frames;

[0028] Step S53: If the intersection-over-union ratio is greater than the third threshold for two consecutive times, it is determined as a foreign object intrusion event; otherwise, it is ignored.

[0029] In an implementation manner of the present application, the intersection-over-union ratio calculation of the minimum bounding rectangle in step S52 adopts a coordinate alignment method to exclude the error caused by object deformation.

[0030] The present application also provides a track foreign object intrusion detection system based on SAM, and the system includes:

[0031] A coordinate acquisition unit, configured to acquire the video stream data captured by the track site camera and the corresponding warning area contour coordinates;

[0032] A mask generation unit, configured to decode the video stream data into a set of image frames, segment each frame of the image through an optimized SAM model to obtain a mask image, and filter the masks of non-warning areas according to the warning area contour coordinates;

[0033] A mask analysis unit, configured to perform an overlap analysis on the mask images of two adjacent frames and filter the inherent semantic information of the track scene;

[0034] A similarity analysis unit, configured to perform a similarity analysis on the filtered masks to screen out effective information;

[0035] An alarm unit, configured to extract hidden danger information from the effective information, confirm a foreign object intrusion event through multi-frame sequence reliability analysis, and output an alarm.

[0036] The present application also provides a track foreign object intrusion detection device based on SAM, and the device includes:

[0037] At least one processor; and,

[0038] A memory communicatively connected to the at least one processor; wherein,

[0039] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to complete the foregoing track foreign object intrusion detection method based on SAM.

[0040] The present application also provides a non-volatile computer storage medium for track foreign object intrusion detection based on SAM, storing computer-executable instructions, and the computer-executable instructions are executed by a processor to be used to implement the foregoing track foreign object intrusion detection method based on SAM.

[0041] A method, system, device and medium for detecting foreign object intrusion on tracks based on SAM provided by this application adjust the structure size of the image encoder, making the overall structure more lightweight and improving the edge inference ability. The segmentation model based on the optimized SAM performs automatic segmentation in a point-sampling manner, solves the problem that the categories of foreign object intrusion cannot be exhausted, and improves the capture ability of foreign object intrusion events. The mask difference processing based on SAM further improves the generalization ability of the model, thus ensuring the safe operation of the track line. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of this application and form a part of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation to this application. In the drawings:

[0043] Figure 1 It is a flowchart of a method for detecting foreign object intrusion on tracks based on SAM provided by an embodiment of this application;

[0044] Figure 2 It is a schematic diagram of scenarios at different time points provided by an embodiment of this application;

[0045] Figure 3 It is a schematic diagram of the cropping of the warning area provided by an embodiment of this application;

[0046] Figure 4 It is a schematic diagram of interference filtering on the track surface provided by an embodiment of this application;

[0047] Figure 5 It is a schematic diagram of identifying potential hazards provided by an embodiment of this application;

[0048] Figure 6 It is a composition diagram of a system for detecting foreign object intrusion on tracks based on SAM provided by an embodiment of this application;

[0049] Figure 7 It is a schematic diagram of a device for detecting foreign object intrusion on tracks based on SAM provided by an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0050] To make the objectives, technical solutions and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments of this application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts shall fall within the scope of protection of this application.

[0051] The embodiments of the present application provide a method, system, device, and medium for detecting foreign object intrusion on tracks based on SAM. The technical solutions proposed in the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0052] Figure 1 It is a flowchart of a method for detecting foreign object intrusion on tracks based on SAM provided by the embodiments of the present application. As Figure 1 shown, the method mainly includes the following steps:

[0053] Step S1: Obtain the video stream data captured by the on-site camera of the track and the corresponding contour coordinates of the warning area;

[0054] Step S2: Decode the video stream data into a set of image frames, segment each frame of the image through an optimized SAM model to obtain a mask image, and filter the masks of non-warning areas according to the contour coordinates of the warning area;

[0055] Step S3: Analyze the overlap degree of the mask images of two adjacent frames to filter out the inherent semantic information of the track scene;

[0056] Step S4: Analyze the similarity of the filtered masks to screen out effective information;

[0057] Step S5: Extract potential hazard information from the effective information, confirm the foreign object intrusion event through multi-frame sequence reliability analysis, and output an alarm.

[0058] In the embodiments of the present application, first, step S2 specifically includes: obtaining the video stream data V_data captured by the on-site camera device of the track and the corresponding contour coordinates P(X, Y) of the warning area at this point; the warning area coordinates are actually [(30, 125), (92, 122), (35, 980), (95, 988)].

[0059] As Figure 2 shown, the adjacent frame images in V_data show that the track surface in the detection scene is covered with crushed stones, which belongs to a field scene and is extremely vulnerable to the influence of crushed stones on the track surface and the disturbance of trees and grass outside the track;

[0060] Furthermore, decode the video stream data V_data according to S1 into a set of image frames I_data, segment each frame of the image through an optimized SAM model to obtain a mask image, and filter the mask information according to the contour coordinates P(X, Y) of the warning area in S1, and retain the masks within the warning area, as Figure 3 shown;

[0061] Furthermore, analyze the overlap degree of the mask images I1_seg and I2_seg of the previous and next frames to filter out the inherent semantic information under the track scene, as Figure 4 shown;

[0062] Further, similarity analysis is performed on the filtered mask image to obtain validity information;

[0063] Further, potential hazard information is screened out from the validity information, reliability analysis of multiple-frame sequences is carried out, and finally, according to the potential hazard coordinates bbox(41, 140, 61, 160), an alarm is output, as Figure 5 shown.

[0064] Specifically: The track scene data captured on-site is used as training images, and a training sample set is constructed by using points as prompts in the form of points; then, the Tiny-Vit structure is selected for the image encoder, and CNN convolutional blocks are introduced to replace some Transformer modules; secondly, in order to improve the calculation efficiency, a local attention mechanism is introduced to replace the global self-attention mechanism; thirdly, the number of attention nesting times and the number of channels in the decoding stage are reduced to reduce the scale of model parameters; then, based on the track scene dataset, the SAM model is fine-tuned, the image encoder is frozen, and only the mask decoder is fine-tuned, so as to improve the transfer ability of few samples in the business scenario; further, an optimized SAM model is obtained through model fine-tuning, a set of points is obtained by uniformly sampling points across the entire image as prompts and input into the model, and the segmentation mask output by the model is obtained; further, the segmentation mask output by the model is filtered in the local area, and the mask information outside the area is filtered through the alarm area information to prevent interference.

[0065] Mask overlap analysis specifically includes the following detailed steps: First, through the mask information of two mask images, the external contour coordinates are extracted and the intersection over union (IOU) of the mask areas at the same position is calculated;

[0066]

[0067] Area(A∩B) represents the area of the intersection region of two masks at the same position, and Area(A∪B) represents the area of the union region of two masks at the same position.

[0068] Further, an IOU logical judgment is made. If it is greater than the first threshold (the first threshold can specifically be taken as 0.7 here, and the embodiments of the present application do not make specific limitations on the specific value), it is considered to be the same mask in the scene, and this mask is removed, so as to remove the influence of unclear boundaries caused by problems such as environmental light interference and local shaking caused by wind; if it is less than 0.7, it is considered a suspected potential hazard or a mask at a neighboring position.

[0069] The process of mask image similarity analysis specifically includes the following detailed steps:

[0070] First, perform similarity analysis on suspected potential hazards or adjacent position masks. Select local regions at the same position in two images through mask information, and use SSIM for similarity measurement;

[0071] Secondly, if the similarity is less than the second threshold (the second threshold can specifically be taken as 0.3 here, and the embodiments of the present application do not specifically limit the specific value to 0.3), it is considered a suspected potential hazard. If the similarity is greater than 0.3, it is considered an adjacent position mask, and the interference information caused by segmenting the track surface crushed stones is removed.

[0072] In step S5, the process of screening potential hazard information includes the following detailed steps:

[0073] S51: Extract the suspected potential hazard information of consecutive frames, and perform reliability analysis on the sequence data of consecutive three frames with potential hazard information;

[0074] S52: Use the three-frame sequence data in S51 to circumscribe the minimum rectangle of the potential hazard contour, and calculate the intersection over union (IoU) of adjacent frames of the three rectangular frames;

[0075] S53: Given that the consecutive three frames of image frames containing potential hazards are F t , F t+1 , F t+2 , calculate IOU(F t , F t+1 ). If it is greater than the third threshold (the third threshold can specifically be taken as 0.5 here, and the embodiments of the present application do not specifically limit the specific value to 0.5), then start to calculate IOU(F t+1 , F t+2 ). If it is still greater than 0.5, it is considered a potential hazard, which is a foreign object intrusion event. Otherwise, it is still a sudden pixel change caused by environmental interference or noise problems, and the potential hazard discrimination result is ignored.

[0076] The above is a method for detecting foreign object intrusion on tracks based on SAM provided by the embodiments of the present application. Based on the same inventive concept, the embodiments of the present application also provide a system for detecting foreign object intrusion on tracks based on SAM. Figure 6 It is a composition diagram of a system for detecting foreign object intrusion on tracks based on SAM provided by the embodiments of the present application, as shown in Figure 6As shown in the figure, the system mainly includes: a coordinate acquisition unit 601, which is used to acquire the video stream data captured by the on-site camera of the track and the corresponding contour coordinates of the warning area; a mask generation unit 602, which is used to decode the video stream data into a set of image frames, segment each frame of the image through an optimized SAM model to obtain a mask image, and filter the masks of non-warning areas according to the contour coordinates of the warning area; a mask analysis unit 603, which is used to analyze the overlap degree of the mask images of two adjacent frames and filter the inherent semantic information of the track scene; a similarity analysis unit 604, which performs similarity analysis on the filtered masks to screen out valid information; and an alarm unit 605, which is used to extract potential hazard information from the valid information, confirm the foreign object intrusion event through multi-frame sequence reliability analysis, and output an alarm.

[0077] The above is a SAM-based track foreign object intrusion detection system provided by an embodiment of the present application. Based on the same inventive concept, an embodiment of the present application also provides a SAM-based track foreign object intrusion detection device. Figure 7 The following is a schematic diagram of a SAM-based track foreign object intrusion detection device provided by an embodiment of the present application. As Figure 7 shown, the device mainly includes: at least one processor 701; and a memory 702 communicatively connected to the at least one processor; wherein, the memory 702 stores instructions executable by the at least one processor 701, and the instructions are executed by the at least one processor 701 to enable the at least one processor 701 to complete the aforementioned SAM-based track foreign object intrusion detection method.

[0078] In addition, an embodiment of the present application also provides a non-volatile computer storage medium for SAM-based track foreign object intrusion detection, storing computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the aforementioned SAM-based track foreign object intrusion detection method.

[0079] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, and the combination of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0080] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device that implements the functions specified in one or more of the procedures Figure 1 in one or more of the procedures and / or blocks Figure 1 specified in one or more of the blocks or multiple blocks.

[0081] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, such that a series of operational steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one or more of the procedures Figure 1 in one or more of the procedures and / or blocks Figure 1 specified in one or more of the blocks or multiple blocks.

[0082] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0083] Each embodiment in this application is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and reference can be made to the partial description of the method embodiments for the relevant parts.

[0084] It should also be noted that the term "comprising", "including" or any other variation thereof is intended to cover a non-exclusive inclusion, such that a process, method, commodity or device including a series of elements not only includes those elements but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, commodity or device including the element.

[0085] The above are only the embodiments of the present application and are not used to limit the present application. For those skilled in the art, various modifications and changes can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A method for detecting intrusion of foreign objects on tracks based on SAM, characterized in that, The method includes the following steps: Step S1: Obtain the video stream data captured by the on-site camera of the track and the corresponding contour coordinates of the warning area; Step S2: Decode the video stream data into a set of image frames, segment each frame of the image through the optimized SAM model to obtain a mask image, and filter the masks of non-warning areas according to the contour coordinates of the warning area; Step S3: Analyze the overlap degree between two adjacent frames of mask images to filter out the inherent semantic information of the track scene; Step S4: Analyze the similarity of the filtered masks to screen out valid information; Step S5: Extract potential hazard information from the valid information, confirm the foreign object intrusion event through multi-frame sequence reliability analysis, and output an alarm.

2. The method for detecting intrusion of foreign objects on tracks based on SAM according to claim 1, wherein, The specific steps of step S2 include the following steps: Step S21: Construct a training sample set in the form of point prompts using track scene data; Step S22: The image encoder adopts the Tiny-Vit structure, and replaces some Transformer modules with CNN convolutional blocks; Step S23: Introduce a local attention mechanism to replace the global self-attention mechanism; Step S24: Reduce the number of attention nesting times and the number of channels in the decoding stage to compress the model parameters; Step S25: Freeze the image encoder and fine-tune the mask decoder to complete model transfer training; Step S26: Generate prompt points in a uniform sampling manner across the entire image and input them into the model to output a segmentation mask.

3. The method for detecting intrusion of foreign objects on a track based on SAM according to claim 2, characterized in that, During the model fine-tuning in step S25, a track scene dataset is used, and during the training process, the parameters of the image encoder are frozen, and only the parameters of the mask decoder are updated.

4. A method for detecting foreign object intrusion on a track based on SAM according to claim 1, characterized in that, The overlap degree analysis in step S3 includes: Step S31: Extract the external contour coordinates of two adjacent frames of mask images, and calculate the intersection over union (IOU) of the masks at the same position; Step S32: If the IOU is greater than the first threshold, it is determined as the same mask and filtered; if the IOU is less than the first threshold, it is retained as a suspected hazard or a mask of a neighboring position.

5. The method for detecting intrusion of foreign objects on a track based on SAM according to claim 1, characterized in that, The similarity analysis in step S4 includes: Step S41: For a suspected hazard mask, extract the local area at the same position and measure it using the structural similarity index (SSIM); Step S42: If the SSIM is less than the second threshold, it is determined as a suspected hazard; if the SSIM is greater than the second threshold, it is determined as interference and filtered.

6. The method for detecting intrusion of foreign objects on tracks based on SAM according to claim 1, characterized in that, The multi-frame sequence reliability analysis in step S5 includes: Step S51: Extract the suspected hazard information of three consecutive frames; Step S52: Generate the minimum bounding rectangle for the hazard contour and calculate the intersection over union of the rectangular areas of adjacent frames; Step S53: If the intersection over union is greater than the third threshold for two consecutive times, it is determined as a foreign object intrusion event, otherwise it is ignored.

7. The method for detecting rail foreign object intrusion based on SAM according to claim 6, wherein, In step S52, the intersection over union calculation of the minimum bounding rectangle adopts a coordinate alignment method to exclude errors caused by object deformation.

8. An SAM-based orbital foreign object intrusion detection system, characterized in that, The system includes: A coordinate acquisition unit for obtaining the video stream data captured by the on-site camera of the track and the corresponding contour coordinates of the warning area; A mask generation unit for decoding the video stream data into a set of image frames, segmenting each frame of the image through the optimized SAM model to obtain a mask image, and filtering the masks of non-warning areas according to the contour coordinates of the warning area; A mask analysis unit for analyzing the overlap degree of mask images of two adjacent frames and filtering the inherent semantic information of the track scene; A similarity analysis unit for analyzing the similarity of the filtered mask and screening valid information; An alarm unit for extracting potential hazard information from the valid information, confirming a foreign object intrusion event through multi-frame sequence reliability analysis, and outputting an alarm.

9. An SAM-based track foreign object intrusion detection device, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can complete a method for detecting track foreign object intrusion based on SAM according to any one of claims 1-7.

10. A non-volatile computer storage medium for detecting intrusion of foreign objects on an orbit based on SAM, storing computer-executable instructions, characterized in that, The computer-executable instructions are executed by the processor to implement a method for detecting track foreign object intrusion based on SAM according to any one of claims 1-7.

Images