Multi-mode full-category railway obstacle intrusion detection method

Abstract

The invention provides a multi-mode full-category railway obstacle intrusion detection method. The method comprises the following steps: acquiring video images of each road section of a rail transit perimeter, decoding and analyzing the video images, splitting the video images into continuous video frames, and transmitting the video frames to a feature extraction module; the feature extraction module performs feature extraction on the video frame by adopting a Rail-IDET algorithm to obtain coordinate information of a detected object in the video frame; drawing and prompting the coordinates of the detected object in the original video frame to obtain detected video frames, synthesizing the detected video frames into a new video stream frame by frame, and outputting the new video stream; and taking the coordinate of the detected object displayed in the new video stream as the coordinate of the invading object of the rail transit. According to the multi-mode full-category railway obstacle intrusion detection method provided by the invention, the image-text understanding capability of a large model is applied to the field, the problem that the existing railway obstacle detection algorithm is incomplete in target detection is solved, and meanwhile, the forgetting problem of continuous learning of large-scale data is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of rail transit safety status identification, and in particular to a multi-modal all-category railway obstacle intrusion detection method. Background Art

[0002] High-speed railways are a top priority for transportation in all countries. The "Opinions on Strengthening Safety and Environmental Governance Along Railways" issued in 2021 proposed that we should make full use of video monitoring, artificial intelligence and other means to improve the ability to detect and manage hidden dangers of safety and environmental problems along railways. In order to better ensure the efficiency and safety of high-speed rail transportation, countries have begun to study auxiliary intelligent driving systems to help the reliability of high-speed railway operations. In the research of assisted driving systems, the advance perception technology of obstacles is an indispensable part. However, the existing intelligent driving systems based on deep learning rely on large-scale data and cannot face the complex and changeable railway environment. And radar-based research has never been able to break through the limitations of perception distance. Therefore, it is necessary to further study visual perception technology to enhance the perception of obstacles by railway assisted driving systems.

[0003] Railway visual obstacle detection is mainly divided into obstacle detection within the dataset and obstacle detection outside the dataset. Category detection within the dataset is trained and learned based on labeled information and depends on the quality of the dataset. Therefore, this method has high detection accuracy and speed in general railway scenarios (such as pedestrian intrusion, animal intrusion, etc.). However, due to the limitation of the dataset, only the learning categories within the dataset can be detected, and most of the categories outside the dataset are classified as known categories. For railway intelligent driving scenarios, the obstacles are complex and changeable, and all objects in the real world have the possibility of intrusion. It is a very difficult task to detect all categories of obstacles, and the model is relatively complex.

[0004] Difficulties such as the above are pain points in the railway industry, especially in the detection of categories outside the data set, which have always been difficult to solve and effectively prevent.

[0005] In response to the above pain points, many scholars have done a lot of research on vision. In terms of visual obstacle detection, some scholars have proposed a lightweight adaptive multi-scale obstacle detection network for small target obstacle detection in complex railway scenes. This network improves the feature extraction and feature fusion modules of the YOLOv3 network to improve the detection accuracy of small target obstacles. In terms of speed, grouped asymmetric convolution is used to reduce model complexity and parameters while ensuring accuracy. Other scholars use YOLOv5 combined with track feature extraction network for multi-task fusion obstacle detection. First, the track line detection algorithm is used to extract the track and determine the dangerous area; secondly, YOLOv5 is used for obstacle detection, and the time risk level is determined based on the spatial state and historical severity prior information; a complete set of railway obstacle detection algorithms is implemented. Some scholars have also proposed a traffic obstacle detection algorithm based on deep learning - DFF-Net. For small-scale obstacles, a two-stage differential feature fusion neural network detection algorithm combining the prior detection module and the target detection module is proposed. By using pyramid feature maps to detect objects of different scales and using differential feature fusion design to construct robust features for object detection, this method effectively balances accuracy and real-time performance.

[0006] The above-mentioned disadvantages of the visual obstacle detection method in the prior art include: being limited by the inherent categories of the data set, it cannot meet the requirements of the existing intelligent assisted driving system for the complex and changeable environment of the railway.

[0007] Therefore, how to solve the data set limitations and build a full-category detection model for railway scenarios remains a key issue to be resolved. Summary of the invention

[0008] The present invention provides a multi-modal all-category railway obstacle intrusion detection method, which applies the powerful image and text understanding ability of a large model to the rail transit scenario, solving the pain point problem of being unable to detect categories outside the data set in existing railway scenarios.

[0009] In order to achieve the above object, the present invention adopts the following technical scheme.

[0010] A multi-modal all-category railway obstacle intrusion detection method, comprising: Obtain video images of each section of the rail transit perimeter, decode and analyze the video images, decompose the video images into continuous video frames, and transmit the video frames to the feature extraction module; The feature extraction module uses the Rail-IDET algorithm to extract features from the video frame to obtain coordinate information of the detected object in the video frame; The coordinates of the detected object are drawn in the original video frame to obtain a detected video frame, and each detected video frame is synthesized frame by frame to output a new video stream; The coordinates of the detected object displayed in the new video stream are used as the coordinates of the intruding object in the rail transit.

[0011] Preferably, the image feature extraction module includes a convolution module, a CBKD module and an RKVP module, the CBKD module is established based on C2f, the RKVP module includes a graphic feature fusion module T-CBKD based on the CBKD module, and the convolution module, CBKD module and RKVP module, the TransFormer text encoder part of Clip, the YOLOv8 detection head part and the graphic-text joint loss function constitute the Rail-IDET algorithm.

[0012] Preferably, the feature extraction module uses the Rail-IDET algorithm to extract features from the video frame to obtain the coordinate information of the detected object in the video frame, including: Set reminder phrases and files; Extract key text information from the input video frame using a text encoder of the Clip model according to the prompt phrase and the file; The input video frame is preliminarily extracted using convolution, and then the extracted features are preliminarily processed by the CBKD feature extraction module to obtain the image features of the video frame; The key text information and the image features are input into the RKVP image-text fusion module, which analyzes the relationship between the image features and the text features, embeds the text into the image features, classifies and identifies the features through the detection head, and outputs the coordinate information of the detected object.

[0013] It can be seen from the technical solutions provided by the above-mentioned embodiments of the present invention that the multimodal all-category railway obstacle intrusion detection method proposed in the present invention applies the graphic and text understanding ability of the large model to the field, solves the problem of incomplete detection targets of the existing railway obstacle detection algorithm, and solves the forgetting problem of continuous learning of large-scale data.

[0014] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0016] Figure 1 A scene diagram of an accident caused by an intrusion obstacle provided by an embodiment of the present invention; Figure 2 A processing flow chart of a multi-modal all-category railway obstacle intrusion detection method provided by an embodiment of the present invention; Figure 3 An overall framework diagram of the Rail-IDET algorithm of the present invention provided in an embodiment of the present invention; Figure 4 A CBKD module framework diagram provided for an embodiment of the present invention; Figure 5 A framework diagram of an RKVP module provided in an embodiment of the present invention; Figure 6 Schematic diagram of zero-sample detection under different confidence levels of Rail-IDET provided in an embodiment of the present invention Figure 7 A comparison chart of Rail-IDET, YOLOv8, YOLOx, and RTMDET detection results provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0018] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0019] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the field to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0020] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention. Example

[0021] Figure 1 The present invention provides a scene diagram of an accident caused by an intrusion obstacle. The present invention provides a processing flow of a multi-modal all-category railway obstacle intrusion detection method. Figure 2 As shown, the following steps are included: Step S1, obtaining video images of each section of the rail transit perimeter, and extracting basic information such as the time and monitored geographical location of the video images.

[0022] Step S2, decoding and analyzing the video image, decomposing the video image into continuous video frames, and transmitting the video frames to the feature extraction module.

[0023] Step S3: The feature extraction module uses the Rail-IDET algorithm to extract features from the video frame and records the coordinates of the detected object according to the detection result. Figure 3 As shown in the figure, the CBKD (Dual bottleneck of cross stage fusion with KAN network, cross-stage fusion module with KAN network) module framework is as follows Figure 4 As shown in the figure, the RKVP (Robust KAN-based Visual Language PAN) module framework is as follows Figure 5As shown in the figure. The Rail-IDET (Rail Incremental Learning Detection Model) algorithm consists of a CBKD feature extraction module based on C2f (Cross Stage Partial Fusion Bottleneck with 2 Convolutions, a cross-stage fusion module with two convolutions), an RKVP image-text feature fusion module developed based on the CBKD module, a TransFormer text encoder part of Clip (Contrastive Language-Image Pre-training, the name of a large visual language model), a YOLOv8 detection head part, and a joint image-text loss function. For the input image, convolution is used to initially extract features, and then the extracted features are preliminarily processed by the CBKD feature extraction module. Then, according to the prompt phrase, the TransFormer text encoder part and the image features are input into the RKVP image-text fusion module to enhance the fusion of the image features and embed the text into the image features. Finally, the features are classified and recognized by the detection head.

[0024] The specific steps of step S3 should include the following steps: Step 3.1, set the prompt phrase and file. For example, the prompt phrase is "Please pay attention to people and animals crossing the railway", and the file can be the provided warning prompt file (Railway regulations: No animals are allowed on the railway), in the format of docx, doc.

[0025] Step 3.2, according to the above prompt phrases and files, use the text encoder of the Clip model to extract the key text information in the input video frame. The text encoder of the Clip model is part of the existing large image and text model Clip. With the help of the trained Clip, the text can be processed more effectively, and then the processed text information is embedded in the RKVP module built in this paper to establish an association with the image. The text encoder of the Clip model is an important component of the Rail-IDET algorithm.

[0026] Step 3.3, for the input video frame, use the image feature extraction module to extract the image features and output the image feature map. The above-mentioned image feature extraction module includes a convolution module, a CBKD module and a RKVP module; it is an important component of the Rail-IDET algorithm.

[0027] Step 3.4, analyze the above key text information and image feature information and output the coordinates of the detected object.

[0028] Analyze the relationship between image features and text features, and output the coordinates of the object in the image corresponding to the text features. For example, if the text feature is pedestrian crossing, then the coordinates of the pedestrian are in the corresponding image.

[0029] Step 4: Draw hints in the original video frame based on the coordinates of the detected object provided in step 3, and synthesize the detected video frames frame by frame into a new video stream for output.

[0030] Step 5: Perform subsequent decision processing on the intrusion object of rail transit according to the coordinates of the detected object displayed in the new video stream. The above-mentioned detected object is the intrusion object. Example

[0031] Taking a certain rail transit perimeter scene as an example, railway intrusion obstacle detection is performed, video images of each section of the rail transit perimeter are obtained, and the monitoring video is detected using the method proposed in the present invention. In the present invention, we provide a detection embodiment of the Rail-IDET algorithm; It is worth noting that the Clip text extraction part in the network architecture of this embodiment can be replaced by other large image and text model related structures.

[0032] In daily rail transit perimeter scenes, the images captured by the camera are acquired, encoded and transmitted to the dynamic monitoring algorithm for real-time detection, and the detection results are marked.

[0033] First, before detection, the file content of the relevant road section or the name of the obstacle that needs attention is input into the Rail-IDET algorithm for prompting. Secondly, the Rail-IDET algorithm is installed in the edge box or server connected to the camera. It is worth noting that the Rail-IDET algorithm we provide requires less running resources and is suitable for most equipment on site.

[0034] Here you can choose whether to use multi-threaded parallel detection according to the size of the video memory. Multi-threaded parallel detection can load and process the model in blocks, which is faster than serial step-by-step loading.

[0035] Finally, after the detection is completed, the video frames are synthesized frame by frame into a new video output and displayed on the display screen, indicating the intrusion category and the degree of danger.

[0036] Figure 6 A schematic diagram of zero-sample detection under different confidence levels of Rail-IDET provided in an embodiment of the present invention. Figure 7 A comparison chart of Rail-IDET, YOLOv8, YOLOX, and RTMDET detection results provided by an embodiment of the present invention. Figure 7YOLOv8, YOLOX, RTMDET, etc. are all existing cutting-edge algorithms, which are used to compare with the Rail-IDET algorithm to prove the performance superiority of Rail-IDET.

[0037] Table 1 shows the accuracy comparison results of a multimodal all-category railway obstacle intrusion detection method of the present invention and other algorithms on the VOC2007+2012 dataset (using public datasets to demonstrate the superiority of the algorithm of the present invention). AP is the average precision, IOU** represents the intersection-over-union ratio of the regression box and the verification truth box, and AR is the average recall.

[0038] Table 1 Accuracy comparison of methods in VOC dataset Table 1 Accuracy comparison of methods in VOC dataset

[0039]

[0040] Table 2 shows the performance of a multimodal all-category railway obstacle intrusion detection method of the present invention and other algorithms on the EXDark dataset, demonstrating that the Rail-IDET algorithm of the present invention can still guarantee high detection accuracy in harsh environments.

[0041] Table 2 Accuracy comparison of methods in EXDark dataset

[0042] Table 3 shows the performance of a multi-modal all-category railway obstacle intrusion detection method of the present invention compared with other algorithms in a railway dataset, proving that the present invention can be well used in railway applications.

[0043] Table 3 Accuracy comparison of the methods in the railway dataset

[0044]

[0045] Table 4 shows the detection speed of a multi-modal all-category railway obstacle intrusion detection method of the present invention, which proves that the method proposed by the present invention has wide applicability.

[0046] Table 4 Comparison of method parameters and calculation amount

[0047]

[0048] In actual railway operation applications, processing and decision-making can be carried out based on the detection results of the Rail-IDET algorithm, and the pixel area size can be calculated based on the detection results. The distance of the target relative to the camera can be converted according to the ratio of actual and image pixels, and the position of the intruding target can be quickly located based on the detection perception.

[0049] In summary, the embodiments of the present invention aim at the problem that rail transit intrusion obstacles cannot be fully perceived at present, and combine with the actual application scenarios of rail transit obstacle intrusion detection to propose a multimodal all-category railway obstacle intrusion detection method; the graphic and text understanding ability of the large model is applied to the site, and improvements are made for the problem of large-scale data learning, which solves the problem of incomplete detection targets of existing railway obstacle detection algorithms, and at the same time solves the forgetting problem of continuous learning of large-scale data.

[0050] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0051] It can be known from the above description of the implementation mode that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0052] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0053] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multi-modal all-category railway obstacle intrusion detection method, characterized in that: include: Obtain video images of each section of the rail transit perimeter, decode and analyze the video images, decompose the video images into continuous video frames, and transmit the video frames to the feature extraction module; The feature extraction module uses the Rail-IDET algorithm to extract features from the video frame to obtain coordinate information of the detected object in the video frame; The coordinates of the detected object are drawn in the original video frame to obtain a detected video frame, and each detected video frame is synthesized frame by frame to output a new video stream; The coordinates of the detected object displayed in the new video stream are used as the coordinates of the intruding object in the rail transit.

2. The method according to claim 1, characterized in that The image feature extraction module includes a convolution module, a CBKD module and an RKVP module. The CBKD module is established based on C2f. The RKVP module includes a graphic feature fusion module T-CBKD based on the CBKD module. The convolution module, the CBKD module and the RKVP module, the TransFormer text encoder part of Clip, the YOLOv8 detection head part and the graphic-text joint loss function constitute the Rail-IDET algorithm.

3. The method according to claim 2, characterized in that The feature extraction module uses the Rail-IDET algorithm to extract features from the video frame to obtain the coordinate information of the detected object in the video frame, including: Set reminder phrases and files; Extract key text information from the input video frame using a text encoder of the Clip model according to the prompt phrase and the file; The input video frame is preliminarily extracted using convolution, and then the extracted features are preliminarily processed by the CBKD feature extraction module to obtain the image features of the video frame; The key text information and the image features are input into the RKVP image-text fusion module, which analyzes the relationship between the image features and the text features, embeds the text into the image features, classifies and identifies the features through the detection head, and outputs the coordinate information of the detected object.

Images