Railway foreign matter intrusion monitoring system and monitoring method based on AI agent

Abstract

The application discloses a railway foreign matter invasion monitoring system and a monitoring method based on an AI intelligent agent, wherein the monitoring system comprises a perception device, a visual perception module, a context management module, a cognitive decision module, an alarm linkage module and a database module; the visual perception module is connected with the perception device; the context management module is connected with the visual perception module, the cognitive decision module, the alarm linkage module and the database module; the context management module constructs and maintains a context state object based on a model context protocol; the cognitive decision module jointly constructs prompt words from a detection result output by the visual perception module and a context state provided by the context management module, inputs the prompt words into a pre-trained large language model for semantic understanding and risk judgment, and outputs a structured analysis text. The application realizes multi-model cooperation by introducing a model context protocol, performs semantic reasoning by using a large language model, and significantly improves the intelligent level and practical value of the system.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and railway safety monitoring technology, specifically to a railway foreign object intrusion monitoring system and method based on AI intelligent agents, which can be used to achieve intelligent, semantic-level risk identification and decision support in key sections of high-speed railways, conventional railways, subways and other rail lines (such as level crossings, platform edges, tunnel entrances, and high slope areas). Background Technology

[0002] With the rapid development of my country's rail transit network, operational safety issues have become increasingly prominent. Foreign object intrusion (such as pedestrians entering the tracks, animals crossing, falling rocks, and floating debris) is a significant hidden danger that could trigger emergency braking or even derailment accidents. Traditional video surveillance systems rely on manual patrols, which suffer from problems such as delayed response, fatigue-related misjudgments, and difficulties in nighttime identification.

[0003] In recent years, deep learning-based object detection algorithms (such as the YOLO series) have been widely used in railway foreign object detection, enabling automatic identification of abnormal objects in images. For example, the invention patent CN120526379B discloses a real-time track foreign object detection method based on YOLOv5, including: synchronous video vibration acquisition, spatial offset compensation parameter generation, dynamic feature enhancement processing, feature weight heatmap generation, hierarchical perception feature aggregation, cross-scale target verification, foreign object positioning command encapsulation, and multi-level early warning execution control. Although this existing technology improves the detection accuracy, the detection system used by this technology is still at the "pixel-level recognition" level, lacking the ability to understand the context of events, making it difficult to distinguish between construction workers and unauthorized intruders, or between a briefly passing plastic bag and a persistent obstacle, resulting in a high false alarm rate and affecting the practical application effect.

[0004] Meanwhile, while Large Language Models (LLMs) have demonstrated powerful capabilities in natural language understanding, reasoning, and generation, their application in industrial vision scenarios is still in the exploratory stage. Current technologies have not yet proposed a solution that combines visual perception results with environmental context and uses a large model for semantic analysis and processing suggestions. Furthermore, the multi-model collaboration mechanism is weak, and there is a lack of a unified context management framework to organize detection data, historical states, and external information, limiting the system's intelligence level.

[0005] Therefore, there is an urgent need for a new type of railway foreign object monitoring system with a closed-loop capability of "perception-cognition-decision" to improve the accuracy, interpretability and practicality of detection. Summary of the Invention

[0006] This invention aims to overcome the shortcomings of existing technologies and provide a railway foreign object intrusion monitoring system based on AI intelligent agents. By introducing a model context protocol to achieve multi-model collaboration and utilizing a large language model for semantic reasoning, the system's intelligence level and practical value are significantly improved.

[0007] To achieve the above objectives, this invention provides a railway foreign object intrusion monitoring system based on an AI intelligent agent, comprising a sensing device, a visual perception module, a context management module, a cognitive decision-making module, an alarm linkage module, and a database module. The visual perception module acquires video streams or image frames of the railway monitoring area and uses a target detection model to identify foreign objects in the image, outputting detection results including category, confidence level, bounding box, and timestamp. The context management module is connected to the visual perception module, the cognitive decision-making module, the alarm linkage module, and the database module. The context management module constructs and maintains context state objects based on a model context protocol. The cognitive decision-making module constructs prompt words by combining the detection results output by the visual perception module with the context state provided by the context management module, inputting them into a pre-trained large language model for semantic understanding and risk assessment, outputting structured analysis text. The alarm linkage module triggers corresponding response actions based on the risk level output by the cognitive decision-making module. The database module persistently stores the detection events of the context management module.

[0008] Furthermore, the context state includes location, time, weather, train operation plan, recent detection events and historical alarm counts, and supports time-series memory and cross-camera context sharing.

[0009] Furthermore, the cognitive decision-making module uses prompt word templates including location, current time, weather conditions, train operation plan, and recent detection list, and guides the large language model to output analysis conclusions in a predetermined format.

[0010] Furthermore, the cognitive decision-making module uses prompt word templates including location, current time, weather conditions, train operation plan, and recent detection list, and guides the large language model to output analysis conclusions in a predetermined format.

[0011] Furthermore, after performing semantic reasoning on the input prompts, the large language model generates structured analysis text containing judgments, levels, reasons, and suggestions.

[0012] Furthermore, the response actions include local audible and visual warnings, pushing alarm information to the dispatch center, automatically generating work orders, and linking the signal system to limit speed.

[0013] Furthermore, the database module uses SQLite or Redis database, with its effective time window set to 5-15 minutes, for statistical analysis of historical occurrence frequency and evolution trends.

[0014] This invention also provides a railway foreign object intrusion monitoring method based on AI intelligent agents, which uses the above-mentioned railway foreign object intrusion monitoring system and includes the following steps:

[0015] S1. The sensing device acquires image frames of the railway area in real time. The visual perception module extracts the image from the sensing device and determines whether there is any foreign object intrusion. If foreign object intrusion is detected, the detection object information is generated.

[0016] S2. Store the detected object information generated by the visual perception module as an event in the context management module;

[0017] S3. The context management module extracts the current context state, including environmental parameters and historical behavior;

[0018] S4. The cognitive decision-making module constructs a prompt word by combining the detection object information output by the visual perception module with the context state provided by the context management module, and calls the large language model to perform semantic reasoning and risk judgment.

[0019] S5. The large language model parses the output results and generates a risk assessment report;

[0020] S6. The alarm linkage module executes alarm or linkage operations based on the risk level.

[0021] Furthermore, the large language model is Qwen or its private deployment version.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] (1) High level of intelligence: It can not only "see", but also "understand", realizing the leap from "detection" to "understanding";

[0024] (2) Significantly reduced false alarm rate: Effectively distinguishes normal activities from real threats through contextual memory and semantic reasoning;

[0025] (3) Strong decision interpretability: Outputs analysis reports in natural language format, which are easy for operation and maintenance personnel to understand and trust;

[0026] (4) Good system scalability: The MCP architecture supports the access of more sensors (radar, infrared) and scheduling system interfaces;

[0027] (5) Strong privacy and security: No need to upload the original image to the cloud, only structured metadata is transmitted;

[0028] (6) It conforms to the trend of intelligent development of railways: It is in line with the construction direction of "smart railway" of China State Railway Group and has broad application prospects.

[0029] In addition to the objectives, features, and advantages described above, the present invention has other objectives, features, and advantages. The invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0030] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings:

[0031] Figure 1 This is a block diagram of the overall architecture of a preferred railway foreign object intrusion monitoring system in an embodiment of the present invention;

[0032] Figure 2 This is a detailed schematic diagram of the MCP tool service module in an embodiment of the present invention;

[0033] Figure 3 This is a schematic diagram illustrating the application of the railway foreign object intrusion monitoring system in a railway scenario according to an embodiment of the present invention;

[0034] Figure 4 The flowchart illustrates the preferred method for detecting foreign object intrusion in railways according to an embodiment of the present invention. Detailed Implementation

[0035] The present invention will now be described in detail with reference to the embodiments shown in the accompanying drawings. However, it should be noted that these embodiments are not intended to limit the present invention. Equivalent transformations or substitutions in function, method, or structure made by those skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0036] Please see Figure 1 This embodiment provides a railway foreign object intrusion monitoring system based on an AI intelligent agent, including sensing devices, a visual perception module, a context management module, a cognitive decision-making module, an alarm linkage module, and a database module. The system is deployed in an edge-cloud collaborative architecture, with the visual perception module running on the edge device and only uploading detection metadata to the central cognitive decision-making module. The specific structure is as follows:

[0037] The sensing device connects to the visual perception module to acquire image frames of the railway area in real time. The visual perception module uses an NVIDIA Jetson Orin edge device equipped with a YOLOv8s model and connects to high-definition cameras (1080P@30fps) along the railway line. It extracts frames per second for inference, identifies potential intrusion objects such as people, dogs, bicycles, and debris, and outputs detection results including category, confidence level, bounding box, and timestamp.

[0038] The context management module is connected to the visual perception module, cognitive decision-making module, alarm linkage module, and database module respectively. The context management module is used to build and maintain context state objects based on the Model Context Protocol (MCP). The context state includes location, time, weather, train operation plan, recent detection events, and historical alarm counts, and supports time-series memory and cross-camera context sharing.

[0039] The cognitive decision-making module uses a central server to deploy a locally fine-tuned Qwen model. It constructs prompts by combining the detection results from the visual perception module with the contextual state provided by the context management module. These prompts are then input into a pre-trained large language model, which performs semantic understanding and risk assessment, generating structured analysis text containing judgments, levels, reasons, and suggestions. The prompt templates used by the cognitive decision-making module include location, current time, weather conditions, train schedules, and a recent detection list, guiding the large language model to output analysis conclusions according to a predetermined format.

[0040] The alarm linkage module is used to trigger corresponding response actions based on the risk level output by the cognitive decision-making module. These actions include local audible and visual alarms, pushing alarm information to the dispatch center, automatically generating work orders, and linking the signal system to speed limits. Specifically, if the risk is determined to be "high," instructions are sent to the on-site audible and visual alarms via the MQTT protocol, and the on-duty dispatcher is notified simultaneously via a WeChat chatbot.

[0041] The database module is used to persistently store the detection events of the context management module; the database module uses SQLite or Redis database, and sets its effective time window to 5 to 15 minutes to statistically analyze the historical frequency and evolution trend.

[0042] Example

[0043] 1. Data Preparation

[0044] (1) Collect surveillance videos along the railway line, covering different times and weather conditions.

[0045] (2) Conduct experimental videos of various typical intrusion scenarios in a closed and secure experimental area.

[0046] (3) Use the labeling tool LabelImg to label foreign objects in the image: person, vehicle, animal, debris, other.

[0047] (4) In the monitoring video, the sampling polygon marks the monitoring area of ​​the railway intrusion.

[0048] (5) Divide the dataset into training set, validation set and test set.

[0049] 2. Model Training and Deployment

[0050] (1) Using the YOLOv11 model as the basic object detection model, the model is trained using a custom dataset based on the pre-trained model.

[0051] (2) Integrate ByteTrack on the basis of training the detection model to realize the target tracking function.

[0052] (3) Quantize the trained target detection and tracking model and deploy it on the Jetson Orin edge computing device.

[0053] 3. MCP server development

[0054] (1) Context encapsulation: Pack the detected events, environmental parameters, and historical behaviors into a standard Context object.

[0055] (2) Time series memory: Store all detection records within the past 10 minutes (SQLite / Redis).

[0056] (3) Status tracking: Determine whether it is a recurring occurrence, a continuous stay, or a change in movement trajectory.

[0057] (4) Prompt word construction: Automatically generate Prompt templates suitable for large models to understand.

[0058] (5) Multi-model scheduling: Coordinate the task chain of YOLO → Qwen → execution module.

[0059] 4. Construction of a large language model semantic reasoning engine

[0060] (1) The Qwen3-1.7B model was used as the basic LLM inference model, and the prompt input model was constructed by combining the sample labeled data labels with the Prompt template in MCP for inference.

[0061] (2) The inference output text results are given to the operation and maintenance personnel for correction, forming a data sample for fine-tuning the LLM model.

[0062] (3) The LoRA sampling method was used to fine-tune the Qwen3-1.7B model.

[0063] 5. Deployment and application of intelligent agents

[0064] The sampling is deployed in a hybrid edge and center mode. Edge nodes are deployed in station / section equipment rooms, including cameras, edge computing boxes (running YOLO), local databases, etc.; the central platform is deployed in data centers / cloud servers, including MCP services, Qwen calls, log aggregation, web console, etc.

[0065] Please see Figure 4 The present invention also provides a railway foreign object intrusion monitoring method based on AI intelligent agents, which uses the above-mentioned railway foreign object intrusion monitoring system for monitoring. The method includes the following steps:

[0066] S1. The sensing device acquires real-time surveillance videos of the railway line area, covering different times and weather conditions; the visual perception module extracts images from the sensing device and determines whether there is any foreign object intrusion; if foreign object intrusion is detected, the detection object information is generated.

[0067] S2. Store the detected object information generated by the visual perception module as an event in the context management module.

[0068] S3. The context management module extracts the current context state, including environment parameters and historical behavior.

[0069] S4. The cognitive decision-making module combines the detection object information output by the visual perception module with the context state provided by the context management module to construct prompt words, and calls the large language model to perform semantic reasoning and risk judgment; where the large language model is Qwen or its private deployment version.

[0070] S5 analyzes the output of the large language model and generates a risk assessment report.

[0071] S6. The alarm linkage module executes alarm or linkage operations based on the risk level.

[0072] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A railway foreign object intrusion monitoring system based on AI intelligent agents, characterized in that, The system includes a sensing device, a visual perception module, a context management module, a cognitive decision-making module, an alarm linkage module, and a database module. The visual perception module is used to acquire video streams or image frames of the railway monitoring area and uses a target detection model to identify foreign objects in the image, outputting detection results including category, confidence level, bounding box, and timestamp. The context management module is connected to the visual perception module, the cognitive decision-making module, the alarm linkage module, and the database module. The context management module is used to construct and maintain context state objects based on a model context protocol. The cognitive decision-making module is used to construct prompt words by combining the detection results output by the visual perception module with the context state provided by the context management module, and input them into a pre-trained large language model for semantic understanding and risk assessment, outputting structured analysis text. The alarm linkage module is used to trigger a corresponding level of response action based on the risk level output by the cognitive decision-making module; the database module is used to persistently store the detection events of the context management module.

2. The railway foreign object intrusion monitoring system according to claim 1, characterized in that, The context status includes location, time, weather, train operation plan, recent detection events and historical alarm counts, and supports time-series memory and cross-camera context sharing.

3. The railway foreign object intrusion monitoring system according to claim 1, characterized in that, The cognitive decision-making module provides prompt word templates including location, current time, weather conditions, train operation plan, and recent detection list, and guides the large language model to output analysis conclusions in a predetermined format.

4. The railway foreign object intrusion monitoring system according to claim 1, characterized in that, The cognitive decision-making module provides prompt word templates including location, current time, weather conditions, train operation plan, and recent detection list, and guides the large language model to output analysis conclusions in a predetermined format.

5. The railway foreign object intrusion monitoring system according to claim 1, characterized in that, The large language model performs semantic reasoning on the input prompts and generates structured analysis text containing judgments, levels, reasons, and suggestions.

6. The railway foreign object intrusion monitoring system according to claim 1, characterized in that, The response actions include local audible and visual warnings, pushing alarm information to the dispatch center, automatically generating work orders, and linking the signal system to limit speed.

7. The railway foreign object intrusion monitoring system according to claim 1, characterized in that, The database module uses SQLite or Redis databases, with its effective time window set to 5-15 minutes, to statistically analyze historical occurrence frequency and evolution trends.

8. A method for monitoring foreign object intrusion in railways based on AI intelligent agents, characterized in that, Monitoring using the railway foreign object intrusion monitoring system as described in any one of claims 1-7 includes the following steps: S1. The sensing device acquires image frames of the railway area in real time. The visual perception module extracts the image from the sensing device and determines whether there is any foreign object intrusion. If foreign object intrusion is detected, the detection object information is generated. S2. Store the detected object information generated by the visual perception module as an event in the context management module; S3. The context management module extracts the current context state, including environmental parameters and historical behavior; S4. The cognitive decision-making module constructs a prompt word by combining the detection object information output by the visual perception module with the context state provided by the context management module, and calls the large language model to perform semantic reasoning and risk judgment. S5. The large language model parses the output results and generates a risk assessment report; S6. The alarm linkage module executes alarm or linkage operations based on the risk level.

9. The railway foreign object intrusion monitoring method according to claim 8, characterized in that, The large language model is Qwen or its private deployment version.

Images