Railway scene dangerous event detection method based on graph neural network
By combining graph neural networks and large language models, the problem of insufficient risk information analysis in existing railway scene security technologies has been solved, risk event prediction and management of railway scenes have been realized, and the accuracy and predictive ability of security detection have been improved.
Patent Information
- Application Number
- CN202510766306.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-10
- Publication Date
- 2025-09-12
AI Technical Summary
Existing railway scene security technology is unable to deeply analyze the risk information implicit in the data, has difficulty identifying the most dangerous targets among multiple intrusion targets, and is difficult to predict dangerous events based on time series information based on single-frame images.
A railway scene graph feature extraction model based on graph neural network is adopted, combined with a large language model to predict railway status. The classification information and risk source information of the scene image are extracted through the graph convolutional network, and a natural text dataset is constructed for in-depth analysis to achieve railway scene status prediction and risk tracing.
It can effectively predict risk events in railway scenarios, identify key nodes and risk sources in the scenarios, conduct risk reasoning and status analysis, and improve the safety management capabilities of railway scenarios.
Smart Images
Figure CN120635778A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of railway scene detection, and in particular to a railway scene dangerous event detection method based on graph neural network. Background Art
[0002] Traditional railway security technology primarily relies on image data for foreign object detection to achieve intuitive detection results. However, this approach suffers from shortcomings in both inference depth and time span: a single neural network architecture cannot deeply analyze the risk information implicit in the data; and a single-frame image cannot capture the dynamic changes of the scene.
[0003] A railway scene safety protection method in the existing technology proposes a new end-to-end scene analysis framework called PerceptionGPT, which uses LLM (Large Language Model) dynamic tag embedding to represent perceptual signals. The method first introduces a special tag to identify the presence of perceptual signals. Unlike the previous P-VLM method that uses static tags, the perceptual signal embedding representation is dynamically variable and can represent various perceptual signals - whether it is a bounding box of any size or a mask with an irregular shape; then a pair of perceptual codecs are constructed to restore the original features from the dynamic embedding. And in order to comprehensively utilize the features of each layer, they introduced a learnable weight coefficient in each layer, and constructed more expressive visual features through weighted summation.
[0004] The shortcomings of the existing railway scene security methods mentioned above include: These methods rely on target detection technology, which can only identify targets within a scene and cannot comprehensively analyze the overall safety status of the scene. When there are multiple intruding targets, it is also difficult to identify the most dangerous target. Furthermore, because existing target detection technologies are mostly based on single-frame images, it is difficult to accurately predict the development of dangerous events based on time series information. Summary of the Invention
[0005] An embodiment of the present invention provides a railway scene dangerous event detection method based on graph neural network to effectively predict railway scene risk events.
[0006] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions.
[0007] A railway scene dangerous event detection method based on graph neural network, comprising:
[0008] Inputting a railway scene image at a single moment into a railway scene graph feature extraction model based on an improved graph convolutional network, the railway scene graph feature extraction model outputs classification information, risk source type information, and node feature information of the railway scene image;
[0009] Constructing a railway scene natural text dataset using the output data of the railway scene graph feature extraction model;
[0010] Using the railway scene natural text dataset to train a railway state prediction large language model to obtain a trained railway state prediction large language model;
[0011] The railway scene image to be identified is input into the railway state prediction large language model, and a natural time-series text description of the railway scene image to be identified is obtained based on the output result of the railway state prediction large language model and the natural text conversion template. The natural time-series text description of the railway scene image to be identified is input into the trained railway state prediction large language model, and the trained railway state prediction large language model outputs a description of the scene state, risk source and risk prediction of the railway scene image to be identified.
[0012] Preferably, the railway scene image at a single moment is input into a railway scene graph feature extraction model based on an improved graph convolutional network, and the railway scene graph feature extraction model outputs classification information, risk source type information, and node feature information of the railway scene image, including:
[0013] The railway scene image at a single moment is input into the railway scene graph feature extraction model based on the improved graph convolutional network. The railway scene graph feature extraction model transforms the railway scene graph into subject-verb-object triples (s i , r i , o i ) description, the triple (s i , r i , o i ) is sent to the first graph convolution layer in the railway scene graph feature extraction model to extract initial features and form an output of size num_node_features*hidden_channels. The first graph convolution layer is followed by a SELU activation function. The activated features are used as the input of the second graph convolution layer. The second graph convolution layer further extracts node features and maintains the same output scale. The features output by the second graph convolution layer are activated by the RELU function. Similarly, the third graph convolution layer also performs similar operations and is activated. After three convolution operations, the output node features are mapped to the specified graph category classification space through the classification head;
[0014] The railway scene graph feature extraction model labels the danger level of each frame of railway scene image and divides the danger levels into four categories: safe scene, low-level danger scene, medium-level danger scene, and high-level danger scene. The Rail-GAT model outputs the node features and scene danger level judgment results corresponding to the railway scene image at each moment.
[0015] Preferably, the method of constructing a railway scene natural text dataset using the output data of the railway scene graph feature extraction model includes:
[0016] Based on the node features corresponding to the railway scene images at each moment output by the railway scene graph feature extraction model, the structured graph data is converted into a natural temporal text description by batch adding conjunctions and prepositions. Based on the scene danger level judgment result generated by the railway scene graph feature extraction model, the scene state description is added to the natural temporal text description to construct a railway scene natural text dataset.
[0017] Preferably, the method of training the railway state prediction large language model using the railway scene natural text dataset to obtain the trained railway state prediction large language model includes:
[0018] The natural temporal text descriptions in the railway scene natural text dataset are input into the railway status prediction large language model. Under the guidance of fine-tuning instructions, the railway status prediction large language model is fine-tuned. The fine-tuned model railway status prediction large language model can judge the status of the entire scene, track the most dangerous elements and predict the scene status at the next moment by comprehensively analyzing the scene graph information at each moment.
[0019] The railway state prediction large language model introduces the large language model LLaMA3.2 as the main body of analysis. The railway state prediction large language model includes: LLaMA3.2 model, scene recognition template and multi-level instructions. The scene recognition template is used to determine the specific type of railway scene. The multi-level instructions include three levels: base layer, association layer and reasoning layer.
[0020] During the training process of the large language model for railway status prediction, the output of the large language model is decomposed into three subtasks: scenario status classification, risk tracing, and risk reasoning. The subtasks are mapped to the task paradigms in the field of graph learning: graph classification, node classification, and event classification. Interpretable quantitative indicators are used to evaluate the training results of the Rail-LLM model. When the change in classification accuracy within a set number of training rounds is less than the set threshold range, the training is terminated, and the trained large language model for railway status prediction is obtained.
[0021] It can be seen from the technical solutions provided by the above-mentioned embodiments of the present invention that the method of the present invention uses a graph neural network to obtain primary railway scene perception information, and uses a large language model technology to perform in-depth analysis based on the obtained primary perception information to obtain scene status information, risk tracing information and risk reasoning information, thereby effectively predicting and managing railway scene risk events.
[0022] Additional aspects and advantages of the present invention will be set forth in part in the following description, will become apparent from the following description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0024] Figure 1 The implementation principle diagram of a railway scene dangerous event detection method based on graph neural network provided by the embodiment of the present invention is as follows Figure 1 shown.
[0025] Figure 2 A processing flow chart of a railway scene dangerous event detection method based on a graph neural network provided in an embodiment of the present invention.
[0026] Figure 3 This figure shows the overall structure of the Rail-GAT model based on the improved graph convolutional network (DA-GCN) provided in an embodiment of the present invention.
[0027] Figure 4 An overall structural diagram of a Rail-LLM model provided by an embodiment of the present invention;
[0028] Figure 5 A schematic diagram illustrating the scene status, risk sources, and risk prediction of a railway scene image output by a trained Rail-LLM model provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0029] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limiting the present invention.
[0030] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the description of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when the present invention refers to an element being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or couplings. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.
[0031] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention pertains. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and, unless defined as such herein, will not be interpreted in an idealized or overly formal sense.
[0032] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings. However, each embodiment does not constitute a limitation on the embodiments of the present invention.
[0033] This paper uses graph neural networks and large language models to perform status analysis, risk tracing, and risk reasoning in railway scenarios. In the first phase, the graph neural network classifies the scene graph and identifies key nodes, thereby obtaining training data for the second phase. In the second phase, the large language model is used to deeply mine and analyze the scene data, realizing status analysis, risk tracing, and risk reasoning in railway scenarios.
[0034] The implementation principle diagram of a railway scene dangerous event detection method based on graph neural network provided by an embodiment of the present invention is as follows: Figure 1 The specific processing flow is as shown in Figure 2 As shown, the processing steps include the following:
[0035] Step S10: Input the railway scene image at a single moment into the railway scene graph feature extraction model based on the improved graph convolutional network (DA-GCN). The Rail-GAT model outputs the classification information and risk source information of the railway scene image, as well as the key node information in the railway scene image.
[0036] The railway scene graph feature extraction model may be a Rail-GAT model. The Rail-GAT model is taken as an example to illustrate the embodiment of the present invention.
[0037] The event analysis of the present invention refers to making a dangerousness judgment on the current events occurring in the railway scene, such as pedestrian intrusion, and pointing out the risk sources, and finally making a prediction on the development of the event.
[0038] A railway scene image at a single moment is obtained and input into the Rail-GAT model based on the improved DA-GCN. The Rail-GAT model classifies the railway scene image and detects the main risk sources. On this basis, the original scene graph information is converted into a natural language description of the scene through a natural text conversion template. This natural language description is used as the input data for the second-stage Rail-LLM model.
[0039] The overall structure of the Rail-GAT model based on the improved graph convolutional network (DA-GCN) provided by the embodiment of the present invention is as follows: Figure 3 As shown in Figure 3, DA-GCN is responsible for generating graph feature representations and then using a classification head to predict the classification labels of the graphs.
[0040] In railway scene images, node representation is the basis for the model to perform graph classification. Each node corresponds to a different target entity in the scene (such as pedestrians, rails, trains, fallen rocks, etc.). The railway scene image at a single moment is input into the above-mentioned Rail-GAT model. The Rail-GAT model identifies the node information in the railway scene image, classifies the nodes by defining the target type, and uses a multi-class one-hot encoding method to assign a unique code to each node. Let N be the total number of node categories, then each node is mapped to an N-dimensional real-valued vector: x i =[0, ..., 1, ..., 0], where there is only one element in the vector that is 1, and its position corresponds to the category index of the node.
[0041] The Rail-GAT model transforms the railway scene graph into subject-verb-object triples (s i , r i , o i ) description, then, the triple type (s i , r i , o i) is sent to the first graph convolution layer in the Rail-GAT model to extract the initial features and form an output of size num_node_features*hidden_channels. The first graph convolution layer is followed by a SELU (scaled exponential linear unit) activation function. The activated features serve as the input of the second graph convolution layer. The second layer is responsible for further feature extraction of the nodes and maintains the same output scale. The features output by the second graph convolution layer are activated by the RELU function. Similarly, the third convolution layer performs similar operations and is activated. After three convolution operations, the output node features are mapped to the specified graph category classification space through the classification head.
[0042] In the node feature extraction process, for each node s in the railway scene image G i ∈S or o i ∈O, its eigenvector x i Stored as a node feature matrix The relationship between nodes (i.e., edge) is represented by edge index and type, and all node relationships are mapped to edge r in the graph. ij ∈R. For each graph G, the present invention normalizes the node features and obtains a normalized feature matrix which can be expressed as:
[0043]
[0044] Where D is the degree matrix.
[0045] Through this conversion method, the Rail-GAT model can efficiently capture the feature differences and structural information between different entity nodes, fully express the target heterogeneity in railway scenarios, and improve the generalization ability of the graph classification model.
[0046] The Rail-GAT model outputs the scene danger level judgment results corresponding to the railway scene images at each moment and the most dangerous event (described in the form of triplets).
[0047] The Rail-GAT model labels the danger level of each frame of railway scene image and divides the danger levels into four categories: safe scenes, mainly including empty scenes and normal train operation scenes; low-level danger scenes, referring to people or objects intruding into the railway perimeter but not into the train operating limits; medium-level danger scenes, referring to people or objects intruding into or intending to intrude into the train operating limits but not into the track limits; high-level danger scenes, referring to people or objects directly intruding into the track limits.
[0048] Step S20: construct a railway scene natural text dataset using the output data of the railway scene graph feature extraction model.
[0049] Based on the node features corresponding to the railway scene images at each moment output by the above-mentioned railway scene graph feature extraction model, the structured graph data is converted into natural temporal text descriptions by batch adding conjunctions and prepositions. Based on the scene danger level judgment results generated by the above-mentioned railway scene graph feature extraction model, the scene status description is added to construct a railway scene natural text dataset for training and testing large language models for railway status prediction.
[0050] The above scene status description includes: 1. Scene status analysis: description of the overall scene status based on the current frame status, indicating whether the scene status is highly dangerous, moderately dangerous, low-level dangerous or safe; 2. Risk tracing: description of the risk source, indicating the most dangerous node in the scene; 3. Risk inference: description of the risk evolution process based on the behavior of the risk source in the next frame scene graph.
[0051] Step S30: Train the railway status prediction large language model using the railway scene natural text dataset to obtain a trained railway status prediction large language model.
[0052] The above-mentioned large language model for railway state prediction may be a Rail-LLM model. The following takes the Rail-LLM model as an example to illustrate the embodiment of the present invention.
[0053] The natural temporal text descriptions in the railway scene natural text dataset are input into the Rail-LLM model. Under the guidance of fine-tuning instructions, the Rail-LLM model is fine-tuned. The fine-tuned Rail-LLM model can judge the status of the overall scene, track the most dangerous elements, and predict the scene status at the next moment by comprehensively analyzing the scene graph information at each moment.
[0054] The Rail-LLM model introduces the large language model llama3.2 as the main body of analysis. The overall structure of the Rail-LLM model provided by the embodiment of the present invention is as follows: Figure 4 As shown, the characteristics of the Rail-LLM model include:
[0055] Rail-LLM is a specialized model based on the LLaMA3.2 model and fine-tuned on a railway scene natural text dataset based on scene recognition templates and multi-level instructions.
[0056] The scene recognition template is used to provide railway expertise, instructing the model to determine the specific type of railway scene based on obvious markers and to focus on possible dangerous targets in the scene based on the specific type.
[0057] The multi-level instructions include three levels: basic layer: by constructing scene state analysis instructions, based on the scene state at the initial moment, focusing on abnormal situations in the scene; association layer: by constructing risk tracing instructions, guiding the large model to find the event with the highest risk level in the scene and the subject target that caused the event; reasoning layer: after clarifying the main risk source in the scene, analyze the events that may be triggered in the next step based on the historical movement pattern and dynamic characteristics or static characteristics of the risk source.
[0058] During the Rail-LLM model training process, by decomposing the model's output into three subtasks: scenario state classification, risk tracing, and risk reasoning, the present invention maps it to classic task paradigms in graph learning: graph classification, node classification, and event classification. This task mapping enables the present invention to use accurate and interpretable quantitative metrics for evaluation.
[0059] The classification accuracy used in this invention is defined as follows:
[0060]
[0061] Among them, TP represents the number of correctly predicted samples, N is the total number of test samples, for graph classification tasks, N represents the total number of graphs in the test set; for node classification tasks, N represents the number of subject nodes in each scene; for event classification tasks, N represents the total number of events that may occur in the entire scene, that is, the number of subjects × the number of relations × the number of objects. is the model’s predicted label for the i-th sample, y i is the true label of the i-th sample, 1(·) is the indicator function, which is 1 when the condition is met and 0 otherwise.
[0062] After the start of each stage of training, the changes in classification accuracy are observed. When the change in classification accuracy within 10 training rounds is less than 0.0005, the training is terminated and the trained Rail-LLM model is obtained.
[0063] Step S30: Input the railway scene image to be identified into the railway scene graph feature extraction model to obtain a natural temporal text description of the railway scene image to be identified, input the natural temporal text description of the railway scene image to be identified into the trained railway state prediction large language model, and the trained railway state prediction large language model outputs a description of the scene state, risk source, and risk prediction of the railway scene image to be identified.
[0064] Specific examples include Figure 5The middle column shows examples of Rail-LLM's input and output. It is divided into four scenarios. The input in the middle column represents the model's input, and the output represents the model's output. The leftmost column shows the original image corresponding to the model input, which is not used as input; the rightmost column shows the ground truth during model training, which is not used as output.
[0065] In summary, the method of the present invention uses graph neural networks to optimize the target nodes of railway scenarios, which can better extract node features and relationship features; the constructed railway scenario natural text sequence dataset is more professional than the general dataset and covers more railway scenario knowledge; the constructed scene type recognition template and multi-level instructions are designed according to the characteristics of railway scenarios, which can provide professional railway knowledge to the general model and improve the model's reasoning results in railway scenarios.
[0066] The method of the present invention can preprocess the basic graph structure data and achieve the conversion from unsupervised training data to supervised training data by supplementing the scene state information and the main risk source information. It can also achieve the reasoning of the main risks in the scene and predict the possible dangerous events in the scene in advance.
[0067] Those skilled in the art will appreciate that the accompanying drawings are merely schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.
[0068] From the above description of the embodiments, it can be seen that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus the necessary general-purpose hardware platform. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.
[0069] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without making any creative efforts.
[0070] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.
Claims
1. A railway scene dangerous event detection method based on graph neural network, characterized by: include: Inputting a railway scene image at a single moment into a railway scene graph feature extraction model based on an improved graph convolutional network, the railway scene graph feature extraction model outputs classification information, risk source type information, and node feature information of the railway scene image; Constructing a railway scene natural text dataset using the output data of the railway scene graph feature extraction model; Using the railway scene natural text dataset to train a railway state prediction large language model to obtain a trained railway state prediction large language model; The railway scene image to be identified is input into the railway state prediction large language model, and a natural time-series text description of the railway scene image to be identified is obtained based on the output result of the railway state prediction large language model and the natural text conversion template. The natural time-series text description of the railway scene image to be identified is input into the trained railway state prediction large language model, and the trained railway state prediction large language model outputs a description of the scene state, risk source and risk prediction of the railway scene image to be identified.
2. The method according to claim 1, characterized in that The railway scene image at a single moment is input into the railway scene graph feature extraction model based on the improved graph convolutional network. The railway scene graph feature extraction model outputs classification information, risk source type information, and node feature information of the railway scene image, including: The railway scene image at a single moment is input into the railway scene graph feature extraction model based on the improved graph convolutional network. The railway scene graph feature extraction model transforms the railway scene graph into subject-verb-object triples (s i , r i , o i ) description, the triple (s i , r i , o i ) is sent to the first graph convolution layer in the railway scene graph feature extraction model to extract initial features and form an output of size num_node_features*hidden_channels. The first graph convolution layer is followed by a SELU activation function. The activated features are used as the input of the second graph convolution layer. The second graph convolution layer further extracts node features and maintains the same output scale. The features output by the second graph convolution layer are activated by the RELU function. Similarly, the third graph convolution layer also performs similar operations and is activated. After three convolution operations, the output node features are mapped to the specified graph category classification space through the classification head; The railway scene graph feature extraction model labels the danger level of each frame of railway scene image and divides the danger levels into four categories: safe scene, low-level danger scene, medium-level danger scene, and high-level danger scene. The Rail-GAT model outputs the node features and scene danger level judgment results corresponding to the railway scene image at each moment.
3. The method according to claim 2, characterized in that The method of constructing a railway scene natural text dataset using the output data of the railway scene graph feature extraction model includes: Based on the node features corresponding to the railway scene images at each moment output by the railway scene graph feature extraction model, the structured graph data is converted into a natural temporal text description by batch adding conjunctions and prepositions. Based on the scene danger level judgment result generated by the railway scene graph feature extraction model, the scene state description is added to the natural temporal text description to construct a railway scene natural text dataset.
4. The method according to claim 3, characterized in that The method of training the railway status prediction large language model using the railway scene natural text dataset to obtain the trained railway status prediction large language model includes: The natural temporal text descriptions in the railway scene natural text dataset are input into the railway status prediction large language model. Under the guidance of fine-tuning instructions, the railway status prediction large language model is fine-tuned. The fine-tuned model railway status prediction large language model can judge the status of the entire scene, track the most dangerous elements and predict the scene status at the next moment by comprehensively analyzing the scene graph information at each moment. The railway state prediction large language model introduces the large language model LLaMA3.2 as the main body of analysis. The railway state prediction large language model includes: LLaMA3.2 model, scene recognition template and multi-level instructions. The scene recognition template is used to determine the specific type of railway scene. The multi-level instructions include three levels: base layer, association layer and reasoning layer. During the training process of the large language model for railway status prediction, the output of the large language model is decomposed into three subtasks: scenario status classification, risk tracing, and risk reasoning. The subtasks are mapped to the task paradigms in the field of graph learning: graph classification, node classification, and event classification. Interpretable quantitative indicators are used to evaluate the training results of the Rail-LLM model. When the change in classification accuracy within a set number of training rounds is less than the set threshold range, the training is terminated, and the trained large language model for railway status prediction is obtained.