A Smart Station Emergency Decision-Making Method Based on Multimodal Large Models and Knowledge Graphs

Abstract

This invention relates to the field of smart station technology, and more particularly to a smart station emergency decision-making method based on a multimodal large model and knowledge graph. The method includes: acquiring multi-source heterogeneous data from the smart station; using a semantic segmentation strategy and a text generation model to transform the multi-source heterogeneous data into semantic descriptive text; inputting the semantic descriptive text into a trained large language model to extract local keywords; performing fusion-based graph node matching using the local keywords based on a semantic-topological dual-path index established by a pre-constructed event knowledge graph to obtain matching results; performing hierarchical confidence assessment and status labeling based on the matching results to obtain final abnormal nodes; and generating and outputting an emergency response plan for the abnormal event based on the final abnormal node and the pre-constructed event knowledge graph. This invention improves the intelligence, collaboration, and efficiency of emergency response.

Description

Technical Field

[0001] This invention relates to the field of smart station technology, and in particular to a smart station emergency decision-making method based on a multimodal large model and knowledge graph. Background Technology

[0002] As urban rail transit networks continue to expand, smart stations have deployed various information systems, including video surveillance, IoT sensors, and automated fare collection, generating massive amounts of multimodal data daily, including text, images, and time-series sensor data. At the same time, stations face unexpected emergencies such as large passenger congestion, equipment failures, fires, and extreme weather, exhibiting characteristics of randomness, multi-source heterogeneity, and dynamic correlation.

[0003] In existing technologies, traditional emergency decision-making mainly relies on human experience and fixed contingency plan databases, lacking deep semantic understanding and structured knowledge organization capabilities for multimodal data. While knowledge graphs, as a core technology for structured knowledge representation, have been applied in some scenarios, their construction process still heavily relies on manual annotation and rule-driven approaches, making them unsuitable for the highly dynamic and complex emergency scenarios of smart stations. For example, patent application CN120806158A (classification G06N) provides a method and device for constructing and making decisions on a large-scale traffic model based on multimodal bidirectional graph reasoning; patent application CN121562748A (classification G06N) provides a method and device for generating intelligent profiles in the power sector based on multimodal knowledge graphs; and patent application CN118966342A (classification G06N) provides an industrial knowledge-generating decision-making method based on multi-granular semantics and large-scale model assistance.

[0004] Therefore, there is an urgent need for a smart station emergency decision-making method based on multimodal large models and knowledge graphs. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] In view of the above-mentioned shortcomings and deficiencies of the existing technology, the present invention provides a smart station emergency decision-making method based on multimodal large model and knowledge graph, which solves the technical problems of low knowledge graph construction efficiency, insufficient multimodal information fusion, lagging knowledge update and one-sided decision-making in the case of multi-source heterogeneous, dynamic, complex and highly correlated emergencies.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the main technical solutions adopted by the present invention include:

[0009] This invention provides a smart station emergency decision-making method based on a multimodal large model and knowledge graph, including:

[0010] The system acquires multi-source heterogeneous data from smart stations and uses a semantic segmentation strategy and text generation model to transform the multi-source heterogeneous data into semantic description text.

[0011] The semantic description text is input into a trained large language model to extract local keywords, which include entity keywords and relation keywords.

[0012] Based on a pre-built event knowledge graph, a semantic-topological dual-path index is established. The local keywords are used to perform fusion-based graph node matching to obtain matching results. The matching results include candidate abnormal event nodes and their matching scores. The dual-path index includes a semantic index based on vector retrieval and a topological index based on graph traversal. The semantic index transforms the descriptive text of entities and relationships in the event knowledge graph into dense vector storage, and the topological index stores entity nodes and relationship edges using an attribute graph model and establishes a multi-level adjacency index.

[0013] Based on the matching results, a graded confidence assessment and status labeling are performed to obtain the final abnormal nodes;

[0014] Based on the final abnormal node and the pre-built event knowledge graph, an emergency response plan for the abnormal event is generated and output.

[0015] Optionally, in some embodiments of this application, the step of using a semantic segmentation strategy and a text generation model to convert the multi-source heterogeneous data into semantic descriptive text includes:

[0016] Identify the modality type of each data in the multi-source heterogeneous data, dynamically select the corresponding semantic segmentation granularity according to the modality type, and divide each data into one or more intermediate blocks according to the semantic segmentation granularity; embed context association identifiers within the intermediate blocks;

[0017] For each intermediate block, a corresponding text generation model is selected according to the modality type to convert the intermediate block into a natural language descriptive text block; the text generation model includes a multimodal large model for processing images, a large language model for processing text, and a template generation model for processing structured data;

[0018] The natural language description text blocks are merged to generate semantic description text.

[0019] Optionally, in some embodiments of this application, fusing the natural language description text blocks to generate semantic description text specifically includes:

[0020] Based on the context association identifier, the fusion order of each natural language description text block is determined;

[0021] Semantic consistency is checked in the boundary region of adjacent natural language description text blocks. If there are conflicts or redundant content, disambiguation or merging is performed according to the source confidence weight of the context association identifier to obtain the disambiguation and merging result.

[0022] Based on the fusion order and disambiguation merging result, the natural language description text blocks are concatenated to generate the semantic description text.

[0023] Optionally, in some embodiments of this application, inputting the semantic description text into a trained large language model to extract local keywords specifically includes:

[0024] The semantic description text is input into the trained large language model for entity boundary recognition and keyword detection, generating a set of candidate keywords and a confidence score for each candidate keyword; the large language model is a domain-specific large language model fine-tuned using corpus from the rail transit domain.

[0025] Based on the terminology database of the rail transit field, each candidate keyword in the candidate keyword set is standardized and mapped, and the mapped standardized words are used as local keyword candidates.

[0026] Select local keyword candidates with confidence scores higher than a preset threshold to obtain local keywords.

[0027] Optionally, in some embodiments of this application, based on the semantic-topological dual-path index established by the pre-built event knowledge graph, graph node matching is performed using the local keywords to obtain matching results including:

[0028] Using the entity keywords in the local keywords, entity matching is performed in the topological index of the knowledge graph to obtain low-level candidate nodes and matching scores for each node;

[0029] The local keywords are converted into query vectors, and an approximate nearest neighbor search is performed in the semantic index of the knowledge graph. The cosine similarity between the query vector and the description vector of each abnormal event node is calculated to obtain the high-level candidate nodes and the semantic similarity score of each node.

[0030] The low-level candidate nodes are fused with the high-level candidate nodes. The weighted sum of the matching score and semantic similarity score of the fused candidate nodes is calculated as the final matching score. The node with the highest final matching score is output as the matching result.

[0031] Optionally, in some embodiments of this application, when fusing the lower-level candidate nodes with the higher-level candidate nodes, a dynamic weighting coefficient is used, the dynamic weighting coefficient specifically including:

[0032] A first dynamic weight factor is calculated based on the confidence score of entity keywords in the local keywords, and a second dynamic weight factor is calculated based on the distribution dispersion of the cosine similarity between the query vector and the description vector of the abnormal event node; wherein, the first dynamic weight factor is positively correlated with the confidence score of entity keywords, and the second dynamic weight factor is negatively correlated with the distribution dispersion of the cosine similarity.

[0033] The first dynamic weight factor is used as the weight of the topology matching score, and the second dynamic weight factor is used as the weight of the semantic similarity score. The two are normalized so that the sum of the normalized weights is 1.

[0034] Optionally, in some embodiments of this application, obtaining the final abnormal node by performing confidence assessment and status marking based on the matching results specifically includes:

[0035] The candidate abnormal event node with the highest final matching score in the matching results is selected as the node to be evaluated, and the historical matching accuracy statistics of the node to be evaluated are extracted.

[0036] The matching score of the node to be evaluated is weighted and fused with the historical matching accuracy statistics to obtain the comprehensive confidence level;

[0037] When the overall confidence level is higher than the first threshold, the node to be evaluated is marked as confirmed and output as the final abnormal node; when the overall confidence level is not higher than the first threshold but higher than the second threshold, it is marked as pending verification and supplementary data collection or manual review is triggered; when the overall confidence level is not higher than the second threshold, it is marked as a false alarm and discarded.

[0038] Optionally, in some embodiments of this application, generating and outputting an emergency response plan for anomalies based on the final abnormal node and the pre-built event knowledge graph specifically includes:

[0039] Extract the first-order neighborhood information of the final abnormal node from the knowledge graph. The first-order neighborhood information includes the attribute relationships of the device nodes, personnel nodes, contingency plan nodes, and connecting edges directly connected to the final abnormal node.

[0040] The first-order neighborhood information is concatenated with the semantic description text to form a solution generation prompt; the solution generation prompt is input into a large language model to generate an emergency response plan.

[0041] Optionally, in some embodiments of this application, the method further includes: simultaneously generating an emergency response plan and performing abnormal pre-event mining and dynamic updating of the knowledge graph, specifically including:

[0042] Using the final abnormal node as the anchor point, a reverse path traversal is performed along the incoming edge direction in the knowledge graph to extract a set of candidate predecessor nodes that have causal, predecessor, or companion relationships with the final abnormal node.

[0043] The real-time data stream within a preset time window before the occurrence of the current abnormal event is obtained. The real-time data stream is time-aligned with the attribute features of each node in the candidate preceding node set, and the pattern matching degree between the real-time data stream and each candidate preceding node is evaluated by a large language model.

[0044] Based on the graph distance traversed by the reverse path and the degree of pattern matching, the confidence score of each candidate preceding node is obtained, and the candidate preceding node with a confidence score higher than a preset threshold is confirmed as a valid preceding event.

[0045] Optionally, in some embodiments of this application, the knowledge graph update includes:

[0046] For each candidate precursor node confirmed as a valid precursor event, check whether there is already a node in the knowledge graph with a semantic similarity higher than a preset repetition threshold; if not, instantiate a new precursor warning node in the knowledge graph; if it exists, reuse the existing node and update the trigger count attribute.

[0047] Establish directed edges from the pre-warning nodes to the final abnormal nodes, and perform vector embedding calculations and adjacency list updates on the newly added pre-warning nodes and their corresponding directed edges to obtain the updated knowledge graph.

[0048] (III) Beneficial Effects

[0049] The beneficial effects of this invention are as follows: The intelligent station emergency decision-making method based on multimodal large model and knowledge graph adopts a semantic segmentation strategy and text generation model to uniformly transform multi-source heterogeneous data into semantic description text. It uses a large language model to extract entity and relation keywords, and combines this with a pre-established semantic-topological dual-path index from the event knowledge graph for fusion node matching. Then, through hierarchical confidence assessment and status marking, it locks the final abnormal node and automatically generates an emergency response plan. Compared with existing technologies, it effectively solves the problems of difficult integration of multi-source heterogeneous data in intelligent stations, low accuracy of abnormal event detection, and delayed response. It achieves high-precision real-time perception of station operation status, accurate identification and rapid location of abnormal events, and intelligent generation and push of emergency response plans, significantly improving the safety of station operation and the efficiency of emergency management.

[0050] Furthermore, this invention employs a modality-type adaptive selection of semantic block granularity and text generation model, combined with context association identification and semantic consistency verification, which effectively solves the problems of conflict and redundancy in multi-source data at the temporal, spatial, and semantic levels, significantly improving the accuracy and coherence of semantic description text. Utilizing a finely tuned large language model and standardized terminology mapping in the rail transit field improves the domain adaptability of keyword extraction and the accuracy of professional terminology. Introducing dynamic weight coefficients in semantic-topological dual-path index matching allows the fusion of entity matching and semantic similarity to adaptively adjust according to the confidence distribution and semantic dispersion of the input data, enhancing the robustness and generalization ability of the matching process. Through hierarchical confidence assessment and state labeling, reliable identification of abnormal events and human-machine collaborative decision-making are achieved. Emergency response plans are generated based on first-order neighborhood information, and combined with reverse path traversal and temporal alignment for pre-event mining and dynamic updating of the knowledge graph, enabling not only real-time emergency response but also proactive identification of potential early warning signals and automatic iteration of the knowledge base, achieving a leap from "passive alarm" to "proactive early warning and knowledge self-evolution." Compared to existing technologies, the above features together enable the smart station emergency decision-making system to have stronger multimodal adaptive processing capabilities, domain knowledge-driven high-precision analysis capabilities, and knowledge graph continuous learning and evolution capabilities, thereby significantly improving the accuracy of abnormal event detection, the level of intelligence in emergency response, and the predictability of the overall safe operation of the station. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating an intelligent station emergency decision-making method based on a multimodal large model and knowledge graph according to an embodiment of this application;

[0052] Figure 2 This is a schematic diagram of a process for generating semantic description text according to an embodiment of this application;

[0053] Figure 3 This is a flowchart illustrating the process of obtaining local keywords according to an embodiment of this application;

[0054] Figure 4 This is a schematic diagram of the process for obtaining matching results according to one embodiment of this application;

[0055] Figure 5 This is a diagram showing the overall data flow of an intelligent station emergency decision-making method based on a multimodal large model and knowledge graph according to an embodiment of this application.

[0056] Figure 6 This is a schematic diagram of the knowledge graph before updating according to an embodiment of this application;

[0057] Figure 7 This is a schematic diagram of an updated knowledge graph according to one embodiment of this application. Detailed Implementation

[0058] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] In related technologies, knowledge graph technology has been attempted to be introduced into the rail transit field for structured storage of accident or operational knowledge. However, most solutions are built based on a single text data source (such as dispatch logs), failing to uniformly analyze other modal data such as images and sensors, leading to a one-sided understanding of events. Furthermore, the constructed knowledge graphs are usually static, lacking event-triggered or incremental update capabilities, making it difficult to adapt to highly dynamic changes such as station equipment status and passenger flow. Some solutions attempt to achieve multimodal data fusion and knowledge graph construction, but remain at the knowledge storage and retrieval level, failing to form a closed loop with anomaly detection, emergency decision-making, and solution generation, and thus unable to drive the dynamic emergency process of "perception → decision → execution." Other solutions use a cloud-edge collaborative architecture to achieve anomaly identification and response plan push, but lack structured knowledge representation, resulting in a black box reasoning process with poor interpretability; or they rely solely on video analysis to focus on passenger flow management, with weak perception capabilities for non-visual events such as equipment failure and fires, resulting in limited functionality. Furthermore, existing technologies generally lack human-machine collaborative intervention mechanisms, making it difficult to effectively introduce expert review and knowledge correction when faced with ambiguous, conflicting, or low-confidence information; they also lack modeling of the entire life cycle of abnormal events (such as from early warning signs to the occurrence of the event and then to its evolution and spread), making it difficult to support early intervention and event evolution chain analysis, and even more difficult to reverse update the knowledge base with newly occurring handling experience to achieve the continuous evolution of the knowledge system.

[0060] Therefore, this application aims to solve the above problems and provide a smart station emergency decision-making method based on multimodal large model and knowledge graph. This application first acquires multi-source heterogeneous data of smart stations, and uses a semantic segmentation strategy and text generation models adapted to different modalities (multimodal large model, large language model, template generation model) to unify them into semantic description text; then, it uses a domain-fine-tuned large language model to extract entity and relation keywords, and performs fusion node matching based on a pre-established semantic-topological dual-path index (vector semantic index and attribute graph topological index) of the event knowledge graph to obtain candidate abnormal event nodes and their matching scores; then, it locks the final abnormal node through hierarchical confidence assessment and status marking (confirmed / pending verification / false alarm), and then extracts its first-order neighborhood information to generate an interpretable emergency response plan; in addition, it also mines anomaly precursor events through reverse path traversal and time sequence alignment, and dynamically updates the knowledge graph, thereby realizing the entire process of smart station emergency management from multimodal perception to accurate decision-making and then to knowledge closed-loop evolution.

[0061] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0062] Figure 1 This is a flowchart illustrating an embodiment of an intelligent railway station emergency decision-making method based on a multimodal large model and knowledge graph according to this application. Figure 1 As shown, the emergency decision-making method for this smart station includes:

[0063] S10. Acquire multi-source heterogeneous data of the smart station, and use a semantic segmentation strategy and text generation model to transform the multi-source heterogeneous data into semantic description text.

[0064] In the context of smart stations, multi-source heterogeneous data refers to multiple data sets originating from different information systems and sensor devices, with varying data structures, formats, semantics, and sampling frequencies. Specifically, it mainly includes the following categories:

[0065] Video surveillance data includes real-time video streams from areas such as station halls, platforms, and passageways, as well as structured tags extracted through AI visual analysis, such as personnel density, movement trajectories, falls, and items left behind. This type of data is essentially an image sequence, information-dense but lacking direct semantic description.

[0066] IoT sensor data includes time-series numerical data generated by devices such as temperature and humidity sensors, smoke detectors, water immersion sensors, noise monitors, and door magnetic switches. This type of data is usually sampled at high frequency (e.g., once per second) and reflects real-time changes in the physical environment of the station, but isolated numerical values ​​themselves do not possess event semantics.

[0067] Automated Fare Collection (AFC) data includes passenger card swipe records for entering and exiting the station, ticket type, transaction time, and gate number. This is a structured or semi-structured transaction log that can be used to analyze indicators such as passenger flow, passage speed, and gate passage time, helping to identify anomalies such as congestion and gate malfunctions.

[0068] Equipment operation status data: operation logs, fault codes, status bits (normal / alarm / fault), and maintenance records of critical equipment such as elevators, escalators, platform screen doors, ventilation and air conditioning systems, and signaling systems. This data is typically stored in industrial protocols (such as Modbus, OPC UA) or in log files, and is characterized by both periodic reporting and sudden alarms.

[0069] Operational scheduling and event logs: Text data including dispatch instructions, broadcast messages, customer service work orders, duty logs, and emergency drill records. The data is discrete and unstructured, but contains key information such as event descriptions, time and location, and handling processes.

[0070] Communication and wireless positioning data: Signal strength, location coordinates, timestamps, etc., generated by employee handheld terminals, passenger Wi-Fi probes, UWB positioning tags, etc. These can be used for personnel trajectory tracking and area thermal analysis.

[0071] Its multi-source heterogeneity is reflected in: modal heterogeneity (images, time-series numerical values, text, structured tables); format heterogeneity (video streams, JSON, XML, CSV, log strings); semantic heterogeneity (the same event is described differently in different systems, such as "gate malfunction" being error code "E002" in the AFC log, but "gate B stuck" in the maintenance record); spatiotemporal scale heterogeneity (video frame rate 25fps, sensor 1Hz, logs may be recorded on a minute-by-minute basis); and quality heterogeneity (significant differences in data integrity, noise level, and reliability).

[0072] Therefore, the core of step S10 is how to transform these data from different sources and with different structures into semantic description text that can be understood by machines and read by humans through semantic segmentation strategies and adapted text generation models, so as to provide high-quality input for keyword extraction and graph matching of subsequent large language models.

[0073] Therefore, see Figure 2 S10 specifically includes:

[0074] S11. Identify the modality type of each data in multi-source heterogeneous data, dynamically select the corresponding semantic block granularity according to the modality type, and divide each data into one or more intermediate blocks according to the semantic block granularity; embed context association identifiers within the intermediate blocks.

[0075] In practical implementation, the semantic segmentation strategy for different modal data types can be summarized as follows:

[0076] For images and videos: static images are processed as a single complete image block; continuous videos are processed at the keyframe level, extracting keyframes when there are significant scene changes or at fixed intervals (e.g., every five seconds). If an abnormal event is detected, short segments before and after the event are packaged into a single block. Each block embeds identifiers such as camera ID, timestamp, and location.

[0077] For plain text (logs, work orders, etc.): short texts are divided into single records as intermediate blocks; long documents are segmented into semantic paragraphs (each paragraph is a subtopic). The identifier includes the source system and log level.

[0078] For structured data (such as automatic fare collection records and equipment status tables): Generally, single records are used as the granularity; if multiple consecutive records belong to the same event chain (such as an elevator going from normal to overload and then to shutdown), they are merged into an intermediate block, using event-triggered granularity. The identifier includes the table name, primary key, timestamp, and equipment location.

[0079] For time-series sensor data (temperature, humidity, vibration, etc.): Due to the weak semantics of single-point numerical values, a time window granularity is adopted. Slowly changing quantities are calculated using a longer window (e.g., one minute) to determine the mean and rate of change; rapidly changing quantities that are event-related are calculated using a shorter window (e.g., five seconds) while retaining abrupt change points; the window length can be dynamically adjusted as the data stabilizes or fluctuates. The identifier includes the sensor number, location, and the start and end points of the time window.

[0080] For audio: continuous audio is segmented into independent segments based on silence using speech activity detection; excessively long segments are further subdivided according to semantic pauses; continuous ambient noise is segmented into blocks of fixed duration (e.g., ten seconds). The label includes microphone location, time interval, and preliminary acoustic tagging.

[0081] For multimodal composite data: First, parse out each embedded modal component, divide it into blocks according to the above strategy, and establish parent-child associations through identifiers to indicate that they belong to the same original data packet.

[0082] Through the above dynamic segmentation, each modality is divided into semantically reasonable and appropriately sized intermediate blocks, with rich spatiotemporal and source identifiers, laying the foundation for subsequent cross-modal fusion.

[0083] S12. For each intermediate block, select the corresponding text generation model according to the modality type and convert the intermediate block into a natural language description text block; the text generation model includes a multimodal large model for processing images, a large language model for processing text, and a template generation model for processing structured data.

[0084] For example, for image or video modalities, multimodal large models (such as GPT-4V, Qwen2.5-VL, etc.) are used. These models are trained on massive amounts of text and image pairs and can be fine-tuned using rail transit images, enabling them to convert visual content into natural language descriptions. For pure text modalities (such as logs, work orders), large language models (such as GLM-4, Qwen series, etc.) are used. These models are pre-trained on large-scale general text corpora and then fine-tuned using rail transit domain corpora (operation logs, fault manuals, etc.) to improve their understanding of station terminology. For structured data (such as automatic fare collection records, sensor values), template generation models are used—that is, lightweight models based on preset rules or fine-tuned with small-scale data (such as BART). By mapping fields to predefined text templates or performing symbolic multi-step reasoning, standardized natural language descriptions are quickly output.

[0085] S13. Merge the natural language description text blocks to generate semantic description text.

[0086] Specifically, S13 includes:

[0087] The fusion order of each natural language description text block is determined based on the context association identifier;

[0088] Semantic consistency is checked in the boundary region of adjacent natural language description text blocks. If there are conflicting or redundant contents, disambiguation or merging is performed according to the source confidence weight of the context association identifier to obtain the disambiguation and merging result.

[0089] Based on the fusion order and disambiguation merging results, the natural language description text blocks are concatenated to generate semantic description text.

[0090] In the specific implementation process, each natural language description text block is embedded with a context-related identifier in the S11 stage. The identifier includes three dimensions: spatial location (e.g., platform C area, next to gate G12), timestamp (accurate to the second or millisecond), and data source type (video, log, sensor, etc.). During fusion, the spatiotemporal identifiers of each block are extracted first, and the fusion order is determined according to the principle of "time priority, spatial aggregation".

[0091] Next, semantic consistency is checked at the boundary regions of adjacent text blocks, and disambiguation or merging is performed based on the source confidence weights. Specifically, the content describing the same object or attribute in the tail of the preceding block and the head of the following block is checked. If a conflict (mutually exclusive information) is detected, the source confidence weights of each block are compared. Assertions from the higher-weighted block are retained, while conflicting parts from the lower-weighted block are discarded or labeled; if the weights are similar, both are retained using the ambiguous word ("maybe") and triggered for further verification. If redundancy (semantically identical or highly similar) is detected, the information is merged: the more detailed or higher-confidence version is retained, and supplementary data from the redundant block (such as more precise numerical values ​​or additional timestamps) is incorporated into the final description.

[0092] The source confidence weight is composed of two weighted components:

[0093] Basic weights: Static values ​​pre-set based on modal type and inherent equipment attributes. General rules are as follows: Structured logs (such as automated fare collection systems and equipment status registers) have the highest accuracy, with a weight of approximately 0.90-0.95; images / videos after multimodal large-scale model analysis have a weight of approximately 0.80-0.85; time-series sensors (temperature, humidity, etc.) have a weight of approximately 0.75-0.85, with sensors less affected by environmental interference having slightly higher weights; manually entered text logs or work orders have the lowest weight, approximately 0.60-0.70.

[0094] Dynamic correction factor: Adjusts the base weights based on real-time data quality. For example, if video analysis outputs a confidence score (such as a face detection score of 0.92), this score is used as a correction factor and multiplied by the base weights; if a sensor has a low signal-to-noise ratio or a recent fault record, its weight is reduced; the older the data timestamp, the lower the weight.

[0095] The basic weights are pre-configured based on expert experience, while the dynamic correction factor is determined by the acquisition quality of the current text block and the reliability of the model's inference. Specifically, for time-series data such as sensor data, the signal-to-noise ratio (SNR), data missing rate, and whether the signal exceeds a reasonable range are calculated in real time. For example, the correction factor is set to 1.0 when the SNR is higher than 20dB, 0.7 when the SNR is between 10dB and 20dB, and 0.4 when the SNR is lower than 10dB; the factor is further reduced if there are missing or abrupt data changes. For image or video data, multimodal large models typically include a confidence score when outputting natural language descriptions, such as 0.92 for object detection and 0.85 for scene classification; this score is directly used as the correction factor. For structured logs, if the log itself contains check bits or integrity markers, the correction factor is 1.0 if it is complete and error-free; it is reduced accordingly if there are format errors or missing fields. Finally, the dynamic correction factor is equal to the product of the signal quality factor and the model confidence factor, multiplied by the basic weights, to obtain the source confidence weight of the intermediate block.

[0096] Finally, the blocks are concatenated in the determined order, with appropriate transition words (such as "at the same time" and "subsequently") automatically added between adjacent blocks, and simple coreference resolution is performed (such as replacing duplicate device names with pronouns). The final output is a conflict-free, non-redundant, and fluent semantic description text for subsequent keyword extraction and graph matching.

[0097] For example, consider a scenario where "platform congestion causes slow passage through turnstiles" at a smart station. The system simultaneously collects three types of multi-source heterogeneous data: first, video surveillance images of the platform area, showing densely packed queues of passengers; second, structured logs uploaded by the automatic fare collection system turnstiles, recorded at 3:23:05 PM with the error code "gate closing timeout"; and third, IoT temperature sensor readings, showing that the temperature sensor located next to the turnstile measured 38.5 degrees Celsius at 3:23:10 PM.

[0098] First, the modal types of each data point are identified. For video images, a keyframe granularity is used, directly treating each frame as an intermediate block. For structured logs, a single event granularity is used, with each log record treated as an independent intermediate block. For time-series values, a one-minute time window granularity is used, aggregating the sensor values ​​within that window into an intermediate block. Each intermediate block embeds a context association identifier to identify its location, device, and time information, such as the camera in area C of the platform, the gate number, and the corresponding timestamp, to determine the spatiotemporal order during subsequent fusion.

[0099] Secondly, for each intermediate block, a corresponding text generation model is selected and transformed according to the modality type. For image-type intermediate blocks, a multimodal large model is called, inputting an image and prompting a description of the platform's crowd density and abnormal situation. The model outputs a natural language description: "The passenger queue length in area C of the platform exceeds the warning line, the distance between people is less than 0.5 meters, indicating a high-density congestion, and the estimated waiting time is more than three minutes." For structured log intermediate blocks, a template generation model is called to map log fields to a preset text template, outputting the description: "At 3:23:05 PM, the turnstile experienced a gate closing timeout failure." For temperature sensor value intermediate blocks, the template generation model is also used, outputting the description: "At 3:23:10 PM, the temperature sensor located next to the turnstile measured a temperature of 38.5 degrees Celsius."

[0100] Finally, the three natural language description text blocks were fused. The system determined the fusion order based on the context association identifiers embedded in each intermediate block (all pointing to the same location and close in time), arranging them chronologically. Semantic consistency checks were performed at the boundaries of adjacent text blocks, and no obvious conflicts or redundant content were found. During the fusion process, it was identified that rising temperatures might cause the gate controller to malfunction, leading to a closing timeout. Simultaneously, platform congestion causing continuous high-frequency use of the gates would exacerbate overheating. Therefore, the three text blocks were integrated according to a causal chain to generate the final semantic description text: "Around 3:23 PM, severe passenger congestion occurred in area C of platform, with queue spacing less than 0.5 meters. At the same time, the temperature of the sensor next to the gate in this area rose to 38.5 degrees Celsius, and subsequently, the gate experienced a gate closing timeout failure at 3:23:05. Based on the combined analysis, high temperature combined with high-frequency use may have led to a decrease in gate performance, while congestion further increased the pressure on the gate's passage." This semantic description text retains the core information of the original multi-source data and incorporates cross-modal association and preliminary causal reasoning, providing high-quality input for subsequent keyword extraction and graph matching.

[0101] Through the above step S10, data from different sources and with different structures, such as video, sensors, logs, and device status, can be efficiently integrated into a unified, coherent, and conflict-free semantic description text. This significantly improves cross-modal semantic alignment capabilities and data readability, providing a high-quality, low-noise input foundation for subsequent keyword extraction and graph matching in large language models. This effectively solves the pain point of difficulty in integrating and utilizing multi-source heterogeneous data in smart stations.

[0102] S20. Input the semantic description text into the trained large language model and extract local keywords, including entity keywords and relation keywords.

[0103] The entity types corresponding to entity keywords include equipment, fault, operation, personnel, location, and tool; the relationship types corresponding to relation keywords include causal, spatial, operational, and dependent. Entity attributes of entity keywords include equipment model, fault code, operation duration, and personnel qualifications; relationship attributes of relation keywords include probability of occurrence, impact level, handling timeliness, and confidence score.

[0104] See Figure 3 S20 specifically includes:

[0105] S21. Input the semantic description text into the trained large language model to perform entity boundary recognition and keyword detection, and generate a set of candidate keywords and the confidence score of each candidate keyword; the large language model is a domain large language model that has been fine-tuned using corpus from the rail transit field.

[0106] S22. Based on the terminology database of the rail transit field, standardize and map each candidate keyword in the candidate keyword set, and use the mapped standardized words as local keyword candidates.

[0107] For example, the same entity or relation may have multiple synonyms in natural language descriptions. For instance, "G12 gate" might be written as "gate number twelve" or "ticket gate number twelve," and "crowded" might be written as "crowded passenger flow" or "stuck with people." However, the node names and relation labels used in the knowledge graph are standardized and unique, therefore normalization is necessary. The terminology database stores entity synonyms (mapping various aliases to standard device codes or location codes), relation synonyms, and attribute standardization rules (such as converting "thirty-eight.5 degrees Celsius" to a standard format with a numerical value and unit). For each candidate keyword, preprocessing is performed first, and then a matching standard word is searched in the terminology database. If a match is found, it is replaced with the standard word; if a match fails, the original text is retained and marked as unmapped, awaiting manual review or further learning. After mapping, each candidate keyword is transformed into a local keyword candidate, accompanied by the standard word, original word, type, original confidence, and optional mapping confidence.

[0108] S23. Filter out local keyword candidates with confidence scores higher than a preset threshold to obtain local keywords.

[0109] In the specific implementation process, the original confidence score and the mapped confidence score (1.0 if the mapping is successful, and a lower value such as 0.5 otherwise) are weighted and fused to obtain the final confidence score. Only candidates with a final confidence score higher than a preset threshold are retained as the final local keywords. For relational keywords, it is further checked whether both the head and tail entities exist in the already screened entity keyword set. If either side is missing, the relation is discarded to ensure the integrity of the triples. This screening process can effectively filter out uncertain or erroneous extraction results from the model, avoiding noise from being introduced into the subsequent graph matching stage.

[0110] As a preferred method, during entity recognition and relation extraction, each entity and relation is also populated with corresponding attributes: entity attributes include equipment model, fault code, operation duration, and personnel qualifications; relation attributes include occurrence probability, impact level, handling timeliness, and confidence score. These attributes will serve as important features for subsequent graph matching.

[0111] Step S20 uses a large language model fine-tuned from the corpus of the rail transit field to identify entity boundaries and detect keywords. It also combines the domain terminology database to standardize and map candidate keywords, and then filters them based on confidence thresholds and relation integrity checks. This significantly improves the accuracy and domain adaptability of local keyword extraction, effectively eliminates synonym ambiguity and noise interference in natural language, and provides standardized, reliable, and high-confidence entity and relation keyword inputs for subsequent semantic-topology dual-path index matching. This enhances the information processing accuracy and robustness of the entire smart station emergency decision-making system.

[0112] S30. Based on the pre-built event knowledge graph, a semantic-topological dual-path index is established. Local keywords are used to perform fusion graph node matching to obtain matching results. The matching results include candidate abnormal event nodes and their matching scores. The dual-path index includes a semantic index based on vector retrieval and a topological index based on graph traversal. The semantic index transforms the descriptive text of entities and relationships in the event knowledge graph into dense vector storage. The topological index stores entity nodes and relationship edges in an attribute graph model and establishes a multi-level adjacency index.

[0113] In the specific construction of the dual-path index, the semantic index generates an index key for each entity and relation: the entity index key is formatted as "entity name + entity type + key attribute summary", and the relation index key is formatted as "source entity + relation type + target entity + global topic term". The index keys and their corresponding vector descriptions are stored together in a distributed vector database. The topological index establishes a three-level adjacency index: the first level is direct neighbors, supporting fast association queries; the second level is community aggregation, supporting macro-topic retrieval; and the third level is cross-subgraph paths, supporting root cause tracing reasoning.

[0114] In the specific implementation process, see Figure 4 Based on a pre-built event knowledge graph, a semantic-topological dual-path index is established. Local keywords are used to perform graph node matching, yielding matching results including:

[0115] S31. Using entity keywords in local keywords, perform entity matching in the topological index of the knowledge graph to obtain low-level candidate nodes and matching scores for each node.

[0116] S32. Convert local keywords into query vectors, perform approximate nearest neighbor search in the semantic index of the knowledge graph, calculate the cosine similarity between the query vector and the description vector of each abnormal event node, and obtain the semantic similarity score of the high-level candidate nodes and each node.

[0117] S33. Merge the low-level candidate nodes with the high-level candidate nodes, calculate the weighted sum of the matching score and semantic similarity score of the merged candidate nodes as the final matching score, and output the node with the highest final matching score as the matching result.

[0118] For example, suppose the preceding steps have extracted the following local keywords from the semantic description text: entity keywords include equipment model "M-01A05-1", fault code "F023", operation time "5 minutes", and personnel qualification "senior maintenance worker"; relation keywords include probability of occurrence "0.8", impact level "level 2", handling time "10 minutes", and confidence score "0.9".

[0119] First, a low-level search is performed: the entity keywords (i.e., entity attribute values) are used for matching in the topological index of the knowledge graph. Each entity node in the topological index is associated with specific attribute values ​​(such as equipment model, fault code, etc.). Unique identifiers are matched first. For example, using the fault code "F023" for an exact match in the entity node attribute, the corresponding fault node "door motor current fluctuation" is found, with a matching score of 1.0; the equipment model "M-01A05-1" is matched to the door motor equipment node, with a matching score of 1.0; the operation time "5 minutes" and the personnel qualification "senior repairman" are used to verify the reasonableness of the matching results (for example, as an auxiliary constraint, if the operation time of the node attribute is inconsistent, the score is reduced). The final low-level candidate node set includes "door motor current fluctuation" (matching score 1.0) and "door equipment node" (matching score 0.95), etc.

[0120] Then, a high-level retrieval is performed: all local keywords (entity attributes + relational attributes) are integrated into a query vector. For example, "F023, M-01A05-1, probability of occurrence 0.8, impact level 2" are concatenated into a text description and converted into a dense vector Q. In the semantic index of the knowledge graph, each anomalous event node has a pre-stored vector representation of its descriptive text (the description includes the typical attribute value range of this type of event). The system calculates the cosine similarity between Q and the descriptive vectors of each anomalous event node. For example, the descriptive vector of the "door motor current fluctuation" node has a similarity of 0.94 with Q, the "door cannot be closed" node has a similarity of 0.88, and the similarity of other unrelated nodes is all below 0.5. A similarity threshold of 0.7 is set to obtain a high-level candidate node set.

[0121] Finally, the nodes are merged and sorted: the union of low-level and high-level candidate nodes is taken, and a weighted sum is calculated for each candidate node as the final matching score. Assume the low-level matching score weight is 0.6 and the high-level semantic score weight is 0.4. For the "door motor current fluctuation" node: final score = 0.6 × 1.0 + 0.4 × 0.94 = 0.976; for the "door cannot close" node: final score = 0.6 × 0.5 (low-level matching was not accurate, only the device model score was low) + 0.4 × 0.88 = 0.652. The node with the highest score, "door motor current fluctuation" (0.976), is output as the matching result. If this score is higher than a high threshold (e.g., 0.9), it is directly marked as a complete match; if it is between the thresholds, it is pushed for manual review.

[0122] In this way, low-level retrieval uses precise entity attributes (fault codes, equipment models, etc.) to quickly locate the target, while high-level retrieval uses relational attributes (probability of occurrence, impact level, etc.) for semantic similarity compensation. The fusion of the two achieves high-precision matching of standardized and non-standardized emergency events.

[0123] Furthermore, in the process of merging low-level and high-level candidate nodes, dynamic weighting coefficients are used instead of fixed weights to further improve matching accuracy. The dynamic weighting coefficients specifically include:

[0124] The first dynamic weight factor is calculated based on the confidence scores of entity keywords within the local keywords. These confidence scores, derived from the preceding large language model extraction stage, reflect the reliability of attribute values ​​such as equipment model, fault code, operation time, and personnel qualifications. The first dynamic weight factor is positively correlated with this confidence score; that is, when the confidence score of the entity keyword is high (e.g., a precise match to the fault code reaching 0.95), the first factor value approaches 1; when the confidence score is low (e.g., only 0.3 due to OCR misidentification), the first factor decreases accordingly.

[0125] Next, a second dynamic weight factor is calculated based on the distribution dispersion of the cosine similarity between the query vector and the description vector of the abnormal event node. The second dynamic weight factor is negatively correlated with the distribution dispersion of the cosine similarity. If the semantic similarity of each candidate node is relatively close (small dispersion), it indicates that the high-level retrieval discrimination is insufficient, and the second factor is reduced. If the similarity is significantly different (large dispersion), the second factor is increased.

[0126] The first dynamic weight factor is used as the weight of the topology matching score, and the second dynamic weight factor is used as the weight of the semantic similarity score. The two are normalized so that the sum of the normalized weights is 1.

[0127] For example, when the fault code is precisely matched with a confidence level as high as 0.95, and the semantic similarity distribution has low dispersion, the first factor ≈ 1, the second factor ≈ 0.2, the normalized topological weight ≈ 0.83, and the semantic weight ≈ 0.17, indicating that low-level matching dominates the fusion. Conversely, if there is no explicit fault code, the entity keyword confidence level is only 0.3, and the semantic similarity distribution has high dispersion, the normalized semantic weight may rise to above 0.7. Finally, the weighted sum of each candidate node is calculated using the normalized dynamic weight: Final matching score = Normalized topological weight × Topological matching score + Normalized semantic weight × Semantic similarity score. This dynamic weight mechanism allows the fusion strategy to automatically adapt to the quality of the input data and the explicitness of the search conditions, improving the robustness and accuracy of the matching.

[0128] S40. Based on the matching results, perform hierarchical confidence assessment and status labeling to obtain the final abnormal nodes.

[0129] S40 specifically includes:

[0130] The candidate abnormal event node with the highest final matching score in the matching results is selected as the node to be evaluated, and the historical matching accuracy statistics of the node to be evaluated are extracted.

[0131] The matching score of the node to be evaluated is weighted and fused with the historical matching accuracy statistics to obtain the overall confidence level;

[0132] When the overall confidence level is higher than the first threshold, the node to be evaluated is marked as confirmed and output as the final abnormal node; when the overall confidence level is not higher than the first threshold but higher than the second threshold, it is marked as pending verification and supplementary data collection or manual review is triggered; when the overall confidence level is not higher than the second threshold, it is marked as a false alarm and discarded.

[0133] Specifically, when the overall confidence level is higher than the first threshold (e.g., 0.9), the node to be evaluated is marked as "confirmed," meaning a complete hit, and is directly output as the final abnormal node without manual intervention. When the overall confidence level is not higher than the first threshold but still higher than the second threshold (e.g., 0.7), the node is marked as "pending verification," meaning a fuzzy hit. At this time, supplementary data collection (such as retrieving sensor data from a longer time window or relevant video clips) will be triggered and pushed to the manual review interface for verification by operations personnel or domain experts. During the manual review process, experts can correct the node matching relationship or confirm fuzzy information. This feedback will be used for subsequent optimization of the knowledge graph and updates to the confidence level statistics. When the overall confidence level is not higher than the second threshold, it is marked as "false alarm," meaning a miss. At this time, it is directly discarded and will not be included in the subsequent decision-making process.

[0134] Through the above step S40, the matching results are evaluated for confidence level and marked with status. Abnormal nodes are divided into three levels: confirmed, pending verification and false alarm. This can achieve automated rapid response in high confidence scenarios, and introduce human-machine collaboration in ambiguous scenarios by triggering manual review. This effectively balances the efficiency and accuracy of emergency decision-making. At the same time, the manual feedback is used for subsequent knowledge graph optimization, which significantly reduces the false alarm rate and the risk of missed alarms.

[0135] S50. Based on the final abnormal node and the pre-built event knowledge graph, generate and output an emergency response plan for the abnormal event.

[0136] The S50 specifically includes:

[0137] Extract the first-order neighborhood information of the final abnormal node from the knowledge graph. The first-order neighborhood information includes the attribute relationships of the device nodes, personnel nodes, contingency plan nodes, and connecting edges directly connected to the final abnormal node.

[0138] First-order neighborhood information is concatenated with semantic description text to form a solution generation prompt; the solution generation prompt is then input into a large language model to generate an emergency response plan.

[0139] Specifically, starting from the final abnormal node, a one-hop traversal is performed in the topological index of the knowledge graph to obtain the following first-order neighborhood information:

[0140] Directly connected device nodes: Extract device entities that are physically or logically related to the abnormal event through relationships such as associated devices and systems. Each device node carries its attributes (such as device model, location code, and maintenance cycle).

[0141] Directly connected personnel nodes: Extract personnel role entities with disposal authority or qualification requirements through relationships such as those responsible for processing and reporting, including job title, qualification level, and contact information.

[0142] Directly connected contingency plan nodes: Predefined response plan entities are extracted through relational edges such as reference contingency plans and applicable standards. Each contingency plan node contains structured operation process steps, tool list, and safety instructions.

[0143] Attribute relationships of connecting edges: Extract the key metadata stored in all the above relationship edges, mainly including: impact level (e.g., level 1 to 5), handling time (e.g., within 5 minutes), historical confidence (the frequency with which the relationship has been successfully used in the past), required tool encoding, reporting hierarchy chain, etc.

[0144] This first-order neighborhood information is organized into a temporary context object, in which equipment, personnel, and contingency plan nodes are presented with their standard names and core attributes, while relation edge attributes are attached to the corresponding connections in the form of key-value pairs.

[0145] Next, the semantic description text generated in the preceding steps is structurally concatenated with the temporary context object to form the input prompt for the large language model. The specific construction method is as follows:

[0146] First, a fixed prefix is ​​used to clearly define the task: "You are an emergency decision-making assistant for a smart station. Please generate a response plan based on the following fault description and its knowledge graph context." Then, semantic description text is embedded. Next, under the title "Knowledge Graph Matching Results," first-order neighborhood information is listed item by item: device node name and key attributes, personnel roles that need to be reported / executed, names and steps of referenced contingency plans, and key parameters of relationship edges (impact level, response time, etc.). Finally, an instruction suffix is ​​added: "Please output a response plan containing four parts: [Operation Steps], [Tool List], [Reporting Process], and [Precautions]. The execution order and expected effect of each step must be clearly stated."

[0147] The prompt string is converted into the model input format, which then calls a pre-trained large language model. After the model generates natural language text, post-processing verification is performed: the output is checked to see if it covers the four required parts; if any are missing, a secondary completion prompt is triggered; if there are obvious contradictions (such as a mismatch between the tool list and the contingency plan), semantic verification and correction are performed using tool nodes in the graph. The final emergency response plan is output in structured text format to the human-computer interaction interface or directly pushed to the terminals of relevant personnel.

[0148] Through the above detailed extraction and generation process, static knowledge graph knowledge is transformed into dynamic, executable, and traceable emergency decision-making instructions, realizing the automated generation of "knowledge-driven" solutions.

[0149] In addition, in the specific implementation process, the smart station emergency decision-making method based on multimodal large model and knowledge graph of this application also includes steps such as entity deduplication and graph fusion, abnormal pre-event mining, dynamic updating of knowledge graph and continuous iterative maintenance.

[0150] Specifically, entity deduplication and graph fusion include:

[0151] During the initial construction or incremental update of the knowledge graph, a three-layer deduplication and fusion strategy is adopted to address duplicate descriptions of the same entity in multi-source data:

[0152] First layer: Name standardization. Establish a dictionary of rail transit terminology and unify the mapping of synonyms. For example, "door controller", "door control unit", and "EDCU" are all mapped to the same standard name.

[0153] The second layer: attribute similarity calculation. For candidate entity pairs, the cosine similarity of their attribute vectors is calculated. If the similarity is greater than a preset threshold (e.g., 85%), they are merged, retaining the most complete attribute set.

[0154] The third layer: topology verification. Check the overlap of the neighbor sets of candidate entities. If the overlap rate of the neighbors is greater than a threshold (e.g., 80%), they are confirmed as the same entity, and the connecting edges are merged.

[0155] By employing a three-layer progressive verification method, the problem of duplicate descriptions of the same entity in multi-source data can be effectively solved.

[0156] Furthermore, the mining of anomaly precursor events specifically includes:

[0157] Using the final anomalous node as the anchor point, a reverse path traversal is performed in the knowledge graph along the incoming edges. The traversal direction follows the "incoming edges" pointing to the current anomalous node, focusing only on relational edges with the current anomalous node as the tail (target) and ignoring outgoing edges pointing to other nodes from this node. A set of candidate predecessor nodes that have causal, pre-existing, or accompanying relationships with the final anomalous node is extracted. Optionally, the traversal depth is controlled by preset parameters, typically starting with a one-hop search (direct predecessor node). If not enough candidate nodes are found, the traversal is recursively expanded to two or three hops, but infinite loops must be avoided. The traversal algorithm combines breadth-first search and depth-first search: breadth-first search comprehensively covers potential predecessor events on different paths, while depth-first search traces the root cause on a single causal chain. After the traversal, all nodes satisfying the relational type conditions are collected, forming a set of candidate predecessor nodes. Each candidate node includes its node type (e.g., device malfunction, state change, operational behavior), attribute characteristics (e.g., threshold, duration, frequency), and distance (hop count) from the anomalous node in the graph.

[0158] The system acquires real-time data streams within a preset time window before the occurrence of the current abnormal event, aligns the real-time data streams with the attribute features of each node in the candidate preceding node set in terms of time sequence, and evaluates the pattern matching degree between the real-time data streams and each candidate preceding node using a large language model.

[0159] Specifically, the process involves extracting all multi-source real-time data streams from historical data storage or real-time data pipelines within a preset time window (e.g., five minutes) preceding the occurrence of the current anomaly. These data streams include time-series values ​​from device sensors related to the anomaly (e.g., motor current, temperature), device status logs (e.g., gate opening / closing records), and video analysis results. Subsequently, the extracted real-time data streams are time-aligned with the attribute features of each node in the candidate preceding node set. The alignment process includes unifying the timestamps in the data streams to the same timeline, using the anomaly occurrence time as the baseline zero point, and calculating the specific time point or time period within the current time window where the characteristic pattern described by each candidate node appears. For example, if a candidate preceding node is described as "motor current fluctuating by more than 10% three times consecutively," then segments matching this pattern (fluctuation amplitude, number of consecutive occurrences, duration) are searched in the real-time current data. After alignment, a large language model is used to evaluate the pattern matching degree of each candidate node. The evaluation inputs include: a natural language description of the candidate node (e.g., "motor current fluctuates by more than 10% three times consecutively"), the standardized feature parameters corresponding to the node (threshold, window length, number of consecutive fluctuations), and the aligned real-time data segment. The large language model outputs a semantic matching score (e.g., between 0 and 1), representing the similarity between the current data stream and the preceding event pattern. The model can also output qualitative judgments (e.g., "highly matched," "partially matched," "not matched") and the matching criteria.

[0160] Based on the graph distance and pattern matching degree of the reverse path traversal, the confidence score of each candidate predecessor node is obtained. Candidate predecessor nodes with confidence scores higher than a preset threshold are confirmed as valid predecessor events. Confirmed valid predecessor events will enter the subsequent knowledge graph dynamic update process.

[0161] Knowledge graph updates specifically include:

[0162] For each candidate precursor node confirmed as a valid precursor event, check whether there is already a node in the knowledge graph with a semantic similarity higher than the preset repetition threshold; if not, instantiate a new precursor warning node in the knowledge graph, with node attributes including feature description, key parameters (threshold, number of consecutive events, etc.), creation time, and initial trigger count; if it exists, reuse the existing node and update the trigger count attribute.

[0163] Next, directed edges are established from the early warning nodes to the final abnormal nodes. Vector embedding calculations and adjacency list updates are then performed on the newly added early warning nodes and their corresponding directed edges to obtain the updated knowledge graph. The entire update process uses an incremental approach to ensure real-time evolution of the knowledge graph while minimizing computational overhead. For distance details, see [link to documentation]. Figure 6 and Figure 7 , Figure 6 This is the knowledge graph before the update. Figure 7 This is the updated knowledge graph.

[0164] During knowledge graph updates, conflict detection is performed on newly extracted entities and relationships. Specifically, this includes checking whether new entities have the same name but different meanings or synonyms with existing entities; and checking whether new relationships contradict existing relationships (e.g., both "A causes B" and "B causes A" exist simultaneously). If a conflict is detected, the system does not automatically merge the relationships; instead, it triggers a manual review module, pushing the conflicting entities or relationships to a human-computer collaboration interface for judgment and correction by domain experts. Merging is only performed after the review is approved. Conflicting parts are directly incorporated into the knowledge graph.

[0165] After the knowledge graph is dynamically updated, the monitoring rule instantiation operation is further performed. Based on the attributes of the newly generated advance warning nodes (such as feature description, threshold, consecutive count, and time window), they are automatically converted into monitoring rules that can be recognized by the real-time data stream engine. The conversion methods include: converting natural language descriptions (such as motor current fluctuations exceeding 10% for three consecutive times) into structured query language or rule expressions and registering them with the rule engine. Subsequently, during the daily operation of the station, once the real-time data stream matches the rule, an early warning is immediately triggered, and proactive preventive information is output (such as it is expected that the door may fail to close in ten minutes, please check in advance), without waiting for the fault to actually occur.

[0166] Furthermore, the intelligent station emergency decision-making method based on multimodal large models and knowledge graphs in this application also includes:

[0167] Establish an effect tracking log for each early warning node, recording the following indicators: number of triggers, number of true positives (an anomaly actually occurred after triggering or a potential risk was confirmed by manual intervention), number of false positives (no anomaly occurred after triggering and no potential risk was found during manual investigation), and number of successful preventions (anomalies were successfully avoided after intervention following triggering).

[0168] Based on the tracking results, the weight of the relationship edge between the early warning node and the abnormal event node in the graph is dynamically adjusted: if the prevention is successful or it is confirmed as a true positive, the edge weight is increased by w (e.g., by 0.05 each time), which improves the priority of the early warning event in subsequent reasoning; if false positives occur continuously, the edge weight is reduced by w. When the weight is lower than the threshold (e.g., 0.3), the edge is automatically marked as invalid or triggers manual review.

[0169] For preceding events with high false positive rates, historical data is re-analyzed using a large language model to optimize the judgment threshold (e.g., adjusting the current fluctuation threshold from 10% to 12%). Regular pruning operations are performed to remove long-term invalid or redundant nodes and edges, keeping the knowledge graph concise and efficient.

[0170] Through a closed loop of "anomaly occurrence → pre-emptive detection → map update → early warning → feedback optimization", the system achieves the evolutionary effect of continuously improving fault coverage, gradually increasing early warning time, and continuously reducing operation and maintenance costs.

[0171] In summary, see Figure 5 This application proposes a smart station emergency decision-making method based on a multimodal large model and knowledge graph. It unifies multi-source heterogeneous data into semantic description text through semantic segmentation and text generation models. It extracts high-confidence entity and relation keywords using a domain-fine-tuned large language model and achieves accurate graph node matching based on semantic-topological dual-path indexing and dynamic weight fusion. Combining hierarchical confidence assessment and human-machine collaboration, it automatically generates structured emergency response plans. Simultaneously, it mines pre-events through reverse path traversal and the large language model, instantiates them as early warning nodes, and automatically generates monitoring rules, realizing a shift from passive response to proactive early warning. Furthermore, it dynamically adjusts the weights and trigger thresholds of graph edges based on the early warning true positive rate feedback, forming a closed-loop intelligent operation and maintenance system of "anomaly occurrence → pre-event mining → graph evolution → proactive early warning → feedback self-optimization," significantly improving the timeliness, accuracy, and adaptability of emergency response.

[0172] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0173] In the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0174] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make modifications, alterations, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A smart station emergency decision-making method based on multimodal large models and knowledge graphs, characterized in that, include: The system acquires multi-source heterogeneous data from smart stations and uses a semantic segmentation strategy and text generation model to transform the multi-source heterogeneous data into semantic description text. The semantic description text is input into a trained large language model to extract local keywords, which include entity keywords and relation keywords. Based on a pre-built event knowledge graph, a semantic-topological dual-path index is established. The local keywords are used to perform fusion-based graph node matching to obtain matching results. The matching results include candidate abnormal event nodes and their matching scores. The dual-path index includes a semantic index based on vector retrieval and a topological index based on graph traversal. The semantic index transforms the descriptive text of entities and relationships in the event knowledge graph into dense vector storage, and the topological index stores entity nodes and relationship edges using an attribute graph model and establishes a multi-level adjacency index. Based on the matching results, a graded confidence assessment and status labeling are performed to obtain the final abnormal nodes; Based on the final abnormal node and the pre-built event knowledge graph, an emergency response plan for the abnormal event is generated and output.

2. The intelligent station emergency decision-making method based on multimodal large model and knowledge graph as described in claim 1, characterized in that, The process of transforming the multi-source heterogeneous data into semantic descriptive text using a semantic segmentation strategy and a text generation model includes: Identify the modality type of each data in the multi-source heterogeneous data, dynamically select the corresponding semantic segmentation granularity according to the modality type, and divide each data into one or more intermediate blocks according to the semantic segmentation granularity; embed context association identifiers within the intermediate blocks; For each intermediate block, a corresponding text generation model is selected according to the modality type to convert the intermediate block into a natural language descriptive text block; the text generation model includes a multimodal large model for processing images, a large language model for processing text, and a template generation model for processing structured data; The natural language description text blocks are merged to generate semantic description text.

3. The intelligent station emergency decision-making method based on multimodal large model and knowledge graph as described in claim 2, characterized in that, The process of fusing the natural language description text blocks to generate semantic description text specifically includes: Based on the context association identifier, the fusion order of each natural language description text block is determined; Semantic consistency is checked in the boundary region of adjacent natural language description text blocks. If there are conflicts or redundant content, disambiguation or merging is performed according to the source confidence weight of the context association identifier to obtain the disambiguation and merging result. Based on the fusion order and disambiguation merging result, the natural language description text blocks are concatenated to generate the semantic description text.

4. The intelligent station emergency decision-making method based on multimodal large model and knowledge graph as described in claim 1, characterized in that, Inputting the semantic description text into a trained large language model to extract local keywords specifically includes: The semantic description text is input into the trained large language model for entity boundary recognition and keyword detection, generating a set of candidate keywords and a confidence score for each candidate keyword; the large language model is a domain-specific large language model fine-tuned using corpus from the rail transit domain. Based on the terminology database of the rail transit field, each candidate keyword in the candidate keyword set is standardized and mapped, and the mapped standardized words are used as local keyword candidates. Select local keyword candidates with confidence scores higher than a preset threshold to obtain local keywords.

5. The intelligent station emergency decision-making method based on multimodal large model and knowledge graph as described in claim 1, characterized in that, Based on the semantic-topological dual-path index established by the pre-constructed event knowledge graph, the local keywords are used to perform fusion-based graph node matching, and the matching results include: Using the entity keywords in the local keywords, entity matching is performed in the topological index of the knowledge graph to obtain low-level candidate nodes and matching scores for each node; The local keywords are converted into query vectors, and an approximate nearest neighbor search is performed in the semantic index of the knowledge graph. The cosine similarity between the query vector and the description vector of each abnormal event node is calculated to obtain the high-level candidate nodes and the semantic similarity score of each node. The low-level candidate nodes are fused with the high-level candidate nodes. The weighted sum of the matching score and semantic similarity score of the fused candidate nodes is calculated as the final matching score. The node with the highest final matching score is output as the matching result.

6. The intelligent station emergency decision-making method based on multimodal large model and knowledge graph as described in claim 5, characterized in that, When fusing the low-level candidate nodes with the high-level candidate nodes, a dynamic weighting coefficient is used, which specifically includes: A first dynamic weight factor is calculated based on the confidence score of entity keywords in the local keywords, and a second dynamic weight factor is calculated based on the distribution dispersion of the cosine similarity between the query vector and the description vector of the abnormal event node; wherein, the first dynamic weight factor is positively correlated with the confidence score of entity keywords, and the second dynamic weight factor is negatively correlated with the distribution dispersion of the cosine similarity. The first dynamic weight factor is used as the weight of the topology matching score, and the second dynamic weight factor is used as the weight of the semantic similarity score. The two are normalized so that the sum of the normalized weights is 1.

7. The intelligent station emergency decision-making method based on multimodal large model and knowledge graph as described in claim 1, characterized in that, Based on the matching results, confidence assessment and status labeling are performed to obtain the final abnormal nodes, specifically including: The candidate abnormal event node with the highest final matching score in the matching results is selected as the node to be evaluated, and the historical matching accuracy statistics of the node to be evaluated are extracted. The matching score of the node to be evaluated is weighted and fused with the historical matching accuracy statistics to obtain the comprehensive confidence level; When the overall confidence level is higher than the first threshold, the node to be evaluated is marked as confirmed and output as the final abnormal node; when the overall confidence level is not higher than the first threshold but higher than the second threshold, it is marked as pending verification and supplementary data collection or manual review is triggered; when the overall confidence level is not higher than the second threshold, it is marked as a false alarm and discarded.

8. The intelligent station emergency decision-making method based on multimodal large model and knowledge graph as described in claim 1, characterized in that, Based on the final abnormal node and the pre-built event knowledge graph, the emergency response plan for the abnormal event is generated and output, specifically including: Extract the first-order neighborhood information of the final abnormal node from the knowledge graph. The first-order neighborhood information includes the attribute relationships of the device nodes, personnel nodes, contingency plan nodes, and connecting edges directly connected to the final abnormal node. The first-order neighborhood information is concatenated with the semantic description text to form a solution generation prompt; the solution generation prompt is input into a large language model to generate an emergency response plan.

9. The intelligent station emergency decision-making method based on multimodal large model and knowledge graph as described in claim 1, characterized in that, The method further includes: simultaneously generating an emergency response plan and performing abnormal pre-event mining and dynamic knowledge graph updates, specifically including: Using the final abnormal node as the anchor point, a reverse path traversal is performed along the incoming edge direction in the knowledge graph to extract a set of candidate predecessor nodes that have causal, predecessor, or companion relationships with the final abnormal node. The real-time data stream within a preset time window before the occurrence of the current abnormal event is obtained. The real-time data stream is time-aligned with the attribute features of each node in the candidate preceding node set, and the pattern matching degree between the real-time data stream and each candidate preceding node is evaluated by a large language model. Based on the graph distance traversed by the reverse path and the degree of pattern matching, the confidence score of each candidate preceding node is obtained, and the candidate preceding node with a confidence score higher than a preset threshold is confirmed as a valid preceding event.

10. The intelligent station emergency decision-making method based on multimodal large model and knowledge graph according to claim 9, characterized in that, The knowledge graph update includes: For each candidate precursor node confirmed as a valid precursor event, check whether there is already a node in the knowledge graph with a semantic similarity higher than a preset repetition threshold; if not, instantiate a new precursor warning node in the knowledge graph; if it exists, reuse the existing node and update the trigger count attribute. Establish directed edges from the pre-warning nodes to the final abnormal nodes, and perform vector embedding calculations and adjacency list updates on the newly added pre-warning nodes and their corresponding directed edges to obtain the updated knowledge graph.

Images