A multimodal urban rail intelligent agent question-answering system

Abstract

This invention discloses a multimodal intelligent agent question-answering system for urban rail transit, relating to the field of urban rail transit operation and maintenance technology. The method includes: collecting train dispatch communication voice data, train event log data, maintenance regulation text data, and historical handling work order data, and unifying the timestamps; standardizing and mapping the terminology of the speech transcription results; establishing a temporal correlation between voice and event logs based on a sliding time window; constructing an evidence chain set containing voice segment identifiers, event log numbers, and regulatory clause references, and filtering the evidence chain through a multi-factor scoring mechanism; calling a preset tool interface to generate handling suggestions or work order drafts when trigger conditions are met; and simultaneously generating audit logs to achieve full-process traceability. This invention achieves temporal correlation fusion of multimodal data and structured evidence support, improving the accuracy, interpretability, and security of the urban rail question-answering system.

Description

Technical Field

[0001] This invention relates to the field of urban rail transit operation and maintenance technology, and in particular to a multimodal urban rail intelligent agent question-and-answer system. Background Technology

[0002] Urban rail transit systems are characterized by complex operation organization, a wide variety of equipment, highly specialized dispatching instructions, and high safety requirements. During daily operation and maintenance, train dispatchers, station staff, and equipment maintenance personnel frequently need to review regulations, procedures, and historical cases to quickly assess and handle train anomalies, equipment alarms, and emergencies.

[0003] In existing technologies, urban rail transit systems typically support on-site question-and-answer and decision-making in the following ways: first, by manually reviewing paper or electronic regulations; second, by relying on operation and maintenance knowledge bases or retrieval systems for keyword searches; and third, by generating text answers based on simple voice question-and-answer or knowledge graph systems.

[0004] However, the above methods generally suffer from the following problems: dispatching communication voice messages, event logs, regulations, and historical handling records are usually scattered across different systems, lacking a unified correlation mechanism, especially making it difficult to automatically align the communication voice content with the event logs. Dispatch communication voice messages contain a large number of colloquial expressions, abbreviations, and non-standard terms, making accurate matching difficult through direct text retrieval. Existing question-and-answer systems typically respond based on text matching, failing to dynamically correlate event occurrence time with voice message time, resulting in a lack of specificity in the question-and-answer results. Question-and-answer results are mostly directly generated text, without providing clear voice segments, event numbers, or regulatory references, failing to meet the traceability requirements of the urban rail transit industry. Urban rail transit production control networks and information networks are usually physically or logically isolated, making it difficult for general question-and-answer systems to operate securely in isolated environments. Therefore, this paper proposes a multimodal urban rail transit intelligent agent question-and-answer system. Summary of the Invention

[0005] The main objective of this invention is to provide a multimodal urban rail intelligent agent question-answering system that can effectively solve the problems in the background art.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A multimodal urban rail intelligent agent question-answering system includes the following steps:

[0008] S1. Collect multimodal data, which includes at least vehicle dispatch communication voice data, vehicle event log data, operation and maintenance regulations text data, and historical handling work order data, and perform timestamp unification and data source identification on the multimodal data;

[0009] S2. The train dispatch communication voice data is transcribed into speech, and the station names, section names, equipment numbers, operation passwords and fault phenomenon words in the transcribed text are normalized and encoded based on the terminology standard library to form a unified terminology sequence.

[0010] S3. Construct a sliding time window association relationship based on the timestamps of voice segments and event logs, generate a cross-modal time alignment index, and bind the alignment index to the unified terminology sequence;

[0011] S4. Map and match the unified terminology sequence with the driving event code, and combine the references to the operation and maintenance regulations and similar records of historical handling work orders to construct an evidence chain set to support the question and answer output;

[0012] S5. Generate question and answer results based on the evidence chain set, and call the preset tool interface to output disposal suggestions or generate a work order draft when the preset trigger conditions are met.

[0013] S6. Generate an audit log, which includes at least the question and answer request identifier, trigger time, input summary, cited evidence identifier, output summary, tool call record and result write-back identifier, so as to achieve full traceability.

[0014] Furthermore, in step S2, when performing semantic matching between the speech-transcribed text and entries in the terminology standard library, a joint similarity calculation model is used for term mapping. The joint similarity function is:

[0015]

[0016] in, Transcribe speech into text; Candidate terms from the terminology standard library; A function for representing text vectors; Cosine similarity; To normalize the edit distance function; , Let be the weight parameters, and satisfy ? When the joint similarity is greater than a preset threshold, the speech-to-text is mapped to the corresponding standard term encoding.

[0017] Furthermore, the sliding time window constructed in step S3 includes at least: window start time, window end time, and window type; wherein the window type includes at least "alarm pre-window", "alarm synchronization window", and "alarm post-window"; and the correlation between the voice segment and the event log is calculated by a time correlation function. The time correlation decreases monotonically as the difference between the timestamp of the voice segment and the timestamp of the event log increases. When the correlation is greater than a preset threshold, a cross-modal correlation between the voice segment and the event log is established, and the correlation result is written into the cross-modal time alignment index.

[0018] Furthermore, in step S4, the generated chain of evidence is scored for credibility, and the credibility scoring function is:

[0019]

[0020] in, The temporal correlation between audio clips and event logs; For term matching similarity; Similarity to historical disposal records; , , Let be the weight parameters, and satisfy ? The evidence chains are sorted according to the credibility score, and the evidence chain with the highest score is selected as the main reference for the question and answer output.

[0021] Furthermore, the tool interface calls in step S5 satisfy the whitelist mechanism, which at least limits: the types of tools that can be called include alarm query, event tracing query, handling process query and work order draft generation; calling interfaces that write driving control parameters or execute control commands is prohibited; the tool call triggering conditions at least include: the question and answer intent belongs to the handling request, the risk level is not lower than the threshold or the evidence chain score is not lower than the threshold; and after calling the tool interface, the input parameter summary, return result summary and interface return code of the tool call are written into the audit log.

[0022] Furthermore, when generating the question-and-answer results in step S5, the output content includes at least: a conclusive answer; a list of suggested steps for handling; risk level identifiers and handling priority identifiers; and a set of identifiers for the cited evidence chain. The risk level is determined comprehensively by the event level factor, the frequency factor of similar historical events, and the risk label factor of regulations and clauses. When the risk level reaches a preset level, the "escalation reporting target" and "suggested contact position" information are automatically added.

[0023] A multimodal urban rail transit intelligent agent question-answering system includes:

[0024] The multimodal data acquisition module is used to collect and identify train dispatch communication voice data, train event log data, operation and maintenance regulations text data and historical handling work order data, and to complete the timestamp unification;

[0025] The speech-semantic normalization module is used for speech transcription and terminology normalization encoding, and outputs a unified terminology sequence;

[0026] The time series alignment module is used to construct sliding time windows and generate cross-modal time alignment indexes;

[0027] The evidence chain construction module is used to complete term-event mapping, rule and clause references, and association with similar work order records, and output a set of evidence chains;

[0028] The intelligent agent decision-making module is used to generate question-and-answer results based on the evidence chain set and to call the preset tool interface when the triggering conditions are met.

[0029] The security audit module is used to generate and store audit logs.

[0030] Furthermore, the system is deployed at the edge of the urban rail production control network and receives external knowledge update data through a one-way data transmission interface; the knowledge update data includes at least the version of the regulation text, the version of the terminology standard library, and the version of the handling process template; and the security audit module records the update source identifier, version number, update time, and effective scope identifier for each knowledge update to meet security isolation and audit requirements.

[0031] Furthermore, the audit logs generated by the security audit module include at least: the user identifier and role permission identifier of the question and answer initiator; the question and answer input summary and intent classification result; the set of evidence chain identifiers and evidence chain score; the output content summary, risk level and handling priority; tool call records, return result summary and write-back status; and the audit logs are verified for tamper-proofing using hash chain or digital signature, and support traceability retrieval by event number or work order number.

[0032] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method steps.

[0033] The present invention has the following beneficial effects:

[0034] 1. Compared with existing technologies, this solution constructs a sliding time window association mechanism between voice segments and driving event logs, enabling communication voice, event logs and regulations to form a unified index in the time dimension, thereby improving the relevance and accuracy of question and answer results.

[0035] 2. Compared with existing technologies, this solution establishes a terminology standard library and adopts a joint similarity matching mechanism to achieve the mapping between colloquial expressions and standard terms, thereby reducing the impact of speech transcription errors on question-and-answer results.

[0036] 3. Compared with existing technologies, this solution forms a structured chain of evidence by associating voice segment identifiers, event log numbers, regulatory clause references, and historical handling records, so that the question and answer results have clear source evidence and enhance the interpretability and verifiability of the system.

[0037] 4. Compared with existing technologies, this solution, by deploying at the edge of the production control network and combining whitelist tool calls and audit log recording mechanisms, enhances decision support capabilities while ensuring the security and auditability of system operation. Attached Figure Description

[0038] Figure 1 This is a system module diagram of a multimodal urban rail intelligent agent question-answering system according to the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0040] Example 1

[0041] This invention provides a multimodal urban rail intelligent agent question-answering method and system, which is deployed in the edge server environment of the urban rail production control network. It is used to uniformly process and integrate multi-source data such as train dispatch communication voice, train event logs, operation and maintenance regulations texts and historical handling work orders, so as to generate question-answering results with evidence support and traceability.

[0042] During the data acquisition phase, the system accesses dispatch communication voice data, vehicle event log data, maintenance regulation text data, and historical work order data through a multimodal data acquisition module. To ensure accurate correlation between different data sources, this embodiment timestamps all real-time data at the acquisition entry point using a unified system clock and converts it to a unified time format for storage. For historical text data, since some time information is embedded in the text description, the system extracts date, time, and interval information from the text using regular expression matching or natural language processing technology and adds standard timestamps, ensuring consistency across different data sources in terms of time.

[0043] During the speech processing stage, the system transcribes the dispatch communication speech and inputs the transcription results into the speech semantic normalization module. This module pre-constructs a terminology standard library, which includes a site name mapping table, a section number mapping table, a device level encoding table, a dispatch password phrase table, and a fault phenomenon synonym table. This standard library is formed through historical data statistical analysis and expert experience compilation and supports periodic updates. To improve the accuracy of terminology matching, this embodiment uses a joint similarity model to match the transcribed speech with entries in the terminology standard library. The joint similarity function is:

[0044]

[0045] in For speech-to-text transcription, Candidate terms, For text vector representation functions, The cosine similarity function is used. To normalize the edit distance function, and Let be the weight parameters, and satisfy ? When the joint similarity exceeds a preset threshold, the transcribed text is mapped to the corresponding standard terminology encoding. Actual testing shows that the joint similarity model significantly improves terminology matching accuracy compared to a single similarity matching method when dealing with colloquial expressions and synonym substitution scenarios, thereby reducing false matches.

[0046] During the time alignment phase, the system constructs a sliding time window mechanism based on the timestamps of the voice segments and the driving event logs. This window includes an alarm pre-window, an alarm synchronization window, and an alarm post-window. Cross-modal time alignment is achieved by calculating the temporal correlation between the voice segments and the event logs within different time intervals. The temporal correlation monotonically decreases as the time difference between the two increases. When the correlation exceeds a preset threshold, a correlation is established between the voice segment and the corresponding event log, and the correlation result is written into the cross-modal time alignment index table, providing a foundation for subsequent evidence chain construction.

[0047] In the mapping and evidence chain construction phase, the system pre-builds a mapping rule base between terms and event codes. This rule base is established based on historical event log statistics and expert rule configurations. For example, standard terms such as "signal malfunction" and "section red light strip" are associated one-to-one with their corresponding event codes. The system matches the normalized term sequence with the traffic event codes and constructs a structured evidence chain by combining regulatory clause references and similar records of historical handling work orders. To improve the reliability of evidence selection, this embodiment uses a multi-factor weighted scoring mechanism to rank the evidence chain. The scoring function is as follows:

[0048] in For time correlation, For term matching similarity, For the similarity of historical disposal records, , , Let be the weight parameters, and satisfy ? By scoring and ranking the chains of evidence, the highest-scoring chain is selected as the primary basis for the question-and-answer output, thus avoiding judgment bias caused by a single factor.

[0049] During the question-and-answer generation and tool invocation phase, the intelligent agent decision-making module generates question-and-answer results based on the filtered evidence chain and invokes preset tool interfaces when preset trigger conditions are met. These trigger conditions include at least the user's question-and-answer intent being identified as a handling request, the current event risk level exceeding a preset threshold, or the evidence chain credibility score exceeding a threshold. The risk level is determined comprehensively by an event level factor, a historical frequency factor for similar events, and a regulatory clause risk label factor. To ensure system operational security, tool interface invocation employs a whitelist mechanism, allowing only alarm queries, event tracing queries, handling process queries, and work order draft generation; writing to vehicle control parameters or executing control commands is prohibited.

[0050] During the audit phase, the security audit module records the entire question-and-answer process. The audit log includes the user identifier that initiated the question-and-answer session, the question-and-answer trigger time, the input summary, the identifier of the cited evidence, the output summary, and the tool call records. Verification values ​​are generated through hash chains or digital signatures to achieve log tamper-proofing and traceability.

[0051] Through the coordinated operation of the above-mentioned multimodal data unified processing, terminology normalization and matching, time alignment, evidence chain scoring and security audit control, this invention can provide Q&A results with clear evidence support, risk warnings and disposal suggestions in urban rail transit operation and maintenance scenarios, thereby improving on-site disposal efficiency and meeting industry safety and audit requirements.

[0052] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal question-answering method for urban rail transit intelligent agents, characterized in that, Includes the following steps: S1. Collect multimodal data, which includes at least vehicle dispatch communication voice data, vehicle event log data, operation and maintenance regulations text data, and historical handling work order data, and perform timestamp unification and data source identification on the multimodal data; S2. The train dispatch communication voice data is transcribed into speech, and the station names, section names, equipment numbers, operation passwords and fault phenomenon words in the transcribed text are normalized and encoded based on the terminology standard library to form a unified terminology sequence. In step S2, when semantically matching the speech-transcribed text with entries in the terminology standard library, a joint similarity calculation model is used for term mapping. The joint similarity function is: in, Transcribe speech into text; Candidate terms from the terminology standard library; A function for representing text vectors; Cosine similarity; To normalize the edit distance function; , Let be the weight parameters, and satisfy ? When the joint similarity is greater than a preset threshold, the speech-to-text is mapped to the corresponding standard term encoding. S3. Construct a sliding time window association relationship based on the timestamps of voice segments and event logs, generate a cross-modal time alignment index, and bind the alignment index to the unified terminology sequence; The sliding time window constructed in step S3 includes at least: window start time, window end time, and window type; wherein the window type includes at least "alarm pre-window", "alarm synchronization window", and "alarm post-window"; and the correlation between the voice segment and the event log is calculated by a time correlation function. The time correlation decreases monotonically as the difference between the timestamp of the voice segment and the timestamp of the event log increases. When the correlation is greater than a preset threshold, a cross-modal correlation between the voice segment and the event log is established, and the correlation result is written into the cross-modal time alignment index. S4. Map and match the unified terminology sequence with the driving event code, and combine the references to the operation and maintenance regulations and similar records of historical handling work orders to construct an evidence chain set to support the question and answer output; In step S4, the generated chain of evidence is scored for credibility. The credibility scoring function is as follows: in, The temporal correlation between audio clips and event logs; For term matching similarity; Similarity to historical disposal records; , , Let be the weight parameters, and satisfy ? The evidence chains are sorted according to the credibility score, and the evidence chain with the highest score is selected as the main reference for the question and answer output. S5. Generate question and answer results based on the evidence chain set, and call the preset tool interface to output disposal suggestions or generate a work order draft when the preset trigger conditions are met. S6. Generate an audit log, which includes at least the question and answer request identifier, trigger time, input summary, cited evidence identifier, output summary, tool call record and result write-back identifier, so as to achieve full traceability.

2. The urban rail intelligent agent question-answering method integrating multimodal approaches according to claim 1, characterized in that, The tool interface calls in step S5 satisfy the whitelist mechanism, which at least limits the types of tools that can be called to include alarm query, event tracing query, handling process query and work order draft generation. Calling the interface that writes vehicle control parameters or executes control commands is prohibited; The conditions for triggering a tool call must include at least the following: the question and answer intent is a disposal request, the risk level is not lower than the threshold, or the evidence chain score is not lower than the threshold. Furthermore, after calling the tool interface, the summary of the input parameters, the summary of the return result, and the interface return code are written to the audit log.

3. The urban rail intelligent agent question-answering method integrating multimodal approaches according to claim 1, characterized in that, When generating the question and answer results in step S5, the output content shall include at least: a conclusive answer; a list of suggested steps for handling; risk level identifiers and handling priority identifiers; and a set of identifiers for the chain of evidence. The risk level is determined by a combination of event level factors, frequency factors of similar historical events, and risk label factors of regulations and clauses. When the risk level reaches the preset level, the information of "upgrade reporting target" and "suggested contact position" shall be automatically added.

4. A multimodal urban rail intelligent agent question-answering system, characterized in that, include: The multimodal data acquisition module is used to collect and identify train dispatch communication voice data, train event log data, operation and maintenance regulations text data and historical handling work order data, and to complete the timestamp unification; The speech-semantic normalization module is used for speech transcription and terminology normalization encoding, and outputs a unified terminology sequence; When performing semantic matching between the speech-transcribed text and entries in the terminology standard library, a joint similarity calculation model is used for term mapping. The joint similarity function is: in, Transcribe speech into text; Candidate terms from the terminology standard library; A function for representing text vectors; Cosine similarity; To normalize the edit distance function; , Let be the weight parameters, and satisfy ? When the joint similarity is greater than a preset threshold, the speech-to-text is mapped to the corresponding standard term encoding. The time series alignment module is used to construct sliding time windows and generate cross-modal time alignment indexes; The constructed sliding time window includes at least: window start time, window end time, and window type; wherein the window type includes at least "alarm pre-window", "alarm synchronization window", and "alarm post-window"; and the correlation between the voice segment and the event log is calculated through a time correlation function. The time correlation decreases monotonically as the difference between the timestamp of the voice segment and the timestamp of the event log increases. When the correlation is greater than a preset threshold, a cross-modal correlation between the voice segment and the event log is established, and the correlation result is written into the cross-modal time alignment index. The evidence chain construction module is used to complete term-event mapping, rule and clause references, and association with similar work order records, and output a set of evidence chains; The generated chain of evidence is scored for credibility. The credibility scoring function is as follows: in, The temporal correlation between audio clips and event logs; For term matching similarity; Similarity to historical disposal records; , , Let be the weight parameters, and satisfy ? The evidence chains are sorted according to the credibility score, and the evidence chain with the highest score is selected as the main reference for the question and answer output. The intelligent agent decision-making module is used to generate question-and-answer results based on the evidence chain set and to call the preset tool interface when the triggering conditions are met. The security audit module is used to generate and store audit logs.

5. A multimodal urban rail intelligent agent question-answering system according to claim 4, characterized in that, The system is deployed at the edge of the urban rail production control network and receives external knowledge update data through a one-way data transmission interface; the knowledge update data includes at least the version of the regulation text, the version of the terminology standard library, and the version of the disposal process template. Furthermore, the security audit module updates the source identifier, version number, update time, and effective scope identifier for each knowledge update record to meet security isolation and audit requirements.

6. The urban rail intelligent agent question-answering system integrating multimodal approaches according to claim 4, characterized in that, The audit logs generated by the security audit module include at least: the user identifier and role / permission identifier of the question and answer initiator; the summary of the question and answer input and the intent classification result; the set of evidence chain identifiers and the evidence chain score; the summary of the output content, the risk level and the handling priority; the tool call record, the summary of the returned result and the write-back status; and the audit logs are verified for tamper-proofing using hash chain or digital signature, and support traceability and retrieval by event number or work order number.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 3.

Images