TDCS / CTC terminal operation behavior identification and analysis method based on screen recording file
By adopting a dual-modal fusion architecture of video and logs in the railway dispatching system, combined with the Transformer architecture and rule engine, efficient identification and analysis of terminal operation behavior is achieved. This solves the problems of single data modality and insufficient semantic modeling in existing technologies, and improves the accuracy of identifying violations and the efficiency of accident investigation.
Patent Information
- Application Number
- CN202511654024.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-12
- Publication Date
- 2026-02-27
AI Technical Summary
Existing technologies in railway dispatching systems suffer from problems such as single data modality, lack of semantic modeling, and difficulty in balancing real-time performance and accuracy. They are unable to effectively identify and analyze subtle differences in terminal operation behavior, especially violations without log records.
A dual-modal fusion architecture of video and logs is adopted. By collecting and synchronizing video and log data from TDCS/CTC terminals, features are extracted and multimodal fusion is performed. Behavior recognition is performed using an improved Transformer architecture and railway domain knowledge graph. Anomaly detection and adaptive learning are combined with a rule engine to generate anomaly alarms and perform visual backtracking.
It improves the accuracy of identifying violations, enhances the semantic understanding of operational behaviors, improves the efficiency of full-scenario traceability and accident investigation, and reduces the frequency of model retraining and manual costs.
Smart Images
Figure CN121582844A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of railway dispatching automation technology, and more specifically to a method for identifying and analyzing the operational behavior of TDCS / CTC terminals based on screen recording files. It is particularly applicable to scenarios involving terminal operation monitoring, abnormal behavior detection, and safety management in railway train dispatching and command systems (TDCS) and centralized dispatching systems (CTC). This invention integrates computer vision, deep learning, edge computing, and railway field knowledge, belonging to the interdisciplinary technology field of intelligent transportation and safety monitoring. Background Technology
[0002] Currently, the analysis of operational behavior in railway dispatching systems mainly relies on the following technical solutions: (1) Analysis method based on CTC log data: By collecting database data and file data of the CTC system (including alarm information, signal status changes, interlocking route sequences, etc.), a rule engine is used to identify dangerous operation behaviors. The system generates dangerous operation records and provides graphical display by associating signal equipment status with operation records. However, this solution only relies on system log data and does not involve video image analysis of terminal operations, so it cannot identify violations without log records (such as unrecorded keyboard and mouse operations, abnormal interface interaction, etc.).
[0003] (2) Single-modal state recognition technology: Some solutions use eye movement features or facial expressions to analyze the dispatcher's state, such as extracting eye movement indicators such as pupil size and saccade speed, and combining them with an SVM classifier to identify states such as fatigue and tension. However, such methods focus on the operator's physiological state and cannot directly relate to the compliance judgment of specific operational behaviors, and do not integrate the spatiotemporal features of terminal operations.
[0004] (3) Traditional video surveillance solutions: These solutions employ manual inspection or simple motion detection techniques, which suffer from poor real-time performance, high false negative rates, and an inability to semantically understand operational intentions. In existing technologies, although cloud-edge-device collaborative architectures have been applied to smart transportation scenarios, improving efficiency through real-time analysis at edge nodes and deep processing in the cloud, they have not been customized for specific operational scenarios of TDCS / CTC terminals (such as dispatch command input and route arrangement).
[0005] (4) Operation diagram playback technology: By recording and playing back the static and dynamic changes of the operation diagram, it can assist in troubleshooting and training. However, this solution only focuses on data analysis at the level of train operation plan and cannot restore the details of personnel behavior during the terminal operation process.
[0006] Existing technologies generally suffer from the following drawbacks: ① They have a single data modality, relying on a single data source such as logs or videos; ② They lack semantic modeling of operational behavior, making it difficult to distinguish the subtle differences between normal and erroneous operations; ③ They are difficult to balance real-time performance and accuracy, and cannot meet the high reliability requirements of scheduling systems. Summary of the Invention
[0007] To overcome the shortcomings of existing technologies, this invention discloses a method for identifying and analyzing TDCS / CTC terminal operation behavior based on screen recording files. This invention collects video and log data from the TDCS / CTC terminal, performs clock synchronization and preprocessing, then extracts video and log features from the video and log data, performs multimodal fusion to obtain cross-modal feature vectors, and inputs these cross-modal feature vectors into a behavior recognition model for terminal operation behavior identification and output. Finally, it performs anomaly detection on the output terminal operation behavior, generating anomaly alarms when anomalies are detected, and analyzing and optimizing the behavior recognition model based on the anomaly alarms. This invention adopts a "video + log + AI" architecture to achieve intelligent analysis throughout the entire process. Through the dual-modal fusion of video and logs, the accuracy of identifying violation records is greatly improved. To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for recognizing and analyzing the operation behavior of TDCS / CTC terminals based on screen recording files includes the following steps: I. Data Acquisition and Preprocessing S1. Collect video data and log data from TDCS / CTC terminals, and perform clock synchronization and preprocessing on the video data and log data; Preferably, in step S1, acquiring video data from the TDCS / CTC terminal includes: connecting the HDMI interface of the TDCS / CTC terminal's host to an HDMI splitter; connecting one interface of the HDMI splitter to the terminal display for use by staff; and connecting the other interface of the HDMI splitter to an audio / video acquisition and encoding device. The audio / video acquisition and encoding device acquires video information from the terminal display through the HDMI splitter and sends the video information to a video parsing and recording server via a network cable. The video parsing and recording server retrieves the video stream corresponding to the video information and records the video file to obtain the video data.
[0008] Preferably, in step S1, collecting log data from the TDCS / CTC terminal includes: a log data parsing server connecting to the TDCS / CTC terminal interface to obtain operation logs, and parsing the operation logs to obtain the log data; wherein, the operation logs include command input, route status, and alarm information.
[0009] Preferably, in step S1, the clock synchronization and preprocessing of video data and log data includes: Use an NTP server to synchronize video data and log data using a clock. The video data is denoised, and the software running area in the video data is dynamically extracted to extract the region of interest. Extract logs and data related to the operation behavior from the log data, and remove irrelevant logs and data.
[0010] II. Feature Extraction S2. Extract video features and log features from video data and log data; Preferably, step S2 includes: Video feature extraction: The YOLO model is used to extract the spatiotemporal features of operational behaviors in the clock-synchronized and pre-processed video data in real time, including: operating trains, operating switches, arranging routes, and operating signals; Log feature extraction: Extract the data that needs to be analyzed from the clock synchronization and preprocessed log data. The data that needs to be analyzed is attribute data that is not directly displayed on the TDCS / CTC terminal screen.
[0011] III. Multimodal Fusion Analysis S3. Multimodal fusion of video features and log features is performed to obtain cross-modal feature vectors, and the cross-modal feature vectors are input into the behavior recognition model to perform terminal operation behavior recognition and output. Preferably, in step S3, based on spatiotemporal alignment preprocessing, cross-attention calculation, and feature enhancement output, video features and log features are fused in a multimodal manner to obtain a cross-modal feature vector, including: Empty alignment preprocessing: To address the temporal granularity differences between video features and log features, a linear interpolation method is used to map log features to the timeline of video feature video frames, forming a unified time scale with intervals of T; for each time window, the visual feature sequence and log feature vector within that time period are extracted, and feature alignment with millisecond-level precision is performed using a timestamp hash table; Cross-attention computation: A dual-path feature transformation architecture is adopted. Visual features are reduced in dimensionality by convolutional layers and then mapped to query vectors through MLP. Log features are encoded into key vectors and value vectors through gated recurrent units. Feature enhancement output: The fusion feature matrix adopts a residual connection structure. After concatenating the attention output with the original visual features, it is processed through standardization to finally generate the corresponding cross-modal feature vector. The cross-modal feature vector simultaneously includes: spatiotemporal trajectory features of operation behavior, state change features of interface elements, and semantic association features of operation commands.
[0012] Preferably, in step S3, the behavior recognition model adopts an improved Transformer architecture, with the input end embedded with a standard knowledge graph in the railway scheduling field, and the operation behavior classification is achieved through a pre-training-fine-tuning mode; in the pre-training stage, the model is coarsely trained by relying on self-supervised learning of machine learning, and then fine-tuned by the model, and manually labeled according to the retrieval accuracy of different scenarios.
[0013] Preferably, in step S3, the behavior recognition model defines multiple types of violations, including: manually arranging shunting routes, manually arranging train routes, manual decoy, timeout operation, timeout without operation, misoperation, and interface stagnation.
[0014] IV. Anomaly Detection and Adaptive Learning S4. Perform anomaly detection on the output terminal operation behavior, generate anomaly alarm when anomaly is detected, and analyze and optimize the behavior recognition model based on the anomaly alarm.
[0015] Preferably, step S4 includes: Rule engine and deep learning collaborative detection: Based on the railway safety operation specifications, a basic rule base is built, and abnormal alarms are generated by combining the terminal operation behavior output by the behavior recognition model; Dynamic model optimization: Based on the user's secondary analysis and confirmation of abnormal alarms, the behavior recognition model is further optimized, and corresponding scene data is continuously input into the behavior recognition model to improve the accuracy of the behavior recognition model.
[0016] Preferably, in step S4, the analysis and optimization of the behavior recognition model for abnormal alarms includes: A1. Visualize and trace abnormal operational behaviors; A2. Based on visualization and backtracking, analysts analyze abnormal operational behaviors, obtain analysis results, and import the analysis results into the model database; A3. The behavior recognition model uses data from the model database to be automatically retrained and dynamically optimized on a regular basis. A4. Use the retrained and adjusted behavior recognition model to recognize operation behavior, output terminal operation behavior, generate new alarm information when the terminal operation behavior is abnormal, and return to step A1.
[0017] V. Visualization and Backtracking Preferably, the TDCS / CTC terminal operation behavior identification and analysis method further includes: visualizing and tracing the terminal operation behavior, wherein the visualization and tracing include: It provides a visualization interface for the timing of operation behavior, synchronously displaying video clips, log records, and anomaly markers; it supports one-click playback to restore the complete scene when the abnormal behavior occurred, including: the state of each software interface t seconds before the operation, the mouse actions during the operation, and the system responses after the operation.
[0018] The beneficial effects of this invention are: 1. Multimodal fusion enhances identification completeness: Existing technologies rely solely on log data, which cannot cover unrecorded operations (such as unsaved temporary operations). This invention significantly improves the accuracy of identifying violations by fusing video and log data into a dual-modal approach.
[0019] 2. Semantic Understanding Enhances Decision-Making Capabilities: By introducing domain knowledge graphs and the Transformer architecture, an upgrade from "action detection" to "behavioral intent judgment" is achieved. Compared to traditional rule engines, this reduces the false positive rate of complex operation sequences and solves the problem that existing technologies cannot distinguish semantic differences between similar operations.
[0020] 3. Full-scenario traceability and evidence collection capabilities: The backtracking function, which combines video playback and log association, adds a dimension of personnel operation details compared to existing technologies, significantly improving the efficiency of accident investigation.
[0021] 4. Adaptive learning to adapt to complex scenarios: Through secondary analysis and confirmation of abnormal alarms by users or the labeling of new scenarios, the model continuously learns adaptively to adapt to more and more complex scenarios, reducing the frequency of model retraining and reducing the cost of manual retraining. Attached Figure Description
[0022] Figure 1 This is a flowchart of the TDCS / CTC terminal operation behavior recognition and analysis method based on screen recording files according to the present invention; Figure 2 This is a diagram illustrating the technical solution architecture of the present invention. Figure 3 This is a flowchart of the adaptive learning process of the model of this invention. Detailed Implementation
[0023] The following will provide a clear and complete description of the concept, specific structure, and technical effects of the present invention in conjunction with the embodiments and accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention.
[0024] A method for recognizing and analyzing TDCS / CTC terminal operation behavior based on screen recording files, such as Figure 1 and Figure 2 As shown, it includes the following steps: 1. Data Acquisition and Preprocessing S1. Collect video data and log data from TDCS / CTC terminals, and perform clock synchronization and preprocessing on the video data and log data; (1) Synchronous acquisition and recording of video data: Connect the HDMI interface of the host to an HDMI splitter, then connect one of the interfaces to the terminal monitor for use by relevant dispatchers, duty officers, and other staff, and connect the other interface to the audio and video acquisition and encoding equipment. This equipment sends out video information through a network cable, and the server pulls the corresponding video stream for video file recording. Compared with the solution of directly deploying live streaming software such as OBS on the host side, this solution has the following advantages: it does not occupy host performance, the equipment will not lag, it does not occupy host bandwidth, and it can achieve network isolation without affecting safe production on the production side.
[0025] (2) Log and other data collection: Connect to the TDCS / CTC system interface to obtain operation logs including command input, route status, alarm information, etc.
[0026] All of the above data needs to be synchronized with the NTP server to ensure that the order of data occurrence is correct.
[0027] (3) Data preprocessing: Noise reduction is performed on the original video to ensure the accuracy of the video information; the software operation area in the video is dynamically extracted, and the region of interest (ROI) is extracted to reduce the computational load. Logs and other data are extracted to include logs and data related to the operation behavior, and irrelevant logs and data are removed.
[0028] 2. Feature Extraction S2. Extract video features and log features from video data and log data; (1) Video feature extraction: The current mainstream YOLO model is used to extract the spatiotemporal features of the operation behavior in real time, including: operating trains, operating turnouts, arranging routes, operating signals, etc. Many high-risk operation behaviors are actually combinations of these basic operation behaviors. For example, a turnout with a faulty branch is not locked by a single lock. First, the terminal image can be used to detect which areas are faulty turnouts, and then it can be detected whether the turnout has been set to a single lock.
[0029] (2) Log feature extraction: Since the information obtained from the video is limited, there are still a certain number of attributes that are not directly displayed on the screen. These attributes are stored in the corresponding logs or data. It is necessary to extract these features and couple them with the features extracted from the video.
[0030] 3. Multimodal fusion analysis S3. Multimodal fusion of video features and log features is performed to obtain cross-modal feature vectors, and the cross-modal feature vectors are input into the behavior recognition model to perform terminal operation behavior recognition and output. (1) Feature-level fusion: Feature-level fusion adopts a cross-attention mechanism based on timestamp alignment to achieve deep association between visual features and log features. This mechanism completes cross-modal information integration through a three-level processing flow: Spatiotemporal alignment preprocessing: To address the temporal granularity differences between video streams and log data, linear interpolation is used to map log features to the video frame timeline, forming a unified time scale with 100ms intervals. For each time window, the visual feature sequence and log feature vector within that time period are extracted, and millisecond-level feature alignment is achieved using a timestamp hash table.
[0031] Cross-attention computation: A dual-path feature transformation architecture is adopted. Visual features are reduced in dimensionality by convolutional layers (3×3 convolutional kernels) and then mapped to query vectors by MLP. Log features are encoded into key vectors and value vectors by gated recurrent units (GRU).
[0032] Feature Enhancement Output: The fused feature matrix adopts a residual connection structure. After concatenating the attention output with the original visual features, it undergoes standardization to finally generate the corresponding cross-modal feature vector. This vector simultaneously contains: ① spatiotemporal trajectory features of the operation behavior; ② state change features of interface elements; ③ semantic association features of operation commands.
[0033] (2) Behavior Classification Model: An improved Transformer architecture is adopted, with the input embedded with a standard knowledge graph in the railway scheduling field. Operation behavior classification is achieved through a pre-training-fine-tuning mode. In the pre-training stage, self-supervised learning of machine learning is used for coarse training of the model, followed by fine-tuning. Based on the retrieval accuracy of different scenarios, manual labeling is performed to improve model accuracy. Various types of violations are defined: manual arrangement of shunting routes, manual arrangement of train routes, manual decoupling, timeout operation, timeout without operation, misoperation, interface stagnation, etc.
[0034] 4. Anomaly Detection and Adaptive Learning S4. Perform anomaly detection on the output terminal operation behavior, generate anomaly alarm when anomaly is detected, and analyze and optimize the behavior recognition model based on the anomaly alarm.
[0035] Rule engine and deep learning collaborative detection: A basic rule base is built based on railway safety operation specifications, and abnormal alarms are generated by combining the behavioral confidence of the model output.
[0036] Dynamic model optimization: Based on the user's secondary analysis and confirmation of abnormal alarms, the model is further optimized by continuously inputting corresponding scenario data into the model to improve model accuracy.
[0037] like Figure 3 As shown, adaptive learning includes: A1. Visualize and trace abnormal operational behaviors; A2. Based on visualization and backtracking, analysts analyze abnormal operational behaviors, obtain analysis results, and import the analysis results into the model database; A3. The behavior recognition model uses data from the model database to be automatically retrained and dynamically optimized on a regular basis. A4. Use the retrained and adjusted behavior recognition model to recognize operation behavior, output terminal operation behavior, generate new alarm information when the terminal operation behavior is abnormal, and return to step A1.
[0038] 5. Visualization and Backtracking It provides a time-series visualization interface for operational behavior, synchronously displaying video clips, log records, and anomaly markers. It supports one-click playback to recreate the complete scene when the abnormal behavior occurred, including: the state of each software interface in the 30 seconds before the operation, mouse actions during the operation, and the system responses after the operation.
[0039] The embodiments of the present invention have been described in detail above, but the present invention is not limited to the described embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalents or substitutions are all included within the scope defined by the claims of the present invention.
Claims
1. A method for recognizing and analyzing the operational behavior of a TDCS / CTC terminal based on screen recording files, characterized in that, Includes the following steps: S1. Collect video data and log data from TDCS / CTC terminals, and perform clock synchronization and preprocessing on the video data and log data; S2. Extract video features and log features from video data and log data; S3. Multimodal fusion of video features and log features is performed to obtain cross-modal feature vectors, and the cross-modal feature vectors are input into the behavior recognition model to perform terminal operation behavior recognition and output. S4. Perform anomaly detection on the output terminal operation behavior, generate anomaly alarm when anomaly is detected, and analyze and optimize the behavior recognition model based on the anomaly alarm.
2. The method for identifying and analyzing TDCS / CTC terminal operation behavior based on screen recording files as described in claim 1, characterized in that, In step S1, the acquisition of video data from the TDCS / CTC terminal includes: connecting the HDMI interface of the TDCS / CTC terminal's host to an HDMI splitter; connecting one interface of the HDMI splitter to the terminal display for use by staff; and connecting the other interface of the HDMI splitter to an audio / video acquisition and encoding device. The audio / video acquisition and encoding device acquires video information from the terminal display through the HDMI splitter and sends the video information to a video parsing and recording server via a network cable. The video parsing and recording server retrieves the video stream corresponding to the video information and records the video file to obtain the video data.
3. The method for identifying and analyzing TDCS / CTC terminal operation behavior based on screen recording files as described in claim 1, characterized in that, In step S1, collecting log data from the TDCS / CTC terminal includes: the log data parsing server connecting to the TDCS / CTC terminal interface to obtain operation logs, and parsing the operation logs to obtain the log data; wherein, the operation logs include command input, route status, and alarm information.
4. The method for identifying and analyzing TDCS / CTC terminal operation behavior based on screen recording files as described in claim 1, characterized in that, In step S1, the clock synchronization and preprocessing of video data and log data includes: Use an NTP server to synchronize video data and log data using a clock. The video data is denoised, and the software running area in the video data is dynamically extracted to extract the region of interest. Extract logs and data related to the operation behavior from the log data, and remove irrelevant logs and data.
5. The method for identifying and analyzing TDCS / CTC terminal operation behavior based on screen recording files as described in claim 1, characterized in that, Step S2 includes: Video feature extraction: The YOLO model is used to extract the spatiotemporal features of operational behaviors in the clock-synchronized and pre-processed video data in real time, including: operating trains, operating switches, arranging routes, and operating signals; Log feature extraction: Extract the data that needs to be analyzed from the clock synchronization and preprocessed log data. The data that needs to be analyzed is attribute data that is not directly displayed on the TDCS / CTC terminal screen.
6. The method for identifying and analyzing TDCS / CTC terminal operation behavior based on screen recording files as described in claim 1, characterized in that, In step S3, based on spatiotemporal alignment preprocessing, cross-attention calculation, and feature enhancement output, video features and log features are fused in a multimodal manner to obtain a cross-modal feature vector, including: Empty alignment preprocessing: To address the temporal granularity differences between video features and log features, a linear interpolation method is used to map log features to the timeline of video feature video frames, forming a unified time scale with intervals of T; for each time window, the visual feature sequence and log feature vector within that time period are extracted, and feature alignment with millisecond-level precision is performed using a timestamp hash table; Cross-attention computation: A dual-path feature transformation architecture is adopted. Visual features are reduced in dimensionality by convolutional layers and then mapped to query vectors through MLP. Log features are encoded into key vectors and value vectors through gated recurrent units. Feature enhancement output: The fusion feature matrix adopts a residual connection structure. After concatenating the attention output with the original visual features, it is processed through standardization to finally generate the corresponding cross-modal feature vector. The cross-modal feature vector simultaneously includes: spatiotemporal trajectory features of operation behavior, state change features of interface elements, and semantic association features of operation commands.
7. The method for identifying and analyzing TDCS / CTC terminal operation behavior based on screen recording files as described in claim 1, characterized in that, In step S3, the behavior recognition model adopts an improved Transformer architecture, with the input end embedded with a standard knowledge graph in the railway scheduling field. The operation behavior classification is achieved through a pre-training-fine-tuning mode. In the pre-training stage, the model is coarsely trained by relying on self-supervised learning of machine learning, and then fine-tuned. Based on the retrieval accuracy of different scenarios, labels are manually assigned.
8. The method for identifying and analyzing TDCS / CTC terminal operation behavior based on screen recording files as described in claim 1, characterized in that, Step S4 includes: Rule engine and deep learning collaborative detection: Based on the railway safety operation specifications, a basic rule base is built, and abnormal alarms are generated by combining the terminal operation behavior output by the behavior recognition model; Dynamic model optimization: Based on the user's secondary analysis and confirmation of abnormal alarms, the behavior recognition model is further optimized, and corresponding scene data is continuously input into the behavior recognition model to improve the accuracy of the behavior recognition model.
9. The method for recognizing and analyzing TDCS / CTC terminal operation behavior based on screen recording files as described in claim 1, characterized in that, In step S4, the analysis and optimization of the abnormal alarm behavior recognition model includes: A1. Visualize and trace abnormal operational behaviors; A2. Based on visualization and backtracking, analysts analyze abnormal operational behaviors, obtain analysis results, and import the analysis results into the model database; A3. The behavior recognition model uses data from the model database to be automatically retrained and dynamically optimized on a regular basis. A4. Use the retrained and adjusted behavior recognition model to recognize operation behavior, output terminal operation behavior, generate new alarm information when the terminal operation behavior is abnormal, and return to step A1.
10. The method for identifying and analyzing TDCS / CTC terminal operation behavior based on screen recording files as described in claim 1, characterized in that, The TDCS / CTC terminal operation behavior identification and analysis method further includes: visualizing and retrospectively analyzing terminal operation behavior, wherein the visualization and retrospective analysis includes: It provides a visualization interface for the timing of operation behavior, synchronously displaying video clips, log records, and anomaly markers; it supports one-click playback to restore the complete scene when the abnormal behavior occurred, including: the state of each software interface t seconds before the operation, the mouse actions during the operation, and the system responses after the operation.