A method and system for fault detection and natural language retrieval of subway PIS screens
By deploying a lightweight VL large model on the vehicle-mounted edge computing device, dual-mode switching between subway PIS screen fault detection and natural language retrieval is achieved, solving the problem of the separation between subway PIS screen fault detection and video retrieval in the existing technology, improving the fault identification accuracy and resource reuse rate, and improving the operation and maintenance efficiency of the vehicle system.
Patent Information
- Application Number
- CN202611352786.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-02
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies cannot simultaneously achieve accurate detection of complex faults in subway PIS screens and natural language video retrieval, and have low resource reuse rates, making it difficult to meet the operation and maintenance needs of subway vehicle scenarios.
A lightweight VL large model is deployed on the vehicle edge computing device to achieve dual-mode switching between fault detection and natural language retrieval. End-to-end semantic recognition is performed through the visual language large model, supporting zero-sample target retrieval, and bandwidth consumption is reduced through the edge-cloud collaborative architecture.
It improves the accuracy and range of PIS screen fault identification, reduces the cost of in-vehicle systems, and improves the response efficiency to emergencies, while balancing real-time performance and accuracy.
Smart Images

Figure CN123115338A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of intelligent operation and maintenance and video analysis technology for rail transit, and in particular to a method and system for fault detection and natural language retrieval of subway PIS screens. Background Technology
[0002] With the rapid development of fully automated urban rail transit lines, onboard passenger information system displays (PIS screens) have become the core visual channel for passengers to obtain travel information, and their display status directly affects passenger travel experience and operational safety. According to operating procedures, trains must not depart if multiple PIS screens malfunction; therefore, rapid fault detection and location are crucial for ensuring operational order.
[0003] PIS screens operate under high loads for extended periods, and fault types encompass multiple aspects, including the screen itself, signal transmission, and power supply links. The same fault phenomenon can be caused by different factors, making traditional manual troubleshooting inefficient. Regarding automatic detection, existing solutions mostly rely on traditional computer vision rules, using brightness thresholds and color histograms to identify simple faults like black screens or dim screens. However, they have low recognition rates for complex faults such as distorted images, garbled text, and color casts, and are easily affected by factors like carriage lighting, passenger obstruction, and screen angle, resulting in a high false alarm rate. In terms of video retrieval, existing subway monitoring systems largely depend on manual playback or structured attribute searches. Dispatchers cannot directly retrieve targets using natural language, leading to low response efficiency for emergencies.
[0004] Existing patent document CN122116777A discloses a detection method for vehicle-mounted displays, which identifies display anomalies based on image processing or AI (artificial intelligence) models. However, this solution is geared towards general displays and is not optimized for the operation scenario of subway PIS screens. Furthermore, it uses a traditional detection model and lacks natural language retrieval capabilities. Patent document CN118551077A discloses a natural language interactive security video retrieval system based on a generative large model. However, it only has video retrieval functions and lacks PIS screen fault detection capabilities. Moreover, retrieval and fault detection are independent systems, making it impossible to reuse models and hardware resources, resulting in a waste of vehicle-mounted computing resources.
[0005] Overall, existing technologies cannot simultaneously achieve accurate detection of complex PIS screen faults and natural language video retrieval, and also lack an efficient architecture that reuses the same model and hardware for two sets of functions, making it difficult to meet the operation and maintenance needs of subway vehicle scenarios. Summary of the Invention
[0006] This application provides a method and system for fault detection and natural language retrieval of subway PIS screens, which solves the problems of low recognition rate of complex faults of PIS screens, fragmentation of fault detection and video retrieval systems, and low resource reuse rate in the prior art.
[0007] On the one hand, embodiments of this application provide a method for fault detection and natural language retrieval of subway PIS screens, including the following steps: The display screens on the subway PIS screens and the panoramic video stream of the carriages are captured by the vehicle-mounted cameras. A lightweight VL (Video Language) large model is deployed on the vehicle edge computing device. The lightweight VL large model is configured with a fault detection mode and a natural language retrieval mode. The two modes share the same set of model weights, and the mode can be switched by changing the input prompt words. In the default state, the fault detection mode is run. After the area of the display screen is located and preprocessed, the screen area image and the fault detection prompt are input into the lightweight VL large model. The lightweight VL large model determines whether there is a display fault on the subway PIS screen. If there is a fault, the fault type, severity level and handling suggestions are output. When a prompt word for a natural language search instruction is received, the system automatically switches to natural language search mode. The panoramic video stream of the carriage and the search prompt word containing the user's natural language description are input into the lightweight VL model. The lightweight VL model then locates and matches the target in the video frames of the panoramic video stream and outputs the target position. After the retrieval task is completed, if no new natural language retrieval instruction is received within the preset timeout period, the system will automatically switch back to fault detection mode.
[0008] On the other hand, embodiments of this application also provide a subway PIS screen fault detection and natural language retrieval system, including: The video perception layer includes an onboard camera aimed at the subway PIS screen and a panoramic camera in the carriage, which are used to capture the display screen of the subway PIS screen and the panoramic video stream of the carriage, respectively. The edge computing layer includes a subway PIS screen area positioning module, a lightweight VL large model inference module, a dual-mode switching control module, and a fault timing confirmation module, which are used to perform subway PIS screen area positioning, VL large model inference, dual-mode switching control, and fault timing confirmation on the on-board edge computing device. The edge-cloud collaboration layer includes an edge inference result uploading module, a cloud large model scheduling module, and a retrieval task distribution module, which are used to manage task allocation and data transmission between the edge and the cloud. The dual-mode service layer includes a fault detection service module, a natural language retrieval service module, and a unified event output module, which are used to output fault information in fault detection mode and output target location results in natural language retrieval mode. The interaction and alarm layer includes an in-vehicle display terminal, a dispatch center alarm interface, a natural language interaction interface, and an event evidence retention module, which are used to display alarms and retrieval results, provide natural language input, and save event evidence.
[0009] The subway PIS screen fault detection and natural language retrieval method and system in this application have the following advantages: A large visual language model is used to achieve end-to-end semantic recognition of complex faults in PIS screens, improving the fault recognition range and accuracy, and enabling zero-shot identification of novel faults. Through dual-mode adaptive switching of the same model, hardware and model reuse for fault detection and natural language retrieval is achieved, reducing the cost of in-vehicle systems. Zero-shot natural language target retrieval is supported, improving the efficiency of emergency response. An edge-cloud collaborative architecture is adopted to balance real-time performance and accuracy, while reducing bandwidth consumption. Attached Figure Description
[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Figure 1 This is a flowchart of the fault detection mode in the subway PIS screen fault detection and natural language retrieval method provided in the embodiments of this application.
[0012] Figure 2 The flowchart shows the natural language retrieval mode in the subway PIS screen fault detection and natural language retrieval method provided in the embodiments of this application.
[0013] Figure 3 This is a schematic diagram of the state machine during the dual-mode switching process provided in an embodiment of this application.
[0014] Figure 4 This is a schematic diagram of the task distribution strategy in the end-to-cloud collaboration process provided in the embodiments of this application.
[0015] Figure 5 This is a schematic diagram illustrating fault confirmation and alarm classification provided in the embodiments of this application.
[0016] Figure 6 This is a schematic diagram illustrating the identification results of a black screen fault provided in an embodiment of this application.
[0017] Figure 7 This is a schematic diagram illustrating the identification results of screen flickering faults provided in an embodiment of this application.
[0018] Figure 8 This is a schematic diagram illustrating the identification results of color cast faults provided in an embodiment of this application.
[0019] Figure 9 This is a schematic diagram illustrating the identification results of garbled text faults provided in an embodiment of this application.
[0020] Figure 10 This is a schematic diagram illustrating the identification results of screen cracking faults provided in an embodiment of this application.
[0021] Figure 11 This is a schematic diagram illustrating the identification results of screen freeze faults provided in an embodiment of this application.
[0022] Figure 12 This is a schematic diagram illustrating the identification results of a no-signal fault provided in an embodiment of this application.
[0023] Figure 13 This is a schematic diagram of the software running results provided in an embodiment of this application.
[0024] Figure 14 A schematic diagram illustrating the composition of the subway PIS screen fault detection and natural language retrieval system provided in this application embodiment. Detailed Implementation
[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0026] like Figure 1-13 As shown in the figure, this application provides a method for subway PIS screen fault detection and natural language retrieval, which includes the following steps: The display screens on the subway PIS screens and the panoramic video stream of the carriages are captured by the vehicle-mounted cameras. A lightweight VL large model is deployed on the vehicle edge computing device. The lightweight VL large model is configured with a fault detection mode and a natural language retrieval mode. The two modes share the same set of model weights, and the mode can be switched by changing the input prompt words. In the default state, the fault detection mode is run. After the area of the display screen is located and preprocessed, the screen area image and the fault detection prompt are input into the lightweight VL large model. The lightweight VL large model determines whether there is a display fault on the subway PIS screen. If there is a fault, the fault type, severity level and handling suggestions are output. When a prompt word for a natural language search instruction is received, the system automatically switches to natural language search mode. The panoramic video stream of the carriage and the search prompt word containing the user's natural language description are input into the lightweight VL model. The lightweight VL model then locates and matches the target in the video frames of the panoramic video stream and outputs the target position. After the retrieval task is completed, if no new natural language retrieval instruction is received within the preset timeout period, the system will automatically switch back to fault detection mode.
[0027] For example, in the video acquisition stage, two video streams are acquired using two types of cameras deployed in the train passenger compartment: one type is a dedicated camera aimed at the PIS screen, which can reuse existing onboard cameras or add wide-angle or hemispherical cameras to capture the real-time display of the PIS screen at a fixed frame rate, providing image input for fault detection; the other type is a panoramic camera that captures the complete monitoring image inside the carriage, covering the entire area of the passenger compartment, providing video source for natural language retrieval. The two video streams are transmitted to the edge computing device via the onboard internal bus, where they are uniformly scheduled and processed.
[0028] During model deployment, a lightweight VL (Vehicle Language) model is loaded onto the in-vehicle edge computing device. This model supports both fault detection and natural language retrieval tasks. Both modes share the same set of model weight parameters and inference engine; there are no two independent models. Task switching is completed simply by replacing the prompt word templates input to the model, without reloading model weights or restarting the inference process.
[0029] The lightweight VL model used in this embodiment employs a three-stage cascaded architecture, consisting of a visual encoder, a projection alignment layer, a language model decoder, and an output constraint layer. The visual encoder uses a Vision Transformer architecture, dividing the input image into 14×14 pixel blocks, linearly projecting them into feature vectors, superimposing learnable positional encodings, and then extracting multi-scale visual features through a multi-layer Transformer encoder to output a visual feature sequence. The projection alignment layer uses a multi-layer perceptron structure containing two fully connected layers and GELU (Gaussian error linear unit) activation, mapping visual features to feature vectors with the same hidden dimensions as the language model, achieving alignment between the visual feature space and the language feature space. The language model decoder uses a Decoder-only Transformer architecture, receiving the concatenated prompt word sequence and visual feature sequence, and jointly modeling through self-attention and cross-attention mechanisms, autoregressively generating natural language inference results. The output constraint layer uses JSON (JavaScript Object Notation) format constraints or function call mechanisms to parse the natural language output into structured data.
[0030] The training of the lightweight VL model consists of three stages: The first stage is large-scale image-text pair pre-training, using a publicly available large-scale image-text pair dataset for comparative learning. The InfoNCE (Information-Noise Contrast Estimation) loss function is optimized to make matching image-text pairs closer in the feature space and mismatched pairs farther apart, giving the model basic image understanding and cross-modal retrieval capabilities. The second stage is visual language instruction fine-tuning, using a manually annotated visual question-answering dataset for supervised fine-tuning. The LoRA low-rank adaptation method is used, training only a small number of parameters in the attention layer, enabling the model to learn to follow human instructions and answer image-related questions in natural language. The third stage is PIS screen fault detection domain adaptation fine-tuning, using a PIS screen fault dataset collected from real subway operation scenarios for fine-tuning. This adapts the general VL model to the specific domain of subway PIS screen fault detection, improving the accuracy and standardization of complex fault identification.
[0031] The system defaults to fault detection mode, continuously monitoring the display status of the PIS screen. In this mode, the acquired PIS screen image is first localized and preprocessed. Then, the processed screen area image, along with preset fault detection prompts, is input into a lightweight VL (Visual Language) model. The model performs end-to-end semantic understanding of the screen content to determine if a display fault exists. When a fault is detected, the system outputs the corresponding fault type, severity level, and handling suggestions.
[0032] When dispatchers input prompts via the interactive interface, the system automatically triggers a mode switching process, switching to natural language retrieval mode. In this mode, the video input source changes to the panoramic camera inside the carriage, and the prompt template is replaced with a dedicated template for the retrieval mode, embedding the user-input natural language target description. The lightweight VL large model receives the panoramic video frames and the retrieval prompts, locates the target matching the description within the image, and outputs the target's position information within the image.
[0033] When the retrieval task is completed, including when the retrieval time period is traversed or the user manually terminates the retrieval, the system starts a timeout countdown. If no new natural language retrieval instruction is received within the preset timeout period (e.g., 30 seconds), the system automatically switches back to fault detection mode, restores the original video input source and prompt word configuration, and continues to execute the PIS screen status monitoring task.
[0034] Furthermore, in fault detection mode, the lightweight VL large model identifies at least one of the following fault types: black screen, distorted screen, color distortion, garbled text, cracked screen, frozen screen, or no signal. The fault detection prompts include visual feature descriptions and severity level judgment criteria for each fault type, guiding the lightweight VL large model to output structured results containing fault type, severity level, fault location description, and handling suggestions.
[0035] For example, this application embodiment guides a lightweight VL large model to cover multiple types of PIS screen display faults through preset fault detection prompts and outputs structured detection results. The lightweight VL large model can identify at least seven types of display faults: black screen, characterized by a completely black screen with no display content; distorted screen, characterized by colored stripes, mosaics, or random noise on the screen; color cast, characterized by abnormal overall screen color, such as reddish, greenish, or bluish tints; garbled text, characterized by text displayed as unrecognizable squares or disordered characters; cracked screen, characterized by display abnormalities caused by physical cracks; frozen screen, characterized by a screen that remains static for a long time without updating; and no signal, characterized by a blue screen or the display of a "no signal" message.
[0036] The fault detection prompts are pre-designed instruction texts that clearly list the visual characteristics, severity level determination rules, and output format requirements for each type of fault, guiding the model to judge the screen according to a unified standard. The prompts also specify the field structure of the output content, ensuring that the model output includes four fields: fault type, severity level, fault location description, and handling suggestions. The fault location description includes location results such as full screen, top half screen, bottom half screen, left half screen, right half screen, or local area. The handling suggestions provide maintenance guidance corresponding to different fault types; for example, for a screen flickering fault, it suggests checking the SVGA (Super SVGA) signal line connection, and for a black screen fault, it suggests checking the power supply link and driver board.
[0037] The model output uses a format constraint mechanism to parse the natural language inference results into structured data, which is convenient for subsequent fault timing confirmation, alarm triggering, and operation and maintenance work order docking. No manual secondary parsing is required.
[0038] Furthermore, the displayed screen is localized and preprocessed, including: using a target detection model to detect the rectangular area of the subway PIS screen in the video frames of the displayed screen and outputting the bounding box coordinates; performing perspective transformation correction on the detected rectangular area to eliminate the installation angle tilt; cropping and scaling the corrected screen area to the input resolution of the lightweight VL large model; when no subway PIS screen is detected or the area of the screen that is obscured exceeds a preset threshold, it is marked as an invisible screen state and no fault judgment is triggered.
[0039] For example, this application embodiment uses a lightweight target detection model and image transformation algorithm to extract and correct the PIS screen area, reducing the impact of scene interference on subsequent fault identification. The PIS screen area localization is achieved using a lightweight target detection model, which is pre-trained and can quickly identify the rectangular outline of the PIS screen in video frames, outputting the bounding box coordinates of the screen area. Addressing the issue of tilted installation angles of PIS screens in subway cars, perspective transformation correction is performed on the detected screen area: the perspective transformation matrix is calculated based on the coordinates of the four vertices of the bounding box, mapping the tilted trapezoidal screen area to a regular rectangle, eliminating the visual distortion caused by the tilt, and ensuring that the subsequent model input image is the correct screen display content.
[0040] After perspective correction is completed, the screen area is cropped from the original frame and uniformly scaled to the standard input resolution of the lightweight VL large model, such as 384×384 or 448×444 pixels, to match the input requirements of the model.
[0041] During the preprocessing of each frame, a validity check is performed simultaneously: if the target detection model does not detect the PIS screen area, or if the detected screen area is obscured by an area exceeding a preset threshold (e.g., the obscured area is greater than 50%), the frame is marked as invisible, the fault judgment for this round is skipped, and no fault result is output. This mechanism avoids false positives in scenarios such as passenger obstruction or foreign object obstruction, reducing the false alarm rate.
[0042] The fault detection dataset used for training the object detection model was constructed through the following process: Images of PIS screens were captured from onboard cameras along subway lines, covering screens from different vehicle models, screen brands, installation angles, and lighting conditions. The capture frequency was 1 frame / second, with a cumulative capture time of no less than 30 days. The original images underwent quality screening, removing blurry, overexposed, underexposed, or heavily obscured images. Valid images were annotated by trained annotators, with annotations including fault type, severity level, fault location, and suggested handling. A dual-annotation mechanism plus third-party review was used to ensure annotation quality. Data augmentation strategies such as brightness perturbation, noise addition, affine transformation, random occlusion, and image blending were employed to improve the model's generalization ability. Finally, the dataset was divided into training, validation, and test sets in a 7:2:1 ratio, ensuring that frames from the same video segment did not appear in different subsets to prevent data leakage.
[0043] Furthermore, after the lightweight VL large model determines the fault detection result of the subway PIS screen, if the lightweight VL large model judges the same type of fault in N consecutive frames, the fault is confirmed. Here, N is an integer greater than or equal to 2. If the confirmed fault is judged as normal in subsequent frames, the countdown begins. When the fault is judged as normal in M consecutive frames, the alarm is deactivated. Here, M is an integer greater than or equal to 2.
[0044] For example, this application embodiment filters transient interference through continuous frame verification and a fade-out strategy, improving the reliability of fault detection. After the lightweight VL large model outputs a single-frame fault judgment result, the result is sent to the fault timing confirmation module for timing verification. For fault judgment, a continuous frame confirmation rule is adopted: the fault is officially confirmed only when N consecutive frames are all judged to be the same type of fault, where N is an integer greater than or equal to 2, and in this embodiment, it is typically 3. Single-frame or non-continuous fault judgments are considered transient interference and do not trigger formal alarms. Through continuous frame confirmation, single-frame misjudgments caused by factors such as sudden changes in illumination, image jitter, and brief occlusion can be effectively filtered.
[0045] For confirmed faults, a gradual resolution rule is adopted: when the lightweight VL large model determines that the screen has returned to normal in subsequent frames, the alarm is not immediately cleared, but enters the gradual counting phase; the fault alarm is officially cleared only when M consecutive frames are judged to be in a normal state, where M is an integer greater than or equal to 2, and in this embodiment, the typical value is 5. The gradual resolution mechanism can avoid frequent alarm switching caused by repeated fault fluctuations and improve alarm stability.
[0046] During the timing confirmation process, information such as the duration and frequency of the fault are recorded simultaneously for subsequent fault statistics and maintenance analysis.
[0047] Furthermore, in natural language retrieval mode, the lightweight VL large model directly understands the target description in the natural language retrieval command and locates and matches the target in the video frames of the panoramic video stream of the carriage with zero samples. After finding the target in the first frame, the lightweight target tracking algorithm is started in subsequent frames to continuously track the target and thus obtain the continuous position of the target.
[0048] For example, this application embodiment achieves direct natural language retrieval based on the zero-shot target recognition capability of the VL large model, and combines it with a tracking algorithm to reduce computational power consumption. In natural language retrieval mode, users directly input natural language commands describing the characteristics of the target, such as "a passenger carrying a red suitcase and wearing a white hat," without needing to convert the description into structured attribute query conditions. The lightweight VL large model directly understands the semantics of natural language through cross-modal alignment capabilities, searches for targets matching the description in video frames of the panoramic video stream, and outputs the bounding box position, confidence score, and feature description of the target. This retrieval process does not require pre-training a detection model for a specific target, supports any describable person, object, or behavioral target, and has zero-shot recognition capability.
[0049] Natural language retrieval capabilities are also acquired through a three-stage training process: the first stage is cross-modal contrastive learning pre-training, which learns a general cross-modal aligned representation based on a large-scale image-text pair dataset; the second stage is target localization command fine-tuning, which uses a visual question-answering dataset with target bounding box annotations for supervised fine-tuning, training the model to accurately locate targets based on natural language descriptions; the third stage is subway car scene domain adaptation, which uses a real subway car surveillance video dataset for fine-tuning, and improves the model's ability to distinguish similar targets through hard example mining strategies, adapting to scenarios such as changes in car lighting, dense crowds, and severe occlusion.
[0050] The retrieval dataset was collected from real subway operation monitoring videos, covering scenarios under different time periods, different carriage types, and different passenger flow densities, with a total collection time of no less than 500 hours. Frame sampling and keyframe extraction were performed on the collected videos, removing blurry, overly dark, and redundant frames. Annotators added natural language descriptions and bounding boxes of different granularities to the searchable targets in the keyframes, employing dual-annotation with consistency checks to ensure annotation quality. Model robustness was improved through strategies such as geometric transformation, photometric transformation, occlusion enhancement, hybrid enhancement, and text enhancement. Finally, the video segments were divided into training, validation, and test sets for model training and performance evaluation.
[0051] Once a target is located for the first time in a frame, the system initiates a target tracking algorithm, such as ByteTrack or BoT-SORT (Robust Association Multi-Person Tracking), to continuously track the target. In subsequent frames, the target position is updated using the tracking algorithm, eliminating the need for a frame-by-frame call to the lightweight VL large model for a global search, significantly reducing inference computational power consumption and improving the tracking frame rate. If the target is lost for more than a preset number of frames, such as more than 30 frames, the lightweight VL large model is invoked again to perform a global search in the current frame to relocate the target.
[0052] This application embodiment combines "lightweight VL large model global localization + lightweight algorithm continuous tracking" to balance the flexibility of retrieval and the efficiency of operation. It can output the target's motion trajectory and real-time highlighted image, which is convenient for dispatchers to view.
[0053] Furthermore, when switching modes, only the prompt word templates fed into the lightweight VL large model are changed, while the weights and inference engine of the lightweight VL large model remain unchanged. The video source is also switched simultaneously when switching modes. In fault detection mode, the onboard camera of the subway PIS screen is used, and in natural language retrieval mode, the panoramic camera of the carriage is used.
[0054] For example, this embodiment achieves second-level mode switching by replacing prompt words and switching video sources synchronously, ensuring system response speed. The core of dual-mode switching is prompt word template replacement: the fault detection mode and the natural language retrieval mode share the same set of weight files for a lightweight VL large model and the same inference engine instance. When switching modes, only the prompt word text of the input model is modified, replacing the fault detection prompt words with retrieval prompt words, or vice versa. The model weights and inference engine are always kept loaded and running, without the need to reload the model or restart the inference process. The entire switching process can be completed within 1 second, achieving second-level switching.
[0055] While switching prompt words, the system simultaneously switches the video input source: in fault detection mode, the video input is the vehicle-mounted camera aligned with the PIS screen, ensuring the image is focused on the screen area; in natural language retrieval mode, the video input switches to a panoramic camera in the passenger compartment, providing a complete view of the passenger compartment for target search. The video source switching and prompt word switching are executed synchronously to ensure that the input data matches the task type after the mode switch.
[0056] The mode switching process is uniformly scheduled by the dual-mode switching control module to ensure the continuity and stability of the data stream during the switching process and avoid screen interruption or inference errors.
[0057] Furthermore, it also includes edge-cloud collaboration steps: a lightweight VL large model is deployed at the edge, and all subway PIS screen fault detection tasks are completed at the edge; a full VL large model is deployed in the cloud, with a larger parameter scale than the lightweight VL large model, and the full VL large model is used to handle complex natural language retrieval tasks and difficult fault verification; when the number of target features in the natural language retrieval task does not exceed a preset value and the retrieval time period does not exceed a preset duration, the natural language retrieval task is executed at the edge; when the number of target features exceeds a preset value or the retrieval time period exceeds a preset duration, the natural language retrieval task is executed in the cloud; when the fault judgment confidence obtained at the edge is lower than a preset threshold or the fault type is uncertain, the screen area image is uploaded to the cloud for verification by the full VL large model.
[0058] For example, this application embodiment balances real-time performance, accuracy, and bandwidth efficiency by dividing tasks between a lightweight edge model and a full cloud model. In terms of deployment architecture, a lightweight VL model with 4B parameters is deployed at the vehicle edge, and the model size is further compressed using INT4 (4-bit integer quantization) or INT8 (8-bit integer quantization) techniques to reduce computing power requirements, enabling low-latency inference on the vehicle edge computing device. All real-time PIS screen fault detection tasks are completed locally at the edge, without relying on the vehicle-to-ground communication network. Even if the network is interrupted, the fault monitoring function can still be executed normally, ensuring basic operation and maintenance capabilities.
[0059] Larger full-scale VL models, such as those with 7 or more parameters, are deployed in the cloud. These models possess stronger semantic understanding and complex reasoning capabilities, primarily undertaking two types of tasks: complex natural language retrieval tasks and complex fault verification tasks. The full-scale VL models in the cloud are deployed on GPU (Graphics Processing Unit) server clusters, such as A100 or H100 GPU servers, or using AI inference instances from cloud service providers. A task scheduling server and a storage server are also deployed in the cloud. The task scheduling server uses containerized deployment to manage task distribution, result aggregation, and model version management; the storage server uses cloud storage or object storage services to store historical fault records, retrieval logs, model training data, and an operational knowledge base.
[0060] Task distribution follows a complexity assessment rule: For natural language retrieval tasks, the complexity is evaluated by parsing the number of target features in the command and considering the retrieval time period and the number of cameras involved. If the number of target features does not exceed a preset value (e.g., no more than 2 features), the retrieval time period does not exceed a preset duration (e.g., no more than 5 minutes), and only a single camera is involved, it is considered a simple retrieval task and is executed locally at the edge. If the number of target features exceeds a preset value, the retrieval time period exceeds a preset duration, or multiple cameras are involved, it is considered a complex retrieval task, and the retrieval command and corresponding video segment information are uploaded to the cloud for the full VL large model to perform the retrieval.
[0061] For fault detection tasks, routine faults are directly determined by the edge device. When the confidence level of the fault judgment output by the edge device is lower than the preset threshold, or the fault type cannot be clearly determined, the screen area image of the current frame and the inference result of the edge device are uploaded to the cloud together. The full VL large model is then used for secondary verification, and the local fault record is updated according to the verification result of the cloud to improve the accuracy of complex fault judgment.
[0062] Data transmission between the edge and cloud adopts an on-demand upload strategy: only the retrieval request text, a single screen image of the fault, and the result data are uploaded. The complete video stream of fault detection does not need to be uploaded to the cloud, which significantly reduces bandwidth consumption compared to the full-frame upload solution. At the same time, the system has an offline degradation mechanism. When vehicle-to-ground communication is interrupted, the edge terminal can independently complete all fault detection functions and simple retrieval functions, ensuring the availability of basic services.
[0063] Furthermore, the severity of the fault is divided into three levels: severe level is a full-screen black screen, full-screen distorted screen, or full-screen no signal fault; general level is a fault with color distortion, partial screen distortion, garbled text, or screen freezing; minor level is a fault with abnormal brightness or occasional flickering; different levels correspond to different alarm strategies.
[0064] For example, this application embodiment classifies PIS screen display faults into three severity levels and matches them with differentiated alarm and handling strategies. A severity level fault refers to a fault where information cannot be displayed at all, including a completely black screen, a completely distorted screen, and a completely no signal. This level of fault will prevent passengers from obtaining critical information such as arrival and transfer information, which may affect train departure according to operating procedures. Therefore, the corresponding alarm strategy for the highest level is: immediately triggering the onboard audible and visual alarm, simultaneously pushing it to the dispatch center alarm interface, automatically generating and dispatching a maintenance work order, and simultaneously saving video clips and screenshot evidence before and after the fault.
[0065] A general-level fault refers to a fault that affects the readability of information but still allows for partial display, including color distortion, partial screen flickering, garbled text, and screen freezing. This level of fault reduces information acquisition efficiency but does not completely lose display functionality. The corresponding intermediate-level alarm strategy is to record the fault information and send a notification to maintenance personnel, temporarily suspending emergency audible and visual alarms, and arranging maintenance after the train returns to the depot.
[0066] Minor faults refer to those that do not affect passengers' access to critical information, including abnormal brightness and occasional flickering. These faults have minimal impact on operations and are handled with a low-level alarm strategy: only written to the system log, periodically summarized and statistically analyzed, without proactively pushing alarms.
[0067] The three-tiered fault classification mechanism enables differentiated fault management, avoids excessive alarms from disrupting operational order, and ensures timely response to serious faults.
[0068] This application also provides a subway PIS screen fault detection and natural language retrieval system, such as Figure 14 As shown, the system includes: The video perception layer includes an onboard camera aimed at the subway PIS screen and a panoramic camera in the carriage, which are used to capture the display screen of the subway PIS screen and the panoramic video stream of the carriage, respectively. The edge computing layer includes a subway PIS screen area positioning module, a lightweight VL large model inference module, a dual-mode switching control module, and a fault timing confirmation module, which are used to perform subway PIS screen area positioning, VL large model inference, dual-mode switching control, and fault timing confirmation on the on-board edge computing device. The edge-cloud collaboration layer includes an edge inference result uploading module, a cloud large model scheduling module, and a retrieval task distribution module, which are used to manage task allocation and data transmission between the edge and the cloud. The dual-mode service layer includes a fault detection service module, a natural language retrieval service module, and a unified event output module, which are used to output fault information in fault detection mode and output target location results in natural language retrieval mode. The interaction and alarm layer includes an in-vehicle display terminal, a dispatch center alarm interface, a natural language interaction interface, and an event evidence retention module, which are used to display alarms and retrieval results, provide natural language input, and save event evidence.
[0069] For example, the system in this embodiment adopts a five-layer architecture design, and each layer interacts through standardized interfaces to jointly realize dual-mode functionality.
[0070] The video perception layer serves as the system's video input layer, deployed within the train carriages. It comprises two types of camera equipment: onboard cameras aimed at the PIS screen, specifically capturing the real-time display image; these can utilize existing onboard cameras or newly added wide-angle or dome cameras. The other type is a panoramic camera capturing a full view of the carriage interior, covering the entire passenger compartment. Both types of cameras are connected to the edge computing layer via the onboard video bus, providing a stable video stream input.
[0071] The edge computing layer, deployed on the vehicle-mounted edge computing device, is the core of local computing and comprises four functional modules: a PIS screen area localization module, which runs target detection algorithms to perform screen area detection, perspective correction, and cropping preprocessing; a lightweight VL large model inference module, which loads the quantized lightweight VL large model and performs inference calculations for fault detection and target retrieval; a dual-mode switching control module, which maintains the mode state machine and controls prompt word replacement and video source switching; and a fault timing confirmation module, which executes continuous frame confirmation and fade-out logic to filter false alarms. The vehicle-mounted edge computing device can be an onboard industrial control computer or an edge AI computing box, supporting NPU (Neural Processing Unit) or GPU acceleration. Typical options include Rockchip RK3588, NVIDIA Jetson Orin Nano, and Qualcomm 8295, which can meet the real-time inference requirements of 4B parameter quantization models. Vehicle storage uses solid-state drives or eMMC (embedded multimedia storage card) flash memory to store model weights, configuration files, and fault video clips and screenshot evidence. Vehicle-to-ground communication can use 5G modules, Wi-Fi 6 modules, or LTE-M (LTE machine communication) modules to achieve data transmission between the edge and the cloud.
[0072] The edge-cloud collaboration layer serves as an intermediate scheduling layer between the edge and the cloud, comprising three functional modules: an edge inference result upload module, responsible for uploading fault results, troubleshooting images, and retrieval requests from the edge to the cloud as needed; a cloud large model scheduling module, which manages the inference resources of the full VL model in the cloud and allocates processing tasks; and a retrieval task distribution module, which distributes retrieval tasks to the edge or cloud for execution based on task complexity determination rules.
[0073] The dual-mode service layer is a business logic encapsulation layer, containing three functional modules: a fault detection service module, which encapsulates the complete business logic of fault detection and outputs standardized fault information; a natural language retrieval service module, which encapsulates target retrieval and tracking logic and outputs target location and tracking results; and a unified event output module, which converts the output of the two modes into a standard event structure for easy calling by upper-layer systems.
[0074] The interaction and alarm layer is the user interaction interface layer, which includes four types of terminals and modules: vehicle-mounted display terminal, installed in the train driver's cab or crew room, to display local fault alarm information; dispatch center alarm interface, deployed in the operation dispatch center, to centrally display the fault status and search results of all trains on the line; natural language interaction interface, providing a text input entry point, supporting dispatchers to input natural language search commands; and event evidence retention module, which stores video clips, screenshots, and search records before and after the fault occurred for subsequent review and tracing.
[0075] Furthermore, the dual-mode switching control module maintains the mode state machine, which is in fault detection mode by default. When a prompt word for a natural language retrieval instruction is received, it automatically switches to natural language retrieval mode. When the retrieval is completed and no new natural language retrieval instruction is received within the preset timeout period, it automatically switches back to fault detection mode. During mode switching, only the input prompt word of the VL large model is changed, while the model weights and inference engine remain unchanged.
[0076] For example, this application embodiment uses a state machine mechanism to achieve automatic transition between dual modes, ensuring the orderliness and reliability of mode switching. The dual-mode switching control module internally maintains a mode state machine, which includes four states: fault detection state, switching in progress state, natural language retrieval state, and waiting to switch back state.
[0077] The system defaults to fault detection mode and continuously executes the PIS screen fault detection process. When a natural language search command is received, the state machine jumps to the switching state, simultaneously performing prompt word template replacement and video source switching; after the switching is completed, it enters the natural language search state and performs target search and tracking tasks.
[0078] In natural language retrieval mode, when the retrieval task is completed, including the completion of time period traversal or manual termination by the user, the state machine transitions to the waiting-to-switch-back state and starts a timeout timer. If a new retrieval instruction is received within the preset timeout period, the timer is reset and the system returns to natural language retrieval mode to execute a new retrieval task; if no new instruction is received within the timeout period, the state machine transitions back to the fault detection state, restores the video source and prompt word configuration of fault detection mode, and continues screen monitoring.
[0079] The entire state transition process is automatically controlled by the module without manual intervention. During the switching process, the model weights and inference engine remain loaded, and only the input prompt words are changed, ensuring the switching speed and system stability.
[0080] The following describes the complete operation process of this system in a specific operational scenario.
[0081] Taking a fully automated subway line during the morning rush hour as an example, a 6-car train is running online, with 2 PIS screens in each car, for a total of 12 screens. The onboard edge computing unit deploys a quantized lightweight VL model and performs fault detection on each PIS screen at a frequency of 1 frame / second. The system is in fault detection mode by default.
[0082] During operation, the PIS screen on the left side of carriage 3 experienced a screen distortion fault, displaying colored stripes and a mosaic effect. Traditional brightness threshold methods cannot identify this type of fault because the average brightness of the screen is within the normal range. In this system, the PIS screen area positioning module detects the screen area and performs perspective correction and cropping before sending the screen image into a lightweight VL large model. The model uses fault detection prompts to guide semantic understanding and outputs a judgment result of screen distortion fault, with a moderate severity level, located across the entire screen. The recommended solution is to check the SVGA signal cable connection; if the cable is normal, the screen driver board is faulty and needs to be replaced.
[0083] The fault timing confirmation module verifies the detection results of three consecutive frames, all of which are determined to be the same screen flickering fault. Therefore, the fault is officially confirmed, a general fault is recorded, and a maintenance prompt is generated. After the train arrives at the terminal station, the maintenance personnel first check the SVGA cable according to the system prompt. They find that the cable pins are oxidized. After replacing the cable, the screen returns to normal, and there is no need to replace the screen itself.
[0084] At this time, the dispatch center received a report from a passenger that a red suitcase had been lost in carriage 3. The dispatcher entered the search command into the natural language interface: "Find the passenger carrying the red suitcase in carriage 3 between 8:00 and 8:30."
[0085] Upon receiving a search command, the dual-mode switching control module automatically switches to natural language search mode, replaces the prompt word template, and switches the video source to the panoramic camera in carriage 3. The system assesses the complexity of the search task: there is only one target feature, but the search time period is 30 minutes, exceeding the preset duration. Therefore, it is determined to be a complex search task, and the search request is uploaded to the cloud for execution using the full VL large model.
[0086] The cloud-based full VL (Video Library) model located three time points in a 30-minute video clip where the red suitcase appeared, outputting the target image and corresponding timestamps. After confirming the target, dispatchers contacted station staff in the corresponding time periods to assist in the search.
[0087] After the retrieval task is completed, the system enters the waiting switch-back state. If no new retrieval command is received within 30 seconds, it automatically switches back to the fault detection mode, restores the PIS screen camera video source and fault detection prompt words, and continues to monitor the display status of the entire vehicle's PIS screen.
[0088] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.
[0089] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A method for fault detection and natural language retrieval of subway PIS screens, characterized in that, Includes the following steps: The display screens on the subway PIS screens and the panoramic video stream of the carriages are captured by the vehicle-mounted cameras. A lightweight VL large model is deployed on an in-vehicle edge computing device. The lightweight VL large model is configured with a fault detection mode and a natural language retrieval mode. The two modes share the same set of model weights, and the mode can be switched by changing the input prompt words. In the default state, the fault detection mode is run. After the display screen is located and preprocessed, the screen area image and fault detection prompt are input into the lightweight VL large model. The lightweight VL large model determines whether there is a display fault on the subway PIS screen. If there is a fault, the fault type, severity level and handling suggestions are output. When a prompt word for a natural language search instruction is received, the system automatically switches to the natural language search mode. The panoramic video stream of the carriage and the search prompt word containing the user's natural language description are input into the lightweight VL model. The lightweight VL model then locates and matches the target in the video frames of the panoramic video stream of the carriage and outputs the target position. After the retrieval task is completed, if no new natural language retrieval instruction is received within the preset timeout period, the system will automatically switch back to the fault detection mode.
2. The subway PIS screen fault detection and natural language retrieval method according to claim 1, characterized in that, In the fault detection mode, the fault types identified by the lightweight VL large model include at least one of black screen, screen distortion, color cast, garbled text, cracked screen, frozen screen, or no signal; the fault detection prompts contain visual feature descriptions and severity level judgment criteria for each fault type, guiding the lightweight VL large model to output structured results containing fault type, severity level, fault location description, and handling suggestions.
3. The subway PIS screen fault detection and natural language retrieval method according to claim 1, characterized in that, The display screen is subjected to region positioning and preprocessing, including: The target detection model is used to detect the rectangular area of the subway PIS screen in the video frame of the displayed screen and output the bounding box coordinates. Perspective transformation correction is performed on the detected rectangular area to eliminate the installation angle tilt. The corrected screen area is cropped and scaled to the input resolution of the lightweight VL large model. When no subway PIS screen is detected or the area of the screen that is obscured exceeds a preset threshold, it is marked as an invisible screen state and no fault judgment is triggered.
4. The subway PIS screen fault detection and natural language retrieval method according to claim 1, characterized in that, After the lightweight VL large model determines the fault detection result of the subway PIS screen, if the lightweight VL large model judges the same type of fault in N consecutive frames, the fault is confirmed. N is an integer greater than or equal to 2. If the confirmed fault is judged as normal in subsequent frames, the countdown begins. When M consecutive frames are judged as normal, the alarm is deactivated. M is an integer greater than or equal to 2.
5. The subway PIS screen fault detection and natural language retrieval method according to claim 1, characterized in that, In the natural language retrieval mode, the lightweight VL large model directly understands the target description in the natural language retrieval instruction and locates and matches the target with zero samples in the video frames of the panoramic video stream of the carriage; after finding the target in the first frame, the lightweight target tracking algorithm is started in subsequent frames to continuously track the target and thus obtain the continuous position of the target.
6. The subway PIS screen fault detection and natural language retrieval method according to claim 1, characterized in that, When switching modes, only the prompt word templates sent to the lightweight VL large model are changed, while the weights and inference engine of the lightweight VL large model remain unchanged. The video source is also switched simultaneously when switching modes. In the fault detection mode, the on-board camera of the subway PIS screen is used, and in the natural language retrieval mode, the panoramic camera of the carriage is used.
7. The subway PIS screen fault detection and natural language retrieval method according to claim 1, characterized in that, It also includes edge-cloud collaboration steps: The lightweight VL large model is deployed at the edge, and all subway PIS screen fault detection tasks are completed at the edge. A full VL model is deployed in the cloud. The parameter scale of the full VL model is larger than that of the lightweight VL model. The full VL model is used to handle complex natural language retrieval tasks and troubleshooting. When the number of target features in the natural language retrieval task does not exceed a preset value and the retrieval time period does not exceed a preset duration, the natural language retrieval task is executed at the edge. When the number of target features exceeds a preset value or the retrieval time period exceeds a preset duration, the natural language retrieval task is executed at the cloud. When the fault judgment confidence obtained at the edge is lower than a preset threshold or the fault type is uncertain, the screen area image is uploaded to the cloud for review by the full VL large model.
8. The subway PIS screen fault detection and natural language retrieval method according to claim 1, characterized in that, The severity of the fault is divided into three levels: severe level is a full-screen black screen, full-screen distorted screen, or full-screen no signal fault; general level is a fault with color distortion, partial screen distortion, garbled text, or screen freezing fault; minor level is a fault with abnormal brightness or occasional flickering fault; different alarm strategies are corresponding to different levels.
9. A subway PIS screen fault detection and natural language retrieval system, wherein the system applies the method described in any one of claims 1-8, characterized in that, include: The video perception layer includes an onboard camera aimed at the subway PIS screen and a panoramic camera in the carriage, which are used to capture the display screen of the subway PIS screen and the panoramic video stream of the carriage, respectively. The edge computing layer includes a subway PIS screen area positioning module, a lightweight VL large model inference module, a dual-mode switching control module, and a fault timing confirmation module, which are used to perform subway PIS screen area positioning, VL large model inference, dual-mode switching control, and fault timing confirmation on the on-board edge computing device. The edge-cloud collaboration layer includes an edge inference result uploading module, a cloud large model scheduling module, and a retrieval task distribution module, which are used to manage task allocation and data transmission between the edge and the cloud. The dual-mode service layer includes a fault detection service module, a natural language retrieval service module, and a unified event output module, which are used to output fault information in fault detection mode and output target location results in natural language retrieval mode. The interaction and alarm layer includes an in-vehicle display terminal, a dispatch center alarm interface, a natural language interaction interface, and an event evidence retention module, which are used to display alarms and retrieval results, provide natural language input, and save event evidence.
10. The subway PIS screen fault detection and natural language retrieval system according to claim 9, characterized in that, The dual-mode switching control module maintains the mode state machine, which is in the fault detection mode by default. When a prompt word for a natural language retrieval instruction is received, it automatically switches to the natural language retrieval mode. When the retrieval is completed and no new natural language retrieval instruction is received within a preset timeout period, it automatically switches back to the fault detection mode. During mode switching, only the input prompt word of the VL large model is changed, while the model weights and inference engine remain unchanged.
Citation Information
Patent Citations
Generative large model-based natural language interactive security video retrieval system and device
CN118551077A
Detection method and device of vehicle-mounted display screen, electronic equipment and medium
CN122116777A
