Intelligent Elevator Fault Handling System and Method Based on Multimodal Large Model and Probabilistic Causal Graph
The elevator fault intelligent processing system, which utilizes a multimodal large model and probabilistic causal graph, enables rapid root cause localization and automated handling of elevator faults. This solves the problems of reliance on manual labor and insufficient collaborative capabilities in existing technologies, thereby improving the efficiency and self-healing ability of elevator fault handling.
Patent Information
- Application Number
- CN202611292447.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-25
- Publication Date
- 2026-11-10
AI Technical Summary
Existing elevator fault handling methods rely on manual experience, resulting in long troubleshooting cycles and low efficiency when facing complex scenarios. They also lack automated handling and multi-system collaboration capabilities, making it difficult to adapt to high-load operation requirements.
By employing a multimodal large model and probabilistic causal graph, multi-source heterogeneous data is collected through the elevator Internet of Things. After standardized preprocessing, modal coding fusion is performed. The time-series multimodal large model is used for fault assessment and causal reasoning, automatically generating troubleshooting workflows and realizing cross-system collaborative handling.
It enables rapid fault root cause localization and standardized handling solution generation, improves fault handling efficiency, enhances self-healing capabilities and system synergy, and reduces manual intervention.
Smart Images

Figure CN123069076A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of elevator maintenance technology, specifically, it relates to an intelligent elevator fault handling system and method based on a multimodal large model and probabilistic causal graph. Background Technology
[0002] In the traditional elevator fault handling model, the entire fault response process is centered on manual intervention: when an elevator triggers a fault code alarm, maintenance and dispatch personnel manually receive the report, assess the fault level, and dispatch emergency repair personnel to the site; on-site workers then match the corresponding fault handling plan by consulting fault manuals and requesting remote technical assistance, combining their own experience to troubleshoot components one by one, locate the root cause of the fault, and complete the repair. This model heavily relies on the technical skills and experience of personnel. When faced with complex scenarios such as multiple concurrent faults or hidden faults, the troubleshooting cycle is long, the handling efficiency is low, and the quality of troubleshooting is easily inconsistent due to differences in personnel capabilities, making it difficult to adapt to the emergency response needs of high-load operation scenarios.
[0003] With the development of intelligent operation and maintenance technology, large-scale models and knowledge graphs are gradually being applied to the elevator operation and maintenance field, which has improved the level of intelligence in fault diagnosis to a certain extent, but there are still obvious limitations. For example, patent application CN 122066144 A proposes a decision-making method for elevator full-cycle operation and maintenance that combines large-scale models and knowledge graphs, which can realize fault prediction and maintenance plan generation. However, its solution focuses on pre-emptive predictive maintenance, and the knowledge graph is only used for component association retrieval and maintenance process matching. It has not established a quantitative fault causal propagation reasoning mechanism, and its support for accurate emergency handling of faults that have already occurred is insufficient. Another example is patent application CN 122241051 A, which proposes an elevator fault diagnosis and handling system. It generates a visual fault navigation to assist on-site operations through causal rule matching. However, its handling process is still based on manual execution, and it has not built an automated handling workflow. It cannot achieve graded and differentiated fault self-healing, and the data interoperability between property management, parts and other business systems is poor, lacking a unified collaborative scheduling mechanism.
[0004] Therefore, existing technologies still have technical problems such as weak interpretability of fault root cause localization, low degree of automation in handling, insufficient multi-system collaboration capability, lack of self-healing capability for minor faults, and lagging model capability iteration. They also lack automated fault handling strategies and intelligent troubleshooting mechanisms for collaborative interaction among various business systems. Summary of the Invention
[0005] To address the aforementioned problems in the existing technology, this application provides an intelligent elevator fault handling system and method based on a multimodal large model and probabilistic causal graph.
[0006] In one approach, an intelligent elevator fault handling system based on a multimodal large model and probabilistic causal graph is proposed. The system architecture consists of an elevator, a fault handling execution terminal, an intelligent fault handling system, a knowledge base system, and a third-party interactive platform working together to form a system architecture.
[0007] Among them, the elevator, as the front-end sensing and controlled execution object, uploads the operation data, fault data and status parameters of the whole elevator and its components to the intelligent fault handling system in real time through the data transmission unit. At the same time, it receives control and disposal instructions issued by the system, realizing two-way interaction between status up and instruction down.
[0008] The intelligent fault handling system includes a fault handling subsystem and a multimodal large model subsystem. The fault handling subsystem is responsible for data access, equipment basic information management and business scheduling. The multimodal large model subsystem performs intelligent processing of the entire chain from data input to fault causal reasoning and handling decision-making.
[0009] The knowledge base system provides underlying domain knowledge support for intelligent fault handling systems, storing expert knowledge of structured fault cases and unstructured technical documents.
[0010] The fault handling execution terminal is an interactive platform for maintenance technicians to use in on-site operations, and it is linked with the intelligent fault handling system in two directions.
[0011] The third-party interaction platform connects with the intelligent fault handling system through a standardized API connection pool to achieve data exchange and command coordination with external systems.
[0012] In one embodiment, to achieve the above objectives, this application also provides an intelligent elevator fault handling method based on a multimodal large model and probabilistic causal graph. This method is used to implement the aforementioned intelligent elevator fault handling system based on a multimodal large model and probabilistic causal graph, and includes the following steps: S1. Collect multi-source heterogeneous data on elevator operation, faults, components and spatial scenarios in all dimensions through elevator IoT and system integration. Eliminate differences in data format, frequency and dimension through standardized preprocessing, and output unified format data that is compatible with the model and graph inference. S2. Different modal tensor data are independently encoded using corresponding modal encoders, and then multi-dimensional information is deeply fused through modal identifier embedding and feature concatenation to generate a unified-dimensional multi-modal fusion feature tensor. S3. Utilize the temporal multimodal large model to perform forward computation on the multimodal fusion feature tensor of S2, simultaneously complete the fault anomaly degree assessment and fault type identification, output the data-driven preliminary fault judgment result, and provide the initial confidence input for knowledge graph reasoning; S4. The fault assessment results output by the time series multimodal large model are transformed into the initial confidence of knowledge graph nodes. The fault propagation link is traversed in the forward direction using the probabilistic graph causal reasoning algorithm to complete the location of the root cause of the fault, the restoration of the propagation path and the quantification of the danger level. S5. Based on the root cause location results and comprehensive hazard level, call the troubleshooting decision strategy library, and automatically arrange and generate intelligent troubleshooting workflow through hierarchical recursive decomposition of the large model and rule constraints. Automatically generate standardized handling solutions for different levels of faults. S6. Based on the functional attributes and handling requirements of the troubleshooting intelligent workflow generated by S5, automatically select matching execution entities, and achieve multi-party collaborative linkage through standardized API interfaces to complete the full-link execution, monitoring and early warning of fault handling.
[0013] Furthermore, in step S2, at least the timing numerical signal, audio waveform data, gantry machine image data, and equipment three-dimensional spatial data are independently encoded; Among them, the time-series numerical signal is extracted by using a one-dimensional convolutional network combined with a time-series position embedding encoder to extract the time-domain variation characteristics and trend patterns of the signal; the audio waveform data is first converted from one-dimensional waveform to two-dimensional frequency domain features through Mel spectrum conversion, and then the acoustic features are extracted by the audio Transformer encoder; the door machine image data is extracted by using a lightweight CNN visual encoder to extract spatial visual features and capture visual anomalies such as door deformation, component offset, and foreign object jamming; the equipment three-dimensional spatial data is superimposed on the surface of the real equipment by the elevator digital twin model, equipment standard parameters, and historical anomaly records through the AR three-dimensional anchoring algorithm, completing the binding between the real scene and the digital model, and outputting spatial structural features and component position correlation features.
[0014] Furthermore, in step S4, the system pre-constructs an elevator fault causal knowledge graph. The knowledge graph uses elevator hardware components as graph nodes and the strength of the fault causal relationship between components as edge weights, thus accumulating the fault propagation patterns, causal relationships, and historical fault statistics of all components in the elevator system.
[0015] Furthermore, in step S5, the troubleshooting decision strategy library is generated by a multimodal large model based on pre-trained reasoning of the troubleshooting knowledge base, covering standardized handling procedures, operating procedures and resource allocation requirements for different fault types and hazard levels.
[0016] Furthermore, in step S5, the intelligent troubleshooting workflow includes seven core node types: start / end node, large model node, plug-in node, selector node, loop node, data processing node, and execution node.
[0017] The workflow is defined by several nodes: Start / End nodes define the beginning and end of the workflow lifecycle; Large Model nodes call multimodal large models to complete cognitive tasks such as intelligent analysis and strategy generation; Plugin nodes connect to third-party systems, external tools, and custom plugins to achieve cross-system functional expansion; Selector nodes have built-in judgment conditions and automatically match different branch handling paths based on the execution results; Loop nodes set the number of loops or termination conditions to repeatedly execute specific process segments until the requirements are met; Data Processing nodes execute data processing tasks; and Execution nodes issue operation instructions to each execution object to drive specific handling actions to be implemented.
[0018] Furthermore, the targets of third-party systems include at least four categories: property management personnel, maintenance technicians, technical experts, and third-party platform personnel.
[0019] Furthermore, for minor fault scenarios where system malfunctions do not require on-site intervention, the troubleshooting tasks automatically orchestrated by the workflow are as follows: S511, Start Node; S512, Large Model Node: Generates the overall troubleshooting strategy; S513, Large Model Node: Select technical personnel to issue troubleshooting strategies; S514, Plug-in Node: Technical personnel review troubleshooting strategies; S515, Plug-in Node: Operation execution instructions generated by the system to the cloud service platform; S516, Data Node: The system automatically confirms the fault handling result based on the received elevator status data; S517, Loop Node: If the fault is not eliminated, jump to S514 and repeat the execution; S518 Select Node: If the fault is cleared, end execution; if the fault is not cleared, jump to S512. S519, End Node.
[0020] Furthermore, for severe malfunction scenarios involving passengers trapped in the elevator car, the automatically scheduled troubleshooting tasks in the workflow are as follows: S521, Start Node; S522, Large Model Node: The language model deployed on the elevator edge gateway generates contextualized reassuring voice text based on the image recognition results of the people in the car to provide reassuring prompts to the people in the car; at the same time, it receives voice input from the car intercom, completes intention recognition and automatic response, and dynamically interacts with the people in the car; S523, Large Model Node: Generates the overall troubleshooting strategy; S524, Large Model Node: Generates rescue work orders and selects suitable technical personnel; S525, Plug-in Node: The system sends troubleshooting operation instructions to technicians' mobile smart terminals; S526, Execution Node: Technicians perform obstacle removal and rescue operations according to the obstacle removal operation guidelines; S527, Data Node: The system automatically confirms the fault handling result based on the received elevator status data; S528. Select Node: If the fault is cleared, execution ends; if the fault is not cleared, a technical expert will provide remote guidance. S529, End Node.
[0021] Furthermore, intelligent elevator fault handling methods based on multimodal large models and probabilistic causal graphs also include: S7. Collect full-process data for each execution and build a labeled fault sample library; and use an incremental fine-tuning algorithm to regularly update the model online.
[0022] The beneficial effects of this application are: This application uses a multimodal large model probabilistic causal graph inference algorithm to automatically output fault transmission chains, enabling rapid fault root cause localization; it automatically orchestrates and generates standardized troubleshooting solutions in an intelligent workflow, with each node automatically selecting execution objects and interacting with related platforms. The system acts as the intelligent hub for fault handling, making decisions and scheduling centrally, which greatly improves fault handling efficiency. Attached Figure Description
[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0024] Figure 1 This is a schematic diagram of the system architecture of an intelligent elevator fault handling system based on a multimodal large model and probabilistic causal graph in one embodiment of this application; Figure 2 This is a flowchart illustrating an embodiment of the intelligent elevator fault handling method based on a multimodal large model and probabilistic causal graph in this application. Detailed Implementation
[0025] The specific embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate this application, but are not intended to limit the scope of this application. Similarly, the following examples are only some embodiments of this application, not all embodiments. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0026] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0027] In this application, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0028] To further illustrate the technical means and effects adopted by this application to achieve its technical objectives, the following detailed description of specific implementation methods based on this application is provided in conjunction with representative embodiments: In one embodiment, please refer to Figure 1 An intelligent elevator fault handling system based on a multimodal large model and probabilistic causal graph is proposed. The system architecture consists of an elevator, a fault handling execution terminal, an intelligent fault handling system, a knowledge base system, and a third-party interactive platform working together.
[0029] In this embodiment, the elevator, as the front-end sensing and controlled execution object, uploads multi-source heterogeneous information such as the operation data, fault data, and status parameters of the whole elevator and its components to the intelligent fault handling system in real time through the data transmission unit (DTU). At the same time, it receives control and handling instructions issued by the system, realizing bidirectional interaction between status up and instruction down.
[0030] In this embodiment, the intelligent fault handling system serves as the intelligent hub of the entire architecture, comprising a fault handling subsystem and a multimodal large-scale model subsystem. The fault handling subsystem handles data access, equipment basic information management, and business scheduling. Through connection with the elevator, it acquires real-time fault-related data, elevator operation data, equipment parameters, and stores basic elevator information and maintenance personnel information. The multimodal large-scale model subsystem includes a multimodal time-series preprocessing module, a multimodal feature fusion encoding module, a multimodal analysis and reasoning module, a fault causal graph reasoning module, an AI intelligent workflow generation and decision-making module, a decision execution and interaction module, and an incremental fine-tuning module. It performs end-to-end intelligent processing from data input to fault causal reasoning and response decisions.
[0031] In this embodiment, the knowledge base system provides underlying domain knowledge support for the intelligent fault handling system, storing structured and unstructured expert knowledge including fault cases, fault manuals, technical documents, video images, etc., for use in large model reasoning, causal graph construction, and troubleshooting solution generation; at the same time, it receives new case data that is returned after the fault handling is completed, realizing the dynamic updating of the knowledge system.
[0032] In this embodiment, the fault handling execution terminal is an interactive carrier for maintenance technicians to use in on-site operations. It is linked with the intelligent fault handling system in two directions, pushing information such as diagnostic conclusions, fault transmission paths, and standardized operation guidelines to maintenance personnel and technical experts. It supports on-site troubleshooting in multiple terminal forms such as PDAs, smart wearable devices, and cloud platforms, and transmits on-site handling results and personnel feedback back to the core system in real time.
[0033] In this embodiment, the third-party interactive platform connects with the intelligent fault handling system through a standardized API connection pool to achieve data exchange and command coordination with external systems such as property management and parts management. This supports resource scheduling, work order linkage, and cross-system business collaboration during the troubleshooting process, ensuring the efficient implementation of the handling process. The execution results are synchronously transmitted back to the core system, together forming a complete closed loop of fault handling and capability iteration.
[0034] In one embodiment, as an implementation scheme, please refer to Figure 2 This application also provides an intelligent elevator fault handling method based on a multimodal large model and probabilistic causal graph. The method is used to implement the intelligent elevator fault handling system based on a multimodal large model and probabilistic causal graph in the foregoing embodiments. The method includes the following steps: Step S1: Collect multi-source heterogeneous data on elevator operation, faults, components and spatial scenarios from all dimensions through elevator IoT and system integration. Eliminate differences in data format, frequency and dimension through standardized preprocessing, and output unified format data for adaptation model and graph inference.
[0035] Specifically, the system synchronously collects comprehensive elevator data through multiple channels, including elevator IoT DTU terminals, system integration interfaces, and field sensing devices. This data covers control and electrical signals output from the elevator control cabinet, vibration and audio data collected by sensors installed on key components, image and video data from the car and door operator areas, elevator component parameters and equipment ledger data, and three-dimensional spatial scene data of the equipment. All raw data is uploaded to the intelligent elevator fault handling system via a unified transmission link for distributed storage and centralized management.
[0036] In this step, differentiated preprocessing algorithms are performed for different types of data, outputting standardized multimodal raw tensors. Specifically, for time-series data, uniform interpolation alignment and frame-by-frame sliding window standardized sampling are performed to ensure consistency in the time dimension; for continuous signals such as vibration and current, wavelet transform is used to perform noise reduction processing to remove environmental interference and acquisition noise; for image frames, grayscale normalization and size standardization are performed to unify visual input specifications; and for structured numerical data, range standardization and outlier removal are performed to standardize the numerical distribution.
[0037] Through the above processing, multi-source heterogeneous data is transformed into multimodal raw tensors with uniform dimensions and controllable quality, providing standardized input for subsequent feature extraction and model inference.
[0038] Step S2: Encode the tensor data of different modalities independently using the corresponding modal encoders, and then achieve deep fusion of multi-dimensional information through modal identifier embedding and feature concatenation to generate a multi-modal fusion feature tensor of a unified dimension.
[0039] In this step, at least the timing numerical signal, audio waveform data, gantry machine image data, and equipment 3D spatial data are independently encoded.
[0040] The time-series numerical signals are processed using a one-dimensional convolutional network combined with a time-series position embedding encoder to extract the temporal variation characteristics and trend patterns of the signals, capturing temporal patterns such as parameter fluctuations and abrupt changes. Audio waveform data is first transformed from a one-dimensional waveform into two-dimensional frequency domain features through Mel-frequency conversion, and then acoustic features are extracted by an audio Transformer encoder to identify abnormal acoustic signals such as unusual noises and friction sounds. Door operator image data uses a lightweight CNN visual encoder to extract spatial visual features, capturing visual anomalies such as door deformation, component misalignment, and foreign object jamming. The equipment's three-dimensional spatial data is processed using an AR three-dimensional anchoring algorithm to overlay the elevator digital twin model, equipment standard parameters, and historical anomaly records onto the surface of the real-world equipment, completing the binding between the real-world and digital models, and outputting spatial structural features and component position correlation features.
[0041] Furthermore, after completing the encoding of each single modality, the feature vectors output by each modality are dimensionally aligned and concatenated. Simultaneously, a unique modality identifier embedding vector is superimposed on each modality feature to distinguish the feature source and modality attribute, ultimately generating a multimodal fusion feature tensor of uniform length. Modality identifier embedding can guide subsequent models to adaptively allocate attention weights to different modalities, strengthen the feature contribution of highly relevant modalities, and improve the comprehensive representational ability of the fused features.
[0042] Step S3: Utilize the temporal multimodal large model to perform forward computation on the multimodal fusion feature tensor of S2, simultaneously complete the fault anomaly degree assessment and fault type identification, output the data-driven preliminary fault judgment result, and provide initial confidence input for knowledge graph reasoning.
[0043] Specifically, the system employs a lightweight temporal multimodal large model that has undergone supervised fine-tuning using massive multimodal samples of elevator faults. The model performs parameter quantization and structural optimization specifically for elevator scenarios, and can be adapted to both cloud deployment and low-latency edge inference. After inputting the multimodal fusion feature tensor into the large model, it simultaneously outputs two types of results—fault anomaly scores and fault type results—through forward inference. These results are directly used as initial confidence scores and injected into the subsequent fault knowledge graph to drive the initiation and computation of probabilistic causal inference.
[0044] The fault anomaly score ranges from 0 to 1. A higher score indicates a greater deviation of the elevator's current operating state from normal conditions, and a higher probability of a fault occurring. The fault type results output multi-category fault type labels and corresponding confidence levels, which can accurately identify various typical elevator faults such as door system faults, traction system faults, and control system faults.
[0045] Step S4: Convert the fault assessment results output by the time-series multimodal large model into the initial confidence of knowledge graph nodes, and use the probabilistic graph causal reasoning algorithm to traverse the fault propagation link in a forward direction to complete the location of the root cause of the fault, the reconstruction of the propagation path and the quantification of the hazard level.
[0046] In this step, the system pre-constructs a knowledge graph of elevator fault causes. The knowledge graph uses elevator hardware components as graph nodes and the strength of the fault causal relationship between components as edge weights, thus accumulating the fault propagation patterns, causal relationships, and historical fault statistics of all components in the elevator system.
[0047] At the start of inference, the fault type and anomaly score output by the large model are mapped to the initial activation confidence of the corresponding component nodes in the knowledge graph, thus injecting the judgment results of the large model into the confidence of the knowledge graph. Subsequently, a probabilistic graphical causal inference algorithm is used to perform a forward traversal of the entire link along the fault propagation direction. By calculating the posterior probability of the fault at each node through probability superposition and propagation, three core inference results are finally output: root cause component, fault propagation path, and comprehensive fault hazard level.
[0048] Among these, the root cause component, being the component with the highest posterior probability of fault location, is identified as the root cause of the fault, thus solving the problem of root cause localization when multiple faults occur concurrently. The fault propagation path reconstructs the complete propagation link from the root cause component to the fault manifestation, clearly presenting the causal transmission logic of the fault and ensuring the interpretability of the reasoning process. The comprehensive fault hazard level, combined with the fault's impact range, safety risk, and probability of occurrence, can be divided into three levels: Level 1 is a minor fault, mostly false alarm fault codes or minor anomalies that do not affect operation; Level 2 is a general fault, mostly recurring functional faults; and Level 3 is a serious fault, such as personnel entrapment faults, braking system anomalies, and other high-risk faults involving personal safety or equipment shutdown.
[0049] Step S5: Based on the root cause location results and comprehensive hazard level, call the troubleshooting decision strategy library, automatically arrange and generate a troubleshooting intelligent workflow through hierarchical recursive decomposition of the large model and rule constraints, and automatically generate standardized handling plans for different levels of faults.
[0050] Among them, the troubleshooting decision strategy library is generated by multimodal large model based on the troubleshooting knowledge base through pre-training inference, covering standardized handling procedures, operating procedures and resource allocation requirements for different fault types and hazard levels.
[0051] In this step, the workflow orchestration adopts a hierarchical adaptive self-healing decision algorithm. Based on the prompt word engineering and business rule constraints, the multimodal large model performs hierarchical recursive decomposition of the overall troubleshooting task, and automatically generates a standardized troubleshooting workflow composed of multiple functional nodes. The workflow includes seven types of core nodes: start / end nodes, large model nodes, plug-in nodes, selector nodes, loop nodes, data processing nodes, and execution nodes.
[0052] Specifically, start / end nodes define the beginning and end of the workflow lifecycle; large model nodes call multimodal large models to complete cognitive tasks such as intelligent analysis and strategy generation; plug-in nodes connect to third-party systems, external tools, and custom plug-ins to achieve cross-system functional expansion; selector nodes have built-in judgment conditions and automatically match different branch handling paths based on the execution results; loop nodes set the number of loops or termination conditions to repeatedly execute specific process segments until the requirements are met; data processing nodes execute data processing tasks; and execution nodes issue operation instructions to each execution object to drive specific handling actions to be implemented.
[0053] Step S6: Based on the functional attributes and handling requirements of the troubleshooting intelligent workflow generated in S5, automatically select matching execution entities, and achieve multi-party collaborative linkage through standardized API interfaces to complete the full-link execution, monitoring and early warning of fault handling.
[0054] In this step, for the execution requirements of each node in the workflow, the system automatically selects matching execution objects. The execution entities include both natural persons and third-party systems. The execution objects of third-party systems include at least four categories: property management personnel, maintenance technicians, technical experts, and third-party platform personnel.
[0055] Among them, property management personnel connect to the property management system to automatically push fault reports, maintenance notices, fault elimination notifications, and cooperation and assistance requests, ensuring information synchronization and on-site cooperation from the property side.
[0056] Maintenance technicians automatically select the optimal on-site personnel based on factors such as location, skills, and work order load. The system then sends tasks and operational instructions to personnel's smart terminals through the fault handling system to guide on-site troubleshooting operations.
[0057] For new, frequent, difficult, or serious faults such as those causing people to get stuck, technical experts can automatically trigger a remote intervention mechanism. Relying on the equipment's 3D digital twin model, they can complete 3D marking, point selection, dimension annotation, virtual component disassembly animation demonstration, and parameter threshold scale annotation in the real-world scene. All virtual guidance content is immersively superimposed onto the real-world equipment scene in real time, achieving virtual and real synchronous guidance that is hands-on from a distance.
[0058] Through the configured API connection pool, the third-party platform allows the plug-in nodes in the workflow to automatically configure connection information, send data requests and issue instructions to the corresponding platform / system, and receive returned data. For example, in the case of parts replacement, it automatically sends parts allocation instructions to the parts management system, and automatically pushes the task to the selected technician after the parts arrive.
[0059] Step S7: Collect full-process data for each execution, build a labeled fault sample library, and use an incremental fine-tuning algorithm to regularly update the model online.
[0060] Specifically, after the fault handling is completed, the system automatically collects all data from the entire process and incorporates it into a cloud-based sample library. The collected data includes original multimodal tensors, multimodal fusion features, fault label information, causal root cause localization results, a complete record of intelligent workflow execution, manual maintenance operation records, and evaluation of handling effectiveness, among other multi-dimensional data. The system automatically cleans, verifies the quality of, and standardizes the returned data to build a continuously expanding elevator fault labeling sample library.
[0061] The optimization phase employs an incremental fine-tuning algorithm, periodically performing online updates on the multimodal large model. It fine-tunes the top-level parameters of the model based solely on new samples, eliminating the need for full retraining. This significantly reduces computational consumption and update costs, continuously improving the model's fault feature extraction accuracy, type identification accuracy, and the rationality of solution generation. Simultaneously, the system updates the edge weights and node attributes of the fault causal knowledge graph based on new fault cases, achieving collaborative optimization of the probabilistic reasoning system and ensuring continuous iterative upgrades of system capabilities as data accumulates.
[0062] In one embodiment, as a preferred implementation of this application, for minor fault scenarios where system malfunctions do not require on-site intervention, such as gantry crane torque adaptive adjustment algorithms or door opening / closing retry loop control logic, the troubleshooting tasks automatically orchestrated by the workflow are as follows: S511, Start Node; S512, Large Model Node: Generates the overall troubleshooting strategy; S513, Large Model Node: Select technical personnel to issue troubleshooting strategies; S514, Plug-in Node: Technical personnel review troubleshooting strategies; S515, Plug-in Node: Operation execution instructions generated by the system to the cloud service platform; S516, Data Node: The system automatically confirms the fault handling result based on the received elevator status data; S517, Loop Node: If the fault is not eliminated, jump to S514 and repeat the execution; S518 Select Node: If the fault is cleared, end execution; if the fault is not cleared, jump to S512. S519, End Node.
[0063] In one embodiment, as a preferred implementation of this application, for serious malfunction scenarios such as passengers trapped in the elevator car, the automatically scheduled troubleshooting tasks are as follows: S521, Start Node; S522, Large Model Node: The language model deployed on the elevator edge gateway generates contextualized reassuring voice text based on the image recognition results of the people in the car to provide reassuring prompts to the people in the car; at the same time, it receives voice input from the car intercom, completes intention recognition and automatic response, and dynamically interacts with the people in the car; S523, Large Model Node: Generates the overall troubleshooting strategy; S524, Large Model Node: Generates rescue work orders and selects suitable technical personnel; S525, Plug-in Node: The system sends troubleshooting operation instructions to technicians' mobile smart terminals (PDAs); S526, Execution Node: Technicians perform obstacle removal and rescue operations according to the obstacle removal operation guidelines; S527, Data Node: The system automatically confirms the fault handling result based on the received elevator status data; S528. Select Node: If the fault is cleared, execution ends; if the fault is not cleared, a technical expert will provide remote guidance. S529, End Node.
[0064] In summary, this application achieves multi-task reasoning for fault type identification and anomaly score assessment by standardizing and preprocessing multi-source heterogeneous elevator data and encoding its modal features, combined with a lightweight multimodal large model specifically for elevators. The output of the large model is then used as the initial confidence level and injected into a probabilistic causal knowledge graph. Through a probabilistic graphical causal reasoning algorithm, the fault propagation chain is traversed forward to automatically output the complete fault transmission path. This approach can quickly and accurately locate the root cause component of the fault and quantify the comprehensive hazard level, combining data-driven diagnostic accuracy with knowledge-driven logical interpretability.
[0065] This application also generates a standardized intelligent troubleshooting workflow based on adaptive orchestration of fault hazard levels. Each node automatically matches and selects the corresponding execution object. The system acts as an intelligent hub to uniformly schedule collaborative operations among multiple entities and platforms, significantly improving the organizational efficiency and cross-process collaboration of fault handling. At the same time, it adopts a hierarchical adaptive decision-making strategy, which can automatically perform self-healing operations such as parameter adaptive adjustment and control logic retry for minor faults, eliminating abnormalities without the need for technical personnel to be present, significantly improving the elevator's fault self-healing capability and effectively shortening the fault downtime. Through a standardized API connection pool, it connects with multiple third-party platforms such as property management and parts management to achieve cross-system data interaction and command collaboration. It can automatically collect elevator operation data to verify the handling effect, effectively reducing manual intervention and accelerating the entire fault handling process.
[0066] Furthermore, relying on the end-to-end fault data feedback mechanism and incremental fine-tuning algorithm, this application can also continuously optimize the feature extraction capability and fault identification accuracy of the large model as fault samples accumulate, thereby achieving a positive closed-loop iteration of diagnosis and treatment capabilities.
[0067] The above are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. An intelligent elevator fault handling system based on a multimodal large model and probabilistic causal graph, characterized in that, The system is composed of an elevator, a fault handling execution terminal, an intelligent fault handling system, a knowledge base system, and a third-party interactive platform working together to form a system architecture. Among them, the elevator, as the front-end sensing and controlled execution object, uploads the operation data, fault data and status parameters of the whole elevator and its components to the intelligent fault handling system in real time through the data transmission unit, and at the same time receives the control and handling instructions issued by the system, realizing two-way interaction between status up and instruction down; The intelligent fault handling system includes a fault handling subsystem and a multimodal large model subsystem. The fault handling subsystem is responsible for data access, equipment basic information management and business scheduling. The multimodal large model subsystem performs intelligent processing across the entire chain from data input to fault causal reasoning and handling decision-making. The knowledge base system provides underlying domain knowledge support for the intelligent fault handling system, storing expert knowledge of structured fault cases and unstructured technical documents; The fault handling execution terminal is an interactive platform for maintenance technicians to use in on-site operations, and it is linked bidirectionally with the intelligent fault handling system. The third-party interaction platform connects with the intelligent fault handling system through a standardized API connection pool to achieve data exchange and command coordination with external systems.
2. A method for intelligent elevator fault handling based on a multimodal large model and probabilistic causal graph, characterized in that, The method is used to implement the elevator fault intelligent processing system based on multimodal large model and probabilistic causal graph as described in claim 1, and the method includes the following steps: S1. Collect multi-source heterogeneous data on elevator operation, faults, components and spatial scenarios in all dimensions through elevator IoT and system integration. Eliminate differences in data format, frequency and dimension through standardized preprocessing, and output unified format data that adapts to the model and graph inference. S2. Different modal tensor data are independently encoded using corresponding modal encoders, and then multi-dimensional information is deeply fused through modal identifier embedding and feature concatenation to generate a unified-dimensional multi-modal fusion feature tensor. S3. Utilize the temporal multimodal large model to perform forward computation on the multimodal fusion feature tensor of S2, simultaneously complete the fault anomaly degree assessment and fault type identification, output the data-driven preliminary fault judgment result, and provide the initial confidence input for knowledge graph reasoning; S4. The fault assessment results output by the time series multimodal large model are transformed into the initial confidence of knowledge graph nodes. The fault propagation link is traversed in the forward direction using the probabilistic graph causal reasoning algorithm to complete the location of the root cause of the fault, the restoration of the propagation path and the quantification of the danger level. S5. Based on the root cause location results and comprehensive hazard level, call the troubleshooting decision strategy library, and automatically arrange and generate intelligent troubleshooting workflow through hierarchical recursive decomposition of large model and rule constraints. Automatically generate standardized handling solutions for different levels of faults. S6. Based on the functional attributes and handling requirements of the troubleshooting intelligent workflow generated by S5, automatically select matching execution entities, and achieve multi-party collaborative linkage through standardized API interfaces to complete the full-link execution, monitoring and early warning of fault handling.
3. The intelligent elevator fault handling method based on a multimodal large model and probabilistic causal graph as described in claim 2, characterized in that, In step S2, at least the time-series numerical signal, audio waveform data, gantry crane image data, and equipment three-dimensional spatial data are independently encoded. Among them, the time-series numerical signal is extracted by using a one-dimensional convolutional network combined with a time-series position embedding encoder to extract the time-domain variation characteristics and trend patterns of the signal; the audio waveform data is first converted from one-dimensional waveform to two-dimensional frequency domain features through Mel spectrum conversion, and then the acoustic features are extracted by the audio Transformer encoder; the door machine image data is extracted by using a lightweight CNN visual encoder to extract spatial visual features and capture visual anomalies such as door deformation, component offset, and foreign object jamming; the equipment three-dimensional spatial data is superimposed on the surface of the real equipment by the elevator digital twin model, equipment standard parameters, and historical anomaly records through the AR three-dimensional anchoring algorithm, completing the binding between the real scene and the digital model, and outputting spatial structural features and component position correlation features.
4. The intelligent elevator fault handling method based on multimodal large model and probabilistic causal graph as described in claim 2, characterized in that, In step S4, the system pre-constructs an elevator fault causal knowledge graph. The knowledge graph uses elevator hardware components as graph nodes and the strength of the fault causal relationship between components as edge weights to precipitate the fault propagation rules, causal relationships and historical fault statistics of all elevator system components.
5. The intelligent elevator fault handling method based on multimodal large model and probabilistic causal graph as described in claim 2, characterized in that, In step S5, the troubleshooting decision strategy library is generated by a multimodal large model based on pre-trained reasoning of the troubleshooting knowledge base, covering standardized handling procedures, operating procedures and resource allocation requirements for different fault types and hazard levels.
6. The intelligent elevator fault handling method based on multimodal large model and probabilistic causal graph as described in claim 5, characterized in that, In step S5, the intelligent troubleshooting workflow includes seven types of core nodes: start / end node, large model node, plug-in node, selector node, loop node, data processing node, and execution node. The workflow is defined by the start / end nodes, which define the beginning and end points of the workflow lifecycle. The large model node invokes a multimodal large model to complete cognitive tasks such as intelligent analysis and strategy generation. The plug-in node interfaces with third-party systems, external tools, and custom plug-ins to achieve cross-system functional expansion. The selector node has built-in judgment conditions and automatically matches different branch handling paths based on the execution results. The loop node sets the number of loops or termination conditions, repeatedly executing specific process segments until the requirements are met. The data processing node executes data processing tasks. The execution node issues operation instructions to each execution object, driving specific handling actions to be implemented.
7. The intelligent elevator fault handling method based on multimodal large model and probabilistic causal graph as described in claim 6, characterized in that, The targets of the third-party system include at least four categories: property management personnel, maintenance technicians, technical experts, and third-party platform personnel.
8. The intelligent elevator fault handling method based on multimodal large model and probabilistic causal graph as described in claim 7, characterized in that, For minor system malfunctions that do not require on-site intervention, the workflow automatically orchestrates the following troubleshooting tasks: S511, Start Node; S512, Large Model Node: Generates the overall troubleshooting strategy; S513, Large Model Node: Select technical personnel to issue troubleshooting strategies; S514, Plug-in Node: Technical personnel review troubleshooting strategies; S515, Plug-in Node: Operation execution instructions generated by the system to the cloud service platform; S516, Data Node: The system automatically confirms the fault handling result based on the received elevator status data; S517, Loop Node: If the fault is not eliminated, jump to S514 and repeat the execution; S518 Select Node: If the fault is cleared, end execution; if the fault is not cleared, jump to S512. S519, End Node.
9. The intelligent elevator fault handling method based on multimodal large model and probabilistic causal graph as described in claim 7, characterized in that, For severe malfunction scenarios where passengers are trapped in the elevator car, the automatically scheduled troubleshooting tasks are as follows: S521, Start Node; S522, Large Model Node: The language model deployed on the elevator edge gateway generates contextualized reassuring voice text based on the image recognition results of the people in the car to provide reassuring prompts to the people in the car; at the same time, it receives voice input from the car intercom, completes intention recognition and automatic response, and dynamically interacts with the people in the car; S523, Large Model Node: Generates the overall troubleshooting strategy; S524, Large Model Node: Generates rescue work orders and selects suitable technical personnel; S525, Plug-in Node: The system sends troubleshooting operation instructions to technicians' mobile smart terminals; S526, Execution Node: Technicians perform obstacle removal and rescue operations according to the obstacle removal operation guidelines; S527, Data Node: The system automatically confirms the fault handling result based on the received elevator status data; S528. Select Node: If the fault is cleared, execution ends; if the fault is not cleared, a technical expert will provide remote guidance. S529, End Node.
10. The intelligent elevator fault handling method based on a multimodal large model and probabilistic causal graph according to any one of claims 2 to 9, characterized in that, The method further includes: S7. Collect full-process data for each execution and build a labeled fault sample library; and use an incremental fine-tuning algorithm to regularly update the model online.
Citation Information
Patent Citations
Elevator full-period operation and maintenance decision-making method based on cooperation of large model and knowledge graph
CN122066144A
Elevator fault diagnosis and processing system and method
CN122241051A
