Elevator detection report generation method based on multi-modal data fusion and semantic analysis

Abstract

This invention discloses a method for generating elevator inspection reports based on multimodal data fusion and semantic analysis, comprising: acquiring multi-source monitoring data generated during elevator system operation; inputting a first feature sequence, a second feature sequence, and a third feature sequence into a pre-trained large model; and automatically generating a structured elevator inspection report containing fault type, fault location, fault cause, prediction confidence, and maintenance recommendations. This invention fuses three types of heterogeneous multi-source monitoring data generated during elevator operation: sensor time-series data, monitoring image data, and operation and maintenance text logs. These data are then preprocessed according to modality specificities to transform them into structured first, second, and third feature sequences, which are input into a pre-trained large model. A cross-modal attention mechanism is used to achieve deep feature alignment and fusion, generating a unified multi-dimensional health embedding vector representing the elevator's health status. Finally, a complete structured elevator inspection report is automatically generated.

Description

Technical Field

[0001] This invention belongs to the field of intelligent detection and artificial intelligence technology, specifically relating to a method for generating elevator inspection reports based on multimodal data fusion and semantic analysis. Background Technology

[0002] With the rapid development of high-rise buildings in cities, the operational safety and intelligent maintenance of elevators, as core vertical transportation equipment, are becoming increasingly urgent. Traditional elevator inspections rely on periodic manual inspections and alarm mechanisms based on fixed thresholds, making it difficult to detect early-stage hidden faults. Furthermore, the diagnostic process is highly subjective and suffers from delayed responses. In recent years, the Internet of Things (IoT) and edge intelligence technologies have driven the real-time collection of multi-source elevator data, providing a foundation for building data-driven intelligent diagnostic systems. However, effectively integrating heterogeneous information and generating actionable decision outputs remains a pain point for the industry.

[0003] Multimodal perception has become an important direction for improving elevator status monitoring capabilities. Existing research attempts to combine video, sensor, and time-series operational data for anomaly identification. For example, Chinese invention patent CN120951051B, "A Real-time Detection Method and System for Elevator Abnormal Behavior Based on Deep Learning," proposes a real-time detection framework that integrates video streams, sensor data, and operational status sequences: it analyzes passenger behavior through a passenger behavior recognition model, predicts vibration patterns for future cycles using a vibration prediction model, and triggers protective measures using a rule base. This solution improves safety in scenarios where abnormal passenger behavior and mechanical vibration are coupled, demonstrating the value of multimodal collaboration. However, it is still limited to the dual-modal correlation of behavior and vibration, and relies on a pre-set rule base for decision-making. It lacks the ability to mine semantic knowledge from historical maintenance texts (such as maintenance records and expert logs), making it difficult to support in-depth fault attribution and the generation of professional reports.

[0004] The development of large-scale model technology has provided a new path to overcome the aforementioned limitations. Architectures represented by Transformer possess powerful cross-modal alignment and semantic reasoning capabilities, enabling the unified encoding of temporal signals, images, and text into token sequences, achieving end-to-end feature fusion and causal inference. Compared to the separate behavior recognition, vibration prediction, and rule matching processes in CN120951051B, large-scale models can dynamically learn complex intermodal dependencies without requiring manual rule design, and naturally support natural language generation, creating conditions for automated output of structured diagnostic conclusions. However, a systematic solution is still lacking for constructing a multimodal fusion and report generation mechanism that considers both physical features and semantic knowledge, tailored to the operating mechanism and maintenance logic of elevator systems.

[0005] Current intelligent elevator diagnostic technology still faces challenges such as the disconnect between perception and semantics, and the lack of integration between detection and generation. On the one hand, most systems (including CN120951051B) fail to effectively utilize expert experience from unstructured text logs, resulting in insufficient fault interpretation. On the other hand, diagnostic results are mostly presented in the form of alarms or protection commands, failing to automatically generate standardized, readable, structured inspection reports that include fault type, location, cause, confidence level, and maintenance recommendations, making seamless integration into existing maintenance workflows difficult. Therefore, a new method is urgently needed that deeply integrates multimodal data from sensors, images, and text logs, achieves semantic-level correlation analysis through large models, and automatically outputs professional-grade elevator inspection reports. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide an elevator inspection report generation method based on multimodal data fusion and semantic analysis, so as to solve the problem that the existing system is unable to deeply integrate heterogeneous data such as sensors, images and text logs, and cannot automatically generate structured inspection reports with complete content, clear cause and effect, and strong interpretability.

[0007] The objective of this invention is achieved through the following technical solution: A method for generating elevator inspection reports based on multimodal data fusion and semantic analysis, comprising: Acquire multi-source monitoring data generated during the operation of the elevator system. The multi-source monitoring data includes at least time-series data collected by sensors, image data collected by monitoring equipment, and text log data from operation and maintenance records. The multi-source monitoring data is subjected to modality-specific preprocessing to generate a structured first feature sequence, a second feature sequence, and a third feature sequence, respectively. The first feature sequence, the second feature sequence, and the third feature sequence are input into a pre-trained large model, and feature fusion is performed through a cross-modal attention mechanism to generate a multi-dimensional health embedding vector for uniformly representing the health status of the elevator. Based on the multidimensional health embedding vector, state anomaly detection and fault evolution prediction are performed, and key fault features are extracted. By performing correlation analysis between the key fault features and the semantic information in the text log data, a structured elevator inspection report is automatically generated, which includes fault type, fault location, fault cause, prediction confidence level, and maintenance suggestions.

[0008] As a preferred embodiment, the mode-specific preprocessing includes: Wavelet denoising and segmented aggregation approximate dimensionality reduction are performed on the time-series data, which are then encoded into time-series tokens with time-position codes to form the first feature sequence; The image data is subjected to contrast enhancement and key component region cropping. Spatial features are extracted by a lightweight visual Transformer and encoded into a visual token with two-dimensional spatial location encoding to form the second feature sequence. Domain segmentation and semantic embedding are performed on the text log data to generate a semantic token, forming the third feature sequence.

[0009] As a preferred approach, the cross-modal attention mechanism dynamically focuses on highly relevant regions by calculating the mutual information weights between temporal features and visual features; specifically, a query matrix is ​​set. The key matrix is ​​obtained from the time-series token mapping. AND-value matrix Obtained from the visual token mapping, the cross-attention output is: Attention softmax

[0010] in, This represents the number of time-series tokens. For the number of visual tokens, and These are the dimensions of the key vector and the value vector, respectively; this output is used for subsequent multimodal fusion.

[0011] As a preferred method, the state anomaly detection adopts the reconstruction error method: during the training phase, the autoencoder is trained using healthy embedding vectors from historical normal states; during the detection phase, the current healthy embedding vector is input into the autoencoder, and the Euclidean distance between the original vector and the reconstructed vector is calculated as an anomaly score; when the anomaly score exceeds a preset threshold, it is determined to be an abnormal state.

[0012] As a preferred approach, the fault evolution prediction includes: inputting a historical health embedding vector sequence into a large model encoder, generating a predicted health embedding sequence for the next H steps through an autoregressive decoder; and based on the predicted sequence, identifying trajectories in the embedding space that approach a preset fault prototype vector to determine the predicted fault occurrence time.

[0013] As a preferred method, the remaining useful life is estimated using the following formula:

[0014] in, Indicates the remaining useful life, in hours (h); For predicting the timing of a fault occurrence, use an offset long index; Provide a long index for the current step distance; The actual time interval corresponding to each step size, in hours (h); It is determined by minimizing the cosine distance between the predicted healthy embedding vector and the fault prototype vector.

[0015] As a preferred approach, prediction uncertainty is quantified using Monte Carlo Dropout: the Dropout mechanism is enabled multiple times during the prediction phase to generate... Group RUL estimates Calculate its standard deviation As an indicator of uncertainty; when When the preset bandwidth is exceeded, a high uncertainty warning is marked in the elevator inspection report.

[0016] As a preferred approach, association analysis includes: retrieving relevant maintenance records and expert procedures from text logs using fault type and location as keywords; constructing a two-layer causal network based on complex network theory, with the upper layer consisting of semantic causal nodes extracted from the text and the lower layer consisting of physical causal nodes from historical fault cases; and identifying the main causal chain by calculating node betweenness centrality and path reachability.

[0017] As a preferred embodiment, the multi-source monitoring data includes the following: the time-series data includes vibration acceleration (m / s²), traction machine current (A), wire rope tension (kN), and operating speed (m / s); the image data includes visible light or infrared images of the car door area, traction sheave, guide rail, and top of the hoistway; and the text log data includes electronic work orders, inspection records, and shift handover notes.

[0018] As a preferred approach, an online model update mechanism is also included: deploying and running a feedback module to collect actual maintenance results and real fault labels in real time; triggering an incremental learning process when the daily average prediction error or RUL deviation exceeds the threshold; adding new samples from the last 7 days to the training set and using an elastic weight solidification (EWC) strategy to fine-tune the large model parameters to prevent catastrophic forgetting.

[0019] The present invention has at least the following beneficial effects: This solution integrates three types of heterogeneous multi-source monitoring data generated during elevator operation: sensor time-series data, monitoring image data, and maintenance text logs. These data are preprocessed modally to transform them into structured first, second, and third feature sequences, which are then input into a pre-trained large model. A cross-modal attention mechanism is used to achieve deep feature alignment and fusion, generating a unified multi-dimensional health embedding vector representing the elevator's health status. Based on this, the system simultaneously performs anomaly detection and fault evolution prediction, accurately extracting key fault features. These features are then correlated with semantic information in the text logs, ultimately automatically generating a complete structured elevator inspection report. This report covers fault type, fault location, fault cause, prediction confidence level, and maintenance recommendations, significantly improving the accuracy, interpretability, and automation level of elevator fault diagnosis and report generation. Attached Figure Description

[0020] To reveal the technical details of the embodiments of the present invention, the accompanying drawings involved in the embodiments will be briefly described below. It should be emphasized that these drawings only present several embodiments of the present invention and should not be considered as defining the scope of the invention. For those skilled in the art, other related drawings can still be derived based on these drawings without inventive effort.

[0021] Figure 1 A flowchart illustrating a method for generating elevator inspection reports based on multimodal data fusion and semantic analysis; Figure 2 A schematic diagram of a modality-specific preprocessing flow. Detailed Implementation

[0022] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the following description.

[0023] It should be noted that the specific details provided below are intended to help readers gain a more comprehensive understanding of the example embodiments, but are not all the conditions necessary to implement these embodiments. Those skilled in the art should understand that the example embodiments can still be effectively implemented even if these specific details are omitted. For example, the system may be illustrated using a block diagram to avoid obscuring the clear expression of the overall structure due to excessive implementation details; in other cases, to highlight the core idea and maintain the simplicity and clarity of the example, some well-known common-sense processes, general structures, or mature technologies may be intentionally omitted, as they have no substantial impact on understanding the key innovative points of this solution.

[0024] like Figure 1 As shown, a method for generating elevator inspection reports based on multimodal data fusion and semantic analysis includes: Acquire multi-source monitoring data generated during the operation of the elevator system. The multi-source monitoring data includes at least time-series data collected by sensors, image data collected by monitoring equipment, and text log data from operation and maintenance records. The multi-source monitoring data is subjected to modality-specific preprocessing to generate a structured first feature sequence, a second feature sequence, and a third feature sequence, respectively. The first feature sequence, the second feature sequence, and the third feature sequence are input into a pre-trained large model (such as a multimodal Transformer), and feature fusion is performed through a cross-modal attention mechanism to generate a multidimensional health embedding vector for uniformly representing the health status of the elevator. Based on the multidimensional health embedding vector, state anomaly detection and fault evolution prediction are performed, and key fault features are extracted. By performing correlation analysis between the key fault features and the semantic information in the text log data, a structured elevator inspection report is automatically generated, which includes fault type, fault location, fault cause, prediction confidence level, and maintenance suggestions.

[0025] This invention first simultaneously collects three types of multi-source heterogeneous monitoring data generated during elevator operation: sensor time-series data, monitoring image data, and maintenance text logs. Then, it preprocesses each modal characteristic, converting the time-series signal into a first feature sequence (time-series token) with time-location encoding, the key component images into a second feature sequence (visual token) with spatial-location encoding, and the unstructured text logs into a third feature sequence (semantic token) in semantic embedding form. Next, the three feature sequences are input into a pre-trained large model, which uses a cross-modal attention mechanism to dynamically align and fuse the correlation information between different modalities, generating a multi-dimensional health embedding vector that can uniformly and compactly represent the overall health status of the elevator. Based on this embedding vector, the system simultaneously performs anomaly detection and fault evolution trend prediction, accurately extracting key fault features. Finally, it performs deep correlation analysis with semantic information such as historical maintenance records and expert experience in the text logs, driving the large model to autoregressively generate a structured elevator inspection report.

[0026] By deeply integrating physical perception data with semantic knowledge, the accuracy and interpretability of fault diagnosis are significantly improved. The generated reports not only include the fault type, location, and cause, but also provide prediction confidence and targeted maintenance suggestions. The content is complete and logically consistent, effectively replacing manual interpretation and greatly improving the automation, intelligence, and decision reliability of elevator operation and maintenance.

[0027] In a preferred embodiment, to improve the adaptability of multimodal fusion to different operating conditions, this invention introduces a dynamic weight allocation strategy based on operating state awareness. Let the... Elevator in operation status at all times Start, maintain constant speed, brake, stop The state is determined by clustering the temporal features of velocity and acceleration. For different states, the three modalities (temporal, visual, and textual) contribute differently to the health assessment.

[0028] Define the fusion weight vector:

[0029] It is in running state. One-hot encoded vector; It is a learnable state modality importance mapping matrix (trained end-to-end using historical data); , , They represent temporal, visual, and textual modalities at time points. Normalized weights. Final multidimensional health embedding vector. Generated by weighted fusion:

[0030] These are the feature vectors of the three modes after being encoded by the large model; The numerical distribution reflects the elevator's health status. During the start / brake phase, vibration and sudden changes in timing current are more critical, requiring automatic system lifting. During the docking phase, the gate area images and text logs are more relevant. and Automatic enhancement. This example significantly improves the sensitivity of anomaly detection under non-steady-state conditions.

[0031] Specifically, the temporal modality (seq) originates from multi-source sensor signals during elevator operation, including continuous time series such as traction machine current, voltage, car speed, acceleration, and vibration, which can accurately capture transient anomalies in the dynamic process; the visual modality (vis) uses images or videos collected by cameras installed in the car door area to identify whether the door opening and closing is stuck, whether the guide rail is misaligned, whether there are foreign objects obstructing or mechanical structural deformation; and the text modality (txt) integrates maintenance records, fault alarm logs, and operating events (such as structured or semi-structured text such as "overload", "door lock abnormality", and "leveling deviation") to provide high-level semantic context support.

[0032] For example, during the startup phase, when an elevator starts moving upwards, the timing mode detects a sudden increase in current to 1.8 times the rated value, accompanied by high-frequency vibrations above 50Hz. The system recognizes this as a startup state and automatically... The weighting was increased to 0.75, while the weights for visual (normal gate area image) and text (no relevant alarms) were suppressed. The fusion result accurately predicted "early wear of motor bearings," avoiding over-reliance on irrelevant modalities.

[0033] During the docking phase, when another elevator stopped on the 10th floor, the visual modality detected a persistent obstruction at the lower right corner of the car door, while the text modality recorded a door lock circuit disconnection event. The system determined this to be a "docked" state and dynamically allocated... , The fusion analysis confirmed that the fault was caused by dust accumulation in the door slot, which obstructed the door from closing. At this time, the timing signal was stable, and its weight was automatically reduced, effectively preventing misjudgment as an electrical or drive system fault.

[0034] In a preferred embodiment, see Figure 2 The modality-specific preprocessing includes: Wavelet denoising and segmented aggregation approximate dimensionality reduction are performed on the time-series data, which are then encoded into time-series tokens with time-position codes to form the first feature sequence; The image data is subjected to contrast enhancement and key component region cropping. Spatial features are extracted by a lightweight visual Transformer and encoded into a visual token with two-dimensional spatial location encoding to form the second feature sequence. Domain segmentation and semantic embedding are performed on the text log data to generate a semantic token, forming the third feature sequence.

[0035] To ensure that raw data from different sources can be effectively processed by a unified model, the system first performs targeted preprocessing on data from each modality, transforming it into a feature sequence with clear structure, condensed information, and location semantics. For time-series data acquired by sensors (such as current and vibration), which is susceptible to noise interference and has a large data volume, the system first uses wavelet denoising technology to filter out high-frequency interference, then compresses redundant information using a segmented aggregation approximation method, reducing computational complexity while retaining key dynamic features, and finally adds a time-location code to each time segment to form an ordered sequence of "time-series tokens". For image data, the system first performs contrast enhancement to highlight details and automatically crops key component areas such as the car door and traction wheel to avoid irrelevant background interference; then, it uses a lightweight visual Transformer model to extract the spatial structural features of these areas and assigns a two-dimensional spatial location code to each image block to generate a sequence of "visual tokens", thereby preserving the relative positional relationships of components in the image. For unstructured text logs (such as maintenance records and alarm information), the system uses a professional dictionary in the elevator field for word segmentation to ensure that terms such as "door lock malfunction" and "leveling deviation" are accurately identified. Then, a semantic embedding model is used to convert the words into vectors that reflect their meanings, forming a sequence of "semantic tokens." These three types of preprocessed feature sequences not only meet the input requirements of subsequent large-scale models, but also retain the core information and structural characteristics of the original modality, laying the foundation for high-quality multimodal fusion.

[0036] In a preferred embodiment, the cross-modal attention mechanism dynamically focuses on highly relevant regions by calculating the mutual information weights between temporal features and visual features; specifically, a query matrix is ​​set. The key matrix is ​​obtained from the time-series token mapping. AND-value matrix Obtained from the visual token mapping, the cross-attention output is: Attention softmax

[0037] in, This represents the number of time-series tokens. For the number of visual tokens, and These are the dimensions of the key vector and the value vector, respectively; this output is used for subsequent multimodal fusion.

[0038] The core objective of cross-modal attention mechanisms is to enable data from different sources to "understand each other," thereby achieving more accurate fusion. In this system, time-series data (such as current and vibration) and visual data (such as gate area images) are encoded into a series of feature units (i.e., tokens). To determine which visual regions are most relevant to time-series anomalies, the system uses time-series features as a "query" to "ask" which parts of the visual features are most strongly associated with them. Specifically, the model dynamically calculates the correlation score between the time-series signal and each local region of the image. The higher the score, the more likely the image region is to be related to the current operating state or potential fault. For example, when abnormal vibration is detected, the model may automatically focus on the image features of the traction wheel or guide rail region, ignoring irrelevant background. This correlation-based weighted selection process allows the fused representation to highlight truly diagnostically valuable information, suppress noise and redundancy, thereby improving the accuracy and robustness of health status assessment. Finally, this intelligently aligned cross-modal information is integrated into a unified health embedding vector, providing high-quality input for subsequent anomaly detection and fault prediction.

[0039] In a preferred embodiment, the state anomaly detection employs the reconstruction error method: during the training phase, an autoencoder is trained using healthy embedding vectors from historical normal states; during the detection phase, the current healthy embedding vector is input into the autoencoder, and the Euclidean distance between the original vector and the reconstructed vector is calculated as an anomaly score; when the anomaly score exceeds a preset threshold, it is determined to be an abnormal state.

[0040] The autoencoder (e.g., a multilayer perceptron autoencoder, MLP autoencoder, existing technology) is a fully connected autoencoder, comprising an encoder and a decoder connected sequentially. The encoder consists of an input layer, a first hidden layer, and a latent representation layer, while the decoder consists of a second hidden layer and an output layer. The input and output layers have the same dimension, equal to the dimension of the healthy embedding vector. The first and second hidden layers have the same number of neurons, and the latent representation layer has a smaller dimension than the input layer. Adjacent layers are fully connected. The hidden layers in the encoder use the ReLU activation function, and the output layer uses linear activation. During training, historically collected healthy embedding vectors confirmed to be in a normal state are input into the autoencoder, and unsupervised training is performed by minimizing the mean square error between the input vector and the reconstructed output. During detection, the current healthy embedding vector is input into the trained autoencoder, and the Euclidean distance between it and the reconstructed vector is calculated as an anomaly score. When this score exceeds a preset threshold, the system is determined to be in an abnormal state.

[0041] In a preferred embodiment, the typical fault evolution of an elevator system follows a three-stage pattern: latent degradation, feature manifestation, and functional failure. In the initial latent degradation stage, slight degradation has already occurred within the system, but its representation in the 512-dimensional healthy embedding space changes extremely slowly, making it difficult for traditional static threshold-based reconstruction error methods to respond in a timely manner, resulting in a delayed early warning. To address this problem, this invention proposes introducing the healthy embedding evolution rate as a highly sensitive criterion for early dynamic anomalies.

[0042] Specifically, let the current sampling time be... The time interval between two adjacent samples is (Unit: hours, h), the corresponding health embedding vectors are respectively and The healthy embedding itself is obtained by mapping normalized multimodal features through an encoder. Based on this, the healthy embedding evolution velocity vector is defined as:

[0043] The vector The physical unit is This reflects the direction and intensity of the displacement of the health status within the embedded space per unit time. To further obtain quantifiable and comparable scalar indices, the system calculates the L2 norm of this velocity vector, i.e., the evolution rate (ER):

[0044] in Representing vectors The Components. Scalar. Also has units Its numerical value directly represents the severity of changes in current health status. Even If the absolute distance from the historical normal embedding does not exceed the limit, then ER A significant sudden increase (e.g., a sudden change in the vibration spectrum structure caused by microcracks in the bearing) can be regarded as a sensitive signal of early degradation.

[0045] Based on this, the present invention designs two collaborative application modes: First, set a low early warning threshold. .when When this occurs, the system enters a "potential risk" state, automatically triggering a high-frequency monitoring strategy (such as adjusting the sampling interval). The time complexity is reduced to 1 / 5 of the original, and the number of Monte Carlo Dropout inferences is increased to assess uncertainty, but formal fault reports are not generated yet to avoid false alarms. Secondly, the evolution rate and traditional reconstruction error are weighted and fused to form a better comprehensive anomaly score. Let... This represents the reconstruction error at the current moment (i.e., the Euclidean distance between the input and output of the autoencoder). The average evolution rate (in units) is statistically obtained from the system's historical steady-state operation. If the anomaly score is 1, then the final anomaly score is:

[0046] in The configurable balancing coefficients enable the system to focus not only on "deviations from normal states" but also on "rapid deterioration of states," thus enhancing its sensitivity to sudden or accelerated failures. In summary, this solution is fully compatible with existing architectures, requiring only the addition of a lightweight difference and norm calculation module after the multimodal data encoding layer output, without modifying the core model, to significantly improve the detection capability and early warning timeliness of early-stage latent failures.

[0047] In a preferred embodiment, the fault evolution prediction includes: inputting a historical health embedding vector sequence into a large model encoder, generating a predicted health embedding sequence for the next H steps through an autoregressive decoder; and based on the predicted sequence, identifying trajectories in the embedding space that approach a preset fault prototype vector to determine the predicted fault occurrence time.

[0048] The core idea behind the fault evolution prediction is to infer future trends from past states. Specifically, the system first collects elevator health status data over a period of time and represents this data as a series of health embedding vectors (which can be understood as high-dimensional digital fingerprints describing the elevator's current health status). Next, these historical vector sequences are fed into a large model encoder. For example, it can be imagined as the encoder part of a Transformer model, whose role is to deeply understand the dynamic changes and potential patterns contained in this entire historical sequence, compressing and refining it into an internal representation rich in contextual information. Subsequently, this internal representation is passed to an autoregressive decoder (such as the decoder in a Transformer model). This decoder works by predicting step by step: it first generates a health status prediction for the first step (i.e., the next time step) based on the encoder's information; then, it uses the just-predicted result as known information and combines it with the previous context to predict the second step; this process is repeated until the health embedding sequence prediction for the next H time steps is completed. Finally, the system observes the trajectory of this path formed by the predicted points in the embedding space. If it clearly moves toward a pre-defined fault prototype vector (representing a typical state of a specific fault), it indicates that the elevator is likely heading toward that fault, and based on this, the specific time point when the fault is expected to occur can be inferred.

[0049] In a preferred embodiment, to ensure that the fault evolution prediction conforms to the physical laws of the elevator mechanical system, the present invention uses a preset fault prototype vector. (corresponding to the first) For faults such as "traction wheel wear" and "gantry crane jamming", energy conservation constraints are introduced for calibration.

[0050] Considering the dynamic equations of the traction system, under normal conditions, the input electrical energy... With output mechanical energy satisfy:

[0051] in For system efficiency. When a specific fault occurs (such as wire rope slippage), efficiency... The decline is manifested as an increase in current and a decrease in velocity. Therefore, a fault prototype is defined. The corresponding energy efficiency deviation threshold should be met. .

[0052] During the calibration phase, for each faulty prototype Calculate the corresponding synthetic time-series feature inversion value. Extract the average current from it. (Unit: A) and average speed (Unit: m / s). Assume the theoretical speed under rated load is... (m / s), rated current is (A), then the calibration conditions are:

[0053] The ratio on the left reflects the "multiple of input current required per unit output speed"; This represents the minimum acceptable level of energy efficiency degradation for this type of failure (set by experts or historical data). If this level is not met, fine-tuning can be performed using gradient descent. This process continues until physical consistency is achieved. This approach avoids generating "false fault trajectories" in large models that do not conform to physical laws, thus improving the reliability and engineering applicability of RUL predictions.

[0054] The above calibration mechanism ensures that each faulty prototype... The location within the embedded space not only possesses semantic distinctiveness but also clear physical interpretability. For example, regarding faults like "traction sheave wear," the corresponding... It is usually set to a small positive value (such as 0.15) to reflect the characteristic of slow efficiency decay; while "severe wire rope slippage" corresponds to a larger value. (e.g., 0.4) to match its rapid energy efficiency degradation. This differentiated threshold design allows different fault prototypes to form an identifiable distribution structure in the energy efficiency dimension, providing a reliable benchmark for subsequent trajectory matching.

[0055] In practical deployments, the calibration process can be performed later in model training or during fine-tuning. The system iterates through all preset fault categories. Verify and correct in sequence It is worth noting that the decoder (Decoder(·)) needs to have good timing reconstruction capabilities to ensure that... Inverse It can accurately reflect the current and velocity characteristics under steady-state operating conditions. To this end, this invention introduces auxiliary reconstruction loss during the decoder training stage, forcing it to retain the statistical characteristics of key physical quantities (such as power and velocity), thereby improving the reliability of calibration.

[0056] The physically constrained fault prototype library significantly enhances the stability of fault evolution prediction. This is particularly evident when large models generate future healthy embedding sequences. Then, the system calculates its destinations. The dynamic distance. Due to all All meet Any approaching The trajectory inevitably corresponds to the actual energy efficiency degradation pattern, effectively eliminating non-physical explanations caused by model overfitting or data noise. This makes the predicted failure time and remaining useful life (RUL) closer to engineering reality, providing a high-confidence basis for maintenance decisions.

[0057] In a preferred embodiment, the remaining useful lifetime is estimated using the following formula:

[0058] in, Indicates the remaining useful life, in hours (h); For predicting the timing of a fault occurrence, use an offset long index; Provide a long index for the current step distance; The actual time interval corresponding to each step size, in hours (h); It is determined by minimizing the cosine distance between the predicted healthy embedding vector and the fault prototype vector.

[0059] In a preferred embodiment, prediction uncertainty is quantified using Monte Carlo Dropout: the Dropout mechanism is enabled multiple times during the prediction phase to generate... Group RUL estimates Calculate its standard deviation As an indicator of uncertainty; when When the preset bandwidth is exceeded, a high uncertainty warning is marked in the elevator inspection report.

[0060] In a preferred embodiment, the association analysis includes: retrieving relevant maintenance records and expert procedures from text logs using fault type and location as keywords; constructing a two-layer causal network based on complex network theory, with the upper layer consisting of semantic causal nodes extracted from the text and the lower layer consisting of physical causal nodes from historical fault cases; and identifying the main causal chain by calculating node betweenness centrality and path reachability.

[0061] In a preferred embodiment, although the above-described correlation analysis can effectively uncover causal chains, the large language model may still produce illusory results that contradict the physical laws of elevators or the topological relationships of components when generating diagnostic conclusions (for example, predicting door operator jamming but corresponding to a healthy embedding trajectory in the traction machine area). To ensure the engineering credibility of the diagnostic logic, this invention introduces a lightweight elevator domain knowledge graph (DKG) as an external strong constraint to perform consistency checks on the prediction results.

[0062] The specific implementation method is as follows: First, construct the elevator DKG, which is a directed graph. , where the node set Includes key elevator components (such as traction machines, car doors, frequency converters, PLCs, etc.), side collection The physical or control relationships between encoded components (e.g., traction machine driving wire rope, PLC controlling gantry crane). Secondly, a learnable mapping function is designed. embed any health vector Converted into activation intensity vectors for each component:

[0063] in, , quantity The larger the value, the more likely it is to be the corresponding component. The higher the probability of an anomaly. During the verification phase, let the decoder predict the future fault occurrence time as... The corresponding health embedding is The system selects the top [units] with the highest activation strength. These components constitute the predictive failure component set. Simultaneously, the most likely set of faulty components is extracted from the synchronized text logs through keyword matching and entity recognition. Define the consistency check score (CCS) as the Jaccard similarity coefficient between the two:

[0064] The score ranges from [0, 1]. If the CCS is lower than a preset threshold (e.g., 0.3), the prediction is deemed inconsistent with the observed facts. The system will abandon the prediction and initiate a correction process based on K-nearest neighbor retrieval, selecting the case from the historical fault prototype library that is most similar to the current embedding and satisfies the knowledge graph path constraints as an alternative diagnostic basis. This method uses structured domain knowledge to logically correct the data-driven model, fundamentally suppressing the risk of "illusion" and ensuring that the final fault diagnosis output conforms to both data characteristics and engineering common sense, significantly improving the system's reliability and interpretability.

[0065] In a preferred embodiment, the structured elevator inspection report is output using a standardized template and includes the following fields: equipment number, inspection time, health score (0–100 points), main abnormal modes, fault category, risk level (low / medium / high), RUL predicted value and confidence interval, and maintenance recommendations.

[0066] After completing multimodal data fusion, anomaly detection, and fault prediction, the system will automatically generate a structured elevator inspection report. This report uses a unified standardized template to ensure that the information is clear, complete, and easy for maintenance personnel to understand quickly. The core fields of the report include equipment number and inspection time, used to uniquely identify the inspected elevator and the inspection time; health score (0–100 points) intuitively reflects the current overall operating status, with lower scores indicating higher risks; main abnormal modes indicate whether the abnormal warning was triggered first or most significantly by time-series signals, visual images, or text logs, helping to locate the source of the problem; fault category specifies the possible fault types, such as "door operator jamming" or "traction sheave wear"; risk level (low / medium / high) comprehensively considers the anomaly intensity, evolution speed, and historical similar cases to classify the urgency; RUL (remaining useful life) prediction value and its confidence interval provide a time window for future faults to occur, along with uncertainty assessment, avoiding blind reliance on a single prediction point; finally, the system automatically generates targeted maintenance suggestions based on correlation analysis and domain knowledge, such as "suggest cleaning the door slot and checking the door lock circuit" or "schedule a vibration retest of the traction machine bearing within 7 days." The entire report generation process deeply integrates physical perception and semantic knowledge, not only presenting "what" but also explaining "why" and "how," significantly improving the operability and engineering practicality of the diagnostic results.

[0067] In a preferred embodiment, the multi-source monitoring data includes the time-series data such as vibration acceleration (m / s²), traction machine current (A), wire rope tension (kN), and running speed (m / s); the image data includes visible light or infrared images of the car door area, traction sheave, guide rail, and top of the hoistway; and the text log data includes electronic work orders, inspection records, and shift handover notes.

[0068] In a preferred embodiment, the model also includes an online update mechanism: deploying and running a feedback module to collect actual maintenance results and real fault labels in real time; triggering an incremental learning process when the average daily prediction error or RUL deviation exceeds a threshold; adding new samples from the last 7 days to the training set and using an elastic weight solidification (EWC) strategy to fine-tune the large model parameters to prevent catastrophic forgetting.

[0069] In a preferred embodiment, the large model is deployed on an edge computing node or a cloud server, supporting concurrent processing of multiple elevators; the health embedding vectors of each elevator are generated independently, but share the same set of fusion and generation modules, realizing efficient resource reuse and unified report format.

[0070] This invention proposes an intelligent elevator inspection report generation method based on multimodal data fusion and semantic analysis. By simultaneously collecting sensor time-series data, monitoring images, and maintenance text logs, a unified health embedding representation is constructed. An innovative health embedding evolution rate mechanism is introduced to enhance early sensitivity to latent degradation stages. A fault prototype library calibrated with physical constraints ensures that the predicted trajectory conforms to elevator dynamics. Simultaneously, a dynamic modal weighting strategy based on operational status perception improves the robustness of anomaly detection under unsteady conditions, and domain knowledge graph consistency verification effectively suppresses large model "illusions," ensuring the engineering credibility of diagnostic conclusions. The entire system achieves end-to-end automatic generation from raw heterogeneous data to structured, interpretable, and high-confidence inspection reports, combining technological advancement with engineering practicality. It significantly improves the intelligence level, early warning timeliness, and decision reliability of elevator operation and maintenance, providing a scalable technical paradigm for predictive maintenance of special equipment.

[0071] The above description, by way of preferred embodiments, exemplifies the present invention and should not be construed as limiting the scope of protection of the present invention. Those skilled in the art, upon mastering the core ideas of the present invention, can make various equivalent substitutions, structural adjustments, or technical optimizations to the above embodiments. Any modifications, combinations, simplifications, or derivative schemes made based on the technical principles disclosed in this invention should be considered to fall within the protection scope of this invention. The claims are intended to cover all such variations and equivalent embodiments.

Claims

1. A method for generating elevator inspection reports based on multimodal data fusion and semantic analysis, characterized in that, include: Acquire multi-source monitoring data generated during the operation of the elevator system. The multi-source monitoring data includes at least time-series data collected by sensors, image data collected by monitoring equipment, and text log data from operation and maintenance records. The multi-source monitoring data is subjected to modality-specific preprocessing to generate a structured first feature sequence, a second feature sequence, and a third feature sequence, respectively. The first feature sequence, the second feature sequence, and the third feature sequence are input into a pre-trained large model, and feature fusion is performed through a cross-modal attention mechanism to generate a multi-dimensional health embedding vector for uniformly representing the health status of the elevator. Based on the multidimensional health embedding vector, state anomaly detection and fault evolution prediction are performed, and key fault features are extracted. By performing correlation analysis between the key fault features and the semantic information in the text log data, a structured elevator inspection report is automatically generated, which includes fault type, fault location, fault cause, prediction confidence level, and maintenance suggestions.

2. The elevator inspection report generation method based on multimodal data fusion and semantic analysis according to claim 1, characterized in that, The modality-specific preprocessing includes: Wavelet denoising and segmented aggregation approximate dimensionality reduction are performed on the time-series data, which are then encoded into time-series tokens with time-position codes to form the first feature sequence; The image data is subjected to contrast enhancement and key component region cropping. Spatial features are extracted by a lightweight visual Transformer and encoded into a visual token with two-dimensional spatial location encoding to form the second feature sequence. Domain segmentation and semantic embedding are performed on the text log data to generate a semantic token, forming the third feature sequence.

3. The elevator inspection report generation method based on multimodal data fusion and semantic analysis according to claim 1, characterized in that, The cross-modal attention mechanism dynamically focuses on highly correlated regions by calculating the mutual information weights between temporal and visual features; specifically, a query matrix is ​​set. The key matrix is ​​obtained from the time-series token mapping. AND-value matrix Obtained from the visual token mapping, the cross-attention output is: Attention softmax ; in, This represents the number of time-series tokens. For the number of visual tokens, and These are the dimensions of the key vector and the value vector, respectively; this output is used for subsequent multimodal fusion.

4. The elevator inspection report generation method based on multimodal data fusion and semantic analysis according to claim 1, characterized in that, The state anomaly detection adopts the reconstruction error method: during the training phase, the autoencoder is trained using the healthy embedding vectors of historical normal states; During the detection phase, the current healthy embedding vector is input into the autoencoder, and the Euclidean distance between the original vector and the reconstructed vector is calculated as an anomaly score; when the anomaly score exceeds a preset threshold, it is determined to be an abnormal state.

5. The elevator inspection report generation method based on multimodal data fusion and semantic analysis according to claim 1, characterized in that, The fault evolution prediction includes: inputting the historical health embedding vector sequence into a large model encoder, generating a predicted health embedding sequence for the next H steps through an autoregressive decoder; and based on the predicted sequence, identifying trajectories in the embedding space that approach a preset fault prototype vector to determine the predicted fault occurrence time.

6. A method for generating elevator inspection reports based on multimodal data fusion and semantic analysis according to claim 1 or 5, characterized in that, The remaining useful life is estimated using the following formula: ； in, Indicates the remaining useful life, in hours (h); For predicting the timing of a fault occurrence, use an offset long index; Provide a long index for the current step distance; The actual time interval corresponding to each step size, in hours (h); It is determined by minimizing the cosine distance between the predicted healthy embedding vector and the fault prototype vector.

7. The elevator inspection report generation method based on multimodal data fusion and semantic analysis according to claim 6, characterized in that, Prediction uncertainty is quantified using Monte Carlo Dropout: The Dropout mechanism is enabled multiple times during the prediction phase to generate... Group RUL estimates Calculate its standard deviation As an indicator of uncertainty; when When the preset bandwidth is exceeded, a high uncertainty warning is marked in the elevator inspection report.

8. The elevator inspection report generation method based on multimodal data fusion and semantic analysis according to claim 1, characterized in that, Association analysis includes: retrieving relevant maintenance records and expert procedures from text logs using fault type and location as keywords; constructing a two-layer causal network based on complex network theory, with the upper layer consisting of semantic causal nodes extracted from the text and the lower layer consisting of physical causal nodes from historical fault cases; and identifying the main causal chain by calculating node betweenness centrality and path reachability.

9. The elevator inspection report generation method based on multimodal data fusion and semantic analysis according to claim 1, characterized in that, The multi-source monitoring data includes time-series data such as vibration acceleration, traction machine current, wire rope tension, and running speed; image data including visible light or infrared images of the car door area, traction sheave, guide rail, and top of the hoistway; and text log data including electronic work orders, inspection records, and shift handover notes.

10. The elevator inspection report generation method based on multimodal data fusion and semantic analysis according to claim 1, characterized in that, It also includes an online model update mechanism: deploy and run a feedback module to collect actual maintenance results and real fault labels in real time; when the daily average prediction error or RUL deviation exceeds the threshold, the incremental learning process is triggered; new samples from the last 7 days are added to the training set, and the elastic weight solidification (EWC) strategy is used to fine-tune the large model parameters to prevent catastrophic forgetting.

Images