Elevator door system based on deep vision and hoistway key component inspection system

Abstract

The application relates to the technical field of elevator safety monitoring, and particularly discloses an elevator door system and a hoistway key component inspection system based on deep vision, aiming to solve the problems of low recognition accuracy and weak generalization ability caused by environmental interference, lack of time sequence analysis and decision-making separation from physical mechanisms in the existing visual inspection technology. The system comprises a multi-modal data acquisition and synchronization module, a time sequence feature fusion and physical constraint embedding module, a physical mechanism driven adaptive decision module and a robot motion and data correlation control module. By synchronously acquiring multi-modal time sequence data, fusing time sequence attention and physical prior to extract robust features, and dynamically generating an adaptive decision threshold based on a physical wear model, a quantitative detection report is finally output, so that high-precision, adaptive intelligent inspection and evaluation of the state of elevator key components are realized.

Description

Technical Field

[0001] This invention belongs to the field of elevator safety monitoring technology, specifically relating to an elevator door system and shaft key component inspection system based on depth vision. Background Technology

[0002] As a core special equipment for vertical transportation, the long-term reliability and safety of elevators are of paramount importance. Regular inspections of elevator door systems and key components inside the shaft (such as guide rails, counterweights, and safety brakes) are crucial for preventing accidents and ensuring passenger safety. Traditional manual inspection methods are inefficient, costly, and rely on the subjective experience of maintenance personnel, making it difficult to achieve comprehensive, objective, and standardized testing.

[0003] With technological advancements, automated inspection technology based on machine vision has become an important direction, aiming to automatically identify component anomalies through image analysis. However, existing technologies typically rely on single-frame static images for defect identification, facing significant bottlenecks: First, oil stains, dust, and other deposits inside the wellbore are easily confused with actual wear features in the image, leading to high rates of model misjudgment and missed detections; second, single-frame processing cannot distinguish between instantaneous interference and progressive wear, ignoring the temporal characteristics of defect evolution, resulting in insufficient system stability; third, model decision thresholds often rely on fixed empirical values, failing to incorporate physical parameters such as material properties and stress states, resulting in weak generalization ability and difficulty in adapting to diverse component materials and operating conditions.

[0004] Based on this, there is still room for improvement and optimization of the existing technology, given the aforementioned technical problems. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a depth vision-based elevator door system and shaft key component inspection system, comprising:

[0006] The multimodal data acquisition and synchronization module is used to simultaneously acquire visible light images, infrared thermal imaging images, and 3D point cloud data of the target component, and bind timestamps and spatial pose indexes.

[0007] The robot motion and data association control module is used to drive the robot to move and position, record pose information at detection points, and associate data with the multimodal data acquisition and synchronization module.

[0008] The temporal feature fusion and physical constraint embedding module is used to receive multimodal temporal data streams that have been time-synchronized and pose-associated. It extracts and fuses temporal features and physical prior features through a dual-branch deep neural network that includes a temporal feature extraction branch and a physical prior encoding branch, and outputs a fused feature vector.

[0009] The physical mechanism-driven adaptive decision-making module has a built-in physical wear knowledge base, which is used to identify the component type and parse the physical parameters based on the fused feature vector, calculate the adaptive threshold based on the physical model, and make defect judgment.

[0010] The cloud-based collaborative analysis and model iteration module is used to store reports and data, provide a platform for manual review, and iteratively update the dual-branch deep neural network based on the review results.

[0011] Preferably, the multimodal data acquisition and synchronization module includes a visible light camera, an uncooled infrared thermal imager, and a three-dimensional lidar;

[0012] The multimodal data acquisition and synchronization module is equipped with a high-precision hardware synchronization trigger to generate trigger pulse signals. After synchronization delay calibration, the sent trigger pulses are time-compensated so that the visible light camera, uncooled infrared thermal imager and three-dimensional lidar can start acquiring data simultaneously.

[0013] Preferably, the robot motion and data association control module is integrated on the inspection robot. The inspection robot calculates its trajectory by fusing data from the inertial measurement unit and the photoelectric encoder, and performs periodic pose correction using at least one of the following methods: visual reference point correction, laser reflector positioning, or preset zero point reset.

[0014] The robot motion and data association control module automatically plans an inspection path consisting of a series of detection points containing position coordinates and observation attitudes, based on the imported digital three-dimensional wellbore model.

[0015] Preferably, the temporal feature extraction branch includes:

[0016] The shallow feature extraction and stitching unit is used to organize the preprocessed visible light image and infrared thermal imaging image into a sequence, extract features through a three-dimensional convolutional layer, and stitch them together in the channel dimension to obtain a multimodal spatiotemporal feature tensor.

[0017] A spatiotemporal convolutional backbone network is used to perform convolution processing on the multimodal spatiotemporal feature tensors to compress spatial size and increase feature abstraction ability, and output high-order spatiotemporal feature tensors;

[0018] The temporal attention pooling unit is used to reshape the high-order spatiotemporal feature tensor into a feature matrix, input it into the temporal self-attention layer, and then perform global average pooling through a multi-head attention mechanism to output a temporal feature vector of fixed length.

[0019] Preferably, the physical prior coding branch includes:

[0020] The point cloud segmentation unit is used to segment the target component's point cloud clusters from the coordinate-transformed 3D point cloud data using a region growing algorithm based on Euclidean distance.

[0021] The geometric descriptor calculation unit is used to calculate the surface normal statistics, curvature statistics, size measurement, position measurement and shape ratio of the point cloud cluster to form a low-dimensional geometric feature vector.

[0022] The encoding network, a multilayer perceptron, is used to perform high-level encoding on the low-dimensional geometric feature vector and output the physical prior vector.

[0023] Preferably, the temporal feature fusion and physical constraint embedding module further includes:

[0024] The cross-modal temporal adaptive calibration submodule is used to extract shallow features of visible light image sequences and infrared thermal imaging image sequences through a lightweight convolutional network, estimate the time delay offset through cross-correlation analysis, and resample and align the infrared thermal imaging image sequences accordingly.

[0025] The physical prior-guided temporal attention modulation submodule is used to map the physical prior vector into an attention bias vector through a modulation network, and to introduce the bias into the attention score calculation of the temporal self-attention layer for modulation.

[0026] Preferably, the physical mechanism-driven adaptive decision-making module performs the following steps:

[0027] Part type identification and parameter retrieval: The part type is determined based on the detection point coordinate-part type mapping table and / or the built-in classifier, and the corresponding material property parameters and standard stress model are retrieved from the physical wear knowledge base;

[0028] Feature parameter parsing: The fused feature vector is parsed into estimated values ​​of defect physical parameters and corresponding estimated values ​​of parameter uncertainty through a parsing network;

[0029] Physical model calculation and adaptive threshold generation: Select the corresponding wear evolution equation according to the component type, combine the analyzed physical parameters and their uncertainties to perform forward physical deduction, and introduce a safety factor that is dynamically related to the uncertainty to generate an adaptive decision threshold.

[0030] Decision-making and report generation: The comprehensive evaluation value representing the severity of the defect is compared with the adaptive decision threshold to determine the anomaly, and the defect is located in three dimensions by combining the robot pose to generate a detection report containing the decision basis parameters.

[0031] Preferably, the physical mechanism-driven adaptive decision-making module further includes a cumulative evolutionary inference mechanism based on historical damage archives;

[0032] The cumulative evolution and deduction mechanism establishes and maintains a long-term damage accumulation state file for each component, and the unique identification key of the file is consistent with the spatial pose index key generated by the robot motion and data association control module.

[0033] The initial state parameters used for physical model calculations are weighted fusion values ​​of the instantaneous values ​​from this analysis and the historical cumulative equivalent damage, where the fusion weights are negatively correlated with the uncertainty estimate of the parameters from this analysis.

[0034] Preferably, the system adopts a hierarchical decision-making process, including:

[0035] Rapid screening layer: When the robot approaches the detection point, the network in the temporal feature fusion and physical constraint embedding module is used to quickly process the short sequence multimodal images acquired in real time to obtain a primary alarm score, and determine whether to trigger an alarm based on a preset threshold.

[0036] Fine-grained analysis layer: After the alarm is triggered, the robot is controlled to remain stable at the current detection point, and the complete temporal feature fusion and physical constraint embedding module and the physical mechanism-driven adaptive decision-making module are started to perform a complete analysis of historical multimodal data with a longer time window and generate a detailed detection report.

[0037] Preferably, the cloud-based collaborative analysis and model iteration module performs the following steps:

[0038] Data archiving: Receive and structure the detection reports and corresponding raw fusion feature vectors, and create a joint index for each record containing elevator number, component spatial location index key and detection timestamp;

[0039] Manual review and correction: Authorized engineers review and correct the system's automatic annotation results through a web-based secure interaction platform, generating corrected annotation files;

[0040] Model Iteration Update: The corrected data pairs form an incremental training dataset. When the sample size reaches a preset threshold, the model fine-tuning is automatically initiated, and the validated new model is sent to the front-end device for update.

[0041] In summary, this application includes at least one of the following beneficial technical effects:

[0042] 1. By integrating multimodal data from visible light, infrared thermal imaging, and three-dimensional point clouds, and introducing a cross-modal temporal calibration and physical prior guidance attention modulation mechanism, the system can effectively distinguish between transient optical interference caused by environmental deposits (such as oil and dust) and real progressive physical wear, thereby reducing the false positive rate and false negative rate and enhancing the detection stability in the complex dynamic environment of elevator shafts.

[0043] 2. This invention establishes a knowledge base of component physical wear and uses a physical model combined with real-time analytical defect parameters and their uncertainties to perform forward deduction, dynamically generating adaptive decision thresholds that match the specific component material, operating conditions, and detection confidence level. It abandons empirical fixed thresholds, upgrading maintenance decisions from post-response or periodic prevention to state prediction, making them more scientific and targeted, and improving the personalized management level of safety margin.

[0044] 3. This invention adopts a hierarchical decision-making process that includes rapid screening and fine analysis. It uses a lightweight network to filter a massive number of detection points at high speed, and only initiates computationally intensive full-depth analysis and physical modeling for a few alarm points. Combined with the robot's high-precision autonomous positioning and path planning, it achieves efficient execution of the entire inspection task while ensuring in-depth analysis of complex defects, effectively balancing the dual requirements of detection accuracy and real-time performance in large-scale inspection scenarios. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating the elevator door system and shaft key component inspection system based on depth vision according to the present invention.

[0046] Figure 2 This is a flowchart illustrating the temporal feature fusion and physical constraint embedding module of the present invention;

[0047] Figure 3 This is a flowchart illustrating the adaptive decision-making module driven by the physical mechanism of this invention.

[0048] Figure 4 This is a flowchart illustrating the robot motion and data association control module and the multimodal data acquisition module of the present invention. Detailed Implementation

[0049] This invention provides an elevator door system and shaft key component inspection system based on depth vision, including a multimodal data acquisition and synchronization module, a temporal feature fusion and physical constraint embedding module, a physical mechanism driven adaptive decision-making module, a robot motion and data association control module, and a cloud-based collaborative analysis and model iteration module, thus forming a complete closed loop from data acquisition, feature analysis, intelligent decision-making to continuous optimization.

[0050] To further illustrate the technical means and effects adopted by the present invention in order to achieve the intended purpose, the following detailed description is provided in conjunction with the accompanying drawings and preferred embodiments, based on specific implementation methods of the present invention.

[0051] The elevator door system and shaft key component inspection system based on depth vision specifically includes the following:

[0052] The multimodal data acquisition and synchronization module is the data source of the system. Its purpose is to simultaneously acquire visible light images, infrared thermal imaging images and three-dimensional point cloud data of the target component at the same physical moment, and assign a precise timestamp and spatial pose index to each frame of data.

[0053] The multimodal data acquisition and synchronization module is integrated into a mobile inspection robot or a device fixedly installed in the wellbore. The specific implementation method is as follows:

[0054] S101: The multimodal data acquisition and synchronization module includes:

[0055] A high-resolution industrial visible light camera with a global shutter sensor, a resolution of no less than 20 megapixels, and a frame rate configurable between 1 and 30 frames per second; the camera is equipped with a low-distortion industrial lens and an adjustable brightness ring light to ensure high-contrast images can be acquired even when the shaft lighting is insufficient.

[0056] An uncooled infrared thermal imager, with its operating band strictly limited to the 8-14 micrometer long-wave infrared range, a spatial resolution of no less than 640×480 pixels, and a thermal sensitivity better than 0.05 degrees Celsius, is used to capture localized temperature rises caused by abnormal friction of components, loose electrical connections, etc.

[0057] A three-dimensional lidar, employing a multi-line rotating scan or solid-state array scheme, has a ranging accuracy better than ±2 mm and an angular resolution better than 0.1 degrees, and is used to acquire dense three-dimensional point clouds on the surface of components.

[0058] S102: The multimodal data acquisition and synchronization module is equipped with a high-precision hardware synchronization trigger. The reference clock is provided by a temperature-controlled crystal oscillator. The trigger generates a precise pulse signal according to the acquisition frequency configured by the system (e.g., 5 times per second).

[0059] To achieve strict synchronization of the three sensor acquisition actions, the system performs synchronization delay calibration before leaving the factory or during regular maintenance: the actual delay time after the trigger signal arrives, when the visible light camera starts exposure, the infrared thermal imager starts acquisition, and the lidar starts scanning is measured respectively. Based on these delay measurements, the firmware inside the synchronization trigger performs timing compensation on the trigger pulses sent to each sensor port (for example, adding a programmable delay to the sensor port with a faster response), so that when the three sensors receive the compensated trigger command, their data acquisition actions can start almost simultaneously.

[0060] The calibrated signal is simultaneously transmitted via physical cables to the exposure trigger port of the visible light camera, the acquisition trigger port of the infrared thermal imager, and the scanning trigger port of the lidar.

[0061] The aforementioned synchronization mechanism ensures that the three sensors start acquiring data simultaneously within a microsecond time error, achieving absolute alignment of visible light images, infrared thermal images, and 3D point cloud data in the time dimension.

[0062] S103: While acquiring data, the multimodal data acquisition and synchronization module receives robot spatial pose information from the robot motion and data association control module in real time through a high-speed data interface. This information is represented by a 4×4 homogeneous transformation matrix, which describes the rotation and translation relationship of the robot body coordinate system relative to the predefined world coordinate system in the well.

[0063] S104: The multimodal data acquisition and synchronization module binds each set of synchronously acquired multimodal data frame packets (i.e., a visible light image, an infrared thermal image, and a 3D point cloud) with a unique timestamp and a corresponding spatial pose transformation matrix to form a complete data unit.

[0064] Before transmission, the data is initially encapsulated. The encapsulation format includes a fixed-length header, which records the data unit number, acquisition timestamp, pose matrix data, and byte offset of the original data from each sensor. The encapsulated data units are then transmitted in real time to the subsequent processing module via Gigabit Ethernet or wireless LAN.

[0065] The robot motion and data association control module is responsible for driving the inspection robot to complete autonomous movement and precise positioning, and ensuring that the robot's precise spatial pose is recorded simultaneously when collecting data at specific points. The implementation process is as follows:

[0066] S201: The hardware foundation of the robot motion and data association control module is a mobile inspection robot. The robot uses a wheeled or tracked chassis and can crawl on elevator rails or move smoothly on sills. To achieve accurate self-positioning, the robot integrates the following sensors:

[0067] A high-precision inertial measurement unit, comprising a three-axis gyroscope, a three-axis accelerometer, and a three-axis magnetometer, is used to measure the robot's angular velocity, linear acceleration, and heading angle.

[0068] A high-resolution photoelectric encoder mounted on the drive wheel is used to measure the wheel's rotation angle and travel.

[0069] The data from the aforementioned inertial measurement unit and photoelectric encoder are fused using an extended Kalman filter to enable real-time trajectory calculation of the robot within the elevator shaft.

[0070] To suppress the cumulative error of trajectory estimation and ensure that the absolute positioning accuracy is better than 5 mm during long-term inspections, the system uses one or more of the following methods for periodic pose correction:

[0071] Visual reference point correction: Specific visual markers with known world coordinates (such as AprilTag QR codes) are pre-placed on the inner wall of the shaft. When the visible light camera on the robot recognizes these markers during movement, the current absolute pose is calculated through the PnP algorithm, and the state of the extended Kalman filter is updated.

[0072] Laser reflector positioning: A laser reflector is installed at a fixed position inside the shaft. When the lidar on the robot scans the reflector, it calculates the precise absolute position and angular deviation by matching the feature point cloud of the reflector, which is then used for correction.

[0073] Preset zero-point reset: At the start or end of each inspection task, the robot automatically returns to a physically defined zero-point position in the well (such as the mechanical slot in the pit) to reset the positioning system and eliminate the accumulated error of the task.

[0074] By combining sensor fusion with periodic absolute correction, the robot's real-time positioning accuracy is kept stable at a level better than 5 millimeters throughout the entire inspection path.

[0075] S202: Maintenance personnel first import the elevator's digital three-dimensional shaft model through the host computer software. This model defines the shaft boundaries, the positions of the doors on each floor, and the spatial coordinates of the key components to be inspected (such as guide rails, car, counterweight, wire rope, landing door locks, sills, etc.).

[0076] Based on this 3D model, the robot motion and data association control module uses a path planning algorithm based on A* or fast random exploration tree to automatically plan an inspection path that covers all the parts to be inspected and has a better total travel distance. During planning, the algorithm considers the robot's minimum turning radius and the reachable attitude angle of the gimbal, and calculates one or more observation angles that can completely cover its key surfaces for each part. These observation angles are converted into specific inspection points.

[0077] The aforementioned inspection path consists of a series of discrete detection points. Each detection point not only includes the target position coordinates (X,Y,Z), but also defines the posture required for the robot to obtain the best observation angle when it arrives at the point, such as the pitch angle and yaw angle of the gimbal, as well as the observation distance between the robot body and the components.

[0078] During task execution, the navigation controller drives the robot chassis and gimbal to move sequentially to each preset detection point.

[0079] S203: When the robot determines through its localization algorithm that the error between its current pose and the expected pose of the target detection point is less than a preset threshold, it is considered that the robot has entered a stable state.

[0080] To ensure a precise correspondence between the data acquisition time and the pose recording time, this module performs the following operations:

[0081] First, immediately lock and record the optimal estimated pose after filtering by the extended Kalman filter at the current moment, and generate a pose snapshot (i.e., a 4×4 homogeneous transformation matrix).

[0082] A high-precision timestamp is assigned to the pose snapshot. This timestamp is generated by the system clock inside the robot motion and data association control module. This clock is synchronized with the synchronous trigger clock of the multimodal data acquisition module during system initialization, thereby ensuring the time base of the entire system is unified.

[0083] Subsequently, the data packet containing the aforementioned pose snapshot and its timestamp, along with a digital ready signal, is sent to the multimodal data acquisition and synchronization module via a low-latency, high-determinism shared memory communication interface. In a distributed system architecture, network packets with precise transmission time stamps (such as those based on the PTP protocol) can also be used for transmission, but the network transmission delay needs to be estimated and compensated at the receiving end. The timestamp in the pose snapshot provides the data acquisition module with the key basis for accurately aligning its internal acquisition time with the robot pose recording time.

[0084] S204: The robot motion and data association control module also maintains a spatial pose index database. This database records the pose information (i.e., homogeneous transformation matrix) of all detection points (regardless of whether data is collected) in each inspection task, and associates and stores this pose information with the corresponding data unit number fed back by the multimodal data acquisition and synchronization module.

[0085] Specifically, a globally unique index key is generated for each detection point. The composition of this index key must ensure its uniqueness and readability. For example, it can adopt the format of "elevator number_inspection task ID_part type_coarse position code". The "coarse position code" can be automatically generated according to the floor and relative area (such as upper, middle, lower) of the detection point in the 3D model of the shaft. The index key is stored in both the pose information record and the corresponding data unit packet header. The database establishes and maintains the mapping relationship between pose information and data units through this index key, thereby realizing accurate and efficient historical data retrieval and spatial alignment.

[0086] This database provides a unified and traceable spatial benchmark framework for subsequent 3D reconstruction, accurate comparison of historical data from different periods, and trend analysis of component status.

[0087] The temporal feature fusion and physical constraint embedding module is the core of the system's intelligent analysis, responsible for transforming multimodal time-series data into high-dimensional fused feature vectors that comprehensively characterize the dynamic changes and physical properties of components. Its core is a two-branch deep neural network, and the specific implementation process is as follows:

[0088] S301: The input to the temporal feature fusion and physical constraint embedding module is a multimodal temporal data stream from the multimodal data acquisition and synchronization module, which has already been time-synchronized and pose-associated. First, the image sequence is preprocessed:

[0089] S3011: Denoising, contrast enhancement, and size normalization are performed on the visible light image sequence. Specifically, Gaussian filtering is used for denoising, with the kernel size adaptively set to 5×5 or 7×7 according to the image resolution, and the standard deviation σ is set to 1.5 to suppress noise while preserving edge details; contrast enhancement uses a limited contrast adaptive histogram equalization method to improve the visual effect of unevenly lit areas within the well channel; during size normalization, if the original image is not square, it is first scaled to 512 pixels by the shorter side, and then the longer side is cropped in the center or padded with zero values ​​to ensure that the input size is uniformly 512×512 pixels.

[0090] S3012: For infrared thermal imaging image sequences, the original grayscale values ​​are first converted into absolute temperature values ​​based on the radiation response curve and nonlinear correction coefficient in the thermal imager's factory calibration file; then, the same size normalization process as for visible light images is performed, and bilinear interpolation is used to scale the image to 512×512 pixels to maintain consistent spatial resolution.

[0091] S3013: The preprocessed visible light image sequence and infrared thermal imaging image sequence are organized into four-dimensional tensors with dimensions [B,T,512,512,C]. The batch size B is set to 16 or 32 during the training phase and 1 during the inference phase. The time step T is set according to the motion characteristics of the component. For example, for guide rail wear detection, T can be 10, corresponding to continuous image sampling within approximately 2 seconds. The sampling interval is dynamically adjusted according to the robot's moving speed and image acquisition frame rate to ensure coverage of a complete observation cycle. The number of visible light image channels C=3 (corresponding to R, G, and B channels) and the number of infrared thermal imaging channels C=1 (corresponding to a single channel temperature value).

[0092] S3014: Synchronously receive 3D point cloud data (from LiDAR) and its spatial pose matrix (4×4 homogeneous transformation matrix) that are strictly aligned with the above image sequence in terms of timestamp. The timestamp alignment relies on the hardware synchronization mechanism of S102 mentioned above to ensure that the error between the point cloud and the image acquisition time is in the microsecond range. The spatial alignment is achieved by transforming the point cloud from the radar coordinate system to the camera coordinate system through the pre-calibrated sensor extrinsic parameter matrix, and then unifying it with the pose matrix into the shaft world coordinate system, providing a consistent spatial reference for subsequent multimodal fusion.

[0093] The above-mentioned S301 completes the standardization and tensor organization of multimodal image data, as well as the synchronous preparation of point cloud data, providing a uniform input format for subsequent deep feature extraction.

[0094] S302: The temporal feature extraction branch processes image sequences. This branch consists of three parts: shallow feature extraction, spatiotemporal convolutional backbone network, and temporal attention pooling. Its aim is to extract dynamic features of component appearance and temperature evolution over time. Specifically:

[0095] S3021: Shallow feature extraction and splicing unit.

[0096] The preprocessed visible light image tensor (dimension: [B,T,512,512,3]) and infrared thermal image tensor (dimension: [B,T,512,512,1]) are fed into separate 3D convolutional layers. These two convolutional layers have the same configuration: a kernel size of 3×3×3, a stride of 1×1×1, 16 output channels, and the ReLU activation function. After convolution, the dimensions of both feature maps become [B,T,512,512,16]. Subsequently, these two feature maps are concatenated along the channel dimension (i.e., the 5th dimension) to obtain a multimodal spatiotemporal feature tensor that fuses visible light and infrared information, with dimensions [B,T,512,512,32].

[0097] S3022: Spatiotemporal Convolutional Backbone Network.

[0098] The above multimodal spatiotemporal feature tensors are input into a backbone network containing four three-dimensional convolutional blocks. The structure of each convolutional block is as follows: a three-dimensional convolutional layer, a three-dimensional batch normalization layer, and a ReLU activation function.

[0099] To gradually compress the spatial size and increase the feature abstraction capability, the first two convolutional blocks use a stride of 1×2×2 for the 3D convolutional layers (i.e., downsampling by 2 times in the spatial dimension while keeping the temporal dimension unchanged), while the last two convolutional blocks use a stride of 1×1×1 to maintain the size. The kernel size of all convolutional layers is 3×3×3.

[0100] After processing by this backbone network, the spatial size of the input tensor gradually decreases, while the number of channels increases. Specifically, the dimensionality changes of an input tensor [B,T,512,512,32] after passing through each block are illustrated in the following example:

[0101] After the first convolutional block (stride 1×2×2): [B,T,256,256,64];

[0102] After the second convolutional block (stride 1×2×2): [B,T,128,128,128];

[0103] After the third and fourth convolutional blocks (stride 1×1×1): [B,T,128,128,256].

[0104] Finally, the backbone network outputs a high-order spatiotemporal feature tensor, denoted as F. high Its dimensions are [B,T,H',W',C'], where H'=128, W'=128, C'=256, and the time dimension T remains unchanged.

[0105] S3023: Temporal attention pooling unit.

[0106] This step aims to extract from the high-order spatiotemporal feature tensor F high In this process, a fixed-length feature vector is calculated that can represent the evolution of the component's state within the entire time window.

[0107] First, to compute attention in the temporal dimension, the spatial and channel features at each time step t are fused. Specifically, this involves fusing the high-order spatiotemporal feature tensor F... high (shape [B,T,H',W',C']) is reshaped into a three-dimensional feature matrix. Where D = H' × W' × C'. Thus, each time step t in X corresponds to a D-dimensional eigenvector, which represents the overall state of the component at that moment.

[0108] Next, the feature matrix X is input into the temporal self-attention layer. This layer first maps the input feature X into a query matrix Q, a key matrix K, and a value matrix V through three independent linear transformations (i.e., fully connected layers):

[0109] ;

[0110] in, It is the weight matrix learned through the network training process. k and d v Define the projection dimensions for the query / key vector and value vector respectively. In this embodiment, d is typically set to... k =d v .

[0111] Subsequently, the scaled dot product attention is calculated, and the formula for calculating the attention output matrix Z is as follows:

[0112] .

[0113] In the formula: QK T Calculate the dot product similarity between the query vector and all key vectors to obtain an attention score matrix of shape [B,T,T], which reflects the correlation strength between features at different time steps.

[0114] It is a scaling factor used to adjust the size of the dot product result, preventing it from entering the gradient saturation region of the softmax function, which helps stabilize training.

[0115] The softmax function normalizes along the last dimension (i.e., the dimension of the key), converting the attention scores into a probability distribution to obtain the attention weights.

[0116] Finally, the attention weight matrix is ​​multiplied by the value matrix V to achieve weighted aggregation of features in the time dimension, outputting the enhanced feature matrix. .

[0117] To enhance the model's representational power, this embodiment employs a multi-head attention mechanism. Specifically, the linear transformation and attention calculation processes described above are executed in parallel h times (i.e., h heads, for example, h=4). For the i-th head, a different weight matrix is ​​used. Generate independent Q i K i V i And calculate the output head Z i In this embodiment, the dimension d of each head is typically set. k =d v =D / h (For example, when D=256, h=4, d k =d v =64).

[0118] Output Z of all heads i (i=1,…,h) are concatenated along the feature dimension to obtain a shape of [B,T,h·d]. v The matrix is ​​then passed through a learnable linear projection layer. Mapped to the target dimension:

[0119] ;

[0120] in, This is the final output of the multi-head attention layer. Typically, D'=D can be set to maintain the dimension, or set to other values ​​(e.g., 256) depending on the needs of subsequent processing.

[0121] Finally, in order to obtain a fixed-length vector representation from the temporal feature matrix, Z is plotted along the time dimension T. multi Global average pooling is performed, which averages the features across all time steps. This operation outputs a feature matrix of dimension [B, D']. During the inference phase, the batch size B is typically 1, so this branch ultimately outputs a D'-dimensional temporal feature vector, which encapsulates the key dynamic information weighted by attention within the time window.

[0122] S302 uses a three-dimensional convolutional network and a self-attention mechanism to achieve focused extraction of progressive wear dynamic features in multimodal image sequences, thereby improving the discriminative power of features in the temporal dimension.

[0123] S303: The Physical Prior Encoding branch processes point cloud data. This branch aims to encode the physical structural properties of components from their 3D geometry. Specifically:

[0124] S3031: Based on the spatial pose matrix (4×4 homogeneous transformation matrix) provided by S3014, the raw point cloud data collected by the lidar is transformed from the robot carrier coordinate system to a unified wellbore world coordinate system to ensure that all data have a common spatial reference benchmark.

[0125] S3032: For the world coordinate system point cloud within the field of view of the current detection point, it is necessary to separate the point cloud of the target component from the background. This embodiment uses a region growing algorithm based on Euclidean distance to complete this segmentation. In specific operation, a seed point can be manually specified or automatically determined according to the prior position of the component. The algorithm will iteratively group neighboring points that are less than a set threshold (e.g., 5 mm) from the current point cloud cluster into the same cluster. In this way, the complete point cloud of the target component to be detected (such as a section of guide rail or a door lock roller) can be effectively extracted to form an independent point cloud cluster.

[0126] S3033: For the segmented component point cloud clusters, calculate a set of low-level geometric descriptors to quantify their surface morphology and structural properties. Specifically, extract the following features:

[0127] Surface normal statistics: Based on principal component analysis, the surface normal vectors of each point in the point cloud are calculated, and the consistency of the directions of these normal vectors is statistically analyzed. For example, the variance of the normal vectors is calculated to reflect the overall smoothness or regularity of the surface.

[0128] Curvature statistics: By analyzing the eigenvalues ​​of the local neighborhood covariance matrix of each point, the curvature of that point is estimated, and the mean and standard deviation of the curvature of the entire point cloud cluster are statistically analyzed. The curvature distribution can represent the surface roughness or local unevenness.

[0129] Dimensional measurement: Calculate the smallest circumscribed axis aligned bounding box (AABB) of the point cloud cluster to obtain its length, width, and height dimensions, describing the macroscopic volume of the component.

[0130] Position measurement: Calculate the three-dimensional centroid coordinates of the point cloud cluster to determine its spatial location.

[0131] Shape ratio: The approximate ratio of the surface area to the volume of a point cloud cluster, which is a dimensionless quantity related to shape compactness.

[0132] The statistical quantities obtained from the above calculations (e.g., normal vector variance, mean and standard deviation of curvature, three AABB dimensions, three centroid coordinates, and one surface area-to-volume ratio, totaling 12 scalars) are organized to form a fixed-dimensional low-dimensional geometric feature vector (e.g., 12-dimensional) to digitally describe the geometric properties of the component point cloud from different perspectives.

[0133] S3034: Although the aforementioned low-dimensional geometric feature vectors have clear physical meaning, the relationships between features are relatively independent and at a low level. To obtain a more compact and representative feature representation, they are input into a multilayer perceptron (MLP) for high-level encoding. This MLP is designed as a simple three-layer fully connected network, specifically:

[0134] The input layer receives 12-dimensional geometric features, the hidden layer is set to 64 neurons and uses the ReLU activation function to introduce non-linearity, and the output layer is set to 32 neurons and uses the Tanh activation function to normalize the output value to the range of [-1,1] to facilitate subsequent fusion and optimization.

[0135] The output of the MLP is the physical prior vector, which is not a simple list of geometric parameters, but a fused and abstracted representation learned through network weights. It encodes key physical structural information such as the regularity and smoothness of the component surface and the presence of overall geometric anomalies (such as large-area deformation) in a compact form.

[0136] S304: In the actual operating environment of elevator shafts, there are often slight temporal discrepancies between visible light and infrared images due to factors such as thermal conduction lag and differences in sensor response. Direct fusion may lead to feature confusion, affecting the distinction between real wear and instantaneous interference. Furthermore, traditional temporal attention mechanisms treat all time steps equally, making it difficult to focus on real progressive damage in complex backgrounds. Therefore, this embodiment introduces two enhancement mechanisms in the dual-branch network, aiming to improve the model's discriminative ability in dynamic and complex environments from two aspects: temporal alignment and feature modulation. The specific implementation is as follows:

[0137] S3041: Cross-modal timing adaptive calibration submodule.

[0138] During elevator operation, abnormal frictional heating of components can be observed in infrared thermography. However, since heat conduction takes time, its manifestation is often slightly delayed compared to mechanical movements or surface changes captured in visible light images. To ensure the accuracy of subsequent fusion, more precise temporal alignment of the data from these two modalities is required before feature extraction.

[0139] The workflow of the calibration submodule is as follows:

[0140] First, parallel shallow feature extraction is performed: two structurally identical and lightweight 2D convolutional networks are used to process visible light image sequences and infrared thermal imaging image sequences, respectively. Each network can be designed with two layers. The first layer uses a 3x3 convolutional kernel, outputting 16 feature channels, followed by ReLU activation; the second layer also uses a 3x3 convolutional kernel, outputting 8 channels, followed by ReLU activation. These two networks do not perform downsampling; the purpose is to quickly extract low-dimensional spatiotemporal feature maps that are sensitive to image content.

[0141] Secondly, cross-correlation analysis and delay estimation are performed: For the two feature sequences extracted above, the normalized cross-correlation function is calculated channel by channel in the time dimension. By locating the peak of the cross-correlation function curve, the overall frame offset of the infrared sequence relative to the visible light sequence can be obtained. In order to achieve sub-frame accuracy, a second interpolation fitting can be performed near the peak to estimate the accurate time delay offset δ (e.g., δ=0.3 frames).

[0142] Finally, time-series compensation and alignment are performed: based on the calculated time delay offset δ, the original infrared thermal imaging image sequence is resampled and aligned. If δ is not an integer, a bilinear temporal interpolation algorithm is used to calculate the infrared image frame at the offset time. If δ is an integer, the entire frame is directly slid, thereby enabling the visible light image sequence and the infrared thermal imaging image sequence to achieve higher-precision synchronization when representing the same physical event, providing quality assurance for subsequent cross-modal fusion.

[0143] Through the aforementioned adaptive calibration, the visible light and infrared sequences achieve higher-precision synchronization at the feature level, which helps the subsequent fusion network to more accurately correlate mechanical surface changes and temperature rise effects, and improves the detection consistency of complex defects such as friction anomalies.

[0144] S3042: Physically prior-guided temporal attention modulation.

[0145] The aim is to dynamically inject the structural regularity of components perceived by the physical prior branch as a soft constraint into the temporal attention computation, thereby guiding the network to focus on real gradual changes and ignoring transient artifacts.

[0146] First, let the vector output by the physical prior coding branch be p (32-dimensional), and map it to an attention bias vector b through a lightweight modulation network M. M can be a simple single-hidden-layer fully connected network, i.e., the input layer is 32-dimensional, corresponding to p; the hidden layer has 16 neurons, using the ReLU activation function; and the output layer dimension is designed to be the projection dimension d of the query vector Q and key vector K in the temporal attention layer. k Same (e.g., 64 dimensions). That is: .

[0147] This bias is introduced for modulation when calculating the attention score at the temporal self-attention layer. Specifically, the attention score calculation formula referenced from S3023... The modulation process modifies it to:

[0148] ;

[0149] Here, 1 is a row vector with all elements being 1, and its length is the same as the sequence length of the key vector K. This operation means that the bias vector b is broadcast and added to each column of the score matrix S, which is equivalent to applying a uniform bias adjustment to all key vectors.

[0150] Subsequently, the modulated score matrix S mod Softmax normalization is performed to obtain the final attention weights. Through training, the modulation network M learns to generate meaningful biases b based on the physical prior vector p. For example, when p represents a part with highly regular surface geometry, M may output a negative value b, thereby generally reducing the attention score. This causes the network to spontaneously reduce its attention to short-term abrupt changes, effectively suppressing feature responses caused by transient disturbances such as splashed oil.

[0151] This modulation mechanism enables the network to dynamically adjust the attention distribution according to the physical regularity of the component structure, thereby suppressing the interference of transient artifacts in the temporal dimension, enhancing the ability to focus on progressive wear characteristics, and further improving the stability of the system in complex environments such as oil and dust.

[0152] S305: Feature Fusion and Output. Following the aforementioned steps, feature vectors reflecting the dynamic changes in component appearance and temperature over time, as well as prior feature vectors encoding the physical properties of the component's geometric structure, have been extracted. To construct a unified feature representation suitable for physical model-driven decision-making, these two types of information need to be effectively fused. While simple concatenation can preserve the integrity of each type of information, it may overlook the deep correlations between them. However, a learnable fusion layer can adaptively uncover the interaction between temporal patterns and static structural properties. For example, certain specific wear dynamic patterns may have a significantly increased risk level when combined with abnormal geometric structural features. Therefore, this embodiment adopts the following fusion strategy:

[0153] S3051: The temporal feature vector (assumed to be 256-dimensional) output by the temporal feature extraction branch and the physical prior vector (32-dimensional) output by the physical prior coding branch are concatenated in terms of feature dimensions to obtain a joint feature vector of 288 dimensions.

[0154] S3052: The aforementioned 288-dimensional joint feature vector is input into the feature fusion fully connected layer. This fully connected layer not only reduces the dimensionality to a fixed target dimension (e.g., 256 dimensions) for easier subsequent processing, but more importantly, its weights learn during training how to weight and nonlinearly map temporal dynamic features and physical prior features according to the needs of elevator component state assessment. For example, the network might learn to reinforce patterns of persistent, small temperature increases that occur simultaneously with slight irregularities in component geometry, as this combination may indicate early anomalies in the friction pair. This layer typically includes activation functions such as ReLU to introduce nonlinearity.

[0155] S3053: The output of this fully connected fusion layer is the final product of the temporal feature fusion and physical constraint embedding module, namely a 256-dimensional fusion feature vector. Unlike a simple feature set, this fusion feature vector is a deep collaborative representation of temporal dynamic information and physical structural information learned by the network. It not only carries both types of information simultaneously, but also encodes the correlation strength and combination pattern between them, thereby enabling a more accurate representation of the state of the component, rather than just its appearance or shape.

[0156] This fused feature vector provides key input to the subsequent physical mechanism-driven adaptive decision-making module, enabling the decision-making process to be based on a unified feature representation that simultaneously embodies the change process and the structural essence.

[0157] In summary, through the process from S301 to S305, a complete processing link is constructed from multimodal data input to high-dimensional fusion feature vector output. It is not a simple data splicing, but rather a deep fusion of temporal dynamic information and component physical properties through specific technical means such as dual-branch network structure, temporal attention mechanism and physical prior modulation. The generated fusion feature vector provides a key input with both temporal discriminability and physical interpretability for the subsequent physical mechanism-driven adaptive decision module.

[0158] The physical mechanism-driven adaptive decision-making module is the core of the system's decision-making. Its core lies in abandoning the empirical fixed defect judgment threshold and adopting dynamic threshold calculation based on the physical wear model. The module has a pre-stored structured component physical wear knowledge base, which is organized in the form of a database to create a physical profile for each type of inspected component.

[0159] The physical wear and tear knowledge base includes the following three parts:

[0160] Material property parameters: such as the elastic modulus, Poisson's ratio, yield strength, and fracture toughness of guide rail steel; and the hardness, wear resistance coefficient, and glass transition temperature of engineering plastics for door lock rollers.

[0161] Standard stress model: describes the type, magnitude and direction of loads borne by components under normal elevator operation conditions, such as the bending and contact stress distribution of guide rails and the tensile fatigue stress spectrum of wire ropes.

[0162] Wear evolution equation: Based on physical theory, it correlates material properties, stress state and observable surface state parameters to predict wear development trend.

[0163] After the module receives the fused feature vector sent by the temporal feature fusion and physical constraint embedding module, the decision-making process starts according to the following steps:

[0164] S401: Component type identification and parameter call.

[0165] The adaptive decision module determines the specific method for identifying the type of component being detected, and can choose one or a combination of these methods depending on system configuration and data reliability:

[0166] (1) Recognition based on spatial location mapping: The system has a pre-stored mapping table of "detection point coordinates - component type". This mapping table is automatically generated when importing the digital shaft model (see S202) or manually calibrated and confirmed by the operation and maintenance personnel. When the robot arrives at a detection point and records its spatial pose, the module can directly query the mapping table to determine the preset component type corresponding to the current field of view center. This method relies on accurate robot positioning and pre-calibrated detection points, and the result is direct and certain.

[0167] (2) Feature-based identification: The module has a built-in lightweight component type classifier (e.g., a 2-3 layer fully connected neural network). The input to this classifier is the fused feature vector (e.g., 256-dimensional) finally output by the temporal feature fusion and physical constraint embedding module. In the cloud, the system uses fused feature vector samples accumulated from historical inspection tasks, which have been labeled with the correct component types by experts in the review reports, to supervise the training of this classifier. In the real-time inference of the inspection process, after the temporal feature fusion and physical constraint embedding module generates the fused feature vector of the current detection point, the adaptive decision module inputs it into this pre-trained classifier. The classifier outputs the probability distribution of the current data belonging to each preset component type (e.g., guide rail, door lock roller, guide shoe, etc.). The module takes the type with the highest probability as the identification result. This method does not strictly rely on absolute positioning accuracy and has a certain adaptive identification capability for detection points that are not accurately marked in the mapping table or for cases where there are slight positional changes due to component installation.

[0168] In practical applications, recognition results based on spatial location mapping can be given priority and used as a reference for verification or error correction of recognition results based on feature classification, so as to improve recognition capabilities.

[0169] In practical applications, recognition results based on spatial location mapping can be given priority and used as a reference for verification or error correction of feature-based classification recognition results. For example, when the highest probability output by the classifier is lower than a certain confidence threshold, the result of location mapping can be used to improve the overall recognition accuracy.

[0170] Once the component type is determined, the system automatically retrieves the material property parameter set and standard stress model that are completely corresponding to the component type from the physical wear knowledge base, providing accurate and matching basic data for subsequent physical model calculations.

[0171] S402: Feature parameter analysis.

[0172] The adaptive decision module has a built-in analytical network. Its function is to convert the fused feature vector output by the temporal feature fusion and physical constraint embedding module into quantifiable engineering parameters (such as crack length, corrosion area, etc.) and corresponding parameter uncertainty estimates that are directly connected to the physical wear model, thereby realizing the connection from deep learning features to physical calculation model.

[0173] The specific implementation includes the following essential technical features:

[0174] S4021: The analytical network employs a shallow fully connected structure and is trained through supervised learning. The training data comes from a cloud-based historical database and consists of labeled sample pairs. Each sample contains a historical fusion feature vector and real physical parameter values ​​(such as actual crack length and depth) determined by experts based on the original data (such as images and point clouds) and actual measurements. During training, the mean squared error (MSE) loss function is used, and the network is optimized through backpropagation to learn the mapping relationship from features to parameters.

[0175] S4022: The trained network, given the input fused feature vector, outputs estimated physical parameters of the corresponding defect. For example:

[0176] For surface scratch defects, output the equivalent scratch depth and length.

[0177] For corrosion defects, output the area percentage and average depth of the corrosion zone.

[0178] For abnormal temperatures, output the maximum temperature rise and area of ​​the overheated region.

[0179] S4023: To evaluate the reliability of the output parameters, the network synchronously outputs the uncertainty (such as standard deviation) of each parameter, which is achieved through any of the following specific technical means:

[0180] Bayesian neural network method: Replace the fully connected layers in the network with Bayesian layers (weights are probability distributions). During inference, the weight distribution is sampled multiple times and forward propagation is performed. The uncertainty is calculated based on the output parameter distribution.

[0181] Monte Carlo Dropout method: During both training and inference phases, Dropout activation is maintained in specific layers of the network (such as the last hidden layer). During inference, forward propagation is performed multiple times (such as 100 times) on the same input, and the statistical variance of the output is used as an estimate of uncertainty.

[0182] The final output of the analytical network includes physical parameter estimates and uncertainty estimates for each parameter, both of which serve as inputs for the physical model in S403 to perform adaptive threshold calculations.

[0183] S403: Physical model calculation and adaptive threshold generation.

[0184] Based on the component type identified by S401, the system retrieves the corresponding material properties and standard stress model from the knowledge base, and combines them with the feature parameters parsed by S402. These are then substituted into a preset wear evolution equation for forward physical deduction, thereby generating an adaptive decision threshold that matches the current component's specific state and the reliability of the input parameters. Based on the component failure mechanism, the system primarily employs two types of physical models:

[0185] For components subjected to cyclic loads and prone to fatigue cracking (such as guide rails), the Paris formula model based on fracture mechanics is adopted.

[0186] For components with sliding friction (such as guide shoes and door lock rollers), the tribological Arcard adhesive wear model is adopted.

[0187] Taking fatigue crack detection of elevator guide rails as an example, the calculation process of its adaptive threshold is as follows:

[0188] First, the Paris formula is used to describe crack propagation: .

[0189] da / dN: The amount of crack length extension per load cycle.

[0190] C,m: Material constants, retrieved from the knowledge base (depending on the fracture toughness of the guide rail steel, etc.).

[0191] ΔK: Stress intensity factor amplitude, its value is determined by the alternating stress amplitude σ provided by the standard stress model and the current equivalent crack length a0 analyzed by S402. Specifically, the appropriate calculation formula can be selected according to the crack model (such as surface crack, through crack). For example, for a simple model, there is... Where Y is a shape factor related to the geometry of the crack and the component. Next, taking the currently analyzed equivalent crack length a0 as the initial state, a prediction time span T is set (e.g., until the next planned inspection or a preset evaluation cycle). Based on the average number of daily operating cycles obtained from the elevator's historical operating data, the estimated number of load cycles N within time T is calculated. Subsequently, the Paris formula is solved using numerical integration methods (such as the Euler method or the Runge-Kutta method) to predict the length a_p that the crack may extend to after N cycles.

[0192] Next, a baseline safety factor γ0 is set (typically selected within the range of 1.5 to 3.0, depending on the importance level of the component). To adjust the confidence level of the current parameter, the predicted length a is... p The final safety factor γ is calculated by dynamically correlating the parameter uncertainty estimate U (which has been normalized) provided by S402. For example, the formula is: α is a preset adjustment factor (e.g., 0.5), which makes the safety factor γ increase as the uncertainty U of the analytical parameters increases, and the decision threshold automatically tends to be conservative, forming a safe boundary for intelligent fault tolerance.

[0193] Finally, by combining the predicted crack length and the dynamic safety factor, the final adaptive decision threshold Threshold is calculated:

[0194] ;

[0195] The threshold is the critical size at which the current guide rail crack has developed to the point where early warning, maintenance, or replacement is required.

[0196] For components using the Archard adhesive wear model (such as guide shoes and rollers), a tribologically based Archard adhesive wear model is employed. The approach to generating the adaptive threshold is consistent with that of the fatigue crack model, but the specific calculation process is as follows:

[0197] First, the basic formula for the Archard wear model is:

[0198] ;

[0199] Where V is the wear volume, K is the dimensionless wear coefficient, P is the contact pressure, S is the sliding distance, and H is the hardness of the softer material.

[0200] When the system is executed, it calls specific parameters from the knowledge base records corresponding to the identified component types. The wear coefficient K and material hardness H are obtained directly from the material property set; the rated contact pressure P is obtained from the standard force model. This value may be a constant determined according to the component design specifications, or it may be a simplified calculation formula related to the car load.

[0201] The estimation of the sliding distance S requires the use of elevator operating data. Based on the elevator's historical operating logs, the system calculates the average daily number of trips and the average travel distance, and then estimates the total sliding distance S within a future predicted time period T (before the next scheduled inspection). T .

[0202] Then, using the current wear depth d0 or wear volume v0 output by the S402 analytical network as the initial state, according to the Arcard formula, it can be deduced that after experiencing a sliding distance S T Then, the predicted value of the wear amount. For example, the predicted wear depth d. p It can be estimated using the following formula:

[0203] ;

[0204] Where A is the nominal area of ​​the wear contact surface, and this value is stored in the knowledge base as a component geometric parameter.

[0205] Next, similar to the crack model approach, a basic safety factor γ0 is set (e.g., 2.0 for safety-related components). Simultaneously, the uncertainty estimate U regarding the current wear depth parameter provided by the S402 analytical network is introduced. This is then calculated using the formula... Calculate the dynamic safety factor γ, where α is a preset sensitivity factor.

[0206] Finally, an adaptive decision threshold is generated. For example, for wear depth, the alarm threshold is:

[0207] ;

[0208] In this way, the system can dynamically calculate a warning threshold that matches the specific service status of friction components such as guide shoes and rollers, taking into account their material properties, design stress, actual operating intensity, and the reliability of current testing, thereby achieving personalized predictive maintenance judgments.

[0209] S404: Cumulative evolutionary projection mechanism based on historical damage archives.

[0210] To enable continuous tracking of progressive damage, the system establishes and maintains a long-term damage accumulation status file for each uniquely identifiable component. The file is stored in the form of a database record, and its unique identification key is consistent with the spatial pose index key generated in S204 (e.g., "elevator number_component type_rough position code"), ensuring a strong correlation between the file and the spatial location of the physical component.

[0211] The specific mechanism is as follows:

[0212] File creation and updating: After each inspection, the system will generate estimated values ​​of physical parameters (such as crack length a) from the output of the S402 parsing network. curr Wear depth d curr The parameter value is compared with the previous value recorded in the archive. If the current parameter value does not exceed the instantaneous adaptive threshold calculated by S403, the difference between it and the historical value is regarded as the micro-damage increment (e.g., Δa=a). curr -a prev This increment will be appended to the increment list of the component file along with the timestamp.

[0213] Initial State Fusion: When inspecting the same component again, the initial state parameters required for the physical model calculation (S403) are not directly based on the instantaneous values ​​from the current analysis, but rather on a weighted fusion value. Specifically, assuming the historical cumulative equivalent damage recorded in the archive is H (obtainable by accumulating past micro-damage increments), and the instantaneous value from the current analysis is M, then the initial value I used for model derivation is: ;

[0214] Here, the fusion weight β is a dynamic parameter, and its value is negatively correlated with the uncertainty estimate U of the parameters analyzed in this study. For example, it can be set to... Where k is the adjustment coefficient, that is, when the uncertainty U of the current detection result is high, the system trusts the historical cumulative data (H) more; when the current detection is very accurate (U is small), it tends to use the current instantaneous value (M).

[0215] File Reset: When the maintenance records confirm that a certain component has been repaired or replaced, the system will reset and clear the corresponding damage accumulation file for that component.

[0216] This mechanism transforms the basis of physical model deduction from "isolated snapshots" to "continuous history," enabling the calculated adaptive thresholds to more realistically reflect the individualized safety margins of components under specific service histories.

[0217] S405: Decision-making and report generation.

[0218] The decision requires two key inputs: a comprehensive evaluation value that characterizes the severity of the current defect, and an adaptive decision threshold calculated by S403.

[0219] The comprehensive evaluation value is obtained from the temporal feature fusion and physical constraint embedding module. Specifically, after the feature fusion fully connected layer of S305, the network adds an additional regression output head. This output head takes the fused feature vector output by S3052 as input, passes through a fully connected layer (output dimension 1) and a sigmoid activation function, and outputs a scalar value between 0 and 1. During the training phase, this output head and the severity labels of the defects (labeled by experts based on historical data) undergo supervised learning (using mean squared error loss), thereby learning to map the high-dimensional fused features into a comprehensive evaluation value S that can characterize the overall severity of the defects.

[0220] Decision-making: The module compares the comprehensive evaluation value S with the adaptive decision threshold Threshold generated by S403. If S≥Threshold, it is determined that there is an anomaly in the component that needs to be handled.

[0221] Defect localization and report generation: Simultaneously, the system performs precise defect localization. The classification output head in the temporal feature fusion and physical constraint embedding module will give the defect category. For defects that need to be localized (such as cracks and corrosion pits), the system combines the defect region information in the image parsed by S402 (such as bounding boxes or pixel masks), the precise spatial pose matrix recorded by the robot when acquiring the frame data (from S203), and the pre-calibrated camera intrinsic parameters, and calculates the three-dimensional spatial coordinate range of the defect region in the wellbore world coordinate system through visual geometric back projection.

[0222] Finally, the module automatically generates a structured inspection report. The report includes not only basic information such as elevator number, inspection time, component location, defect type and level, and maintenance recommendations, but also key technical parameters upon which the decision was based, such as: the physical model used, model input parameters (e.g., initial crack length a0), predicted future state (a_p), calculation uncertainty (U), dynamic safety factor (γ), and the final adaptive threshold (Threshold). This report deeply integrates and interprets the results of deep learning perception, physical model deduction, and historical data analysis, greatly improving the reliability and engineering acceptability of the results.

[0223] To ensure the system balances real-time performance and detection accuracy in complex inspection tasks, this system employs a hierarchical decision-making process. This process mainly consists of two levels: the first level is a rapid screening layer, designed to efficiently filter a large number of normal status points; the second level is a fine-grained analysis layer, responsible for in-depth and precise analysis of potential anomalies. The specific implementation process is as follows:

[0224] S501: When the inspection robot approaches or arrives at a preset detection point, the fast screening layer is immediately activated. This layer only uses the first two convolutional blocks of the three-dimensional convolutional backbone network in the temporal feature fusion and physical constraint embedding module (i.e., the first two convolutional blocks with a stride of 1×2×2 described in S3022) to maximize the inference speed. The system takes the latest continuous multimodal image sequence (e.g., the last 5 frames) acquired in real time, and after the same preprocessing process as S301, inputs it into these two convolutional blocks for fast forward propagation. The network outputs a preliminary spatiotemporal feature map, and then performs global average pooling on the feature map in the spatial dimension to obtain a low-dimensional preliminary feature vector of fixed length (e.g., 128 dimensions).

[0225] S502: The preliminary feature vector obtained in S501 is input into a pre-trained lightweight fully connected layer (this layer is trained as an independent output head of the backbone network). This layer outputs a scalar value, namely the primary alarm score. The system presets a primary alarm threshold, which is determined by calculating the distribution of primary alarm scores on a large number of historical normal component samples. For example, it can be set above twice the standard deviation of the mean of all normal sample scores to ensure high sensitivity (i.e., high recall) to potential anomalies.

[0226] If the score is below the threshold: the current component is determined to be in normal condition. The process terminates, and the system only saves the original image snapshot and robot pose at that point (S203), and instructs the robot to proceed to the next detection point.

[0227] If the score reaches or exceeds the threshold: it is determined that there is a potential anomaly, an alarm is triggered, the current detection point is locked, and the process enters the fine analysis layer.

[0228] S503: After the alarm is triggered, the system controls the robot to maintain a stable pose at the current detection point (reaching the stable state of S203); at the same time, the complete temporal feature fusion and physical constraint embedding module is started (that is, the complete network from S302 to temporal attention pooling, S303 physical prior coding branch and S305 feature fusion is enabled).

[0229] The system extracts historical multimodal time-series data (e.g., the sequence of synchronized images from the past 20 frames) and corresponding 3D point cloud data from a pre-maintained circular data buffer, which have a longer time window for the current detection point, as input to this layer. These data also undergo standard preprocessing (S301).

[0230] S504: The complete dual-branch network processes the rich input data. The temporal feature extraction branch (including attention mechanism) extracts dynamic evolution features, the physical prior encoding branch extracts geometric structure features, and finally generates a high-dimensional fusion feature vector that integrates temporal dynamics and physical properties through the feature fusion layer (S305).

[0231] S505: The fused feature vector generated in S504 is fed into the physical mechanism-driven adaptive decision-making module to execute the complete process from component identification (S401), parameter parsing (S402), adaptive threshold calculation based on the physical model (S403) to final decision (S405). During this process, the system automatically queries and fuses the long-term historical damage file of the component (S404), performs cumulative evolutionary deduction, and finally generates a detailed inspection report containing physical evidence and three-dimensional positioning.

[0232] Through the aforementioned hierarchical mechanism, the system concentrates its main computing resources on a small number of suspicious points that are quickly identified through screening. This allows for efficient and rapid filtering of a massive number of normal detection points while ensuring in-depth physical modeling and analysis of complex defects. This effectively balances the stringent requirements of real-time performance and accuracy in large-scale inspection tasks.

[0233] The cloud-based collaborative analysis and model iteration module is deployed on a remote cloud server or enterprise private cloud platform, and connects to the front-end inspection robot or edge computing device via a network. Its core function is to continuously optimize the system's detection capabilities and accumulate operational knowledge over a long period. The specific operation mechanism of this module is as follows:

[0234] S601: Each final detection report and its corresponding original fusion feature vector generated by the physical mechanism-driven adaptive decision-making module are uploaded to the cloud module in batches via MQTT or HTTP / 2 protocol after the inspection task is completed or at a preset cycle (such as daily).

[0235] After receiving the data, the cloud verifies and decompresses it, and then stores it in a structured time-series database or relational database. When archiving, a composite index is forcibly created for each record. The index key must include at least: the elevator's unique number, the component's spatial location index key (consistent with S204), the precise detection timestamp, and the report ID, thereby supporting efficient historical data tracking and retrieval.

[0236] S602: The cloud module deploys a web-based secure interaction platform. Authorized maintenance engineers can log in using their account, password, and two-factor authentication to select reports to be reviewed from the platform's task list.

[0237] The platform adopts a front-end and back-end separation architecture. The front-end uses WebGL technology to render 3D point clouds and displays synchronized visible light and infrared images side by side. Engineers can review the system's automatically labeled results through a graphical interface and can directly perform correction operations on the image or point cloud view, such as redrawing defective areas, selecting the correct definition type from the drop-down menu, and adjusting the severity level with a slider. All manual correction actions generate structured correction annotation files.

[0238] S603: All data pairs that have been verified or corrected by experts (the initial output of the system and the results of manual correction) are automatically combined into a high-quality labeled sample with version labels and stored in a dedicated incremental training dataset. This dataset is managed independently of the original training set and records the source report, component type, defect category and correction status of each sample, providing a data foundation for subsequent targeted training.

[0239] S604: The cloud maintains a complete model training environment. When the number of samples in the incremental training dataset reaches a preset threshold (e.g., 500), the fine-tuning process is automatically initiated.

[0240] The weights of the dual-branch network in the current front-end deployment's temporal feature fusion and physical constraint embedding module are used as initialization.

[0241] The training set mainly consists of the aforementioned incremental dataset.

[0242] The loss function is a weighted sum of cross-entropy loss and mean squared error loss. To enhance the learning of historical misclassified / missed samples, a sample-level weighting strategy is adopted. That is, the system dynamically calculates a weight coefficient greater than 1.0 based on whether the sample has been corrected and the magnitude of the correction, and multiplies it into the loss calculation for that sample.

[0243] The optimizer can be Adam or SGD with Momentum.

[0244] S605: The weight file of the newly trained model needs to be tested by an offline verification gateway.

[0245] Its performance (e.g., overall F1 score, recall rate for previously misclassified categories) is evaluated on an independent, labeled validation set to ensure that key metrics are improved or do not decline. After validation, the model is tagged with a version number and stored in the repository. The update package is distributed to each front-end device through an HTTPS secure channel. The front-end device receives and verifies the integrity in an idle state, loads the new model by hot update or restarting the service, and confirms the update status with the cloud. This ensures that the iteration process does not affect normal inspection tasks.

[0246] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0247] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment includes only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A depth vision-based inspection system for elevator door systems and key components of elevator shafts, characterized in that, include: The multimodal data acquisition and synchronization module is used to simultaneously acquire visible light images, infrared thermal imaging images, and 3D point cloud data of the target component, and bind timestamps and spatial pose indexes. The robot motion and data association control module is used to drive the robot to move and position, record pose information at detection points, and associate data with the multimodal data acquisition and synchronization module. The temporal feature fusion and physical constraint embedding module is used to receive multimodal temporal data streams that have been time-synchronized and pose-associated. It extracts and fuses temporal features and physical prior features through a dual-branch deep neural network that includes a temporal feature extraction branch and a physical prior encoding branch, and outputs a fused feature vector. The physical mechanism-driven adaptive decision-making module has a built-in physical wear knowledge base, which is used to identify the component type and parse the physical parameters based on the fused feature vector, calculate the adaptive threshold based on the physical model, and make defect judgment. The cloud-based collaborative analysis and model iteration module is used to store reports and data, provide a platform for manual review, and iteratively update the dual-branch deep neural network based on the review results.

2. The elevator door system and shaft key component inspection system based on depth vision according to claim 1, characterized in that, The multimodal data acquisition and synchronization module includes a visible light camera, an uncooled infrared thermal imager, and a three-dimensional lidar; The multimodal data acquisition and synchronization module is equipped with a high-precision hardware synchronization trigger to generate trigger pulse signals. After synchronization delay calibration, the sent trigger pulses are time-compensated so that the visible light camera, uncooled infrared thermal imager and three-dimensional lidar can start acquiring data simultaneously.

3. The elevator door system and shaft key component inspection system based on depth vision according to claim 2, characterized in that, The robot motion and data association control module is integrated on the inspection robot. The inspection robot calculates its trajectory by fusing data from the inertial measurement unit and the photoelectric encoder, and performs periodic pose correction using at least one of the following methods: visual reference point correction, laser reflector positioning, or preset zero point reset. The robot motion and data association control module automatically plans an inspection path consisting of a series of detection points containing position coordinates and observation attitudes, based on the imported digital three-dimensional wellbore model.

4. The elevator door system and shaft key component inspection system based on depth vision according to claim 1, characterized in that, The temporal feature extraction branch includes: The shallow feature extraction and stitching unit is used to organize the preprocessed visible light image and infrared thermal imaging image into a sequence, extract features through a three-dimensional convolutional layer, and stitch them together in the channel dimension to obtain a multimodal spatiotemporal feature tensor. A spatiotemporal convolutional backbone network is used to perform convolution processing on the multimodal spatiotemporal feature tensors to compress spatial size and increase feature abstraction ability, and output high-order spatiotemporal feature tensors; The temporal attention pooling unit is used to reshape the high-order spatiotemporal feature tensor into a feature matrix, input it into the temporal self-attention layer, and then perform global average pooling through a multi-head attention mechanism to output a temporal feature vector of fixed length.

5. The elevator door system and shaft key component inspection system based on depth vision according to claim 4, characterized in that, The physical prior coding branch includes: The point cloud segmentation unit is used to segment the target component's point cloud clusters from the coordinate-transformed 3D point cloud data using a region growing algorithm based on Euclidean distance. The geometric descriptor calculation unit is used to calculate the surface normal statistics, curvature statistics, size measurement, position measurement and shape ratio of the point cloud cluster to form a low-dimensional geometric feature vector. The encoding network, a multilayer perceptron, is used to perform high-level encoding on the low-dimensional geometric feature vector and output the physical prior vector.

6. The elevator door system and shaft key component inspection system based on depth vision according to claim 5, characterized in that, The temporal feature fusion and physical constraint embedding module also includes: The cross-modal temporal adaptive calibration submodule is used to extract shallow features of visible light image sequences and infrared thermal imaging image sequences through a lightweight convolutional network, estimate the time delay offset through cross-correlation analysis, and resample and align the infrared thermal imaging image sequences accordingly. The physical prior-guided temporal attention modulation submodule is used to map the physical prior vector into an attention bias vector through a modulation network, and to introduce the bias into the attention score calculation of the temporal self-attention layer for modulation.

7. The elevator door system and shaft key component inspection system based on depth vision according to claim 1, characterized in that, The physical mechanism-driven adaptive decision-making module performs the following steps: Part type identification and parameter retrieval: The part type is determined based on the detection point coordinate-part type mapping table and / or the built-in classifier, and the corresponding material property parameters and standard stress model are retrieved from the physical wear knowledge base; Feature parameter parsing: The fused feature vector is parsed into estimated values ​​of defect physical parameters and corresponding estimated values ​​of parameter uncertainty through a parsing network; Physical model calculation and adaptive threshold generation: Select the corresponding wear evolution equation according to the component type, combine the analyzed physical parameters and their uncertainties to perform forward physical deduction, and introduce a safety factor that is dynamically related to the uncertainty to generate an adaptive decision threshold. Decision-making and report generation: The comprehensive evaluation value representing the severity of the defect is compared with the adaptive decision threshold to determine the anomaly, and the defect is located in three dimensions by combining the robot pose to generate a detection report containing the decision basis parameters.

8. The elevator door system and shaft key component inspection system based on depth vision according to claim 7, characterized in that, The physical mechanism-driven adaptive decision-making module also includes a cumulative evolutionary inference mechanism based on historical damage archives. The cumulative evolution and deduction mechanism establishes and maintains a long-term damage accumulation state file for each component, and the unique identification key of the file is consistent with the spatial pose index key generated by the robot motion and data association control module. The initial state parameters used for physical model calculations are weighted fusion values ​​of the instantaneous values ​​from this analysis and the historical cumulative equivalent damage, where the fusion weights are negatively correlated with the uncertainty estimate of the parameters from this analysis.

9. The elevator door system and shaft key component inspection system based on depth vision according to claim 1, characterized in that, The system adopts a hierarchical decision-making process, including: Rapid screening layer: When the robot approaches the detection point, the network in the temporal feature fusion and physical constraint embedding module is used to quickly process the short sequence multimodal images acquired in real time to obtain a primary alarm score, and determine whether to trigger an alarm based on a preset threshold. Fine-grained analysis layer: After the alarm is triggered, the robot is controlled to remain stable at the current detection point, and the complete temporal feature fusion and physical constraint embedding module and the physical mechanism-driven adaptive decision-making module are started to perform a complete analysis of historical multimodal data with a longer time window and generate a detailed detection report.

10. The elevator door system and shaft key component inspection system based on depth vision according to claim 1, characterized in that, The cloud-based collaborative analysis and model iteration module performs the following steps: Data archiving: Receive and structure the detection reports and corresponding raw fusion feature vectors, and create a joint index for each record containing elevator number, component spatial location index key and detection timestamp; Manual review and correction: Authorized engineers review and correct the system's automatic annotation results through a web-based secure interaction platform, generating corrected annotation files; Model Iteration Update: The corrected data pairs form an incremental training dataset. When the sample size reaches a preset threshold, the model fine-tuning is automatically initiated, and the validated new model is sent to the front-end device for update.

Images