An escalator step detection method fusing YOLO algorithm and digital twin technology

Abstract

This invention discloses a method for detecting escalator steps that integrates the YOLO algorithm and digital twin technology. First, sensor and video stream data are collected to construct a step dataset. Then, YOLOv8n is modified by introducing the CCA attention mechanism and reconstructing the EL-Head detection head. After pruning and INT8 quantization, a lightweight TJ-YOLOv8 model is obtained and trained. The model is deployed to the edge to achieve real-time step detection, constructing topological relationships and marking suspected missing steps, generating abnormal metadata with 3D coordinates. A digital twin of the escalator is constructed to achieve data injection, virtual-real synchronization, and sensor calibration. A decision signal is generated by fusing multi-source heterogeneous features, and a three-level filtering system is formed by combining gradient accumulation and historical backtracking to determine anomalies. This invention offers high detection accuracy and fast response, achieving closed-loop management of missing steps and improving escalator safety and intelligent operation and maintenance.

Description

Technical Field

[0001] This invention belongs to the field of escalator detection technology, specifically relating to a real-time escalator step missing detection method that integrates the YOLO algorithm and digital twin. Background Technology

[0002] As special equipment in the public transportation sector, escalators, due to their continuous high-load operation, place stringent requirements on safety inspections. As the direct carrier of passengers, the missing or faulty steps can easily lead to serious safety accidents such as passengers falling or limbs getting caught, making them one of the most serious accident hazards of escalators. Therefore, it is crucial to achieve real-time and reliable inspection of them.

[0003] Current methods for detecting missing steps on escalators have the following shortcomings: ① Escalators adhering to current standards generally use photoelectric detection devices for missing step detection. While this method can achieve basic escalator stop protection, the photoelectric detectors are located in the upper and lower machine compartments, making them unable to detect missing steps in inclined areas. Furthermore, they are susceptible to interference from ambient light, dust, and oil, leading to the risk of false alarms or missed alarms. ② Many escalators still in service and adhering to older standards lack this detection function entirely, and their safety relies entirely on manual inspections during regular maintenance. Manual inspections suffer from low efficiency, high cost, strong subjectivity, and the inability to achieve 24-hour uninterrupted monitoring. They heavily depend on the experience and sense of responsibility of maintenance personnel, making it difficult to form a stable and reliable safety assurance loop.

[0004] In recent years, deep learning object detection algorithms (such as the YOLO series) and digital twin technology have been gradually applied to the field of escalator inspection. However, existing technologies still have significant shortcomings: the application of YOLO algorithms suffers from "technology silos," remaining only at the level of visual recognition and simple alarms. The detection results cannot be deeply correlated with the physical entity of the escalator and its real-time operating status, and the decision support value of massive amounts of detection data has not been explored. The application of digital twin technology is in its early stages, mostly consisting of static or simple dynamic 3D displays. It has failed to be deeply integrated with high-precision real-time anomaly detection algorithms, lacks closed-loop decision support capabilities, and operation and maintenance decisions still rely on offline experience and on-site judgment. Summary of the Invention

[0005] The purpose of this invention is to provide an escalator step detection method that integrates the YOLO algorithm and digital twins, thereby solving the problems mentioned in the background art.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following solution:

[0007] A method for detecting escalator steps that integrates the YOLO algorithm and digital twin technology includes the following steps:

[0008] Step S1: Synchronous acquisition and preprocessing of multi-source heterogeneous data: Acquire sensor data and video information of escalator step operation; and perform preprocessing. Based on the preprocessed data, construct a step state annotation dataset to provide a data foundation for model training.

[0009] Step S2: Improve the construction and training of the lightweight YOLOv8n detection model:

[0010] Based on the YOLOv8n architecture, the network was modified by reparameterizing the backbone network, inserting the Rep CCA attention mechanism into the neck network, and reconstructing the detection head into an EL-Head structure. Lightweighting was achieved through channel pruning and INT8 quantization. The modified model was trained and validated using the dataset constructed in step S1, resulting in a TJ-YOLOv8 model that meets the requirements of tiered detection accuracy.

[0011] Step S3: Real-time edge detection and abnormal metadata generation: Deploy the TJ-YOLOv8 model to the edge computing node, perform step detection on the real-time acquired and pre-processed escalator step operation video stream, output the detection results and dynamically construct the step chain topology relationship, mark suspected missing steps and generate abnormal event metadata with step 3D spatial coordinates;

[0012] Step S4: Dynamic bidirectional driving of digital twin model: Construct a digital twin of the escalator containing a 3D model of the steps, a kinematic model and a physical parameter database. Through the stream processing engine, real-time sensor data and abnormal event metadata are injected into the digital twin in real time to drive the virtual model to synchronize with the physical entity. At the same time, the kinematic model of the digital twin is used to reverse calibrate the sensor data drift and correct the reference coordinates of the step topology.

[0013] Step S5: Multimodal data fusion decision: Design a cross-domain feature fusion module to generate an instantaneous fusion decision signal by normalizing and weighting three types of heterogeneous features: the step height mutation amount collected by the sensor, the confidence decay amount of the TJ-YOLOv8 model, and the topological chain break length after digital twin calibration; introduce a gradient accumulation mechanism to determine whether the accumulated decision signal of multiple consecutive frames exceeds the threshold.

[0014] Then, by using a digital twin to backtrack and query the historical operating parameters of the suspected abnormal tiers for verification, a three-level decision filtering system of "instantaneous features → short-term accumulation → long-term backtracking" is formed to determine whether the abnormality persists.

[0015] Step S6: Millisecond-level alarm feedback and model self-update: If the verification anomaly persists, an emergency braking signal is sent to the escalator control system within 50ms, and the missing position is highlighted on the human-machine interface. At the same time, the data related to the anomaly event is packaged into training samples, and incremental learning is performed on the TJ-YOLOv8 model based on the federated learning framework. The digital twin synchronously generates a simulation fault case library to optimize the topology chain breakage prediction algorithm.

[0016] Further optimization involves using infrared ranging sensors, photoelectric sensor arrays, and industrial camera groups deployed in the upper and lower compartments of the escalator to collect escalator operation sensor data and video information under various conditions such as strong light, weak light, backlight, dust obstruction, and oil stains in different scenarios like shopping malls, subways, and train stations.

[0017] Preprocessing includes microsecond-level time alignment of sensor data and video stream using a hardware synchronization module, GPU-accelerated inter-frame difference processing of the video stream, extraction of step motion regions, and resolution compression. The dataset is labeled with three categories in YOLO format: "normal steps," "missing steps," and "occluded steps." The category ID, bounding box center point coordinates, and bounding box width and height are labeled according to the YOLO standard format. The labeled dataset is divided into training, validation, and test sets in an 8:1:1 ratio.

[0018] Further optimization, step S2 specifically includes:

[0019] Step S2.1: Complete the network transformation based on the YOLOv8n architecture, specifically including:

[0020] Backbone network modification: The basic structure of YOLOv8n's C2f module is retained, and all 3×3 standard convolutions in the module are replaced with RepVGG-style reparameterized convolutions; a three-branch parallel structure of 3×3 convolution, 1×1 convolution and residual connection is adopted in the training phase, and fused into a single 3×3 convolution in the inference phase.

[0021] Neck network enhancement: A RepCCA attention mechanism module is inserted after the YOLOv8n neck SPPF module. The RepCCA attention mechanism first performs pooling processing on the input features in three directions (global, height, and width) through the CCA module and then weighted and fused to obtain the CCA feature map. Then, a three-branch parallel structure is introduced for reparameter optimization.

[0022] Detection Head Reconstruction: The native YOLOv8n decoupling head is replaced with the efficient and lightweight EL-Head detection head. The two parallel 3×3 convolutions in the original decoupling head are merged into a single 3×3 convolution. A lightweight substructure consisting of a 1×1 convolution and a RepCCA module is introduced. The 1×1 convolution is responsible for channel dimensionality reduction and information integration, while the RepCCA module focuses on spatial feature extraction. The EL-Head output format is consistent with the native decoupling head, including target bounding box coordinates, confidence score, and class probability.

[0023] Step S2.2: Lightweight processing: Perform channel pruning and INT8 quantization on the modified network model to reduce the number of model parameters and computational cost;

[0024] Channel pruning specifically involves: performing L1 regularized sparsity training on the modified model, calculating the feature contribution of each convolutional channel, removing redundant channels with a feature contribution of <0.01, and fine-tuning the pruned model to recover from the accuracy loss.

[0025] The INT8 quantization process involves: selecting 1000 tiered samples covering normal, missing, and occluded conditions to construct a calibration dataset; using the Post-TrainingQuantization static quantization method to convert the model weights from 32-bit floating-point to 8-bit integer; verifying the model accuracy after quantization; if the tiered missing detection mAP@0.5 decreases by more than 2%, then re-selecting calibration samples for quantization.

[0026] Step S2.3: Model Training: Using the tiered state dataset, a customized training strategy is adopted to train the lightweight model; based on the PyTorch framework, the SGD optimizer, EIoU loss function and class balance factor are used for model training, with the initial learning rate set to 0.01, weight decay coefficient to 0.0005, batch size to 32 and training epochs to 100, and the learning rate is adjusted using a cosine annealing strategy.

[0027] Step S2.4: Model Validation: Using mAP@0.5 as the core evaluation metric, when the stepwise missing detection mAP@0.5 ≥ 95%, the false negative rate < 0.5%, and the false positive rate < 0.1% on the validation set, the TJ-YOLOv8 model is determined to be successfully trained.

[0028] Further optimization involves the CCA module processing the input feature map X in step S2. CCA ∈R C×H×W The processing procedure specifically includes:

[0029] 1) Feature splitting and multi-directional pooling: The input feature map X is split into... CCA Split by channel, for the first cThe features of each channel are respectively subjected to horizontal pooling, vertical pooling, and global pooling to obtain corresponding feature tensors, where 0 ≤ c <C, specifically:

[0030] Horizontal pooling: Traverse and sum along the width direction for the rows with a fixed height of h and take the average to obtain the horizontal pooling feature tensor :

[0031] ;

[0032] where h is the position index in the height direction and 0 ≤ h <H, i is the traversal index in the width direction and 0 ≤ i <W, x c ( h , i ) is the pixel feature value at the height h , width i in the c-th channel;

[0033] Vertical pooling: Traverse and sum along the height direction for the columns with a fixed width of w and take the average to obtain the vertical pooling feature tensor :

[0034] ;

[0035] where w is the position index in the width direction and 0 ≤ w <W, j is the traversal index in the height direction and 0 ≤ j <H, x c ( j , w ) is the pixel feature value at the height j , width w in the c-th channel;

[0036] Global pooling: Traverse and sum the entire feature space and take the average to obtain the global pooling feature tensor: ;

[0037] z c ( w , h ) is the global pooling feature value at the width c , height w in the h -th channel; C is the total number of channels of the feature map, H is the total number of pixels in the height of the feature map, and W is the total number of pixels in the width of the feature map.

[0038] 2) Global feature weight generation: This involves generating global pooling feature tensors. z c ( w , h The channels are compressed using a 1×1 convolution transform function F1, and then the global feature weights g are obtained through a non-linear activation function Sigmoid. c The formula is:

[0039] ;

[0040] 3) Intermediate mapping feature map generation: To enhance the information exchange capability in both the horizontal and vertical directions, the horizontal pooling feature tensor is generated. and vertical pooling feature tensor The data is stitched together in the spatial dimension, and then processed by a 1×1 convolutional transform function F1 to compress the channels, batch normalization, and nonlinear activation to obtain the intermediate mapping feature map. f The formula is:

[0041] ;

[0042] in, This indicates a concat operation performed in two directions, where δ represents the non-linear activation function. r This indicates the channel compression ratio.

[0043] 4) Generation of horizontal and vertical feature weights: This involves mapping the intermediate feature map... f Split along the spatial dimension to obtain independent feature tensors. And, then convolved separately with 1×1. F h , F w This operation, combined with the Sigmoid activation function, yields feature weights in both the horizontal and vertical directions. g hor and g ver The formula is:

[0044] ;

[0045] ;

[0046] 5) Weighted fusion to obtain CCA features: The global feature weights are then combined. g c Horizontal feature weights g hor Vertical feature weights g verElement-wise weighted fusion is used to obtain the feature map output by the CCA module. Y CCA ( w , h The formula is:

[0047] ;

[0048] The input feature map has dimensions of C (number of channels) × H (height) × W (width). w The current position in the width dimension. i Indicates the traversal index in the width direction. h The current position in the height dimension. j Indicates the traversal index in the height direction.

[0049] Further optimization involves using a three-branch parallel structure for the RepCCA attention mechanism in step S2. During training, 3×3 convolutions, 1×1 convolutions, and residual connections are used to process the CCA module's output features in parallel. The output formula is as follows:

[0050] ;

[0051] During the inference phase, the three branches are merged into a single 3×3 convolution, and the output formula is as follows: ;where X RepCCA The input feature map is the RepCCA module, where W1 and W2 are the convolution weights, and b1 and b2 are the convolution biases. For convolution operations, f CCA () is the CCA module mapping function.

[0052] Further optimization, in step S3, the real-time edge detection and anomaly metadata generation specifically includes:

[0053] Step S3.1: Model Deployment: Convert the TJ-YOLOv8 model to the ONNX / TensorRT lightweight format and deploy it to the edge computing node;

[0054] Step S3.2: Real-time video stream input and detection inference: The edge computing node receives the pre-processed tiered video stream from the front end, performs format conversion, normalization, and dimension adjustment on the video frames, and obtains single-frame data for the adaptation model input.

[0055] The TJ-YOLOv8 model is used for inference on a single frame of data, outputting the diagonal bounding box coordinates (x, y, y) of each ladder instance. n,1 ,y n,1 ), (x n,2 ,y n,2), object confidence and class probability distribution of normal / missing / occluded tiers; merging object confidence and class probability to calculate the class confidence score S for tiers of normal tiers. norm S norm = Object confidence score × Normal ladder category probability

[0056] Step S3.3: Construction and Update of Ladder-like Chain Topology: Extract the coordinates P of the center point of each ladder step in each frame. center (x n ,y n )=((x n,1 +x n,2 ) / 2,(y n,1 +y n,2 ) / 2), calculate the relative displacement vector (Δx,Δy) of the center points of adjacent steps, and dynamically construct and update the step chain topology graph;

[0057] The ladder-chain topological relationship is the set of coordinate differences of the center points of adjacent ladder steps in the x-axis and y-axis directions of the image plane within the same frame, Δx=x n -x n-1 Δy=y n -y n-1 , where (x n y n (x) represents the coordinates of the center point of the nth step. n-1 y n-1 ) represents the coordinates of the center point of the (n-1)th step; the normal range of Δx is [20,25] pixels, and the normal range of Δy is [0,5] pixels.

[0058] Step S3.4: Mark suspected missing steps: Set confidence threshold T conf =0.85, when S norm <T conf If the relative displacement vector of adjacent steps exceeds the normal range, mark the corresponding step as suspected missing.

[0059] The rule for marking suspected missing steps is as follows: assign a unique ID to a suspected missing step, in the format of "escalator ID-timestamp-step number", and record the reason for marking, including insufficient confidence, broken topology chain, or both.

[0060] Step S3.5: Abnormal Event Metadata Generation: Fusing the pixel coordinates of the ladder center point with the height data collected by the infrared ranging sensor, and converting the intrinsic and extrinsic parameters of the camera to obtain the three-dimensional spatial coordinates P in the world coordinate system of the digital twin. 3D (X word ,Y word Z word Generate abnormal event metadata containing the coordinates, detection timestamp, and topology chain break information;

[0061] The transformation formula for the three-dimensional spatial coordinates of the ladder is:

[0062]

[0063] in x pixel , y pixel These are the pixel coordinates of the center point of the ladder step. c x , c y These are the coordinates of the camera's principal point. f x , f y For camera focal length, Z infra The data represents the height of the steps, collected by an infrared ranging sensor.

[0064] The abnormal event metadata is in JSON format and includes a unique abnormal ID, escalator number, microsecond-level timestamp, step number, three-dimensional spatial coordinates, model detection results, topology chain information, marking reason, and synchronous sensor data fields. The time deviation between the abnormal metadata timestamp and the sensor data time is no more than 100μs.

[0065] Further optimization involves the digital twin model being dynamically driven bidirectionally in step S4, specifically including:

[0066] Step S4.1: Construct a digital twin of the escalator, import the 3D model of the steps into the geometry layer, and embed the kinematic formula L of the steps into the kinematic layer. k (t)=L0+v×t+1 / 2a×t 2 The data layer stores the escalator's rated parameters and sensor calibration parameters; among them, L step (t) represents the displacement / position of the k-th step at time t, and L0 represents the initial position of the step. v The rated linear speed of the cascade. a To accelerate the cascade operation, t The current time;

[0067] Step S4.2: Inject sensor data and abnormal event metadata into the digital twin using the Kafka stream processing engine. In the virtual model, normal steps are kept displayed, while steps suspected of being missing are hidden or deformed.

[0068] Step S4.3: Calculate the theoretical height H of the steps based on the kinematic model. theory (t), calculate the actual height value H of the sensor. real The deviation ΔH = H of (t) real (t)-Htheory (t), correcting the sensor data to H cal (t)=H real (t)-ΔH;

[0069] Step S4.4: Using the calibrated height data as a reference, correct the reference coordinates of the ladder chain topology, using the formula P. 3D-cal (X cal ,Y cal Z cal )= P 3D (X word ,Y word Z word )+(0,0,ΔH).

[0070] Further optimization, in step S5, the multimodal data fusion decision specifically includes:

[0071] Step S5.1: Extract three types of heterogeneous features: extract the step height mutation amount F. h (t)=∣H cal (t)-H cal (t-1)∣、Confidence decay F c (t)=S norm (t-1)-S norm (t) and topological chain break length ;

[0072] Step S5.2: Perform min-max normalization on the three types of features to obtain N. h (t), N c (t), N t (t), weighted fusion generates instantaneous decision signal S(t)=αN h (t)+βN c (t)+γN t (t), where α, β, and γ are weighting coefficients adjusted according to the actual application environment;

[0073] Step S5.3: Calculate the sum of the instantaneous decision signals of three consecutive frames as the cumulative decision signal. Set the cumulative threshold T A When A(t) > T A This triggers historical rewind;

[0074] Step S5.4: Query the operating parameters of the suspected abnormal step over the past 5 seconds using the digital twin, and calculate the historical backtracking evaluation value at time t. Set the backtracking threshold T R When S(t) > 0.5 and A(t) > 1.5 and R(t) > T R When the anomaly is determined to persist, then... P (tg ) represents the quantized value of the single-frame anomaly feature corresponding to the ladder level at time tg, which is obtained by normalizing the instantaneous decision signal S(ti), and g=0,1,2,3,4.

[0075] Further optimization, in step S6, the millisecond-level alarm feedback and model self-update specifically include:

[0076] Step S6.1: If the abnormality is determined to persist, an emergency braking signal is sent through the Profinet / EtherCAT industrial bus within 50ms, and the three-dimensional spatial coordinates of the suspected missing step are highlighted on the human-machine interface.

[0077] Step S6.2: Package the abnormal video clips, sensor time series data, decision signals and judgment labels into a training sample set and supplement it to the tiered state dataset in step S1;

[0078] Step S6.3: Based on the federated learning framework, freeze the backbone network of the TJ-YOLOv8 model, and only fine-tune the neck network and EL-Head detection head. The fine-tuning parameters are learning rate 0.001, batch size 16, and training epochs 10. After completion, distribute the updated model weights to the edge nodes.

[0079] Step S6.4: The digital twin generates a simulation fault case library based on abnormal metadata and optimizes the judgment threshold of the topology chain break prediction algorithm.

[0080] Compared with the prior art, the present invention has the following beneficial effects:

[0081] 1. This invention modifies the YOLOv8n model in multiple dimensions, introducing the CCA attention mechanism and the EL-Head lightweight detection head. Combined with channel pruning and INT8 quantization for lightweight processing, the resulting TJ-YOLOv8 model achieves a stepwise detection mAP@0.5≥95% while maintaining inference speed. This effectively solves the problems of traditional detection methods being susceptible to environmental interference and having high false alarm and false positive rates. At the same time, it constructs a three-level decision filtering system that fuses multimodal heterogeneous features, integrating sensor, visual inspection, and digital twin data to effectively distinguish between real faults and transient interference, further improving detection accuracy.

[0082] 2. This invention deploys a lightweight TJ-YOLOv8 model to edge computing nodes to achieve local real-time inference of video streams, with a single-frame inference time of <16ms. Combined with the virtual-real synchronization and rapid calibration of the digital twin, it can complete the emergency braking signal transmission and visual alarm within 50ms after determining that the anomaly persists, meeting the millisecond-level response requirements for escalator step missing faults.

[0083] 3. This invention constructs a digital twin of an escalator to achieve real-time injection of sensor data and abnormal event metadata. Through bidirectional mapping and reverse calibration between physical and virtual spaces, it achieves precise synchronization between the virtual model and the physical entity, solving the problem of existing digital twin technology being "heavy on visuals, light on interaction, and lacking in intelligence". It enables precise three-dimensional positioning and historical status backtracking of missing step faults.

[0084] 4. After an abnormal event occurs, the system automatically packages the relevant data into training samples, realizes incremental training and iterative optimization of the model based on federated learning, and optimizes the prediction algorithm by generating a simulation fault case library from the digital twin, enabling the detection system to have self-learning and self-optimization capabilities. As the running time increases, the detection accuracy continues to improve, significantly reducing the cost of manual inspection and model updates, and realizing intelligent operation and maintenance of escalators. Attached Figure Description

[0085] Figure 1 This is a flowchart of an escalator step detection method that integrates the YOLO algorithm and digital twin technology, as described in this invention.

[0086] Figure 2 This is a system architecture and data flow diagram for multi-source heterogeneous data acquisition and preprocessing in this invention;

[0087] Figure 3 This is a schematic diagram of the network structure of the TJ-YOLOv8 model of the present invention;

[0088] Figure 4 This is a schematic diagram of the network structure of the CCA attention mechanism of the present invention;

[0089] Figure 5 To improve the comparison image of the detection head before and after;

[0090] Figure 6 This is the performance evaluation curve of the tiered missing detection model in this embodiment of the invention;

[0091] Figure 7 This is a timing diagram for the bidirectional driving of the digital twin of the present invention;

[0092] Figure 8 This is a flowchart illustrating the logic of multimodal data fusion decision-making in this invention. Detailed Implementation

[0093] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0094] like Figure 1 As shown in the figure, this embodiment provides a method for detecting missing steps in escalators that integrates the YOLO algorithm and digital twin technology. This method is applied to the detection of missing steps in shopping mall escalators. The specific implementation steps are as follows:

[0095] Step S1: Multi-source heterogeneous data acquisition, preprocessing, and dataset construction: Infrared ranging sensors, photoelectric sensor arrays, and a 4-channel industrial camera group are deployed in the upper and lower compartments of the shopping mall escalator. The infrared ranging sensors acquire time-series data of the escalator step height at a frequency of 100Hz, and the industrial camera group captures the video stream of the escalator steps at a frame rate of 60fps. The sensor data and video stream are aligned at the microsecond level using a hardware synchronization module. GPU acceleration is used to perform inter-frame difference preprocessing on the video stream, extracting the motion region of the escalators and compressing the resolution to 640×480 pixels. Figure 2 As shown.

[0096] A total of 100,000 pre-processed video frames and sensor data were collected under various conditions such as strong light, weak light, dust obstruction, and oil stains during the daily operation of the shopping mall. The data were labeled with three categories of labels in YOLO format: "normal ladder", "missing ladder", and "occluded ladder". The labeled dataset was divided into 80,000 training frames, 10,000 validation frames, and 10,000 test frames in an 8:1:1 ratio to construct a ladder state labeling dataset.

[0097] Step S2, TJ-YOLOv8 model construction and training, based on the YOLOv8n architecture, according to... Figure 3 The TJ-YOLOv8 model network structure shown has undergone full network transformation and lightweighting. The transformation process combines... Figure 3 , Figure 4 The specific process is as follows:

[0098] Backbone network modification: The basic structure of YOLOv8n's C2f module is retained, and all 3×3 standard convolutions in the module are replaced with RepVGG-style reparameterized convolutions. During the training phase, a three-branch parallel structure of 3×3 convolutions, 1×1 convolutions and residual connections is adopted, and during the inference phase, it is fused into a single 3×3 convolution, which improves the inference speed while ensuring the feature extraction capability.

[0099] Neck network enhancement: Insert a RepCCA attention mechanism module after the YOLOv8n neck SPPF module, such as... Figure 4 As shown, the RepCCA attention mechanism first processes the input feature map X through the CCA module. CCA ∈R 256×80×80(Channel compression ratio r=16) Pooling is performed in three directions: horizontal, vertical and global. After weight generation and weighted fusion, the CCA feature map is obtained. Then, the model is optimized by reparameterization through a three-branch parallel structure to enhance the model's ability to extract features of the escalator space and solve the feature confusion problem of continuous escalator steps.

[0100] Detection head reconstruction: The original decoupled head is replaced with the EL-Head high-efficiency and lightweight detection head. The two parallel 3×3 convolutions in the original decoupled head are merged into a single 3×3 convolution. A lightweight substructure of 1×1 convolution and RepCCA module is introduced. The 1×1 convolution is responsible for channel dimensionality reduction and information integration, such as... Figure 5 As shown, where Figure 5 (a) in the diagram is a schematic diagram of the detection head network structure before the improvement. Figure 5 (b) in the diagram is a schematic of the improved detection head network structure. The RepCCA module focuses on spatial feature extraction, and the EL-Head output format is consistent with the native decoupled head, including the target bounding box coordinates, detection confidence, and class probability distribution.

[0101] Lightweight processing: The modified model was trained with L1 regularization sparsity, redundant convolutional channels with feature contribution less than 0.01 were removed, and after pruning, it was fine-tuned until the validation set mAP@0.5 recovered to more than 95%. A calibration dataset was constructed by selecting 1000 step samples covering all working conditions. The Post-Training Quantization static quantization method was used to complete INT8 quantization, converting the model weights from 32-bit floating-point type to 8-bit integer type. After quantization, the model mAP@0.5 decreased by <1%, which met the accuracy requirements.

[0102] Model Training and Validation: The model was trained using the PyTorch framework, employing the SGD optimizer, EIoU loss function, and a missing class balancing factor of 5.0. Data augmentation operations, including Mosaic enhancement, random pruning, and color gamut transformation, were performed on the training set. Training parameters were set as follows: initial learning rate 0.01, weight decay coefficient 0.0005, batch size 32, and training epochs 100. A cosine annealing strategy was used to dynamically adjust the learning rate.

[0103] After training, the model performance is validated on the test set, and the precision-recall (PR) curve is obtained as shown below. Figure 6 As shown in the figure, the curves are close to the upper right corner (Precision≈1.0, Recall≈1.0), indicating that the model maintains a very high recall rate while the precision is almost unaffected. The final model has a stepwise missing detection mAP@0.5=96.8%, a false negative rate Rmiss=0.3%, and a false positive rate Rfalse=0.08%, all of which meet the preset threshold requirements, and is determined to be a successfully trained TJ-YOLOv8 model.

[0104] Step S3, Real-time Edge Detection and Anomaly Metadata Generation: The TJ-YOLOv8 model is converted to ONNX format and deployed to an NVIDIA Jetson Xavier NX edge computing node. Frame-by-frame detection of the pre-processed video stream from the shopping mall escalator is performed. Specific steps are as follows:

[0105] Step S3.1, Real-time Detection and Inference: The edge computing node receives the pre-processed ladder-running video stream from the front end, performs format conversion, normalization, and dimension adjustment on the video frames, calls the TJ-YOLOv8 model for frame-by-frame inference, and outputs the diagonal bounding box coordinates (x, y, y) of each ladder instance. n,1 ,y n,1 ), (x n,2 ,y n,2 ), object confidence and the category probability distribution of the normal / missing / occluded hierarchy, and calculate the category confidence score S of the normal hierarchy. norm =Object confidence score × Normal class probability, set confidence threshold T conf =0.85.

[0106] Step S3.2, Construction of ladder-like chain topology: Extract the coordinates P of the center point of the ladder-like bounding box in each frame. center (x n ,y n )=((x n,1 +x n,2 ) / 2,(y n,1 +y n,2 ) / 2), calculate the relative displacement vector (Δx,Δy) of the center points of adjacent steps. The normal range of Δx is [20,25] pixels and the normal range of Δy is [0,5] pixels. Dynamically construct and update the step chain topology graph. When Δx is not within the range of [20,25] pixels or Δy is not within the range of [0,5] pixels, it is determined to be an abnormal topology chain displacement.

[0107] Step S3.3, Suspected Missing Marker and Metadata Generation: When a certain ladder S norm When the value is 0.72 (below the threshold of 0.85) and the topology chain is broken, the step is marked as a suspected missing step, assigned a unique ID—shopping mall escalator ID—microsecond-level timestamp—step sequence number, and the triggering reason is recorded as: insufficient confidence and broken topology chain. The pixel coordinates of the step's center point are fused with the height data collected by the infrared ranging sensor, and the 3D spatial coordinates P in the digital twin world coordinate system are obtained through camera intrinsic and extrinsic parameter transformation. 3D (X word ,Y word Z word= (1250.5, 820.3, 150.8) (unit: mm), generates JSON format exception event metadata, including fields such as exception unique ID, escalator number, microsecond-level timestamp, step number, three-dimensional spatial coordinates, model detection results, topology chain information, marking reason and synchronous sensor data.

[0108] Step S4, Digital Twin Data Injection and Calibration: Construct a digital twin of the shopping mall's escalator, including a 3D CAD model of the steps, a kinematic model of the steps, and a database of escalator physical parameters. Using a Kafka streaming engine, sensor data and abnormal event metadata are injected into the digital twin in real time, driving the corresponding step models within the twin to perform hiding operations, thus synchronizing the virtual scene with the physical entity's state. The kinematic model of the digital twin is used to calculate the theoretical height of the steps, and the height time-series data from the infrared ranging sensor is calibrated to correct the reference coordinates of the step chain topology, improving the accuracy of the topology. Figure 7 As shown.

[0109] Step S5: Multimodal heterogeneous feature fusion decision, such as... Figure 8 As shown, the abrupt change in the height of the steps, F, collected by the photoelectric sensor, is extracted. h (t)=0.08, confidence decay F of the TJ-YOLOv8 model c (t)=0.24, the topological chain break length F after digital twin calibration t (t)=12, and N is obtained by performing min-max normalization on the three types of features. h (t)=0.85, N c (t)=0.92, N t (t)=0.88; setting weight coefficients α=0.4, β=0.35, γ=0.25, the instantaneous fusion decision signal S(t)=0.4×0.85+0.35×0.92+0.25×0.88=0.882 is calculated; the sum of the instantaneous fusion decision signals of the current and the past 3 consecutive frames is calculated, and the cumulative decision signal A(t)=0.882+0.856+0.821=2.559 is obtained, which exceeds the preset threshold T. A =1.5, triggering historical state backtracking; by backtracking the digital twin to query the operating status parameters of the suspected missing step in the past 5 seconds, the historical state backtracking evaluation value R(t) = 10.23 is calculated, and three-level judgment conditions are set: S(t) > 0.5, A(t) > T A =1.5、R(t)>T R =5.0. All three conditions were met in this test, and it was ultimately determined that the missing step anomaly persisted.

[0110] Step S6, Millisecond-level alarm feedback and model iterative optimization: Within 45ms after the abnormality is detected, an E-stop emergency braking signal is sent to the escalator control system via the Profinet industrial bus, and the escalator stops immediately. At the same time, the three-dimensional spatial location of the missing step is highlighted on the human-machine interface of the mall's operation and maintenance center.

[0111] The video clips, sensor time-series data, real-time fusion decision signals, and cumulative decision signals of the abnormal event are packaged into a training sample set and added to the tiered state dataset to enrich the model's fault sample library and solve the problem of scarce fault samples. Based on the federated learning framework, the backbone network of the TJ-YOLOv8 model is frozen, and only the neck and detection head are fine-tuned, and incremental training is performed on the edge model. The digital twin automatically generates simulated fault cases of the missing tier event and adds them to the fault case library. The topology chain break prediction algorithm is optimized to improve the accuracy of subsequent fault detection.

[0112] Based on the preferred embodiments of the present invention and the above description, those skilled in the art can make various changes and modifications without departing from the technical concept of the present invention. The technical scope of the present invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. An escalator step detection method fusing YOLO algorithm and digital twin technology, characterized in that, Includes the following steps: Step S1: Synchronous Acquisition and Preprocessing of Multi-Source Heterogeneous Data: Acquire sensor data and video information of escalator step operation; The data is then preprocessed, and tiered state labeling data is constructed based on the preprocessed data. Step S2: Improve the construction and training of the lightweight YOLOv8n detection model: Based on the YOLOv8n architecture, the network was modified by reparameterizing the backbone network, inserting the RepCCA attention mechanism into the neck network, and reconstructing the detection head into an EL-Head structure. Lightweighting was achieved through channel pruning and INT8 quantization. Then, the modified model was trained and validated using the dataset constructed in step S1 to obtain the TJ-YOLOv8 model that meets the requirements of tiered detection accuracy. Step S3: Real-time edge detection and abnormal metadata generation: Deploy the TJ-YOLOv8 model to the edge computing node, perform step detection on the real-time acquired and pre-processed escalator step operation video stream, output the detection results and dynamically construct the step chain topology relationship, mark suspected missing steps and generate abnormal event metadata with step 3D spatial coordinates; Step S4: Dynamic bidirectional driving of digital twin model: Construct a digital twin of the escalator containing a 3D model of the steps, a kinematic model and a physical parameter database. Through the stream processing engine, real-time sensor data and abnormal event metadata are injected into the digital twin in real time to drive the virtual model to synchronize with the physical entity. At the same time, the kinematic model of the digital twin is used to reverse calibrate the sensor data drift and correct the reference coordinates of the step topology. Step S5: Multimodal data fusion decision: Design a cross-domain feature fusion module to generate an instantaneous fusion decision signal by normalizing and weighting three types of heterogeneous features: the step height mutation amount collected by the sensor, the confidence decay amount of the TJ-YOLOv8 model, and the topological chain break length after digital twin calibration; introduce a gradient accumulation mechanism to determine whether the accumulated decision signal of multiple consecutive frames exceeds the threshold. Then, by using a digital twin to backtrack and query the historical operating parameters of the suspected abnormal tiers for verification, a three-level decision filtering system of "instantaneous features → short-term accumulation → long-term backtracking" is formed to determine whether the abnormality persists. Step S6: Millisecond-level alarm feedback and model self-updating: If the verification anomaly persists, an emergency braking signal is sent to the escalator control system within 50ms, and the missing location is highlighted on the human-machine interface. At the same time, the relevant data of the anomaly event is packaged into training samples, and incremental learning is performed on the TJ-YOLOv8 model based on the federated learning framework. The digital twin synchronously generates a simulation fault case library to optimize the topology chain breakage prediction algorithm.

2. The escalator step detection method of claim 1, wherein, In step S1, infrared ranging sensors, photoelectric sensor arrays and industrial camera groups are arranged in the upper and lower compartments of the escalator to collect escalator operation sensing data and video information under different scenarios and working conditions. Preprocessing includes microsecond-level time alignment of sensor data and video stream using a hardware synchronization module, GPU-accelerated inter-frame difference processing of the video stream, extraction of step motion regions, and resolution compression. Label the data in the YOLO format with three categories: "normal step", "missing step" and "occluded step". Label the category ID, bounding box center point coordinates and bounding box width and height in the YOLO standard format. Divide the labeled dataset into training set, validation set and test set in an 8:1:1 ratio.

3. The escalator step detection method of claim 1, wherein, Step S2 specifically includes: Step S2.1: Complete the network transformation based on the YOLOv8n architecture, specifically including: Backbone network modification: The basic structure of YOLOv8n's C2f module is retained, and all 3×3 standard convolutions in the module are replaced with RepVGG-style reparameterized convolutions; a three-branch parallel structure of 3×3 convolution, 1×1 convolution and residual connection is adopted in the training phase, and fused into a single 3×3 convolution in the inference phase. Neck network enhancement: A RepCCA attention mechanism module is inserted after the YOLOv8n neck SPPF module. The RepCCA attention mechanism first performs pooling processing on the input features in three directions (global, height, and width) through the CCA module and then weighted and fused to obtain the CCA feature map. Then, a three-branch parallel structure is introduced for reparameter optimization. Detection Head Reconstruction: The native YOLOv8n decoupling head is replaced with the efficient and lightweight EL-Head detection head. The two parallel 3×3 convolutions in the original decoupling head are merged into a single 3×3 convolution. A lightweight substructure consisting of a 1×1 convolution and a RepCCA module is introduced. The 1×1 convolution is responsible for channel dimensionality reduction and information integration, while the RepCCA module focuses on spatial feature extraction. The EL-Head output format is consistent with the native decoupling head, including target bounding box coordinates, confidence score, and class probability. Step S2.2: Lightweight processing: Perform channel pruning and INT8 quantization on the modified network model to reduce the number of model parameters and computational cost; Step S2.3: Training the modified network model: Using the tiered state labeling dataset, a customized training strategy is adopted to train the lightweight network model; Step S2.4: Verify the trained network model: Use the step-by-step missing detection mAP@0.5, false negative rate, and false positive rate as evaluation indicators to verify the accuracy of the trained network model. If the preset threshold is met, it is determined to be the TJ-YOLOv8 model.

4. The escalator step detection method of claim 3, wherein, In the step S2, the CCA module processes the input feature map X CCA ∈R C×H×W , and specifically includes: 1), Feature splitting and multi-directional pooling: Split the input feature map X CCA by channels, and perform horizontal pooling, vertical pooling, and global pooling on the features of the c -th channel respectively to obtain corresponding feature tensors, where 0 ≤ c < C, specifically: Horizontal pooling: sum and average along the width direction for each row with fixed height h to get the horizontal pooling feature tensor ​ ； in h The position index is in the height direction and 0 ≤ h <H, i The index is used for traversing in the width direction, and 0 ≤ i <W, x c ( h , i ) represents the height in the c-th channel. h ,width i Pixel feature value at; Vertical pooling: sum and average along the height direction of the columns with fixed width w to get the vertical pooling feature tensor ​ ； wherein w is a width direction position index and 0≤ w < W, j is a height direction traversal index and 0≤ j < H, x c j w is a pixel feature value at height j , width w in the cth channel;​​ Global directional pooling: Iterates through the entire feature space, sums the results, and takes the average to obtain the global directional pooling feature tensor. ； z c ( w , h ) for the first c th channel in the global pooling feature value at the width w , height h ; C is the total number of feature map channels, H is the total number of feature map height pixels, and W is the total number of feature map width pixels. 2) Global feature weight generation: This involves generating global pooling feature tensors. z c ( w , h The channels are compressed using a 1×1 convolution transform function F1, and then the global feature weights g are obtained through a non-linear activation function Sigmoid. c The formula is: ； 3) intermediate mapping feature map generation: the horizontal direction pooled feature tensor and the vertical direction pooled feature tensor are spliced in the spatial dimension, compressed in the channel by the 1x1 convolution transformation function F1, batch normalized and nonlinearly activated to obtain the intermediate mapping feature map f , the formula is: ； wherein, denotes a Concat concatenation operation in both directions, denotes a non-linear activation function, r denotes a channel compression rate; 4) Generation of horizontal and vertical feature weights: This involves mapping the intermediate feature map... f Split along the spatial dimension to obtain independent feature tensors. and Then, each is convolved with a 1×1 matrix. F h , F w This operation, combined with the Sigmoid activation function, yields feature weights in both the horizontal and vertical directions. g hor and g ver The formula is: ； ； 5) Weighted fusion to obtain CCA features: the global feature weight g c , horizontal feature weight g hor , vertical feature weight g ver Element-wise weighted fusion to obtain CCA module output feature map Y CCA ( w , h ), the formula is: ； wherein the input feature map dimension is channel number C x height H x width W, w is the current position in the width dimension, i represents the traversal index in the width direction, h is the current position in the height dimension, j represents the traversal index in the height direction.

5. The escalator step detection method integrating YOLO algorithm and digital twin technology according to claim 3, characterized in that, In the step S2, the RepCCA attention mechanism is a three-branch parallel structure, and a 3x3 convolution, a 1x1 convolution, and a residual connection are used in parallel to process the CCA module output features in the training stage, and the output formula is ; The inference stage merges the three branches into a single 3x3 convolution, and the output formula is ; wherein X RepCCA is the input feature map of the RepCCA module, W1 and W2 are convolution weights, b1 and b2 are convolution biases, is a convolution operation, f CCA , and f() is a CCA module mapping function.

6. The escalator step detection method of claim 1, wherein, In step S3, the real-time edge detection and abnormal metadata generation specifically include: Step S3.1: Model Deployment: Convert the TJ-YOLOv8 model to the ONNX / TensorRT lightweight format and deploy it to the edge computing node; Step S3.2: Real-time video stream input and detection inference: The edge computing node receives the pre-processed tiered video stream from the front end, performs format conversion, normalization, and dimension adjustment on the video frames, and obtains single-frame data for the adaptation model input. The TJ-YOLOv8 model is used for inference on a single frame of data, outputting the diagonal bounding box coordinates (x, y, y) of each ladder instance. n,1 ,y n,1 ), (x n,2 ,y n,2 ), object confidence and class probability distribution of normal / missing / occluded tiers; merging object confidence and class probability to calculate the class confidence score S for tiers of normal tiers. norm S norm =Object confidence score × Normal tier category probability; Step S3.3: Construction and Update of Ladder-like Chain Topology: Extract the coordinates P of the center point of the boundary box of each ladder step. center (x n ,y n )=((x n,1 +x n,2 ) / 2,(y n,1 +y n,2 ) / 2), calculate the relative displacement vector (Δx,Δy) of the center points of adjacent steps, and dynamically construct and update the step chain topology graph; The ladder-chain topological relationship is the set of coordinate differences of the center points of adjacent ladder steps in the x-axis and y-axis directions of the image plane within the same frame, Δx=x n -x n-1 Δy=y n -y n-1 , where (x n y n (x) represents the coordinates of the center point of the nth step. n-1 y n-1 () represents the coordinates of the center of the (n-1)th step; Step S3.4: Suspected missing step marker: set a confidence threshold T conf When S norm <T conf or the relative displacement vector of adjacent steps exceeds the normal range, mark the corresponding step as a suspected missing step; The rule for marking suspected missing steps is as follows: assign a unique ID to a suspected missing step, in the format of "escalator ID-timestamp-step number", and record the reason for marking, including insufficient confidence, broken topology chain, or both. Step S3.5: Abnormal Event Metadata Generation: Fusing the pixel coordinates of the ladder center point with the height data collected by the infrared ranging sensor, and converting the intrinsic and extrinsic parameters of the camera to obtain the three-dimensional spatial coordinates P in the world coordinate system of the digital twin. 3D (X word ,Y word Z word Generate abnormal event metadata containing the coordinates, detection timestamp, and topology chain break information; The metadata of the abnormal event is in JSON format, including the unique abnormal ID, escalator number, microsecond-level timestamp, step number, three-dimensional spatial coordinates, model detection results, topology chain information, marking reason, and synchronous sensor data fields.

7. The escalator step detection method of claim 6, wherein, In step S4, the digital twin model is dynamically driven bidirectionally, specifically including: Step S4.1: Construct a digital twin of the escalator, import the 3D model of the steps into the geometry layer, and embed the kinematic formula L of the steps into the kinematic layer. k (t)=L0+v×t+1 / 2a×t 2 The data layer stores the escalator's rated parameters and sensor calibration parameters; among them, L step (t) represents the displacement / position of the k-th step at time t, and L0 represents the initial position of the step. v The rated linear speed of the cascade. a To accelerate the cascade operation, t The current time; Step S4.2: Inject sensor data and abnormal event metadata into the digital twin using the Kafka stream processing engine. In the virtual model, normal steps are kept displayed, while steps suspected of being missing are hidden or deformed. Step S4.3: Calculate step theoretical height value H based on kinematic model theory (t), calculate deviation AH = H real (t) - H real (t) - H theory (t), correct sensor data to H cal (t) = H real (t) - AH; Step S4.4: Using the calibrated height data as a reference, correct the reference coordinates of the ladder chain topology, using the formula P. 3D-cal (X cal ,Y cal Z cal )= P 3D (X word ,Y word Z word )+(0,0,ΔH).

8. The escalator step detection method of claim 7, wherein, In step S5, the multimodal data fusion decision specifically includes: Step S5.1: Extracting three types of isomerism features: extracting the gradient height mutation variable F h (t) = |H cal (t) - H cal (t-1), the confidence attenuation variable F c (t) = S norm (t-1) - S norm (t) and the topological chain break length ; Step S5.2: Perform min-max normalization on the three types of features to obtain N. h (t), N c (t), N t (t), weighted fusion generates instantaneous decision signal S(t)=αN h (t)+βN c (t)+γN t (t), where α, β, and γ are weighting coefficients adjusted according to the actual application environment; Step S5.3: Calculate the sum of the 3 consecutive frame instantaneous decision signals as the cumulative decision signal , set a cumulative threshold T A , trigger a history backtracking when A(t) > T A ; Step S5.4: Query the operating parameters of the suspected abnormal step over the past 5 seconds using the digital twin, and calculate the historical backtracking evaluation value at time t. Set the backtracking threshold T R When S(t) > 0.5 and A(t) > 1.5 and R(t) > T R When the anomaly is determined to persist, then... P ( tg ) represents the quantized value of the single-frame anomaly feature corresponding to the ladder level at time tg, which is obtained by normalizing the instantaneous decision signal S(ti), and g=0,1,2,3,4.

9. The escalator step detection method integrating YOLO algorithm and digital twin technology according to claim 1, characterized in that, In step S6, the millisecond-level alarm feedback and model self-update specifically include: Step S6.1: If the abnormality is determined to persist, an emergency braking signal is sent through the Profinet / EtherCAT industrial bus within 50ms, and the three-dimensional spatial coordinates of the suspected missing step are highlighted on the human-machine interface. Step S6.2: Package the abnormal video clips, sensor time-series data, decision signals, and judgment labels into a training sample set; Step S6.3: Based on the federated learning framework, freeze the backbone network of the TJ-YOLOv8 model, and only fine-tune the neck network and EL-Head detection head. The fine-tuning parameters are learning rate 0.001, batch size 16, and training epochs 10. After completion, distribute the updated model weights to the edge nodes. Step S6.4: The digital twin generates a simulation fault case library based on abnormal metadata and optimizes the judgment threshold of the topology chain break prediction algorithm.

Images