Escalator pedestrian falling detection method based on CEF-YOLOv10 and attitude time sequence modeling

Abstract

The invention discloses an escalator pedestrian falling detection method based on CEF-YOLOv10 and attitude time sequence modeling, and belongs to the field of computer vision pedestrian real-time behavior detection. According to the method, an improved model CEF-YOLOv10 is constructed, and collaborative modeling of local convolution and image overall perception is realized by combining local window attention and grouping attention mechanisms. A bidirectional feature fusion and context information enhancement strategy is adopted to construct a context enhanced bidirectional feature pyramid network and a dynamic background suppression and hierarchical feature decoupling module, and an improved Openpose attitude estimation model is adopted to perform attitude estimation on a confidence frame region image. And a long-short-term memory network model is introduced to perform tumble behavior reasoning and judgment, so that real-time and reliable tumble detection and intelligent early warning are realized. The method has the advantages of being high in spatial-temporal feature expression ability, flexible in deployment, rapid in response and the like, and is suitable for setting an intelligent security system in a public environment of the escalator.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer vision real-time pedestrian behavior detection, and particularly relates to an escalator pedestrian falling detection method based on CEF-YOLOv10 and posture time sequence modeling. BACKGROUND

[0002] As an important part of modern urban infrastructure, escalators have been widely used in public places such as shopping malls, subway stations, and airports. However, while escalators bring convenience, they also pose certain safety risks, especially with the frequent occurrence of pedestrian falling incidents. According to relevant data, escalator accidents account for more than 50% of passenger injury incidents in subway stations.

[0003] Traditional escalator safety monitoring mainly relies on safety personnel inspection and real-time observation of video monitoring images, which not only requires a large number of manpower, but also has problems such as low efficiency, slow response, and fatigue. In recent years, video analysis based on computer vision has gradually become a research hotspot. It realizes real-time detection and alarm of abnormal events by automatically analyzing the behavior of pedestrians in video images. With the rapid development of deep learning technology in the field of visual detection, pedestrian falling detection based on deep learning has been widely studied, which effectively improves the detection accuracy in complex scenes by automatically learning the features of pedestrian falling.

[0004] However, in the escalator scene, pedestrian falling detection still faces many challenges. First, the escalator environment is complex, and factors such as light changes, background interference, and pedestrian mutual occlusion can affect detection efficiency. Second, the pedestrian falling behavior is sudden and diverse, and how to accurately distinguish between normal walking and falling behavior is the key to system design. In addition, the system also needs to have real-time performance, which can quickly complete detection and alarm to ensure timely response to emergencies. Hou Ying et al. published an article titled "Escalator passenger safety detection algorithm based on region of interest YOLO-BFROI" in Computer Engineering and Applications (2025, 61(06): 84-95), which proposed an improved YOLOv8 lightweight detection method based on the region of interest. The method introduces BiFormer_ROI attention mechanism and SPD-Conv module in the backbone network to enhance the small target feature extraction capability, and combines GhostSlimPAFPN lightweight structure and PIoU v2 loss function to realize high-precision real-time detection of escalator passenger falling behavior, effectively reducing the problem of missed detection of small targets at a distance and false detection caused by background interference. However, the algorithm only relies on the YOLO model for single-frame detection, which is difficult to capture the complete time sequence features of the falling behavior, resulting in insufficient ability to distinguish dynamic processes.

[0005] In their paper "Monitoring the Safety Operations of Escalators Using a Multi-Level Fusion Network," published at the 2024 IEEE 17th International Conference on Signal Processing (ICSP 2024, 509-513), S. Zhao et al. proposed a multi-level fusion network method. This method integrates optical flow features, depth information, and motion trends of key human points to achieve real-time monitoring of escalator operation status, abnormal passenger behavior, and luggage falling events. However, in escalator environments with significant reflections or shadows, optical flow features generate substantial noise, leading to unstable motion estimation. Summary of the Invention

[0006] This invention addresses the problems of existing technologies by providing a pedestrian fall detection method for escalators based on CEF-YOLOv10 and attitude temporal modeling. This method solves the problem that existing escalator pedestrian fall detection methods rely solely on models such as YOLO for single-frame detection, resulting in limited discriminative ability during dynamic processes. Furthermore, it addresses the issue that factors such as high pedestrian traffic increase the difficulty of pedestrian tracking, thus limiting the improvement of detection performance.

[0007] To address the above technical problems, this invention provides the following technical solution: a method for detecting pedestrian falls on escalators based on CEF-YOLOv10 and attitude temporal modeling, comprising the following steps:

[0008] S1. Construct the CEF-YOLOv10 neural network for detecting pedestrian falls on escalators. Specifically, improve the C2f module of the YOLOv10 network with a cascaded group attention mechanism, construct the C2f_CGA module, introduce window attention mechanism and group attention mechanism to achieve the collaborative work of local convolution and global attention, and adopt a bidirectional feature fusion strategy.

[0009] The feature fusion method of YOLOv10 network is improved by constructing CEBiFPN network, which performs adaptive pooling and weighted fusion on local features and global context information to improve the network's feature extraction.

[0010] Input pedestrian video sequences into the CEF-YOLOv10 escalator pedestrian fall detection neural network, output pedestrian detection results and region localization, and preliminarily determine whether a pedestrian has fallen through image frame recognition. If so, proceed to step S2; otherwise, proceed to step S3.

[0011] S2, construct an ROI cache module to perform picture rollback, specifically: cache the confidence box region of the last N frames detected by the escalator pedestrian fall detection neural network CEF-YOLOv10; picture rollback extracts 2N frames of confidence box region images before and after the trigger time to cover the entire fall process; then perform step S4;

[0012] S3, use the ByteTrack algorithm to associate the detected pedestrians in time and space, specifically: design a trajectory management algorithm to associate the detection box across frames, use low-confidence detection boxes to complete the trajectory, realize multi-target tracking, and use a threshold-based state transition algorithm to model the state transition of pedestrian speed and confidence box height features; then perform threshold judgment of speed and baseline value, and if it is lower than the threshold, execute step S2, and if it is higher than the threshold, execute step S3;

[0013] S4, construct a dynamic background suppression module DBS and a hierarchical feature decoupling module DHF to improve the OpenPose pose estimation model, and use the OpenPose pose estimation model to extract the joint nodes of the detected pedestrians for the confidence box region images of the ROI cache module;

[0014] S5, construct the time sequence features of the joint node coordinates extracted by the OpenPose pose estimation model, input the time sequence features into the LSTM model for recursive inference to determine whether the pedestrian has fallen, and if so, perform early warning, otherwise return to step S3.

[0015] Further, in the aforementioned step S1, the window attention mechanism dynamically divides the input feature map into multiple local windows, and independently calculates the attention weight inside the window. When the size of the input feature map is smaller than the size of the window , the module directly applies a self-attention mechanism to the feature map as follows:

[0016] ,

[0017] wherein, represents the target position that needs to be calculated for attention, which is used to actively query the information of other positions; represents the identification information of all positions, which is used for matching to calculate the similarity; represents the actual feature content of each position, which is weighted and aggregated to generate the output feature according to the attention weight. , , and are linear mappings of the input feature map

[0018] When the input feature map size is greater than the window size, the input feature map is first padded and divided into multiple local windows with a size of Then, the local self-attention calculation is performed on each local window, and the results are combined to balance the calculation efficiency and global perception ability.

[0019] The grouping attention mechanism divides the channel dimension of the input feature into multiple subspaces, and each subspace independently calculates the attention weight. The features are gradually refined by using a cascaded depth separable convolution as shown in formula (2):

[0020] ,

[0021] wherein, is the processed output feature, is a depth separable convolution operation, is the input feature of the i-th group channel, is the number of channel groups. Further, in the aforementioned step S1, the CEBiFPN network is specifically a context-enhanced bidirectional feature pyramid network, which only uses a single set of weight coefficients to uniformly evaluate the global contribution of each level of feature. The weighting process of feature fusion is shown in formula (3):

[0022]

[0023] , wherein,

[0024] is the fused feature, is the i-th layer input feature, and the weight is obtained by adaptive weighting according to the global feature contribution. The fused feature map is refined by the context enhancement module CE to eliminate the distribution difference between different levels of features. First, 1x1 convolution and 3x3 convolution are used to capture local spatial information, and then global average pooling and channel attention mechanism are used to generate context weights to achieve adaptive reconstruction of the fused feature as shown in formula (4):

[0025]

[0026] , wherein,

[0027] is the enhanced feature obtained after weighting, represents a Sigmoid activation function, represents a fully connected layer, GAP represents a global average pooling, represents a convolution. ​​

[0028] Further, in the aforementioned step S3, the trajectory management algorithm completes the trajectory, specifically:

[0029] CEF-YOLOv10 output based on neural network confidence box original score , through function mapping to get the confidence , when , the detection box is divided into high confidence; when , it is classified as low confidence;

[0030] High confidence detection results are first used to match with historical trajectories, and then Kalman filter is used to predict the trajectory state, and the target state vector is set as:

[0031] ,

[0032] Among them, represents the state vector of the target at time , and are the position coordinates of the target in the image coordinate system, and are the velocity components of the target in the x and y directions, and the state prediction process is shown in formula (6):

[0033] ,

[0034] Among them, is the state obtained by predicting the state at time at time , is the state transition matrix, is the process noise; using the predicted state, the similarity measure between the predicted bounding box and the detection box is constructed, and the intersection over union is used as the measurement index, which is defined as formula (7):

[0035] ,

[0036] Among them and are the predicted bounding box and the detection bounding box respectively. The value between all trajectories and detection boxes is composed into a matrix , which is defined as formula (8):

[0037] ,

[0038] Among them, For the first The trajectory prediction box and the first IoU value between each detection box;

[0039] Solving the optimal matching scheme using the Hungarian algorithm As shown in formula (9):

[0040] ,

[0041] in, To match the scheme, for trajectories that do not match at this stage, the confidence requirement for the detection box is relaxed, and the detection box is recalculated using a lower confidence level. This is to obtain additional matching opportunities for secondary association.

[0042] Furthermore, in step S3 above, a threshold-optimized state transition algorithm is used to model the state transition of pedestrian speed and confidence box height features. Specifically, the center point coordinates of each pedestrian confidence box are extracted from the video frame sequence. and height The speed calculation formula is shown in formula (10):

[0043] ,

[0044] in, Indicates the first Frame rate, This indicates the position of the pedestrian's center point in the current frame, which is the time interval between adjacent frames;

[0045] The median of the velocity and altitude sequences of the most recent P-frames is used as the baseline velocity of the pedestrian at the current moment. and reference height As shown in formula (11):

[0046] ,

[0047] If, within an L-frame, a pedestrian's speed drops below 70% of the baseline speed or their height drops below 60% of the baseline height, an anomaly count is initiated. If either condition persists for an L-frame, the pedestrian is deemed to have experienced a suspected fall, and an alarm is triggered.

[0048] Furthermore, in step S4 above, the dynamic background suppression module DBS generates a weight matrix with the same resolution as the input feature map to indicate that the spatial location belongs to the background region that needs to be suppressed. Specifically, this includes:

[0049] A gating mechanism is used to reconstruct the feature map using weighted weighting. Let the input feature map be... with spatial dimension , and channel number ; the single-channel background mask generated by the attention module is .

[0050] The gating mechanism dynamically weights each spatial position of the feature map by introducing the mask as a learnable control factor, achieving suppression of background region features and effective preservation of foreground information. The final output feature map is calculated as shown in equation (12):

[0051]

[0052] wherein, represents an element-wise multiplication operation, is a background residual factor used to retain a small amount of information while suppressing the background to enhance the robustness of the network.

[0053] Further, in the aforementioned step S4, the hierarchical feature decoupling module DHF decouples human posture features from the microscopic deformation and macroscopic structure levels respectively through a double-path feature separation method, forming a complement in terms of details and structure. It includes a multi-branch feature enhancement module that fuses a high-frequency detail path, a semantic structure path, and an adaptive fusion layer, which are respectively used to extract edge contours, action semantics, and dynamically fused features,

[0054] The high-frequency detail path adopts a dilated convolution with a dilation rate to enhance the response of the edge region, normalize the local features, and improve the model's ability to perceive changes in detailed contours such as limb extension and arm opening. The normalization operation is shown in equation (13):

[0055]

[0056] wherein, is the channel feature of the feature map, is the normalized channel feature, , are the mean and variance of the channel, respectively, is a stabilizing term;

[0057] The semantic structure path uses cascaded asymmetric convolutions to capture transverse limb movement features and axial tilting of the torso. It also uses group normalization to alleviate the problem of normalization failure in small batch training and enhance the model's ability to understand the structure at different posture stages before and after falling.

[0058] Further, the aforementioned step S5 includes the following sub-steps:

[0059] S5.1, normalizing the torso length of the pedestrian, translating all joint coordinates with the midpoint of the hip as the origin, and scaling with the torso length as the unit of measure, reducing the error caused by the deformation of the bounding box;

[0060] S5.2, completing the missing coordinate information: first, using the PCHIP algorithm, based on the coordinate information of all detected frames, estimating the missing joint positions, using bone length constraints to ensure that the estimated positions conform to the geometric proportions of the human body skeletal structure; finally, using the Savitzky-Golay smoothing filter to smooth the joint trajectory in the time series to suppress noise and maintain the continuity of the motion curve;

[0061] S5.3, recording the coordinate information of the processed head, hip and knee joints, and constructing the time sequence feature with it, and inputting the time sequence feature into the LSTM model for recursive reasoning.

[0062] Further, in the aforementioned step S5.1, the torso length of the pedestrian is normalized, specifically including:

[0063] The OpenPose pose estimation model is used to extract the joint points of the detected pedestrian, extract the head, hip midpoint and right knee joint information, and normalize the torso length of the pedestrian. The coordinates of the head joint and the hip midpoint joint are extracted from the human joint information, and the Euclidean distance between the two is calculated as the torso length, as shown in formula (14):

[0064]

[0065] wherein , , , is the hip center joint and the head joint coordinate in the frame, represents the size of the human body in the image;

[0066] The coordinates of any joint are , and the normalized coordinates are represented as formula (15):

[0067]

[0068] In the normalization process, all joint coordinates are translated with the hip center joint as the origin, and scaled with the torso length as the unit of measure.

[0069] Further, the aforementioned step S5.2 of completing the missing coordinate information specifically includes the following sub-steps:

[0070] S5.2.1, performing integrity check on each frame of human body joint coordinate sequence output by the pose estimation algorithm, if the confidence of a joint is lower than a preset threshold, determining that the joint is missing in the frame, and recording the missing frame number;

[0071] S5.2.2, for each missing joint , respectively, and and are completed by PCHIP interpolation on the time axis, given the observation values , , , at time , , , , the interpolation polynomial of PCHIP in the interval is shown in formula (16):

[0072]

[0073] wherein is the time length of the current interpolation interval, is the interval normalization parameter, , , , is the cubic Hermite basis function; the slope adopts the monotonicity protection formula, as shown in formula (17):

[0074]

[0075] wherein, represents the time step of the th interpolation interval;

[0076] S5.2.3, the joint coordinate obtained after interpolation may cause bone proportion distortion, therefore, bone length constraint is adopted, and the reference bone length from a non-missing frame is:

[0077]

[0078] wherein is the reference bone length, and are the connected joint pairs, for the joint in the set of non-missing frames, and for the joint at time t and joint two-dimensional coordinates;

[0079] S5.2.4, if the relative error between the interpolated bone length and the reference length exceeds a set threshold, the joint position is scaled in proportion to make the corrected bone length meet the human bone proportion relationship, and the proportional correction formula is shown as formula (19):

[0080]

[0081] wherein, is the corrected joint coordinate, and is the interpolated and joint coordinate, is the interpolated bone length;

[0082] S5.2.5, the time sequence joint coordinates corrected by the bone length constraint are smoothed by using a Savitzky-Golay smoothing filter, and the processed joint coordinates are updated to the human body posture data of the frame as the final completion result of the missing joint.

[0083] Compared with the prior art, the beneficial technical effects of the above technical solutions of the present application are as follows:

[0084] The method of the application is suitable for a safety monitoring system in indoor and outdoor public places. The method optimizes the existing YOLOv10 neural network structure and constructs an improved model CEF-YOLOv10. By combining local window attention and grouping attention mechanism, a C2f_CGA module is constructed to realize the collaborative modeling of local convolution and overall image perception. A context-enhanced bidirectional feature pyramid network (CEBiFPN) is constructed by using bidirectional feature fusion and context information enhancement strategy to enhance the recognition ability of pedestrians in escalator scenes. In view of the problem that the large number of people in the escalator scene leads to difficulty in tracking pedestrians, the cross-frame trajectory association is carried out by combining pedestrian detection and ByteTrack algorithm. By completing the trajectory with low confidence boxes and using a trajectory management algorithm, stable tracking of the motion state of pedestrians is realized. A detection region cache module (Region of Interest, ROI) is designed to store the image information of the confidence box region of the last N frames. When the CEF-YOLOv10 model identifies a fall event or detects an abnormal motion trajectory through the state transition algorithm, the ROI cache module extracts the confidence box region image of 2N frames before and after the triggering time to cover the entire fall process. In view of the feature interference caused by the similarity of the clothes of pedestrians and the color of the escalator, the following strategies are adopted. First, a dynamic background suppression and hierarchical feature decoupling module is constructed, and an inflation convolution and branch structure is used to make the Openpose network have large-scale receptive field and local fine-grained feature extraction capability. Then, the improved Openpose pose estimation model is used to estimate the pose of the confidence box region image, and the head (node 0), hip midpoint (node 8) and right knee (node 10) joint information is extracted. The extracted joint information is normalized in terms of torso length to reduce the error caused by confidence box deformation. Finally, for the still missing joints, the Piecewise Cubic Hermite Interpolating Polynomial (PCHIP) and Savitzky-Golay filter are used to complete the joints to construct a time sequence feature expression of complete joint coordinate information. The Long Short-term Memory Networks (LSTM) model is introduced to infer and judge the fall behavior, realizing real-time and reliable fall detection and intelligent early warning. The application has the advantages of strong spatio-temporal feature expression ability, flexible deployment and rapid response, and is suitable for intelligent security systems in complex environments such as public transportation and shopping malls with escalators. BRIEF DESCRIPTION OF DRAWINGS

[0085] Figure 1 CEF-YOLOv10 network structure diagram.

[0086] Figure 2Structure diagram of improved C2f_CGA.

[0087] Figure 3 Structure diagram of CGA.

[0088] Figure 4 Structure diagram of improved CEBiFPN.

[0089] Figure 5 Structure diagram of CEBiFPN in the improved CEBiFPN.

[0090] Figure 6 Structure diagram of improved OpenPose.

[0091] Figure 7 Structure diagram of improved DBS and structure diagram of DHF.

[0092] Figure 8 Contrast diagram of image frames before and after interpolation.

[0093] Figure 9 Falling case LSTM model detection result diagram.

[0094] Figure 10 Flow chart of the detection method of the present application. DETAILED DESCRIPTION

[0095] In order to better understand the technical content of the present application, specific embodiments are described below with reference to the accompanying drawings.

[0096] Aspects of the present application are described in this disclosure by reference to the drawings, in which various illustrative embodiments are shown. The embodiments of the present application are not limited to the illustrative embodiments depicted in the drawings. It should be understood that the inventive concepts and embodiments disclosed herein can be implemented in any of the variety of ways described above and in any of the alternative ways described below, and that the inventive concepts and embodiments disclosed herein are not limited to any particular implementation. Further, some aspects of the present application can be used alone, or in any combination with other aspects of the present application.

[0097] As Figure 10 shown, the present application provides an escalator pedestrian falling detection method based on CEF-YOLOv10 and posture time sequence modeling, characterized in that it comprises the following steps:

[0098] S1, build an escalator pedestrian falling detection neural network CEF-YOLOv10, specifically: improve the C2f module of the YOLOv10 network with a cascade group attention mechanism, build a C2f_CGA module, introduce a local window attention mechanism (LWA) and a group attention mechanism, realize the cooperative work of local convolution and global attention, and adopt a bidirectional feature fusion strategy;

[0099] Improve the feature fusion method of the YOLOv10 network, build a CEBiFPN network, perform adaptive pooling processing on local features and global context information, and weighted fusion, to improve the network's feature extraction;

[0100] Input the pedestrian video sequence into the escalator pedestrian falling detection neural network CEF-YOLOv10, output the pedestrian detection result and region positioning, and preliminarily determine whether the pedestrian falls through image frame recognition, if yes, execute step S2, otherwise execute step S3;

[0101] S2, build an ROI cache module for picture backtracking, specifically: cache the confidence box region of the last N frames detected by the escalator pedestrian falling detection neural network CEF-YOLOv10; picture backtracking extracts 2N frames of confidence box region images before and after the trigger time to cover the entire falling process; then execute step S4;

[0102] S3, use the ByteTrack algorithm to associate the detected pedestrians in time and space, specifically: design a trajectory management algorithm to associate the detection box across frames, use low confidence detection boxes to complete the trajectory, realize multi-target tracking, and use a threshold-based state transition algorithm to model the state transition of pedestrian speed and confidence box height features; then perform threshold judgment on speed and baseline value, if below the threshold, execute step S2, if above the threshold, execute step S3;

[0103] S4, build a dynamic background suppression module DBS and a hierarchical feature decoupling module DHF, and improve the OpenPose pose estimation model, for the confidence box region image of the ROI cache module, use the OpenPose pose estimation model to extract the joints of the detected pedestrians;

[0104] S5, construct the time sequence features of the joint coordinates extracted by the OpenPose pose estimation model, input the time sequence features into the LSTM model for recursive reasoning to determine whether the pedestrian has fallen, if yes, perform early warning, otherwise return to execute step S3.

[0105] As Figure 1As shown, based on the existing YOLOv10 neural network, an escalator pedestrian falling detection neural network CEF-YOLOv10 is constructed, like Figure 2 In step S1, a window attention mechanism (Local Window Attention, LWA) is introduced after the convolution layer in the C2f module, and a C2f_CGA module is constructed.

[0106] The core of the LWA module is a cascaded group attention mechanism (Cascaded Group Attention, CGA), which includes two parts: window division strategy and grouped cascaded attention mechanism.

[0107] The LWA module dynamically divides the input feature map into multiple local windows, and independently calculates the attention weight inside the window to effectively avoid the quadratic complexity problem of traditional global attention mechanism. When the size of the input feature map is smaller than the size of the window , the module directly applies self-attention mechanism to the feature map. The core principle of self-attention mechanism is to give the model the ability to dynamically adjust its calculation strategy according to the local characteristics of the input data, and to realize resource on-demand allocation through a learnable decision function. As shown in equation (1):

[0108]

[0109] wherein represents the target position that needs to calculate attention, which is used to actively query the information of other positions; represents the identification information of all positions, which is used for matching to calculate similarity; represents the actual feature content of each position, which is weighted and aggregated to generate output features according to the attention weight. , , are linear mappings of the input feature map , and d is the feature dimension.

[0110] When the size of the input feature map is greater than the size of the window, it is first divided into multiple local windows with a size of by appropriate padding, and then the above local self-attention calculation is performed respectively, and the results are recombined to balance the calculation efficiency and global perception ability.

[0111] In addition, as Figure 3As shown, the CGA module uses a grouped attention mechanism to divide the channel dimension of the input features into multiple subspaces. Each subspace independently calculates the attention weights, and the features are gradually refined using cascaded depthwise separable convolutions, as shown in formula (2).

[0112]

[0113] in, The processed output features For depthwise separable convolution operations, For the first Input characteristics of group channels, The number of channel groups is specified. This "local-global" progressive feature fusion design can capture local detailed features through convolution and model broader cross-regional semantic dependencies through attention mechanisms, thereby improving the network's ability to express multi-scale semantic information.

[0114] like Figure 4 As shown, the CEBiFPN network is specifically a context-enhanced bidirectional feature pyramid network. It abandons the local weighting strategy of layer-by-layer node setting in traditional bidirectional feature pyramid networks, and instead uses only a single set of weight coefficients to uniformly evaluate the global contribution of features at each level. The number of weights is reduced by more than 50% compared to traditional bidirectional feature pyramid networks. The weighting process of feature fusion is shown in formula (3):

[0115] ,

[0116] in, The characteristics after fusion For the first Layer input features, weights It is obtained through adaptive weighting based on the contribution of global features;

[0117] like Figure 5 As shown, to further eliminate the distribution differences between features at different levels, the fused feature map is finely adjusted through the context enhancement module (CE). First, 1×1 convolution and 3×3 convolution are used to capture local spatial information. Then, context weights are generated through global average pooling and channel attention mechanisms to achieve adaptive reconstruction of the fused features, as shown in formula (4):

[0118] ,

[0119] in, The enhanced features obtained after weighting, This represents the Sigmoid activation function. This indicates a fully connected layer, and GAP indicates global average pooling. This represents convolution. With this design, the feature map effectively preserves local details while also possessing stronger global semantic awareness and expressive capabilities.

[0120] In step S3, the trajectory is completed using a trajectory management algorithm, specifically:

[0121] The CEF-YOLOv10 model was used for pedestrian detection and region localization. The ByteTrack algorithm was employed to assign different IDs to the detected pedestrians for continuous tracking. The raw scores of the confidence boxes output by the CEF-YOLOv10 neural network are presented. ,go through Confidence is obtained by function mapping ,when When, the detection box is classified as high confidence; when If so, it is classified as low confidence.

[0122] The high-confidence detection results are first used to match historical trajectories, and then a Kalman filter is used to predict the trajectory state. Let the target state vector be:

[0123] ,

[0124] in, Indicates the target at time. The state vector, and Let the coordinates be the position coordinates of the target in the image coordinate system. and Let x be the velocity components of the target in the x and y directions. Its state prediction process is shown in Equation (6):

[0125] ,

[0126] in, For time-based For time The state obtained by predicting the state. Here is the state transition matrix. This is process noise. Using the predicted state, a similarity metric between the predicted bounding box and the detection box is constructed, with the intersection-union ratio (IUU) used as the metric. Its definition is shown in formula (7):

[0127] ,

[0128] in and respectively, are the predicted bounding box and the detected bounding box. The matrix is composed of the values between all trajectories and detection boxes

[0129]

[0130]

[0131]

[0132]

[0133]

[0134] In step S3, a state transition algorithm based on threshold optimization is used to model the state transition of pedestrian speed and confidence box height. Specifically, the center point coordinates of each pedestrian confidence box and the height are extracted from the video frame sequence, and the speed calculation formula is as shown in formula (10):

[0135]

[0136]

[0137] The median of the speed sequence and the height sequence of the nearest P frames are taken as the baseline speed and the reference height of the pedestrian at the current time, as shown in formula (11):

[0138]

[0139] If the pedestrian's speed drops below 70% of the baseline speed or the height drops below 60% of the reference height within L frames, the abnormal count is started. If either condition lasts for L frames, it is determined that the pedestrian has a suspected fall event and an alarm is triggered.​​​​​​​​​​​​​​​​​​​

[0140] Step S4, as follows Figure 6 As shown, to address the issues of missed detections and false detections caused by the similarity between pedestrian clothing and escalator colors, the OpenPose pose estimation model was improved by designing a dynamic background suppression module DBS and a hierarchical feature decoupling module DHF.

[0141] like Figure 7 As shown, the Dynamic Background Suppression (DBS) module generates a weight matrix with the same resolution as the input feature map to indicate spatial locations belonging to the background regions that need to be suppressed. Specifically, it includes:

[0142] A gating mechanism is used to reconstruct the feature map using weighted weighting. Let the input feature map be... Its spatial dimension is The number of channels is The single-channel background mask generated by the attention module is .

[0143] The gating mechanism introduces this mask as a learnable control factor to dynamically weight each spatial location of the feature map, thereby suppressing background features and effectively preserving foreground information, ultimately outputting a feature map. The calculation is shown in equation (12):

[0144]

[0145] in, This represents element-wise multiplication. This is a background residual factor used to retain a small amount of information while suppressing the background, thereby enhancing the robustness of the network.

[0146] The hierarchical feature decoupling module (DHF) employs a dual-path design, including a high-frequency detail path and a semantic structure path. The high-frequency detail path uses a hole ratio control. of Hollow convolution, its effective receptive field is equivalent to Convolution provides stronger local perception capabilities. To enhance the response in edge regions, local features are standardized to improve the model's ability to perceive subtle contour changes such as limb abduction and arm opening. The normalization process is shown in formula (13):

[0147]

[0148] in, For the feature map of the th Channel characteristics, For the normalized first Channel characteristics, , These are the mean and variance of the channel, respectively. It is a stable term.

[0149] The semantic structure pathway employs cascaded asymmetric convolutions to capture lateral limb movement features and trunk axial tilt, combined with group normalization to alleviate the problem of normalization failure under small-batch training, thereby enhancing the model's ability to understand the structure of different posture stages before, during and after a fall.

[0150] Step S5 includes the following sub-steps:

[0151] S5.1. Normalize the torso length of the pedestrian. All key node coordinates are translated with the midpoint of the hip as the origin and scaled using the torso length as the unit to reduce the error caused by the deformation of the confidence box. Specifically, the improved OpenPose pose estimation model is used to extract the key points of the detected pedestrian, extracting the head, midpoint of the hip, and right knee joint information. The torso length of the pedestrian is normalized, and the coordinates of the head joint and midpoint of the hip joint are extracted from the human body key point information. The Euclidean distance between the two is calculated as the torso length. The calculation formula is shown in (14):

[0152]

[0153] in , , , In the first Midpoint of hip in frame and head coordinate, This indicates the size of the human body in the image.

[0154] For any joint coordinates Its normalized coordinate representation is shown in formula (15):

[0155]

[0156] During the normalization process, all node coordinates are scaled with the midpoint of the hip as the origin and the length of the torso as the unit of measurement.

[0157] S5.2, Completing the missing coordinate information: first, based on the coordinate information of all detected frames, the missing joint position is estimated by using the PCHIP algorithm, and the bone length constraint is used to ensure that the estimated position conforms to the geometric proportion of the human body bone structure; finally, the Savitzky-Golay smoothing filter is used to smooth the joint trajectory in the time sequence to suppress noise and maintain the continuity of the motion curve, including the following sub-steps:

[0158] S5.2.1, Integrity check is performed on each frame of human joint coordinate sequence output by the pose estimation algorithm, if the confidence of a joint is lower than a preset threshold, it is determined that the joint is missing in the frame, and the missing frame number is recorded;

[0159] S5.2.2, as shown in Figure 8 , for the missing joints in the 6th frame, 13th frame, 14th frame and 15th frame in the falling event , PCHIP interpolation completion is performed on and in the time axis, the observation values at time , , , are known , , , , the interpolation polynomial of PCHIP in the interval is shown in formula (16):

[0160]

[0161] wherein is the time length of the current interpolation interval, is the interval normalization parameter, , , , is the cubic Hermite basis function; the slope uses the monotonicity protection formula, as shown in formula (17):

[0162]

[0163] wherein, represents the time step of the th interpolation interval, this method avoids the oscillation of the traditional cubic spline, and can keep the shape monotonic when the joint motion direction changes abruptly;

[0164] S5.2.3, the joint coordinates obtained after interpolation may cause bone proportion distortion, therefore bone length constraint is adopted, and the reference bone length from the non-missing frame is:

[0165]

[0166] wherein is the reference bone length, and is the connected joint pair, is the set of the joint in the non-missing frame, and is the joint two-dimensional coordinate at t moment; and

[0167] S5.2.4, if the relative error of the interpolated bone length and the reference length exceeds the set threshold, the joint position is scaled to make the corrected bone length meet the human bone proportion relationship, and the proportional correction formula is shown in formula (19):

[0168]

[0169] wherein, is the corrected joint coordinate, and is the interpolated joint coordinate, and is the interpolated bone length.

[0170] S5.2.5, for the time sequence joint coordinates corrected by the bone length constraint, Savitzky-Golay smoothing filter is adopted for smoothing processing, and the processed joint coordinates are updated to the human posture data of the frame as the final completion result of the missing joint.

[0171] S5.3, as shown in formula (20), the coordinate information of the processed head, hip and knee three joints is recorded, and the time sequence characteristics are constructed, and the time sequence characteristics are input into the LSTM model for recursive inference. Figure 9

[0172] Although the present application has been described as above with the preferred embodiments, it is not intended to limit the present application. Those skilled in the art can make various modifications and decorations without departing from the spirit and scope of the present application. Therefore, the protection scope of the present application shall be subject to the definition of the claims.​​​​

Claims

1. A method for escalator passenger fall detection based on CEF-YOLOv10 and pose timing modeling, characterized in that, Comprising the following steps: S1, construct escalator pedestrian fall detection neural network CEF-YOLOv10, specifically: improve the C2f module of YOLOv10 network with cascaded group attention mechanism, construct C2f_CGA module, introduce window attention mechanism and grouping attention mechanism, realize the cooperative work of local convolution and global attention, adopt bidirectional feature fusion strategy; Improve the feature fusion method of YOLOv10 network, construct CEBiFPN network, perform adaptive pooling processing on local features and global context information, and weighted fusion, to improve the network's feature extraction; Input the pedestrian video sequence into the escalator pedestrian fall detection neural network CEF-YOLOv10, output the pedestrian detection result and region positioning, and preliminarily determine whether the pedestrian falls through image frame recognition, if yes, execute step S2, otherwise execute step S3; S2, build ROI cache module for picture backtracking, specifically: cache the confidence box region of the last N frames detected by the escalator pedestrian fall detection neural network CEF-YOLOv10; picture backtracking extracts 2N frames of confidence box region images before and after the trigger time to cover the entire fall process; then execute step S4; S3, use ByteTrack algorithm to associate the detected pedestrians in time and space, specifically: design a trajectory management algorithm to associate the detection box across frames, use low confidence detection box to complete the trajectory, realize multi-target tracking, and use threshold-based state transition algorithm to model the state transition of pedestrian speed and confidence box height features; then perform threshold judgment on speed and baseline value, if below the threshold, execute step S2, if above the threshold, execute step S3; S4, build dynamic background suppression module DBS and hierarchical feature decoupling module DHF, and improve the OpenPose pose estimation model, for the confidence box region image of the ROI cache module, use the OpenPose pose estimation model to extract the joint points of the detected pedestrians; S5, construct the time sequence features of the joint point coordinates extracted by the OpenPose pose estimation model, input the time sequence features into the LSTM model for recursive reasoning to determine whether the pedestrian has fallen, if yes, perform early warning, otherwise return to step S3.

2. The escalator fall detection method based on CEF-YOLOv10 and pose timing modeling according to claim 1, characterized in that, In step S1, the window attention mechanism dynamically divides the input feature map into multiple local windows, and independently calculates the attention weight inside the window. When the size of the input feature map is smaller than the size of the window , the module directly applies a self-attention mechanism to the feature map as follows: ， wherein, represents the target position that needs to be calculated attention at present, and is used for actively querying information of other positions; represents the identification information of all positions, and is used for being queried and matched to calculate similarity; represents the actual feature content of each position, and is weighted and aggregated according to the attention weight to generate an output feature; , , respectively are linear mappings of input feature maps , d is a feature dimension; When the input feature map size is greater than the window size, the input feature map is first filled with appropriate padding and divided into multiple local windows with a size of The local self-attention calculation is then performed on each local window, and the results are combined to balance the calculation efficiency and global perception ability. The grouping attention mechanism divides the channel dimension of the input features into multiple subspaces, each of which independently calculates the attention weight, and uses cascaded depth separable convolution to gradually refine the features as shown in formula (2): ， wherein, is the processed output feature, is a depthwise separable convolution operation, is the first input feature of the group of channels, is the number of channel groups.

3. The escalator fall detection method based on CEF-YOLOv10 and pose timing modeling according to claim 1, characterized in that, In step S1, the CEBiFPN network is a context-enhanced bidirectional feature pyramid network that only uses a single set of weight coefficients to uniformly evaluate the global contribution of each hierarchical feature; the weighted process of feature fusion is shown in formula (3): ， wherein, is the fused feature, is the first layer input feature, the weight is obtained by adaptive weighting according to the global feature contribution degree; The fused feature map is fine-tuned by a context enhancement module CE to eliminate the distribution difference between different level features. First, 1x1 convolution and 3x3 convolution are used to capture local spatial information, and then global average pooling and channel attention mechanism are used to generate context weights to realize adaptive reconstruction of the fused features, as shown in formula (4): ， wherein, is the weighted enhanced feature, denotes a Sigmoid activation function, denotes a fully connected layer, and GAP denotes a global average pooling, denotes a convolution.

4. The escalator fall detection method based on CEF-YOLOv10 and pose timing modeling according to claim 1, characterized in that, In step S3, the trajectory management algorithm completes the trajectory, specifically: The confidence box original score based on the neural network CEF-YOLOv10 output , is mapped to the confidence through function When , the detection box is divided into high confidence; when , it is classified as low confidence; The high-confidence detection result is first used to match with the historical trajectory, and then a Kalman filter is used to predict the trajectory state. The target state vector is set as: ， wherein denotes the state vector of the target at time and are the position coordinates of the target in the image coordinate system, and are the velocity components of the target in the x and y directions, the state prediction process of which is shown in equation (6):​ ， wherein, is based on the time the time state obtained by predicting the state, is a state transition matrix, is process noise; using the predicted state, a similarity measure between the predicted bounding box and the detection box is constructed, and the intersection over union is used as a measurement index, which is defined as shown in equation (7): ， wherein and are a predicted bounding box and a detected bounding box, respectively; all trajectories and bounding boxes are composed into a matrix of values as defined in equation (8): as defined in equation (8): ， in, For the first The trajectory prediction box and the first IoU value between each detection box; Solving the optimal matching scheme using the Hungarian algorithm As shown in equation (9): ， wherein, For matching scheme, for the track that is not matched at this stage, the confidence requirement of the detection box is relaxed, and low confidence is used to calculate again to obtain additional matching opportunities for secondary association.

5. The escalator fall detection method based on CEF-YOLOv10 and pose timing modeling according to claim 1, characterized in that, In step S3, a state transition algorithm based on threshold optimization is used to model the state transition of pedestrian speed and confidence box height. Specifically, the center point coordinates of each pedestrian confidence box are extracted from the video frame sequence and the height , and the speed calculation formula is shown in formula (10): ， wherein, represents the speed of the frame, represents the frame pedestrian center point position, the time interval of the adjacent frames; respectively taking the median of the recent P-frame speed sequence and the height sequence as the baseline speed of the pedestrian at the current time and the reference height as shown in equation (11): ， If the pedestrian's speed drops below 70% of the baseline speed or the height drops below 60% of the baseline height within L frames, the abnormal count is started, and if any condition lasts for L frames, it is determined that the pedestrian has a suspected falling event and triggers an alarm.

6. The escalator fall detection method based on CEF-YOLOv10 and pose timing modeling according to claim 1, characterized in that, In step S4, the dynamic background suppression module DBS generates a weight matrix with the same resolution as the input feature map to indicate the spatial position belonging to the background area that needs to be suppressed, which specifically includes: The input feature map is weighted and reconstructed by using a gating mechanism , the spatial dimension of which is , the number of channels is ; and the single-channel background mask generated by the attention module is ; The gating mechanism dynamically weights each spatial location of the feature map by introducing the mask as a learnable control factor, realizes the suppression of the background region feature and the effective reservation of the foreground information, and finally outputs the feature map is calculated as shown in equation (12): ， wherein, denotes an element-wise multiplication operation, is a background residual factor used to preserve a small amount of information while suppressing the background to enhance the robustness of the network.

7. The escalator fall detection method based on CEF-YOLOv10 and pose timing modeling according to claim 1, characterized in that, In step S4, the hierarchical feature decoupling module DHF decouples the human posture features from the microscopic deformation and macroscopic structure levels through a double-path feature separation method to form a complementary relationship in detail and structure. It includes a multi-branch feature enhancement module in the fusion high-frequency detail path, semantic structure path and adaptive fusion layer, which is used to extract edge contour, action semantics and dynamic fusion features, High-frequency detail passage adopts hollow rate of hollow convolution, to enhance the edge area response, standardize the local features, and improve the model's perception of limb extension, arm opening, and other detailed contour changes. The normalization operation is shown in formula (13): ， wherein, is the i-th channel feature of the feature map, is the i-th normalized channel feature, , are the mean and variance of the channel, respectively, is a stabilizing term;​​ The semantic structure path uses cascaded asymmetric convolution to capture transverse limb movement features and axial tilt of the torso, and group normalization to alleviate the problem of normalization failure under small batch training, thereby enhancing the model's ability to understand the structure at different posture stages before, during and after falling.

8. The escalator fall detection method based on CEF-YOLOv10 and pose timing modeling according to claim 1, characterized in that, Step S5 includes the following sub-steps: S5.1, normalize the length of the pedestrian's torso, translate all joint coordinates with the midpoint of the hip as the origin, and scale them with the length of the torso as the unit to reduce errors caused by confidence box deformation; S5.2, complete the missing coordinate information: first, use the PCHIP algorithm to estimate the missing joint positions based on the coordinate information of all detected frames, and use bone length constraints to ensure that the estimated positions conform to the geometric proportions of human bone structures; finally, use the Savitzky-Golay smoothing filter to smooth the joint trajectories in the time series to suppress noise and maintain the continuity of the motion curve; S5.3, record the coordinates of the processed head, hip and knee joints, and use them to construct a time series feature, which is input into the LSTM model for recursive reasoning.

9. The escalator fall detection method based on CEF-YOLOv10 and pose timing modeling according to claim 8, characterized in that, In step S5.1, the length of the pedestrian's torso is normalized, specifically including: The OpenPose pose estimation model is used to extract the joint information of the detected pedestrian, including the head, hip midpoint and right knee joint information. The coordinates of the head joint and hip midpoint joint are extracted from the human joint information, and the Euclidean distance between them is calculated as the torso length, as shown in formula (14): ， wherein , , , is the center of the hip joint in the first frame and the head joint coordinates, denotes the size of the human body in the image; The coordinates of any joint point whose normalized coordinate representation is given by equation (15):​ ， In the normalization process, all joint coordinates are taken as the origin of the hip center joint and scaled with the torso length as the unit of measurement.

10. The escalator fall detection method based on CEF-YOLOv10 and pose timing modeling according to claim 9, characterized in that, In step S5.2, the missing coordinate information is completed, specifically including the following sub-steps: S5.2.1, integrity check is performed on each frame of human body joint coordinate sequence output by the pose estimation algorithm, if the confidence of a joint is lower than a preset threshold, it is determined that the joint is missing in the frame, and the missing frame number is recorded; S5.2.2, for each missing link node , respectively, to and complete the PCHIP interpolation on the time axis, given the observations at times , , , , , , The PCHIP interpolating polynomial on the interval is given by equation (16):​ ， wherein is the time length of the current interpolation interval, is the interval normalization parameter, , , , is a cubic Hermite basis function; slope A monotonicity preserving formula is adopted, as shown in equation (17): ， wherein, denotes the time step of the th interpolation interval; S5.2.3, the joint coordinates obtained after interpolation may cause bone proportion distortion, therefore, bone length constraint is adopted, and the reference bone length from the non-missing frame is: ， wherein is a reference bone length, and is a connected joint pair, is a set of the joint in non-missing frames, and is a joint two-dimensional coordinate at time t and joint two-dimensional coordinate; S5.2.4, if the relative error between the interpolated bone length and the reference length exceeds the set threshold, the joint position is scaled in proportion to make the corrected bone length satisfy the human body bone proportion relationship, and the proportional correction formula is shown in formula (19): ， wherein, is the corrected node coordinates, and is the interpolated and node coordinates, is the interpolated bone length; S5.2.5, the time sequence joint coordinates corrected by the bone length constraint are smoothed by using the Savitzky-Golay smoothing filter, and the processed joint coordinates are updated to the human body pose data of the frame as the final completion result of the missing joint.

Images