Robust escalator passenger flow statistics system adaptive to different hardware scenarios

Abstract

This invention discloses a robust escalator passenger flow statistics system adaptable to different hardware and application scenarios. It adaptively selects the optimal detection and tracking models suitable for different hardware platforms and pedestrian movement information in various escalator scenarios based on the computing power of those platforms and the specific needs of the users. An improved tracking model enhances tracking performance, and software acceleration algorithms speed up the detection model. An improved quantization algorithm improves inference speed and reduces the number of model parameters. The system adaptively determines the location and size of the Region of Interest (ROI) for passenger flow statistics, counts the number of people passing through the ROI at the escalator entrance, and proposes algorithms for pedestrian detection deduplication and pedestrian count correction to improve the accuracy of passenger count statistics. This invention is applicable to hardware platforms with varying computing power and different escalator scenarios, and it improves model tracking performance and detection speed, exhibiting excellent robustness across different hardware and application scenarios.

Description

Technical Field

[0001] This invention relates to the technical field of escalator security, and in particular to a robust escalator passenger flow statistics system that is adaptable to different hardware and various scenarios. Background Technology

[0002] In modern society, with the continuous emergence of high-rise buildings, escalators have become an indispensable piece of equipment to facilitate people's access to different floors. Compared to elevators, they have the advantage of a larger passenger capacity and are widely used in various places, especially shopping malls, subways, and other high-traffic areas. The foot traffic in a shopping mall can reflect its operational success and has certain commercial value. Furthermore, foot traffic statistics at subway entrances and exits can effectively monitor public transportation and prevent safety issues. Therefore, passenger flow statistics in escalator scenarios are of great significance.

[0003] A commonly used method for passenger flow statistics is computer vision-based. This method detects and tracks passengers passing through escalator floors to obtain their location information and then performs passenger flow statistics. The combination of detection and tracking is a relatively accurate method. However, to track pedestrians, a fast detection speed is required to match the detected and tracked targets. If the detection speed is too slow, tracking will fail, resulting in ineffective acquisition of passenger location information and thus, the passenger flow statistics algorithm will not achieve its intended effect. In practical applications, the computing power of AI edge products and AI chips from different manufacturers varies. While high-performance AI hardware platforms can achieve better results, they are also relatively expensive. Furthermore, different application scenarios have different accuracy requirements for passenger flow statistics, necessitating the selection of different AI hardware platforms. Therefore, different detection or tracking algorithms must be selected for different hardware platforms, application scenarios, and task accuracy requirements. This often leads to the same algorithm performing poorly or even being unusable on different hardware platforms, posing a significant challenge to the practical application of AI projects. For escalator passenger flow statistics, it's often only necessary to select the floor area at the escalator entrance and exit, and the size of this area is usually manually selected. However, the size of the manually selected ROI area is not always the optimal choice. How to adaptively select the most suitable ROI area based on the hardware's computing power is a problem that needs to be solved. Furthermore, for hardware with poor performance or complex escalator scenarios, it's necessary to improve the detection speed and the accuracy of passenger count. Therefore, a robust passenger flow statistics system that can adapt to different hardware and various scenarios is needed. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings and deficiencies of existing technologies and propose a robust escalator passenger flow statistics system adaptable to different hardware and multiple scenarios. This system adaptively selects the optimal detection and tracking models suitable for different hardware platforms and application scenarios based on the computing power of each platform and pedestrian movement information in different escalator scenarios. It uses an improved tracking model to enhance tracking performance, employs software acceleration algorithms to speed up the detection model, and uses improved quantization algorithms to improve model inference speed and reduce the number of model parameters. Then, it adaptively determines the location and size of the ROI region used for passenger flow statistics, counts the number of people passing through the ROI region at the escalator entrance, and proposes algorithms such as pedestrian detection deduplication and pedestrian number correction to improve the accuracy of the number of people count. This invention is applicable to hardware platforms with different computing power and different escalator scenarios, and features improved model tracking performance, increased model detection speed, reduced model memory usage through quantization, and accelerated performance. It exhibits good robustness to different hardware and application scenarios and effectively reduces algorithm deployment problems caused by differences in hardware platforms and application scenarios.

[0005] To achieve the above objectives, the technical solution provided by this invention is: a robust escalator passenger flow statistics system adaptable to different hardware and multiple scenarios, comprising:

[0006] The hardware detection and performance testing module can acquire CPU, memory and GPU information of the application platform and test the computing power of the hardware and the inference speed of the deep learning model. Other modules in the system share the information and test data acquired by the hardware detection and performance testing module and improve and optimize their corresponding algorithms based on the acquired information and data.

[0007] A pedestrian information detection module adaptable to multiple scenarios is used to detect pedestrian information in different scenarios, including the density of pedestrian flow in video frames, the direction of pedestrian movement, the trajectory of pedestrian movement, and the speed of pedestrian movement. CrowdNet is used to estimate the density of the crowd, thereby quickly locating the initially selected ROI pedestrian flow statistics area. Based on YOLOv4 and the tracking model, it can accurately detect and track pedestrians, and obtain the speed of pedestrian movement based on the direction of movement and the trajectory of movement.

[0008] The tracking model adaptive module adaptively determines whether a tracking model is needed and the type of tracking model to be used based on the inference speed obtained from the hardware detection and performance testing modules. For the tracking model used, in order to overcome the problems of the original tracking model that only uses IoU as weights and cannot determine the two disjointnesses and does not consider the shape of the detection box and the tracking box, an improved matrix weight calculation method for the matching of detection boxes and tracking boxes is proposed. This method takes into account the overlap between the detection box and the tracking box, the distance between the two boxes and the size difference of the boxes, and provides a scaling factor to adapt to the degree of attention to the shape of the detection box and the tracking box in multiple scenarios, thereby improving the tracking accuracy.

[0009] The detection model adaptive module adaptively selects the optimal model based on the hardware information obtained from the hardware detection and performance testing modules, the hardware inference speed, and the tracking model type determined by the tracking model adaptive module. It also improves existing corresponding models to match hardware devices and enhance hardware utilization. This module features inference acceleration, achieved through improvements to the software processing of the detection model post-processing NMS, an asymmetric truncation mapping, a KL divergence-based evaluation quantization method, and a multi-threaded image processing and target detection acceleration method.

[0010] The ROI region adaptive module determines densely populated areas as preliminary ROI regions for passenger flow statistics based on the crowd density estimation of CrowdNet, determines the direction of the ROI region based on the walking direction of pedestrians, and determines the length and width of the ROI region based on the inference speed of the detection model in the detection model adaptive module.

[0011] The pedestrian counting module, after the ROI region adaptive module determines the ROI region, detects and tracks pedestrians passing through the ROI region based on the detection model and the tracking model, and then performs passenger flow statistics. For cases where the tracking model adaptive module does not use a tracking algorithm, a gray-scale histogram comparison algorithm is used to compare the similarity of heads in the ROI entrance and ROI exit areas to determine whether the same person is counted repeatedly. To address the issue of missed detections caused by occlusion in the detection model when the crowd is congested, a passenger count correction algorithm is proposed to improve the accuracy of passenger flow statistics.

[0012] Furthermore, the hardware detection and performance testing module includes the following functions:

[0013] a. This module integrates a series of hardware detection functions, which can obtain CPU information of the hardware, including CPU performance and number of cores; it can detect whether a GPU exists and obtain information such as the GPU model, architecture and memory size.

[0014] b. This module tests the computational speed of deep learning-based object detection methods on both CPU and GPU, using YOLOv4 as the detection model. For devices with GPUs, convolution operations are performed on the GPU; otherwise, they are performed on the CPU. The module first loads pre-trained weight parameters, then inputs videos from different real-world application scenarios into the detection model. The model performs inference detection on each frame of the video, records the detection time for each frame, sums these times, and finally calculates the average detection time t for each frame. det .

[0015] Furthermore, the pedestrian information detection module adapted to multiple scenarios includes the following functions:

[0016] a) Functionality for locating and selecting ROI regions based on crowd density; CrowdNet is used to estimate the crowd density in the image, and the region with the highest crowd density is recorded as the initial selected region for passenger flow statistics, denoted as ROI. pre ;

[0017] b. Functionality for acquiring pedestrian movement information; using detection and tracking models to analyze the densely populated ROI selected through CrowdNet. pre All pedestrians are detected and tracked, and their movement trajectories and directions are obtained. Then, the pedestrians' speeds are calculated based on their trajectories and directions, and the initial video frame of each person just entering the floor slab is recorded. in and position (x) in ,y in ), where x in and y in They are video frames. in The height and width coordinates are used to record the last video frame number of each person appearing on the floor slab. out and position (x) out ,y out ), where x out and y out They are video frames. out The coordinates in the height and width directions are calculated; then (x) in ,y in ) and (x out ,y out The straight-line distance dis, where Then calculate the speed of the i-th pedestrian. Where i∈[1,pn], pn represents the total number of pedestrians, and then the speeds of all pedestrians, v1,v2,...,v, are calculated. pn To obtain average speed The average speed of pedestrians detected by the pedestrian information detection module; based on θ i Let θi represent the direction of movement of the i-th pedestrian, and find the directions of movement of all pedestrians θ1, θ2, ..., θi. pn Then calculate the average direction of movement of the pedestrians.

[0018] Furthermore, the tracking model adaptive module includes the following functions:

[0019] a. This module has a tracking model selection function: This module includes three different tracking models, including: TRACK no TRACK indicates that no tracking model is used; re , indicating a tracking model based on correlation filtering; TRACK kam This represents a Kalman filter-based tracking model. The adaptive process of the tracking model involves first obtaining the average detection time t for each frame, calculated by the hardware detection and performance testing module. det Then, a detection time threshold t is set according to different application scenarios. thres The final track model type, TrackType, is determined according to the following formula:

[0020]

[0021] b. For cases using a tracking model, this module proposes an improved algorithm to optimize tracking performance. To overcome the limitations of the original algorithm, which only uses IoU as weights and cannot determine the disjointness of two boxes, and which does not consider the shapes of the detection and tracking boxes, an improved matrix weight calculation method for detection and tracking box matching is proposed. This method considers not only the overlap between the detection and tracking boxes, but also the distance between the two boxes and the size difference between the boxes, and provides a scaling factor to adapt to the degree of attention paid to the shape of the detection and tracking boxes in various scenarios. For the matching of detection box D and tracking box T, its matching degree is defined as MATCH(D,T), and its calculation method is as follows:

[0022]

[0023] In the formula, S D∩T and S D∪T w represents the area of ​​the intersection and union regions of D and T, respectively. D h D To detect the width and height of bounding box D, w T h T For the width and height of the tracking box T; L c L is the straight-line distance between the center points of D and T. lLet be the diagonal length of the bounding rectangle of D and T; α is an adjustable factor, with a value range of [0, 0.5], which can be fine-tuned according to the actual application scenario to reflect the importance of the scale in matching; the value range of MATCH(D, T) is (-2, 1]. For the convenience of subsequent matrix processing, the formula is used. Normalize to the range (0,1). This represents the matching weights of the detection box D and the tracking box T after normalization, which are ultimately used... Used as the matching weight between the detection box D and the tracking box T.

[0024] Furthermore, the detection model adaptive module includes the following functions:

[0025] a. It has the function of adaptive selection of detection algorithms, including two major categories of selectable detection models, one-stage detection model DET. one and the two-stage detection model DET two One-stage detection model DET one For YOLO series, CenterNet or SSD, the two-stage detection model DET two The detection model is either Faster R-CNN or Cascade R-CNN; the selection of the detection model is adaptive based on the type of tracking model, and the process is as follows: if the tracking is TRACK... no Then the detection model is selected as DET. two Otherwise, select DET as the detection model. one After selecting a detection model, the selected model is used to test the given video, obtaining the detection time for each frame, and then the final average detection time is calculated and denoted as t. det_select ;

[0026] b. A software acceleration method for detection models is provided: On GPU hardware devices, the original non-maximum suppression (NMS) process, which was executed on the CPU, is improved by using CUDA programming and placing the NMS process on the GPU, thus achieving detection acceleration. On devices containing only a CPU, the original single-threaded method is improved by dividing the tensor operation process of the detection model into two parts: the backbone, neck, and head operations are implemented in one thread, and the output tensors are saved in a queue. The post-processing NMS operation is executed in another thread, with the input of the input tensors in the queue and the output predicted bounding boxes, thus achieving detection acceleration.

[0027] c. Provides an asymmetric truncation mapping for model weight parameter quantization and a KL divergence-based quantization method; firstly, the weight parameters are asymmetrically truncated, transforming the original 32-bit floating-point precision weight parameter W... o Mapped to 8-bit integer weight parameter W p The formula for the asymmetric truncation mapping is as follows:

[0028]

[0029] In the formula, T n Satisfy min(W) o )≤-T n <0 indicates the cutoff point in the negative range; T p Satisfying 0 < T p ≤max(W o ), representing the cutoff point of the positive interval; w is the weight parameter W before quantization. o The elements in the tensor, with respect to W o Each element in the array is mapped using f(w) to obtain W. p For each element in the matrix, f(w) is the function value after the mapping operation; when T satisfies n and T p Traversing T under constraints n and T p Given the values ​​of , calculate the activation values ​​before quantization and after asymmetric truncated mapping quantization. Then, use the KL divergence formula below to calculate the difference between the activation value distributions and find the T value corresponding to the smallest KL divergence value. n and T p As the final mapping cutoff point;

[0030]

[0031] In the formula, x represents the discrete activation value before and after quantization, and A o (x) and A p (x) represent the probability distributions of the network layer output activation values ​​corresponding to the weight parameters before and after the above asymmetric truncation mapping, respectively. KL (A p ||A o ) represents A o (x) and A p The KL divergence of (x) is used to represent the similarity of their distributions.

[0032] Furthermore, the ROI region adaptive determination module includes the following functions:

[0033] This module has the function of determining the direction and size of the ROI region; it obtains the preliminary ROI based on the pedestrian information detection module adapted to multiple scenarios. preBased on the optimal region, the average pedestrian speed v, and the average pedestrian direction θ, the final ROI used for passenger flow statistics is determined with θ along its width. w For ROI pre Width; Height of the ROI region h The average detection time t is obtained from the pedestrian's average moving speed v and the adaptive module of the detection algorithm. det_select Together, it was determined to be roi h =0.9*v*t det_select .

[0034] Furthermore, the pedestrian counting module includes the following functions:

[0035] a) This module provides two different improved passenger flow statistics algorithms depending on whether the tracking model is used in the adaptive module, to adapt to the performance of different hardware or the needs of the task; for the case where the tracking model is used: the determined ROI area is divided into two regions equally according to the direction of pedestrian movement, namely the upper ROI area. up and the lower region roi down If the same pedestrian appears both above and below the ROI region, then their number is counted, and the total number of people in the pedestrian flow statistics is incremented by 1; for cases where the tracking model is not used: the detection model accumulates the number of people in the ROI region in each frame of the image it detects;

[0036] b. For passenger flow statistics methods that do not use tracking models, a new algorithm is proposed to avoid the same person being detected repeatedly: A detection box near a pedestrian exiting the ROI region is denoted as B, and the most likely detection box near a pedestrian entering the ROI region is denoted as A, indicating that they are the same target. This algorithm determines whether they are the same target by comparing the gray-level histograms and shape similarity between the two detection boxes. d(A,B) represents the difference between the two detection boxes. The closer the difference is to 0, the more similar the two targets are, and the more likely they are the same target. The specific calculation formula is as follows:

[0037]

[0038] In the formula, H A (i) and H B (i) are the normalized histograms of the images corresponding to detection boxes A and B, respectively, where N represents the number of bars in the histogram, and i∈[1,N] represents one of the histogram bars; w A and h A These are the width and height of the detection box A, respectively. B and h BLet α be the width and height of the detection box B, β be the weight of the difference in width between the two boxes, and β be the weight of the difference in height between the two boxes. Then, the calculation result of the above formula is combined with the set threshold d. thres Compare, if greater than d thres This indicates that there is no duplicate count between the detection box B in the current frame and the detection box A in the previous frame. Detection box B is then included in the passenger flow statistics. If the count is less than d... thres This means that the detection box B in the current frame and the detection box A in the previous frame are the same target, so detection box B is not included in the passenger flow count.

[0039] c. For passenger flow statistics methods that do not use tracking models, in order to correct the missed detection problem caused by congestion and occlusion, a passenger flow statistics correction algorithm based on passenger flow density is proposed. The idea is that the denser the passenger flow, the more severe the occlusion, and the greater the probability of the model missing detections. Therefore, the number of people counted by the model will be less than the actual number. This algorithm defines the dense representation of passenger flow within the ROI area by the area ratio of the detection box, and then proposes a correction algorithm based on the density. The specific formula is as follows:

[0040]

[0041]

[0042] In the formula, R represents the density of pedestrian traffic within the ROI area, and S... bbox (j) represents the area of ​​the j-th detection box, N p This represents the total number of people detected in the video frame, j∈[1,N]. p [] represents any person detected in a video frame, S ROI Let N represent the area of ​​the ROI region; f(R) represents the number of people that need to be corrected; the final number of people detected in the current video frame is N. p +f(R).

[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0044] 1. This invention can adaptively select the optimal detection and tracking model suitable for the hardware platform and application scenario based on the hardware information and computing power of different hardware platforms and pedestrian movement information in various escalator scenarios.

[0045] 2. Compared with other methods, the present invention improves the tracking model and enhances the tracking effect.

[0046] 3. Compared with other methods, this invention uses an improved quantization algorithm and software acceleration method to achieve improved deceleration.

[0047] 4. Compared with other methods, the present invention can adaptively determine the location and size of the ROI area for passenger flow statistics.

[0048] 5. Compared with other methods, this invention provides algorithms for pedestrian detection deduplication and pedestrian count correction, which improves the robustness of pedestrian detection.

[0049] 6. Compared with other methods, the deployment of this invention is simpler and faster. Attached Figure Description

[0050] Figure 1 This is a diagram showing the module composition of the system of the present invention.

[0051] Figure 2 This is a similarity matching graph between detection boxes and tracking boxes in the adaptive module of the tracking model.

[0052] Figure 3 This is a schematic diagram for detecting asymmetric truncation mappings in the adaptive module of the detection model.

[0053] Figure 4 This is a diagram illustrating the process of deduplicating pedestrian counts in the pedestrian counting module. Detailed Implementation

[0054] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0055] like Figure 1 As shown, this embodiment provides a robust escalator passenger flow statistics system adaptable to different hardware and multiple scenarios, including a hardware detection and performance testing module, a pedestrian information detection module adaptable to multiple scenarios, a tracking model adaptive module, a detection model adaptive module, an ROI region adaptive module, and a pedestrian number counting module, the specific details of which are as follows:

[0056] In the hardware detection and performance testing module, the program retrieves hardware information stored in the system files, including CPU and memory information, and saves it as shared information for various modules. The detection model is YOLOv4. First, it acquires video of the escalator scene, then extracts one frame every 20 frames and saves the resulting image. Since the target to be detected is human heads, the human heads in the images are labeled. The labeled dataset is divided into training and testing sets, and then YOLOv4 is trained. Then, the trained YOLOv4 is used to test the inference speed of videos in a new escalator application scenario. The selected videos include both congested and uncongested situations, and the duration is set to 30 minutes. Then, each frame of the test video is extracted and denoted as IMG = {img1, img2, ..., img...} nThen, the inference speed of YOLOv4 was tested using the IMG dataset, and the inference time for each image was denoted as T = t1, t2, ..., t n The average detection time obtained by the hardware inference time detection module Where n represents the total number of frames in the video, and img k t represents the k-th frame image in the video. k Let represent the time taken to perform inference on the k-th frame image, where k satisfies k∈[1,n];

[0057] In the pedestrian information detection module adapted to multiple scenarios, CrowdNet is used to estimate the crowd density in the image, and the region with the highest crowd density is recorded. This region accounts for approximately [a certain percentage] of the entire image size. The area selected initially for passenger flow statistics is denoted as ROI. pre The YOLOv4 and Kalman filter tracker group records each person's entry into the ROI. pre Video frame rate in and position (x) in ,y in ), where x in and y in They are video frames. in The height and width coordinates are used to record the last video frame number of each person appearing on the floor slab. out and position (x) out ,y out ), where x out and y out They are video frames. out The coordinates in the height and width directions; the straight-line distance of the motion trajectory. The difference in frame count Δframe = frame out -frame in Generally, if the frame rate of a video is 24 fps, then the speed of a pedestrian's movement... Find the velocities v1, v2, ..., v of all pedestrians. pn To obtain average speed The average speed of pedestrians detected by the pedestrian information detection module, where pn represents the total number of pedestrians, v i Let i represent the speed of the i-th pedestrian, where i ∈ [1, pn]; according to θ i Let θi represent the direction of movement of the i-th pedestrian, and find the directions of movement of all pedestrians θ1, θ2, ..., θi. pn Then calculate the average direction of movement of the pedestrians. In the adaptive tracking model module, a detection time threshold t is set. thres =1s, the hardware model inference time t obtained from the hardware detection and performance testing module. det Adaptively determine the type of tracking model to use; if t det ≥t thres If the detection time is too long and exceeds the threshold, use TRACK. no This indicates that the detection time is too long, and the detection results cannot be matched with the tracking results, so the tracking model is not used; if 0.1*t thres ≤t det <t thres If the detection time is less than the threshold but greater than 0.1 times the threshold, the TRACK tracking model, based on the correlation filtering principle, provided by the adaptive tracking model module, can be used. re If t det <0.1*t thres The detection time is short, the hardware computing power is relatively good, and the effective Kalman filter tracking model TRACK can be used. kam For cases using a tracking model, an improved algorithm to optimize tracking performance is as follows: (see attached image) Figure 2 As shown, for the matching of detection box D and tracking box T, the matching degree is defined as MATCH(D,T), and its calculation method is as follows:

[0058]

[0059] In the formula, S D∩T and S D∪T w represents the area of ​​the intersection and union regions of D and T, respectively. D ,h D To detect the width and height of bounding box D, w T h T For the width and height of the tracking box T; L c L is the straight-line distance between the center points of D and T. l Let be the diagonal length of the bounding rectangle of D and T; α is an adjustable factor, with a value range of (0, 0.5]. This parameter can be fine-tuned according to the actual application scenario. For scenarios with large changes in the head frame, this value can be set larger to reflect the importance of scale in matching; the value range of MATCH(D, T) is (-2, 1]. For the convenience of subsequent matrix processing, the formula is used. Normalize to the range (0,1). This represents the matching weights of the detection box D and the tracking box T after normalization, which are ultimately used... Used as the matching weight between the detection box D and the tracking box T.

[0060] In the adaptive detection model module, the first-stage detection model DET one Including YOLO series, CenterNet, SSD, and the two-stage detection model DET. two Including Faster R-CNN, Cascade R-CNN, etc., the detection model is adaptively selected based on the type of tracking model. The process is as follows: if the tracking is selected as TRACK... no Then the detection model is selected as DET. two Otherwise, select DET as the detection model. one After selecting a detection model, the selected model is used to test the given video, obtaining the detection time for each frame, and then the final average detection time is calculated and denoted as t. det_select Furthermore, the algorithm flow for software acceleration of the detection model is as follows: On GPU hardware devices, the original non-maximum suppression (NMS) process, which was executed on the CPU, is improved by using CUDA programming and placing the NMS process on the GPU, thus achieving detection acceleration. On devices containing only a CPU, the original single-threaded method is improved by dividing the tensor operation process of the detection model into two parts: the backbone, neck, and head operations are implemented in one thread, and the output tensors are saved in a queue. The post-processing NMS operation is executed in another thread, with the input of the input being the tensors in the queue and the output being the predicted bounding boxes, thus achieving detection acceleration. Furthermore, an asymmetric truncation mapping and KL divergence-based evaluation quantization method for model weight parameters is proposed: First, the weight parameters are subjected to asymmetric truncation mapping, followed by... Figure 3 This is a schematic diagram of the mapping process, where the original 32-bit floating-point precision weight parameter W is... o Mapped to 8-bit integer weight parameter W p The formula for the asymmetric truncation mapping is as follows:

[0061]

[0062] In the formula, T n Satisfy min(W) o )≤-T n <0 indicates the cutoff point in the negative range; T p Satisfying 0 < T p ≤max(W o ), representing the cutoff point of the positive interval; w is the weight parameter W before quantization. o The elements in the tensor, with respect to W o Each element in the array is mapped using f(w) to obtain W. p For each element in the matrix, f(w) is the function value after the mapping operation; when T satisfies n and Tp Traversing T under constraints n and T p Generally, the preferred interval for traversal iterations is to satisfy the requirement of 1000 iterations. Calculate the activation values ​​before network quantization and after asymmetric truncation mapping quantization. Then, use the KL divergence formula below to calculate the difference between the activation value distributions, and find the T value corresponding to the smallest KL divergence value. n and T p As the final mapping cutoff point;

[0063]

[0064] In the formula, x represents the discrete activation value before and after quantization, and A o (x) and A p (x) represent the network layer output activation values ​​of the weight parameters before and after the above asymmetric truncation mapping, respectively, D KL (A p ||A o ) represents A o (x) and A p The KL divergence of (x) is used to represent the similarity of their distributions.

[0065] In the ROI region adaptive module, the initial ROI is obtained based on the pedestrian information detection module adapted to multiple scenarios. pre Based on the optimal region, the average pedestrian speed v, and the average pedestrian direction θ, the final ROI used for passenger flow statistics is determined with θ along its width. w For ROI pre Width; Height of the ROI region h The average detection time t is obtained from the pedestrian's movement speed v and the detection model's adaptive module. det_select Together, it was determined to be roi h =0.9*v*t det_select .

[0066] In the pedestrian counting module, two different improved pedestrian flow statistics algorithms are provided depending on whether the tracking model is used in the adaptive module, to adapt to the performance of different hardware or the needs of the task. For the case where the tracking model is used: the determined ROI region is divided into two regions based on the direction of pedestrian movement, namely the upper ROI region and the lower ROI region. up and the lower region roi downIf the same pedestrian appears both above and below the ROI region, then their presence is counted, and the total number of people in the pedestrian flow statistics is incremented by 1. For cases where a tracking model is not used: the detection model accumulates the number of people in the ROI region for each detected image frame. Furthermore, the algorithm to avoid the same person being detected repeatedly is implemented as follows: (see attached image). Figure 4 As shown, the detection box near the pedestrian exiting the ROI region, denoted as B, and B should be within 0.2 times the ROI height at the exit, is most likely the same target as the detection box near the pedestrian entering the ROI region, denoted as A, and A should be within 0.2 times the ROI height at the entrance. The algorithm determines whether they are the same target by comparing the grayscale histograms and shape similarity between the two detection boxes. d(A,B) represents the difference between the two detection boxes; the closer the difference is to 0, the more similar the two targets are, and the more likely they are the same target. The specific calculation formula is as follows:

[0067]

[0068] In the formula, H A (i) and H B (i) are the normalized histograms of the images corresponding to detection boxes A and B, respectively, where N represents the number of bars in the histogram, and i∈[1,N] represents one of the histogram bars; w A and w A These are the width and height of the detection box A, respectively. B and h B Let α be the width and height of the detection box A, β be the weight of the difference in width between the two boxes, and β be the weight of the difference in height between the two boxes. Then, the calculation result of the above formula is combined with the set threshold d. thres For comparison, preferably, the threshold is set to 0.5; if it is greater than d... thres This indicates that there is no duplicate count between the detection box B in the current frame and the detection box A in the previous frame. Detection box B is then included in the passenger flow statistics. If the count is less than d... thres This indicates that the detection box B in the current frame and the detection box A in the previous frame are the same target, so detection box B is not included in the passenger flow count. Furthermore, for passenger flow statistics methods that do not use a tracking model, to correct the missed detection problem caused by congestion and occlusion, a passenger flow statistics correction algorithm based on passenger flow density is proposed. The idea is that the denser the passenger flow, the more severe the occlusion, and the greater the probability of the model missing detections; therefore, the number of people counted by the model will be less than the actual number. This algorithm defines the dense representation of passenger flow within the ROI area by the area ratio of the detection boxes, and then proposes a correction algorithm based on the density, the specific formula of which is as follows:

[0069]

[0070]

[0071] In the formula, R represents the density of pedestrian traffic within the ROI area, and S... bbox (j) represents the area of ​​the j-th detection box, N p This represents the total number of people detected in the video frame, j∈[1,N]. p [] represents any person detected in a video frame, S ROI Let N represent the area of ​​the ROI region; f(R) represents the number of people that need to be corrected; the final number of people detected in the current video frame is N. p +f(R).

[0072] In summary, by adopting the above solutions, this invention can adaptively select the optimal detection and tracking models suitable for different hardware platforms and application scenarios based on the computing power of different hardware platforms and pedestrian movement information in different escalator scenarios. It uses an improved tracking model to enhance tracking performance, employs software acceleration algorithms to speed up the detection model, and uses improved quantization algorithms to improve model inference speed and reduce the number of model parameters. Then, it adaptively determines the location and size of the ROI region used for passenger flow statistics, counts the number of people passing through the ROI region at the escalator entrance, and proposes algorithms such as pedestrian detection deduplication and pedestrian count correction to improve the accuracy of the count. This invention is applicable to hardware platforms with different computing power and different escalator scenarios, and has the advantages of improving model tracking performance, increasing model detection speed, reducing model memory usage and accelerating performance through quantization. It exhibits good robustness to different hardware and application scenarios, and effectively reduces algorithm deployment problems caused by differences in hardware platforms and application scenarios. This invention has practical promotional value and is worth promoting.

[0073] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A robust escalator passenger flow statistics system adaptable to different hardware and multiple scenarios, characterized in that: include: The hardware detection and performance testing module can acquire CPU, memory and GPU information of the application platform and test the computing power of the hardware and the inference speed of the deep learning model. Other modules in the system share the information and test data acquired by the hardware detection and performance testing module and improve and optimize their corresponding algorithms based on the acquired information and data. A pedestrian information detection module adaptable to multiple scenarios. This module is used to detect pedestrian information in different scenarios, including the density of pedestrian flow in video frames, the direction of pedestrian movement, the trajectory of pedestrian movement, and the speed of pedestrian movement. CrowdNet is used to estimate crowd density, thereby quickly locating the initially selected ROI passenger flow statistics area. Based on YOLOv4 and the tracking model, pedestrians can be accurately detected and tracked, and the movement speed of pedestrians can be obtained according to their movement direction and trajectory. The tracking model adaptive module adaptively determines whether a tracking model is needed and the type of tracking model to be used based on the inference speed obtained from the hardware detection and performance testing modules. For the tracking model used, in order to overcome the problems of the original tracking model that only uses IoU as weights and cannot determine the two disjointnesses and does not consider the shape of the detection box and the tracking box, an improved matrix weight calculation method for the matching of detection boxes and tracking boxes is proposed. This method takes into account the overlap between the detection box and the tracking box, the distance between the two boxes and the size difference of the boxes, and provides a scaling factor to adapt to the degree of attention to the shape of the detection box and the tracking box in multiple scenarios, thereby improving the tracking accuracy. The detection model adaptive module adaptively selects the optimal model based on the hardware information and inference speed of the hardware obtained by the hardware detection and performance testing modules and the tracking model type determined by the tracking model adaptive module. It also improves the existing corresponding models to match the hardware devices and improve the utilization of the hardware. This module features inference acceleration, achieved through improvements to the software processing of the detection model post-processing NMS, an asymmetric truncation mapping, a KL divergence-based evaluation quantization method, and a multi-threaded image processing and target detection acceleration method. The ROI region adaptive module determines densely populated areas as preliminary ROI regions for passenger flow statistics based on the crowd density estimation of CrowdNet, determines the direction of the ROI region based on the walking direction of pedestrians, and determines the length and width of the ROI region based on the inference speed of the detection model in the detection model adaptive module. The pedestrian counting module detects and tracks pedestrians passing through the ROI area after the ROI area adaptive module has determined the ROI area, and then performs passenger flow statistics based on the detection model and the tracking model. For cases where the tracking algorithm is not used in the adaptive module of the tracking model, a gray-scale histogram comparison algorithm is used to compare the similarity of heads in the ROI entrance and ROI exit areas in order to increase the accuracy of passenger flow statistics, thereby determining whether the same person is counted repeatedly. In order to solve the problem of missed detection caused by occlusion in the detection model when the crowd is crowded, a passenger number correction algorithm is proposed to improve the accuracy of passenger flow statistics.

2. The robust escalator passenger flow statistics system adaptable to different hardware scenarios as described in claim 1, characterized in that, The hardware detection and performance testing module includes the following functions: a. This module integrates a series of hardware detection functions, which can obtain CPU information of the hardware, including CPU performance and number of cores; it can detect whether a GPU exists and obtain information such as the GPU model, architecture and memory size. b. This module tests the computational speed of deep learning-based object detection methods on both CPU and GPU, using YOLOv4 as the detection model. For devices with GPUs, convolution operations are performed on the GPU; otherwise, they are performed on the CPU. The module first loads pre-trained weight parameters, then inputs videos from different real-world application scenarios into the detection model. The model performs inference detection on each frame of the video, records the detection time for each frame, sums these times, and finally calculates the average detection time t for each frame. det .

3. The robust escalator passenger flow statistics system adaptable to different hardware scenarios according to claim 2, characterized in that, The pedestrian information detection module adapted to multiple scenarios includes the following functions: a) Functionality for locating and selecting ROI regions based on crowd density; CrowdNet is used to estimate the crowd density in the image, and the region with the highest crowd density is recorded as the initial selected region for passenger flow statistics, denoted as ROI. pre ; b. Functionality for acquiring pedestrian movement information; using detection and tracking models to analyze the densely populated ROI selected through CrowdNet. pre All pedestrians are detected and tracked, and their movement trajectories and directions are obtained. Then, the pedestrians' speeds are calculated based on their trajectories and directions, and the initial video frame of each person just entering the floor slab is recorded. in and position (x) in ,y in ), where x in and y in They are video frames. in The height and width coordinates are used to record the last video frame number of each person appearing on the floor slab. out and position (x) out ,y out ), where x out and y out They are video frames. out The coordinates in the height and width directions are calculated; then (x) in ,y in ) and (x out ,y out The straight-line distance dis, where Then calculate the speed of the i-th pedestrian. Where i∈[1,pn], pn represents the total number of pedestrians, and then the speeds of all pedestrians, v1,v2,…,v, are calculated. pn To obtain average speed The average speed of pedestrians detected by the pedestrian information detection module; based on θ i Let θi represent the direction of movement of the i-th pedestrian, and find the directions of movement of all pedestrians θ1, θ2, ..., θi. pn Then calculate the average direction of movement of the pedestrians.

4. The robust escalator passenger flow statistics system adaptable to different hardware and multiple scenarios according to claim 3, characterized in that, The tracking model adaptive module includes the following functions: a. This module has a tracking model selection function: This module includes three different tracking models, including: TRACK no TRACK indicates that no tracking model is used; re , indicating a tracking model based on correlation filtering; TRACK kam This represents a Kalman filter-based tracking model. The adaptive process of the tracking model involves first obtaining the average detection time t for each frame, calculated by the hardware detection and performance testing module. det Then, a detection time threshold t is set according to different application scenarios. thres The final track model type, TrackType, is determined according to the following formula: b. For cases using a tracking model, this module proposes an improved algorithm to optimize tracking performance. To overcome the limitations of the original algorithm, which only uses IoU as weights and cannot determine the disjointness of two boxes, and which does not consider the shapes of the detection and tracking boxes, an improved matrix weight calculation method for detection and tracking box matching is proposed. This method considers not only the overlap between the detection and tracking boxes, but also the distance between the two boxes and the size difference between the boxes, and provides a scaling factor to adapt to the degree of attention paid to the shape of the detection and tracking boxes in various scenarios. For the matching of detection box D and tracking box T, its matching degree is defined as MATCH(D,T), and its calculation method is as follows: In the formula, S D∩T and S D∪T w represents the area of ​​the intersection and union regions of D and T, respectively. D h D To detect the width and height of bounding box D, w T h T For the width and height of the tracking box T; L c L is the straight-line distance between the center points of D and T. l Let be the diagonal length of the bounding rectangle of D and T; α is an adjustable factor, with a value range of [0, 0.5], which can be fine-tuned according to the actual application scenario to reflect the importance of the scale in matching; the value range of MATCH(D, T) is (-2, 1]. For the convenience of subsequent matrix processing, the formula is used. Normalize to the range (0,1). This represents the matching weights of the detection box D and the tracking box T after normalization, which are ultimately used... Used as the matching weight between the detection box D and the tracking box T.

5. The robust escalator passenger flow statistics system adaptable to different hardware scenarios according to claim 4, characterized in that, The adaptive detection model module includes the following functions: a. It has the function of adaptive selection of detection algorithms, including two major categories of selectable detection models, one-stage detection model DET. one and the two-stage detection model DET two One-stage detection model DET one For YOLO series, CenterNet or SSD, the two-stage detection model DET two The detection model is either Faster R-CNN or Cascade R-CNN; the selection of the detection model is adaptive based on the type of tracking model, and the process is as follows: if the tracking is TRACK... no Then the detection model is selected as DET. two Otherwise, select DET as the detection model. one After selecting a detection model, the selected model is used to test the given video, obtaining the detection time for each frame, and then the final average detection time is calculated and denoted as t. det_select ; b. A software acceleration method for detection models is provided: On GPU hardware devices, the original non-maximum suppression (NMS) process, which was executed on the CPU, is improved by using CUDA programming and placing the NMS process on the GPU, thus achieving detection acceleration. On devices containing only a CPU, the original single-threaded method is improved by dividing the tensor operation process of the detection model into two parts: the backbone, neck, and head operations are implemented in one thread, and the output tensors are saved in a queue. The post-processing NMS operation is executed in another thread, with the input of the input tensors in the queue and the output predicted bounding boxes, thus achieving detection acceleration. c. Provides an asymmetric truncation mapping for model weight parameter quantization and a KL divergence-based quantization method; firstly, the weight parameters are asymmetrically truncated, transforming the original 32-bit floating-point precision weight parameter W... o Mapped to 8-bit integer weight parameter W p The formula for the asymmetric truncation mapping is as follows: In the formula, T n Satisfy min(W) o )≤-T n <0 indicates the cutoff point in the negative range; T p Satisfying 0 < T p ≤max(W o ), representing the cutoff point of the positive interval; w is the weight parameter W before quantization. o The elements in the tensor, with respect to W o Each element in the array is mapped using f(w) to obtain W. p For each element in the matrix, f(w) is the function value after the mapping operation; when T satisfies n and T p Traversing T under constraints n and T p Given the values ​​of , calculate the activation values ​​before quantization and after asymmetric truncated mapping quantization. Then, use the KL divergence formula below to calculate the difference between the activation value distributions and find the T value corresponding to the smallest KL divergence value. n and T p As the final mapping cutoff point; In the formula, x represents the discrete activation value before and after quantization, and A o (x) and A p (x) represent the probability distributions of the network layer output activation values ​​corresponding to the weight parameters before and after the above asymmetric truncation mapping, respectively. KL (A p ||A o ) represents A o (x) and A p The KL divergence of (x) is used to represent the similarity of their distributions.

6. The robust escalator passenger flow statistics system adaptable to different hardware and multiple scenarios according to claim 5, characterized in that, The ROI region adaptive module includes the following functions: This module has the function of determining the direction and size of the ROI region; it obtains the preliminary ROI based on the pedestrian information detection module adapted to multiple scenarios. pre Based on the optimal region, the average pedestrian speed v, and the average pedestrian direction θ, the final ROI used for passenger flow statistics is determined with θ along its width. w For ROI pre Width; Height of the ROI region h The average detection time t is obtained from the pedestrian's average moving speed v and the adaptive module of the detection algorithm. det_select Together, it was determined to be roi h =0.9*v*t det_select .

7. The robust escalator passenger flow statistics system adaptable to different hardware and multiple scenarios according to claim 6, characterized in that, The pedestrian counting module includes the following functions: a. This module provides two different improved passenger flow statistics algorithms depending on whether the tracking model is used in the adaptive module, in order to adapt to the performance of different hardware or the needs of the task; For cases using a tracking model: the determined ROI region is divided into two regions based on the direction of pedestrian movement, namely the upper ROI region and the lower ROI region. up and the lower region roi down If the same pedestrian appears both above and below the ROI region, then their number is counted, and the total number of people in the pedestrian flow statistics is incremented by 1; for cases where the tracking model is not used: the detection model accumulates the number of people in the ROI region in each frame of the image it detects; b. For passenger flow statistics methods that do not use tracking models, a new algorithm is proposed to avoid the same person being detected repeatedly: A detection box near a pedestrian exiting the ROI region is denoted as B, and the most likely detection box near a pedestrian entering the ROI region is denoted as A, indicating that they are the same target. This algorithm determines whether they are the same target by comparing the gray-level histograms and shape similarity between the two detection boxes. d(A,B) represents the difference between the two detection boxes. The closer the difference is to 0, the more similar the two targets are, and the more likely they are the same target. The specific calculation formula is as follows: In the formula, H A (i) and H B (i) are the normalized histograms of the images corresponding to detection boxes A and B, respectively, where N represents the number of bars in the histogram, and i∈[1,N] represents one of the histogram bars; w A and h A These are the width and height of the detection box A, respectively. B and h B Let α be the width and height of the detection box B, respectively, and let β be the weight of the difference in width between the two boxes. Then, the calculation result of the above formula and the set threshold d are used. thres Compare, if greater than d thres This indicates that there is no duplicate count between the detection box B in the current frame and the detection box A in the previous frame. Detection box B is then included in the passenger flow statistics. If the count is less than d... thres This means that the detection box B in the current frame and the detection box A in the previous frame are the same target, so detection box B is not included in the passenger flow count. c. For passenger flow statistics methods that do not use tracking models, in order to correct the missed detection problem caused by congestion and occlusion, a passenger flow statistics correction algorithm based on passenger flow density is proposed. The idea is that the denser the passenger flow, the more severe the occlusion, and the greater the probability of the model missing detections. Therefore, the number of people counted by the model will be less than the actual number. This algorithm defines the dense representation of passenger flow within the ROI area by the area ratio of the detection box, and then proposes a correction algorithm based on the density. The specific formula is as follows: In the formula, R represents the density of pedestrian traffic within the ROI area, and S... bbox (j) represents the area of ​​the j-th detection box, N p This represents the total number of people detected in the video frame, j∈[1,N]. p [] represents any person detected in a video frame, S ROI Let N represent the area of ​​the ROI region; f(R) represents the number of people that need to be corrected; the final number of people detected in the current video frame is N. p +f(R).

Images