Building entrance and exit people flow statistical method and system based on multi-modal perception

Abstract

The application provides a kind of building access flow statistics method and system based on multi-modal perception, it is related to the technical field of intelligent building management and computer vision, the method comprises: obtaining the real-time video stream of building entrance and exit, ambient light intensity and access control service data;Based on ambient light intensity, dynamically adjust the processing parameters of visual perception link, and based on the current population density, adaptively configure the multi-modal feature fusion strategy, process the real-time video stream, generate pedestrian tracking trajectory and visual statistical number;Based on tracking trajectory, access control service data and ambient light intensity, calibrate the visual statistical number, output the final number of people in the building, by introducing and deeply using ambient light intensity, dynamically adjust the whole process of video processing and number statistics, solve the technical problems of low detection accuracy, poor accuracy and reliability of people flow statistics result, realize robust, accurate real-time number of people in the building statistics.

Description

Technical Field

[0001] This invention relates to the technical fields of intelligent building management and computer vision, and more specifically, to a method and system for counting pedestrian flow in and out of buildings based on multimodal perception. Background Technology

[0002] With the rapid development of smart buildings and building automation management, higher demands are being placed on the accurate counting and management of real-time pedestrian traffic within buildings. Currently, a common practice is to deploy visual monitoring systems at entrances and exits. These systems typically monitor video streams, using relevant target detection and tracking algorithms to identify and track pedestrians, and determine the direction of entry and exit by setting counting areas in the video frame, thereby achieving headcount.

[0003] However, in actual building applications, such solutions that rely primarily on visual analysis face certain limitations. For example, existing technologies have the following technical problems: in complex environmental conditions and densely populated scenarios, the accuracy of vision-based pedestrian detection and statistics methods is poor; in long-term operation, the biases introduced by the visual analysis process will affect the stability of the final statistical data, resulting in insufficient reliability of long-term statistical results. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for counting pedestrian traffic in and out of buildings based on multimodal perception, so as to solve the technical problems of low detection model accuracy and poor accuracy and reliability of pedestrian traffic statistics results in the prior art.

[0005] In a first aspect, embodiments of the present invention provide a method for counting pedestrian flow in and out of buildings based on multimodal perception, including: acquiring real-time video streams of building entrances and exits, real-time ambient light intensity of the corresponding monitoring area, and access control business data; The processing parameters of the visual perception process are dynamically adjusted based on the ambient light intensity, and a multimodal feature fusion strategy is adaptively configured based on the current population density to process the real-time video stream and generate pedestrian tracking trajectories and visual population statistics. Based on the tracking trajectory, the access control data, and the ambient light intensity, the visually counted number of people is fused and corrected to output the final real-time number of people inside the building.

[0006] In some optional implementations, the dynamic adjustment of the processing parameters of the visual perception stage includes: determining an image enhancement coefficient based on the ambient light intensity using a preset nonlinear mapping function, and using the image enhancement coefficient to perform brightness transformation on the video frames of the real-time video stream; wherein the image enhancement coefficient varies with the degree to which the light intensity deviates from the reference light value.

[0007] In some optional implementations, the dynamic adjustment of the processing parameters of the visual perception stage includes: dynamically adjusting the confidence threshold of the target detector based on the ambient light intensity using a preset nonlinear mapping function.

[0008] In some optional implementations, the adaptive configuration of the multimodal feature fusion strategy based on the current personnel density includes: The population density is determined based on the number of pedestrian targets detected in the current frame of the real-time video stream; The fusion weight is calculated using the personnel density through a preset mapping relationship; the fusion weight is used to weight and combine motion feature similarity and appearance feature similarity to generate a comprehensive similarity between the detection box and the predicted trajectory for tracking association.

[0009] In some optional implementations, generating pedestrian tracking trajectories includes: The motion state of pedestrians is predicted and updated using a Kalman filter, wherein the measurement noise matrix of the Kalman filter is dynamically adjusted according to the ambient light intensity; the greater the deviation of the light intensity from the reference value, the greater the measurement noise.

[0010] In some optional implementations, the fusion and correction of the visual headcount based on the tracking trajectory, the access control service data, and the ambient light intensity includes: An environmental reliability factor is calculated based on the ambient light intensity, and the environmental reliability factor is used to represent the reliability of visual statistics under the current lighting conditions; The difference between the net number of people entering and exiting obtained from the access control business data and the visually counted number of people is weighted using the environmental reliability factor, the visually counted number of people is corrected, and the final real-time number of people in the building is output.

[0011] In some optional implementations, the fusion correction further includes: Obtain the number of people who have not left the building at the moment, as determined by the access control service data; The preset reset threshold is dynamically adjusted based on the cumulative deviation trend between the visually counted number of people and the access control business data over a historical period. During a preset quiet period, the real-time number of people in the final building is compared with the number of people who have not left the current building. If the difference exceeds the dynamically adjusted reset threshold, the number of people who have not left the current building is used as the reset value of the real-time number of people in the final building.

[0012] Secondly, embodiments of the present invention provide a building access pedestrian flow statistics system based on multimodal perception, including: a data acquisition module, used to acquire real-time video streams of building entrances and exits, real-time ambient light intensity of the corresponding monitoring area, and access control business data; The parameter adjustment module is used to dynamically adjust the processing parameters of the visual perception link based on the ambient light intensity, and to adaptively configure the multimodal feature fusion strategy based on the current personnel density to process the real-time video stream and generate pedestrian tracking trajectories and visual counts of people. The data calibration module is used to fuse and correct the visually counted number of people based on the tracking trajectory, the access control business data, and the ambient light intensity, and output the final real-time number of people in the building.

[0013] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the steps of the method described in any of the first aspects above.

[0014] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to perform the method described in any of the first aspects above.

[0015] This invention provides a method and system for counting pedestrian traffic in buildings based on multimodal perception, relating to the technical fields of intelligent building management and computer vision. The method includes: acquiring real-time video streams, ambient light intensity, and access control data at building entrances and exits; dynamically adjusting processing parameters of the visual perception stage based on ambient light intensity, and adaptively configuring a multimodal feature fusion strategy based on current pedestrian density to process the real-time video stream, generating pedestrian tracking trajectories and visually counted pedestrian numbers; calibrating the visually counted pedestrian numbers based on the tracking trajectories, access control data, and ambient light intensity, and outputting the final number of people inside the building. This invention uses ambient light intensity as a feedforward control variable throughout the entire process of visual perception and fusion correction, and adaptively switches the motion feature and appearance feature fusion strategy in multi-target tracking according to pedestrian density, solving the technical problems of low detection accuracy and poor accuracy and reliability of pedestrian traffic statistics, and achieving robust and accurate real-time pedestrian counting within buildings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1A flowchart illustrating a method for counting pedestrian flow in and out of a building based on multimodal perception, provided in an embodiment of the present invention; Figure 2 A flowchart illustrating another method for counting pedestrian flow in and out of buildings based on multimodal perception, provided in an embodiment of the present invention; Figure 3 A schematic diagram of a building entrance and exit pedestrian flow statistics system based on multimodal perception provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Traditional methods for counting pedestrian flow at building entrances and exits primarily rely on computer vision technology. This involves using surveillance cameras to capture video streams and employing target detection and tracking algorithms to identify, track, and count pedestrians entering and exiting. However, office building entrances and exits present unique challenges: drastic changes in light and shadow throughout the day, often resulting in strong backlighting, localized overexposure, or low-light conditions; high pedestrian traffic during morning and evening rush hours, with dense crowds and significant mutual obstruction; and the presence of glass curtain walls, turnstiles, revolving doors, and decorative partitions within the buildings themselves, making them highly susceptible to visual misdetection and missed detection.

[0020] The aforementioned complex conditions lead to the following technical shortcomings in existing pure vision solutions: First, under complex lighting fluctuations, high illumination noise typically causes a large number of false detections, while low illumination results in a large number of missed detections, significantly reducing counting accuracy. Second, in high-density, crowded scenes, pedestrians occlude each other, and existing tracking algorithms are prone to identity tag changes, causing duplicate counting or omissions. Third, visual statistical methods often cannot effectively utilize access control gate data; access control systems can provide accurate card swipe counts but cannot detect tailgating, cardless entry, etc., and the two remain disconnected for a long time, making it difficult to eliminate systematic errors accumulated over time.

[0021] Based on this, this invention proposes a method for statistically analyzing building entry and exit pedestrian flow, which is based on visual multimodal perception and dynamically corrects the data by combining environmental modal data and access control gate business modal data. By introducing ambient light intensity as a feedforward control variable to dynamically adjust image preprocessing parameters, detection thresholds, and Kalman filter noise, and by dynamically adjusting tracking weights through personnel density, the method ultimately utilizes an environmental reliability factor to achieve nonlinear closed-loop calibration of visual statistics and access control data.

[0022] To facilitate understanding of this embodiment, a detailed description of a building entry and exit pedestrian flow statistics method based on multimodal perception disclosed in this embodiment of the invention will be provided first. (See [link to relevant documentation]). Figure 1 The diagram shows a flowchart of a building entry and exit pedestrian flow statistics method based on multimodal perception.

[0023] Before implementing this invention, a multimodal perception model needs to be built and trained in advance. Preferably, the multimodal perception model includes an object detector and an appearance feature extraction network.

[0024] Specifically, the object detector can be designed using YOLO, Faster R-CNN, SSD, CenterNet, EfficientDet, or Vision Transformer (ViT) architectures, and fine-tuned using a large-scale labeled dataset containing office building-specific scenes (such as turnstiles, revolving doors, columns, and glass curtain wall backgrounds).

[0025] The appearance feature extraction network employs deep convolutional neural networks or attention-based networks such as the Swing Transformer. It is pre-trained on a pedestrian dataset at the entrances and exits of office buildings. Through constraints such as triplet loss functions or diagonal edge loss, the model learns appearance representations robust to changes in lighting, including clothing color, texture, and human body shape. Once trained, the model can be used for online inference.

[0026] Figure 1 The multimodal sensing method for counting pedestrian traffic in and out of buildings, as shown, can be executed by electronic devices and mainly includes the following steps S102 to S106: Step S102: Obtain the real-time video stream of the building entrance and exit, the real-time ambient light intensity of the corresponding monitoring area, and access control business data.

[0027] The real-time video stream can be captured by high-definition cameras deployed at building entrances and exits to capture pedestrian traffic, including several video frames; the corresponding monitoring area refers to the passage area at the entrance / exit covered by the camera. Real-time light intensity can be synchronously collected by a light sensor to reflect the current lighting conditions of the monitored area. Access control data can be provided in real time by the access control controller, including at least the number of people entering by swiping their cards, the number of people leaving by swiping their cards, and the number of people still inside the building based on access control records. The real-time video stream, real-time ambient light intensity, and access control data are synchronized in time, together constituting a multimodal input.

[0028] Step S104: Dynamically adjust the processing parameters of the visual perception process based on the ambient light intensity, and adaptively configure the multimodal feature fusion strategy based on the current population density to process the real-time video stream and generate pedestrian tracking trajectories and visual counts of people.

[0029] The visual perception stage refers to the entire processing flow from inputting raw video frames to outputting structured pedestrian detection and tracking results, including but not limited to image preprocessing, target detection, and multi-target tracking; the processing parameters refer to various adjustable parameters that affect the processing effect in the above stages.

[0030] Specifically, this step uses ambient light intensity as a feedforward control signal to adaptively adjust the processing parameters affected by lighting in the visual perception process, ensuring stable pedestrian perception under different lighting conditions. Simultaneously, based on the current scene's pedestrian density perceived from the real-time video stream, it adaptively adjusts the fusion method of motion and appearance features during multi-target tracking, maintaining the continuity of the tracking trajectory's identity even under crowded and occluded conditions. Furthermore, by determining and counting pedestrian entry and exit directions, it generates pedestrian tracking trajectories and real-time number of people inside the building based on visual statistics.

[0031] Step S106: Based on the tracking trajectory, access control business data and ambient light intensity, the visual count of people is fused and corrected to output the final real-time number of people inside the building.

[0032] In this step, the number of people obtained by visual perception in S104 is used as the basis. The independent entry and exit records provided by the access control business data are combined with the current visual perception conditions reflected by the ambient light intensity to dynamically compensate and correct the errors in the visual number of people.

[0033] Preferably, when visual perception conditions are good, the fusion result tends to be more visually statistical; when visual perception conditions are poor, the fusion result relies more on access control business data, thereby achieving a credibility-driven balance between the two types of data and outputting a more accurate real-time number of people inside the building.

[0034] In summary, the embodiments of the present invention solve the technical problem of inaccurate crowd counting in complex lighting and crowded scenes by dynamically adjusting the parameters of ambient light intensity throughout the visual perception process and fusing and correcting multi-source data, and by using a feature fusion strategy based on adaptive switching of multi-target tracking according to personnel density. This results in a robust and accurate real-time number of people inside a building.

[0035] In one embodiment, the processing parameters of dynamically adjusting the visual perception link in step S104 above are further explained.

[0036] Specifically, the processing parameters include the image enhancement coefficient, and the specific dynamic adjustment method is as follows: based on the ambient light intensity, the image enhancement coefficient is determined through a preset nonlinear mapping function, and the brightness of the video frames of the real-time video stream is transformed using the image enhancement coefficient; wherein, the image enhancement coefficient changes with the degree to which the light intensity deviates from the reference light value.

[0037] As a preferred implementation method, relying on the building video security system, video frames at time t are acquired through high-definition cameras at the entrance and exit of the office building. The light intensity of the corresponding monitoring area is obtained through a light sensor. An adaptive gamma correction coefficient based on the Logistic function is constructed as the image enhancement coefficient:

[0038] in, The optimal imaging reference illumination value collected by the illumination sensor; This represents the maximum Gamma compensation value under low illumination. This represents the minimum Gamma compression value under strong backlighting. This is the response sensitivity coefficient.

[0039] Perform non-linear brightness transformation on the video frames and output the preprocessed video frames. :

[0040] It can be seen that when Below hour, Approaching Brighten dark areas; when Higher than hour, Approaching By compressing highlights, the image enhancement coefficient achieves an adaptive response to illumination deviations.

[0041] As an alternative, in addition to using the Logistic function, the nonlinear mapping function can also employ piecewise linear functions, exponential mapping tables, or histogram equalization to enhance the image based on the illumination intensity.

[0042] In another embodiment, the above processing parameters also include the confidence threshold of the target detector. The specific dynamic adjustment method is as follows: based on the ambient light intensity, the confidence threshold of the target detector is dynamically adjusted through a preset nonlinear mapping function to balance the detection effect between high light noise and low light missed detection.

[0043] As a preferred implementation method, to achieve a balance between light noise and low-light missed detections, the light intensity of the corresponding monitoring area is used to address the changes in light and shadow at the entrances and exits of office buildings. Dynamically adjust the confidence threshold τ of the pedestrian target detector:

[0044] in, The target detection baseline confidence threshold; τ represents the maximum allowable fluctuation range of the confidence threshold; The reference light intensity; This is the illumination adjustment factor. Only the confidence level is retained. The detection results for targets exceeding the confidence threshold τ. When the illumination deviates from the baseline value, the tanh function allows τ to fluctuate appropriately, thereby increasing the threshold to suppress false detections in high-illumination environments with abundant noise, and decreasing the threshold to reduce missed detections in low-illumination environments.

[0045] As an alternative, the nonlinear mapping function for dynamically adjusting the confidence threshold, besides the tanh function, can be replaced by the Sigmoid function, Gaussian mapping, or simple linear scaling.

[0046] Based on the illumination-adaptive adjustment of the processing parameters of the visual perception stage in the above embodiments, before sending the video frames into the multimodal perception model for inference, this method can also adopt the following processing flow to ensure perception accuracy and efficiency.

[0047] First, based on the preset ROI parameters of the office building entrance and exit layout, the preprocessed video frames are... t The image is cropped to retain only the key monitoring area, eliminating visual noise interference from non-traffic areas in the background (such as rest areas and decorative walls).

[0048] Then, a pre-tuned multimodal perception model is used to infer the video frames cropped by the ROI to achieve pedestrian target detection and appearance feature extraction. Preferably, the multimodal perception model includes an object detector and an appearance feature extraction network: the object detector can be a YOLO, Faster R-CNN, SSD, CenterNet, EfficientDet, or VisionTransformer (ViT) architecture, and is fine-tuned using a large-scale labeled dataset containing office building-specific scenes (such as turnstiles, revolving doors, columns, and glass curtain wall backgrounds); the appearance feature extraction network adopts a deep convolutional neural network or an attention mechanism network such as SwinTransformer, and is pre-trained on a pedestrian dataset at the entrance and exit of an office building. Through constraints such as triplet loss function or diagonal edge loss, the model learns appearance representations that are robust to changes in lighting, including clothing color, texture, and human body shape.

[0049] The model can transform each frame of an image into a structured set of pedestrian features, and output a detection set. Each detection box Includes: position vector Where (x, y) are the coordinates of the center point of the detection box, and w and h are the width and height of the detection box; confidence level Foot feature point location This is used for subsequent cross-line detection and can be achieved by using the bottom midpoint of the detection box. Representation; and appearance feature vectors .

[0050] Then, non-maximum suppression is applied to the candidate detection boxes, they are sorted in descending order of confidence, the intersection-union ratio is calculated, and redundant detection boxes with an overlap higher than a preset threshold are removed to form the final detection set for the current frame.

[0051] After the above target detection and feature extraction process, the multi-target tracking stage begins, the specific implementation of which will be discussed in subsequent embodiments.

[0052] In one embodiment, the multimodal feature fusion strategy adaptively configured based on the current personnel density in step S104 specifically includes: determining the personnel density based on the number of pedestrian targets detected in the current frame of the real-time video stream; calculating the fusion weight using the personnel density through a preset mapping relationship; the fusion weight is used to weightedly combine motion feature similarity and appearance feature similarity to generate a comprehensive similarity between the detection box and the predicted trajectory for tracking association.

[0053] In the multi-target tracking phase, the motion state vector at time t is first constructed for each tracked target i. ,in , , , These represent the instantaneous rates of change of the corresponding variables. The prior predicted value of the target's state vector at time t+1 is calculated using the Kalman prediction equation. With prior prediction covariance matrix :

[0054]

[0055] in, The state transition matrix is ​​defined as follows:

[0056] It is a symmetric positive definite matrix, which is the posterior covariance matrix at the current time t, representing the state vector. The estimation error of each component in the target will be considered when a new trajectory is created. The covariance matrix is ​​initialized as a diagonal matrix. :

[0057] in The diagonal elements are the preset initial uncertainty constants for each component.

[0058] The process noise matrix characterizes the uncertainty of the system's motion model itself.

[0059] in, For the uncertainty of the position component, To address the uncertainty of the velocity component, a method based on the detection frame height is adopted. Dynamic noise model calculation:

[0060]

[0061] in This is the position error coefficient. This is the speed error coefficient.

[0062] As a preferred implementation, after obtaining all detection boxes in the current frame, the comprehensive similarity matrix C between each detection box and all predicted target trajectories is calculated. C is an N×M dimensional matrix, where N is the number of all predicted target trajectories in the current frame, and M is the number of all detection boxes in the current frame.

[0063]

[0064]

[0065] in, To characterize the spatiotemporal motion consistency based on the spatial location similarity of the intersection-union ratio of the predicted bounding box and the detection box; The cosine similarity of the appearance feature vectors represents the consistency of visual identity. , Detection boxes And the appearance feature vector corresponding to the predicted target trajectory i.

[0066] The fusion weight ω is calculated based on the current personnel density ρ through a preset mapping relationship. One implementation method is as follows:

[0067] ρ can be determined by the number of pedestrian targets detected in the current frame; This serves as a baseline value for population density in the detection area; ω is the personnel density adjustment coefficient. When the entrance and exit of the office building become crowded and the personnel density ρ increases, ω approaches 0, and trajectory tracking automatically shifts from relying on spatiotemporal motion characteristics to relying on visual appearance characteristics, thereby maintaining identity continuity under occlusion conditions.

[0068] In the correlation solution phase, decision variables are defined. , Represents the detection box The trajectory i was successfully matched and associated. Represents the detection box With the predicted target trajectory If no match is found, no association is performed. The objective function is to maximize the global association similarity Z:

[0069] The constraints are that each detection box can be assigned to at most one trajectory, and each trajectory can match at most one detection box. For example: Uniqueness constraint (row constraint): Each detection box can be assigned to at most one trajectory to prevent multiple trajectories from competing for the same detection target.

[0070] Consistency constraint (column constraint): Each trajectory can match at most one bounding box to prevent ID splitting.

[0071] The solution methods can include the Hungarian algorithm, branch and bound method, KM algorithm, or greedy matching algorithm.

[0072] Based on the above embodiments, the generation of pedestrian tracking trajectory in step S104 may specifically include: using a Kalman filter to predict and update the pedestrian's motion state, wherein the measurement noise matrix of the Kalman filter is dynamically adjusted according to the ambient light intensity, and the greater the deviation of the light intensity from the reference value, the greater the measurement noise.

[0073] For a successfully associated trajectory-detection pair (i,j), the observation vector is used. Perform a posteriori correction.

[0074] Preferably, an observation matrix H is first defined to map the 8-dimensional prediction space to the 4-dimensional observation space: ; Then, the measurement noise matrix R is adjusted based on the illumination intensity, representing the observation noise of the detection box itself:

[0075] in, To account for the detection uncertainty of each component. , This is the detection error coefficient; This is the light response sensitivity coefficient.

[0076] When the light Deviation from benchmark value The farther away, the more the exponential term increases. The larger the value, the amplified the measurement noise.

[0077] The state update process is as follows: calculate the measurement residual y and Kalman gain K, and update the posterior state. With covariance matrix :

[0078]

[0079]

[0080]

[0081] In poor lighting conditions, the measurement noise matrix R is amplified, the Kalman gain K decreases, and the filter trusts the predictions of the motion model more than unreliable detection observations, thereby improving tracking robustness.

[0082] After updating the trajectory, the direction of pedestrian movement needs to be determined. Specifically, a virtual counting line L:ax+by+c=0 is preset on the video image plane, and its unit normal vector pointing from the outside of the building to the inside is defined. .

[0083] For the tracked pedestrian trajectory, extract its trajectory points within k consecutive frames. Based on foot feature points (The midpoint of the bottom of the detection frame can be taken) Substitute it into the equation of the virtual counting line to calculate. ,like The system determines whether the target has crossed the virtual counting line. For the moment of crossing the line, a displacement vector is constructed. Calculate the inner product Cross = V·n. If Cross > 0, it is determined that a person has entered; if Cross < 0, it is determined that a person has left.

[0084] This allows for real-time accumulation of visual statistics on the number of people entering and leaving. As an alternative, the virtual counting line can be replaced by virtual area determination, which involves counting by determining whether a target moves from an entry area to a exit area.

[0085] In one embodiment, step S106 above, based on the tracking trajectory, access control service data, and ambient light intensity, performs a fusion correction on the visually counted number of people, including: calculating an environmental reliability factor based on the ambient light intensity, the environmental reliability factor being used to represent the reliability of the visual statistics under the current lighting conditions; using the environmental reliability factor to weight the difference between the net number of people entering and exiting obtained from the access control service data and the visually counted number of people, correcting the visually counted number of people, and outputting the final real-time number of people in the building.

[0086] After determining and counting the direction of entry and exit, the visual count of the number of people inside the building is updated in real time. :

[0087] in , These represent the number of people entering and leaving the building at time t, respectively.

[0088] To integrate access control business data, the number of people is corrected based on the access control statistics, and the final real-time number of people in the building is output. :

[0089] in, , These represent the number of people entering and exiting the building at time t, obtained through the access control system; η is the personnel accompanying coefficient, defined as the ratio of the total number of visual statistical observations within the historical time window T to the total number of access control card swipes, i.e.:

[0090] The normal proportion of cardless passage (tailgating / visitors) is used to compensate for the current time t. T is the set of historical time windows of length W before the current time t.

[0091] Environmental reliability factor Based on the calculation of light intensity, one implementation method is as follows:

[0092] Where κ and σ are given constants. When the light intensity is L... t When it is close to the reference value L0, When the light intensity approaches 0, visual statistics are reliable, and the weight of the correction term is small; when the illumination deviates from the baseline value... As the weight of the correction term increases, the fusion result becomes more dependent on the access control data.

[0093] As an alternative, the fusion of access control data and visual data is not limited to the above modified formula. It can be replaced by state fusion using a distributed Kalman filter, or confidence fusion can be performed by using evidence theory to treat the two sets of data as sources of evidence for mutual verification.

[0094] In another embodiment, the fusion correction in step S106 above may further include: obtaining the number of people who have not left the building at present, obtained from the access control service data; dynamically adjusting a preset reset threshold based on the cumulative deviation trend between the visually counted number of people and the access control service data over a historical time period; and comparing the final real-time number of people in the building with the current number of people who have not left the building during a preset silent period. If the difference exceeds the dynamically adjusted reset threshold, the current number of people who have not left the building is used as the reset value of the final real-time number of people in the building.

[0095] As a preferred implementation, the system continuously records the deviation between visually counted people and access control data, and calculates the cumulative trend of the deviation over a historical period (e.g., through a moving average or cumulative sum control chart). When the cumulative deviation shows a continuously increasing trend, the reset threshold is automatically lowered to trigger calibration in advance; conversely, the reset threshold is increased to avoid unnecessary resets.

[0096] The preset silent period can be the daily early morning silent period (e.g., 3:00 AM). During this period, due to extremely low personnel entry and exit, visual statistics and access control data should be highly consistent. Compare this with the currently output real-time number of people inside the building. The number of people who have not left the building as recorded by the access control system. (i.e., the number of cardholders who have not yet left the area, and the difference between the two exceeds the dynamically adjusted reset threshold). ( If the cumulative error threshold is used, then forced execution will be performed. This eliminates the cumulative error of visual statistics by using access control data as an anchor point.

[0097] As an alternative, the silent period can also be automatically identified by the system based on the frequency of people entering and exiting in the real-time video stream: when the frequency of people entering and exiting is detected to be continuously lower than the preset activity threshold and maintained for a preset duration, the above comparison and reset are triggered.

[0098] By introducing a multimodal perception and adaptive fusion mechanism, this invention can effectively overcome the interference of complex lighting on image quality and algorithm performance, and improve the robustness of target detection and tracking. By intelligently fusing visual statistical results with access control business data, errors are dynamically corrected, significantly improving the long-term accuracy and reliability of people flow statistics.

[0099] To facilitate understanding, this invention also provides an application example of a building entry and exit pedestrian flow statistics method based on multimodal perception. Specifically, this invention proposes a building entry and exit pedestrian flow statistics method and system that uses visual multimodal perception as the core and combines environmental modal data and access control gate business modal data for dynamic correction. This method dynamically adjusts image preprocessing parameters, detection thresholds, and Kalman filter noise by introducing ambient light intensity as a feedforward control variable, dynamically adjusts tracking weights through personnel density, and finally uses an environmental reliability factor to achieve nonlinear closed-loop calibration of visual statistics and access control data. The system mainly consists of a multimodal perception layer (including surveillance cameras, ambient light sensors, and access control controllers), a multimodal data processing module (including target detection, trajectory tracking, and cross-line determination), and a business application module.

[0100] See Figure 2 The diagram shows another method for counting pedestrian traffic in and out of buildings based on multimodal perception. The specific implementation of this method mainly includes the following steps S201 to S206: Step S201: Acquisition and preprocessing of video stream and environmental modal data; S1.1: Relying on the building video security system, high-definition cameras at the entrances and exits of the office building are used to acquire data. Moment video frame .

[0101] S1.2: Obtain the light intensity of the corresponding monitoring area through a light sensor. .

[0102] S1.3: Constructing adaptive gamma correction coefficients based on the Logistic function:

[0103] in, The optimal imaging reference illumination value collected by the illumination sensor; This represents the maximum Gamma compensation value under low illumination. This represents the minimum Gamma compression value under strong backlighting. This is the response sensitivity coefficient.

[0104] S1.4: Perform non-linear brightness transformation on the video frames and output the preprocessed video frames. : .

[0105] Step S202, Multimodal perception model training; S2.1: Fine-tuning of object detection models: Fine-tuning object detection models (such as YOLO, Faster R-CNN, etc.) using a large-scale labeled dataset containing office building-specific scenes (such as turnstiles, revolving doors, columns, and glass curtain wall backgrounds).

[0106] S2.2: Pre-training of the Appearance Feature Extraction Network: A deep convolutional neural network was used for pre-training on a pedestrian dataset at building entrances and exits. A triplet loss function was used to constrain the model, enabling it to learn highly discriminative appearance representations that are robust to changes in lighting, including clothing color, texture, and human body shape.

[0107] Step S203, pedestrian target detection; S3.1: ROI clipping: Based on the layout of the building entrances and exits, preset ROI parameters and process only the images within the key monitoring range to eliminate visual noise from non-passage areas in the background (such as rest areas and decorative walls).

[0108] S3.2: Using the multimodal perception model trained in step S202, the ROIs are cropped. Inference is performed to achieve object detection, transforming each frame of image into a structured set of pedestrian features, and outputting a detection set. Each detection box Include: Position vector ,in The coordinates of the center point of the detection box. and To detect the width and height of the frame; Confidence ; Foot feature point location For determining if a line is crossed, the bottom midpoint of the detection box can be used. express; The appearance feature vector is extracted using a pre-trained appearance feature extraction network. It represents visual modal information such as clothing color, texture, and human body shape.

[0109] S3.3: Regarding the changes in light and shadow at building entrances and exits, in order to achieve a balance between high-light noise and low-light missed detections, the light intensity of the corresponding monitoring area is used... Dynamically adjust the confidence threshold of the pedestrian target detector :

[0110] in, The target detection baseline confidence threshold; This represents the maximum allowable fluctuation range for the confidence threshold. The reference light intensity; This is the illumination adjustment factor. Only the confidence level is retained. Greater than the confidence threshold The target detection results.

[0111] S3.4: Non-maximum suppression: Sort candidate detection boxes in descending order of confidence, calculate the intersection-union ratio, and remove redundant detection boxes with an overlap higher than a preset threshold.

[0112] Step S204: Continuous tracking of multiple targets; S4.1: For each tracked target Built on Motion state vector at time 1 ,in , , , These represent the instantaneous rates of change of the corresponding variables. The prior predicted value of the target's state vector at time t+1 is calculated using the Kalman prediction equation. With prior prediction covariance matrix :

[0113]

[0114] in It is a symmetric positive definite matrix, which is the posterior covariance matrix at the current time t, representing the state vector. The estimation error of each component in the target will be considered when a new trajectory is created. The covariance matrix is ​​initialized as a diagonal matrix. :

[0115] in The diagonal elements are the preset initial uncertainty constants for each component.

[0116] The process noise matrix characterizes the uncertainty of the system's motion model itself.

[0117] in, For the uncertainty of the position component, To address the uncertainty of the velocity component, a method based on the detection frame height is adopted. Dynamic noise model calculation:

[0118]

[0119] in This is the position error coefficient. This is the speed error coefficient.

[0120] The state transition matrix is ​​defined as follows: .

[0121] S4.2: Calculate all detection boxes in the current frame. With all predicted targets state vector The comprehensive similarity matrix between , for 3D matrix This represents the total number of predicted target trajectories in the current frame. The total number of detection boxes in the current frame:

[0122]

[0123]

[0124]

[0125] in, The similarity of spatial location represents the consistency of spatiotemporal motion; The similarity of appearance feature vectors represents the consistency of visual identity. State vector The predicted bounding box is obtained by restoring the first 4 dimensions of the bounding box. , Detection boxes and prediction targets The corresponding appearance feature vector; These are modal weighting coefficients, based on the current personnel density. Make dynamic adjustments. The number of pedestrian targets detected can be determined by the number of pedestrian targets calculated in step S203; This serves as a baseline value for population density in the detection area; This is the personnel density adjustment coefficient. When congestion occurs at building entrances and exits, the personnel density increases, and trajectory tracking automatically shifts from relying on spatiotemporal motion characteristics to relying on visual appearance characteristics.

[0126] S4.3: Solve the following optimization problem using the branch and bound method: Define decision variables , Represents the detection box With the predicted target trajectory Matching successful and association established. Represents the detection box With the predicted target trajectory If no match is found, no association will be performed. The objective function is to maximize the global association similarity. :

[0127] The constraints are: (1) Uniqueness constraint (row constraint): Each detection box can be assigned to at most one trajectory to prevent multiple trajectories from competing for the same detection target:

[0128] (2) Consistency constraint (column constraint): Each trajectory can only match at most one detection box to prevent ID splitting.

[0129] S4.4: State Update and Cross-Modal Parameter Correction: For successfully associated trajectory detection pairs Using observation vectors Perform a posteriori correction: S4.4.1: Define the observation matrix Used to map an 8-dimensional prediction space to a 4-dimensional observation space.

[0130] S4.4.2: Adjusting the measurement noise matrix based on illumination intensity This indicates the noise of the detection frame itself.

[0131] in, To account for the detection uncertainty of each component. , This is the detection error coefficient; This is the light response sensitivity coefficient.

[0132] S4.4.3: Calculate measurement residuals Kalman gain Update post-verification status With covariance matrix :

[0133]

[0134]

[0135]

[0136] Step S205: Direction determination based on virtual counting lines and vector analysis; S5.1: Virtual Counting Line Definition: Define a virtual counting line in the video frame image plane. And define the unit normal vector of the virtual counting line. The direction is from the outside of the building inwards.

[0137] S5.2: Trajectory Acquisition: Extracting Tracked Targets In continuous Intra-frame trajectory .

[0138] S5.3: Cross-line detection: Calculate target foot feature points Substitute the value of the virtual counting line equation .like This indicates that the target has crossed the virtual counting line.

[0139] S5.4: Direction Determination: For the moment of crossing the line Construct the displacement vector Calculate the inner product of the target trajectory displacement vector and the virtual counting line direction vector. .like 0 indicates entry is permitted; if It was determined that the person had left.

[0140] Step S206: The number of people entering and leaving the building and inside the building is updated in real time; S6.1: Real-time update of the number of people currently inside the building. :

[0141] in , They are respectively The number of people entering and leaving Xiamen at any given time.

[0142] S6.2: Adjust the number of people based on access control statistics:

[0143] in This is the final output showing the real-time number of people inside the building. , They were obtained through the access control system. The number of people entering and leaving Xiamen at any given time; The accompanying personnel coefficient is defined as follows: ; That is, sliding windows in historical time. The ratio of internal visual statistical observations to access control card swipes is used to compensate for the normal proportion of cardless access (tailgating / visitors). For the current moment The previous length was A collection of historical time windows; Environmental reliability factors based on light intensity:

[0144] in, and is a given constant.

[0145] S6.3: Zero-point reset calibration: Utilizing the daily early morning quiet period (e.g., 03:00), compare the number of people inside the building calculated through visual analysis algorithms. The number of people who have not left Xiamen obtained through the access control system ,like Then enforce ,in This is the cumulative error threshold.

[0146] The above-mentioned embodiments of the invention mainly solve the technical pain points of traditional people flow statistics methods based on computer vision when applied to building entrance and exit scenarios, such as visual perception failure under complex light and shadow fluctuations, tracking ID jump caused by high-density crowding and occlusion, and difficulty in effectively complementing visual statistics methods and access control gate data.

[0147] Specifically, by introducing light sensor data to implement end-to-end feedforward control of the Gamma correction coefficient, target detection threshold, and Kalman filter measurement noise, the problem of missed detections and false alarms in office buildings under strong backlight and low nighttime illumination is effectively solved. By dynamically adjusting the similarity matrix weights based on personnel density, fast tracking is achieved by relying on IOU when the flow of people is sparse, and automatically switching to relying on appearance features when the flow of people is crowded and obstructed. Combined with state vector prediction, ID switching and duplicate counting are significantly reduced during peak periods. By constructing a nonlinear fusion model of visual statistics and access control data, and utilizing environmental confidence factors, the advantages of computer vision in perceiving tailgating personnel are preserved when the visual environment is ideal, and the accumulated error is forcibly pulled back by the access control ground truth when the visual environment is poor, achieving zero drift in long-term operation.

[0148] Based on the same inventive concept, this invention also provides a building entry and exit pedestrian flow statistics system based on multimodal perception, see [link to relevant documentation]. Figure 3 As shown, the system mainly includes the following parts: The data acquisition module 310 is used to acquire real-time video streams from building entrances and exits, real-time ambient light intensity of the corresponding monitoring area, and access control business data. The parameter adjustment module 320 is used to dynamically adjust the parameters of the multimodal perception model based on the ambient light intensity mentioned above, and to generate the pedestrian tracking trajectory and the number of people in the building as visually counted. The data calibration module 330 is used to calibrate the real-time number of people in the building based on the above-mentioned tracking trajectory, access control business data and ambient light intensity, and output the final number of people in the building.

[0149] The building access pedestrian flow statistics system based on multimodal perception provided in this invention embodiment can be specific hardware on a device or software or firmware installed on the device. The system provided in this invention embodiment has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the system embodiment section can be referred to the corresponding content in the aforementioned method embodiment. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0150] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, specifically, the electronic device includes a processor and a storage device; the storage device stores a computer program, and the computer program, when run by the processor, executes the method described in any of the above embodiments.

[0151] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device 400 includes: a processor 410, a memory 420, a communication interface 430, and a bus 440. The memory 420 stores machine-readable instructions that can be executed by the processor 410. When the electronic device is running, the processor 410 communicates with the memory 420 through the bus 440. The processor 410 executes the machine-readable instructions to perform the steps of the method described above.

[0152] Specifically, the memory 420 and processor 410 can be general-purpose memory and processor, without any specific limitations. When the processor 410 runs the computer program stored in the memory 420, it can execute the above method.

[0153] Processor 410 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 410 or by instructions in software form. The processor 410 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 420, and processor 410 reads the information from memory 420 and, in conjunction with its hardware, completes the steps of the above method.

[0154] Corresponding to the above method, this embodiment of the invention also provides a computer-readable storage medium storing machine-executable instructions. When the computer-executable instructions are called and run by a processor, the computer-executable instructions cause the processor to perform the steps of the above method.

[0155] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0156] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0157] Furthermore, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0158] It should be noted that if the functionality is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0159] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0160] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for counting pedestrian flow in and out of buildings based on multimodal perception, characterized in that, include: Acquire real-time video streams from building entrances and exits, real-time ambient light intensity of the corresponding monitored areas, and access control business data; The processing parameters of the visual perception process are dynamically adjusted based on the ambient light intensity, and a multimodal feature fusion strategy is adaptively configured based on the current population density to process the real-time video stream and generate pedestrian tracking trajectories and visual population statistics. Based on the tracking trajectory, the access control data, and the ambient light intensity, the visually counted number of people is fused and corrected to output the final real-time number of people inside the building.

2. The method according to claim 1, characterized in that, The processing parameters for dynamically adjusting the visual perception stage include: determining an image enhancement coefficient based on the ambient light intensity using a preset nonlinear mapping function, and using the image enhancement coefficient to perform brightness transformation on the video frames of the real-time video stream; wherein the image enhancement coefficient varies with the degree to which the light intensity deviates from the reference light value.

3. The method according to claim 1 or 2, characterized in that, The processing parameters for dynamically adjusting the visual perception stage include: dynamically adjusting the confidence threshold of the target detector based on the ambient light intensity using a preset nonlinear mapping function.

4. The method according to claim 1, characterized in that, The adaptive configuration multimodal feature fusion strategy based on the current personnel density includes: The population density is determined based on the number of pedestrian targets detected in the current frame of the real-time video stream; The fusion weight is calculated using the personnel density through a preset mapping relationship; the fusion weight is used to weight and combine motion feature similarity and appearance feature similarity to generate a comprehensive similarity between the detection box and the predicted trajectory for tracking association.

5. The method according to claim 4, characterized in that, The generation of pedestrian tracking trajectories includes: The motion state of pedestrians is predicted and updated using a Kalman filter, wherein the measurement noise matrix of the Kalman filter is dynamically adjusted according to the ambient light intensity; the greater the deviation of the light intensity from the reference value, the greater the measurement noise.

6. The method according to claim 1, characterized in that, The step of fusing and correcting the visual headcount based on the tracking trajectory, the access control service data, and the ambient light intensity includes: An environmental reliability factor is calculated based on the ambient light intensity, and the environmental reliability factor is used to represent the reliability of visual statistics under the current lighting conditions; The difference between the net number of people entering and exiting obtained from the access control business data and the visually counted number of people is weighted using the environmental reliability factor, the visually counted number of people is corrected, and the final real-time number of people in the building is output.

7. The method according to claim 6, characterized in that, The fusion correction also includes: Obtain the number of people who have not left the building at the moment, as determined by the access control service data; The preset reset threshold is dynamically adjusted based on the cumulative deviation trend between the visually counted number of people and the access control business data over a historical period. During a preset quiet period, the real-time number of people in the final building is compared with the number of people who have not left the current building. If the difference exceeds the dynamically adjusted reset threshold, the number of people who have not left the current building is used as the reset value of the real-time number of people in the final building.

8. A building entry and exit pedestrian flow statistics system based on multimodal perception, characterized in that, include: The data acquisition module is used to acquire real-time video streams from building entrances and exits, real-time ambient light intensity of the corresponding monitoring area, and access control business data. The parameter adjustment module is used to dynamically adjust the processing parameters of the visual perception link based on the ambient light intensity, and to adaptively configure the multimodal feature fusion strategy based on the current personnel density to process the real-time video stream and generate pedestrian tracking trajectories and visual counts of people. The data calibration module is used to fuse and correct the visually counted number of people based on the tracking trajectory, the access control business data, and the ambient light intensity, and output the final real-time number of people in the building.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images