Entrance door passenger flow detection and defense organization method and device based on multi-modal feature fusion

Abstract

The invention relates to the field of security monitoring, in particular to a multi-modal feature fusion entrance door passenger flow detection and deployment method and device, and the method comprises the steps: carrying out the space-time alignment of a preprocessed moving target feature and an image feature, determining a dynamic weight parameter based on a preset factor, and carrying out the detection of the passenger flow of an entrance door based on a space-time alignment result and the dynamic weight parameter. Determining a fusion feature vector after space-time alignment; and performing target association and access direction judgment based on the fusion feature vector, performing statistics on effective passenger flow of an entrance door according to a preset rule, determining an abnormal behavior judgment result of a target, and executing defense organization or defense withdrawing operation of the entrance door. According to the embodiment of the invention, the method combines the advantages of the millimeter wave radar and the visual sensor, and improves the detection accuracy and reliability in a complex environment through a dynamic weight fusion algorithm.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of security monitoring, in particular to a multi-modal feature fusion-based visitor flow detection and defense method and device for a front door. BACKGROUND

[0002] With the continuous development and intelligent upgrading of security monitoring technology, the security protection demand for the front door area of enterprise, family and other scenarios is increasing.

[0003] However, the single sensor solution in the related art has obvious limitations: the camera is easily affected by light, resulting in a decrease in night detection accuracy; although the millimeter wave radar can stably obtain motion parameters, it cannot obtain the appearance information of the target, and it is difficult to accurately distinguish different target individuals in complex environments. Therefore, how to perform visitor flow detection and defense for the front door through the fusion of multiple sensors, so as to improve the accuracy and reliability of security monitoring, has become a problem to be solved. SUMMARY

[0004] Therefore, the present disclosure provides a multi-modal feature fusion-based visitor flow detection and defense method and device for a front door, to solve the problem of how to perform visitor flow detection and defense for the front door through the fusion of multiple sensors, so as to improve the accuracy and reliability of security monitoring.

[0005] In one aspect, the present disclosure provides a multi-modal feature fusion-based visitor flow detection and defense method for a front door, the method comprising:

[0006] acquiring motion target features of the front door area through a millimeter wave radar, acquiring image features of the front door area through a vision sensor, and acquiring an opening and closing state of the front door through a door state sensor; and preprocessing the motion target features and the image features;

[0007] spatially and temporally aligning the preprocessed motion target features and image features, determining dynamic weight parameters corresponding to the motion target features and the image features based on a preset factor, and determining a fusion feature vector after spatial and temporal alignment based on the spatial and temporal alignment result and the dynamic weight parameters;

[0008] performing target association and entry and exit direction judgment based on the fusion feature vector, counting effective visitor flow of the front door according to a preset rule, and determining an abnormal behavior judgment result of the target;

[0009] dynamically adjusting a defense threshold according to a preset time period and historical visitor flow data, and performing a defense or disarm operation of the front door based on the defense threshold, actual visitor flow data, the opening and closing state of the front door, and the abnormal behavior judgment result.

[0010] In another aspect, the present disclosure also provides a multi-modal feature fusion-based visitor flow detection and defense device for a front door, the device comprising:

[0011] The multimodal feature acquisition module is used to acquire moving target features in the entrance door area through millimeter-wave radar, acquire image features in the entrance door area through a vision sensor, acquire the opening and closing status of the entrance door through a door status sensor, and preprocess the moving target features and image features.

[0012] The feature alignment and fusion module is used to perform spatiotemporal alignment of preprocessed moving target features and image features. It determines the dynamic weight parameters corresponding to the moving target features and image features based on preset factors, and determines the spatiotemporal aligned fused feature vector based on the spatiotemporal alignment result and the dynamic weight parameters.

[0013] The effective passenger flow detection module is used to perform target association and entry / exit direction judgment based on fused feature vectors, count the effective passenger flow at the entrance door according to preset rules, and determine the abnormal behavior judgment results of the target.

[0014] The arming / disarming execution module is used to dynamically adjust the arming threshold based on preset time periods and historical passenger flow data. Based on the arming threshold, actual passenger flow data, the opening and closing status of the entrance door, and the judgment results of abnormal behavior, it executes the arming or disarming operation of the entrance door.

[0015] This disclosure also provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the above-mentioned multimodal feature fusion method for detecting and deploying entrance passenger flow.

[0016] This disclosure also provides a computer-readable storage medium storing computer instructions for enabling a computer to implement the aforementioned multimodal feature fusion method for detecting and deploying entrance passenger flow.

[0017] This disclosure also provides a computer program product, including computer instructions for causing a computer to execute the above-described method for multimodal feature fusion-based entryway passenger flow detection and deployment.

[0018] The multimodal feature fusion-based entrance passenger flow detection and deployment method and apparatus of the above embodiments of this disclosure combine the advantages of millimeter-wave radar and visual sensors. Through dynamic weight fusion algorithm, it avoids the problem of low detection accuracy of a single sensor in scenarios such as night and strong light, and improves the accuracy and reliability of detection in complex environments.

[0019] Furthermore, by leveraging the preprocessing of moving target features and image features, spatiotemporal alignment, and dynamic weight fusion, the consistency and correlation of multimodal data are improved, enabling the fused feature vector to more accurately reflect the actual state of the target, thereby further enhancing the accuracy and security of security monitoring. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a multimodal feature fusion-based method for detecting and deploying passenger flow at a residential entrance, as provided in an embodiment of this disclosure.

[0022] Figure 2 This is a schematic diagram of another multimodal feature fusion-based entrance door passenger flow detection and deployment device provided in this embodiment of the present disclosure;

[0023] Figure 3 This is a schematic diagram of another multimodal feature fusion-based entrance door passenger flow detection and deployment device provided in this embodiment. Detailed Implementation

[0024] With the deep integration of security monitoring technology with IoT and AI technologies, the security requirements for key entrances and exits in businesses and homes have evolved from traditional passive monitoring to proactive sensing and intelligent decision-making. Beyond basic intrusion alarm functions, these systems need to accurately track entry and exit flows, identify abnormal behavior in real time, and dynamically adjust security strategies based on the specific scenario. For example, home scenarios need to adapt to a daytime no-attendance, nighttime alert mode, while business scenarios need to match the differences in customer flow between peak and off-peak hours. This places higher demands on the multi-source data fusion capabilities and adaptive decision-making abilities of security systems.

[0025] However, existing technologies have significant shortcomings in meeting the above requirements:

[0026] On the one hand, in solutions that rely on a single sensor, visual sensors are easily affected by lighting and occlusion (such as blurred images at night or loss of targets due to backlighting), making it impossible to reliably obtain the motion parameters of the target. Although millimeter-wave radar can stably output motion characteristics in complex environments, it lacks spatial detail information (such as whether the target is a human body or whether it has completely passed through the door frame), resulting in large errors in passenger flow statistics and behavior judgment.

[0027] On the other hand, some multi-sensor fusion solutions have not addressed core technical pain points: either they have not strictly aligned radar and visual data in time and space, resulting in incorrect association of the same target in multimodal data; or the fusion weights are fixed, making it impossible to dynamically adjust the degree of sensor dependence according to environmental changes, which makes it difficult to meet the precise security needs in diverse scenarios.

[0028] To address the aforementioned issues, various embodiments of this disclosure provide a multimodal feature fusion-based method for detecting and deploying passenger flow at an entrance door. The method includes: acquiring moving target features in the entrance door area using millimeter-wave radar; acquiring image features in the entrance door area using a visual sensor; acquiring the opening / closing status of the entrance door using a door status sensor; preprocessing the moving target features and image features; performing spatiotemporal alignment on the preprocessed moving target features and image features; determining dynamic weight parameters corresponding to the moving target features and image features based on preset factors; determining a spatiotemporally aligned fused feature vector based on the spatiotemporal alignment result and the dynamic weight parameters; performing target association and entry / exit direction judgment based on the fused feature vector; statistically analyzing the effective passenger flow at the entrance door according to preset rules; and determining the abnormal behavior judgment result of the target; dynamically adjusting the deployment threshold based on preset time periods and historical passenger flow data; and executing the deployment or disarming operation of the entrance door based on the deployment threshold, actual passenger flow data, the opening / closing status of the entrance door, and the abnormal behavior judgment result.

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0030] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a multimodal feature fusion-based method for detecting and deploying security measures at a residential entrance, as provided in this embodiment. The method may include the following steps:

[0031] Step S101: Obtain moving target features in the entrance door area using millimeter-wave radar, obtain image features in the entrance door area using a visual sensor, obtain the opening and closing status of the entrance door using a door status sensor, and preprocess the moving target features and image features.

[0032] In this embodiment, millimeter-wave radar is used to transmit millimeter-wave signals and receive echoes, enabling the measurement of motion parameters such as target distance, velocity, and angle. Millimeter-wave radar is characterized by strong resistance to light and obstruction, making it suitable for moving target detection in complex environments.

[0033] Here, the model of the millimeter-wave radar can be selected according to actual needs, and there is no specific limitation. For example, the Texas Instruments AWR1843 model can be selected.

[0034] In addition, the installation location of the millimeter-wave radar can be determined based on the height of the door frame to ensure that the installed millimeter-wave radar can cover a horizontal angle of -60° to 60° and a vertical angle of -30° to 30°.

[0035] A millimeter-wave radar installed at the entrance door frame acquires a set of features of moving targets within the entrance door area. Here, the geometry of the moving target features can include quantitative parameters such as the target's distance from the radar, its speed, azimuth angle, and trajectory. The target can refer to a person within the entrance door area.

[0036] The visual sensor is used to collect image features of the entrance door area. These image features may include, but are not limited to, spatial features such as the door frame boundary, the position and shape of people in the area, etc.

[0037] Here, the choice of vision sensor model can be set according to actual needs, and no specific limitation is made. For example, a TSC-WL001-AHD2 camera with a minimum illumination of 0.001 illuminance and a resolution of 1920×1080 can be selected.

[0038] Furthermore, the installation position of the visual sensor maintains a certain horizontal distance from the installation position of the millimeter-wave radar.

[0039] In one possible implementation, the door status sensor can be a Honeywell reed switch, which is installed at the contact point between the door frame and the door leaf of the entrance door to ensure that a signal is triggered when the door is closed.

[0040] Furthermore, the moving target features and image features are preprocessed to remove redundant information and transform the raw data into a structured feature form, ensuring that the features from different sensors can be efficiently fused in subsequent steps.

[0041] Step S102: Spatiotemporally align the preprocessed moving target features with the image features, determine the dynamic weight parameters corresponding to the moving target features and the image features based on preset factors, and determine the spatiotemporally aligned fused feature vector based on the spatiotemporal alignment result and the dynamic weight parameters.

[0042] In this embodiment, the moving target features acquired by the millimeter-wave radar and the image features acquired by the visual sensor are unified into the same spatiotemporal coordinate system through coordinate mapping, time synchronization and other methods, so as to ensure that the same physical target corresponds to the same spatiotemporal position in different sensor data.

[0043] Here, the moving target features and image features are spatiotemporally aligned to ensure that the data has spatiotemporal consistency before fusion.

[0044] Furthermore, the preset factors are a set of parameters that affect the dynamic weight allocation, which can be used to quantitatively evaluate the reliability of each sensor in the current scenario.

[0045] Here, preset factors may include, but are not limited to, ambient light intensity, the distance between the target and the sensor, the sensor's detection confidence level, and the target's speed. For example, the weight of millimeter-wave radar features may be increased in nighttime scenes, while the weight of visual features may be enhanced when the target is at close range.

[0046] Furthermore, based on the spatiotemporal alignment results and dynamic weight parameters, the moving target features and image features are weighted and fused to generate a fused feature vector containing multi-dimensional information. Here, the fused feature vector combines the advantages of both sensors, and can more comprehensively describe the target state in the entrance door area.

[0047] Step S103: Based on the fused feature vector, target association and entry / exit direction judgment are performed. The effective passenger flow at the entrance is counted according to preset rules, and the abnormal behavior judgment result of the target is determined.

[0048] In this embodiment, by fusing feature matching of moving targets and visual targets in the feature vector, the same target perceived by different sensors is associated, ensuring continuous tracking of the same target and avoiding target recognition confusion caused by the heterogeneity of multi-sensor data.

[0049] By combining the target's movement trajectory and changes in spatial location, it can be determined whether the target is entering or leaving the entrance area, providing a directional basis for passenger flow statistics.

[0050] Based on preset rules, valid passenger flow is selected from all detected targets as actual passenger flow data.

[0051] Here, effective passenger flow refers to targets that meet the preset rules and are judged to be normal visitors entering and exiting the entrance; preset rules refer to the criteria used to screen effective passenger flow.

[0052] For example, preset rules may include, but are not limited to: whether the target has completely entered or left the door frame area, whether the movement trajectory is continuous and uninterrupted, and whether the dwell time conforms to normal entry and exit logic.

[0053] Furthermore, based on the analysis of the target's motion state and dwell characteristics using fused feature vectors, abnormal situations that do not conform to normal behavior patterns are identified. If an abnormal situation exists in the target, the abnormal behavior of the target is determined to be abnormal.

[0054] For example, abnormal situations may include, but are not limited to: rapid intrusion, prolonged stay, etc.

[0055] Step S104: Dynamically adjust the deployment threshold based on the preset time period and historical passenger flow data. Based on the deployment threshold, actual passenger flow data, the opening and closing status of the entrance door, and the judgment result of abnormal behavior, perform the deployment or disarming operation of the entrance door.

[0056] In this embodiment, the deployment threshold adapted to the current scenario is dynamically calculated based on the preset time period division and the regularity of historical passenger flow data, so as to avoid the rigidity of the strategy caused by the fixed threshold.

[0057] The preset time period refers to a time interval pre-divided according to security needs, with different risk levels and deployment strategies corresponding to different time intervals to adapt to the differences in scenarios over time. Historical passenger flow data refers to the statistical data of effective passenger flow at the entrance door within a preset number of days.

[0058] By comprehensively monitoring real-time passenger flow data, the opening and closing status of entrance doors, and the results of abnormal behavior judgments, and comparing them with dynamically adjusted deployment thresholds, it is determined whether the current scenario requires activation or deactivation of the defense.

[0059] Here, based on the judgment result, when the arming conditions are met, the arming operation of the entrance door is executed, triggering the security measures; when the disarming conditions are met, the disarming operation is executed, deactivating the security measures.

[0060] For example, security measures may include, but are not limited to: activating audible and visual alarms, triggering video recording, linking door locks to enhance protection, and notifying homeowners.

[0061] The multimodal feature fusion-based entrance door passenger flow detection and deployment method and apparatus of the above embodiments of this disclosure combine the advantages of millimeter-wave radar and visual sensors. Through a dynamic weight fusion algorithm, it avoids the low detection accuracy problem of single sensors in scenarios such as nighttime and strong light, improving the accuracy and reliability of detection in complex environments. By leveraging the preprocessing, spatiotemporal alignment, and dynamic weight fusion of moving target features and image features, the consistency and correlation of multimodal data are enhanced, enabling the fused feature vector to more accurately reflect the actual state of the target, further improving the accuracy and security of security monitoring. Based on the fused feature vector, target association and direction judgment are achieved. Effective passenger flow is filtered through preset rules to avoid invalid interference. Simultaneously, abnormal behavior is identified by combining target movement status and dwell characteristics, significantly improving the accuracy of passenger flow statistics and the ability to identify abnormal risks. The deployment threshold is dynamically adjusted according to preset time periods and historical passenger flow data. Combined with real-time passenger flow, door status, and abnormal results, intelligent adaptation of deployment / disarming is achieved. This avoids false alarms and missed alarms caused by fixed thresholds and balances security reliability with ease of use in various scenarios.

[0062] In one possible implementation of step S101 above, preprocessing of the moving target features and image features includes:

[0063] Raw point cloud data, which serves as features of moving targets, is acquired from millimeter-wave radar. The raw point cloud data is then clustered to extract the trajectory of the moving targets. The raw point cloud data includes at least one of the following: distance, velocity, and angle.

[0064] Image features are acquired from a visual sensor, and the U-Net network is used to segment the door frame of the entrance door in the image features to determine the set of door frame pixel coordinates; the set of door frame pixel coordinates is used to define the physical boundary of the entrance door.

[0065] An improved YOLOv5 model is used to detect human bodies in image features and determine the coordinates of human body bounding boxes.

[0066] In this embodiment, raw point cloud data of the target is acquired in real time from millimeter-wave radar. The raw point cloud data includes at least: range (d), velocity (v), and angle (θ) data. The raw point cloud data is then clustered to extract the trajectory of the moving target.

[0067] Among them, the trajectory T of the moving target r It includes n trajectory points; d i v represents the distance between the target corresponding to the i-th trajectory point and the millimeter-wave radar; i Let θ be the velocity corresponding to the i-th trajectory point; i Let t be the angle of the target relative to the millimeter-wave radar corresponding to the i-th trajectory point; i Let be the timestamp corresponding to the i-th trajectory point.

[0068] Image features are acquired from a visual sensor, and a U-Net network is used to segment the door frame of the entrance door in the image features to determine the set of pixel coordinates of the door frame.

[0069] Among them, S d Let x be the set of pixel coordinates of the door frame. j ,y j ) represents the coordinates of the j-th pixel within the door frame region in the image coordinate system; m represents the total number of pixels contained within the door frame region.

[0070] The U-Net network is an encoder-decoder architecture based on convolutional neural networks that can accurately delineate the boundaries of different objects in an image.

[0071] In one possible implementation, the method may also include: pre-training the U-Net network.

[0072] Specifically, a large number of entrance door images are collected from different scenes (such as day or night, different lighting conditions, and different door frame styles). The door frame area in each entrance door image is labeled, and the labeling results are used as the supervision signal for training. The U-Net network is trained using the cross-entropy loss function.

[0073] For example, during training, the training parameters can be specifically set to batch size equal to 16, learning rate = 0.001, and 50.

[0074] An improved YOLOv5 model is used to detect human bodies in image features and determine the coordinates B of the human body bounding box. v ={(x min ,y min ,x max ,y max ,c,s)}.

[0075] Where, x min ,y min The x-coordinate represents the pixel coordinates of the top-left corner of the bounding box. max ,y max represents the pixel coordinates of the bottom right corner of the bounding box; c represents the target's category confidence; and s represents the scale factor, used to reflect the target size.

[0076] The YOLOv5 model is a fast and efficient object detection algorithm that can identify different categories of objects in an image and provide bounding boxes for each object.

[0077] In one possible implementation, the method may also include: pre-improving the YOLOv5 model.

[0078] Specifically, target annotation data (such as people, packages, etc.) in the entrance door area are collected. The width / height of all targets are clustered using the K-means algorithm to obtain the set of anchor boxes that best represent the target sizes in the entrance door scene. This makes the anchor boxes more closely match the target sizes, reducing the burden of model learning offsets and improving the accuracy of bounding box prediction. The anchor boxes are used as initial guesses for target bounding boxes. A Spatial Pyramid Pooling-Fast (SPPF) module is inserted into the feature fusion path of the Neck section to enhance the feature extraction capability for small targets, addressing the problem of small, easily missed distant targets in the entrance door scene.

[0079] The multimodal feature fusion-based entrance door passenger flow detection and deployment method and apparatus of the above embodiments of this disclosure clusters and extracts trajectories from the original point cloud data, which can filter environmental interference, enhance the effectiveness of moving target features, and thus improve the accuracy of target detection. By obtaining the door frame coordinate set through pixel-level segmentation, the spatial boundary of the entrance door is clearly defined, allowing for more accurate judgment of effective passenger flow and reducing statistical errors caused by unclear boundaries.

[0080] In one possible implementation of step S102 above, the preprocessed moving target features and image features are spatiotemporally aligned; dynamic weight parameters corresponding to the moving target features and image features are determined based on preset factors; and the spatiotemporally aligned fused feature vector is determined based on the spatiotemporal alignment result and the dynamic weight parameters, including:

[0081] A transformation relationship is established between the radar coordinate system corresponding to the millimeter-wave radar and the image coordinate system corresponding to the vision sensor. The moving target features in the radar coordinate system are projected to the image coordinate system through the calibration matrix to determine the spatial coordinate mapping result. The calibration matrix is ​​obtained by Zhang Zhengyou calibration method.

[0082] By combining hardware triggering and software interpolation, the motion target features after spatial coordinate mapping are time-synchronized with the door frame pixel coordinate set and human body bounding box coordinates in the image features, and the time synchronization processing result is determined; among them, linear interpolation is used to compensate for the time misalignment of feature data.

[0083] The first weight parameter corresponding to the moving target features and the second weight parameter corresponding to the image features are calculated based on preset factors. The preset factors include ambient light intensity, target distance and movement speed. The sum of the first weight parameter and the second weight parameter is 1.

[0084] Based on the spatial coordinate mapping results, time synchronization processing results, first weight parameter and second weight parameter, the trajectory of the moving target is weighted and fused with the set of door frame pixel coordinates and the human body bounding box coordinates to determine the spatiotemporally aligned fused feature vector.

[0085] In this embodiment, a transformation relationship is established between the radar coordinate system corresponding to the millimeter-wave radar and the image coordinate system corresponding to the visual sensor, and the moving target features are projected onto the image coordinate system through the calibration matrix M.

[0086] The transformation relationship is as follows:

[0087]

[0088] Where M is the calibration matrix, obtained by Zhang Zhengyou calibration method; (x′,y′) are the transformed image coordinates, used to characterize the pixel position of the target in the image captured by the vision sensor.

[0089] Furthermore, the following linear interpolation method is used to compensate for the time-displaced feature data:

[0090]

[0091] The logic of linear interpolation is to assume that the feature data between two time points changes linearly with time (i.e., the rate of change of data is constant). Using the known t1 and t2 data, the virtual data at time t is calculated, so that the features collected at different times are aligned in the time dimension.

[0092] Furthermore, the first weight parameter w corresponding to the moving target features is calculated based on preset factors. r The second weight parameter w corresponding to the image features v =1-w r .

[0093] Specifically, the first weight parameter w r The formula is shown below:

[0094]

[0095] Where k1 is the light sensitivity coefficient, L is the ambient light intensity, and L th d is the illumination threshold; d is the target distance, d0 is the distance reference value; v is the target speed, v0 is the speed reference value.

[0096] Based on the spatial coordinate mapping results, time synchronization processing results, first weight parameter and second weight parameter, the trajectory of the moving target is weighted and fused with the set of door frame pixel coordinates and the human body bounding box coordinates to avoid the shortcomings of a single sensor in complex scenes.

[0097] For example, when the ambient light intensity is low, the reliability of visual sensors decreases because they are more susceptible to noise interference in this scenario, while millimeter-wave radar remains unaffected. Therefore, the second weighting parameter w corresponding to the visual sensor is reduced. v Increase the first weighting parameter w corresponding to millimeter-wave radar r .

[0098] The multimodal feature fusion-based entrance flow detection and deployment method and apparatus of the above embodiments of this disclosure accurately projects the moving target features of the radar coordinate system to the image coordinate system through a calibration matrix. This avoids the problem of inconsistent descriptions of the same target position by radar and visual sensors, ensuring a clear spatial correspondence between the moving target trajectory and the door frame and human body bounding box, reducing spatial mapping errors, and thus improving data consistency. Combining hardware triggering and linear interpolation effectively compensates for the temporal misalignment between radar and visual data, controlling the time deviation within a small range, ensuring temporal consistency of feature data within the same time window, and avoiding target association errors caused by time differences. Through dynamic adjustment of the first and second weighting parameters, the fused features maintain high reliability even in complex scenarios.

[0099] In one possible implementation of step S103 above, target association and entry / exit direction determination are performed based on the fused feature vector, effective passenger flow at the entrance is counted according to preset rules, and the abnormal behavior judgment result of the target is determined, including:

[0100] An improved Hungarian algorithm is used to perform cross-modal target association between the trajectory of moving targets in the fused feature vector and the coordinates of human bounding boxes. The association cost is calculated by combining Mahalanobis distance and visual feature cosine similarity to determine the same target after association.

[0101] Define the door plane normal vector of the entrance door, calculate the dot product of the velocity vector of the associated target and the door plane normal vector to determine the target's entry and exit direction, and combine the human body's bounding box coordinates with the human body's orientation to determine the entry and exit direction; where the human body's orientation is determined by OpenPose keypoint detection.

[0102] The effective passenger flow at the entrance is determined based on the number of targets that meet the preset rules. The preset rules include: the center point of the target's human bounding box passes through the door frame area corresponding to the set of pixel coordinates of the door frame; the target's movement trajectory is continuous and in the same direction; and the target's dwell time exceeds a preset threshold.

[0103] If a target satisfies both the preset rules and the preset abnormal behavior conditions, the abnormal behavior judgment result of the target is determined to be abnormal.

[0104] In this embodiment, the improved Hungarian algorithm is as follows:

[0105] D association =α·D M +(1-α)·D V

[0106] Among them, D association The association cost is used to measure whether the trajectory of a moving target matches the coordinates of the human bounding box; the smaller the value, the higher the matching degree. MD is the Mahalanobis distance, used to measure the degree of matching of spatial locations. M The smaller the value, the better the spatial location match; D V α is the visual feature cosine similarity, used to measure the similarity of appearance features; the smaller the value, the more similar the features. α is the weighting coefficient.

[0107] Furthermore, the formula for the Mahalanobis distance is as follows:

[0108]

[0109] Where, x v x represents the target spatial coordinates output by the vision sensor. r These are the target spatial coordinates output by the millimeter-wave radar.

[0110] The formula for cosine similarity of visual features is shown below:

[0111]

[0112] Among them, f v f is the target feature vector extracted by the visual sensor. r The target feature vector extracted by millimeter-wave radar; cosine(f v ,f r The cosine similarity between two feature vectors is denoted by , which ranges from -1 to 1. The closer the value is to 1, the more consistent the vector directions are.

[0113] Define the door plane normal vector, calculate the dot product of the target's velocity vector and the door plane normal vector to determine the target's entry / exit direction, including:

[0114] Define the door plane normal vector of the entrance door. Calculate the target velocity vector AND gate plane normal vector The dot product is used to determine the direction of entry and exit of the target:

[0115]

[0116] If the dot product is greater than 0, the direction of the target is entering; if the dot product is less than or equal to 0, the direction of the target is leaving.

[0117] Furthermore, OpenPose is used to detect key points on the human body (such as the shoulders and hips) and calculate the human body orientation angle θ; if the target's human body orientation angle θ is related to the target velocity vector... If the directions are consistent, then the direction judgment result is determined.

[0118] In one possible implementation, the preset abnormal behavior conditions may include, but are not limited to: rapid intrusion, prolonged loitering, or reverse flow. If a target satisfies both the preset rules and the preset abnormal behavior conditions, the abnormal behavior of the target is determined to be abnormal.

[0119] The multimodal feature fusion-based entrance flow detection and deployment method and apparatus of the above embodiments of this disclosure, by fusing Mahalanobis distance and visual feature cosine similarity through an improved Hungarian algorithm and combining dynamic weights to adapt to environmental differences, solves the association ambiguity caused by spatiotemporal misalignment and feature heterogeneity between radar and visual data, ensuring accurate matching of motion trajectories and human targets, especially in complex scenarios, significantly improving the association accuracy. By judging the physical trend of the velocity vector and the dot product of the door plane normal vector, and combining it with semantic verification of human orientation, the dual mechanism reduces directional misjudgment caused by complex target postures (such as turning sideways or turning back), making the entry / exit recognition more consistent with actual movement logic.

[0120] In one possible implementation of step S104 above, the deployment threshold is dynamically adjusted based on a preset time period and historical passenger flow data. Based on the deployment threshold, actual passenger flow data, the opening / closing status of the entrance door, and the result of abnormal behavior judgment, the deployment or disarming operation of the entrance door is performed, including:

[0121] The basic deployment threshold is adjusted based on the preset time period division results, and the adaptive deployment threshold is adjusted based on historical passenger flow data and preset safety factor. The maximum value between the basic deployment threshold and the adaptive deployment threshold is used as the deployment threshold.

[0122] If the actual passenger flow data corresponding to the effective passenger flow is not less than the deployment threshold, the status of the entrance door is closed and the duration is greater than the first closing time, and the abnormal behavior judgment result of the target is abnormal, the deployment operation of the entrance door is executed.

[0123] If the actual passenger flow data is 0, the entrance door is closed for a duration longer than the second closure duration, and the current time period is a low-risk period, the entrance door will be disarmed; wherein the second closure duration is longer than the first closure duration.

[0124] In this embodiment, the basic deployment threshold is adjusted according to the preset time period division results. The basic deployment threshold is as follows:

[0125]

[0126] Specifically, when the preset time period falls within the high-risk period of [8:00, 22:00), the basic deployment threshold N is... set (t)=N day When the preset time period belongs to other time periods, the basic deployment threshold N is adjusted. set (t)=N night .

[0127] It is understandable that the high-risk periods [8:00, 22:00] are only used as examples for illustration.

[0128] For example, N day It can be preset to 3, N night It can be preset to 1.

[0129] Furthermore, the adaptive deployment threshold N is adjusted based on historical passenger flow data and a preset safety factor. adaptive Based on N actual ≥max(N set (t),N adaptive ), set the basic deployment threshold N set (t) and adaptive deployment threshold N adaptive The maximum value in the range is used as the deployment threshold.

[0130] If the actual passenger flow data N corresponds to the effective passenger flow actual If the threshold is not less than the arming threshold, the entrance door is closed for a duration greater than the first closing time, and the target's abnormal behavior is judged as abnormal, then arm the entrance door.

[0131] If the actual passenger flow data N actual The value is 0, the entrance door is closed for a duration longer than the second closing time, and the current time period is a low-risk period (i.e., ), and perform the disarming operation on the entrance door; wherein, the second closing time is longer than the first closing time.

[0132] For example, the first shutdown duration can be set to 5 minutes, and the second shutdown duration can be set to 30 minutes.

[0133] The multimodal feature fusion-based entrance door passenger flow detection and deployment method and apparatus of the above embodiments of this disclosure achieves scene-adaptive security strategy by dynamically adjusting the deployment threshold and combining passenger flow data, door status and abnormal behavior to determine deployment / disarming under multiple conditions. This ensures that the deployment threshold conforms to the risk of the time period and historical patterns, and through strict verification of deployment / disarming conditions, it balances security and ease of use while accurately responding to risks, thereby improving the intelligence and reliability of entrance door security.

[0134] In one possible implementation of step S104 above, the basic deployment threshold is adjusted based on the preset time period division results, and the adaptive deployment threshold is adjusted based on historical passenger flow data and a preset safety factor, including:

[0135] The preset time period is divided into high-risk period and low-risk period; the basic deployment threshold for high-risk period is greater than the basic deployment threshold for low-risk period.

[0136] Record the effective passenger flow at the entrance door every day, and calculate historical passenger flow data based on the effective passenger flow over a preset number of days; the historical passenger flow data includes: the moving average and standard deviation over the preset number of days;

[0137] The adaptive deployment threshold is adjusted based on the moving average, standard deviation, and preset safety factor of the preset number of days.

[0138] In this embodiment, the preset time period is divided into a high-risk period t∈[8:00,22:00) and a low-risk period.

[0139] Based on the moving average of the preset number of days The standard deviation σ and the preset safety factor k2 are used to adjust the adaptive deployment threshold.

[0140] The multimodal feature fusion method and apparatus for detecting and deploying passenger flow at the entrance of a house through the above embodiments of this disclosure calculates an adaptive threshold based on the moving average, standard deviation and safety factor of a preset number of days. This not only fits the historical average passenger flow level, but also covers the passenger flow fluctuation range through the standard deviation, so that the threshold can dynamically adapt to different passenger flow differences and avoid misjudgment when the fixed threshold is used when the passenger flow fluctuates abnormally.

[0141] In one embodiment, a multimodal feature fusion-based entrance door passenger flow detection and deployment device 200 is provided, which corresponds one-to-one with the multimodal feature fusion-based entrance door passenger flow detection and deployment method described in the above embodiments. For example... Figure 2 As shown, the multimodal feature fusion-based entrance door passenger flow detection and arming device 200 includes a multimodal feature acquisition module 201, a feature alignment and fusion module 202, an effective passenger flow detection module 203, and an arming / disarming execution module 204. The detailed descriptions of each functional module are as follows:

[0142] The multimodal feature acquisition module 201 is used to acquire moving target features in the entrance door area through millimeter-wave radar, acquire image features in the entrance door area through a visual sensor, acquire the opening and closing status of the entrance door through a door status sensor, and preprocess the moving target features and image features.

[0143] The feature alignment and fusion module 202 is used to perform spatiotemporal alignment of the preprocessed moving target features and image features, determine the dynamic weight parameters corresponding to the moving target features and image features respectively based on preset factors, and determine the spatiotemporal aligned fused feature vector based on the spatiotemporal alignment result and the dynamic weight parameters.

[0144] The effective passenger flow detection module 203 is used to perform target association and entry / exit direction judgment based on fused feature vectors, count the effective passenger flow at the entrance door according to preset rules, and determine the abnormal behavior judgment result of the target.

[0145] The arming / disarming execution module 204 is used to dynamically adjust the arming threshold based on the preset time period and historical passenger flow data. Based on the arming threshold, actual passenger flow data, the opening and closing status of the entrance door, and the judgment results of abnormal behavior, it executes the arming or disarming operation of the entrance door.

[0146] In one embodiment, the multimodal feature acquisition module 201 is used to acquire raw point cloud data as features of moving targets from millimeter-wave radar, perform clustering processing on the raw point cloud data, and extract the trajectory of moving targets; wherein, the raw point cloud data includes at least one of the following: distance, velocity, and angle;

[0147] Image features are acquired from a visual sensor, and the U-Net network is used to segment the door frame of the entrance door in the image features to determine the set of door frame pixel coordinates; the set of door frame pixel coordinates is used to define the physical boundary of the entrance door.

[0148] An improved YOLOv5 model is used to detect human bodies in image features and determine the coordinates of human body bounding boxes.

[0149] In one embodiment, the feature alignment and fusion module 202 is used to establish the transformation relationship between the radar coordinate system corresponding to the millimeter-wave radar and the image coordinate system corresponding to the visual sensor. The moving target features in the radar coordinate system are projected to the image coordinate system through the calibration matrix to determine the spatial coordinate mapping result. The calibration matrix is ​​obtained by Zhang Zhengyou calibration method.

[0150] By combining hardware triggering and software interpolation, the motion target features after spatial coordinate mapping are time-synchronized with the door frame pixel coordinate set and human body bounding box coordinates in the image features, and the time synchronization processing result is determined; among them, linear interpolation is used to compensate for the time misalignment of feature data.

[0151] The first weight parameter corresponding to the moving target features and the second weight parameter corresponding to the image features are calculated based on preset factors. The preset factors include ambient light intensity, target distance and movement speed. The sum of the first weight parameter and the second weight parameter is 1.

[0152] Based on the spatial coordinate mapping results, time synchronization processing results, first weight parameter and second weight parameter, the trajectory of the moving target is weighted and fused with the set of door frame pixel coordinates and the human body bounding box coordinates to determine the spatiotemporally aligned fused feature vector.

[0153] In one embodiment, the effective passenger flow detection module 203 is used to perform cross-modal target association between the trajectory of moving targets in the fused feature vector and the coordinates of human bounding boxes using an improved Hungarian algorithm, and to calculate the association cost by combining Mahalanobis distance and visual feature cosine similarity to determine the same target after association;

[0154] Define the door plane normal vector of the entrance door, calculate the dot product of the velocity vector of the associated target and the door plane normal vector to determine the target's entry and exit direction, and combine the human body's bounding box coordinates with the human body's orientation to determine the entry and exit direction; where the human body's orientation is determined by OpenPose keypoint detection.

[0155] The effective passenger flow at the entrance is determined based on the number of targets that meet the preset rules. The preset rules include: the center point of the target's human bounding box passes through the door frame area corresponding to the set of pixel coordinates of the door frame; the target's movement trajectory is continuous and in the same direction; and the target's dwell time exceeds a preset threshold.

[0156] If a target satisfies both the preset rules and the preset abnormal behavior conditions, the abnormal behavior judgment result of the target is determined to be abnormal.

[0157] In one embodiment, the arming / disarming execution module 204 is used to adjust the basic arming threshold according to the preset time period division result, adjust the adaptive arming threshold according to historical passenger flow data and preset safety factor, and take the maximum value of the basic arming threshold and the adaptive arming threshold as the arming threshold.

[0158] If the actual passenger flow data corresponding to the effective passenger flow is not less than the deployment threshold, the status of the entrance door is closed and the duration is greater than the first closing time, and the abnormal behavior judgment result of the target is abnormal, the deployment operation of the entrance door is executed.

[0159] If the actual passenger flow data is 0, the entrance door is closed for a duration longer than the second closure duration, and the current time period is a low-risk period, the entrance door will be disarmed; wherein the second closure duration is longer than the first closure duration.

[0160] In one embodiment, the arming / disarming execution module 204 is used to divide a preset time period into a high-risk time period and a low-risk time period; wherein, the basic arming threshold for the high-risk time period is greater than the basic arming threshold for the low-risk time period.

[0161] Record the effective passenger flow at the entrance door every day, and calculate historical passenger flow data based on the effective passenger flow over a preset number of days; the historical passenger flow data includes: the moving average and standard deviation over the preset number of days;

[0162] The adaptive deployment threshold is adjusted based on the moving average, standard deviation, and preset safety factor of the preset number of days.

[0163] It should be noted that the multimodal feature fusion-based entrance door passenger flow detection and deployment device provided in the above embodiments is only illustrated by the division of the above-described program modules when implementing the corresponding multimodal feature fusion-based entrance door passenger flow detection and deployment method. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the above system can be divided into different program modules to complete all or part of the processing described above. In addition, the system provided in the above embodiments and the corresponding Figure 1 The embodiments of the methods shown belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0164] This disclosure also provides an electronic device having the above-described features. Figure 2 The device shown is a multimodal feature fusion-based entrance door passenger flow detection and deployment device.

[0165] Please see Figure 3 , Figure 3 This is a schematic diagram of another multimodal feature fusion-based entrance door passenger flow detection and deployment device provided in this disclosure embodiment, as shown below. Figure 3 As shown, the electronic device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 3 Take a processor 10 as an example.

[0166] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GPA), or any combination thereof.

[0167] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0168] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0169] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0170] The electronic device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.

[0171] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touch screen.

[0172] The electronic device also includes a communication interface for communicating with other devices or communication networks.

[0173] This disclosure also provides a computer-readable storage medium in which the methods described in this disclosure can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium after being downloaded over a network. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium may be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium may also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0174] A portion of this disclosure can be applied to computer program products, such as computer program instructions, which, when executed by a computer, can invoke or provide methods and / or technical solutions according to this disclosure through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, and installation package files. Accordingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions; the computer compiling the instructions and then executing the corresponding compiled program; the computer reading and executing the instructions; or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0175] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for detecting and deploying security measures for passenger flow at residential entrances using multimodal feature fusion, characterized in that, The method includes: The moving target features in the entrance door area are obtained by millimeter-wave radar, the image features of the entrance door area are obtained by a visual sensor, and the opening and closing status of the entrance door is obtained by a door status sensor. The moving target features and the image features are then preprocessed. The preprocessed moving target features and image features are spatiotemporally aligned. Dynamic weight parameters corresponding to the moving target features and image features are determined based on preset factors. Based on the spatiotemporal alignment result and the dynamic weight parameters, the spatiotemporally aligned fused feature vector is determined. Based on the fused feature vector, target association and entry / exit direction determination are performed. The effective passenger flow at the entrance door is statistically analyzed according to preset rules, and the abnormal behavior judgment result of the target is determined. The deployment threshold is dynamically adjusted based on the preset time period and historical passenger flow data. Based on the deployment threshold, actual passenger flow data, the opening and closing status of the entrance door, and the abnormal behavior judgment result, the deployment or disarming operation of the entrance door is executed.

2. The method according to claim 1, characterized in that, The preprocessing of the moving target features and the image features includes: Raw point cloud data, which serves as features of moving targets, is acquired from millimeter-wave radar. The raw point cloud data is then clustered to extract the trajectory of the moving targets. The raw point cloud data includes at least one of the following: distance, velocity, and angle. Image features are acquired from a visual sensor, and a U-Net network is used to perform region segmentation processing on the door frame of the entrance door in the image features to determine the set of door frame pixel coordinates; wherein, the set of door frame pixel coordinates is used to define the physical boundary of the entrance door; An improved YOLOv5 model is used to detect human bodies in the image features and determine the coordinates of the human body bounding box.

3. The method according to claim 2, characterized in that, The process of spatiotemporally aligning the preprocessed moving target features with image features, determining dynamic weight parameters corresponding to the moving target features and image features based on preset factors, and determining the spatiotemporally aligned fused feature vector based on the spatiotemporal alignment result and the dynamic weight parameters includes: A transformation relationship is established between the radar coordinate system corresponding to the millimeter-wave radar and the image coordinate system corresponding to the vision sensor. The moving target features in the radar coordinate system are projected to the image coordinate system through a calibration matrix to determine the spatial coordinate mapping result. The calibration matrix is ​​obtained by Zhang Zhengyou calibration method. By combining hardware triggering and software interpolation, the motion target features after spatial coordinate mapping are time-synchronized with the door frame pixel coordinate set and human body bounding box coordinates in the image features, and the time synchronization processing result is determined; among them, linear interpolation is used to compensate for the time misalignment of feature data. The first weight parameter corresponding to the moving target feature and the second weight parameter corresponding to the image feature are calculated based on preset factors; wherein, the preset factors include: ambient light intensity, target distance and movement speed, and the sum of the first weight parameter and the second weight parameter is 1; Based on the spatial coordinate mapping result, the time synchronization processing result, the first weight parameter, and the second weight parameter, the trajectory of the moving target is weighted and fused with the set of door frame pixel coordinates and the human body bounding box coordinates to determine the spatiotemporally aligned fused feature vector.

4. The method according to claim 3, characterized in that, The process of associating targets and determining entry / exit directions based on the fused feature vectors, statistically analyzing the effective passenger flow at the entrance door according to preset rules, and determining the abnormal behavior judgment results of the targets includes: An improved Hungarian algorithm is used to perform cross-modal target association between the trajectory of moving targets in the fused feature vector and the coordinates of human bounding boxes. The association cost is calculated by combining Mahalanobis distance and visual feature cosine similarity to determine the same target after association. Define the door plane normal vector of the entrance door, calculate the dot product of the velocity vector of the associated target and the door plane normal vector to determine the target's entry and exit direction, and combine the human body orientation corresponding to the human body bounding box coordinates to determine the entry and exit direction; wherein, the human body orientation is determined by detecting key points through OpenPose. The effective passenger flow of the entrance door is determined based on the number of targets that meet the preset rules; wherein the preset rules include: the center point of the target's human body bounding box passes through the door frame area corresponding to the set of pixel coordinates of the door frame, the target's movement trajectory is continuous and the direction is consistent, and the target's dwell time exceeds a preset threshold. If a target satisfies the preset rules and the preset abnormal behavior conditions, the abnormal behavior judgment result of the target is determined to be abnormal.

5. The method according to claim 4, characterized in that, The step of dynamically adjusting the deployment threshold based on a preset time period and historical passenger flow data, and executing the arming or disarming operation of the entrance door based on the deployment threshold, actual passenger flow data, the opening and closing status of the entrance door, and the abnormal behavior judgment result, includes: The basic deployment threshold is adjusted according to the preset time period division results, and the adaptive deployment threshold is adjusted according to historical passenger flow data and preset safety factor. The maximum value between the basic deployment threshold and the adaptive deployment threshold is used as the deployment threshold. If the actual passenger flow data corresponding to the effective passenger flow is not less than the deployment threshold, the status of the entrance door is closed and the duration is greater than the first closing time, and the abnormal behavior judgment result of the target is abnormal, the deployment operation of the entrance door is executed. If the actual passenger flow data is 0, the entrance door is closed for a duration longer than the second closing duration, and the current time period is a low-risk period, the disarming operation of the entrance door is executed; wherein, the second closing duration is longer than the first closing duration.

6. The method according to claim 5, characterized in that, The adjustment of the basic deployment threshold based on the preset time period division results, and the adjustment of the adaptive deployment threshold based on historical passenger flow data and a preset safety factor, include: The preset time period is divided into high-risk period and low-risk period; the basic deployment threshold for high-risk period is greater than the basic deployment threshold for low-risk period. Record the effective passenger flow at the entrance door every day, and calculate historical passenger flow data based on the effective passenger flow over a preset number of days; wherein, the historical passenger flow data includes: the moving average and standard deviation of the preset number of days; The adaptive deployment threshold is adjusted based on the moving average, standard deviation, and preset safety factor of the preset number of days.

7. A multimodal feature fusion-based entrance door passenger flow detection and deployment device, characterized in that, The device includes: The multimodal feature acquisition module is used to acquire moving target features in the entrance door area through millimeter-wave radar, acquire image features of the entrance door area through a visual sensor, acquire the opening and closing status of the entrance door through a door status sensor, and preprocess the moving target features and the image features. The feature alignment and fusion module is used to perform spatiotemporal alignment of preprocessed moving target features and image features, determine dynamic weight parameters corresponding to moving target features and image features respectively based on preset factors, and determine the spatiotemporal aligned fused feature vector based on the spatiotemporal alignment result and the dynamic weight parameters. The effective passenger flow detection module is used to perform target association and entry / exit direction judgment based on the fused feature vector, count the effective passenger flow of the entrance door according to preset rules, and determine the abnormal behavior judgment result of the target; The arming / disarming execution module is used to dynamically adjust the arming threshold based on a preset time period and historical passenger flow data. Based on the arming threshold, actual passenger flow data, the opening and closing status of the entrance door, and the abnormal behavior judgment result, it executes the arming or disarming operation of the entrance door.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the steps of the multimodal feature fusion-based entrance passenger flow detection and deployment method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the multimodal feature fusion method for detecting and deploying passenger flow at a doorway as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal feature fusion method for detecting and deploying passenger flow at the entrance as described in any one of claims 1 to 6.

Images