Crowd counting method and system based on WiFi and video modal cross-level attention

Abstract

The invention relates to the technical field of crowd counting, in particular to a crowd counting method and system based on WiFi and video modal cross-level attention, and the method comprises the steps: constructing a WiFi density map at a WiFi sensing side, and converting an irregular detection record into a fixed-size image representation; on a video sensing side, marking a region of interest for video frames collected by cameras with different visual angles, and cutting the region of interest to serve as a video side image; respectively carrying out feature coding on the WiFi density map and the video side image by adopting a convolutional neural network and self-attention combined mode; gradually aligning the WiFi modal feature embedded representation and the video modal feature embedded representation through multi-layer stacked cross-modal attention to obtain a cross-modal fusion feature; and inputting the cross-modal fusion features into a lightweight multilayer perceptron, and outputting crowd count. According to the invention, through hierarchical alignment and fusion of WiFi signals and video features, accurate estimation of the number of crowds in a large-scale complex scene can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of crowd counting, and more particularly to a crowd counting method and system based on WiFi and video modal cross-level attention. BACKGROUND

[0002] With the acceleration of urbanization, large-scale crowd monitoring becomes increasingly important in social governance and public safety. Although significant progress has been made in crowd counting technology in recent years, there are still many challenges in achieving high-precision counting in wide and complex monitoring areas. The accuracy of crowd counting is crucial for crowd management, traffic control, urban planning, and emergency response to public emergencies. The occurrence of multiple major stampede accidents in reality highlights the urgency and necessity of achieving accurate and real-time crowd estimation in practical application scenarios.

[0003] Visual information is the most direct basis for crowd counting, so most existing methods rely on images or videos to extract pedestrian features. However, problems such as light changes, occlusions, and scale differences can severely affect the reliability of visual methods. At the same time, the limited perspective of a single camera makes it difficult to cover large-scale, complexly distributed scenes, thus restricting its deployment in practice. In contrast, passive sensing based on WiFi has advantages in coverage and scalability, and can infer crowd size by capturing device signals. However, due to the fact that a person may carry multiple devices, the existence of intermittent signals, and address randomization, the accuracy of WiFi alone is limited, making it difficult to meet the high-precision requirements in large-scale scenarios.

[0004] In recent years, the development of multi-modal deep learning has provided an opportunity for the fusion of visual and wireless signals. By combining the complementary characteristics of different modalities, the robustness and accuracy of crowd counting can be significantly improved. However, existing fusion methods still have deficiencies in multi-scale feature alignment, cross-camera information integration, and noise suppression, resulting in unstable performance in complex dynamic environments. The existing WiFi-video fusion crowd counting algorithm has the following deficiencies:

[0005] 1) Most existing technologies only perform inter-modal feature interaction at a single level, making it difficult to balance global scene information and local details simultaneously, resulting in insufficient counting accuracy in large-scale, multi-view scenarios.

[0006] 2) Existing methods usually use simple feature averaging or concatenation for cross-view fusion, lacking adaptive regulation of different camera data quality, which can easily introduce noise and reduce overall robustness when there are redundant or low-quality views.

[0007] 3) In dynamic complex environments such as light changes, severe occlusions, and wireless signal interference, the optimization process of existing algorithms is unstable, sensitive to noise, and prone to counting deviation, making it difficult to meet the needs of actual large-scale monitoring.

[0008] Therefore, how to realize efficient fusion of WiFi and video multi-modal data and crowd counting to overcome the limitations of single modal and improve the robustness and adaptability of the system while ensuring counting accuracy is a problem that technicians in the field need to solve. SUMMARY

[0009] Therefore, the present application provides a crowd counting method and system based on WiFi and video modal cross-level attention, which realizes accurate estimation of the number of people in large-scale complex scenes by aligning and fusing WiFi signals and video features at different levels.

[0010] To achieve the above purpose, the present application adopts the following technical solutions:

[0011] In a first aspect, the present application provides a crowd counting method based on WiFi and video modal cross-level attention, comprising the following steps:

[0012] A multi-modal hybrid perception network covering the monitoring area is constructed, which is composed of multiple WiFi detection nodes and multiple cameras with different perspectives;

[0013] On the WiFi perception side, a WiFi density map is constructed to convert irregular detection records into a fixed-size image representation; on the video perception side, the region of interest is labeled and cropped for each video frame collected by the camera as a video side image;

[0014] Convolutional neural networks and self-attention are combined to encode WiFi density maps and video side images respectively, generating WiFi modal feature embedding representation and video modal feature embedding representation with context perception;

[0015] The WiFi modal feature embedding representation and the video modal feature embedding representation are gradually aligned through multi-layer stacked cross-modal attention, the initial layer establishes the global and local correlation, and the subsequent layers gradually focus on local key features to obtain cross-modal fusion features;

[0016] The cross-modal fusion features are input into a lightweight multilayer perception machine to output crowd counting.

[0017] Further, the construction process of the WiFi density map includes:

[0018] Map continuous physical coordinates to discrete pixel coordinates, with one pixel corresponding to a 1m x 1m area on the physical plane; let the number of detected devices be n wThe position information set of which is: The pixel coordinate set is obtained through coordinate transformation:

[0019] The intermediate image I(p) is generated based on the pixel coordinate set:

[0020]

[0021] Wherein, p represents the two-dimensional coordinates of the current pixel point, p i represents the coordinate position of the i-th detected device mapped to the pixel plane, and δ represents the impulse function.

[0022] The WiFi density map is obtained by Gaussian kernel convolution and smoothing processing on the intermediate image I(p):

[0023] WDM(p) = I(p) * G σ (p)

[0024] Wherein, G σ (p) represents the Gaussian kernel function; the size of the finally obtained WiFi density map is represented as C w ×H w ×W w , wherein C w = 1 represents the number of channels, H w and W w are the pixel height and width, respectively.

[0025] Further, the Gaussian kernel function G σ (p) is represented as:

[0026]

[0027] Wherein, || || 2 represents the squared Euclidean distance, and σ represents the standard deviation of the Gaussian kernel.

[0028] Further, on the video perception side, after cropping and scaling the video frames collected by each camera to a unified resolution C v ×H v ×W v , wherein C v = 1 represents the number of channels, H v and W v are the height and width, respectively.

[0029] Further, the extraction process of the WiFi modal feature embedding representation includes:

[0030] Local features of the WiFi density map are extracted using the LeNet convolutional neural network and serialized into token representations. Then, the global context information is aggregated using a lightweight Transformer module. The entire process is represented as follows:

[0031] X w =FlattenProj(CNN) w (I w ))

[0032]

[0033] Here, FlattenProj represents flattening the convolutional feature map and projecting it through a linear mapping, transforming it into a fixed-dimensional serialized vector representation; CNN w X is a WiFi image encoder. w This represents the token sequence obtained by convolutional feature extraction and flattening projection from the WiFi density map. TEnc represents the WiFi modal feature embedding representation. w For the WiFi branch of the Transformer encoder, d model Let h represent the model dimension, h represent the number of heads in the multi-head attention mechanism, and d represent the multi-head attention mechanism. ff This indicates the dimension of the feedforward network.

[0034] Furthermore, on the video perception side, each camera is equipped with an independent feature extraction backbone network. The feature extraction process of each feature extraction backbone network for the video frames captured by the current camera is as follows: extracting the convolutional features of the video image through a ResNet residual network, serializing them into token representations, and inputting them into the Transformer module for long-range context aggregation; the Transformer module includes multi-head self-attention, residual connections, normalization, and position-wise feedforward layers, ultimately generating a video modality feature embedding representation.

[0035] Furthermore, the process of progressively aligning the WiFi modal feature embedding representation and the video modal feature embedding representation through multi-layered stacked cross-modal attention is represented as follows:

[0036] The video modal features of each camera are embedded and projected through a fully connected layer to obtain... Where FC represents a fully connected layer. This represents the video modal feature embedding representation corresponding to the video frame captured by the m-th camera;

[0037] Embedding WiFi modal features into representation and embedding of each video feature projected through a fully connected layer Perform three-layer cascaded cross-modal attention interactions:

[0038]

[0039] Where MHA represents multi-head attention mechanism and LN represents layer normalization; This represents the WiFi sequence after cross-modal alignment.

[0040] For the video streams captured by the M-channel cameras, the cross-modal aligned WiFi sequence Concatenating the original WiFi modal feature embeddings with the token dimension yields cross-modal fused features:

[0041]

[0042] Among them, X fuse This indicates cross-modal fusion features.

[0043] Furthermore, the process of predicting population size includes:

[0044] For cross-modal fusion features X fuse Each token in the algorithm is assigned a reliability score. Each original token is multiplied by its corresponding reliability score and then concatenated to obtain the filtered token sequence.

[0045] The refined token sequence Inputting data into a lightweight multilayer perceptron, the prediction of crowd size is expressed as follows:

[0046]

[0047] h2 = ReLU(W2h1)

[0048]

[0049] in, This represents the filtered token sequence; ReLU represents the activation function; h1 and h2 represent the intermediate hidden layer features of the multilayer perceptron, respectively; W1, W2, and W3 represent the weight matrices of the prediction head, respectively; vec() represents the flattening operation; This indicates the final predicted output.

[0050] Secondly, the present invention provides a crowd counting system based on WiFi and video modal cross-level attention, which employs the method described above, including:

[0051] The multimodal data acquisition module is used to collect WiFi detection data through multiple WiFi detection nodes and to collect video streams from different perspectives through multiple cameras;

[0052] The multimodal data preprocessing module is used to construct a WiFi density map on the WiFi sensing side, transforming irregular detection records into fixed-size image representations; on the video sensing side, it annotates the regions of interest for each video frame captured by the camera and crops them as video-side images.

[0053] The modal feature encoding module is used to encode the WiFi density map and video side image respectively using a combination of convolutional neural network and self-attention, to generate context-aware WiFi modal feature embedding representation and video modal feature embedding representation;

[0054] The hierarchical cross-modal alignment module is used to gradually align the WiFi modal feature embedding representation and the video modal feature embedding representation through multiple stacked cross-modal attention layers. The initial layer establishes the global and local correlation, and subsequent layers gradually focus on key local features to obtain cross-modal fusion features.

[0055] The prediction module is used to input cross-modal fused features into a lightweight multilayer perceptron and output crowd counts.

[0056] Thirdly, a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the crowd counting method based on WiFi and video modal cross-level attention as described above.

[0057] As can be seen from the above technical solution, compared with the prior art, the present invention has the following beneficial effects:

[0058] 1. This invention utilizes a hierarchical cross-modal attention mechanism (HCMA) to establish coarse-to-fine semantic alignment and token matching relationships layer by layer. In complex environments with occlusion, blurring, and changing viewpoints, WiFi and video modalities can complement each other to effectively suppress false matches and noise interference, thereby improving the stability and accuracy of cross-modal fusion.

[0059] 2. This invention introduces convolutional neural networks and self-attention modules into the WiFi and video branches respectively, overcoming the bottleneck of slow expansion of the receptive field in convolutional networks. This design can both ensure the integrity of local features and achieve global context modeling, generating context-aware, high-quality embedding representations, providing a solid foundation for cross-modal alignment and prediction.

[0060] 3. For low-quality or heavily interfered video streams, this invention achieves implicit feature selection through the response strength of tokens aligned across modalities. Weak response tokens are naturally suppressed, while strong response tokens are retained, thus focusing on cross-modal consistency evidence during the prediction stage. The fusion head employs direct vectorization and a lightweight MLP, rather than attention pooling, avoiding noise amplification effects and significantly enhancing the model's robustness in complex scenarios.

[0061] 4. In multi-camera scenarios, this invention independently configures a backbone network and a hierarchical cross-modal attention module for each video stream, achieving multi-view complementarity through post-processing feature fusion. This design can significantly reduce the impact of low-quality input from a single path, effectively prevent error coupling, and maintain stability during fusion training, thereby improving the robustness and generalization ability of the overall system.

[0062] This invention not only achieves hierarchical cross-modal alignment and implicit feature selection of WiFi and video signals for the first time in multimodal crowd counting scenarios, but also proposes a fusion framework that takes into account both local details and global context, and is robust and scalable. This framework can significantly improve the accuracy and stability of crowd counting, providing important technical support for fields such as public safety, smart cities and group behavior analysis, while also showing broad application potential in other multimodal perception tasks. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0064] Figure 1 This is a schematic diagram of the structure of the multimodal hybrid sensing network provided by the present invention.

[0065] Figure 2 This is a framework diagram of the crowd counting method based on WiFi and video modal cross-level attention provided by the present invention. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] This invention discloses a crowd counting method based on cross-level attention between WiFi and video modalities, comprising the following steps:

[0068] S1. Construct a multimodal hybrid sensing network covering the monitoring area, which consists of multiple WiFi detection nodes and multiple cameras with different perspectives;

[0069] S2. On the WiFi sensing side, construct a WiFi density map to transform irregular detection records into fixed-size image representations; on the video sensing side, label the regions of interest for each video frame captured by the camera and crop them to serve as video-side images.

[0070] S3. A combination of convolutional neural networks and self-attention is used to encode the WiFi density map and video side image respectively, generating context-aware WiFi modal feature embedding representations and video modal feature embedding representations;

[0071] S4. The WiFi modal feature embedding representation and the video modal feature embedding representation are gradually aligned through multi-layer stacked cross-modal attention. The initial layer establishes the relationship between the global and local features, and the subsequent layers gradually focus on key local features to obtain cross-modal fusion features.

[0072] S5. Input the cross-modal fusion features into a lightweight multilayer perceptron and output the crowd count.

[0073] This invention is developed for large-scale crowd counting and monitoring tasks. It leverages the complementarity of WiFi signals and video features, combined with a hierarchical cross-modal attention mechanism, to achieve robust application in complex real-world scenarios. Specifically, the core method of this invention involves two key aspects: first, multimodal data preprocessing and modal feature encoding; and second, a hierarchical cross-modal alignment and robust prediction mechanism.

[0074] (I) Multimodal data preprocessing and modal feature encoding:

[0075] Because raw WiFi data is redundant, noisy, and irregular in dimensionality, and video data is limited by background interference and different camera perspectives, existing methods struggle to uniformly process multimodal information. To address this issue, this invention proposes the following improvements:

[0076] First, on the WiFi sensing side, this invention constructs a WiFi density map (WDM) to transform irregular detection records into a fixed-size image representation, thereby eliminating redundant information and ensuring uniform input dimensions. Gaussian kernel smoothing is also used to enhance spatial continuity. Second, on the video side, for each camera's fixed viewpoint, regions of interest (ROIs) where pedestrians may appear are pre-labeled. Through cropping and uniform resolution scaling, irrelevant background noise is effectively eliminated, highlighting visual cues useful for crowd counting. Finally, in the feature encoding stage, this invention employs a combination of convolutional neural networks and lightweight self-attention, enabling both local feature extraction and global context modeling to generate context-aware WiFi and video modal embedding representations, laying the foundation for subsequent cross-modal fusion.

[0077] (II) Hierarchical cross-modal alignment and robust prediction mechanism:

[0078] Existing methods for WiFi-video fusion often rely on single-interaction attention, making it difficult to consider both global and local features, and are easily affected by low-quality viewpoints in multi-camera scenarios. To address these issues, this invention proposes a hierarchical cross-modal attention and robust prediction mechanism:

[0079] First, in the cross-modal alignment stage, WiFi features and video features are gradually aligned through multi-layered stacked interactive attention, refining from global to local. The initial layer establishes stable global associations, while subsequent layers focus on key local features, thereby improving matching performance under conditions of occlusion and changing viewpoints. Second, in the multi-camera fusion stage, this invention retains the independent feature streams of each camera to avoid being misled by low-quality perspectives, and implements adaptive weighting through the fusion module to highlight reliable information and reduce redundant or noisy inputs. Finally, in the prediction stage, this invention employs a robust regression head based on a lightweight multilayer perceptron to flatten and regress the unified features after cross-modal fusion, emphasizing multimodal consistency signals and reducing the impact of low-quality perspectives on the results, thus ensuring the stability and accuracy of crowd counting in complex environments.

[0080] The following provides further explanation of each of the above steps.

[0081] S1. Acquisition of multimodal data.

[0082] To meet the needs of large-scale population monitoring, this invention constructs a multimodal hybrid sensing network (HSN), the structure of which is as follows: Figure 1As shown, this network consists of multiple low-cost WiFi detection nodes and a small number of cameras, enabling flexible and adaptable deployment. Specifically, the WiFi nodes can be dedicated multi-module detectors or programmable commercial access points (APs). The nodes run detection scripts and periodically upload data to the server. Simultaneously, standard surveillance cameras transmit video streams to the server via wired connections.

[0083] S2. Preprocessing of multimodal data.

[0084] On the WiFi sensing side, raw detection data typically appears in the form of record entries, including fields such as device MAC address, received signal strength (RSS), channel number, and timestamp. These fields are redundant and contain information unrelated to population size; furthermore, the number of record entries within a time window is not fixed, resulting in irregular input dimensions and hindering unified modeling. Therefore, this invention proposes using WiFi density maps (WDM) for data transformation. This method compactly encodes the detection data into a fixed-size image representation, removing redundant attributes while ensuring input standardization.

[0085] Specifically, the process of constructing a WiFi density map includes:

[0086] Continuous physical coordinates are mapped to discrete pixel coordinates, with one pixel corresponding to a 1-meter × 1-meter area on the physical plane; let the number of detected devices be n. w Its location information set is as follows:

[0087]

[0088] Location information is obtained through WiFi fingerprinting, which involves pre-dividing sampling points within the monitoring area and having multiple deployed WiFi detection nodes collect wireless signal characteristics of terminal devices (such as received signal strength RSS, channel state information CSI, channel number, timestamp, etc.) and establish a fingerprint database by combining the physical coordinates of the sampling points. In actual operation, the WiFi detection nodes collect the real-time signal characteristics of the terminal and perform similarity matching or regression prediction with the samples in the fingerprint database (such as using algorithms such as K-nearest neighbors, support vector machines, or deep learning networks) to estimate the two-dimensional coordinate position of the device on the plane, ultimately forming a set of location information.

[0089] The set of pixel coordinates is obtained after coordinate transformation:

[0090]

[0091] The intermediate image I(p) is generated based on the pixel coordinate set. The specific process is as follows: First, traverse each pixel point p on the pixel grid corresponding to the monitoring area, and then determine whether the pixel point is related to a certain device position p in the pixel coordinate set P. i Overlapping; if one or more devices fall at the pixel location, the value is assigned the corresponding number of devices, i.e., I(p) = 1, 2, 3..., otherwise it is assigned 0. Through this point-by-point statistical process, the resulting I(p) is formally a two-dimensional sparse matrix, where non-zero pixels represent the distribution locations of detected devices at the current time, and zero-value pixels represent areas without devices, thus forming the intermediate image I(p), which is represented as:

[0092]

[0093] Where p represents the two-dimensional coordinates of the current pixel, p i This represents the coordinate position of the i-th detected device mapped onto the pixel plane, and δ represents the impulse function;

[0094] Gaussian kernel convolution and smoothing are applied to the intermediate image I(p) to obtain the WiFi density map:

[0095] I w =I(p)*G σ (p) (4)

[0096] Essentially, a WiFi density map is a two-dimensional matrix where the value at each position represents the number of devices in the corresponding area, and the position of the value reflects the spatial distribution of the devices throughout the monitoring area.

[0097] Gaussian kernel function G σ (p) is represented as:

[0098]

[0099] Among them, G σ (p) represents the Gaussian kernel function, || || 2 The square of the Euclidean distance is represented by σ, which represents the standard deviation of the Gaussian kernel and controls the smoothing range; the final WiFi density map is represented by the size C. w ×H w ×W w , where C w =1, indicating the number of channels, H w and W w These are the pixel height and width, respectively.

[0100] On the video perception side, each camera has a fixed viewing angle, thus reliably annotating regions of interest (ROIs) where pedestrians may appear, eliminating redundant backgrounds introduced by complex textures and environmental interference. To further enhance the effectiveness of visual cues and ensure consistency in subsequent processing, this invention crops the ROIs of the annotated image and scales it to a uniform resolution Ci. v ×H v ×W v , where C v =1 indicates the number of channels, H v and W v These are the height and width, respectively.

[0101] Through the aforementioned WiFi density map construction, video ROI annotation, and standardization processing steps, this invention achieves unified and standardized input of multimodal raw data, reduces redundant information and noise interference, and provides a reliable data foundation for subsequent modal feature extraction and cross-modal fusion.

[0102] S3. Multimodal Feature Encoding: The modal feature encoding of this invention consists of two parts: first, a convolutional neural network (CNN) backbone (LeNet-5 for the WiFi branch and ResNet34 for the video branch); and second, an intramodal self-attention mechanism stacked on top of the convolutional features. The CNN backbone, by utilizing translation equivariance and weight sharing mechanisms, can stabilize and accelerate the training process, extract compact and efficient feature representations, and possess suitable capacity and receptive field. The subsequently introduced self-attention module compensates for the shortcomings of convolution in expanding the receptive field, enabling features to perform global-scale interaction and long-distance context modeling, thereby generating context-aware embedding representations. and This provides support for subsequent cross-modal fusion and regression prediction.

[0103] On the WiFi sensing side, local features of the WiFi density map are first extracted using the convolutional neural network LeNet and serialized into token representations. Then, the lightweight Transformer module is used to aggregate global context information. The entire process is represented as follows:

[0104]

[0105] Here, FlattenProj represents flattening the convolutional feature map and projecting it through a linear mapping, transforming it into a fixed-dimensional serialized vector representation. CNN w X is a WiFi image encoder. w This represents the token sequence obtained by convolutional feature extraction and flattening projection from the WiFi density map. TEnc represents the WiFi modal feature embedding representation. wFor the WiFi branch of the Transformer encoder, d model The dimension of the Transformer module is represented by h, which represents the number of heads in the multi-head attention mechanism, and d represents the number of heads in the multi-head attention mechanism. ff This indicates the dimension of the feedforward network. The feedforward network refers to the two fully connected neural networks in the Transformer module that operate at each position to enhance the feature representation capability.

[0106] The self-attention mechanism is defined as:

[0107]

[0108] By combining residual connections, layer normalization, and position-wise feedforward networks (ReLU activation), long-distance dependency modeling can be achieved while avoiding over-parameterization.

[0109] On the video perception side, each camera is equipped with an independent feature extraction backbone network. The feature extraction process of each feature extraction backbone network on the video frames captured by the current camera is as follows: the convolutional features of the video image are extracted through a ResNet residual network, serialized into a token representation, and input into the Transformer module for long-range context aggregation. The Transformer module includes multi-head self-attention, residual connections, normalization, and position-wise feedforward layers, and finally generates a video modal feature embedding representation.

[0110] Through the above modal feature encoding steps, the present invention can extract efficient representations that combine local details and global context in both WiFi and video modalities, providing a solid foundation for subsequent hierarchical cross-modal alignment and fusion.

[0111] S4. Hierarchical Cross-Modal Alignment: The proposed hierarchical cross-modal attention (HCMA) module enhances the fusion effect of WiFi and video modalities. This module progressively refines cross-modal alignment from coarse-grained semantic level to fine-grained token-level matching. The initial layer establishes stable global and local associations to constrain the optimization process and suppress false matches; subsequent layers gradually focus on more discriminative tokens, thus maintaining alignment accuracy even in the presence of occlusion or changes in viewpoint. Simultaneously, residual connections and layer normalization mechanisms ensure the preservation of modality-specific information and maintain stability during training.

[0112] Specifically, the video modal features of each camera are embedded and projected through a fully connected layer to obtain... Where FC represents a fully connected layer. This represents the video modal feature embedding representation corresponding to the video frame captured by the m-th camera;

[0113] Embedding WiFi modal features into representation and embedding of each video feature projected through a fully connected layer Perform three-layer cascaded cross-modal attention interactions:

[0114]

[0115] MHA stands for Multi-head Attention Mechanism, which adopts a batch-first layout, using WiFi modal features as query Q and video modal features as both key K and value V, thereby achieving alignment between WiFi and video features. LN stands for Layer Normalization. This represents the WiFi sequence after cross-modal alignment.

[0116] In multi-camera scenarios, the complementarity and redundancy of different perspectives can effectively mitigate the impact of low-quality video streams, thus maintaining high robustness even under noise, blur, and occlusion conditions. This invention configures an independent feature extraction backbone network for each camera to avoid error coupling; in the later fusion stage, reliable features from different perspectives are aggregated to further improve overall performance.

[0117] For the video streams captured by the M-channel cameras, the cross-modal aligned WiFi sequence Embedded representation with original WiFi modal features By concatenating the tokens, we obtain the cross-modal fusion features:

[0118]

[0119] Among them, X fuse This indicates cross-modal fusion features.

[0120] In the implementation process, each video stream uses an independent ResNet34 and Transformer as encoders and interacts with the corresponding HCMA module through the projection layer. To maintain feature stability, the parameters of the backbone network are frozen during the fusion training phase, and only the fusion layer is optimized. The optimizer adopts Adam and combines a cosine annealing learning rate scheduling strategy.

[0121] Through the aforementioned hierarchical cross-modal alignment mechanism, this invention can achieve a progressively refined fusion of WiFi and video features, significantly improving the accuracy and robustness of crowd counting in complex scenarios.

[0122] S5. Cross-modal selection and prediction:

[0123] After completing cross-modal alignment, this invention further introduces an implicit cross-modal feature selection mechanism (ICTS) to reduce the interference of low-quality cameras on overall prediction performance. During cross-modal interaction, WiFi features are enhanced by video features, and the interaction results naturally highlight stable tokens that are supported in all modalities. For low-quality viewpoints affected by noise, blur, or occlusion, the corresponding tokens will show weaker responses. Based on this characteristic, this invention concatenates the outputs of all HCMA modules with the original WiFi tokens to form a unified token set.

[0124] Subsequently, a reliability score is assigned to each token using the ICTS mechanism, thereby strengthening high-quality, multimodal consistency features and weakening tokens affected by noise during the fusion process. Specifically, let the concatenated sequence be... The reliability score for each token is then calculated as follows:

[0125]

[0126] Here, f(·) captures cross-modal consistency, and σ(·) is the Sigmoid function. This yields the refined token sequence.

[0127] Under the ICTS mechanism, video features from different cameras are first concatenated into a unified token sequence, and each token is evaluated in conjunction with cross-modal consistency information. By calculating a reliability score, the model can amplify clear viewpoint features that are consistent with other modalities, while suppressing blurry, occluded, or poor-quality viewpoints, thereby achieving adaptive selection and weighting of video stream data.

[0128] ICTS is called an "implicit" selection mechanism because it doesn't explicitly discard a camera or modality. Instead, it automatically adjusts weights through token-by-token reliability modeling. The contribution of low-confidence tokens is weakened, while the contribution of high-confidence tokens is amplified. This internal, continuous weight adjustment process is essentially an implicit cross-modal feature selection.

[0129] During the prediction phase, the fusion head of this invention no longer uses additional attention pooling operations to avoid amplifying noise from specific viewpoints. Instead, it adopts a direct vectorization (flattening) approach before inputting into a lightweight multilayer perceptron (MLP), thereby highlighting cross-modal consistency signals and reducing interference from low-quality or unreliable cameras.

[0130] Formalistically, the calculation process of the prediction head is as follows:

[0131]

[0132] in, This represents the token sequence after ICTS filtering; ReLU represents the activation function; h1 and h2 represent the intermediate hidden layer features of the MLP, respectively; W1, Q2, and Q3 represent the weight matrices of the prediction head, respectively; vec() represents the flattening operation; This represents the final predicted output, corresponding to the actual count O. gt .

[0133] During the training phase, the mean squared error (MSE) loss function is used to optimize for multiple output targets. During the inference phase, the final crowd count is obtained by summing the prediction results.

[0134] Through the aforementioned implicit cross-modal feature selection and prediction mechanism, this invention can effectively suppress interference caused by low-quality or poor-quality perspective inputs while maintaining the effectiveness of cross-modal fusion, thereby further improving the accuracy and robustness of the system in crowd counting in complex scenarios.

[0135] In another embodiment, the present invention also provides a crowd counting system based on WiFi and video modal cross-level attention, which employs the method described above, including:

[0136] The multimodal data acquisition module is used to collect WiFi detection data through multiple WiFi detection nodes and to collect video streams from different perspectives through multiple cameras;

[0137] The multimodal data preprocessing module is used to construct a WiFi density map on the WiFi sensing side, transforming irregular detection records into fixed-size image representations; on the video sensing side, it annotates the regions of interest for each video frame captured by the camera and crops them as video-side images.

[0138] The modal feature encoding module is used to encode the WiFi density map and video side image respectively using a combination of convolutional neural network and self-attention, to generate context-aware WiFi modal feature embedding representation and video modal feature embedding representation;

[0139] The hierarchical cross-modal alignment module is used to gradually align the WiFi modal feature embedding representation and the video modal feature embedding representation through multiple stacked cross-modal attention layers. The initial layer establishes the global and local correlation, and subsequent layers gradually focus on key local features to obtain cross-modal fusion features.

[0140] The prediction module is used to input cross-modal fused features into a lightweight multilayer perceptron and output crowd counts.

[0141] In other embodiments, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the crowd counting method based on WiFi and video modality cross-level attention as described above.

[0142] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0143] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A crowd counting method based on cross-level attention between WiFi and video modalities, characterized in that, Includes the following steps: A multimodal hybrid sensing network covering the monitoring area is constructed, which consists of multiple WiFi detection nodes and multiple cameras with different perspectives; On the WiFi sensing side, a WiFi density map is constructed to transform irregular detection records into a fixed-size image representation; on the video sensing side, regions of interest are labeled and cropped for each video frame captured by the camera to serve as video-side images. A combination of convolutional neural networks and self-attention is used to encode the WiFi density map and video side image respectively, generating context-aware WiFi modal feature embedding representations and video modal feature embedding representations; WiFi modal feature embedding representation and video modal feature embedding representation are gradually aligned through multi-layer stacked cross-modal attention. The initial layer establishes the relationship between global and local features, and subsequent layers gradually focus on key local features to obtain cross-modal fusion features. The cross-modal fusion features are input into a lightweight multilayer perceptron, which outputs a crowd count.

2. The crowd counting method based on cross-level attention of WiFi and video modalities according to claim 1, characterized in that, The process of constructing a WiFi density map includes: Continuous physical coordinates are mapped to discrete pixel coordinates, with one pixel corresponding to a 1-meter × 1-meter area on the physical plane; let the number of detected devices be n. w Its location information set is as follows: The set of pixel coordinates is obtained after coordinate transformation: Generate an intermediate image I(p) based on the set of pixel coordinates: Where p represents the two-dimensional coordinates of the current pixel, p i This represents the coordinate position of the i-th detected device mapped onto the pixel plane, and δ represents the impulse function; Gaussian kernel convolution and smoothing are applied to the intermediate image I(p) to obtain the WiFi density map: WDM(p)=I(p)*G σ (p) Among them, G σ (p) represents the Gaussian kernel function; the final WiFi density map size is represented by C. w ×H w ×W w , where C w =1, indicating the number of channels, H w and W w These are the pixel height and width, respectively.

3. The crowd counting method based on cross-level attention of WiFi and video modalities according to claim 2, characterized in that, Gaussian kernel function G σ (p) is represented as: Among them, || || 2 σ represents the squared Euclidean distance, and σ represents the standard deviation of the Gaussian kernel.

4. The crowd counting method based on cross-level attention of WiFi and video modalities according to claim 1, characterized in that, On the video sensing side, the video frames captured by each camera are cropped and scaled to a uniform resolution C. v ×H v ×W v , where C v =1 indicates the number of channels, H v and W v These are the height and width, respectively.

5. The crowd counting method based on cross-level attention of WiFi and video modalities according to claim 1, characterized in that, The extraction process of WiFi modal feature embedding representation includes: Local features of the WiFi density map are extracted using the LeNet convolutional neural network and serialized into token representations. Then, the global context information is aggregated using a lightweight Transformer module. The entire process is represented as follows: X w =FlattenProj(CNN w (I w )) Here, FlattenProj represents flattening the convolutional feature map and projecting it through a linear mapping, transforming it into a fixed-dimensional serialized vector representation; CNN w X is a WiFi image encoder. w This represents the token sequence obtained by convolutional feature extraction and flattening projection from the WiFi density map; TEnc represents the WiFi modal feature embedding representation. w For the WiFi branch of the Transformer encoder, d model Let h represent the model dimension, h represent the number of heads in the multi-head attention mechanism, and d represent the multi-head attention mechanism. ff This indicates the dimension of the feedforward network.

6. The crowd counting method based on cross-level attention of WiFi and video modalities according to claim 1, characterized in that, On the video perception side, each camera is equipped with an independent feature extraction backbone network. The process of each feature extraction backbone network extracting features from the video frames captured by the current camera is as follows: extract the convolutional features of the video image through the ResNet residual network, serialize them into token representations, and input them into the Transformer module for long-distance context aggregation. The Transformer module includes multi-head self-attention, residual connections, normalization, and position-wise feedforward layers, ultimately generating video modal feature embedding representations.

7. The crowd counting method based on cross-level attention of WiFi and video modalities according to claim 1, characterized in that, The process of progressively aligning the WiFi modal feature embedding representation and the video modal feature embedding representation through multi-layered stacked cross-modal attention is represented as follows: The video modal features of each camera are embedded and projected through a fully connected layer to obtain... Where FC represents a fully connected layer. This represents the video modal feature embedding representation corresponding to the video frame captured by the m-th camera; Embedding WiFi modal features into representation and embedding of each video feature projected through a fully connected layer Perform three-layer cascaded cross-modal attention interactions: Where MHA represents multi-head attention mechanism and LN represents layer normalization; This represents the WiFi sequence after cross-modal alignment. For the video streams captured by the M-channel cameras, the cross-modal aligned WiFi sequence Concatenating the original WiFi modal feature embeddings with the token dimension yields cross-modal fused features: Among them, X fuse This indicates cross-modal fusion features.

8. The crowd counting method based on cross-level attention of WiFi and video modalities according to claim 7, characterized in that, The process of predicting population size includes: For cross-modal fusion features X fuse Each token in the algorithm is assigned a reliability score. Each original token is multiplied by its corresponding reliability score and then concatenated to obtain the filtered token sequence. The refined token sequence Inputting data into a lightweight multilayer perceptron, the prediction of crowd size is expressed as follows: h2 = ReLU(W2h1) in, This represents the filtered token sequence; ReLU represents the activation function; h1 and h2 represent the intermediate hidden layer features of the multilayer perceptron, respectively; W1, W2, and W3 represent the weight matrices of the prediction head, respectively; vec() represents the flattening operation; This indicates the final predicted output.

9. A crowd counting system based on cross-level attention using WiFi and video modalities, characterized in that, It employs the method described in any one of claims 1-8, comprising: The multimodal data acquisition module is used to collect WiFi detection data through multiple WiFi detection nodes and to collect video streams from different perspectives through multiple cameras; The multimodal data preprocessing module is used to construct a WiFi density map on the WiFi sensing side, transforming irregular detection records into fixed-size image representations; on the video sensing side, it annotates the regions of interest for each video frame captured by the camera and crops them as video-side images. The modal feature encoding module is used to encode the WiFi density map and video side image respectively using a combination of convolutional neural network and self-attention, to generate context-aware WiFi modal feature embedding representation and video modal feature embedding representation; The hierarchical cross-modal alignment module is used to gradually align the WiFi modal feature embedding representation and the video modal feature embedding representation through multiple stacked cross-modal attention layers. The initial layer establishes the global and local correlation, and subsequent layers gradually focus on key local features to obtain cross-modal fusion features. The prediction module is used to input cross-modal fused features into a lightweight multilayer perceptron and output crowd counts.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the crowd counting method based on WiFi and video modality cross-level attention as described in any one of claims 1-8.

Images