Multiscale crowd counting methods, systems, and media based on VSSM and mask reconstruction

Abstract

This invention discloses a multi-scale crowd counting method, system, and medium based on VSSM and mask reconstruction, belonging to the fields of computer vision and deep learning. The invention performs multi-scale transformation and segmentation operations on the input crowd image to obtain image patches at each scale. Each image patch is analyzed to obtain an information entropy map, and a mask operation is performed on each image patch based on the information entropy map. The masked image patches are then input into VSSM for feature extraction to obtain coded features. The coded features are decoded and reconstructed using a Transformer decoder to finally obtain a reconstructed feature image. A multi-scale fusion module is used to fuse the reconstructed feature images at different scales to generate fused features. A crowd density map is generated based on the fused features, and the crowd counting result is obtained by analyzing the crowd density map. This avoids the counting errors caused by single-scale analysis and significantly improves the crowd counting accuracy across the entire scale range.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, specifically to a multi-scale crowd counting method, system, and medium based on VSSM and mask reconstruction. Background Technology

[0002] Crowd counting, as an important branch of computer vision, plays a vital role in fields such as public safety and urban planning. With the rapid development of deep learning technology, crowd counting methods have evolved from early detection methods based on traditional image processing to density estimation methods based on deep neural networks. Current mainstream crowd counting methods are based on density estimation techniques, generating a density map from the input image by training a convolutional neural network. Each pixel value in the density map represents the crowd density at that location, and the final crowd count result is obtained by integrating the entire density map. This method not only provides information on the number of people but also shows the spatial distribution of the crowd, often exhibiting good robustness in complex scenes. However, this method still has certain limitations.

[0003] The main limitations of existing technology:

[0004] First, existing crowd counting techniques exhibit significant robustness limitations when dealing with partially missing images and complex scenes. In real-world deployment environments, input images often suffer from missing local information due to factors such as occlusion, limited field of view, or sensor malfunction. Furthermore, complex lighting conditions, such as shadowed areas, strong light, and low-light conditions at night, further complicate feature extraction. Traditional supervised learning methods heavily rely on complete, high-quality labeled data for training and lack the ability to adapt to incomplete inputs. When faced with images containing partially missing information, model performance drops sharply.

[0005] Second, existing crowd counting techniques face dual challenges in terms of feature representation learning depth and computational efficiency. Current mainstream methods employ convolutional neural networks (CNNs) or visual Transformers architectures, which have inherent technical bottlenecks: CNNs are limited by local receptive fields, making it difficult to model long-distance dependencies, resulting in an inability to effectively capture global contextual information in dense crowd scenes; while Transformers, although possessing global modeling capabilities, suffer from a significant computational burden due to the quadratic time complexity (O(n²)) of their self-attention mechanism when processing high-resolution crowd images, severely limiting the system's real-time performance. Summary of the Invention

[0006] The purpose of this invention is to provide a multi-scale crowd counting method, system, and device based on VSSM and mask reconstruction. By innovatively using a visual state space model and integrating a mask self-supervised learning mechanism with multi-scale feature processing technology, it solves the technical problems of existing crowd counting technologies, such as low accuracy in missing images and complex scenes, poor adaptability to multi-scale targets, low depth of feature representation learning, and low computational efficiency.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A multi-scale crowd counting method based on VSSM and mask reconstruction includes the following steps:

[0009] Step 1: Standardize and multi-scale transform the input crowd image to obtain scaled input crowd images at different scales, then perform segmentation to obtain a set of image patches at each scale;

[0010] Step 2: Analyze each image block in the image block set to obtain an information entropy map that reflects the density complexity of each image block, and perform a masking operation on each image block based on the information entropy map to obtain a masked image block set;

[0011] Step 3: Input the masked image block into VSSM for feature extraction to obtain the encoded features at each scale. The VSSM establishes a global receptive field through a 2D selective scanning mechanism to generate encoded features with long-range dependencies.

[0012] Step 4: Decode and reconstruct the encoded features using the Transformer decoder to generate reconstructed blocks, calculate the joint loss for self-supervised pre-training, and generate corresponding reconstructed feature images based on the reconstructed blocks at each scale;

[0013] Step 5: Employ an attention-based multi-scale fusion module to fuse reconstructed feature images at different scales to generate fused features;

[0014] Step 6: Generate a population density map based on the fusion features, and obtain the population count results by analyzing the population density map.

[0015] Furthermore, the input crowd images are standardized to The size of the input crowd image is determined, and a downsampling-based scaling transformation is performed on the standardized input crowd image to obtain three scaled input crowd images: 1 / 2 scale, 1 / 4 scale, and 1 / 8 scale. The 1 / 2 scale, 1 / 4 scale, and 1 / 8 scaled input crowd images are then segmented into... The image blocks are of various sizes, each with a specific dimensions. ,in .

[0016] Furthermore, the image patch is divided into Divide the regions into equal-sized areas, calculate the information entropy of each area to form an information entropy map, and calculate the difference between the average information entropy of each area and its neighboring areas. This difference is called the information gap. Sort the regions according to their information entropy from largest to smallest to obtain an information entropy ranking sequence. Sort the regions according to their information gap from smallest to largest to obtain an information gap ranking sequence. Then, sort the regions in the information entropy ranking sequence from top to bottom. The regions constitute a set of information entropy regions, and the information gaps are arranged in a sequence. Each region constitutes a set of information gap regions; with a step size of 1, starting from... Initially, iterations are performed. When the intersection of the information entropy set and the information gap set is greater than 20% of the total area of ​​the image patch, a random selection is made from the intersection of the information entropy set and the information gap set. A set of regions is formed, and a random selection operation is performed 10 times. This is a rounding up operation. The number of regions in the intersection of the information entropy set and the information gap set;

[0017] For each entropy set, calculate the distance between the center coordinates of each region and the center coordinates of the image patch, and calculate the standard deviation of all distances. Set a standard deviation threshold and compare the standard deviation of the distances with the standard deviation threshold.

[0018] If, in the 10 random selection operations, there exists an entropy set whose distance standard deviation is greater than the standard deviation threshold, the iteration terminates, and a random entropy set is selected to perform a full masking operation on the region in the entropy set. If, in the 10 random selection operations, there is no entropy set whose distance standard deviation is greater than the standard deviation threshold, the next iteration begins, and so on, until the iteration terminates.

[0019] Furthermore, the VSSM adopts a visual state space model architecture, containing 4-12 VSS blocks, each VSS block containing the following components:

[0020] The layer normalization module normalizes the input mask image blocks;

[0021] The linear projection layer projects the normalized mask image patch onto the hidden dimension to obtain the mask image patch features of the hidden dimension.

[0022] Depthwise separable convolutional layers perform depthwise separable convolution operations on the hidden dimension mask image patch features to obtain the input feature map, thereby enhancing the ability to model local features.

[0023] The two-dimensional selective scanning module is used to perform four-way scanning processing on the input feature map to obtain directional fusion features;

[0024] The output projection layer maps the oriented fusion features to the original dimensions;

[0025] First residual connection layer: The directional fusion features mapped to the original dimension are residually connected to obtain the first encoded output feature;

[0026] The feedforward neural network performs a nonlinear feature transformation on the first encoded output features;

[0027] The second residual connection layer: The output of the feedforward neural network is residually connected to obtain the encoded features.

[0028] Furthermore, the two-dimensional selective scanning module is used to perform four-directional scanning processing on the input feature map, including:

[0029] The cross-scanning unit unfolds the input feature map into a one-dimensional sequence along four different directions to obtain four scan sequences. The four different directions include from left to right, from right to left, from top to bottom, and from bottom to top.

[0030] The selective state-space processing unit independently applies the S6 block to each scan sequence for feature transformation. The S6 block is a state-space model with a selective mechanism.

[0031] The cross-merging unit merges the scan sequences processed in the four directions to generate directional fusion features.

[0032] Furthermore, the multi-scale fusion module includes the following components:

[0033] The multi-scale feature alignment unit is used to unify encoded features at different scales to the same scale through a downsampling network, treating each scale as a channel.

[0034] The channel attention computation unit is used to learn the importance weights of different channels through global average pooling and fully connected layers;

[0035] The spatial attention computation unit is used to determine the attention distribution at different spatial locations of the reconstructed feature image through a combination of average pooling and max pooling.

[0036] The adaptive fusion unit is used to perform weighted fusion of aligned multi-scale features based on channel attention and spatial attention to generate fused features.

[0037] Furthermore, the formula for calculating the joint loss is as follows:

[0038] ;

[0039] in, For joint losses, To rebuild the losses, For density loss, For counting losses, , , These are preset weighting coefficients;

[0040] The reconstruction loss is calculated using the mean squared error, and the specific formula is as follows:

[0041] ;

[0042] in, To rebuild the losses, Represents the first in the original image The pixel value of the pixel at the masked position. Indicates the first in the reconstructed image The pixel value of the pixel at the masked position. This represents the set of masked positions. The number of masked positions;

[0043] The density loss is calculated using mean square error, and the counting loss is calculated using absolute error.

[0044] This invention further provides a multi-scale crowd counting system based on mask reconstruction, the system being used in the aforementioned multi-scale crowd counting method based on VSSM and mask reconstruction, comprising:

[0045] Image preprocessing and multi-scale construction unit is used to standardize and multi-scale transform the input crowd image to obtain scaled input crowd images at different scales, and then perform segmentation operation to obtain a set of image patches at each scale;

[0046] The mask generation unit is used to analyze each image block in the image block set to obtain an information entropy map that reflects the density complexity of each image block, and to perform a masking operation on each image block according to the information entropy map to obtain a masked image block set.

[0047] The VSSM feature extraction unit is used to input the masked image block into the VSSM for feature extraction to obtain the encoded features at each scale. The VSSM establishes a global receptive field through a 2D selective scanning mechanism to generate encoded features with long-range dependencies.

[0048] The Transformer decoding and image reconstruction unit is used to decode and reconstruct the encoded features through the Transformer decoder, generate reconstruction blocks, and calculate the joint loss for self-supervised pre-training; and generate corresponding reconstructed feature images based on the reconstruction blocks at each scale.

[0049] The multi-scale feature fusion unit is used to fuse reconstructed feature images at different scales using an attention-based multi-scale fusion module to generate fused features.

[0050] The density map generation and crowd counting unit is used to generate a crowd density map based on fused features and obtain crowd counting results by analyzing the crowd density map.

[0051] The present invention further provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multi-scale crowd counting method based on VSSM and mask reconstruction.

[0052] Compared with existing technologies, this invention has the following significant advantages: This invention innovatively introduces a mask reconstruction task, fundamentally changing the model's dependence on input integrity. By selectively masking each image patch during the training phase based on information entropy and information gap, it not only ensures that the masked region is the key learning region but also avoids large-scale masking of adjacent regions. Through masking, the VSSM's 2D selective scanning mechanism is forced to strengthen cross-scale feature association. The post-masking reconstruction task enhances the model's inference ability regarding local deficiencies. Even with partial visual information loss, the model can accurately infer the distribution of people in occluded or missing areas through contextual cues from the surrounding visible areas.

[0053] The Visual State Space Model (VSSM) encoder is introduced, achieving a combination of global modeling capabilities and linear time complexity. Through its unique state-space modeling mechanism, VSSM can capture long-range dependencies in images with a linear time complexity of O(n). It can accurately identify partially occluded or highly overlapping targets by fully utilizing global contextual information, while maintaining efficient computational performance in high-resolution image processing. Furthermore, VSSM exhibits excellent scalability—even when facing 4K or higher resolution surveillance video streams, the system can maintain real-time processing capabilities without compromising between accuracy and speed.

[0054] The technical solution of this invention provides a more accurate and robust solution for the problem of crowd counting in complex scenarios. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0056] Figure 2 This is a framework diagram of the single-scale processing part based on image reconstruction in this invention;

[0057] Figure 3 This is a framework diagram of the feature fusion and density map generation part of the present invention;

[0058] Figure 4 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0060] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects.

[0061] Example 1: Basic Implementation Plan

[0062] Please see Figure 1 , Figure 2 and Figure 3 The present invention provides a technical solution:

[0063] Step 1: Standardize and multi-scale transform the input crowd image to obtain scaled input crowd images at different scales, then perform segmentation to obtain a set of image patches at each scale;

[0064] Furthermore, the input crowd images are standardized to The size of the input crowd image is determined, and a downsampling-based scaling transformation is performed on the standardized input crowd image to obtain three scaled input crowd images: 1 / 2 scale, 1 / 4 scale, and 1 / 8 scale. The 1 / 2 scale, 1 / 4 scale, and 1 / 8 scaled input crowd images are then segmented into... The image blocks are of various sizes, each with a specific dimensions. ,in .

[0065] Step 2: Analyze each image block in the image block set to obtain an information entropy map that reflects the density complexity of each image block, and perform a masking operation on each image block based on the information entropy map to obtain a masked image block set;

[0066] Furthermore, the image patch is divided into Divide the regions into equal-sized areas, calculate the information entropy of each area to form an information entropy map, and calculate the difference between the average information entropy of each area and its neighboring areas. This difference is called the information gap. Sort the regions according to their information entropy from largest to smallest to obtain an information entropy ranking sequence. Sort the regions according to their information gap from smallest to largest to obtain an information gap ranking sequence. Then, sort the regions in the information entropy ranking sequence from top to bottom. The regions constitute a set of information entropy regions, and the information gaps are arranged in a sequence. These regions constitute a set of information gap areas.

[0067] The information entropy of each region is the average of the information entropy of the R channel, the information entropy of the G channel, and the information entropy of the B channel for each region.

[0068] The local information entropy of each image patch is calculated to reflect the density complexity of the region. Higher information entropy indicates a more complex population distribution in the region, and high-entropy areas are more prone to occlusion. Masking forces the VSSM's 2D selective scanning mechanism to strengthen cross-scale feature association. Post-masking reconstruction enhances the model's inference ability regarding local missing information. Information gap reflects the difference between the analyzed region and its adjacent regions; a larger value indicates a greater difference between the analyzed region and its adjacent regions, suggesting potential anomalous jumps in the analyzed location. These jumps are irregular and difficult to identify during training. Masking regions with large jumps not only fails to enhance the robustness of the subsequent model but also increases training difficulty and reduces robustness.

[0069] Set the step size to 1, from Initially, iterations are performed. When the intersection of the information entropy set and the information gap set is greater than 20% of the total area of ​​the image patch, a random selection is made from the intersection of the information entropy set and the information gap set. A set of regions is formed, and a random selection operation is performed 10 times. This is a rounding up operation. The number of regions in the intersection of the information entropy set and the information gap set;

[0070] For each entropy set, calculate the distance between the center coordinates of each region and the center coordinates of the image patch, and calculate the standard deviation of all distances. Set a standard deviation threshold and compare the standard deviation of the distances with the standard deviation threshold.

[0071] Since this embodiment uses information entropy for masking, large areas of adjacent regions may be masked in areas with complex information. The subsequent VSSM technique is designed to explore long-range dependencies. Once large areas of adjacent regions are masked, it will result in large semantic gaps in the image, making it impossible to find long-range dependencies through semantic relationships, causing the model to completely fail. By comparing the standard deviation of the distance with the standard deviation threshold, masking schemes that mask large areas of adjacent regions can be filtered out.

[0072] If, in the 10 random selection operations, there exists an entropy set whose distance standard deviation is greater than the standard deviation threshold, the iteration terminates, and a random entropy set is selected to perform a full masking operation on the region in the entropy set. If, in the 10 random selection operations, there is no entropy set whose distance standard deviation is greater than the standard deviation threshold, the next iteration begins, and so on, until the iteration terminates.

[0073] The complete masking operation is to set the R, G, and B values ​​of the masked pixels to 0.

[0074] Step 3: Input the masked image block into VSSM for feature extraction to obtain the encoded features at each scale. The VSSM establishes a global receptive field through a 2D selective scanning mechanism to generate encoded features with long-range dependencies.

[0075] Existing crowd counting methods often encounter challenges such as crowding, blurring, and occlusion in complex scenes. CNN-based methods tend to overfit local details, while Transformer-based methods excel in global modeling but suffer from heavy computational burdens due to their quadratic complexity in terms of image size. These limitations led us to introduce VSSM, which can capture comprehensive long-range information in linear time complexity for crowd counting tasks.

[0076] The VSSM used in this embodiment is based on the VMamba architecture and is a state-space model specifically optimized for processing two-dimensional visual data. Unlike traditional one-dimensional sequence state-space models, VSSM achieves efficient modeling of image features through an innovative 2D selective scan (SS2D) mechanism. The core of VSSM is the VSS (Visual State Space) block, each of which includes: input layer normalization and linear projection; SS2D module for two-dimensional selective scan; feedforward network for feature enhancement; and residual connections to ensure gradient flow.

[0077] The introduction of the Visual State Space Model (VSSM) encoder achieves a perfect combination of global modeling capabilities and linear time complexity. In stark contrast to the local receptive field limitations of CNNs and the quadratic time complexity burden of Transformers, VSSM, through its unique state-space modeling mechanism, captures long-range dependencies in images with a linear time complexity of O(n). This groundbreaking design allows the model to accurately identify partially occluded or highly overlapping targets by fully utilizing global contextual information when processing dense crowd scenes, while maintaining efficient computational performance in high-resolution image processing. Particularly in practical deployment scenarios, VSSM demonstrates excellent scalability—even when facing 4K or higher resolution surveillance video streams, the system maintains real-time processing capabilities without compromising between accuracy and speed. This efficient global modeling capability not only significantly reduces hardware deployment costs but also enables the widespread application of high-precision crowd counting systems in resource-constrained environments such as edge computing devices, truly achieving a technological leap from "high performance dependent on high computing power" to "high performance and high efficiency coexisting." The need for high efficiency is particularly prominent in multi-scale processing scenarios.

[0078] Furthermore, the VSSM adopts a visual state space model architecture, containing 4-12 VSS blocks, each VSS block containing the following components:

[0079] The layer normalization module normalizes the input mask image blocks;

[0080] The linear projection layer projects the normalized mask image patch onto the hidden dimension to obtain the mask image patch features of the hidden dimension.

[0081] Depthwise separable convolutional layers perform depthwise separable convolution operations on the hidden dimension mask image patch features to obtain the input feature map, thereby enhancing the ability to model local features.

[0082] The two-dimensional selective scanning module is used to perform four-directional scanning processing on the input feature map, resulting in:

[0083] The cross-scanning unit unfolds the input feature map into a one-dimensional sequence along four different directions to obtain four scan sequences. The four different directions include:

[0084] Path 1: Scan line by line from top left to bottom right;

[0085] Path 2: Scan in reverse order from bottom right to top left;

[0086] Path 3: Scan column by column from top to bottom;

[0087] Path 4: Reverse column scan from bottom to top;

[0088] The selective state-space processing unit independently applies the S6 block to perform feature transformation on each scan sequence to obtain the scan change features of the four scan paths;

[0089] Block S6 is a state-space model with a selective mechanism, and its state-space model is represented as follows:

[0090] ;

[0091] ;

[0092] in, They represent Time and The hidden state at any given moment. express Time and Input at any time , , , yes The weight parameters for the next iteration are initialized. , , , All are 0;

[0093] The weight parameters are generated through input dependencies:

[0094] ;

[0095] ;

[0096] ;

[0097] ;

[0098] ;

[0099] in, , , , , For the preset linear layer, The iteration step size, for Activation function. This input-dependent parameterization allows the model to adaptively adjust state transitions based on image content.

[0100] The S6 block (Selective Scan 6) is a core component of the state-space model, representing an improvement over the traditional state-space model. The S6 block implements a state-space model with a selective mechanism, capable of selectively retaining or forgetting information based on the input. Furthermore, it exhibits linear complexity compared to traditional attention mechanisms.

[0101] The cross-merging unit merges the scan sequences processed in the four directions to generate directional fusion features.

[0102] ;

[0103] in, For directional fusion features, For the first The scanning variation characteristics of each scanning path, This is an index for scanning change features.

[0104] The output projection layer maps the oriented fusion features to the original dimensions;

[0105] First residual connection layer: The directional fusion features mapped to the original dimension are residually connected to obtain the first encoded output feature;

[0106] The feedforward neural network performs a nonlinear feature transformation on the first encoded output features;

[0107] The second residual connection layer: The output of the feedforward neural network is residually connected to obtain the encoded features.

[0108] Step 4: Decode and reconstruct the encoded features using the Transformer decoder to generate reconstructed blocks, calculate the joint loss for self-supervised pre-training, and generate corresponding reconstructed feature images based on the reconstructed blocks at each scale;

[0109] The existing technology uses a Transformer decoder to decode and reconstruct the encoded features. The Transformer decoder comprises L decoding layers. Each decoding layer includes a multi-head self-attention mechanism, a feedforward network, and residual connections. Specifically, the first layer performs a linear transformation on the input encoded features, rearranging the transformed visible features and placeholder vectors according to their original spatial positions to obtain a complete sequence representation containing visible information and placeholders. The placeholder vectors fill in the mask portion. Based on this, positional encoding is performed to obtain the input to the first layer of the decoder. For each decoding layer, the output of the previous layer is fed into the multi-head self-attention mechanism to obtain the dependency representation of different positions in the sequence. The attention output is residually connected (added) to the original input, and then normalized to obtain an intermediate representation. This intermediate representation is input into the feedforward neural network, and the output of the feedforward neural network is connected to the residual to obtain the output of this layer.

[0110] Furthermore, the formula for calculating the joint loss is as follows:

[0111] ;

[0112] in, For joint losses, To rebuild the losses, For density loss, For counting losses, , , These are preset weighting coefficients;

[0113] The reconstruction loss is calculated using the mean squared error, and the specific formula is as follows:

[0114] ;

[0115] in, To rebuild the losses, Represents the first in the original image The pixel value of the pixel at the masked position. Indicates the first in the reconstructed image The pixel value of the pixel at the masked position. This represents the set of masked positions. The number of masked positions;

[0116] The density loss is calculated using the mean square error, and the specific formula is as follows:

[0117] ;

[0118] in, Indicates density loss, Indicates the true density map at location The value, Indicates the location of the predicted density map The value, and These represent the height and width of the density map, respectively.

[0119] The counting loss is calculated using absolute error, and the specific formula is as follows:

[0120] ;

[0121] in, Indicates count loss, Indicates the actual number of people. This indicates the predicted number of people.

[0122] Step 5: Employ an attention-based multi-scale fusion module to fuse reconstructed feature images at different scales to generate fused features;

[0123] The multi-scale fusion module includes the following components:

[0124] The multi-scale feature alignment unit is used to unify encoded features at different scales to the same scale through a downsampling network, treating each scale as a channel.

[0125] The channel attention computation unit is used to learn the importance weights of different channels through global average pooling and fully connected layers;

[0126] The spatial attention computation unit is used to determine the attention distribution at different spatial locations in the reconstructed feature image through a combination of average pooling and max pooling.

[0127] The adaptive fusion unit is used to perform weighted fusion of aligned multi-scale features based on channel attention and spatial attention to generate fused features.

[0128] Since the reconstructed feature images at the three scales have different spatial resolutions (1 / 2, 1 / 4, 1 / 8), they need to be unified to the same spatial scale for effective fusion. Scale alignment is achieved using a branch-specific downsampling network.

[0129] For 1 / 2 scale features :

[0130] ;

[0131] For 1 / 4 scale features :

[0132] ;

[0133] For 1 / 8 scale features :

[0134] ;

[0135] in, Indicates the first Each convolutional layer has a stride of 2 and a kernel size of 2×2, followed by a LeakyReLU activation function. This design downsamples the reconstructed feature images at three different scales to a uniform 1 / 8 scale. Reconstructed feature images at 1 / 2, 1 / 4, and 1 / 8 scales before alignment. These are the reconstructed feature images at aligned 1 / 2, 1 / 4, and 1 / 8 scales. This is the index of the convolutional layer.

[0136] The aligned multi-scale features are then concatenated along the channel dimension:

[0137] ;

[0138] in, To stitch together and reconstruct the feature image;

[0139] Based on feature concatenation, an attention mechanism is introduced to learn the importance weights of features at different scales.

[0140] Channel attention calculation:

[0141] ;

[0142] in, For channel attention, GAP represents global average pooling. and For dimensionality reduction, the default dimensionality reduction ratio is 4. This is the Sigmoid activation function.

[0143] Spatial attention calculation:

[0144] ;

[0145] in, For spatial attention, AvgPool and MaxPool represent the average pooling and max pooling operations across channels, respectively.

[0146] Adaptive feature fusion:

[0147] ;

[0148] in, As a feature of fusion, This represents element-wise multiplication. This represents a 1×1 convolutional layer, where the final 1×1 convolution is used to map the fused 3C channel features to the desired number of output channels.

[0149] The multi-scale feature fusion module employs a mechanism combining channel attention and spatial attention to weightedly fuse features from three different scales. By learning the importance weights of each scale feature at different spatial locations and feature channels, it generates a fused feature representation rich in semantic and geometric information. This fusion process effectively integrates multi-scale information, enhancing the perception capability for populations of varying densities.

[0150] Step 6: Generate a population density map based on the fusion features, and obtain the population count results by analyzing the population density map.

[0151] Generate crowd density maps based on fused features:

[0152] ;

[0153] in, For population density maps, As a feature of fusion, This is a density regression head containing multiple convolutional layers:

[0154] ;

[0155] The population count results are obtained by integrating the population density map:

[0156] ;

[0157] in, The results of the calculations for the population are here. .

[0158] During training, standard deep learning optimization algorithms are used to iteratively update network parameters to minimize the joint loss function.

[0159] Please see Figure 4 The present invention further provides a multi-scale crowd counting system based on mask reconstruction, the system being used in the aforementioned multi-scale crowd counting method based on VSSM and mask reconstruction, comprising:

[0160] Image preprocessing and multi-scale construction unit is used to standardize and multi-scale transform the input crowd image to obtain scaled input crowd images at different scales, and then perform segmentation operation to obtain a set of image patches at each scale;

[0161] The mask generation unit is used to analyze each image block in the image block set to obtain an information entropy map that reflects the density complexity of each image block, and to perform a masking operation on each image block according to the information entropy map to obtain a masked image block set.

[0162] The VSSM feature extraction unit is used to input the masked image block into the VSSM for feature extraction to obtain the encoded features at each scale. The VSSM establishes a global receptive field through a 2D selective scanning mechanism to generate encoded features with long-range dependencies.

[0163] The Transformer decoding and image reconstruction unit is used to decode and reconstruct the encoded features through the Transformer decoder, generate reconstruction blocks, and calculate the joint loss for self-supervised pre-training; and generate corresponding reconstructed feature images based on the reconstruction blocks at each scale.

[0164] The multi-scale feature fusion unit is used to fuse reconstructed feature images at different scales using an attention-based multi-scale fusion module to generate fused features.

[0165] The density map generation and crowd counting unit is used to generate a crowd density map based on fused features and obtain crowd counting results by analyzing the crowd density map.

[0166] This invention further provides a multi-scale crowd counting system based on mask reconstruction, the system being used in the aforementioned multi-scale crowd counting method based on VSSM and mask reconstruction, comprising:

[0167] Image preprocessing and multi-scale construction unit is used to standardize and multi-scale transform the input crowd image to obtain scaled input crowd images at different scales, and then perform segmentation operation to obtain a set of image patches at each scale;

[0168] The mask generation unit is used to analyze each image block in the image block set to obtain an information entropy map that reflects the density complexity of each image block, and to perform a masking operation on each image block according to the information entropy map to obtain a masked image block set.

[0169] The present invention further provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multi-scale crowd counting method based on VSSM and mask reconstruction.

[0170] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0171] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0172] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0173] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that cannot be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A multi-scale crowd counting method based on VSSM and mask reconstruction, characterized in that, The specific steps include: Step 1: Standardize and multi-scale transform the input crowd image to obtain scaled input crowd images at different scales, then perform segmentation to obtain a set of image patches at each scale; Step 2: Analyze each image block in the image block set to obtain an information entropy map that reflects the density complexity of each image block, and perform a masking operation on each image block based on the information entropy map to obtain a masked image block set; Step 3: Input the masked image block into VSSM for feature extraction to obtain the encoded features at each scale. The VSSM establishes a global receptive field through a 2D selective scanning mechanism to generate encoded features with long-range dependencies. Step 4: Decode and reconstruct the encoded features using the Transformer decoder to generate reconstructed blocks, calculate the joint loss for self-supervised pre-training, and generate corresponding reconstructed feature images based on the reconstructed blocks at each scale; Step 5: Employ an attention-based multi-scale fusion module to fuse reconstructed feature images at different scales to generate fused features; Step 6: Generate a population density map based on the fusion features, and obtain the population counting results by analyzing the population density map; Divide the image patch into Divide the regions into equal-sized areas, calculate the information entropy of each area to form an information entropy map, and calculate the difference between the average information entropy of each area and its neighboring areas. This difference is called the information gap. Sort the regions according to their information entropy from largest to smallest to obtain an information entropy ranking sequence. Sort the regions according to their information gap from smallest to largest to obtain an information gap ranking sequence. Then, sort the regions in the information entropy ranking sequence from top to bottom. The regions constitute a set of information entropy regions, and the information gaps are arranged in a sequence. These regions constitute a set of information gap areas; Set the step size to 1, from The iteration begins. When the intersection of the information entropy set and the information gap set is greater than 20% of the total area of ​​the image patch, a random selection is made from the intersection of the information entropy set and the information gap set. A set of regions is formed, and a random selection operation is performed 10 times. This is a rounding up operation. The number of regions in the intersection of the information entropy set and the information gap set; For each entropy set, calculate the distance between the center coordinates of each region and the center coordinates of the image patch, and calculate the standard deviation of all distances. Set a standard deviation threshold and compare the standard deviation of the distances with the standard deviation threshold. If, in the 10 random selection operations, there exists an entropy set whose distance standard deviation is greater than the standard deviation threshold, the iteration terminates, and a random entropy set is selected to perform a full masking operation on the region in the entropy set. If, in the 10 random selection operations, there is no entropy set whose distance standard deviation is greater than the standard deviation threshold, the next iteration begins, and so on, until the iteration terminates.

2. The multi-scale crowd counting method based on VSSM and mask reconstruction according to claim 1, characterized in that: Standardize the input crowd image to The size of the input crowd image is determined, and a downsampling-based scaling transformation is performed on the standardized input crowd image to obtain three scaled input crowd images: 1 / 2 scale, 1 / 4 scale, and 1 / 8 scale. The 1 / 2 scale, 1 / 4 scale, and 1 / 8 scaled input crowd images are then segmented into... The image blocks are of various sizes, each with a specific dimensions. ,in .

3. The multi-scale crowd counting method based on VSSM and mask reconstruction according to claim 2, characterized in that: The VSSM adopts a visual state space model architecture, containing 4-12 VSS blocks, each VSS block containing the following components: The layer normalization module normalizes the input mask image blocks; The linear projection layer projects the normalized mask image patch onto the hidden dimension to obtain the mask image patch features of the hidden dimension. Depthwise separable convolutional layers perform depthwise separable convolution operations on the hidden dimension mask image patch features to obtain the input feature map, thereby enhancing the ability to model local features. The two-dimensional selective scanning module is used to perform four-way scanning processing on the input feature map to obtain directional fusion features; The output projection layer maps the oriented fusion features to the original dimensions; First residual connection layer: The directional fusion features mapped to the original dimension are residually connected to obtain the first encoded output feature; The feedforward neural network performs a nonlinear feature transformation on the first encoded output features; The second residual connection layer: The output of the feedforward neural network is residually connected to obtain the encoded features.

4. The multi-scale crowd counting method based on VSSM and mask reconstruction according to claim 3, characterized in that: A two-dimensional selective scanning module is used to perform four-directional scanning processing on the input feature map, including: The cross-scanning unit unfolds the input feature map into a one-dimensional sequence along four different directions to obtain four scan sequences. The four different directions include from left to right, from right to left, from top to bottom, and from bottom to top. The selective state-space processing unit independently applies the S6 block to each scan sequence for feature transformation. The S6 block is a state-space model with a selective mechanism. The cross-merging unit merges the scan sequences processed in the four directions to generate directional fusion features.

5. The multi-scale crowd counting method based on VSSM and mask reconstruction according to claim 1, characterized in that: The multi-scale fusion module includes the following components: The multi-scale feature alignment unit is used to unify reconstructed feature images at different scales to the same scale through a downsampling network, treating each scale as a channel. The channel attention computation unit is used to learn the importance weights of different channels through global average pooling and fully connected layers; The spatial attention computation unit is used to determine the attention distribution at different spatial locations in the reconstructed feature image through a combination of average pooling and max pooling. The adaptive fusion unit is used to perform weighted fusion of aligned multi-scale features based on channel attention and spatial attention to generate fused features.

6. The multi-scale crowd counting method based on VSSM and mask reconstruction according to claim 1, characterized in that: The formula for calculating the joint loss is: in, For joint losses, To rebuild the losses, For density loss, For counting losses, , , These are preset weighting coefficients; The reconstruction loss is calculated using the mean squared error, and the specific formula is as follows: in, To rebuild the losses, Represents the first in the original image The pixel value of the pixel at the masked position. Indicates the first in the reconstructed image The pixel value of the pixel at the masked position. This represents the set of masked positions. The number of masked positions; The density loss is calculated using mean square error, and the counting loss is calculated using absolute error.

7. A multi-scale crowd counting system based on mask reconstruction, characterized in that, The system is used to execute the multi-scale crowd counting method based on VSSM and mask reconstruction as described in any one of claims 1 to 6, including: Image preprocessing and multi-scale construction unit is used to standardize and multi-scale transform the input crowd image to obtain scaled input crowd images at different scales, and then perform segmentation operation to obtain a set of image patches at each scale; The mask generation unit is used to analyze each image block in the image block set to obtain an information entropy map that reflects the density complexity of each image block, and to perform a masking operation on each image block according to the information entropy map to obtain a masked image block set. The VSSM feature extraction unit is used to input the masked image block into the VSSM for feature extraction to obtain the encoded features at each scale. The VSSM establishes a global receptive field through a 2D selective scanning mechanism to generate encoded features with long-range dependencies. The Transformer decoding and image reconstruction unit is used to decode and reconstruct the encoded features through the Transformer decoder, generate reconstruction blocks, and calculate the joint loss for self-supervised pre-training; and generate corresponding reconstructed feature images based on the reconstruction blocks at each scale. The multi-scale feature fusion unit is used to fuse reconstructed feature images at different scales using an attention-based multi-scale fusion module to generate fused features. The density map generation and crowd counting unit is used to generate a crowd density map based on fused features and obtain crowd counting results by analyzing the crowd density map.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the multi-scale crowd counting method based on VSSM and mask reconstruction as described in any one of claims 1 to 6.

Images