Multi-scale crowd counting method and system based on VSSM and mask reconstruction, and medium

Abstract

The invention discloses a multi-scale crowd counting method and system based on VSSM and mask reconstruction and a medium, and belongs to the field of computer vision and deep learning, and the method comprises the steps: carrying out the multi-scale transformation and segmentation operation of an input crowd image, so as to obtain an image block under each scale; analyzing each image block to obtain an information entropy graph, performing masking operation on each image block according to the information entropy graph, inputting the masked image block into a VSSM to perform feature extraction so as to obtain coding features, performing decoding reconstruction on the coding features through a Transform decoder, and finally obtaining a reconstructed feature image; fusing the reconstructed feature images of different scales by using a multi-scale fusion module to generate fused features; and generating a crowd density map based on the fusion features, and analyzing the crowd density map to obtain a crowd counting result. Counting errors caused by single-scale analysis are avoided, and the crowd counting precision in a full-scale range is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and artificial intelligence, and in particular to a multi-scale crowd counting method, system and medium based on VSSM and mask reconstruction. Background Art

[0002] Crowd counting technology, as a key branch of computer vision, plays a vital role in public safety, urban planning, and other fields. With the rapid development of deep learning technology, crowd counting methods have evolved from early detection methods based on traditional image processing to density estimation methods based on deep neural networks. Current mainstream crowd counting methods are based on density estimation techniques. They train convolutional neural networks to generate a density map from an input image. Each pixel in the density map represents the crowd density at that location, and the final crowd count result is obtained by integrating the entire density map. This method not only provides information on crowd size but also displays the spatial distribution of the crowd, often demonstrating good robustness in complex scenarios. However, this method still has certain limitations.

[0003] The main limitations of existing technologies are: First, existing crowd counting technologies exhibit significant robustness deficiencies when dealing with partially missing images and complex scenarios. In real-world deployments, input images often suffer from partial information loss due to factors such as occlusion, limited field of view, or sensor failure. Furthermore, complex lighting conditions such as shadowed areas, strong sunlight, and low light conditions at night further complicate feature extraction. Traditional supervised learning methods rely heavily on complete, high-quality annotated data for training and lack the ability to adapt to incomplete input. Consequently, model performance plummets when faced with images with partially missing information.

[0004] Second, existing crowd counting technologies face dual challenges in terms of feature representation learning depth and computational efficiency. Current mainstream approaches, which employ convolutional neural networks (CNNs) or visual Transformer architectures, suffer from inherent technical bottlenecks. CNNs are limited by their local receptive field and struggle to model long-range dependencies, resulting in an inability to effectively capture global context in dense crowd scenes. While Transformers possess global modeling capabilities, the quadratic time complexity (O(n²)) of their self-attention mechanism imposes a significant computational burden when processing high-resolution crowd images, severely limiting the system's real-time performance. Summary of the Invention

[0005] The purpose of the present invention is to provide a multi-scale crowd counting method, system and device based on VSSM and mask reconstruction. By innovatively using a visual state space model and integrating a mask self-supervised learning mechanism with multi-scale feature processing technology, the present invention solves the technical problems of existing crowd counting technology, such as low accuracy in image missing and complex scenes, poor adaptability to multi-scale targets, low feature representation learning depth and low computational efficiency.

[0006] To achieve the above object, the present invention adopts the following technical solutions: A multi-scale crowd counting method based on VSSM and mask reconstruction includes the following steps: Step 1: After normalizing and multi-scale transforming the input crowd image to obtain scaled input crowd images of different scales, a segmentation operation is performed to obtain a set of image blocks at each scale; Step 2: Analyze each image block in the image block set to obtain an information entropy map reflecting the density complexity of each image block, and perform a mask operation on each image block according to the information entropy map to obtain a masked image block set; Step 3: Input the mask image block into VSSM for feature extraction to obtain the encoding features at each scale. The VSSM establishes a global receptive field through a 2D selective scanning mechanism to generate encoding features with long-range dependencies. Step 4: Decode and reconstruct the encoded features through the Transformer decoder to generate reconstructed blocks, calculate the joint loss for self-supervised pre-training, and generate corresponding reconstructed feature images based on the reconstructed blocks at each scale; Step 5: Use a multi-scale fusion module based on the attention mechanism to fuse the reconstructed feature images of different scales to generate fusion features; Step 6: Generate a crowd density map based on the fusion features, and obtain the crowd counting results by analyzing the crowd density map.

[0007] Furthermore, the input crowd image is normalized to The scaled input crowd images of three scales, namely 1 / 2 scale, 1 / 4 scale and 1 / 8 scale, are obtained. The scaled input crowd images of 1 / 2 scale, 1 / 4 scale and 1 / 8 scale are divided into The image blocks of the block, each image block has a size of ,in .

[0008] Furthermore, the image blocks are divided into The information entropy of each area is calculated to form an information entropy map, and the difference between the average information entropy of each area and its adjacent areas is calculated. This difference is called information drop. The divided areas are sorted in order from large to small according to the information entropy to obtain the information entropy arrangement sequence. The information drop arrangement sequence is obtained by sorting the divided areas in order from small to large according to the information drop. The first part of the information entropy arrangement sequence is sorted. The regions constitute the information entropy region set, and the information gap is arranged in the front of the sequence. The regions constitute the information gap region set; set the step size to 1, from Start and iterate. When the intersection area of ​​the information entropy set and the information gap set is greater than 20% of the total area of ​​the image block, randomly select regions, forming an entropy drop set, and performing random selection operations 10 times. For the rounding operation, is the number of regions in the intersection of the information entropy set and the information gap set; For each entropy drop set, calculate the distance between the center coordinates of each block and the center coordinates of the image block, and calculate the standard deviation of all distances. Preset the standard deviation threshold and compare the standard deviation of the distance with the standard deviation threshold. If there is an entropy drop set whose standard deviation of the distance is greater than the standard deviation threshold in 10 random selection operations, the iteration is terminated, and an entropy drop set is randomly selected to perform a full masking operation on the area in the entropy drop set; if there is no entropy drop set whose standard deviation of the distance is greater than the standard deviation threshold in 10 random selection operations, the next iteration is performed until the iteration is terminated.

[0009] Furthermore, the VSSM adopts a visual state space model architecture, including 4-12 VSS blocks, each of which contains the following components: Layer normalization module, which normalizes the input mask image blocks; The linear projection layer projects the normalized mask image block to the hidden dimension to obtain the mask image block features of the hidden dimension; The depth-wise separable convolution layer performs a depth-wise separable convolution operation on the mask image block features of the hidden dimension to obtain the input feature map to enhance the local feature modeling capability; The two-dimensional selective scanning module is used to perform four-directional scanning on the input feature map to obtain directional fusion features; Output projection layer, mapping the directional fusion features to the original dimension; The first residual connection layer: performs residual connection on the directional fusion features mapped to the original dimension to obtain the first encoded output features; A feedforward neural network performs nonlinear feature transformation on the first encoded output features; The second residual connection layer: performs residual connection on the output results of the feedforward neural network to obtain the encoding features.

[0010] Furthermore, the two-dimensional selective scanning module is used to perform four-directional scanning processing on the input feature map, including: A cross scanning unit expands the input feature map into a one-dimensional sequence along four different directions to obtain four scanning sequences. The four different directions include from left to right, from right to left, from top to bottom, and from bottom to top. The selective state space processing unit applies the S6 block to each scan sequence independently for feature transformation. The S6 block is a state space model with a selective mechanism. The cross-merging unit merges the scan sequences processed in four directions to generate directional fusion features.

[0011] Furthermore, the multi-scale fusion module includes the following components: Multi-scale feature alignment unit, used to unify the encoding features of different scales to the same scale through the downsampling network, treating each scale as a channel; Channel attention calculation unit, used to learn the importance weights of different channels through global average pooling and fully connected layers; A spatial attention calculation unit, which is used to determine the attention distribution of different spatial positions of the reconstructed feature image through a combination of average pooling and maximum pooling; The adaptive fusion unit is used to perform weighted fusion of the aligned multi-scale features based on channel attention and spatial attention to generate fused features.

[0012] Furthermore, the calculation formula of the joint loss is: ; in, For joint losses, To rebuild the losses, is the density loss, is the counting loss, 、 、 is the preset weight coefficient; The reconstruction loss is calculated using mean square error, and the specific formula is: ; in, To rebuild the losses, Indicates the original image The pixel value of the pixel at the masked position, Indicates the reconstructed image The pixel value of the pixel at the masked position, represents the set of masked positions, is the number of masked positions; The density loss is calculated using mean square error, and the counting loss is calculated using absolute error.

[0013] The present invention further provides a multi-scale crowd counting system based on mask reconstruction, which is used in the multi-scale crowd counting method based on VSSM and mask reconstruction, comprising: An image preprocessing and multi-scale construction unit is used to perform normalization and multi-scale transformation on the input crowd image to obtain scaled input crowd images of different scales, and then perform segmentation operations to obtain a set of image blocks at each scale; a mask generation unit, configured to analyze each image block in the image block set to obtain an information entropy map reflecting the density complexity of each image block, and perform a mask operation on each image block according to the information entropy map to obtain a mask image block set; A VSSM feature extraction unit is used to input the mask image block into the VSSM for feature extraction to obtain coding features at each scale. The VSSM establishes a global receptive field through a 2D selective scanning mechanism to generate coding features with long-range dependencies; A Transformer decoding and image reconstruction unit, configured to decode and reconstruct the encoded features through a Transformer decoder, generate reconstructed blocks, and calculate a joint loss for self-supervised pre-training; and generate corresponding reconstructed feature images based on the reconstructed blocks at each scale; A multi-scale feature fusion unit is used to fuse reconstructed feature images of different scales using a multi-scale fusion module based on an attention mechanism to generate fused features; The density map generation and crowd counting unit is used to generate a crowd density map based on the fusion features and obtain the crowd counting result by analyzing the crowd density map.

[0014] The present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the multi-scale crowd counting method based on VSSM and mask reconstruction.

[0015] Compared with the existing technology, the present invention has the following significant advantages: the present invention innovatively introduces the mask reconstruction task, which fundamentally changes the model's reliance on input integrity. By selectively masking each image block according to information entropy and information gap during the training phase, it can not only ensure that the masked area is the key learning area, but also avoid large-scale masking of adjacent areas; the mask forces the 2D selective scanning mechanism of the VSSM to strengthen cross-scale feature associations. The post-mask reconstruction task can enhance the model's ability to reason about local missing information. Even when some visual information is missing, the model can accurately infer the distribution of people in the occluded or missing area through contextual clues in the surrounding visible areas.

[0016] The introduction of the Visual State Space Model (VSSM) encoder combines global modeling capabilities with linear time complexity. Through its unique state-space modeling mechanism, the VSSM can capture long-range dependencies in images with a linear time complexity of O(n). It can fully leverage global contextual information to accurately identify partially occluded or highly overlapping targets while maintaining efficient computational performance in high-resolution image processing. Furthermore, the VSSM demonstrates excellent scalability—even with 4K or higher-resolution surveillance video streams, the system can maintain real-time processing capabilities without compromising accuracy or speed.

[0017] The technical solution of the present invention provides a more accurate and robust solution to the problem of crowd counting in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Schematic diagram of the process of the present invention; Figure 2 This is a framework diagram of the single-scale processing part based on image reconstruction of the present invention; Figure 3 This is a framework diagram of the feature fusion and density map generation part of the present invention; Figure 4 Schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0019] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to specific embodiments.

[0020] It should be noted that, unless otherwise defined, technical or scientific terms used in this invention should have the same ordinary meaning as those understood by persons of ordinary skill in the art to which this invention belongs. The terms "first," "second," and similar expressions used in this invention do not denote any order, quantity, or importance; they are simply used to distinguish different components. The terms "include" or "comprising" and similar expressions mean that the element or object preceding the word includes the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0021] Example 1: Basic implementation plan See also Figure 1 、 Figure 2 and Figure 3 , the present invention provides a technical solution: Step 1: After normalizing and multi-scale transforming the input crowd image to obtain scaled input crowd images of different scales, a segmentation operation is performed to obtain a set of image blocks at each scale; Furthermore, the input crowd image is normalized to The scaled input crowd images of three scales, namely 1 / 2 scale, 1 / 4 scale and 1 / 8 scale, are obtained. The scaled input crowd images of 1 / 2 scale, 1 / 4 scale and 1 / 8 scale are divided into The image blocks of the block, each image block has a size of ,in .

[0022] Step 2: Analyze each image block in the image block set to obtain an information entropy map reflecting the density complexity of each image block, and perform a mask operation on each image block according to the information entropy map to obtain a masked image block set; Furthermore, the image blocks are divided into The information entropy of each area is calculated to form an information entropy map, and the difference between the average information entropy of each area and its adjacent areas is calculated. This difference is called information drop. The divided areas are sorted in order from large to small according to the information entropy to obtain the information entropy arrangement sequence. The information drop arrangement sequence is obtained by sorting the divided areas in order from small to large according to the information drop. The first part of the information entropy arrangement sequence is sorted. The regions constitute the information entropy region set, and the information gap is arranged in the front of the sequence. The regions constitute a set of information gap regions.

[0023] The information entropy of each area is the average of the R channel information entropy, the G channel information entropy and the B channel information entropy of each area.

[0024] The local information entropy of each image block is calculated to reflect the density complexity of the area. Higher information entropy indicates a more complex population distribution in the area, and high-entropy areas are more prone to occlusion. Masking forces the VSSM's 2D selective scanning mechanism to strengthen cross-scale feature associations. The post-masking reconstruction task enhances the model's ability to reason about local missing information. The information gap reflects the difference between the analysis area and its adjacent areas. A larger value indicates a greater difference between the analysis area and its adjacent areas, indicating possible abnormal jumps in the analysis location. These jumps are irregular, making them difficult to identify during training. Masking areas with large jumps will not only fail to enhance the robustness of the subsequent model, but will actually make training more difficult and reduce robustness.

[0025] Set the step size to 1, from Start and iterate. When the intersection area of ​​the information entropy set and the information gap set is greater than 20% of the total area of ​​the image block, randomly select regions, forming an entropy drop set, and performing random selection operations 10 times. For the rounding operation, is the number of regions in the intersection of the information entropy set and the information gap set; For each entropy drop set, calculate the distance between the center coordinates of each block and the center coordinates of the image block, and calculate the standard deviation of all distances. Preset the standard deviation threshold and compare the standard deviation of the distance with the standard deviation threshold. Since this embodiment performs masking according to information entropy, for areas with more complex information, large adjacent areas may be masked. The subsequent VSSM technology is designed to explore long-range dependencies. Once large adjacent areas are masked, large semantic gaps will appear in the image, and long-range dependencies cannot be found through semantic relationships, resulting in complete failure of the model. By comparing the standard deviation of the distance with the standard deviation threshold, masking schemes that mask large adjacent areas can be screened out.

[0026] If there is an entropy drop set whose standard deviation of the distance is greater than the standard deviation threshold in 10 random selection operations, the iteration is terminated, and an entropy drop set is randomly selected to perform a full masking operation on the area in the entropy drop set; if there is no entropy drop set whose standard deviation of the distance is greater than the standard deviation threshold in 10 random selection operations, the next iteration is performed until the iteration is terminated.

[0027] The full mask operation is to set the R, G, and B values ​​of the masked pixels to 0.

[0028] Step 3: Input the mask image block into VSSM for feature extraction to obtain the encoding features at each scale. The VSSM establishes a global receptive field through a 2D selective scanning mechanism to generate encoding features with long-range dependencies. Existing crowd counting methods often encounter challenges such as crowding, blur, and occlusion in complex scenes. CNN-based methods tend to overfit local details, while Transformer-based methods excel at global modeling but suffer from a heavy computational burden due to their quadratic complexity in image size. These limitations motivate us to introduce VSSM, which captures comprehensive long-range information in linear time for crowd counting tasks.

[0029] The VSSM used in this embodiment is based on the VMamba architecture and is a state-space model optimized for processing two-dimensional visual data. Unlike traditional one-dimensional sequential state-space models, the VSSM achieves efficient modeling of image features through an innovative 2D Selective Sweep (SS2D) mechanism. The core of the VSSM is the VSS (Visual State Space) block, each of which contains: input layer normalization and linear projection; an SS2D module for two-dimensional selective scanning; a feedforward network for feature enhancement; and residual connections to ensure gradient flow.

[0030] The introduction of the Visual State Space Model (VSSM) encoder achieves a perfect combination of global modeling capabilities and linear time complexity. In stark contrast to the local receptive field limitations of CNNs and the quadratic time complexity burden of Transformers, the VSSM, through its unique state-space modeling mechanism, captures long-range dependencies in images with linear time complexity of O(n). This groundbreaking design enables the model to fully leverage global contextual information to accurately identify partially occluded or highly overlapping objects in dense crowd scenes, while maintaining efficient computational performance even in high-resolution image processing. In particular, in real-world deployment scenarios, the VSSM demonstrates exceptional scalability—even with 4K and higher-resolution surveillance video streams, the system maintains real-time processing capabilities without compromising accuracy or speed. This efficient global modeling capability not only significantly reduces hardware deployment costs but also enables the widespread application of high-precision crowd counting systems in resource-constrained environments such as edge computing devices. This truly represents a technological leap from "high performance relying on high computing power" to "high performance and high efficiency coexisting." The demand for high efficiency is particularly prominent in multi-scale processing.

[0031] Furthermore, the VSSM adopts a visual state space model architecture, including 4-12 VSS blocks, each of which contains the following components: Layer normalization module, which normalizes the input mask image blocks; The linear projection layer projects the normalized mask image block to the hidden dimension to obtain the mask image block features of the hidden dimension; The depth-wise separable convolution layer performs a depth-wise separable convolution operation on the mask image block features of the hidden dimension to obtain the input feature map to enhance the local feature modeling capability.

[0032] The two-dimensional selective scanning module is used to perform four-way scanning on the input feature map to obtain: The cross scanning unit expands the input feature map into a one-dimensional sequence along four different directions to obtain four scanning sequences. The four different directions include: Path 1: Scan from upper left to lower right, line by line; Path 2: From the lower right to the upper left, reverse scanning; Path 3: Scan from top to bottom, column by column; Path 4: From bottom to top, reverse column scanning; The selective state space processing unit applies the S6 block to each scanning sequence independently to perform feature transformation and obtain the scanning change features of the four scanning paths; The S6 block is a state space model with a selective mechanism, and its state space model is expressed as: ; ; in, Respectively Moment and The hidden state of the moment, express Moment and Input at the moment, 、 、 、 yes The weight parameters of the iteration are initialized 、 、 、 All are 0; Among them, the weight parameters are generated by input dependency: ; ; ; ; ; in, , , , , is the default linear layer, is the iteration step length, for Activation function. This input-dependent parameterization enables the model to adaptively adjust state transitions based on image content.

[0033] The S6 block (Selective Scan 6) is a core component of the state-space model and an improvement on the traditional state-space model. The S6 block implements a state-space model with a selectivity mechanism, selectively retaining or forgetting information based on the input content. Furthermore, compared to traditional attention mechanisms, the S6 block has linear complexity.

[0034] The cross-merging unit merges the scan sequences processed in four directions to generate directional fusion features.

[0035] ; in, is the direction fusion feature, For the The scanning change characteristics of the scanning path, The index of the scanned changing feature.

[0036] Output projection layer, mapping the directional fusion features to the original dimension; The first residual connection layer: performs residual connection on the directional fusion features mapped to the original dimension to obtain the first encoded output features; A feedforward neural network performs nonlinear feature transformation on the first encoded output features; The second residual connection layer: performs residual connection on the output results of the feedforward neural network to obtain the encoding features.

[0037] Step 4: Decode and reconstruct the encoded features through the Transformer decoder to generate reconstructed blocks, calculate the joint loss for self-supervised pre-training, and generate corresponding reconstructed feature images based on the reconstructed blocks at each scale; The existing technology is to decode and reconstruct the encoded features through the Transformer decoder, which contains L decoding layers. Each decoding layer includes multi-head self-attention, a feedforward network and a residual connection. The specific process is as follows: the first layer performs a linear transformation on the input encoded features, and rearranges the transformed visible features and placeholder vectors according to the original spatial positions, thereby obtaining a complete sequence representation containing visible information and placeholders. Among them, the placeholder vector fills the mask part; on this basis, position encoding is performed to obtain the input of the first layer of the decoder; for each decoding layer, the output of the previous layer is sent to the multi-head self-attention mechanism to obtain the dependency relationship representation of different positions in the sequence. The attention output is residually connected (added) to the original input, and then normalized through the layer to obtain an intermediate representation. The intermediate representation is input into the feedforward neural network, and the output of the feedforward neural network is connected to the residual to obtain the output result of this layer.

[0038] Furthermore, the calculation formula of the joint loss is: ; in, For joint losses, To rebuild the losses, is the density loss, is the counting loss, 、 、 is the preset weight coefficient; The reconstruction loss is calculated using mean square error, and the specific formula is: ; in, To rebuild the losses, Indicates the original image The pixel value of the pixel at the masked position, Indicates the reconstructed image The pixel value of the pixel at the masked position, represents the set of masked positions, is the number of masked positions; The density loss is calculated using mean square error, and the specific formula is: ; in, represents the density loss, Indicates that the true density map is at position The value of Indicates the predicted density map at position The value of and Represent the height and width of the density map respectively; The counting loss is calculated using absolute error, and the specific formula is: ; in, represents the counting loss, Indicates the real number of people, Indicates the predicted number of people.

[0039] Step 5: Use a multi-scale fusion module based on the attention mechanism to fuse the reconstructed feature images of different scales to generate fusion features; The multi-scale fusion module includes the following components: Multi-scale feature alignment unit, used to unify the encoding features of different scales to the same scale through the downsampling network, treating each scale as a channel; Channel attention calculation unit, used to learn the importance weights of different channels through global average pooling and fully connected layers; A spatial attention calculation unit, used to determine the attention distribution of different spatial positions in the reconstructed feature image through a combination of average pooling and maximum pooling; The adaptive fusion unit is used to perform weighted fusion of the aligned multi-scale features based on channel attention and spatial attention to generate fused features.

[0040] Since the reconstructed feature images at three scales have different spatial resolutions (1 / 2, 1 / 4, and 1 / 8), they need to be aligned to the same spatial scale for effective fusion. A branch-specific downsampling network is used to achieve scale alignment: For 1 / 2 scale features : ; For 1 / 4 scale features : ; For 1 / 8 scale features : ; in, Indicates the The convolutional layers are constructed with a stride of 2 and a kernel size of 2×2. Each convolutional layer is followed by a LeakyReLU activation function. Through this design, the reconstructed feature images of three different scales are all downsampled to a unified scale of 1 / 8. Reconstructed feature images at 1 / 2 scale, 1 / 4 scale, and 1 / 8 scale before alignment are the reconstructed feature images at 1 / 2 scale, 1 / 4 scale and 1 / 8 scale after alignment, is the index of the convolutional layer.

[0041] The aligned multi-scale features are concatenated in the channel dimension: ; in, Reconstruct feature images for stitching; Based on feature splicing, the attention mechanism is introduced to learn the importance weights of features at different scales.

[0042] Channel attention calculation: ; in, is channel attention, GAP represents global average pooling, and For dimensionality reduction operation, the default dimensionality reduction ratio is 4. is the Sigmoid activation function.

[0043] Spatial attention calculation: ; in, For spatial attention, AvgPool and MaxPool represent the average pooling and maximum pooling operations across channels, respectively.

[0044] Adaptive feature fusion: ; in, To fusion features, represents element-wise multiplication, Represents a 1×1 convolution layer, and the final 1×1 convolution is used to map the fused 3C channel features to the desired number of output channels.

[0045] The multi-scale feature fusion module uses a combination of channel-wise and spatial-wise attention to perform a weighted fusion of features at three different scales. By learning the importance weights of each scale feature at different spatial locations and feature channels, it generates a fused feature representation rich in semantic and geometric information. This fusion process effectively integrates multi-scale information and improves perception of crowds of varying densities.

[0046] Step 6: Generate a crowd density map based on the fusion features, and obtain the crowd counting results by analyzing the crowd density map.

[0047] Generate a crowd density map based on fusion features: ; in, is the population density map, To fusion features, It is a density regression head, which contains multiple convolutional layers: ; The crowd counting result is obtained by integrating the crowd density map: ; in, Calculate the results for the crowd, here .

[0048] During the training process, a standard deep learning optimization algorithm is used to iteratively update the network parameters to minimize the joint loss function.

[0049] See also Figure 4 The present invention further provides a multi-scale crowd counting system based on mask reconstruction, which is used in the multi-scale crowd counting method based on VSSM and mask reconstruction, comprising: An image preprocessing and multi-scale construction unit is used to perform normalization and multi-scale transformation on the input crowd image to obtain scaled input crowd images of different scales, and then perform segmentation operations to obtain a set of image blocks at each scale; a mask generation unit, configured to analyze each image block in the image block set to obtain an information entropy map reflecting the density complexity of each image block, and perform a mask operation on each image block according to the information entropy map to obtain a mask image block set; A VSSM feature extraction unit is used to input the mask image block into the VSSM for feature extraction to obtain coding features at each scale. The VSSM establishes a global receptive field through a 2D selective scanning mechanism to generate coding features with long-range dependencies; A Transformer decoding and image reconstruction unit, configured to decode and reconstruct the encoded features through a Transformer decoder, generate reconstructed blocks, and calculate a joint loss for self-supervised pre-training; and generate corresponding reconstructed feature images based on the reconstructed blocks at each scale; A multi-scale feature fusion unit is used to fuse reconstructed feature images of different scales using a multi-scale fusion module based on an attention mechanism to generate fused features; The density map generation and crowd counting unit is used to generate a crowd density map based on the fusion features and obtain the crowd counting result by analyzing the crowd density map.

[0050] The present invention further provides a multi-scale crowd counting system based on mask reconstruction, which is used in the multi-scale crowd counting method based on VSSM and mask reconstruction, comprising: An image preprocessing and multi-scale construction unit is used to perform normalization and multi-scale transformation on the input crowd image to obtain scaled input crowd images of different scales, and then perform segmentation operations to obtain a set of image blocks at each scale; a mask generation unit, configured to analyze each image block in the image block set to obtain an information entropy map reflecting the density complexity of each image block, and perform a mask operation on each image block according to the information entropy map to obtain a mask image block set; The present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the multi-scale crowd counting method based on VSSM and mask reconstruction.

[0051] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formulas are set by technicians in this field according to actual conditions.

[0052] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software depends on the specific application and design constraints of the technical solution.

[0053] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, and may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment as needed.

[0054] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technical personnel familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, and they should all be covered by the scope of protection of the present application.

Claims

1. A multi-scale crowd counting method based on VSSM and mask reconstruction, characterized in that: The specific steps include: Step 1: After normalizing and multi-scale transforming the input crowd image to obtain scaled input crowd images of different scales, a segmentation operation is performed to obtain a set of image blocks at each scale; Step 2: Analyze each image block in the image block set to obtain an information entropy map reflecting the density complexity of each image block, and perform a mask operation on each image block according to the information entropy map to obtain a masked image block set; Step 3: Input the mask image block into VSSM for feature extraction to obtain the encoding features at each scale. The VSSM establishes a global receptive field through a 2D selective scanning mechanism to generate encoding features with long-range dependencies. Step 4: Decode and reconstruct the encoded features through the Transformer decoder to generate reconstructed blocks, calculate the joint loss for self-supervised pre-training, and generate corresponding reconstructed feature images based on the reconstructed blocks at each scale; Step 5: Use a multi-scale fusion module based on the attention mechanism to fuse the reconstructed feature images of different scales to generate fusion features; Step 6: Generate a crowd density map based on the fusion features, and obtain the crowd counting results by analyzing the crowd density map.

2. The multi-scale crowd counting method based on VSSM and mask reconstruction according to claim 1 is characterized in that: Normalize the input crowd image to The scaled input crowd images of three scales, namely 1 / 2 scale, 1 / 4 scale and 1 / 8 scale, are obtained. The scaled input crowd images of 1 / 2 scale, 1 / 4 scale and 1 / 8 scale are divided into The image blocks of the block, each image block has a size of ,in .

3. The multi-scale crowd counting method based on VSSM and mask reconstruction according to claim 2 is characterized in that: Divide the image into blocks The information entropy of each area is calculated to form an information entropy map, and the difference between the average information entropy of each area and its adjacent areas is calculated. This difference is called information drop. The divided areas are sorted in order from large to small according to the information entropy to obtain the information entropy arrangement sequence. The information drop arrangement sequence is obtained by sorting the divided areas in order from small to large according to the information drop. The first part of the information entropy arrangement sequence is sorted. The regions constitute the information entropy region set, and the information gap is arranged in the front of the sequence. The regions constitute a set of information gap regions; Set the step size to 1, from Start the iteration. When the intersection area of ​​the information entropy set and the information gap set is greater than 20% of the total area of ​​the image block, randomly select regions, forming an entropy drop set, and performing random selection operations 10 times. For the rounding operation, is the number of regions in the intersection of the information entropy set and the information gap set; For each entropy drop set, calculate the distance between the center coordinates of each block and the center coordinates of the image block, and calculate the standard deviation of all distances. Preset the standard deviation threshold and compare the standard deviation of the distance with the standard deviation threshold. If there is an entropy drop set whose standard deviation of the distance is greater than the standard deviation threshold in 10 random selection operations, the iteration is terminated, and an entropy drop set is randomly selected to perform a full masking operation on the area in the entropy drop set; if there is no entropy drop set whose standard deviation of the distance is greater than the standard deviation threshold in 10 random selection operations, the next iteration is performed until the iteration is terminated.

4. The multi-scale crowd counting method based on VSSM and mask reconstruction according to claim 3, characterized in that: The VSSM adopts a visual state space model architecture, which includes 4-12 VSS blocks. Each VSS block contains the following components: Layer normalization module, which normalizes the input mask image blocks; The linear projection layer projects the normalized mask image block to the hidden dimension to obtain the mask image block features of the hidden dimension; The depth-wise separable convolution layer performs a depth-wise separable convolution operation on the mask image block features of the hidden dimension to obtain the input feature map to enhance the local feature modeling capability; The two-dimensional selective scanning module is used to perform four-directional scanning on the input feature map to obtain directional fusion features; Output projection layer, mapping the directional fusion features to the original dimension; The first residual connection layer: performs residual connection on the directional fusion features mapped to the original dimension to obtain the first encoded output features; A feedforward neural network performs nonlinear feature transformation on the first encoded output features; The second residual connection layer: performs residual connection on the output results of the feedforward neural network to obtain the encoding features.

5. The multi-scale crowd counting method based on VSSM and mask reconstruction according to claim 4 is characterized in that: The two-dimensional selective scanning module is used to perform four-directional scanning on the input feature map, including: A cross scanning unit expands the input feature map into a one-dimensional sequence along four different directions to obtain four scanning sequences. The four different directions include from left to right, from right to left, from top to bottom, and from bottom to top. The selective state space processing unit applies the S6 block to each scan sequence independently for feature transformation. The S6 block is a state space model with a selective mechanism. The cross-merging unit merges the scan sequences processed in four directions to generate directional fusion features.

6. The multi-scale crowd counting method based on VSSM and mask reconstruction according to claim 1, characterized in that: The multi-scale fusion module includes the following components: Multi-scale feature alignment unit, used to unify reconstructed feature images of different scales to the same scale through the downsampling network, treating each scale as a channel; Channel attention calculation unit, used to learn the importance weights of different channels through global average pooling and fully connected layers; A spatial attention calculation unit, used to determine the attention distribution of different spatial positions in the reconstructed feature image through a combination of average pooling and maximum pooling; The adaptive fusion unit is used to perform weighted fusion of the aligned multi-scale features based on channel attention and spatial attention to generate fused features.

7. The multi-scale crowd counting method based on VSSM and mask reconstruction according to claim 1, characterized in that: The formula for calculating the joint loss is: ； in, For joint losses, To rebuild the losses, is the density loss, is the counting loss, 、 、 is the preset weight coefficient; The reconstruction loss is calculated using mean square error, and the specific formula is: ； in, To rebuild the losses, Indicates the original image The pixel value of the pixel at the masked position, Indicates the reconstructed image The pixel value of the pixel at the masked position, represents the set of masked positions, is the number of masked positions; The density loss is calculated using mean square error, and the counting loss is calculated using absolute error.

8. A multi-scale crowd counting system based on mask reconstruction, characterized in that: The system is used to execute the multi-scale crowd counting method based on VSSM and mask reconstruction according to any one of claims 1 to 7, comprising: An image preprocessing and multi-scale construction unit is used to perform normalization and multi-scale transformation on the input crowd image to obtain scaled input crowd images of different scales, and then perform segmentation operations to obtain a set of image blocks at each scale; a mask generation unit, configured to analyze each image block in the image block set to obtain an information entropy map reflecting the density complexity of each image block, and perform a mask operation on each image block according to the information entropy map to obtain a mask image block set; A VSSM feature extraction unit is used to input the mask image block into the VSSM for feature extraction to obtain coding features at each scale. The VSSM establishes a global receptive field through a 2D selective scanning mechanism to generate coding features with long-range dependencies; A Transformer decoding and image reconstruction unit, configured to decode and reconstruct the encoded features through a Transformer decoder, generate reconstructed blocks, and calculate a joint loss for self-supervised pre-training; and generate corresponding reconstructed feature images based on the reconstructed blocks at each scale; A multi-scale feature fusion unit is used to fuse reconstructed feature images of different scales using a multi-scale fusion module based on an attention mechanism to generate fused features; The density map generation and crowd counting unit is used to generate a crowd density map based on the fusion features and obtain the crowd counting result by analyzing the crowd density map.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multi-scale crowd counting method based on VSSM and mask reconstruction according to any one of claims 1 to 7 is implemented.

Images