Crowd counting method based on multi-feature fusion VMama
By introducing the VMamba backbone network and methods such as multi-feature fusion, integrated attention, and elliptical-constrained deformable convolution, the problems of weak global perception and high computational complexity in crowd counting are solved, and real-time high-precision counting in high-density scenarios is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-08
- Publication Date
- 2026-03-10
AI Technical Summary
Existing technologies for crowd counting suffer from problems such as weak global perception, high computational complexity, sample dependency bottlenecks, and poor adaptability of feature fusion modules, resulting in poor counting performance in high-density scenarios.
Using VMamba as the backbone network, combined with multi-feature fusion, integrated attention and elliptical constrained deformable convolution, a four-way scanning module is used to achieve global receptive field and linear computational complexity, thereby enhancing feature fusion and sampling accuracy and generating a crowd density map.
Real-time counting is achieved on high-density and large-size images, improving counting accuracy and robustness, reducing dependence on training samples, and maintaining high accuracy, especially on small datasets.
Smart Images
Figure CN121640382A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer vision, and particularly relates to a crowd counting method based on multi-feature fusion VMamba. BACKGROUND
[0002] In the field of crowd counting, existing technologies mainly revolve around CNN, Transformer and emerging state space model (VMamba), and the core solutions can be divided into the following three categories: The first category is a basic counting architecture based on deep learning, which specifically includes: CNN architecture, mainly represented by CSRNet, MCNN and FGENet, which extracts features through convolution / dilation convolution, and partially combines multi-column branches to adapt to scale changes. However, the receptive field of CNN depends on the stacking of convolution kernels, and the ability to capture global information is weak, and the adaptability to head occlusion and scale difference in high-density scenes is limited.
[0003] Transformer derived architecture, mainly represented by TransCrowd and LoViTCrowd, which captures global pixel dependencies through multi-head attention. According to the VMambaCC document (Ma et al., 2024), this type of architecture has two major defects: (1) high computational complexity: the complexity of attention mechanism grows quadratically with the input image block size (O( )), and the inference time of a 1024x768 image is more than 1.5 times that of VMamba; (2) large sample requirement: a large amount of labeled data (such as 1201 training images of UCF_QNRF) is needed to achieve stable performance, and the MAE error on small datasets (such as UCF_CC_50 with only 50 images) is more than 90, which is much worse than the 64.34 of VMamba.
[0004] The second category is the auxiliary optimization of attention mechanism and deformable convolution, which specifically includes: Attention mechanism, such as CA (coordinate attention) and MLAttn (multi-order attention), which can enhance the focus on target regions, but when used alone, the parameter amount is large (such as MLAttnCNN parameters exceeding ), and lacks deep collaboration with the backbone network; Deformable convolution, such as the traditional deformable convolution used in DadNet, which initializes the sampling point offset randomly, making it difficult to adapt to the elliptical features of the head, and easily leading to insufficient feature aggregation.
[0005] The third category is the emerging application of state space model (VMamba), which specifically includes: As the first visual state space model, VMamba synchronously scans from the four corners of the image through the four-way scanning module (CSM) while maintaining linear computational complexity (O(n)) to achieve a global receptive field. It has been verified to have advantages in image classification and detection tasks, but its application in crowd counting is still blank, and it has not been combined with multi-feature fusion and constraint deformable convolution to solve the problems of scale change and background interference.
[0006] However, when the above prior art is adopted, the following technical problems often exist: First, the CNN global perception ability is weak: relying on local convolution kernels, it is difficult to capture the global distribution relationship of the head in high-density scenes, and the multi-scale feature expression is insufficient; Second, the efficiency and sample dependence bottleneck of Transformer: the computational complexity grows quadratically with the image size (O( )), making real-time deployment difficult; and a large number of training samples are required, with poor generalization on small data sets (such as TransCrowdMAE=97.2 on UCF_CC_50); Third, the sampling accuracy of deformable convolution is low: the sampling point offset of traditional deformable convolution has no geometric constraint and cannot adapt to the elliptical features of the head, resulting in feature sampling deviating from the target area; Fourth, the adaptability of backbone network and fusion module is weak: the feature output format of traditional CNN / Transformer does not match the needs of multi-feature fusion modules, which can easily cause resolution deviation or inconsistent feature distribution, resulting in poor crowd counting effect. SUMMARY
[0007] In order to solve the technical problem of poor crowd counting effect, the present application proposes a multi-feature fusion VMamba-based crowd counting method.
[0008] The present application provides a multi-feature fusion VMamba-based crowd counting method, specifically relates to the technology of crowd counting in target counting, and particularly relates to an end-to-end crowd counting network that takes VMamba (visual state space model) as the backbone network, fuses multi-scale features, integrates attention mechanism and constraint deformable convolution. The present application breaks through the efficiency and accuracy bottleneck of traditional CNN and Transformer in crowd counting through the linear computational complexity and global receptive field advantage of VMamba, and is suitable for real-time counting scenarios of high-density and large-size images. Specifically, a multi-feature fusion VMamba-based crowd counting method comprises: S1, obtaining a crowd image data set to be counted, performing resolution normalization processing on the image, and constructing an end-to-end crowd counting model taking VMamba as the backbone network, extracting four-stage multi-scale features of the image through the VMamba backbone network; The VMamba backbone network is composed of a Stem layer, four visual state space sub-blocks (VSS Blocks) and an adapter layer, and each VSS Block is internally provided with a four-way scanning module that synchronously scans features from four directions, i.e., the upper left, upper right, lower left and lower right of an image, and realizes global correlation modeling through state space transformation. S2, based on the four-stage multi-scale features output in step S1, performing a fusion operation through a multi-feature fusion module to generate multi-scale fused features; inputting the fused features into an integrated attention module to output target enhanced features through channel attention feature weight distribution and coordinate attention spatial positioning enhancement; S3, inputting the target enhanced features output in step S2 into an elliptical constraint deformable convolution module, performing dilated convolution processing on the features output by the elliptical constraint deformable convolution module to generate a crowd density map, summing the pixel values of the density map to obtain a final crowd counting result. The elliptical constraint deformable convolution module guides the sampling point offset of the deformable convolution through a preset elliptical constraint loss, so that the sampling points adapt to the elliptical geometric features of the heads of the crowd; the elliptical constraint loss includes a center sampling point constraint term, a horizontal symmetry constraint term, a vertical symmetry constraint term and an edge elliptical constraint term.
[0009] Optionally, the specific structure and operation parameters of the VMamba backbone network in step S1 include: The Stem layer: using 4x4 convolution, batch normalization operation and ReLU activation function to convert an input image with a resolution of 1024x768 into an initial feature map with a resolution of 256x192 and 64 channels; VSS Block 1: stacking two visual state space sub-blocks with CSM to output a feature map with a resolution of 128x96 and 128 channels; VSS Block 2: connecting three visual state space sub-blocks with CSM and 2x2 max pooling in series to generate a feature map with a resolution of 64x48 and 128 channels; VSS Block 3: connecting four visual state space sub-blocks with CSM and 2x2 max pooling in series to output a feature map with a resolution of 32x24 and 256 channels; VSS Block 4: connecting two visual state space sub-blocks with CSM and 2x2 max pooling in series to obtain a feature map with a resolution of 16x12 and 512 channels; The adapter layer: using 1x1 convolution to uniformly expand the channel number of the features output by VSS Block 1-2 to 128; correcting the resolution deviation of ±1 pixel through 3x3 kernel bilinear interpolation; adding a BN layer at the output end of each VSS Block to normalize the feature distribution to a range where the mean is close to 0 and the variance is close to 1.
[0010] Optionally, the fusion process of the multi-feature fusion module in step S2 includes: Perform 1×1 convolution and 2×2 max pooling on the feature map output by VSS Block 1 to generate intermediate fused features. ; Will The feature maps output by VSS Block 2 are concatenated along the channel dimension and then subjected to 2×2 max pooling to obtain intermediate fused features. ; Will The feature maps output by VSS Block 3 are concatenated along the channel dimension and then subjected to 2×2 max pooling to obtain the final fused features. ; Bilinear interpolation was used to upsample the feature map output by VSS Block 4, adjusting its resolution to 64×48. Together they serve as input features for the integrated attention module.
[0011] Optionally, the computation process of integrating the attention module in step S2 includes: Channel attention calculation includes: Let the input feature map be... Where H is the feature map height, W is the feature map width, and C is the number of feature map channels. Global average pooling and global max pooling are performed separately to obtain two compressed features of dimension 1×1×C. After adding the two compressed features, channel weights are generated by passing them through two layers of 1×1 convolution. After further processing with the Sigmoid activation function, the channel attention tensor is obtained. The mathematical expression is: ; , where σ() represents the Sigmoid activation function; Coordinate attention calculation includes: calculating Row average of channel c with column mean The mathematical expression is: ,in, for The pixel value in the h-th row and i-th column of the c-th channel; ,in for The pixel value in the j-th row and w-th column of the c-th channel; All channels and By concatenating the data along the channel dimensions, we obtain the row mean feature. with column mean features ; perform 1x1 convolution and ReLU activation on the concatenation result of with , to obtain feature ; split f into horizontal feature and vertical feature , respectively pass through 1x1 convolution and Sigmoid activation to obtain horizontal attention tensor and vertical attention tensor ; feature reweighting includes: and , , element-wise multiplication, output target enhanced feature , the mathematical expression is: , wherein represents element-wise multiplication operation.
[0012] Optionally, the elliptical constraint loss in step S3 includes: center sampling point constraint term : constraint the offset of center sampling point E and head target center point e, the expression is , wherein, , are the offset of E in x, y direction relative to e, represents the square of L2 norm; horizontal symmetry constraint term : constraint the symmetric distribution of horizontal direction sampling points D, F relative to e, the expression is , wherein, , are the x, y offset of D relative to e, , are the x, y offset of F relative to e; vertical symmetry constraint term : constraint the symmetric distribution of vertical direction sampling points B, H relative to e, the expression is , wherein, , are the x, y offset of B relative to e, , are the x, y offset of H relative to e; edge ellipse constraint term: including , , , respectively constraint the elliptical distribution of edge sampling points A and C, G and I, A and G, the expression is: , wherein, , , , , , , , These are the x and y offsets of sampling points A, C, G, and I relative to e, respectively. Total loss due to elliptic constraints: .
[0013] Optionally, steps S1 to S3 also include the model training process, including specific parameters and operations: Data augmentation includes: performing random scaling, horizontal flipping, and Gaussian noise perturbation on the preprocessed image; The optimizer configuration includes: using the Adam optimizer, with an initial learning rate set to... The weight decay coefficient is set to 5. Batch size is set to 8, and total training rounds are set to 300. The training strategy includes: freezing the VMamba backbone network parameters for the first 200 rounds, and only fine-tuning the parameters of the multi-feature fusion module, the integrated attention module, and the elliptical constrained deformable convolution module; unfreezing the backbone network for the last 100 rounds and performing joint training of the entire network. The total loss function is: ,in, The expression for the loss prediction of the density map is: Where N is the number of samples in the batch. For the input image The predicted density map for The true density map, For network parameters; This is the balance coefficient.
[0014] Optionally, the reasoning process in step S3 satisfies the following conditions, including: Under NVIDIA T4 GPU hardware, the inference time for extracting four-stage features from the input image to be counted via the VMamba backbone network is ≤31.2ms, and the total network computation is 69.8GFlops. Compared with the quadratic complexity of the Transformer architecture, the linear computational complexity of the VMamba backbone network improves the inference speed by ≥35% on a 1024×768 resolution image. The density map output by inference is used to obtain the number of people by summation, and the pixel values of the density map correspond one-to-one with the spatial distribution of the heads of the real people.
[0015] Optionally, the four-way scanning module of the VMamba backbone network in step S1 realizes global pixel-dependent modeling through state space transformation, without relying on the multi-head attention mechanism of the Transformer; the average absolute error of crowd counting on a small sample dataset is less than or equal to 183.1, which is better than the TransCrowd architecture; the MAE on a high-density dataset is 56.8, which is better than the DMCNet architecture.
[0016] The present application has the following advantages: The present application introduces VMamba (state space model) as the backbone network, replacing the quadratic complexity of the Transformer with linear computational complexity (O (n)), while retaining the global receptive field, solving the real-time counting problem of large-size images; through the deep cooperation of VMamba, MFF (multi-feature fusion), IA (integrated attention), and EDC (elliptical constraint deformable convolution), the robustness to scale changes and background interference is improved; the dependence of the model on training samples is reduced, and high-precision counting performance is still maintained on small datasets (such as UCF_CC_50), thereby improving the crowd counting effect. BRIEF DESCRIPTION OF DRAWINGS
[0017] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.
[0018] Figure 1 A flowchart of a multi-feature fusion VMamba-based crowd counting method of the present application; Figure 2 A VMamba backbone network sub-step flowchart of the present application; Figure 3 A multi-feature fusion module sub-step flowchart of the present application; Figure 4 An integrated attention module sub-step flowchart of the present application; Figure 5 An elliptical constraint deformable convolution sub-step flowchart of the present application; Figure 6 A MFIENet-VMamba overall operation flowchart of the present application; Figure 7 An integrated attention calculation architecture schematic diagram of the present application; Figure 8 An Eclipse-Restrict deformable convolution schematic diagram of the present application; Figure 9 Two visualization examples in the ablation experiment of the present application. DETAILED DESCRIPTION
[0019] In order to further clarify the technical means and effects taken by the present application to achieve the predetermined object of the application, the specific embodiments, structures, features and effects of the technical solutions proposed according to the present application are described in detail below in combination with the drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.
[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.
[0021] The MFIENet-VMamba proposed by the present application adopts an end-to-end architecture of "VMamba backbone + core module in series", and the overall process is: input image → VMamba backbone extracts four-stage features → MFF fuses multi-scale features → IA enhances target features → EDC precise sampling → dilated convolution → density map prediction → crowd counting. The core innovation lies in taking VMamba as the backbone to realize the dual advantages of "global perception + efficient calculation", which specifically includes: core backbone: VMamba (visual state space model) design; collaborative design of core module and VMamba; model training and inference process.
[0022] Referring to the four-way scanning strategy of the VMambaCC document, the VMamba backbone network of the present application is composed of a Stem layer + 4 VSS (visual state space) blocks + a connection adaptation layer. The four-way scanning module (CSM): each VSS block is built-in CSM, which synchronously scans the features from the four corners of the top left, top right, bottom left and bottom right of the image, captures the global pixel dependence through state space transformation, and avoids the O( ) attention calculation of Transformer; linear complexity guarantee: feature interaction only depends on "scanning order + state update", and the complexity grows linearly with the image size (O(n)), GFLOPs of 1024x768 image = 69.8, which is only 70% of TransCrowd (GFLOPs>100).
[0023] Referring to Figure 1 , the flow of some embodiments of a multi-feature fusion VMamba-based crowd counting method of the present application is shown. Figure 2 The sub-step flow of the VMamba backbone network of the present application is shown; Figure 3 The sub-step flow of the multi-feature fusion module (MFF) of the present application is shown; Figure 4An integrated attention module (IA) sub-step flow of the present application is shown; Figure 5 An ellipse-constrained deformable convolution (EDC) sub-step flow of the present application is shown; Figure 6 An MFIENet-VMamba overall operation flow (core module series graph) of the present application is shown; Figure 7 An integrated attention computing architecture of the present application is shown; Figure 8 An Eclipse-Restrict deformable convolution of the present application is shown; Figure 9 Two visualization examples in the ablation experiment of the present application are shown. Specifically, a crowd counting method based on multi-feature fusion VMamba includes the following steps: Step S1, obtain a crowd image dataset to be counted, perform resolution normalization processing on the image, and construct an end-to-end crowd counting model with VMamba as the backbone network, and extract four-stage multi-scale features of the image through the VMamba backbone network.
[0024] In some embodiments, data preprocessing and backbone network feature extraction: obtain a crowd image dataset to be counted, perform resolution normalization processing (uniformly adjusted to 1024x768) on the image, and construct an end-to-end crowd counting model with VMamba (Visual StateSpace Model, visual state space model) as the backbone network; extract four-stage multi-scale features of the image through the above VMamba backbone network.
[0025] The above VMamba backbone network is composed of a Stem layer, 4 visual state space sub-blocks (VSS Block) and a connection adaptation layer, and each VSS Block is built-in a cross-scan module (CSM). The cross-scan module (CSM) synchronously scans features from the top left, top right, bottom left and bottom right of the image, realizes global correlation modeling through state space transformation, and maintains linear computational complexity (O (n), n is the total number of image pixels).
[0026] It should be noted that the cross-scan module (CSM) of the above VMamba backbone network in step S1 realizes global pixel-dependent modeling through state space transformation, without relying on the multi-head attention mechanism of the Transformer; the mean absolute error (MAE) of crowd counting on a small sample dataset (such as UCF_CC_50, containing 50 training images) is ≤183.1, which is better than the TransCrowd architecture (MAE=97.2); the MAE on a high-density dataset (such as ShanghaiTech Part_A, containing 300 training images) is 56.8, which is better than the DMCNet architecture (MAE=58.5).
[0027] As an example, the specific structure and operation parameters of the above-mentioned VMamba backbone network in step S1 are as follows: Stem layer: using 4x4 convolution (step size 4, 64 convolution filters configured), Batch Normalization (BN) operation and ReLU activation function, the input image with a resolution of 1024x768 is converted into an initial feature map with a resolution of 256x192 and 64 channels, realizing the compression of the 1024x768 input image; VSS Block 1: stacking 2 visual state space sub-blocks containing CSM (without pooling operation), four-way scanning capturing local-global feature association, outputting a feature map with a resolution of 128x96 and 128 channels; VSS Block 2: connecting 3 visual state space sub-blocks containing CSM and 2x2 max pooling (step size 2) in series, enhancing the expression of medium-scale features, generating a feature map with a resolution of 64x48 and 128 channels; VSS Block 3: connecting 4 visual state space sub-blocks containing CSM and 2x2 max pooling (step size 2) in series, adapting to the head features of high-density scenes, outputting a feature map with a resolution of 32x24 and 256 channels; VSS Block 4: connecting 2 visual state space sub-blocks containing CSM and 2x2 max pooling (step size 2) in series, extracting high-level semantic features, obtaining a feature map with a resolution of 16x12 and 512 channels; Adaptation layer: using 1x1 convolution (configuring 128 convolution filters) to uniformly expand the channel number of the output features of VSS Block 1-2 to 128; correcting the resolution deviation of ±1 pixel by 3x3 kernel bilinear interpolation; adding a BN layer at the output end of each VSS Block to normalize the feature distribution to a range with a mean close to 0 and a variance close to 1.
[0028] Step S2, based on the four-stage multi-scale features output in step S1, performing fusion operation through multi-feature fusion module, generating multi-scale fusion features; inputting the fusion features into the integrated attention module, enhancing through channel attention feature weight distribution and coordinate attention spatial positioning, outputting target enhanced features.
[0029] In some embodiments, multi-feature fusion and attention enhancement: based on the four-stage multi-scale features output by step S1, a fusion operation of “step-by-step channel splicing-pooling dimension reduction-up sampling completion” is performed by a multi-feature fusion module (MFF) to generate multi-scale fusion features; the above fusion features are input into an integrate attention module (IA), and through the feature weight distribution of channel attention and the spatial positioning enhancement of coordinate attention, background noise is suppressed and crowd region features are strengthened to output target enhanced features .
[0030] As an example, the multi-feature fusion module (MFF) is based on the four-stage features output by VMamba, and adopts a “step-by-step splicing + up sampling completion” strategy. Specifically, the fusion process of the above multi-feature fusion module (MFF) in step S2 can include: performing 1x1 convolution (configuring 128 convolution filters) and 2x2 max pooling (step size 2) on the feature map output by VSS Block 1 to generate intermediate fusion features ; splicing and the feature map output by VSS Block 2 along the channel dimension, and processing by 2x2 max pooling (step size 2) to obtain intermediate fusion features ; splicing and the feature map output by VSS Block 3 along the channel dimension, and processing by 2x2 max pooling (step size 2) to obtain final fusion features ; performing up sampling operation on the feature map output by VSS Block 4 by bilinear interpolation to adjust its resolution to 64x48, and together as input features of the integrate attention module (IA).
[0031] It should be noted that the VMamba has the advantages of synergy: the global features of VMamba make it unnecessary to supplement global information when MFF is fused, and the semantic consistency of the fused features is improved by 40% (ablation experiment verifies that MAE is reduced by 3.2).
[0032] As another example, the above integrate attention module (IA) in step S2 mainly includes fusion channel attention and coordinate attention, and realizes “channel weight distribution + spatial positioning enhancement”. Specifically, it can include: Channel attention calculation includes: let the input feature map be , wherein, "Input Feature Map" is the English name for the input feature map, specifically referring to the input feature tensor of the model. Following the naming convention in deep learning, "F" represents "Feature," H is the feature map height, W is the feature map width, and C is the number of feature map channels. Global average pooling (AvgPool) and global max pooling (MaxPool) are performed separately to obtain two compressed features of dimension 1×1×C. After adding the two compressed features, channel weights are generated by passing them through two layers of 1×1 convolution. After further processing with the Sigmoid activation function, the channel attention tensor is obtained. The mathematical expression is: Channel attention tensor obtained after sigmoid activation , where σ() represents the Sigmoid activation function; The full English name is Channel Attention Tensor, which refers to the final channel attention output obtained after processing by the Sigmoid activation function.
[0033] Coordinate attention calculation includes: calculating Channel c ( row mean with column mean The mathematical expression is: ,in, for The pixel value in the h-th row and i-th column of the c-th channel; h is the row index, which refers to the row position of the feature map, and its value range is usually from 1 to H. H is the height of the input feature map, which is used to locate the specific row in the feature map; i is the column index, which refers to the column position of the feature map, and its value range is usually from 1 to W. W is the width of the input feature map, which is used to locate the specific column in the feature map. ,in for The pixel value in the c-th channel, row j, column w. j is the row index, used to locate the row position of the feature map; w is the column index, used to locate the column position of the feature map; the row mean refers to the value of the input feature map. The c-th channel's feature map is the average value calculated along the "column direction" over all pixel values. The column mean, or column average, refers to the input feature map. The feature map of channel c is the average value of all pixel values calculated along the "row direction".
[0034] All channels and Concatenate along the channel dimension respectively to obtain row mean features and column mean features ; Perform 1x1 convolution (compression ratio r) and ReLU activation on the concatenation result of to obtain features ; split f into horizontal features and vertical features , respectively pass through 1x1 convolution and Sigmoid activation to obtain horizontal attention tensor and vertical attention tensor ; Feature reweighting includes: element-wise multiplication of and , , , output target enhanced features , the mathematical expression is: , wherein represents element-wise multiplication.
[0035] Step S3: input the target enhanced features output by step S2 into the ellipse-constrained deformable convolution module, perform dilated convolution processing on the features output by the ellipse-constrained deformable convolution module, generate a crowd density map, and sum the pixel values of the density map to obtain the final crowd counting result.
[0036] In some embodiments, ellipse-constrained sampling and density map prediction: input output by step S2 into an ellipse-constrained deformable convolution module (EDC), the above ellipse-constrained deformable convolution module (EDC) guides the sampling point offset of deformable convolution through a preset ellipse-constrained loss ( ) to adapt the sampling point to the ellipse geometric features of the crowd head; perform dilated convolution (DC) processing on the EDC output features to generate a crowd density map; sum the pixel values of the above density map to obtain the final crowd counting result.
[0037] Wherein, the above ellipse-constrained loss contains a center sampling point constraint term ( ), a horizontal symmetry constraint term ( ), a vertical symmetry constraint term ( ) and an edge ellipse constraint term ( , , ), which is used to constrain the sampling point distribution to accurately cover the head region.
[0038] It should be noted that the elliptical constraint deformable convolution module (EDC) is constrained by a self-defined loss function to constrain the sampling point offset, adapt to the head elliptical feature, and its convolution parameter setting includes: using a 3x3 convolution kernel, 64 filters, and adding an elliptical constraint loss on the basis of traditional deformable convolution ).
[0039] As an example, the above elliptical constraint loss in step S3 can include: a center sampling point constraint term : constraint the offset of the center sampling point E and the head target center point e, ensure that the center sampling point E is close to the target point e, and the expression is , wherein , E_x and E_y are the offset of E in the x and y directions relative to e, represents the square of the L2 norm; a horizontal symmetry constraint term : constraint the symmetric distribution of horizontal direction sampling points D and F relative to e, ensure that the horizontal distance of D and F to e is equal and close to the x axis, and the expression is , wherein , D_x and D_y are the x and y offset of D relative to e, , F_x and F_y are the x and y offset of F relative to e; a vertical symmetry constraint term : constraint the symmetric distribution of vertical direction sampling points B and H relative to e, ensure that the vertical distance of B and H to e is equal and close to the y axis, and the expression is , wherein , B_x and B_y are the x and y offset of B relative to e, , H_x and H_y are the x and y offset of H relative to e; an edge ellipse constraint term: ensure that A, C, G, and I are distributed on the edge of the ellipse, including , , (the calculation method is the same as ), respectively constrain the elliptical distribution of edge sampling points A and C, G and I, A and G, and the expression is: , wherein , , , , , , , A_x, A_y, C_x, C_y, G_x, G_y, I_x, and I_y are the x and y offset of sampling points A, C, G, and I relative to e; an elliptical constraint total loss: .
[0040] As another example, steps S1 to S3 also include the model training process, and the specific parameters and operations may include: Images were uniformly resized to 1024×768 (bilinear interpolation), and annotation files were scaled proportionally. Data augmentation includes: performing random scaling (scaling factor range of 0.7~1.3), horizontal flipping, and Gaussian noise perturbation (noise standard deviation of 0.01) on the preprocessed image to adapt to the large input size requirements of VMamba and improve the model's generalization ability; The optimizer configuration includes: using the Adam optimizer, with an initial learning rate set to... The weight decay coefficient is set to 5. The batch size is set to 8, and the total number of training rounds is set to 300. The training strategy includes: freezing the VMamba backbone network parameters for the first 200 rounds, and only fine-tuning the parameters of the Multi-Feature Fusion (MFF), Integrated Attention (IA), and Elliptical Constrained Deformable Convolution (EDC) modules; and unfreezing the backbone network for the last 100 rounds to perform joint training of the entire network. The total loss function is: ,in, The expression for the density map prediction loss (using L2 loss) is: Where N is the number of samples in the batch. For the input image The predicted density map for The true density map, For network parameters; The balance coefficient is set to 1 after experimental verification. The model performs relatively optimally at that time.
[0041] Inference process: Input image to be counted → VMamba extracts four-stage features (linear complexity O(n), inference time 31.2ms) → MFF fusion → IA enhancement → EDC sampling → dilated convolution → output density map → sum to obtain the number of people.
[0042] Optionally, the inference process in step S3 satisfies the following conditions: Under the NVIDIA T4 GPU hardware environment, the inference time for extracting four-stage features from the input image to be counted via the VMamba backbone network is ≤31.2ms, and the total network computation (GFLOPs) is 69.8; the linear computational complexity (O(n)) of the VMamba backbone network is significantly lower than the quadratic complexity (O(n)) of the Transformer architecture. ), the inference speed is increased by ≥ 35% on 1024*768 resolution images; the density map output is summed to obtain the crowd number, and the pixel value of the density map corresponds to the spatial distribution of the real crowd head.
[0043] Compared with the sampling point constraint mode of parallelogram (or rectangle), the elliptical constraint proposed in the application has significant advantages in geometric adaptability, sampling accuracy, occlusion robustness and mathematical modeling rigor, which are embodied in the following four aspects: First, the high adaptability of the geometric shape to the biological characteristics of the human head.
[0044] From the biological characteristics and two-dimensional projection geometric characteristics, the human head as a three-dimensional structure of an approximate sphere, its projection profile on the image plane presents typical elliptical characteristics-the left and right diameters (major axis) and the upper and lower diameters (minor axis) have a stable proportional relationship (usually 1.2~1.5:1), and the profile edge is a continuous and smooth quadratic curve without obvious corners or linear transitions. The parallelogram (or rectangle) belongs to a polygonal geometric structure, the boundary of which is composed of four straight line segments, and the opposite sides are parallel and the adjacent sides have a fixed angle (90° for a rectangle), which cannot fit the arc transition characteristics of the head profile: in the forehead, mandible and other areas with large curvature, the straight boundary of the parallelogram will have a "geometric deviation angle" of 5°~15° with the actual head profile, resulting in that the sampling area either exceeds the head range (including background noise) or fails to cover the effective features of the head edge (such as the temporal part and the mandibular line), forming a "sampling blind area". The elliptical constraint can accurately match the elliptical profile of the head projection by pre-setting the long semi-axis (a) and the short semi-axis (b) parameters (which can be dynamically adjusted according to the head size statistics of the data set, such as a=8 pixels and b=6 pixels in the ShanghaiTech Part_A data set), so that the average deviation of the distribution boundary of the sampling points from the actual profile of the head is controlled within 2 pixels, and the geometric adaptation error is reduced by more than 60% compared with the parallelogram constraint.
[0045] Second, the dynamic accuracy and information integrity of the sampling point distribution.
[0046] From the information theory efficiency of the sampling strategy, the parallelogram constraint is based on the linear constraint logic of "parallel opposite sides", and the offset direction of the sampling points is limited to two fixed directions parallel to the adjacent sides, which cannot dynamically adjust the sampling density according to the curvature change of the head profile-in areas with small curvature of the head profile (such as the top of the head), uniform sampling of the parallelogram will cause redundant sampling (repetition of information); in areas with large curvature (such as the mandibular line), the feature is missed due to the large spacing between the sampling points. The elliptical constraint is based on the quadratic curve equation ( ) to construct a sampling point distribution model, and the edge elliptical constraint term ( , , )Non-linear constraint on the offset of sampling points: In areas with large head contour curvature (such as the lower jaw line, with a curvature radius of usually 3-5 pixels), the elliptical constraint will guide the sampling points to be more densely distributed (sampling interval ≤ 1 pixel) by increasing the L2 norm penalty, ensuring the complete capture of curve features; in areas with small curvature (such as the top of the head, with a curvature radius ≥ 10 pixels), the penalty is reduced to allow the sampling points to be moderately sparse (sampling interval ≤ 2 pixels), avoiding redundant calculations. Experimental logic verification shows that this dynamic sampling strategy can improve the sampling coverage rate of head features from 75%-80% under parallelogram constraint to 92%-95%, effectively increasing the amount of information obtained by 15%-20%.
[0047] Third, the robustness improvement in high-density occlusion scenarios.
[0048] In high-density crowd scenarios (such as the ShanghaiTech Part_A dataset, with a single image containing ≥ 500 people), the head occlusion rate often reaches 30%-50%, and the main form of occlusion is "partial overlap" (such as ears and cheeks being occluded, and forehead and eyes being exposed). The parallelogram constraint cannot distinguish between "head effective area" and "occluded background area" due to its reliance on fixed linear boundaries - when the parallelogram boundary covers part of the occluded background, its linear constraint will include background noise (such as clothing texture and hair of adjacent heads) in the sampling range, leading to feature confusion. The elliptical constraint, on the other hand, has "effective area screening capability" through the coordinated action of multiple-dimensional constraint terms: the center sampling point constraint term ( ) ensures that the sampling center focuses on the core area of the head that is not occluded (such as the eyebrow center, usually the geometric center of the head, with an occlusion probability ≤ 10%); the horizontal / vertical symmetry constraint term ( 、 ) excludes occluded background points (such as the outer side of the cheek occluded by an adjacent head) by constraining the symmetric distribution of sampling points; and the edge ellipse constraint term automatically filters background sampling points that exceed the head contour (such as the edge pixels of the occluding object) based on the ellipse boundary. Through this multi-layer screening mechanism, the background noise suppression rate of the elliptical constraint is improved by 40%-45% compared to the parallelogram constraint, and the counting error caused by feature confusion is reduced by 25%-30%.
[0049] Fourth, the rigor of mathematical modeling and the completeness of constraint logic.
[0050] From the mathematical expression of the constraint model, the loss function of the parallelogram constraint is constructed based on the parallelogram rule of linear algebra, and the constraint condition is a system of linear equations (such as 、 The parallelogram constraint can only describe linear geometric relationships such as "parallel sides, equal opposite sides," while the elliptical contour of the head projection is a nonlinear quadratic curve. The linear constraint suffers from a "model mismatch error" with the actual object, causing the constraint strength to become unstable with changes in head posture (such as pitch and yaw angles). When the head pitches within ±10°, the sampling error of the parallelogram constraint increases by 20% to 25%. The loss function of the elliptical constraint (…) Based on the geometric properties of quadratic curves, a complete constraint system of "center positioning - axis alignment - edge contour" is formed: center sampling point constraint term ( The center coordinates of the corresponding ellipse are (x0, y0), which is obtained by using the square of the L2 norm (x0, y0). Precise penalty for center offset; horizontal / vertical symmetry constraint terms ( , The alignment characteristics of the major and minor axes of the corresponding ellipse ensure that the ellipse is consistent with the horizontal / vertical symmetry axis of the head; the edge ellipse constraint term ( , , This corresponds to the boundary equation of the ellipse. By constraining the relative positions of the edge sampling points, it ensures that the sampling area strictly conforms to the elliptical contour. This nonlinear constraint model can adapt to head posture changes within ±15°, and its constraint stability is improved by 35%~40% compared to the parallelogram constraint, with a significant reduction in modeling error.
[0051] In summary, this invention introduces VMamba (Visual State Space Model) as the backbone network, replacing the quadratic complexity of Transformer with linear computational complexity (O(n)) while preserving the global receptive field, thus solving the problem of real-time counting of large-size images. Through deep collaboration between VMamba and MFF (Multi-Feature Fusion), IA (Integrated Attention), and EDC (Elliptical Constrained Deformable Convolution), robustness to scale changes and background interference is improved. The model's dependence on training samples is reduced, and high-precision counting performance is maintained on small datasets (such as UCF_CC_50), thereby improving the crowd counting effect.
[0052] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A crowd counting method based on multi-feature fusion VMamba, characterized in that, The method comprises the following steps: S1, obtaining a to-be-counted crowd image dataset, performing resolution normalization on the image, and constructing an end-to-end crowd counting model with a VMamba backbone network, and extracting four-stage multi-scale features of the image through the VMamba backbone network; The VMamba backbone network is composed of a Stem layer, four visual state space sub-blocks and a connection adaptation layer, and each VSSBlock is internally provided with a four-way scanning module, the four-way scanning module synchronously scans features from four directions of the top left, top right, bottom left and bottom right of the image, and realizes global correlation modeling through state space transformation; S2, based on the four-stage multi-scale features output by step S1, performing fusion operation through a multi-feature fusion module to generate multi-scale fusion features; inputting the fusion features into an integrated attention module, and outputting target enhanced features through channel attention feature weight distribution and coordinate attention spatial positioning enhancement; S3, inputting the target enhanced features output by step S2 into an elliptical constraint deformable convolution module, performing dilated convolution processing on the output features of the elliptical constraint deformable convolution module to generate a crowd density map, and summing pixel values of the density map to obtain a final crowd counting result; The elliptical constraint deformable convolution module guides the sampling point offset of the deformable convolution through a preset elliptical constraint loss, so that the sampling point adapts to the elliptical geometric features of the crowd head; The elliptical constraint loss includes a center sampling point constraint term, a horizontal symmetry constraint term, a vertical symmetry constraint term and an edge elliptical constraint term.
2. The crowd counting method based on multi-feature fusion VMamba according to claim 1, characterized in that, The specific structure and operation parameters of the VMamba backbone network in step S1 include: The Stem layer: using 4x4 convolution, batch normalization operation and ReLU activation function, converts an input image with a resolution of 1024x768 into an initial feature map with a resolution of 256x192 and 64 channels; VSS Block 1: stacking 2 visual state space sub-blocks with CSM, outputting a feature map with a resolution of 128x96 and 128 channels; VSS Block 2: connecting 3 visual state space sub-blocks with CSM and 2x2 maximum pooling in series, generating a feature map with a resolution of 64x48 and 128 channels; VSS Block 3: connecting 4 visual state space sub-blocks with CSM and 2x2 maximum pooling in series, outputting a feature map with a resolution of 32x24 and 256 channels; VSS Block 4: connecting 2 visual state space sub-blocks with CSM and 2x2 maximum pooling in series, obtaining a feature map with a resolution of 16x12 and 512 channels; The connection adaptation layer: using 1x1 convolution to uniformly expand the channel number of the output features of VSS Block 1-2 to 128; correcting the resolution deviation of ±1 pixel by 3x3 kernel bilinear interpolation; adding a BN layer at the output end of each VSS Block to normalize the feature distribution to a range with a mean close to 0 and a variance close to 1.
3. The crowd counting method based on multi-feature fusion VMamba according to claim 2, characterized in that, The fusion process of the multi-feature fusion module in step S2 includes: performing a 1x1 convolution and 2x2 max pooling on the output of the VSS Block 1 to generate intermediate fused features ; Will The feature maps output by VSS Block 2 are concatenated along the channel dimension and then subjected to 2×2 max pooling to obtain the intermediate fused features. ; Will The feature maps output by VSS Block 3 are concatenated along the channel dimension and then subjected to 2×2 max pooling to obtain the final fused features. ; The up-sampling operation is performed on the feature map output by VSS Block 4 using bilinear interpolation to adjust its resolution to 64x48, which is concatenated with the output of VSS Block 3 as the input feature of the integrated attention module. The output of the integrated attention module is concatenated with the output of VSS Block 3 as the input feature of the VSS Block 4.
4. The crowd counting method based on multi-feature fusion VMamba of claim 1, wherein, The calculation process of the integrated attention module in step S2 includes: The channel attention calculation comprises: taking an input feature map as wherein H is a feature map height, W is a feature map width, and C is a feature map channel number, and performing global average pooling and global maximum pooling on the input feature map respectively to obtain two compressed features with a dimension of 1×1×C; adding the two compressed features and sequentially passing the added compressed features through two 1×1 convolution layers to generate a channel weight , and performing Sigmoid activation function processing on the channel weight to obtain a channel attention tensor , and a mathematical expression is as follows: ； where σ() denotes a Sigmoid activation function; The coordinate attention computation includes computing a row mean of the cth channel a column mean of the cth channel where the mathematical expression is: where, is a pixel value of the cth channel in the hth row and the ith column. wherein is a pixel value of the cth channel, the jth row, and the wth column. concatenate all channels of with row mean features along channel dimension respectively and column mean features ; perform 1x1 convolution and ReLU activation on the concatenation result of and to obtain feature ; split f into horizontal feature and vertical feature , respectively pass through 1x1 convolution and Sigmoid activation to obtain horizontal attention tensor and vertical attention tensor ; The feature reweighting includes: multiplying with , , element by element, to output a target enhanced feature , and the mathematical expression is: , wherein represents an element-by-element multiplication operation.
5. The crowd counting method based on multi-feature fusion VMamba of claim 1, wherein, The elliptical constraint loss in step S3 includes: center sample point constraint term : constraint the offset between the center sample point E and the head target center point e, expressed as wherein, , are the offset of E in x, y direction relative to e, denotes the square of L2 norm; horizontal symmetry constraint term : constraint the symmetric distribution of horizontal sampling points D, F relative to e, expressed as wherein, , is the x, y offset of D relative to e, , is the x, y offset of F relative to e; Vertical symmetry constraint term : constraint the symmetric distribution of vertical direction sampling points B, H relative to e, expressed as , wherein , is the x, y offset of B relative to e, , is the x, y offset of H relative to e; Edge ellipse constraint term: including , , , respectively constrain the ellipse distribution of edge sampling points A and C, G and I, A and G, and the expression is: Wherein, , , , , , , , are the x, y offset of sampling points A, C, G, I relative to e. Ellipse constraint total loss: .
6. The crowd counting method based on multi-feature fusion VMamba according to claim 1, characterized in that, The steps S1-S3 further include a model training process, and specific parameters and operations include: Data enhancement includes: performing random scaling, horizontal flipping, and Gaussian noise disturbance on the preprocessed image; The optimizer configuration comprises: adopting an Adam optimizer, setting an initial learning rate to , setting a weight decay coefficient to 5 , setting a batch size to 8, and setting a total training round to 300 rounds; The training strategy includes: freezing the VMamba backbone network parameters for the first 200 rounds, and only fine-tuning the parameters of the multi-feature fusion module, the integrated attention module, and the elliptical constraint deformable convolution module; unfreezing the backbone network for the last 100 rounds, and performing joint training of the whole network; The total loss function is: ,in, The expression for the loss prediction based on the density map is: Where N is the number of samples in the batch. For the input image The predicted density map for The true density map, For network parameters; This is the balance coefficient.
7. The crowd counting method based on multi-feature fusion VMamba according to claim 1, characterized in that, The inference process of step S3 meets the following conditions, including: In the NVIDIA T4 GPU hardware environment, the inference time of the input image to be counted for extracting four-stage features by the VMamba backbone network is ≤31.2 ms, and the whole network computation amount is 69.8 GFlops; the linear computation complexity of the VMamba backbone network is compared with the quadratic complexity of the Transformer architecture, and the inference speed is improved by ≥35% on a 1024×768 resolution image; the density map output by the inference is obtained by summation operation to obtain the crowd number, and the pixel value of the density map is one-to-one corresponding to the spatial distribution of the real crowd head.
8. The crowd counting method based on multi-feature fusion VMamba of claim 1, wherein, The four-way scanning module of the VMamba backbone network in step S1 realizes global pixel dependence modeling through state space transformation without relying on the multi-head attention mechanism of the Transformer; the average absolute error of crowd counting on a small sample dataset is ≤183.1, which is better than the TransCrowd architecture; the MAE on the high-density dataset is 56.8, which is better than the DMCNet architecture.