A target counting method, system and electronic device based on multi-level segmentation attention

Abstract

The present application relates to a kind of target counting method, system based on multi-level segmentation attention, electronic equipment, belong to image processing technical field.The present application includes the following steps: the multi-level feature of input image is extracted by VGG16 main network;Corresponding multi-level segmentation attention map is generated by auxiliary multi-level segmentation network AMSN to multi-level feature, and training is carried out using segmentation loss;Multi-level local feature enhancement module MLFE in main network is used to fuse segmentation attention map and its corresponding multi-level feature, promote the interaction between main network and AMSN, to generate local enhancement feature map;Multi-level feature is captured by three self-attention modules SA to different levels of non-local information, and global enhancement feature map is generated;Local enhancement feature map and global enhancement feature map are fused in the recursive manner by hierarchical feature fusion module HFF, generate fusion feature, then the fusion feature is input into regression head, to obtain the final density map;The present application can significantly improve the performance in object counting task.

Description

Technical Field

[0001] This invention relates to a target counting method, system, and electronic device based on multi-level segmentation attention, belonging to the field of image processing technology. Background Technology

[0002] With the development of deep learning, many counting methods based on deep neural networks have emerged in recent years. These methods are mainly divided into counting methods based on convolutional neural networks (CNNs), counting methods based on attention mechanisms, and counting methods based on auxiliary networks. CNN-based counting methods (such as MCNN, CSRNet, CAN, BL, DC-regression, DM-Count, and STEERER) use CNNs to extract features, generate density maps from input images, and then integrate the density maps to predict the number of objects. This paradigm formulates object counting as a regression task and has been widely used in the field of object counting. However, due to changes in viewpoint and interference from background noise, these methods still face challenges in effectively identifying objects. Inspired by the successful application of attention mechanisms, more and more research is beginning to incorporate them into counting methods to improve performance. For example, SCAR uses spatial and channel attention to capture local and global dependencies in feature maps. KDMG uses a self-attention mechanism to generate adaptive density maps; another study, MAN, uses multiple attention mechanisms for crowd counting in changing environments. PET uses progressive encoder attention to encode crowd information at multiple scales. Gramformer is a graph modulation Transformer that enhances the counting performance of a network by adjusting attention and input node features based on two different types of graphs. However, these attention-based counting methods often rely on ground truth values ​​from labeled or density maps, the quality of which can be affected by background noise or inaccurate labeling, making it difficult to extract high-quality local and global features, thus impacting counting accuracy. Recently, some methods have begun to introduce auxiliary networks to perform different tasks to help improve counting accuracy (e.g., IRAST, P2PNet, MPS, FocusNet). In these auxiliary network-based counting methods, the main network and the auxiliary network share initial features extracted from the input image. Since the feature extraction backbone is also shared with the auxiliary network, these auxiliary tasks help enhance the feature representation of the main network, thereby improving its counting performance. However, many methods lack effective interaction between the main and auxiliary networks, which could further improve counting accuracy.

[0003] CNN-based methods excel at extracting multi-scale local features: utilizing convolutional kernels of varying sizes, these methods better handle viewpoint changes and annotation errors. However, due to the limited receptive field of convolutions, these methods struggle to capture global information in feature maps, making them sensitive to background noise, occlusion, and viewpoint changes. Attention-based methods outperform CNN-based methods in capturing global dependencies, further improving counting performance. However, these methods typically capture long-distance dependencies in an unsupervised manner and are also susceptible to viewpoint changes and background noise, making them difficult to accurately identify individual objects. While auxiliary network-based counting methods leverage the auxiliary network to enhance the performance of the main network, few studies have focused on promoting interaction between the main and auxiliary networks. Furthermore, significant task differences between the main and auxiliary networks can sometimes lead to performance degradation. In fact, deep and effective interaction can enable the auxiliary network to better guide the learning of the main task and contribute to further improving its counting performance.

[0004] Existing counting methods are challenged by complex backgrounds and changing viewpoints. While attention-based methods can focus on regions of interest, these studies often neglect detailed local contextual information around the target object and fail to effectively integrate local and global features to improve the robustness of the counting. Summary of the Invention

[0005] This invention proposes a target counting method, system, and electronic device based on multi-level segmentation attention. This invention can significantly improve the accuracy of target counting and effectively integrate local and global features, thereby improving the robustness of counting.

[0006] The technical solution of this invention is: a target counting method based on multi-level segmentation attention, the method comprising the following:

[0007] Step 1: Extract multi-level features from the input image using the VGG16 backbone network;

[0008] Step 2: Generate multi-level segmentation attention maps corresponding to multi-level features through the Auxiliary Multi-Level Segmentation Network (AMSN), and train the network using segmentation loss.

[0009] Step 3: Integrate the multi-level segmentation attention map generated from the auxiliary multi-level segmentation network AMSN into the main network;

[0010] The segmentation attention map is fused with its corresponding multi-level features by the multi-level local feature enhancement module MLFE in the main network, which promotes the interaction between the main network and AMSN, thereby generating a local enhanced feature map.

[0011] Multi-level features capture different levels of non-local information through three self-attention modules (SA) and generate a global enhanced feature map.

[0012] The Hierarchical Feature Fusion (HFF) module recursively fuses local and global enhanced feature maps to generate fused features, which are then input into the regression head to obtain the final density map.

[0013] Further, Step 1 includes:

[0014] Given an input image I∈R with height H and width W. H×W×3 Its basic characteristics Intermediate features Advanced features From the VGG16 backbone network F vgg Extracted from modules 3, 4, and 5:

[0015] F l ,F m ,F h =F vgg (I).

[0016] Furthermore, Step 2 includes:

[0017] Step 2.1: Extracting multi-level features F from different layers of the VGG16 backbone network l F m and F h The corresponding segmentation attention score is generated by passing through two 3×3 convolutional layers and a ReLU activation function. and

[0018]

[0019] In the formula, Conv(·) represents a 3×3 convolutional layer, followed by a low-level segmentation attention map. Intermediate segmentation attention map Advanced Segmentation Attention Map To be generated:

[0020]

[0021] In the formula, Softmax(·) refers to the Softmax function, and the operator [·]1 only retains the foreground channel of the segmentation attention map; next, the operation is repeated on each segmentation attention map along the channel dimension to adjust its size to match the size of the multi-level features:

[0022]

[0023] In the formula, rep(·) represents the repetition operator, and This is the adjusted segmentation attention map;

[0024] During training, a per-pixel weighted focus loss L is used. seg To supervise the learning of the auxiliary network AMSN; specifically, each segmentation head in AMSN will predict a segmentation mask. Its error compared to the true mask M is measured by the following formula:

[0025]

[0026] In the formula, And |·| return Or M enc The number of pixels involved, in addition, M enc It is the one-hot encoding of the real mask M, where ch∈{0,1} represents the channel index, and ∝=2 is the focus parameter;

[0027] In practice, the training loss of each segmentation head is weighted differently; therefore, the overall training loss of AMSN is defined as:

[0028]

[0029] In the formula, and These are the segmentation masks predicted by different segmentation heads, and the actual masks are respectively... and Meanwhile, the weights λ1, λ2, and λ3 are set to 0.1, 0.2, and 0.3, respectively.

[0030] Furthermore, in Step 3, the multi-level local feature enhancement module (MLFE) includes three parallel local feature enhancement blocks (LFE).

[0031] Each Local Feature Enhancement (LFE) block processes segmentation attention maps from the same level and the initially extracted multi-level features;

[0032] The input to the first Local Feature Enhancement Block (LFE) includes high-level features F. h and its corresponding segmentation attention map First, the high-level feature F... h and its corresponding segmentation attention map Pixel-wise multiplication is performed to highlight regions of higher importance as objects of interest; then a 1×1 convolution is applied to generate a smooth feature map. Next, As a query, the initial high-level feature F hSimultaneously, both are used as keys and values, input into the Cross Attention block, thus enabling F... h Local dependencies are constructed to enhance feature representation; residual connections are also used to mitigate gradient decay, and the results output by the Cross Attention block are processed. Add back Then, local enhanced feature maps of high-level features are obtained;

[0033] Based on the processing method of the first Local Feature Enhancement Block (LFE), the second and third Local Feature Enhancement Blocks (LFE) respectively obtain the local enhanced feature maps of low-level features and the local enhanced feature maps of mid-level features.

[0034] Furthermore, in Step 3, the specific process of obtaining the local enhanced feature map of high-level features is as follows:

[0035]

[0036] In the formula, the symbol ⊙ represents element-wise multiplication. This is a scaling factor; to mitigate gradient decay, residual connections were also used. Add back Then, local enhanced feature maps of high-level features are obtained.

[0037]

[0038] The second and third local feature enhancement blocks (LFEs) respectively produce locally enhanced feature maps of low-level features. Local enhancement feature maps of intermediate features

[0039] Furthermore, in Step 3, the Hierarchical Feature Fusion (HFF) module includes N consecutive fusion layers, and the feature pairs... and First, the data is fed into the initial fusion layer, which contains multiple fusion units, channel attention units, and spatial attention units. These represent the global augmented feature maps for high-level, mid-level, and low-level features, respectively. These represent local enhanced feature maps for high-level, mid-level, and low-level features, respectively. The multi-layer fusion unit uses the Cross Attention module for feature fusion, while the spatial and channel attention units aim to preserve the local and non-local dependencies in the fused features. Each fusion layer after the initial fusion layer contains a multi-head attention unit, a channel attention unit, and a spatial attention unit. The multi-head attention unit uses three attention heads to capture the latent representations of the fused features from different angles.

[0040] Furthermore, in Step 3, the specific processing procedure of the Hierarchical Feature Fusion (HFF) module is as follows:

[0041] set up Let the output of the (l-1)th layer fusion be represented as (where l = 1, 2, ..., N-1), which is also the input of the lth layer fusion. Then, the attention head is obtained through the following formula:

[0042]

[0043] In the formula, i∈{0,1,2} represents the index of the attention head, while This is the output of the Cross Attention module, where d k This is the scaling factor. Next, all attention heads are aggregated to generate the output of the multi-head attention unit:

[0044]

[0045] In the formula, [·,·,·] are concatenation operators, W and b are model parameters. To preserve local and nonlocal dependencies during the fusion process, spatial attention units and channel attention units are used, from f l c Local enhancement feature map F for extracting low-level features l loc and global augmented feature map F l glob :

[0046] F l glob =f l c ⊙σ(Conv(ReLU(Conv(MP(f l c )))))

[0047] F l loc =f l c ⊙σ(BN(Conv(CP(MP)))),

[0048] In the formula, MP(·), CP(·), and BN(·) represent average pooling, channel pooling, and batch normalization, respectively; meanwhile, σ represents the Sigmoid function, where Fl l glob and F l loc They are all outputs of the fusion at layer l;

[0049] Finally, the output of the hierarchical feature fusion module HFF It was obtained through the following methods:

[0050]

[0051] In the formula, N-1 is the index of the last fusion layer, and residual(·) represents the residual module.

[0052] Furthermore, the output F of the hierarchical feature fusion module HFF fuse The data is passed to the regression head, which predicts the final density map. The regression head consists of consecutive 3×3 convolutional layers and a ReLU activation function:

[0053]

[0054] Then, to Pixel-wise integration is performed to obtain the counting results. The main network is trained using the real density map D∈R. H ×W×1 For supervision, the objective function is defined as:

[0055]

[0056] In the formula, and Furthermore, D is uniformly divided into N along both the height and width directions. row ×N col For each block, from the p-th row and q-th column, measure the predicted count. Compared with the true value The error between them is E(p,q), and a binary indicator B(·)∈{0,1} is used to determine whether |E(p,q)| is greater than a given threshold t;

[0057] Combining the segmentation loss function L in the Auxiliary Multi-Level Segmentation Network (AMSN) seg_all The regression loss function L of the main network reg The total training loss function L is defined as:

[0058] L = L seg_all +L reg .

[0059] During the inference phase, the auxiliary multi-level segmentation network AMSN is removed, and the remaining part is retained for prediction.

[0060] The present invention provides a target counting system based on multi-level segmentation attention, the system comprising: a module for executing the target counting method based on multi-level segmentation attention.

[0061] The present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target counting method based on multi-level segmentation attention.

[0062] The relevant technical terms in this invention are explained as follows:

[0063] CNN: A deep learning model that automatically extracts features through convolutional layers and is widely used in image processing and computer vision tasks.

[0064] Self-attention is a mechanism that enhances the representation of information by dynamically adjusting the degree of attention each element pays to other elements by computing the relationships between elements in the input sequence.

[0065] Cross Attention: Cross attention is a mechanism that enhances information fusion in multimodal tasks by computing the interrelationships between two different sequences, enabling elements of one sequence to pay attention to relevant information in the other sequence.

[0066] Transformer: A deep learning model based on self-attention mechanism, widely used in natural language processing and sequence data tasks, which can effectively capture long-range dependencies.

[0067] The beneficial effects of this invention are:

[0068] 1. The method proposed in this invention, abbreviated as MultiSegCount, corresponds to a novel multi-level segmentation attention map guided network for object counting. It consists of three key elements: 1) an Auxiliary Multi-level Segmentation Network (AMSN) that generates segment-attention maps containing multi-level local semantic information surrounding the target object; 2) a Multi-level Local Feature Enhancement (MLFE) module, designed to enhance the representation of extracted multi-level local features using the multi-level segment-attention maps; and 3) a Hierarchical Feature Fusion (HFF) module, which effectively fuses local and global features to further improve counting performance. Extensive evaluation on four datasets (including crowd counting, vehicle counting, wheat ear counting, and rebar counting) demonstrates that MultiSegCount outperforms several state-of-the-art baseline models in object counting tasks.

[0069] 2. MultiSegCount is suitable for various object counting applications, including public safety monitoring, traffic flow analysis, agricultural production management, ecological monitoring, and business analysis. Traditional object counting methods often face challenges due to complex backgrounds and changing perspectives, while MultiSegCount significantly enhances the accuracy and robustness of counting through multi-level segmentation attention. This method demonstrates superior accuracy compared to existing baseline models in different tasks, particularly in handling tasks such as crowd counting and vehicle counting, effectively fusing local and global features to improve performance. In summary, MultiSegCount is particularly suitable for application in various complex counting scenarios, enabling more accurate calculation of the number of objects and meeting diverse practical needs. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of the network structure corresponding to the MultiSegCount method of the present invention;

[0071] Figure 2 This is a structural diagram of the MLFE module in this invention;

[0072] Figure 3 This is a structural diagram of the HFF module in this invention. Detailed Implementation

[0073] Example 1: As Figures 1-3 As shown, a target counting method based on multi-level segmentation attention is proposed, the method comprising the following:

[0074] Step 1: Extract multi-level features from the input image using the VGG16 backbone network;

[0075] Step 2: Generate multi-level segmentation attention maps corresponding to multi-level features through the Auxiliary Multi-Level Segmentation Network (AMSN), and train the network using segmentation loss.

[0076] Step 3: Integrate the multi-level segmentation attention map generated from the auxiliary multi-level segmentation network AMSN into the main network;

[0077] The segmentation attention map is fused with its corresponding multi-level features by the multi-level local feature enhancement module MLFE in the main network, which promotes the interaction between the main network and AMSN, thereby generating a local enhanced feature map.

[0078] Multi-level features capture different levels of non-local information through three self-attention modules (SA) and generate a global enhanced feature map.

[0079] The Hierarchical Feature Fusion (HFF) module recursively fuses local and global enhanced feature maps to generate fused features, which are then input into the regression head to obtain the final density map.

[0080] The AMSN proposed in this invention captures local features from multiple perspectives by generating segment-attention maps using multi-level feature maps, explicitly guiding the main network to focus on the local context of the target object. This network is trained under the supervision of a real segmentation mask. In essence, a segmentation attention map is a weight map, where larger weight values ​​correspond to target object regions and smaller weight values ​​correspond to background regions. These segmentation attention maps are generated from shallow or deep features of the VGG16 backbone network. Notably, deep features excel at extracting high-level semantic information (such as object category and overall structure), while shallow features focus on capturing low-level physical details (such as color, texture, and edges). Therefore, this invention, by fusing segmentation attention maps generated from both shallow and deep features, integrates high-level semantics and low-level details, enabling the main network to learn more robust feature representations, thereby significantly improving the accuracy of crowd counting.

[0081] exist Figure 1In this invention, the main network includes: a Multi-level Local Feature Enhancement (MLFE) module, a Self-attention (SA) module, a Hierarchical Feature Fusion (HFF) module, and a regression head. To achieve more accurate counting, this invention incorporates multi-level segmentation attention maps generated from AMSN into the main network. Specifically, the MLFE module aims to fuse the segmentation attention maps into the multi-level features extracted by the VGG16 backbone network, thereby enhancing the representation of the local context surrounding the target object. In addition to enhancing local information, this invention also utilizes the Self-Attention (SA) module to capture global information at multiple levels. Then, the HFF module deeply combines these multi-level local and global information in a hierarchical manner and integrates the fused features into the regression head. fuse The data is passed to the regression head for final count prediction.

[0082] Further, Step 1 includes:

[0083] like Figure 1 As shown, given an input image I∈R with height H and width W... H×W×3 Its basic characteristics Intermediate features Advanced features From the VGG16 backbone network F vgg Extracted from modules 3, 4, and 5:

[0084] F l ,F m ,F h =F vgg (I).

[0085] Furthermore, Step 2 includes:

[0086] Step 2.1: Extracting multi-level features F from different layers of the VGG16 backbone network l F m and F h The corresponding segmentation attention score is generated by passing through two 3×3 convolutional layers and a ReLU activation function. and

[0087]

[0088] In the formula, Conv(·) represents a 3×3 convolutional layer, followed by a low-level segmentation attention map. Intermediate segmentation attention map Advanced Segmentation Attention Map To be generated:

[0089] A l =[Softmax(F l seg )]1,

[0090]

[0091] In the formula, Softmax(·) refers to the Softmax function, and the operator [·]1 only retains the foreground channel of the segmentation attention map; next, the operation is repeated on each segmentation attention map along the channel dimension to adjust its size to match the size of the multi-level features:

[0092]

[0093] In the formula, rep(·) represents the repetition operator, and This is the adjusted segmentation attention map;

[0094] During training, a per-pixel weighted focus loss L is used. seg To supervise the learning of the auxiliary network AMSN; specifically, each segmentation head in AMSN will predict a segmentation mask. Its error compared to the true mask M is measured by the following formula:

[0095]

[0096] In the formula, And |·| return Or M enc The number of pixels involved, in addition, M enc It is the one-hot encoding of the real mask M, where ch∈{0,1} represents the channel index, and ∝=2 is the focus parameter;

[0097] In practice, the training loss of each segmentation head is weighted differently; therefore, the overall training loss of AMSN is defined as:

[0098]

[0099] In the formula, and These are the segmentation masks predicted by different segmentation heads, and the actual masks are respectively... and Meanwhile, the weights λ1, λ2, and λ3 are set to 0.1, 0.2, and 0.3, respectively.

[0100] Furthermore, in Step 3, the MLFE (Multi-level Local Feature Enhancement) module aims to identify the initially extracted multi-level features F. l F m and F h Local information in the image is used to segment attention maps. and Enhance the representation of the local context surrounding the target object. The multi-level local feature enhancement module (MLFE) includes three parallel local feature enhancement blocks (LFE).

[0101] Each Local Feature Enhancement (LFE) block processes segmentation attention maps from the same level and the initially extracted multi-level features;

[0102] The input to the first Local Feature Enhancement Block (LFE) includes high-level features F. h and its corresponding segmentation attention map First, the high-level feature F... h and its corresponding segmentation attention map Pixel-wise multiplication is performed to highlight regions of higher importance as objects of interest; then a 1×1 convolution is applied to generate a smooth feature map. Next, As a query, the initial high-level feature F h Simultaneously, both are used as keys and values, input into the Cross Attention block, thus enabling F... h By constructing local dependencies and enhancing feature representations, the model can better focus on important regions in high-level feature maps. To mitigate gradient decay, residual connections are also used to process the output of the Cross Attention block. Add back Then, local enhanced feature maps of high-level features are obtained;

[0103] Based on the processing method of the first Local Feature Enhancement Block (LFE), the second and third Local Feature Enhancement Blocks (LFE) respectively obtain the local enhanced feature maps of low-level features and the local enhanced feature maps of mid-level features.

[0104] Furthermore, in Step 3, the specific process of obtaining the local enhanced feature map of high-level features is as follows:

[0105]

[0106] In the formula, the symbol ⊙ represents element-wise multiplication. This is a scaling factor; to mitigate gradient decay, residual connections were also used. Add back Then, local enhanced feature maps of high-level features are obtained.

[0107]

[0108] The second and third local feature enhancement blocks (LFEs) respectively produce locally enhanced feature maps of low-level features. Local enhancement feature maps of intermediate features

[0109] In addition to enhancing local features, this invention also identifies global dependencies among these initially extracted multi-level features. To this end, three self-attention (SA) modules are applied in parallel to F. l F m and F h Therefore, global enhanced features were obtained.

[0110] F l glob =SA(F l ),

[0111]

[0112] In the formula, SA(·) represents the self-attention (SA) module.

[0113] The purpose of the Hierarchical Feature Fusion (HFF) module is to thoroughly fuse these enhanced multi-level features in a hierarchical manner. as well as Ultimately, this module generates a more expressive fusion feature F. fuse It effectively captures local and non-local dependencies, thereby improving counting performance.

[0114] Furthermore, in Step 3, the Hierarchical Feature Fusion (HFF) module includes N consecutive fusion layers, and the feature pairs... and First, the signals are fed into the initial fusion layer, which contains multiple fusion units, channel attention (CA) units, and spatial attention (SA) units. These represent the global augmented feature maps for high-level, mid-level, and low-level features, respectively. These represent local enhanced feature maps for high-level, mid-level, and low-level features, respectively. The multi-layer fusion unit utilizes the Cross Attention module for feature fusion, while the spatial and channel attention units aim to preserve local and non-local dependencies in the fused features. Each fusion layer after the initial fusion layer contains a multi-head attention unit, a channel attention unit, and a spatial attention unit. The multi-head attention unit uses three attention heads to capture the latent representations of the fused features from different angles. Unlike existing multi-head attention, this invention further introduces residual connections to alleviate the gradient vanishing problem.

[0115] Furthermore, in Step 3, the specific processing procedure of the Hierarchical Feature Fusion (HFF) module is as follows:

[0116] set up Let the output of the (l-1)th layer fusion be represented as (where l = 1, 2, ..., N-1), which is also the input of the lth layer fusion. Then, the attention head is obtained through the following formula:

[0117]

[0118] In the formula, i∈{0,1,2} represents the index of the attention head, while This is the output of the Cross Attention module, where d k This is the scaling factor. Next, all attention heads are aggregated to generate the output of the multi-head attention unit:

[0119]

[0120] In the formula, [·,·,·] are concatenation operators, W and b are model parameters, and spatial attention units and channel attention units are used to preserve local and nonlocal dependencies during the fusion process. Local enhancement feature map for extracting low-level features and global augmented feature map

[0121] F l glob =f l c ⊙σ(Conv(ReLU(Conv(MP(f l c )))))

[0122] F l loc =f l c ⊙σ(BN(Conv(CP(MP))(),

[0123] In the formula, MP(·), CP(·), and BN(·) represent average pooling, channel pooling, and batch normalization, respectively; meanwhile, σ represents the Sigmoid function, in which case F l glob and F l loc They are all outputs of the fusion at layer l;

[0124] Finally, the output of the hierarchical feature fusion module HFF It was obtained through the following methods:

[0125]

[0126] In the formula, N-1 is the index of the last fusion layer, and residual(·) represents the residual module.

[0127] Furthermore, the output F of the hierarchical feature fusion module HFF fuse The data is passed to the regression head, which predicts the final density map. The regression head consists of consecutive 3×3 convolutional layers and a ReLU activation function:

[0128]

[0129] Then, to Pixel-wise integration is performed to obtain the counting results. The main network is trained using the real density map D∈R. H ×W×1 For supervision, the objective function is defined as:

[0130]

[0131] In the formula, and Furthermore, D is uniformly divided into N along both the height and width directions. row ×N col For each block, from the p-th row and q-th column, measure the predicted count. Compared with the true value The error between them is E(p,q), and a binary indicator B(·)∈{0,1} is used to determine whether |E(p,q)| is greater than a given threshold t;

[0132] Combining the segmentation loss function L in the Auxiliary Multi-Level Segmentation Network (AMSN) seg_all The regression loss function L of the main network reg The total training loss function L is defined as:

[0133] L = L seg_all +L reg .

[0134] During the inference phase, the auxiliary multi-level segmentation network AMSN is removed, and the remaining part is retained for prediction.

[0135] The AMSN of this invention is an auxiliary network that performs segmentation tasks, designed to help the main network generate more accurate counting results. To better focus on regions of interest that contribute to object counting, the AMSN generates corresponding multi-level features F. l F m F h Segmentation attention (Figure A) l A m A h The network is trained using a segmentation loss. Then, a designed Multi-Level Local Feature Enhancement (MLFE) module generates locally enhanced feature maps F by fusing the segmentation attention map with its corresponding multi-level features to facilitate interaction between the main network and the AMSN. l loc , and Therefore, these feature maps can provide more local information cues about the target object.

[0136] Meanwhile, multi-level features F l F m F h It also uses three self-attention (SA) modules to capture different levels of non-local information and generate a globally enhanced feature map F. l glob , and Our counting prediction relies on combining local and global augmented features. To this end, we introduce an efficient hierarchical feature fusion (HFF) module that performs feature fusion recursively. It generates fused features F. fuse Then, it is input into the regression head to obtain the final density map.

[0137] The present invention also provides a target counting system based on multi-level segmentation attention, the system comprising:

[0138] The VGG16 backbone network is used to extract multi-level features from the input image.

[0139] The Auxiliary Multi-Level Segmentation Network (AMSN) is used to generate multi-level segmentation attention maps corresponding to multi-level features and is trained using segmentation loss.

[0140] The main network is used to implement the following functions:

[0141] The segmentation attention map is fused with its corresponding multi-level features by the multi-level local feature enhancement module MLFE in the main network, which promotes the interaction between the main network and AMSN, thereby generating a local enhanced feature map.

[0142] Multi-level features capture different levels of non-local information through three self-attention modules (SA) and generate a global enhanced feature map.

[0143] The Hierarchical Feature Fusion (HFF) module recursively fuses local and global enhanced feature maps to generate fused features, which are then input into the regression head to obtain the final density map.

[0144] The present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target counting method based on multi-level segmentation attention.

[0145] To verify the effectiveness of this invention, four datasets with different counting objects were used for evaluation in experiments. These datasets included CARPK, GWHD 2021, JHU-CROWD++, and RebarDSC. CARPK is a vehicle counting dataset containing 1448 images and 89774 labeled vehicles. 989 images were used as training samples, and the remaining 459 images were used as test samples. GWHD 2021 collected 6515 images of wheat ears from 16 different institutions in 12 countries. This dataset provides 275468 labeled wheat ears and is divided into 3657 training images, 1476 validation images, and 1382 test images. JHU-CROWD++ is a large-scale crowd counting dataset containing 4372 high-resolution images with a total of 1.51 million annotations. This dataset includes 2272 training images, 500 validation images, and 1600 test images. RebarDSC contains 2215 images, including 1125 training samples and 1000 test samples, with a total of 350348 annotations for rebar counting.

[0146] This invention uses Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) to evaluate counting performance. Their definitions are as follows:

[0147]

[0148] In the formula, n represents the number of test samples, and y i and Let represent the true count value and the predicted count value of the i-th sample, respectively. Here, the lower the values ​​of both metrics, the higher the counting performance.

[0149] Implementation details:

[0150] The model of this invention was implemented using PyTorch on a workstation equipped with an NVIDIA A100 GPU. The network backbone is based on VGG16, which was pre-trained on the ImageNet dataset. The Adam optimizer was used to optimize the model of this invention, with weight decay set to 5 × 10⁻⁶. -4 The learning rate is set to 1×10. -5 The model of this invention was trained for 300 epochs with a batch size of 8. Furthermore, data augmentation was performed on the input samples, including random cropping and vertical flipping. Specifically, image patches of size 512×512 were randomly cropped from the original image and then randomly vertically flipped with a probability of 0.5.

[0151] During training, in order to learn the segmentation attention map, this invention takes the following steps to generate a segmentation mask. First, the ground truth value D∈R of the density map... H×W×1 It is generated by applying Gaussian filtering to the bounding box annotation map. The kernel size of the Gaussian filter is dynamically determined based on the bounding box to ensure that the density distribution of each object is diffused within the bounding box region. Then, the segmentation mask M∈R... H×W×1 Generated:

[0152]

[0153] In the formula, (i,j) represents the coordinates of the mask. Subsequently, the present invention downsamples the mask M to generate a multi-scale mask, including: and Comparison with advanced methods

[0154] This invention compares the proposed MultiSegCount with 16 state-of-the-art baseline methods, including MCNN, CSRNet, SCAR, CAN, BL, IRAST, DM-Count, P2PNet, KDMG, MPS, MAN, DCregression, PET, STEERER, FocusNet, and Gramformer. Among these baseline methods, MCNN, CSRNet, CAN, BL, DM-Count, DC-regression, and STEERER are traditional CNN-based counting methods, while SCAR, KDMG, MAN, PET, and Gramformer are attention-based counting methods. Meanwhile, IRAST, P2PNet, MPS, and FocusNet are counting methods involving auxiliary networks. Table 1 shows the comparison results.

[0155] As shown in Table 1, the proposed MultiSegCount method achieves state-of-the-art performance on all datasets. On the most challenging dataset, JHU-CROWD++, MultiSegCount outperforms the second-best baseline models Gramformer and MAN by 0.19 and 5.62, respectively, especially considering that this dataset contains samples with large viewpoint variations. For other datasets such as CARPK, GWHD 2021, and RebarDSC, the proposed method also achieves state-of-the-art results, with MAE / RMSE exceeding the second-best methods by 1.87 / 1.82, 3.54 / 4.37, and 1.76 / 2.91, respectively. Given that these datasets involve the performance of different objects under various environments, the superiority of the MultiSegCount method demonstrates that the proposed method can better handle a wide range of counting scenarios.

[0156] These results reveal that some methods (e.g., MCNN, CSRNet, SCAR, and CAN) perform unsatisfactorily. Furthermore, some methods show significant differences in performance across different datasets. For example, MAN performs poorly on the CARPK and GWHD 2021 datasets, but performs better on the JHU-CROWD++ and RebarDSC datasets. Among these baselines, DC-regression and DM-Count perform excellently on all four datasets. Notably, the MultiSegCount method of this invention not only demonstrates stable performance but also achieves more accurate counting results on these datasets. This is attributed to its enhanced ability to extract and fuse local and global information.

[0157] Table 1 shows the comparison results of the model of this invention with other baseline models on four datasets.

[0158]

[0159]

[0160] Ablation experiment:

[0161] This invention conducts an ablation study on the CARPK dataset to evaluate the effectiveness of the module proposed in MultiSegCount.

[0162] The MultiSegCount module of this invention comprises four key components: an auxiliary network AMSN for generating multi-level segmentation attention maps, an MLFE module for enhancing local features, an SA module for extracting global features, and an HFF module for fusing local and global features. These modules work together to improve the accuracy and robustness of target counting. Through ablation studies, this invention will evaluate the impact of each module on the overall performance.

[0163] Table 2 shows the performance of different key modules of the model of this invention on the CARPK dataset.

[0164]

[0165] As shown in Table 2, this invention starts with a baseline network from which all critical parts of the proposed model have been removed. Subsequently, critical parts are gradually added to the baseline network to verify their effectiveness. After integration with the auxiliary network AMSN, some improvements are observed, with MAE and RMSE values ​​decreasing by 0.19 and 0.35, respectively. This invention argues that the auxiliary network AMSN improves performance by influencing the learning of encoded features shared by the main and auxiliary networks. Next, this invention uses the MLFE module to facilitate the interaction between the main and auxiliary networks, further improving counting performance. This module fuses the segmentation attention map generated by the auxiliary network to guide the learning of the main network. Therefore, MAE and RMSE values ​​are reduced to 4.58 and 6.76, respectively. Next, the SA module is added to extract global features, further reducing MAE and RMSE values. Finally, the added HFF module significantly enhances counting performance, effectively reducing MAE and RMSE values ​​to 3.30 and 5.12 due to its multi-level hierarchical fusion of features.

[0166] Table 3 shows the effectiveness of multi-level segmentation attention maps.

[0167]

[0168] The generated multi-level segmentation attention maps enable the MultiSegCount of this invention to accurately identify local regions around a target object. Here, the effectiveness of this invention is verified by removing segments of the attention map layer by layer. Specifically, this invention evaluates three variants of MultiSegCount: MultiSegCount without low-level segmentation attention maps (denoted as w / o LSM), MultiSegCount without mid-level segmentation attention maps (denoted as w / o MSM), and MultiSegCount without high-level segmentation attention maps (denoted as w / o HSM). It is noteworthy that when a segmentation attention map at a corresponding level is removed, the invention also removes auxiliary segmentation branches. As shown in Table 3, removing any level of segmentation attention map results in an immediate performance degradation. This indicates that each level of segmentation attention map contributes independently to the achieved counting performance.

[0169] This invention proposes a target counting method, MultiSegCount, based on multi-level segmentation attention. Key aspects of this approach include the MultiSegCount network architecture, AMSN module, MLFF module, and HFF module. MultiSegCount enhances the capture and utilization of multi-level local information by integrating the main and auxiliary networks. The AMSN module aims to generate a segment-attention map containing multi-level local semantic information surrounding the target object. The MLFF module enhances the representation ability of local features and promotes the interaction between the main and auxiliary networks, while the HFF module comprehensively integrates local and global information through a hierarchical fusion strategy. These core technologies significantly improve the model's performance in general object counting tasks and represent the key technical points specifically protected.

[0170] This invention proposes a target counting method, MultiSegCount, based on multi-level segmentation attention. This algorithm can be applied to general object counting tasks. Including the auxiliary multi-level segmentation network proposed in this scheme, other types of auxiliary networks (such as classification networks) can be used to generate feature maps with local semantic information. Furthermore, in the fusion of local and global information, different fusion methods (such as BiFusion) can be used to fuse features.

[0171] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A target counting method based on multi-level segmentation attention, characterized in that: The method includes the following: Step 1: Extract multi-level features from the input image using the VGG16 backbone network; Step 2: Generate multi-level segmentation attention maps corresponding to multi-level features through the Auxiliary Multi-Level Segmentation Network (AMSN), and train the network using segmentation loss. Step 3: Integrate the multi-level segmentation attention map generated from the auxiliary multi-level segmentation network AMSN into the main network; The segmentation attention map is fused with its corresponding multi-level features by the multi-level local feature enhancement module MLFE in the main network, which promotes the interaction between the main network and AMSN, thereby generating a local enhanced feature map. Multi-level features capture different levels of non-local information through three self-attention modules (SA) and generate a global enhanced feature map. The hierarchical feature fusion module (HFF) recursively fuses local and global enhanced feature maps to generate fused features, which are then input into the regression head to obtain the final density map. In Step 3, the Hierarchical Feature Fusion (HFF) module includes... A continuous fusion layer, feature pairs ( , ), ( , )and( , First, it is fed into the initial fusion layer, which contains multiple fusion units, channel attention units, and spatial attention units. , , These represent the global augmented feature maps for high-level, mid-level, and low-level features, respectively. , , These represent local enhanced feature maps for high-level, mid-level, and low-level features, respectively. The multi-layer fusion unit uses the Cross Attention module for feature fusion, while the spatial and channel attention units aim to preserve the local and non-local dependencies in the fused features. Each fusion layer after the initial fusion layer contains a multi-head attention unit, a channel attention unit, and a spatial attention unit. The multi-head attention unit uses three attention heads to capture the latent representations of the fused features from different angles.

2. The target counting method based on multi-level segmentation attention according to claim 1, characterized in that: Step 1 includes: Given a high value Width Input image Its basic characteristics Intermediate features Advanced features From the VGG16 backbone network respectively Extracted from modules 3, 4, and 5: 。 3. The target counting method based on multi-level segmentation attention according to claim 1, characterized in that: Step 2 includes: Step 2.1: Multi-level features extracted from different layers of the VGG16 backbone network , and After two convolutional layers and Activation function to generate corresponding segmentation attention scores , : ； In the formula, express After the convolutional layer, a low-level segmentation attention map is applied. Intermediate segmentation attention map Advanced segmentation attention map To be generated: ； In the formula, It refers to Functions, Operators Only the foreground channel of the segmentation attention map is retained; next, the operation is repeated along the channel dimension for each segmentation attention map to adjust its size to match the size of the multi-level features: ； In the formula, This indicates the repetition operator, and , , This is the adjusted segmentation attention map; During training, a per-pixel weighted focus loss is used. This is used to supervise the learning of the auxiliary network AMSN; specifically, each segmentation head in AMSN will predict a segmentation mask. , and the real mask The error is measured by the following formula: In the formula, ,and return or The number of pixels involved, in addition, It is the real mask. one-hot encoding, Indicates the channel index. It is the focus parameter; In practice, the training loss of each segmentation head is weighted differently; therefore, the overall training loss of AMSN is defined as: ； In the formula, , and These are the segmentation masks predicted by different segmentation heads, and the actual masks are respectively... , and Meanwhile, weight , and Set them to 0.1, 0.2 and 0.3 respectively.

4. The target counting method based on multi-level segmentation attention according to claim 1, characterized in that: In Step 3, the multi-level local feature enhancement module (MLFE) includes three parallel local feature enhancement blocks (LFE). Each Local Feature Enhancement (LFE) block processes segmentation attention maps from the same level and the initially extracted multi-level features; The input to the first Local Feature Enhancement Block (LFE) includes high-level features. and its corresponding segmentation attention map First, for high-level features and its corresponding segmentation attention map Perform pixel-by-pixel multiplication to highlight areas of higher importance as objects of interest; Subsequent application Convolution generates smooth feature maps Next, As a query, and the initial high-level features Simultaneously, it serves as both a key and a value, inputting into the Cross Attention block, thus enabling... Local dependencies are constructed to enhance feature representation; residual connections are also used to mitigate gradient decay, and the results output by the Cross Attention block are processed. Add back Then, local enhanced feature maps of high-level features are obtained; Based on the processing method of the first Local Feature Enhancement Block (LFE), the second and third Local Feature Enhancement Blocks (LFE) respectively obtain the local enhanced feature maps of low-level features and the local enhanced feature maps of mid-level features.

5. The target counting method based on multi-level segmentation attention according to claim 4, characterized in that: In Step 3, the specific process of obtaining the local enhanced feature map of high-level features is as follows: ； In the formula, the symbol ⊙ represents element-wise multiplication. This is a scaling factor; to mitigate gradient decay, residual connections were also used. Add back Then, local enhanced feature maps of high-level features are obtained. : ； The second and third local feature enhancement blocks (LFEs) respectively produce locally enhanced feature maps of low-level features. Local enhancement feature maps of intermediate features .

6. The target counting method based on multi-level segmentation attention according to claim 1, characterized in that: In Step 3, the specific processing procedure of the Hierarchical Feature Fusion (HFF) module is as follows: set up , Represented as the first The output of layer fusion (where, They are also the first The input for layer fusion is then used to obtain the attention head using the following formula: ； In the formula, This represents the index of the attention head, while This is the output of the Cross Attention module, where This is the scaling factor. Next, all attention heads are aggregated to generate the output of the multi-head attention unit: ； In the formula, It is a concatenation operator. and These are model parameters. To preserve local and nonlocal dependencies during the fusion process, spatial attention units and channel attention units are used. Extracting local features and global features : ； In the formula, , and These represent average pooling, channel pooling, and batch normalization, respectively; meanwhile, express Function, in this case, and They are all the first Output of layer fusion; Finally, the output of the hierarchical feature fusion module HFF It was obtained through the following methods: ； In the formula, It is the index of the final fusion layer, and This represents the residual module.

7. The target counting method based on multi-level segmentation attention according to claim 1, characterized in that: The output of the hierarchical feature fusion module HFF The data is passed to the regression head, which predicts the final density map. The regression head is continuous convolutional layers and Activation function components: ； Then, to Pixel-wise integration is performed to obtain the counting results. The main network is trained using the real density map. For supervision, the objective function is defined as: ； In the formula, ,and ,also, Divide evenly along the height and width directions into The first block, for the block from the first row and number Column blocks, measurement prediction count Compared with the true value Error between A binary indicator Used for judgment Is it greater than a given threshold? ; Combining the segmentation loss function in the Auxiliary Multi-Level Segmentation Network (AMSN) The regression loss function of the main network Total training loss function Defined as: ； During the inference phase, the auxiliary multi-level segmentation network AMSN is removed, and the remaining part is retained for prediction.

8. A target counting system based on multi-level segmentation attention, characterized in that, The system includes a module for performing the target counting method based on multi-level segmentation attention as described in any one of claims 1-7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the target counting method based on multi-level segmentation attention as described in any one of claims 1 to 7.

Images