Dense target positioning method based on module optimization

Abstract

The invention discloses a dense target positioning method based on module optimization, and the method comprises the steps: generating a foreground confidence graph through introducing a DINOv2 encoder and a context perception classifier, and generating a point prompt based on the foreground confidence graph to guide an SAM model to pay attention to a human head region. According to the method, the adapter module is embedded in the segmentation trunk of the SAM model, so that the structure perception and detail modeling capability is enhanced. And by adopting a post-processing module based on region segmentation of a watershed algorithm and a minimum enclosing rectangle supplement strategy, the problems of target shielding and overlapping are effectively solved. Through experimental verification of various data sets, the positioning method provided by the invention is superior to the existing mainstream method in indexes such as F1 score, precision and recall rate, and shows strong cross-scene adaptive capacity. The method not only improves the positioning precision under dense crowds and complex backgrounds, but also has high cross-scene adaptability, and is widely applied to the fields of public safety, urban traffic management, large-scale activity analysis and the like.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a dense target localization method based on modular optimization. It is particularly suitable for high-precision pixel-level localization in scenarios with severe occlusion and target overlap, such as dense crowds, cell nuclei, and flocks of sheep. It can be widely applied in practical scenarios such as public safety monitoring, urban traffic management, biomedical image analysis, and agricultural and livestock monitoring. Background Technology

[0002] In fields such as public safety early warning (e.g., identifying the risk of stampedes at large events), traffic flow control (e.g., pedestrian density statistics at intersections), and biomedical research (e.g., counting cell nuclei in pathological slides), accurate individual localization of dense targets is a core technological requirement. This not only necessitates counting the number of targets but also obtaining the specific spatial coordinates of each target to support subsequent tasks such as behavioral analysis and risk prediction. However, existing dense target localization technologies are limited by scene complexity (occlusion, background interference) and target characteristics (scale variations, dense overlap), generally suffering from low localization accuracy, poor robustness, and insufficient generalization ability. Specific shortcomings are as follows: (a) Density map-based localization method These methods (such as CSRNet and FIDTM) convert point labels into density maps using Gaussian kernel diffusion, and then learn the density distribution through a regression model to achieve counting and coarse localization. Their core drawback is that they can only reflect the overall density of the region and cannot distinguish individual boundaries, leading to the fundamental problem of "density integral replacing individual detection." In high-density scenarios, spatial localization is prone to false positives.

[0003] (II) Localization methods based on target detection Such methods (such as YOLO and Faster R-CNN) locate targets by predicting bounding boxes, but they rely heavily on accurate bounding box annotations. In dense occlusion or small target scenarios, they are prone to missed detections and false detections, and the computational complexity increases exponentially with the target density, which limits their practicality.

[0004] (III) Localization method based on traditional mask segmentation Such methods (such as IIM and AdaSEEM) generate target masks through semantic segmentation to distinguish individuals, but they have two major bottlenecks: first, the mask generation process is easily affected by noise in complex backgrounds; second, they are not good at solving the problem of "mask adhesion" in extreme density scenarios, and cannot effectively separate overlapping targets, resulting in poor generalization ability.

[0005] (iv) Localization method based on pre-trained segmentation model In recent years, the Segment Anything Model (SAM), as a general segmentation model, has been attempted for dense target localization due to its zero-shot segmentation capability, but it has significant limitations: single-point cues are prone to generating irregular masks, which leads to confusion between the target and the background; while multi-point cues improve the connectivity of the mask, they can blur individual boundaries and require manual intervention in the selection of cues, making them unsuitable for automated localization needs.

[0006] In summary, existing technologies cannot simultaneously solve the three core problems of "foreground and background confusion, insufficient feature interaction, and mask adhesion leading to missed detections," and there is an urgent need for a modular, efficient, and robust dense target localization scheme. Summary of the Invention

[0007] To address the problems of low positioning accuracy, strong background interference, blurred mask boundaries, and high false positive and false negative rates in existing technologies for dense target localization, this invention provides a dense target localization method based on modular optimization. Through the collaborative operation of three-level modules—foreground guidance, feature enhancement, and post-processing optimization—high-precision and automated target localization in dense scenes can be achieved.

[0008] This invention uses SAM as the segmentation backbone, integrates the feature extraction capabilities of a pre-trained visual Transformer, the feature enhancement capabilities of a dual-branch adapter, and the optimization capabilities of structure-aware post-processing, to construct an end-to-end dense target localization framework, specifically including the following steps: 1. Foreground Cue Generation Step (S1) The input image is acquired, and multi-scale semantic features are extracted using a pre-trained visual Transformer encoder (preferably DINOv2). The output features from layers l to m (l > m ≥ 1, l ≤ 12) of the encoder are concatenated to form a multi-scale semantic feature map. This semantic feature map is then added to the spatial location code along the channel dimension and input into a self-attention mechanism to generate a weighted feature map to capture long-distance dependencies. The weighted feature map is then processed sequentially by a 3×3 convolution (padding=1), ReLU activation, 1×1 convolution, and sigmoid activation. A single-channel foreground confidence map is output through a context-aware classifier (to suppress background interference). In the foreground confidence map, the points with the maximum response in the local neighborhood of 5×5 to 11×11 pixels are selected as foreground keypoints to guide subsequent segmentation.

[0009] 2. Adapter-enhanced mask generation step (S2) The input image, foreground confidence map, and foreground keypoints are input into the pre-trained SAM model. A dual-branch adapter module is inserted in the middle layer of the ImageEncoder ViT layer 4-8 of the SAM. This module includes a parallel semantic adapter and a spatial adapter. The semantic adapter is a bottleneck residual structure that achieves dimensionality reduction and expansion of the channel dimension C→C / 4→C through a fully connected layer. The spatial adapter is a 3×3 depthwise separable convolution (groups=C, padding=1) used to capture local spatial features. The dual-branch output is weighted by a learnable scaling factor γ (initial value 0.1) and then fused with the input features in a residual manner. The SAM backbone parameters are fixed, and only the last two Transformer blocks of the mask decoder are fine-tuned. The target mask is generated by optimizing the joint loss function of Dice loss (weight 0.6) and cross-entropy loss (weight 0.4).

[0010] 3. Structure-aware post-processing step (S3) Region separation: The target mask is normalized to a grayscale range of 0~255, binarized with a threshold of 127, and preprocessed by 2×2 structuring element morphological erosion to obtain a preprocessed mask; the distance transformation map of the preprocessed mask is calculated, local maximum points are extracted and their intersection with the foreground region of the foreground confidence map is taken as seed points, watershed segmentation is performed to obtain non-overlapping regions, and the geometric center of each region is calculated as the initial target coordinates.

[0011] Additional key points: Extract the contour of the target mask and filter candidate contours with an area greater than a preset threshold; fit the minimum bounding rectangle for each candidate contour and calculate the center coordinates. If the Euclidean distance between the coordinates and the initial target coordinates is within the range of [3 pixels, 15 pixels], then supplement it as the target coordinates.

[0012] Accuracy verification: The final target coordinates are matched with the actual labeled coordinates. If the Euclidean distance is ≤σ (σ = 4~8 pixels, adapting to the target overlap), it is considered a valid location.

[0013] (III) Beneficial Effects Compared with the prior art, the beneficial effects of the present invention are as follows: Significantly improved positioning accuracy: On four major datasets, ShanghaiTech Part A (SHA), ShanghaiTech Part B (SHB), NWPU-Crowd, and UCF-QNRF, the F1 score at a 4-pixel threshold is 5.7% to 15.6 percentage points higher than traditional methods; the F1 score at an 8-pixel threshold reaches 76.8% to 85.4%, meeting the requirements for high-precision positioning.

[0014] It is robust and adaptable: the F1 score fluctuation is ≤7% under dual threshold (4 / 8 pixels), which is far lower than the fluctuation range of 15%~40% of traditional methods. It can stably adapt to low, medium, high and ultra-high density scenes. It can outperform the baseline method on other datasets after training only on the SHA dataset, and has outstanding generalization ability.

[0015] Highly efficient parameters and low deployment cost: The dual-branch adapter accounts for only 0.9% of the SAM model parameters, eliminating the need for full fine-tuning and significantly reducing training and inference costs; it can be deployed on conventional GPU devices such as NVIDIA A100, making it practical for engineering applications.

[0016] Good scene transferability: The modular design supports the expansion of target types. In addition to dense crowds, it can be directly transferred to non-crowded target localization scenarios such as cell nucleus counting and sheep flock monitoring without major modifications to the model structure. Attached Figure Description

[0017] To better understand the technical solution of this invention, the following description is provided in conjunction with the accompanying drawings (the drawing numbers correspond to the core content of the technical solution): Figure 1: Overall framework diagram of the present invention, showing the connection relationship and data flow of the foreground prompting module, the adapter enhancement mask generation module, and the structure perception post-processing module; Figure 2: Schematic diagram of the foreground cue module structure, including DINOv2 feature extraction, self-attention fusion, context-aware classifier and key point extraction process; Figure 3: Schematic diagram of the dual-branch adapter module structure, showing the parallel residual fusion structure of the semantic adapter (fully connected layer) and the spatial adapter (depthiable separable convolution); Figure 4: Flowchart of structure-aware post-processing, which shows in detail the steps and parameter settings for watershed segmentation and key point supplementation; Figure 5: Performance comparison visualization of the dataset, comparing the localization performance of the present invention with CAAPN and FIDTM in different density scenarios (red dots are predicted coordinates, blue dots are real coordinates). Figure 6: Comparison of ablation experiment results, showing the differences in positioning accuracy after removing different modules. Detailed Implementation

[0018] (I) Experimental Environment Configuration 1. Hardware Environment Graphics Processor: NVIDIA A6000 40G GPU 2. Software Environment Operating System: Ubuntu 20.04 LTS Deep learning frameworks: PyTorch 2.0.1, CUDA 11.7 Auxiliary libraries: OpenCV 4.6.0 (for post-processing), NumPy 1.23.5, Scikit-learn 1.2.2 Pre-trained model weights: DINOv2-vit-h (self-supervised pre-training), SAM-vit-h (pre-trained segmentation model) (II) Specific Implementation of Each Module 1. Implementation of the foreground hint module The foreground cueing module generates accurate foreground keypoints, providing automated guidance for subsequent segmentation. Its core functionality is achieved through three steps: multi-scale feature extraction, spatial feature fusion, and confidence map and keypoint generation, as detailed below: Multi-scale semantic feature extraction adjusts the input image to a preset resolution (e.g., 224×224), inputs it into a pre-trained visual Transformer encoder (e.g., DINOv2), extracts its output features at different levels (preferably 3-5 intermediate layer features, with each layer having the same feature dimension, such as 768), and stitches them together according to the channel dimension to form a multi-scale semantic feature map—by fusing shallow detail features with deep semantic features, it improves the feature representation ability of dense targets.

[0019] Spatial features and location information are fused to generate a sinusoidal position code that matches the dimensions of the semantic feature map. This sinusoidal position code is then superimposed on the semantic feature map along the channel dimension and input into a self-attention layer (the number of heads and the dimensions of the hidden layer can be set reasonably, such as 12 heads or 768 dimensions) for feature weighting. This captures long-distance spatial dependencies by leveraging the self-attention mechanism, and enhances the spatial localization accuracy of the features by combining the position code, outputting a weighted feature map.

[0020] Foreground confidence map and keypoint generation: Convolution and activation operations are performed sequentially on the weighted feature map: First, local features are extracted through 3×3 convolution (including boundary padding to avoid feature size loss) and ReLU activation. Then, the channels are compressed to 1 dimension through 1×1 convolution. After sigmoid activation, a foreground confidence map with the same resolution as the input image is generated (to suppress background interference and highlight the target area). In the confidence map, a local neighborhood sliding window of a preset size (such as 5×11 pixels, which can be adjusted according to the target scale) is used to extract the points with the maximum response within the window as foreground keypoints (hundreds are extracted for each image to meet the needs of dense target guidance).

[0021] Adapter-enhanced mask generation module implementation The adapter-enhanced mask generation module is used to optimize the feature representation capabilities of SAM, addressing issues such as insufficient feature interaction and mask sticking. Its core implementation is achieved through a "dual-branch adapter insertion - mask decoder fine-tuning" mechanism, as detailed below:

[0022] A dual-branch adapter is inserted into the intermediate layers (preferably 2-3 consecutive intermediate layers) of the pre-trained SAM image encoder (ImageEncoderViT). This adapter includes a parallel semantic adapter and a spatial adapter. The semantic adapter adopts a bottleneck residual structure and uses a fully connected layer to reduce and increase the dimensionality of feature channels (e.g., the number of channels C → C / 4 → C), thereby enhancing the semantic relevance of features. The spatial adapter employs 3×3 depthwise separable convolutions (with boundary padding to preserve feature size) to focus on capturing local spatial features; The outputs of the two branches are weighted by a learnable scaling factor (the initial value is set to a small value, such as 0.1, to avoid interfering with the pre-trained features) and then fused with the encoder input features in a residual manner—compensating for the lack of SAM feature interaction through the dual-dimensional enhancement of semantic and spatial features.

[0023] The image encoder and cue encoder parameters of SAM are fixed (to avoid disrupting pre-trained knowledge), and only the last 2-3 Transformer blocks of its mask decoder are fine-tuned. During training, a joint loss function is used (mainly Dice loss with a weight of 0.5-0.7, and secondary cross-entropy loss with a weight of 0.3-0.5), along with reasonable optimization strategies (such as AdamW optimizer and cosine decay of learning rate) and training batch settings to improve the accuracy of mask generation and reduce mask sticking.

[0024] 3. Implementation of the structure-aware post-processing module

[0025] The following preprocessing and segmentation operations are performed on the target mask to separate overlapping targets and locate initial coordinates: Normalization and binarization: The pixel values ​​of the target mask are normalized to a grayscale range of 0~255, and 127 is used as the threshold for binarization to obtain the preliminary separation results of the foreground (target) and background. Morphological erosion: A 2×2 rectangular structuring element is used to perform one morphological erosion operation on the binarized result to eliminate the small adhesions at the mask edges and optimize the target boundary contour. Seed point extraction: Calculate the L2 distance transform map of the eroded mask, and extract the local maximum points in the distance transform map using the connected component labeling algorithm. Use these local maximum points as seed points (the seed points need to intersect with the foreground region in the foreground confidence map to exclude background interference). Watershed segmentation and center localization: Based on the above seed points, the watershed segmentation algorithm is executed to segment the adhered target mask into independent regions that do not overlap. By calculating the mean of all pixel coordinates in each region, the geometric center of each region is determined as the initial positioning coordinates of the target.

[0026] To compensate for any missed targets during the initial localization, key points are supplemented through contour analysis and coordinate verification. The specific process is as follows: Candidate contour filtering: Perform contour extraction on the target mask (prioritize extracting the outermost contour to avoid nesting interference) and filter out candidate contours with an area greater than 10 pixels - this area threshold is used to exclude false target areas formed by background noise or mask fragments; Contour center localization: Fit the minimum bounding rectangle to each candidate contour, and obtain the candidate coordinates by calculating the geometric center of the rectangle. The minimum bounding rectangle can effectively avoid the influence of contour irregularities on center localization. Valid coordinate verification: Calculate the Euclidean distance between the candidate coordinates and the initial target coordinates. If the distance is within the range of 3 to 15 pixels, then the candidate coordinates are added as valid target coordinates. This distance range can balance the needs of "avoiding duplicate annotations" and "completing missing targets". Example

[0027] (a) Public safety monitoring When this invention is deployed in the monitoring system of large shopping malls and sports venues, it can output the coordinates of individual people in real time by inputting real-time monitoring images (1920×1080); when the number of targets in a unit area (1m²) is greater than 5, a crowding warning is triggered to assist security personnel in guiding the crowd.

[0028] (ii) Biomedical cell nucleus counting By inputting a pathological slide image (400×400), this invention can automatically locate the cell nucleus coordinates with a counting error of <5%, replacing manual counting and improving the efficiency of pathological diagnosis.

[0029] (III) Monitoring of Sheep Flocks in Agriculture By using drone aerial images (2048×2048), this invention can accurately locate the coordinates of each sheep in a flock, count the number of sheep, and identify stray individuals, thus facilitating intelligent management of pastures.

Claims

1. A dense target localization method based on modular optimization, characterized in that, Includes the following steps: S1. Foreground Cue Generation: Acquire the input image, extract multi-scale semantic features of the input image through a pre-trained visual Transformer encoder, fuse the multi-scale semantic features with spatial location encoding and input them into a self-attention mechanism for processing, and then process them through a context-aware classifier to generate a foreground confidence map and foreground key points; The foreground confidence map is used to suppress background interference, and the foreground key points are used to guide subsequent segmentation; S2. Adapter-enhanced mask generation: The input image, foreground confidence map, and foreground key points are input into the pre-trained SAM. A dual-branch adapter module is inserted into the intermediate layer of the ImageEncoderViT of the SAM. The cross-layer feature interaction is enhanced through the dual-branch adapter module. The mask decoder of the SAM is optimized based on the joint loss function to generate the target mask. S3. Post-processing of structure perception: The target mask is sequentially processed by region separation and key point supplementation to obtain the accurate coordinates of dense targets; the region separation is based on the watershed algorithm to segment the contiguous region, and the key point supplementation is based on the minimum bounding rectangle fitting to recover the missed targets.

2. The dense target localization method based on modular optimization according to claim 1, characterized in that, The pre-trained visual Transformer encoder mentioned in step S1 is a DINOv2 encoder; the multi-scale semantic features are obtained by concatenating the intermediate layer output features of the DINOv2 encoder; the fusion method of the multi-scale semantic features and spatial location encoding is as follows: the spatial location encoding and multi-scale semantic features are added according to the channel dimension, and then input into the self-attention mechanism to generate a weighted feature map; the number of attention heads of the self-attention mechanism is 12, and the dropout rate is 0.1; the processing procedure of the context-aware classifier is as follows: the weighted feature map output by the self-attention mechanism is sequentially processed by 3×3 convolution (padding=1), ReLU activation function, 1×1 convolution and sigmoid activation function to output a single-channel foreground confidence map; the extraction rule of the foreground key points is as follows: in the foreground confidence map, the pixel with the largest response value in each local neighborhood of 5×5~11×11 pixels is selected as the foreground key point; the size of the local neighborhood is adaptively adjusted according to the resolution of the input image.

3. The dense target localization method based on modular optimization according to claim 1, characterized in that, The dual-branch adapter module described in step S2 includes a semantic adapter and a spatial adapter configured in parallel. The semantic adapter is a bottleneck residual structure, which performs the following operations in sequence: reducing the channel dimension of the input feature from C to C / 4 through a fully connected layer, performing ReLU activation processing, and restoring the channel dimension from C / 4 to C through a fully connected layer. The spatial adapter is a 3×3 depth-separable convolutional structure, with the number of convolutional groups being the same as the number of input feature channels C, and padding=1; The output of the dual-branch adapter module is weighted by a learnable scaling factor γ and then fused with the input features of the dual-branch adapter module in a residual manner; the initial value of γ is 0.1, which is adaptively updated through backpropagation.

4. The dense target localization method based on modular optimization according to claim 1, characterized in that, In step S2, the dual-branch adapter module is inserted into the intermediate layer of the ImageEncoderViT of the SAM; the joint loss function is a weighted sum of Dice loss and cross-entropy loss with a weight ratio of 0.6:0.4; the mask decoder only fine-tunes the last two Transformer blocks, while the parameters of the remaining layers are fixed.

5. The dense target localization method based on modular optimization according to claim 1, characterized in that, The region separation described in step S3 includes the following sub-steps: S5.

1. Normalize the target mask to a grayscale range of 0~255, use 127 as the threshold for binarization, and then perform morphological erosion operation through a 2×2 pixel rectangular structural element to obtain the preprocessed mask. S5.

2. Perform a distance transformation on the preprocessed mask to obtain a distance transformation map; extract local maximum points from the distance transformation map and take the intersection with the foreground region of the foreground confidence map as seed points for the watershed algorithm; S5.

3. Using the seed point as a marker, perform watershed segmentation on the target mask to obtain multiple non-overlapping regions; calculate the geometric center coordinates of each non-overlapping region as the initial target coordinates.

6. The dense target localization method based on modular optimization according to claim 1, characterized in that, The key point supplementation mentioned in step S3 includes the following sub-steps: S6.

1. Perform contour extraction on the binarization result of the target mask, and filter out contours with an area greater than a preset threshold as candidate contours; S6.

2. Fit the minimum bounding rectangle for each candidate contour, and calculate the center coordinates of the minimum bounding rectangle; S6.

3. Calculate the Euclidean distance between the center coordinates and the initial target coordinates. If the Euclidean distance is within the range of [dmin, dmax], then supplement the center coordinates with the target coordinates; where dmin = 3 pixels and dmax = 15 pixels.

7. The dense target localization method based on modular optimization according to claim 1, characterized in that, The verification rule for the precise coordinates in step S3 is as follows: match the precise coordinates with the marked real target coordinates. If the Euclidean distance between the two is ≤σ, it is determined to be a valid location. The value of σ is 4~8 pixels, which is adaptively selected according to the degree of overlap of dense targets.

Images