A text prompt-based zero-shot dense target counting method

Abstract

This invention discloses a zero-sample dense target counting method with text prompts, belonging to the fields of computer vision and multimodal deep learning. Addressing the problems of ambiguous semantic responses and loss of small target boundaries in dense counting of existing large visual language models, this invention transforms textual semantics into high-purity visual priors through dynamic virtual prototype generation. It recovers spatial details worn down by deep networks through high-resolution feature adaptive fusion, and combines a dual-branch parallel prediction architecture with a hierarchical efficient fine-tuning strategy. Only a single natural language command is needed to achieve high-precision counting and localization of dense targets of unknown categories. Experiments show that this method exhibits good generalization performance and counting accuracy in cross-category and cross-domain scenarios.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and multimodal learning technology, specifically relating to a zero-sample dense target counting method based on text prompts. Background Technology

[0002] The target counting task aims to locate and count the number of target instances belonging to a specific semantic category in complex scenes. Existing few-shot counting methods typically follow a visual matching setting, heavily relying on a small number of visual example boxes manually annotated by the user on the query image as guiding cues. However, in real-world and complex open-world applications, it is difficult to require users to provide high-quality, highly representative targets at any time. To reduce the reliance on target bounding boxes, the number of samples provided by the user needs to be reduced. When the number of samples is reduced to zero, the few-shot method evolves into a zero-shot method.

[0003] Compared to visual bounding boxes, natural language possesses a high degree of semantic abstraction and broad coverage, allowing users to directly perform target search and count through text commands (such as "bird"). In recent years, large-scale visual language models (such as Grounding DINO) have demonstrated zero-shot generalization capabilities in open-vocabulary object detection through cross-modal contrastive learning of massive image-text pairs, making text-driven general object localization and zero-shot counting possible.

[0004] However, when these large open-vocabulary visual language models designed for routine object detection tasks are directly applied to dense counting scenarios with highly overlapping objects, the following significant technical drawbacks exist:

[0005] There is a contradiction between macroscopic semantic alignment and microscopic extremum constraints, which can easily lead to semantic diffusion. Existing large-scale visual language models tend to focus on macroscopic semantic alignment and lack microscopic constraints for dense point distributions. For severely occluded small targets, the semantic-to-pixel mapping of large models is not accurate enough. In densely overlapping regions, the global response map of large models is prone to "semantic diffusion" and background false alarms, causing the model to be unable to accurately distinguish adjacent independent individuals from contiguous response clumps, thus resulting in localization ambiguity at the microscopic instance level.

[0006] Insufficient resolution of deep features leads to the loss of high-frequency physical boundary information. Existing large-scale visual language models typically retain only deep features with high downsampling factors (e.g., 8x, 16x, 32x) to control computational complexity. The physical boundaries and high-frequency textures of small targets are worn away and lost during multiple downsampling operations (e.g., multiple convolution and pooling operations). In extremely dense scenes, multiple small targets that were originally independent but close together exhibit severe "boundary adhesion" on deep feature maps. Furthermore, the low-level decoding and post-processing of large models rely on geometric bounding box repulsion, inevitably leading to missed detections and counting distortion in dense regions.

[0007] Traditional fine-tuning methods for downstream tasks are prone to catastrophic forgetting. The generalization ability of large visual language models stems from their high-dimensional feature manifolds built on massive datasets. When transferring these models to dense counting tasks, if a traditional full-scale fine-tuning approach is used, the model will drastically distort its original high-dimensional feature manifold to accommodate the crowded features in a specific training set. Traditional full-scale fine-tuning may overwrite the original high-dimensional feature manifold of the large model, leading to catastrophic forgetting. This causes the model to completely lose its ability to recognize unseen categories in the open world, thus destroying the model's original zero-shot generalization ability.

[0008] In summary, how to solve the semantic diffusion and boundary adhesion problems in micro-dense scenes while preserving the powerful cross-modal generalization and understanding capabilities of large visual language models, and how to construct localization constraints for extremely dense target groups, are key technical problems that urgently need to be solved in this field. Summary of the Invention

[0009] To address the technical challenges of existing Visual Language Models (VLMs) in dense counting scenarios, such as semantic diffusion leading to ambiguous localization, loss of deep features causing boundary adhesion, and full fine-tuning triggering catastrophic forgetting, this invention provides a zero-sample dense target detection and counting method based on text prompts.

[0010] The technical solution of the present invention is as follows:

[0011] A zero-sample dense target counting method based on text prompts includes the following steps:

[0012] S1, Independent Encoding of Multimodal Features

[0013] The system receives a query image and a natural language text prompt describing the target category. The text modality is processed by a BERT encoder to extract a high-dimensional text feature sequence, denoted as . ,in Where is the number of lexical units, and D is the feature dimension. Visual modalities are extracted using a Swin Transformer backbone network for multi-scale feature extraction. This invention intercepts and extracts high-resolution feature maps with a downsampling rate of 4x (4x) in the first stage of the backbone network's base layer. Meanwhile, the network continues to extract deep feature maps at downsampling rates of 8x, 16x, and 32x. .

[0014] S2, Cross-modal global semantic interaction

[0015] Multi-scale visual and text features are flattened and fed into a bidirectional cross-attention encoder based on stacked multi-layer Transformers. Each layer contains three operations: intra-modal self-attention enhancement, where standard self-attention is used for the text modality and deformable self-attention for the visual modality to reduce computational complexity; image-to-text cross-attention, enabling text terms to retrieve full-image visual content to eliminate semantic ambiguity; and text-to-image cross-attention, injecting text semantics into visual features. After multiple iterations, the encoder output incorporates global memory features infused with prior textual information. and target text features , where L is the total number of spatial pixels after flattening, and D is the dimension of the feature.

[0016] S3, Dynamic Virtual Prototype Generation

[0017] This invention proposes a dynamic virtual prototype generation mechanism based on confidence level, which automatically extracts visual priors from textual semantics.

[0018] S3.1 Cross-modal similarity calculation

[0019] Global memory features By mapping linear projection layers and layer normalization to the contrastive learning space, we obtain... Calculate the dot product similarity between the target text features and each visual token:

[0020]

[0021] The similarity vector is mapped to the Sigmoid function. The interval is reconstructed into a two-dimensional global probability response graph.

[0022] S3.2, Top-K Search and Dynamic Mask Replacement

[0023] Sort all visual tokens in descending order of similarity response value, and extract the first... One as a candidate virtual prototype set and corresponding confidence scores The candidate with the highest confidence level is labeled as the optimal anchor feature. Threshold validation is performed on each candidate feature:

[0024]

[0025] in A pre-set confidence threshold (usually 0.3) is used. Candidate features below the threshold are forcibly replaced with the optimal anchor feature, resulting in a set of virtual example prototypes. .

[0026] S4, High-resolution feature adaptive fusion and recombination

[0027] This step injects the textual semantics implied by the virtual prototype of S3 into the 4x higher resolution features extracted by S1.

[0028] S4.1, Feature Projection

[0029] use Convolution and group normalization will The channel dimension is mapped to the dimension D consistent with the cross-modal encoder, resulting in... .

[0030] S4.2 Semantic Injection and Boundary Refinement

[0031] It adopts a decoupled cascaded architecture, consisting of two serial attention blocks.

[0032] Prototype attention block: Flatten as a query, using a virtual prototype Standard multi-head cross-attention is performed, with the feature map acting as both key and value. A 4x feature map is used to perceive the semantic information of the virtual prototype, and pixels consistent with the textual semantics are activated and enhanced.

[0033] Boundary refinement block: Taking semantically injected features and positional encodings as input, it performs single-scale deformable self-attention, an operation performed only at a single scale of 4x resolution. The network predicts the sampling offset of each pixel through linear projection. With attention weight A, features are extracted at sub-pixel locations using bilinear interpolation and then weighted and aggregated. After this processing, the 4x feature set is transformed into an enhanced feature map that combines textual semantic discriminative ability with high-resolution spatial sharpness. .

[0034] S4.3, Cross-scale pyramid reconstruction

[0035] The deep memory features output by S2 are restored to 8x, 16x, and 32x feature maps according to spatial resolution. A top-down fusion method is used: starting with the 32x features, after bilinear interpolation upsampling, they are concatenated with the 16x features along the channel dimension. Convolution and group normalization are combined. This operation is performed layer by layer upwards, ultimately merging with... The final fusion is performed to output a target-aware feature map that takes into account both macroscopic semantics and microscopic boundaries.

[0036] S5. Joint Optimization of Bi-branch Parallel Prediction and Extreme Value Constraints

[0037] The target perception feature map is fed into the detection branch and the density map estimation branch in parallel. The two branches share the underlying features but undertake prediction tasks of different granularities.

[0038] The detection branch outputs instance-level discrete predictions, including a centrality classification head and a bounding box regression head, which predict the confidence score of each spatial location belonging to the target center and the corresponding normalized bounding box coordinates, respectively. The density map estimation branch generates a continuous density map through a fully convolutional sub-network and uses the Softplus activation function to ensure that the output is non-negative.

[0039] This invention employs a density loss based on near-end mapping for constraint. This loss divides the density map into non-overlapping jurisdictional blocks through nearest-neighbor assignment, and within each block... The norm forces the sum of densities to approximate 1, while driving the density values ​​to converge toward the target center through the centralization distance cost.

[0040] S6, Layered High-Efficiency Fine-Tuning Strategy

[0041] To preserve the open vocabulary generalization capability of the large visual language model, this invention employs hierarchical parameter control: all parameters of the text encoder and the Swing Transformer visual backbone network are frozen and do not participate in gradient updates; the parameters of the first four layers of the cross-modal encoder are frozen, with only the last two layers open for updates and given a very small learning rate (0.01 to 0.1 times the backbone learning rate); lightweight task adapter components such as the feature hooking projection layer in S1, the high-resolution feature adapter in S4, and the dual-branch prediction head in S5 are given normal learning rates and updated in full.

[0042] S7, Adaptive Fusion Counting in the Inference Phase

[0043] During the inference phase, the network obtains a set of candidate detection boxes and a continuous density map through forward propagation. This invention employs an adaptive fusion strategy based on crowding determination: in sparse and medium-density regions, the number of candidate detection boxes is directly used as the count value; in crowded regions determined to have severe feature aliasing, a two-dimensional joint mask is generated using all candidate detection boxes. Spatial integration of the density map is performed only within the effective region defined by the mask:

[0044]

[0045] in These are density map predicted values. This represents element-wise product. The fusion strategy utilizes the localization information of the detection candidate boxes to delineate the integration region to mask background noise, while continuously integrating the density map to compensate for false detections in crowded regions by the detection branches.

[0046] The present invention has the following beneficial effects:

[0047] Users only need to input text to complete the counting, eliminating the need for manual bounding box annotation and significantly reducing manual intervention. The dynamic virtual prototype generation mechanism and hierarchical fine-tuning strategy ensure stable performance for unseen training categories and cross-domain scenarios, demonstrating strong open-world generalization capabilities. Four times higher resolution feature extraction and single-scale deformable boundary refinement restore the spatial details of small targets, while density supervision based on extremum constraints generates sharp extremum responses at the original target center, mitigating response blurring in densely occluded regions. The dual-branch architecture enables detection and density prediction to work collaboratively on the same feature base, and the adaptive fusion strategy in the inference phase maximizes the effectiveness of both paradigms in their respective strengths. Hierarchical fine-tuning trains only a small number of adapter components, freezing most base parameters, reducing memory usage and computational overhead, making it practically feasible to adapt large models to dense counting tasks. Attached Figure Description

[0048] Figure 1 This is the overall flowchart of the present invention;

[0049] Figure 2 This is an example of a global similarity response graph visualization;

[0050] Figure 3 This is a schematic diagram of the high-resolution feature adaptive fusion and recombination process. Detailed Implementation

[0051] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0052] Example:

[0053] This embodiment provides a zero-sample dense target counting method based on text prompts. The overall process is as follows: Figure 1 As shown in the diagram. The visual language large-scale model uses the pre-trained Grounding DINO as its foundation, with BERT as the text encoder and Swin Transformer as the visual backbone network. The network input consists of an image to be counted and a natural language text prompt; the output is the fused count value and the bounding box of the target instance. The training dataset uses FSCD-147, and the test datasets include the unseen class division of FSCD-147 and the CARPK cross-domain dataset.

[0054] S1, Independent Encoding of Multimodal Features

[0055] This step receives a one-to-many modal input: a spatial resolution of [missing information]. The query image, along with a natural language text prompt describing the category of the target to be counted.

[0056] For the text modality, the input natural language text is first processed into a word sequence by a tokenizer, and then fed into the pre-trained BERT text encoder. The BERT encoder consists of 12 stacked Transformer encoder layers. After extraction through multiple layers of self-attention mechanisms, the output is a text feature sequence containing deep syntactic structure and contextual semantic information, denoted as . ,in Let D be the number of lexical units and D be the feature embedding dimension. In this embodiment, D is 256 dimensions.

[0057] For the visual modality, the input image undergoes multi-scale feature extraction through a hierarchical Swin Transformer visual backbone network. The Swin Transformer naturally outputs feature maps with multiple spatial resolutions through a stage-by-stage window-shifting self-attention mechanism and downsampling operations. Specifically, the four stages of the Swin Transformer output feature maps with downsampling rates of 4x, 8x, 16x, and 32x, respectively.

[0058] The native cross-modal encoder of Grounding DINO only receives deep feature maps at three levels: 8x, 16x, and 32x. This invention makes a key modification: an explicit feature extraction hook is injected after the base layer of the first stage (stage 1) of the Swin Transformer. This hook intercepts intermediate feature tensors during the forward propagation of the backbone network, extracting and retaining high-resolution feature maps at a 4x downsampling rate, denoted as... Meanwhile, the network continues to extract deep feature maps from the output of subsequent stages. The number of channels is 256, 512, and 1024 dimensions, respectively. The 4x high-resolution feature map, together with the 8x, 16x, and 32x deep feature maps, constitutes the unique multi-level visual feature pyramid of this invention.

[0059] S2, Cross-modal global semantic interaction

[0060] After obtaining independently encoded text features and multi-scale visual features, this step is responsible for bridging the modal gap between visual pixels and abstract language, enabling visual features to perceive text instructions.

[0061] This step relies on a bidirectional cross-attention cross-modal encoder based on a stack of 6 Transformer layers, called the feature enhancer. Each layer contains three core operations.

[0062] Intramodal self-attention enhancement. Standard multi-head self-attention operations are performed on text feature sequences to capture syntactic associations and long-range dependencies between lexical units. Deformable self-attention operations are performed on multi-scale visual feature sequences.

[0063] Image-to-text cross-attention. Using text features as query vectors and visual features as keys and values, each text term can retrieve the visual content of the entire image, thus incorporating the specific physical context of the current image into the text features.

[0064] Text-to-image cross-attention. Visual features are used as query vectors, and text features as keys and values, enabling each visual pixel to retrieve its corresponding text terms, thus injecting high-level text semantic priors into the low-level visual features. Residual connections and layer normalization are applied after each attention operation.

[0065] After repeated iterations and feature weaving through the six layers of bidirectional attention modules described above, the cross-modal encoder finally outputs a set of global memory features injected with prior textual semantics. and target text features .

[0066] S3, Dynamic Virtual Prototype Generation

[0067] This step proposes a confidence-guided dynamic virtual prototype generation mechanism, which transforms text-driven macro-semantic retrieval into a set of highly discriminative local visual priors.

[0068] S3.1 Cross-modal similarity calculation and global response map generation

[0069] First, all visual tokens in the global memory features M output by S2 are fed into a projection head consisting of a linear projection layer and a layer normalization layer. This maps the feature dimension D to the contrastive learning space, resulting in the projected visual features. .

[0070] Then, calculate the target text features. Dot product similarity with every visual token in the entire image:

[0071]

[0072] The obtained similarity vector After processing with the Sigmoid activation function:

[0073]

[0074] in Indicates the first The semantic matching probability between each visual token and the target text. (All) By rearranging them according to their spatial location, a global probability response map for the current text category can be reconstructed in two-dimensional space. ,in and The spatial dimensions before flattening multi-scale features.

[0075] To visually verify the actual effect of the above cross-modal feature alignment Figure 2 A typical example is given. The left side shows the input image, the middle side shows the similarity response map for the text prompt "glasses," with red areas indicating high semantic matching probabilities, and the right side shows a visualization of the overlay of the response map and the original image. From... Figure 2 As can be seen, after the cross-modal semantic interaction in S2, the high-response extreme value region closely matches the target location, and the background interference region is effectively suppressed. This response map provides a high-quality initial anchor point for the subsequent candidate retrieval in S3.2.

[0076] S3.2 Top-K Candidate Retrieval and Optimal Anchor Point Extraction

[0077] Based on the probability values ​​of the global probability response map, all L visual tokens are sorted in descending order of response value, and the tokens with the highest response values ​​are extracted. One visual feature as a candidate virtual prototype set Simultaneously, the confidence score corresponding to each candidate feature is recorded. In this embodiment Take 5.

[0078] The candidate feature with the highest confidence score, ranked first, is extracted and marked as the optimal anchor feature. Because this feature has the highest text matching response across the entire graph, that is:

[0079]

[0080] It is considered to be a visual feature representation in the current image that is semantically valid and representative of the target category.

[0081] S3.3 Dynamic Mask Replacement Based on Confidence Threshold

[0082] For each candidate feature in the candidate set Calculate its confidence score Compared with the preset replacement threshold A comparison is made. In this embodiment... Let's take 0.3. The decision rule is given by the following formula:

[0083]

[0084] Through the aforementioned dynamic mask replacement mechanism, the network ultimately outputs a set of high-purity virtual example prototypes:

[0085]

[0086] This set of virtual prototypes is semantically faithful to the user's text commands and physically eliminates background noise and low-confidence interference, providing clean and reliable guiding signals for subsequent dense target localization and counting.

[0087] S4, High-resolution feature adaptive fusion and recombination

[0088] Through S2 and S3, the network has obtained global semantic-level memory features and virtual prototypes, and S1 extracted feature maps with a resolution four times higher. The overall structure of this step is as follows: Figure 3 As shown, it is completed in sequence by three sub-modules.

[0089] S4.1 Lossless projection of low-level high-frequency features

[0090] Due to the original 4x higher resolution feature map The channel dimension is 128, which does not match the hidden layer dimension D=256 of the cross-modal encoder, requiring dimension alignment first. This submodule uses a lightweight feature projection operator, derived from... It consists of a two-dimensional convolutional kernel and a group normalization layer (with 32 groups):

[0091]

[0092] in The number of input channels is 128, the number of output channels is 256, and the step size is 1; This is a group normalization operation. After projection, a channel-aligned high-resolution feature map is obtained. .

[0093] S4.2, Semantic Injection and Boundary Refinement Decoupling Architecture

[0094] After obtaining the high-resolution feature map after projection, this submodule adopts a cascaded architecture of semantic injection and spatial refinement.

[0095] Prototype attention block: First, the two-dimensional high-resolution feature map is... Flattened into a one-dimensional sequence As a query vector, the virtual prototype set generated by S3. The key matrix is ​​obtained through linear projection. Sum matrix Then perform the standard multi-head cross-attention operation:

[0096]

[0097]

[0098] in All are learnable projection matrices. For each attention head, the number of heads is set to 4. The multi-head outputs are concatenated and linearly projected to obtain the semantically injected features. .

[0099] Boundary Refinement Block: After semantic injection, the spatial boundaries of the high-resolution feature map are sharpened. This block uses features with injected semantics... Two-dimensional sinusoidal position coding The result of the summation is used as the input feature map. A single-scale deformable self-attention mechanism is employed (applying only to feature maps with 4x resolution). The specific operation is as follows: For each query point on the feature map... (Its reference coordinates are) ), predicted by linear projection Each sampling offset and the corresponding attention weights ( Subsequently, the bilinear interpolation function was used. Extract sub-pixel locations from the input feature map X. The feature values ​​at each location are summed and weighted according to the attention weights. This process can be expressed as:

[0100]

[0101] S4.3 Adaptive Reorganization of Cross-Scale Feature Pyramids

[0102] This submodule splits the deep memory features according to spatial resolution, restoring them into feature maps at three levels: 8x, 16x, and 32x. A top-down feature pyramid fusion path is adopted. Let the... The fusion characteristics of the layers are Deep features are The fusion process is represented as:

[0103]

[0104] in This indicates bilinear interpolation with a 2x upsampling; Concat concatenates along the channel dimension; and Fuse is the fusion module. The fusion module consists of... The concatenation process consists of a 2D convolution (stride 1, padding 1), group normalization (32 groups), and ReLU activation function, arranged sequentially. The concatenation operation starts with 32 times the features (…). ), after 16 times the characteristic ( ), and then to 8 times the features ( Finally, at the bottom layer of the pyramid, it is fused with the 4x enhanced feature map generated by S4.2. After an additional independent 2x upsampling, the network outputs a set of target-aware feature maps with high spatial resolution.

[0105] S5, Bi-branch Parallel Prediction and Joint Optimization

[0106] The target-aware feature maps are fed in parallel into the detection branch and the density map estimation branch. The detection branch consists of a centrality classification head and a bounding box regression head: the centrality classification head outputs a two-dimensional confidence map; the bounding box regression head outputs normalized coordinates. The density map estimation branch generates a density map through a fully convolutional sub-network, and the softplus activation function is used at the end to ensure that the output is strictly non-negative.

[0107]

[0108] To achieve accurate supervision of density branches, this invention employs a density loss based on near-end mapping for constraint. This loss divides the density map into non-overlapping jurisdictional blocks through nearest-neighbor assignment, and within each block... The norm forces the sum of densities to approximate 1, while simultaneously driving the density values ​​to cluster towards the target center through a centralization distance cost. Its overall form is as follows:

[0109]

[0110] in As a background penalty term, the density values ​​are weighted pixel by pixel through a centered distance cost matrix, which enhances values ​​near the center and suppresses values ​​far from the center. As a counting constraint, the density integral within each block is forced to approximate 1.

[0111] Detecting branch loss With density branching loss The weighted summation constitutes the joint optimization objective, with weight coefficients... In this embodiment, it is set to 0.3:

[0112]

[0113] S6, Layered High-Efficiency Fine-Tuning Strategy

[0114] This invention employs a hierarchical parameter control strategy, where all parameters of the text encoder are completely frozen during training and do not participate in gradient updates. Similarly, all parameters of the visual backbone network are also completely frozen.

[0115] In the 6 layers of the cross-modal encoder, the parameters of the first 4 layers are strictly frozen, and only the last 2 layers are allowed to participate in the update, with a very small learning rate assigned to these two layers. The newly added lightweight components of this invention, including the feature hooking projection layer in S1, the high-resolution feature adapter in S4, and the dual-branch prediction head in S5, use a learning rate... Participate in full gradient update.

[0116] S7, Adaptive Fusion Counting in the Inference Phase

[0117] During the inference phase, the network obtains a set of candidate detection boxes through forward propagation. With continuous density map This invention employs an adaptive fusion strategy based on crowding determination: in sparse and medium-density regions, the number of detection candidate boxes is directly used as the count value; in crowded regions determined to have severe feature aliasing, a two-dimensional joint mask is generated using all detection candidate boxes. Spatial integration of the density map is performed only within the effective region defined by the mask:

[0118]

[0119] in Represents element-wise product, mask Defined as:

[0120]

[0121] In the text-driven architecture of this invention, the above-mentioned detection candidate boxes are generated by adapting the virtual prototype to the S4 high-resolution features. Their positioning accuracy is significantly better than that of directly using the native output of Grounding DINO. Therefore, the mask region can more accurately cover the distribution range of the real target, thereby improving the accuracy of fusion counting.

[0122] S8, Experimental Verification

[0123] To verify the effectiveness of this invention, experimental evaluations were conducted on the FSCD-147 and CARPK datasets.

[0124] Table 1. Counting and detection results of different methods in FSCD 147

[0125]

[0126] Specifically, on the FSCD-147 test set, the method of this invention achieved an MAE of 9.78 and an RMSE of 94.69 under text prompt settings, which is better than GroundingREC's 10.12 / 107.19 and COUNTGD's 12.98 / 98.35. On the CARPK cross-domain test set, the method of this invention achieved an MAE of 3.46 and an RMSE of 5.03, which is better than COUNTGD's 3.83 / 5.41, and also better than LOCA's 9.97 / 12.51 and SAFECount's 5.33 / 7.04 using visual examples.

[0127] Table 2 Counting results of different methods in CARPK

[0128]

[0129] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A zero-sample dense target counting method based on text prompts, characterized in that, Includes the following steps: S1. Receive a query image and a natural language text prompt describing the target category. The text modality is processed by a BERT encoder to extract a high-dimensional text feature sequence, denoted as... ,in D represents the number of lexical units and D represents the feature dimension. For visual modalities, a Swin Transformer is used as the backbone network to extract multi-scale visual features. In the first stage of the backbone network, the base layer intercepts and extracts high-resolution feature maps with a downsampling rate of 4 times. Meanwhile, the network continues to extract deep feature maps at downsampling rates of 8x, 16x, and 32x. ; S2. After flattening the multi-scale visual features and text features, they are fed into a bidirectional cross-attention encoder based on multi-layer Transformer stacking. Each layer contains three operations: intra-modal self-attention enhancement, where standard self-attention is used for the text modality and deformable self-attention is used for the visual modality to reduce computational complexity; image-to-text cross-attention, which enables text terms to retrieve the visual content of the entire image to eliminate semantic ambiguity; and text-to-image cross-attention, which injects text semantics into visual features. After multiple iterations, the encoder output is injected with global memory features from prior textual knowledge. and target text features , where L is the total number of spatial pixels after flattening, and D is the dimension of the feature; S3. Employs a confidence-guided dynamic virtual prototype generation mechanism to automatically extract visual priors from textual semantics: S3.1 Cross-modal similarity calculation: Global memory features By mapping linear projection layers and layer normalization to the contrastive learning space, we obtain... Calculate the dot product similarity between the target text features and each visual token: ， The similarity vector is mapped to the Sigmoid function. The interval is reconstructed into a two-dimensional global probability response graph; S3.2, Top-K Search and Dynamic Mask Replacement: Sort all visual tokens in descending order of similarity response value, and extract the first... One as a candidate virtual prototype set and corresponding confidence scores The candidate with the highest confidence level is marked as the optimal anchor feature. For each candidate feature, a threshold check is performed: ， in To set a pre-defined confidence threshold, candidate features below the threshold are forcibly replaced with the optimal anchor feature, thus obtaining a set of virtual example prototypes. ; S4. Inject the textual semantics implied by the virtual prototype in S3 into the 4x higher resolution features extracted in S1, including: S4.1, Feature Projection: use Convolution and group normalization will The channel dimension is mapped to the dimension D consistent with the cross-modal encoder, resulting in... ; S4.2 Semantic Injection and Boundary Refinement: It adopts a decoupled cascaded architecture, consisting of two serial attention blocks: Prototype attention block: Flatten as a query, using a virtual prototype As keys and values, standard multi-head cross attention is performed, the feature map perceives the semantic information of the virtual prototype by 4 times, and pixels consistent with the text semantics are activated and enhanced; Boundary refinement block: Taking semantically injected features and positional encodings as input, it performs single-scale deformable self-attention, an operation performed only at a single scale of 4x resolution; the network predicts the sampling offset of each pixel through linear projection. With attention weight A, features are extracted at sub-pixel locations using bilinear interpolation and then weighted and aggregated. After processing, the 4x feature set is transformed into an enhanced feature map that combines textual semantic discriminative ability with high-resolution spatial sharpness. ; S4.3, Cross-scale pyramid reconstruction: The deep memory features output by S2 are restored to 8x, 16x, and 32x feature maps according to spatial resolution. A top-down fusion method is used: starting with the 32x features, after bilinear interpolation upsampling, they are concatenated with the 16x features along the channel dimension. Convolution and group normalization are combined; this operation is performed layer by layer upwards, ultimately merging with... The final fusion is performed to output a target perception feature map that takes into account both macroscopic semantics and microscopic boundaries. S5. Joint optimization of bi-branch parallel prediction and extreme value constraint: The target perception feature map is fed into the detection branch and the density map estimation branch in parallel. The two branches share the underlying features but undertake prediction tasks of different granularities. The detection branch outputs instance-level discrete predictions, including a centrality classification head and a bounding box regression head, which respectively predict the confidence that each spatial location belongs to the target center and the corresponding normalized bounding box coordinates. The density map estimation branch generates a continuous density map through a fully convolutional sub-network and uses the Softplus activation function to ensure that the output is non-negative. A density loss based on near-end mapping is used for constraint. This loss divides the density map into non-overlapping jurisdictional blocks through nearest-neighbor assignment, and within each block... The norm forces the sum of densities to approximate 1, while simultaneously driving the density values ​​to converge toward the target center through a centralization distance cost. S6. Layered parameter control is adopted: all parameters of the text encoder and the Swin Transformer visual backbone network are frozen and do not participate in gradient updates; the parameters of the first 4 layers of the cross-modal encoder are frozen, only the last 2 layers are opened to participate in updates, and a very small learning rate is given; lightweight task adapter components such as the feature hooking projection layer in S1, the high-resolution feature adapter in S4, and the dual-branch prediction head in S5 are given normal learning rates and updated in full. S7, Adaptive fusion counting during inference phase: During the inference phase, the network obtains a set of candidate detection boxes and a continuous density map through forward propagation, and adopts an adaptive fusion strategy based on crowding determination: in sparse and medium-density regions, the number of candidate detection boxes is directly used as the count value; in crowded regions determined to have severe feature aliasing, a two-dimensional joint mask is generated using all candidate detection boxes. Spatial integration of the density map is performed only within the effective region defined by the mask: ， in These are density map predicted values. This represents element-wise product.

Images