Zero sample target counting method based on visual enhancement and expert cooperation

Abstract

The invention discloses a zero sample target counting method based on visual enhancement and expert collaboration, which comprises the following steps: firstly, extracting text features of text data through a text encoder, and extracting initial image features of image data through an image encoder; embedding a visual prompt tuning model in the image encoder to perform visual enhancement on the initial image features to obtain optimized image features; inputting the optimized image features and text features into a feature enhancer, and performing enhancement processing to obtain image enhancement features and text enhancement features; the two enhanced features are input into a dynamic selection Top-K module, and a high-quality feature proposal is obtained by combining intersection-parallel ratio diversity constraint screening; a high-quality feature proposal is input into a cross-modal decoder containing a hybrid expert model and an intelligent gating network, adaptive expert combination processing features are dynamically selected, finally, a target number estimation result is screened and output through a similarity matrix, and the method is suitable for various target counting tasks in an open world scene.

Description

Technical Field

[0001] This invention belongs to the field of computer pattern recognition technology, specifically relating to a zero-sample target counting method based on visual enhancement and expert collaboration, which is particularly suitable for the task of estimating the number of unlabeled targets in open-world scenarios. Background Technology

[0002] Object counting is an important research direction in computer vision. Its core is to automatically identify and accurately estimate the number of specific objects based on image or video data, and it has wide applications in real-world scenarios. Traditional supervised object counting methods rely on large-scale, precisely labeled datasets, which are not only costly in terms of data acquisition and labeling, but also prone to performance degradation due to inconsistent labeling quality. More importantly, these methods are based on the closed-set assumption, meaning they can only handle predefined object categories during training and struggle to cope with the constantly emerging new categories in open-world scenarios, thus limiting their generalization ability.

[0003] Zero-shot target counting, through natural language description, estimates the number of specified targets in an image without labeled samples, offering a new approach to solving the aforementioned problems. Existing zero-shot target counting methods rely on large-scale pre-trained visual-language models to build cross-modal semantic bridges, mainly falling into two core categories. One category is density map regression-based methods, represented by CLIP-Count, which extract joint features from the image and text using CLIP and regress the density map to predict the overall spatial distribution and total number of targets. This is suitable for dense scenes, but its accuracy is limited in detecting small targets. The other category is detection-based methods, represented by CountGD, which utilizes an open-vocabulary detection model to first locate target instances one by one and then count the total. This approach performs better in terms of target localization accuracy and semantic consistency, but it is prone to missed detections and duplicate detections when handling high-density overlapping scenes.

[0004] Despite significant advancements in existing technologies, notable shortcomings remain. First, models relying solely on verbal descriptions are prone to ambiguity; text struggles to accurately convey the specific visual features of a target, such as the shape, size, and shade of red of a "red apple," limiting recognition accuracy. Second, models tend to overlook the spatial distribution and diversity of target features, selecting features based solely on confidence levels. In complex scenarios, this may lead to the selection of redundant and spatially overlapping features, impacting counting accuracy. Third, models lack dynamic optimization mechanisms for feature processing. Employing a uniform feature processing flow fails to automatically select appropriate feature paths or expert modules based on the task characteristics of different scenarios or target categories, resulting in wasted computational resources and potentially reduced counting performance. Summary of the Invention

[0005] This invention aims to solve the above-mentioned problems and proposes a zero-sample target counting method based on visual enhancement and expert collaboration, characterized by the following steps: (1) Input the text data into the text encoder and output the text feature t; input the image data into the image encoder, and after processing by the encoder backbone network, output four sets of initial image features of different sizes. The sizes of the four sets of initial image features are H / 8×W / 8×256, H / 16×W / 16×256, H / 32×W / 32×256, and H / 64×W / 64×256, respectively. (2) Embed a visual cue tuning model in each layer of the initial image features of the image encoder, perform visual enhancement operations on each layer of the initial image features, and integrate the enhanced features of all layers to obtain the optimized image features f; (3) Input the optimized image feature f and the text feature t into the feature enhancer. The feature enhancer contains 6 feature enhancement layers. Each feature enhancement layer enhances the optimized image feature f through a deformable self-attention mechanism and enhances the text feature t through a normal self-attention mechanism, and outputs the image enhancement feature. Text Enhancement Features ; (4) Enhance image features Text Enhancement Features The input is dynamically selected from the Top-K module. First, the confidence scores and bounding box coordinates of candidate feature proposals are extracted. Then, the proposals are sorted according to their confidence scores. Combined with the intersection-union ratio (IUU) diversity constraint, the top k most diverse candidate feature proposals are selected, and the high-quality feature proposals are output. ; (5) Propose high-quality features Text Enhancement Features The input is a cross-modal decoder, which generates a comprehensive feature through an intra-decoder attention mechanism. The comprehensive feature is processed by a 5-layer feedforward neural network to output a feature tgt. The flattened feature tgt is then input into a gating network to generate a gating score gate_logits. The gating score gate_logits is normalized by Softmax to obtain a gating probability gate_scores. A Top-k selection mechanism is used to select k suitable experts from a pre-defined expert network. The suitable experts process the tokens corresponding to the feature tgt and output the expert processing results. The expert processing results are weighted and accumulated by combining the gating probabilities gate_scores to obtain the comprehensive processing result Output. The comprehensive processing result Output is residually connected to the feature tgt and normalized to output the optimized feature tgt'. (6) Cross-modal decoder calculates image enhancement features Text Enhancement Features The similarity matrix is ​​used to filter high-confidence regions and output the estimated number of target instances. (7) Input the zero-sample target count dataset into the above model, train the model using the Adam optimizer, and set the number of expert networks to 3, the k value in the Top-k selection mechanism to 2, the training batch size to 8, and the learning rate to . The training iterations are 10 times. After the model loss function curve converges and stabilizes, the model weight file is generated. (8) Embed the model weight file into a hardware device containing GPU and CPU, complete the device deployment, and realize the detection and counting of any target in any image.

[0006] Furthermore, the visual enhancement operation in step (2) includes the following sub-steps: (2-1) Select any initial image feature as the feature to be processed, perform feature normalization on the feature to be processed, and output the normalized feature. ; (2-2) Embedding from learnable hints Visual cues are extracted, averaged, and then expanded to the same spatial dimension as the features to be processed, generating spatial cue features. Learnable hints embedded The dimension is ,in To visually indicate depth, Quantity for visual cues; (2-3) Normalized features are processed by 1×1 convolution. Spatial cue features Perform feature projection to reduce the dimensions of both features to one-quarter of their original values. After normalization, output the projected features separately. With projection prompts Projection features With projection prompts All dimensions are 64×H / 64×W / 64; (2-4) Normalized features The input channel attention mechanism first extracts global feature information through adaptive global average pooling and two 1×1 convolutions, then compresses the global information to the range [0,1] using the Sigmoid activation function to generate channel attention weights AW. The channel attention weights AW are multiplied channel by channel by the projected feature fᵢ to output the modulation feature. ; (2-5) Set the learnable fusion weight 'a', with an initial value of 0.1. The modulated features are then processed using the learnable fusion weight 'a'. Weight the values ​​and then combine them with the normalized features. Perform residual connections to output a single-layer enhanced feature; (2-6) Repeat steps (2-1) to (2-5) to perform the same operation on all initial image feature layers, integrate all single-layer enhancement features, and obtain the optimized image feature f.

[0007] Further, the implementation of the intersection-union ratio diversity constraint in step (4) is as follows: the intersection-union ratio of any two candidate feature proposals is calculated using the formula IoU(A,B)=Areaof intersection / Area of ​​union, where A and B are the bounding boxes of the two candidate feature proposals, respectively; the intersection-union ratio threshold is set to 0.7 and the score-weighted threshold is set to 0.1. When the intersection-union ratio of a candidate feature proposal with the selected high-quality feature proposals is less than 0.7, and the confidence score of the candidate feature proposal is higher than the sum of the average confidence score of the selected high-quality feature proposals and the score-weighted threshold, the candidate feature proposal is included in the high-quality feature proposals. .

[0008] Furthermore, in step (5), the gated network consists of two fully connected layers and a ReLU activation function, and the gate score gate_logits is obtained through the formula The calculation yields the result, where G() is the gating network mapping function and Flat() is the feature flattening operation. The temperature parameter is used; the expert network is an independent feedforward neural network, and the overall processing result Output is obtained through a formula. The calculation yields the result, where top_k_scores represents the gating probability corresponding to the fitting expert. This represents the processing result of the i-th matching expert, where the value of i ranges from 1 to k.

[0009] The present invention has the following beneficial effects: (1) The present invention introduces a visual cue tuning model in the image encoder, guides the model to focus on the visual features related to target counting, improves the ability to perceive visual information of the target, effectively alleviates the ambiguity caused by relying solely on text description, and improves counting accuracy.

[0010] (2) By improving the Top-K selection module and introducing the crossover ratio diversity constraint, redundant target feature selection is avoided, the model’s perception of the spatial distribution and diversity of target features is enhanced, and the performance in complex scenarios is improved.

[0011] (3) By introducing a hybrid expert model and an intelligent gating network into the cross-modal decoder, dynamic optimization of feature processing is achieved. The appropriate expert combination is selected based on the characteristics of the input task, thereby improving the model's adaptability and generalization ability. Overall, this invention demonstrates significant advantages in accuracy, adaptability, and flexibility, better addressing various challenges of zero-sample target counting tasks and providing an efficient and accurate solution. Attached Figure Description

[0012] Figure 1 This is an overall flowchart of the present invention; Figure 2 The diagram shows the structure of a zero-shot target counting method based on grounding DINO-based improved visual enhancement and expert collaboration. Figure 3 A schematic diagram of the VPT module structure for embedding the original image encoder; Figure 4 This is a diagram of the overall structure of the MoE model; Figure 5 This method is compared with existing zero-sample target counting models using the public dataset FCS-147; Figure 6 For the visualization comparison of the prediction results of this method and CountGD, column (a) is the image to be counted, column (b) is the ground truth image, column (c) is the visualization of the counting results of CountGD, and column (d) is the visualization of the counting results of this invention. Detailed Implementation

[0013] The present invention will be further described in detail below with reference to the accompanying drawings.

[0014] The overall flowchart of the method proposed in this paper is as follows: Figure 1 As shown in the figure. The structure diagram of the zero-shot target counting network model based on Grounding DINO-improved visual enhancement and expert collaboration is shown below. Figure 2 As shown, improvements were made to the image encoder, Select Top-K, and cross-modal decoder in the original Grounding DINO.

[0015] 1. Text data is input into the text encoder and image encoder to obtain text features t.

[0016] 2. VPT is introduced into the feature layer of the image encoder in the Grounding DINO architecture. The specific embedding method and execution flow are as follows: Figure 3 As shown. First, the image input... After passing through the backbone network of the image encoder, four sets of image features of different sizes are obtained, namely: , , and. We embed VPT into each layer of image features and perform the same visual enhancement operation on each layer of image features. The following explanation uses the fourth layer of features as an example.

[0017] First, we will input image features By performing feature normalization to ensure model stability and convergence speed, normalized features are obtained. Next, we introduce a visual enhancement method (VPT) to guide the model to focus on key visual features relevant to the target counting task. VPT utilizes learnable visual cues to dynamically adjust the region of interest for image features, thereby further enhancing the model's ability to perceive targets. Specifically, we generate spatial features from learnable cues. This process is achieved through the following steps: We embed learnable cueing... Extract visual cues from, among which Depth as a visual cue The number of visual cues. First, they are averaged and then expanded to reflect the features of the input image. Same spatial dimensions The formula is as follows: (1) Next, regarding image features and visual cues Feature projection is performed using 1×1 convolutions, reducing the dimension to 1 / 4 of the original while maintaining the original feature size. This is then normalized to obtain... This process can be represented as: (2) Meanwhile, in order to enhance the expressive power of image features, image features The model employs a channel attention mechanism to adaptively adjust the weights of each channel. Through adaptive global average pooling and two 1×1 convolution operations, the model first extracts global information from the features of each channel and learns the attention weights for each channel. Next, the sigmoid function is used to compress the attention weights to the range [0,1], representing the importance of each channel. Finally, the attention weights generated by the channel attention mechanism are compared with the enhanced features. Channel-by-channel multiplication produces adjusted image features, enhancing useful information relevant to the target counting task and suppressing irrelevant features, thereby improving the model's perception and feature representation capabilities for the target counting task. This process is represented as: (3) in, The calculated channel attention weights represent the importance of each channel. It is the Sigmoid activation function. These are image features that have undergone channel attention modulation.

[0018] Finally, attention-modulated features are applied through learnable fusion weights. Attention modulation features Weighting is performed and compared with the original image features. Perform residual connections to obtain the final output features. .

[0019] (4) in, These are learnable fusion weights, initially set to 0.1, which automatically adjust the weight allocation for augmented and original features in the residual connections. In this way, the model can comprehensively utilize both original and visually augmented features, improving its adaptability and accuracy for target counting tasks.

[0020] After each layer of image features undergoes visual enhancement processing, the model integrates these enhanced features into the final features. This data is then fed into the next processing stage for further feature fusion and task optimization.

[0021] 3. Obtained image features and text features The module then moves to the feature enhancer. It consists of six feature enhancement layers. Each layer enhances image features using a deformable self-attention mechanism and text features using a standard self-attention mechanism. The final output is the image feature model. and text feature output As subsequent input.

[0022] 4. Image feature output of the feature enhancer and text feature output As input to the Dynamic SelectTop-K module, the module first... and We extract the confidence scores and bounding box coordinates of candidate feature proposals. Then, we sort the proposals according to their confidence scores and select the top ones. The most diverse proposals. To account for spatial diversity, we evaluate the spatial overlap of proposals by calculating the Intersection over Union (IoU) between them. Specifically, the formula for calculating IoU is: (5) Here, A and B are two bounding boxes. We ensure sufficient spatial separation between the selected proposals and avoid redundant selections by calculating the IoU between the proposal and the selected proposal.

[0023] During the selection process, we achieve dynamic adjustment by setting an IoU threshold (0.7 in this method) and a score weighting mechanism. Specifically, a proposal will be selected only when the IoU between the proposal and the selected proposals is less than the IoU threshold and higher than the average confidence of the selected proposals plus a score weighting threshold of 0.1.

[0024] Finally, the image features obtained through the Dynamic Select Top-K module are: This serves as the input for subsequent decoding. This module not only effectively reduces the selection of redundant proposals but also enhances the object detection's ability to perceive different regions and targets, thereby improving the model's accuracy in complex scenes.

[0025] 5. First, as Figure 4 As shown, the input to the cross-modal decoder is image features. and text features These input features interact between the image and text through an attention mechanism in the cross-modal decoder, generating a comprehensive feature representation. Next, the resulting comprehensive features undergo nonlinear transformation and further feature processing via a 5-layer feedforward neural network (FFN), ultimately generating new features. ,in It is the number of queries. It is a feature dimension, which serves as the input to the Top-k routing mechanism.

[0026] Input features After being flattened, the sample passes through a gating network, which consists of two fully connected layers (linear layers) and an activation function (ReLU) to generate a gating score. , representing the probability of each expert's choice. The output dimension of the gating network is equal to the number of experts. Furthermore, we introduce a temperature parameter. This is used to control the sharpness of the gating output and adjust the distribution of expert selection. At higher temperatures, the selection distribution is more uniform; at lower temperatures, the model tends to select a smaller number of experts. This process is represented as: (6) Gating score The probability distribution will be normalized using the Softmax function. Let represent the probability of each expert's choice. Then, a Top-k selection mechanism is used to select the expert with the highest probability score. Each expert is selected to process the input. The weight of each selected expert is then obtained. and the index of selected experts The formula is as follows: (7) (8) After the Top-K routing phase, the model enters the expert processing phase. For example... Figure 4 As shown, each expert is an independent feedforward neural network (FFN). At this stage, firstly through... Select the appropriate token to ensure that each selected expert processes only the specified subset of input features. Next, each selected expert network It will process the corresponding tokens and generate their respective outputs. Finally, all expert outputs will be based on... The weighted cumulative formula is as follows: (9) (10) in, For the corresponding expert's output, The output is a weighted cumulative sum of all experts' inputs.

[0027] Finally, the weighted output of the experts Will be with the original input Residual connections are performed and normalized to enhance the model's feature representation and achieve faster convergence, avoiding gradient vanishing or exploding problems. The calculation process is as follows: (11) The above process ensures that the model, through the MoE mechanism, combines the independent computations and weighted combinations of multiple experts, thereby reducing the computational burden while improving the model's expressive power and adaptability. By flexibly weighting and combining expert outputs, the model can effectively integrate the knowledge of various experts based on the differences in input features, thus becoming more efficient and flexible in handling complex tasks.

[0028] 6. Next, the cross-modal decoder calculates the similarity matrix between image features and text features, and filters out high-confidence regions to obtain the final estimated number of target instances in the output image.

[0029] 7. Input the zero-shot target counting dataset into the zero-shot target model with visual enhancement and expert collaboration, and optimize the model using the Adam optimizer. Set the number of experts to 3, K to 2, Batch size to 8, learning rate to 1×10−4, and the corresponding number of training iterations to 10. As the number of training iterations increases, the model's loss function curve gradually converges. When the loss function curve converges and stabilizes, the model is trained to its optimal state, and the corresponding weight file is generated.

[0030] 8. By embedding the trained weights into a device with a GPU and a CPU, it is possible to detect any target in any image.

[0031] To demonstrate the effectiveness of our method, we conducted experiments comparing it with currently popular zero-shot target counting methods on the public dataset FSC-147. The comparison results for MAE and RMSE are shown below. Figure 5 The results show that the method of the present invention has the highest MAE and also has a good RMSE.

[0032] Detailed comparison results chart, such as Figure 6 The images shown encompass different target categories and sparsity levels, aiming to demonstrate the superior adaptability and accuracy of the proposed method in diverse scenarios. State-of-the-art methods exhibit varying degrees of false negatives and false positives when detecting occluded and small targets. Our method, however, correctly detects and counts these targets, with the differences in detection highlighted in gold boxes.

[0033] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A zero-shot target counting method based on visual enhancement and expert collaboration, characterized in that, Includes the following steps: (1) Input the text data into the text encoder and output the text feature t; input the image data into the image encoder, and after processing by the encoder backbone network, output four sets of initial image features of different sizes. The sizes of the four sets of initial image features are H / 8×W / 8×256, H / 16×W / 16×256, H / 32×W / 32×256, and H / 64×W / 64×256, respectively. (2) Embed a visual cue tuning model in each layer of the initial image features of the image encoder, perform visual enhancement operations on each layer of the initial image features, and integrate the enhanced features of all layers to obtain the optimized image features f; (3) Input the optimized image feature f and the text feature t into the feature enhancer. The feature enhancer contains 6 feature enhancement layers. Each feature enhancement layer enhances the optimized image feature f through a deformable self-attention mechanism and enhances the text feature t through a normal self-attention mechanism, and outputs the image enhancement feature. Text Enhancement Features ; (4) Enhance image features Text Enhancement Features The input is dynamically selected from the Top-K module. First, the confidence scores and bounding box coordinates of candidate feature proposals are extracted. Then, the proposals are sorted according to their confidence scores. Combined with the intersection-union ratio (IUU) diversity constraint, the top k most diverse candidate feature proposals are selected, and the high-quality feature proposals are output. ; (5) Propose high-quality features Text Enhancement Features The input to the cross-modal decoder generates a comprehensive feature through an in-decoder attention mechanism. This comprehensive feature is then processed by a 5-layer feedforward neural network to output feature tgt. The flattened feature tgt is input to a gating network to generate gate scores (gate_logits). These gate scores (gate_logits) are normalized using Softmax to obtain gate probabilities (gate_scores). A Top-k selection mechanism is used to select k suitable experts from a pre-defined expert network. Each expert processes the token corresponding to feature tgt and outputs the expert processing results. These expert processing results are then weighted and accumulated using the gate probabilities (gate_scores) to obtain the comprehensive processing result, Output. Output is then regressed and normalized with feature tgt to output the optimized feature. ; (6) Cross-modal decoder calculates image enhancement features Text Enhancement Features The similarity matrix is ​​used to filter high-confidence regions and output the estimated number of target instances. (7) Input the zero-sample target count dataset into the above model, train the model using the Adam optimizer, and generate the model weight file after the model loss function curve converges and stabilizes. (8) Embed the model weight file into a hardware device containing GPU and CPU, complete the device deployment, and realize the detection and counting of any target in any image.

2. The zero-shot target counting method based on visual enhancement and expert collaboration according to claim 1, characterized in that, The visual enhancement operation in step (2) includes the following sub-steps: (2-1) Select any initial image feature as the feature to be processed, perform feature normalization on the feature to be processed, and output the normalized feature. ; (2-2) Embedding from learnable hints Visual cues are extracted, averaged, and then expanded to the same spatial dimension as the features to be processed, generating spatial cue features. ; Learnable hints embedding The dimension is pd×np×256, where To visually indicate depth, Quantity for visual cues; (2-3) Normalized features are processed by 1×1 convolution. Spatial cue features Perform feature projection to reduce the dimensions of both features to one-quarter of their original values. After normalization, output the projected features separately. With projection prompts Projection features With projection prompts All dimensions are 64×H / 64×W / 64; (2-4) Normalized features The input channel attention mechanism first extracts global feature information through adaptive global average pooling and two 1×1 convolutions, then compresses the global information to the [0,1] range using the Sigmoid activation function to generate channel attention weights AW; the channel attention weights AW and the projected features... Channel-by-channel multiplication to output modulation characteristics ; (2-5) Set the learnable fusion weight 'a', with an initial value of 0.

1. The modulated features are then processed using the learnable fusion weight 'a'. Weight the values ​​and then combine them with the normalized features. Perform residual connections to output a single-layer enhanced feature; (2-6) Repeat steps (2-1) to (2-5) to perform the same operation on all initial image feature layers, integrate all single-layer enhancement features, and obtain the optimized image feature f.

3. The zero-sample target counting method based on visual enhancement and expert collaboration according to claim 1, characterized in that, The implementation of the intersection-union ratio diversity constraint in step (4) is as follows: the intersection-union ratio of any two candidate feature proposals is calculated using the formula IoU (A,B)=Area of ​​intersection / Area of ​​union, where A and B are the bounding boxes of the two candidate feature proposals, respectively; and when the confidence score of the candidate feature proposal is higher than the sum of the average confidence score of the selected high-quality feature proposals and the score-weighted threshold, the candidate feature proposal is included in the high-quality feature proposals. .

4. The zero-sample target counting method based on visual enhancement and expert collaboration according to claim 1, characterized in that, In step (5), the gating network consists of two fully connected layers and a ReLU activation function, with a gating score. Through formula The calculation yields the result, where G() is the gated network mapping function and Flat() is the feature flattening operation. The temperature parameter is used; the expert network is an independent feedforward neural network, and the overall processing result Output is obtained through a formula. The calculation yielded, where To adapt to the gating probabilities corresponding to the experts, This represents the processing result of the i-th matching expert, where the value of i ranges from 1 to k.

Images