Text-guided zero-sample target counting method, program product, and electronic device

Abstract

The present invention belongs to the field of image processing technology and provides a text-guided zero-shot target counting method, a program product and an electronic device. The counting method comprises: inputting a query image and a target description text into a target counting model to obtain a density estimation map of the target; the target counting model comprises: a visual feature extraction module; a text feature extraction module; a multi-level polarity dual attention module comprising a plurality of cascaded feature fusion modules; each feature fusion module performs visual polarity cross-attention processing on visual-to-text fusion features or visual features to obtain visual-to-text fusion features of this level, and performs text polarity cross-attention processing on text-to-visual fusion features or text features to obtain text-to-visual fusion features of this level; a decoder decodes the visual-to-text fusion features to obtain a density estimation map of the target; the present invention retains negative polarity part information, thereby improving the accuracy of the density estimation map of the target.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a text-guided zero-sample target counting method, a program product, and an electronic device. Background Art

[0002] Object counting, a crucial task in computer vision, has broad application prospects in fields such as crowds, agriculture, transportation, and healthcare. Zero-shot object counting algorithms (also known as class-agnostic object counting methods) aim to construct an example-free object counter that allows for the specification of the objects to be counted. They assume that the model has never seen samples of the target category during training, but by leveraging the model's prior knowledge and representational capabilities, they can infer and count objects of new categories. Zero-shot object counting algorithms typically utilize the semantics and feature representations learned by the model during training, combined with descriptive information or attribute information about the target category, to estimate the count of objects of new categories.

[0003] In related technologies, Jiang et al. proposed CLIP-Count, a counting method that leverages a pre-trained vision-language model. This method can simultaneously understand images and related text descriptions through the pre-trained vision-language model. Furthermore, CLIP-Count introduces a patch-text contrast loss to guide the model in learning information-rich patch-level visual representations. This allows the model to better understand different objects in an image and generate more accurate density predictions, enabling end-to-end density map prediction based on multimodality (text and vision) in a zero-shot manner.

[0004] In multimodal learning, efficiently fusing information from different modalities (such as vision and text) is crucial for improving model performance. Traditional cross-attention mechanisms achieve inter-modal interaction by calculating attention weights between the query of one modality and the key and value of another modality. However, these methods are often limited in capturing the complex, nonlinear relationships between modalities, especially when dealing with negative polarity information and modality-specific information, resulting in inaccurate target density estimation. Summary of the Invention

[0005] The present application aims to at least solve the technical problems existing in the prior art and provide a text-guided zero-sample target counting method, program product and electronic device.

[0006] In the first aspect, the present application provides a text-guided zero-shot target counting method, including: obtaining a query image and a target description text, inputting the query image and the target description text into a trained target counting model, and the target counting model outputting a density estimation map of the target; the target counting model includes: a visual feature extraction module, used to obtain the visual features of the query image; a text feature extraction module, used to obtain the text features of the target description text; a multi-level polarity dual attention module, including a cascade of multiple levels of feature fusion modules; wherein the feature fusion module of each level in at least one level performs visual polarity cross-attention processing on the visual-to-text fusion features or visual features output by the feature fusion module of the previous level to obtain the visual-to-text fusion features of the current level, and performs text polarity cross-attention processing on the text-to-visual fusion features or text features output by the feature fusion module of the previous level to obtain the text-to-visual fusion features of the current level; a decoder, which decodes the visual-to-text fusion features output by the feature fusion module of at least one level to obtain a density estimation map of the target.

[0007] In a second aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method provided in the first aspect of the present application.

[0008] In a third aspect, the present application provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute a text-guided zero-sample target counting method provided in the first aspect of the present application.

[0009] The beneficial technical effects of the present invention are as follows: the visual features of the query image are obtained by the visual feature extraction module, the text features of the target description text are obtained by the text feature extraction module, and the multi-level polarity dual attention module uses visual polarity cross-attention processing and text polarity cross-attention processing to process the visual features and text features respectively, which can establish a fine-grained bidirectional association between text features and visual features, ensuring that information is not only transmitted from text to vision, but also from vision to text feedback, enhancing the complementarity and consistency of features, and solving the unidirectional limitation of the traditional cross-attention mechanism in the information transmission process. At the same time, the positive and negative polarity part information of the query vector and the key vector are separated in the visual polarity cross-attention processing and the text polarity cross-attention processing, and the same polarity similarity and opposite polarity similarity are calculated, which retains the negative polarity part information, improves the expression ability of the cross-modal relationship of the visual to text fusion features, and thus improves the accuracy of the density estimation map of the target obtained by the decoder. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a schematic diagram of the structure of a target counting model in a preferred embodiment of the present invention;

[0011] Figure 2 This is a schematic structural diagram of a target counting model in another preferred embodiment of the present invention;

[0012] Figure 3 2 is a schematic diagram of the connection relationship and internal structure of the visual feature extraction module and the decoder in another preferred embodiment of the present invention;

[0013] Figure 4 1 is a schematic diagram of the process of visual polarity cross-attention processing and text polarity cross-attention processing in another preferred embodiment of the present invention;

[0014] Figure 5 is a schematic diagram of fine-tuning a text encoder in another preferred embodiment of the present invention;

[0015] Figure 6 2 is a schematic diagram of the latent space for alignment embedding in contrastive learning in another preferred embodiment of the present invention;

[0016] Figure 7 4 is a qualitative comparison result diagram of the target counting model of the present invention and the existing CLIP-Count model in another preferred embodiment of the present invention;

[0017] Figure 8 It is a structural diagram of an electronic device in another preferred embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0019] The execution subject of a text-guided zero-sample target counting method provided by the present invention includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, a text-guided zero-sample target counting method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0020] The present invention provides a text-guided zero-shot target counting method. In a preferred embodiment, the method includes:

[0021] Step S1, obtain the query image and target description text. The target description text is a text including the target category. The target category can be a living object or an inanimate object, such as a cell, a person, an animal, a cup, a fruit, a grape, a pill, etc. For example, Figure 8 As shown, when the number of cups needs to be counted from a query image, the target description text can be "cups".

[0022] Step S2: Input the query image and target description text into the trained target counting model.

[0023] In step S3, the target counting model outputs a density estimation map of the target, i.e., a density estimation heatmap. The pixel value of each pixel in the density estimation heatmap represents whether the target exists at that pixel. For example, the geometric center point of each target in the density estimation heatmap is highlighted.

[0024] Please see the attached Figure 1 and 2 , target counting models include:

[0025] The visual feature extraction module is used to obtain the visual features of the query image.

[0026] Exemplarily, the visual feature extraction module includes a visual encoder, which can employ a CLIP visual encoder. To obtain more efficient visual features and train on any image collection without requiring associated metadata, the visual encoder preferably employs the DINOv2 visual encoder. Specifically, the image block-level feature maps output by the DINOv2 visual encoder are used as visual features.

[0027] The text feature extraction module is used to obtain the text features of the target description text.

[0028] Exemplarily, the text feature extraction module can be a BERT encoder or a CLIP text encoder. Preferably, the text feature extraction module uses the CLIP text encoder, which connects images and natural language by performing large-scale pre-training on image-text pair data, making it easier to align with visual features.

[0029] A multi-level polarity dual attention module includes multiple cascaded levels of feature fusion modules; wherein, the feature fusion module of each level in at least one level performs visual polarity cross-attention processing on the visual-to-text fusion features or visual features output by the feature fusion module of the previous level to obtain the visual-to-text fusion features of the current level, and performs text polarity cross-attention processing on the text-to-visual fusion features or text features output by the feature fusion module of the previous level to obtain the text-to-visual fusion features of the current level.

[0030] In this embodiment, at least one feature fusion module has a text input terminal, a text output terminal, a visual input terminal and a visual output terminal. Text features are input to the text input terminal of the feature fusion module of the first level, visual features are input to the visual input terminal of the feature fusion module of the first level, the text output terminal of the feature fusion module of the first level outputs text-to-visual fusion features to the text input terminal of the feature fusion module of the second level, the visual output terminal of the feature fusion module of the first level outputs visual-to-text fusion features to the visual input terminal of the feature fusion module of the second level, and feature fusion modules of multiple levels are connected in this way. For feature fusion modules of adjacent levels, the text output terminal of the feature fusion module of the previous level is correspondingly connected to the text input terminal of the feature fusion module of the next level, and the visual output terminal of the feature fusion module of the previous level is correspondingly connected to the visual input terminal of the feature fusion module of the next level.

[0031] The decoder decodes the visual-to-text fusion features output by the feature fusion module of at least one level to obtain a density estimation map of the target.

[0032] Exemplarily, the decoder is an existing CNN network-based decoder. It can decode the visual-to-text fusion features output by the feature fusion module of the last level to obtain the density estimation map of the target. It can also decode the comprehensive fusion features after the visual-to-text fusion features output by the feature fusion module of the last level and other (non-last) levels to obtain the density estimation map of the target. Figure 3The decoder structure on the right can also connect the visual-to-text fusion features output by the feature fusion modules of the last level and other (non-last) levels to each node of the decoder network in the form of residuals to obtain the density estimation map of the target.

[0033] In this embodiment, preferably, the feature fusion module of each level in at least one level includes:

[0034] The visual-to-text polarity cross-attention unit performs visual polarity cross-attention processing on the visual features or the visual-to-text fusion features output by the feature fusion module of the previous level to obtain the visual-to-text attention features of this level. For the feature fusion module of the first level, its visual-to-text polarity cross-attention unit performs visual polarity cross-attention processing on the visual features. For feature fusion modules other than the first level, its visual-to-text polarity cross-attention unit performs visual polarity cross-attention processing on the visual-to-text fusion features output by the feature fusion module of the previous level.

[0035] The text-to-visual polarity cross-attention unit performs text-polarity cross-attention processing on text features or the text-to-visual fusion features output by the feature fusion module of the previous level to obtain the text-to-visual attention features of this level. For the feature fusion module of the first level, its text-to-visual polarity cross-attention unit performs text-polarity cross-attention processing on the text features. For feature fusion modules other than the first level, its text-to-visual polarity cross-attention unit performs text-polarity cross-attention processing on the text-to-visual fusion features output by the feature fusion module of the previous level.

[0036] The first fusion unit performs a first fusion process on the visual-to-text attention feature and the text-to-visual attention feature to obtain a visual-to-text fusion feature.

[0037] The second fusion unit performs a second fusion process on the visual-to-text attention feature and the text-to-visual attention feature to obtain a text-to-visual fusion feature.

[0038] In this implementation, see Figure 3 The visual-to-text polarity cross attention unit and the text-to-visual polarity cross attention unit are collectively represented by PDCA (Pola Dual Cross Attention). The first fusion unit and the second fusion unit are feed-forward neural networks (FNNs) with two or more layers. Specifically, the first fusion unit or the second fusion unit can be a cascaded fully connected layer with two or more layers.

[0039] In this implementation, see Figure 4 ,The process of the visual to text polar cross attention unit performing visual polar cross attention processing includes:

[0040] In step A1, the visual features or the visual-to-text fusion features output by the feature fusion module of the previous level are used as the visual query vector vis_q, and the text features or the text-to-visual fusion features output by the feature fusion module of the previous level are used as the visual key vector vis_k and the visual value vector vis_v.

[0041] Step A2: Scale the visual query vector vis_q and the visual value vector vis_v using a first feature scaling factor scale1. Preferably, scale1 is a learned parameter learned during object counting model training to dynamically adjust feature amplitudes. Preferably, the visual query vector vis_q and the visual value vector vis_v are first multi-head reshaped before the amplitude adjustment is performed.

[0042]

[0043] Step A3: Perform positive and negative polarity decomposition and enhancement processing on the amplitude-adjusted visual query vector vis_q' to obtain the first query positive polarity enhancement feature Q1 + And the first query negative polarity enhancement feature Q1 - ; Perform positive and negative polarity decomposition and enhancement processing on the visual key vector vis_k to obtain the first key positive polarity enhancement feature K1 + and the first bond negative polarity enhancement feature K1 - .

[0044] In this embodiment, the decomposition of the positive and negative polarity of the vector means that the vector itself is regarded as the positive polarity part, and the result of the bitwise inversion operation of the vector (such as multiplying each element by -1) is defined as the negative polarity part of the vector.

[0045] Traditional cross-attention often ignores negative information in query-key pairs due to non-negative feature mapping (such as the ReLU activation function), which limits the model's ability to express complex relationships between modalities. To alleviate this problem, this application decomposes the query and key into positive and negative polarity parts and enhances the positive and negative polarity parts separately.

[0046] Further preferably, a learnable power function is introduced for enhancement processing, and the specific calculation formula is as follows:

[0047] Q1 + =ReLU(vis_q') 1+α1·sigmoid ( p1

[0048] Q1 - =ReLU(-vis_q') 1+α1·sigmoid ( p1

[0049] K1+ =ReLU(vis_k) 1+α1·sigmoid ( p1

[0050] K1 - =ReLU(-vis_k) 1+α1·sigmoid ( p1 )

[0051] Here, -vis_q' is obtained by bitwise inverting vis_q', and -vis_k is obtained by bitwise inverting vis_k. For example, bitwise inversion is expressed as: if a number is 1, then its bitwise inversion is -1. p1 represents the first power exponent parameter, and α1 is the first weight parameter used to control the feature scaling strength. Both p1 and α1 are learnable parameters, learned during target count model training. ReLU(·) represents the ReLU activation function.

[0052] Step A4, by first querying the positive polarity enhancement feature Q1 + and the first bond positive polarity enhancement feature K1 + Calculate the first forward similarity S1 pos ; Enhance feature Q1 by first querying negative polarity - and the first bond negative polarity enhancement feature K1 - Calculate the first negative similarity S1 neg .

[0053] In this embodiment, the first forward similarity S1 can be calculated by Euclidean distance or cosine similarity. pos Or the first negative similarity S1 neg Preferably, the first forward similarity S1 is calculated using the following formula: pos Or the first negative similarity S1 neg :

[0054] S1 pos =(Q1 + K1 +

[0055] S1 neg =(Q1 - K1 - )

[0056] Here, “·” represents the dot product operator.

[0057] Step A5: first forward similarity S1 pos With the first negative similarity S1 neg The difference is processed by activation function to obtain the first attention weight attn1. The activation function is preferably softmax(). Specifically:

[0058] attn1=softmax(S1 pos -S1 neg )

[0059] Step A6: Obtain the visual-to-text attention feature vis at this level based on the first attention weight attn1 and the amplitude-adjusted visual value vector vis_v' context , specifically:

[0060] vis context =Dropout(attn1)·vis_v′.

[0061] Dropout(·) represents a dropout operation, which is used to prevent overfitting during training.

[0062] In this embodiment, preferably, see Figure 4 ,The process of text-to-visual polarity cross attention unit performing text polarity cross attention processing includes:

[0063] Step B1: Use text features or text-to-visual fusion features output by the feature fusion module of the previous level as the text query vector txt_q, and use visual features or visual-to-text fusion features output by the feature fusion module of the previous level as the text key vector txt_k and text value vector txt_v.

[0064] Step B2: Scale the text query vector txt_q and the text value vector txt_v using a second feature scaling factor scale2. Preferably, the second feature scaling factor scale2 is a learned parameter learned during target count model training to dynamically adjust feature amplitudes. Preferably, the text query vector txt_q and the text value vector txt_v are first multi-head reshaped before the amplitude adjustment is performed.

[0065]

[0066] Step B3: Perform positive and negative polarity decomposition and enhancement processing on the amplitude-adjusted text query vector txt_q' to obtain the second query positive polarity enhancement feature Q2 + and the second query negative polarity enhancement feature Q2 - ; Perform positive and negative polarity decomposition and enhancement processing on the text key vector txt_k to obtain the second key positive polarity enhancement feature K2 + and the second bond negative polarity enhancement feature K2 - .

[0067] Traditional cross-attention often ignores negative information in query-key pairs due to non-negative feature mapping (such as ReLU), which limits the model's ability to express complex relationships between modalities. To alleviate this problem, this application decomposes the query and key into positive and negative polarity parts and enhances the positive and negative polarity parts separately.

[0068] Further preferably, a learnable power function is introduced for enhancement processing, and the specific calculation formula is as follows:

[0069] Q2 + =ReLU(txt_q') 1+α2·sigmoid(p2)

[0070] Q2 - =ReLU(-txt_q') 1+α2·sigmoid(p2)

[0071] K2 + =ReLU(txt_k) 1+α2·sigmoid(p2)

[0072] K2 - =ReLU(-txt_k) 1+α2·sigmoid(p2)

[0073] Here, -txt_q' is obtained by bitwise inverting txt_q', and -txt_k is obtained by bitwise inverting txt_k. p2 represents the second power exponent parameter, and α2 is the second weight parameter used to control the feature scaling strength. Both p2 and α2 are learnable parameters and are learned during target count model training.

[0074] Step B4, enhance the feature Q2 by the second query positive polarity + and the second bond positive polarity enhancement feature K2 + Calculate the second forward similarity S2 pos ; Enhance feature Q2 by the second query negative polarity - and the second bond negative polarity enhancement feature K2 - Calculate the second negative similarity S2 neg , preferably:

[0075] S2 pos =(Q2 + K2 + )

[0076] S2 neg =(Q2 - K2 - )

[0077] Step B5: For the second forward similarity S2 pos With the second negative similarity S2 negThe difference is processed by activation function to obtain the second attention weight attn2. The activation function is preferably softmax(). Specifically:

[0078] attn2=softmax(S2 pos -S2 neg ).

[0079] Step B6: Obtain the text-to-visual attention feature txt of this level based on the second attention weight attn2 and the amplitude-adjusted text value vector txt_v' context :

[0080] txt context =Dropout(attn2)·txt_v'.

[0081] In this embodiment, a polarity dual cross-attention mechanism is implemented through the visual polarity cross-attention processing of steps A1-A6 and the text polarity cross-attention processing of steps B1-B6. The PDCA unit composed of the visual-to-text polarity cross-attention unit and the text-to-visual polarity cross-attention unit can establish fine-grained bidirectional associations between text and visual features, significantly enhancing the fusion and understanding of multimodal information. This dual information flow allows text and visual features to interact in two directions, ensuring that information is not only transmitted from text to visual but also fed back from visual to text, enhancing the complementarity and consistency of features and addressing the unidirectional limitation of the traditional cross-attention mechanism in the information transmission process. At the same time, by separating the positive and negative components of the query and key and calculating the same polarity and opposite polarity similarities, negative value information is retained, improving the expressiveness of cross-modal relationships. The use of a power function and setting learnable p1, p2 and α1, α2 to implement a dynamic scaling mechanism reduces the entropy of the attention distribution, improves the distinction between strong and weak signals, and helps improve target counting accuracy.

[0082] In a preferred embodiment, see the attached Figure 3 , the feature fusion module of each layer in at least one layer also includes a visual multi-head self-attention unit ( Figure 3 MHSA on the right side of (a)) and the text multi-head self-attention unit ( Figure 3 MHSA on the left in (a).

[0083] The visual multi-head self-attention unit performs multi-head self-attention processing on the visual features or the visual-to-text fusion features output by the feature fusion module of the previous level to obtain visual self-attention features. Specifically, for the first-level feature fusion module, its visual multi-head self-attention unit performs multi-head self-attention processing on the visual features. For feature fusion modules other than the first level, its visual multi-head self-attention unit performs multi-head self-attention processing on the visual-to-text fusion features output by the feature fusion module of the previous level.

[0084] At this time, the visual-to-text polarity cross-attention unit performs visual polarity cross-attention processing on the visual self-attention feature to obtain the visual-to-text attention feature of this level.

[0085] The text multi-head self-attention unit performs multi-head self-attention processing on text features or the text-to-visual fusion features output by the feature fusion module of the previous level to obtain text self-attention features. Specifically, for the first-level feature fusion module, its text multi-head self-attention unit performs multi-head self-attention processing on text features. For feature fusion modules other than the first level, its text multi-head self-attention unit performs multi-head self-attention processing on the text-to-visual fusion features output by the feature fusion module of the previous level.

[0086] At this time, the text-to-visual polarity cross-attention unit performs text polarity cross-attention processing on the text self-attention features to obtain the text-to-visual attention features of this level.

[0087] In this embodiment, visual multi-head self-attention units and text multi-head self-attention units are used to reduce the risk of overfitting, improve the expressiveness of features, and speed up computational efficiency.

[0088] In this embodiment, the process of performing visual polarity cross attention processing on the visual self-attention feature by the visual-to-text polarity cross attention unit is similar to steps A1-A6 of the above preferred embodiment, except that step A1 is replaced by the following steps, and steps A2-A6 remain unchanged:

[0089] The new step A1 uses the visual self-attention features as the visual query vector vis_q, and the textual self-attention features as the visual key vector vis_k and the visual value vector vis_v.

[0090] In this embodiment, the process of the text-to-visual polarity cross-attention unit performing text polarity cross-attention processing on the text self-attention feature is similar to steps B1-B6 of the above preferred embodiment, except that step B1 is replaced by the following steps, and steps B2-B6 remain unchanged:

[0091] The new step B1 uses the text self-attention features as the text query vector txt_q, and the visual self-attention features as the text key vector txt_k and the text value vector txt_v.

[0092] In a preferred embodiment, see the attached Figure 3 A convolution upsampling module (Conv&2xup) is also connected between the feature fusion modules of two adjacent layers. The convolution upsampling module performs convolution and upsampling on the visual-to-text fusion features output by the feature fusion module of the previous layer in the two adjacent layers, and uses the processing results as the input of the feature fusion module of the next layer in the two adjacent layers.

[0093] In this embodiment, the convolution upsampling module includes a convolution unit and an upsampling unit connected in sequence. The convolution unit has a jump connection for further processing the visual to text fusion features output by the feature fusion module of the previous level and capturing local spatial information. The upsampling unit is preferably a 2× bilinear interpolation layer (i.e., 2x upsampling), which is used to double the resolution of the feature map output by the convolution unit. The motivation for such a design is to capture the relationship between text and image in a more fine-grained manner, thereby enhancing the complementarity and consistency of features.

[0094] In this embodiment, any convolutional upsampling module is used to perform convolution and upsampling processing on the visual-to-text fusion features output by the feature fusion module of the previous level, and input the processing results into the visual multi-head self-attention unit or the visual-to-text polarity cross-attention unit of the feature fusion module at the next level.

[0095] In a preferred embodiment, see the attached Figure 2 ,The visual feature extraction module includes a visual encoder that processes the query ,image to obtain visual embeddings; a visual adapter that converts the visual embeddings to ,specified dimensions to obtain visual features.

[0096] And / or, the text feature extraction module includes: a text encoder, which processes the target description text to obtain text embedding; and a text adapter, which converts the text embedding to a specified dimension to obtain text features.

[0097] In this implementation, because the visual encoder (e.g., DINOv2) and text encoder (e.g., CLIP) are pre-trained using different training methods on datasets with different distributions, the image patch embeddings extracted by the visual encoder and the text embeddings extracted by the text encoder may be located in different feature latent spaces. Therefore, a visual adapter and a text adapter are set to transform the visual embeddings and text embeddings to the same specified dimension, respectively, to facilitate subsequent comparison, fusion, and other operations.

[0098] In this embodiment, preferably, both the visual adapter and the text adapter use a multi-layer perceptron (MLP). For example, if the dimension of the visual embedding input to the visual adapter is 768, the dimension of the visual features output by the visual adapter is 512. If the dimension of the text embedding input to the text adapter is 512, the dimension of the text features output by the text adapter is also 512.

[0099] In this embodiment, the visual adapter Av and the text adapter At are expressed as:

[0100]

[0101] Among them, ε I represents visual embedding, ε t represents text embedding, d i represents the visual embedding dimension, d t represents the text embedding dimension, d e Indicates setting dimension.

[0102] In a preferred embodiment, see the attached Figure 3 (b) The decoder includes a plurality of cascaded decoding convolutional upsampling modules and an output convolutional layer, and also includes one or more adding units, which are respectively connected in series between one or more adjacent decoding convolutional upsampling modules, and each adding unit corresponds to a feature fusion module of a level; specifically, the first adding unit corresponds to the feature fusion module of the second level, the second adding unit corresponds to the feature fusion module of the third level, and so on.

[0103] The first-level feature fusion module outputs the visual-to-text fusion feature to the input of the first decoding convolutional upsampling module;

[0104] The addition unit is used to add the output features of the previous decoding convolution upsampling module of the addition unit and the visual to text fusion features output by the feature fusion module of the corresponding level, and input the addition result into the next decoding convolution upsampling module of the addition unit.

[0105] In this embodiment, the decoding convolution upsampling module includes a decoding convolution and a decoding upsampling layer connected in sequence. The decoding upsampling layer is preferably a 2× interpolation layer. The convolution kernel of the decoding convolution is preferably 3×3, and the convolution kernel of the output convolution layer is preferably 1×1. The output convolution layer typically has a sigmoid activation function. Each layer of the decoding convolution upsampling module doubles the spatial resolution and halves the dimensionality.

[0106] For example, Figure 3Taking (b) as an example, the multi-level polarity dual attention module includes two layers of feature fusion modules, the first and second layers, with a convolutional upsampling module placed between them. The decoder includes four cascaded decoding convolutional upsampling modules and one output convolutional layer. The decoder has one addition unit located between the first and second decoding convolutional upsampling modules, corresponding to the feature fusion module of the second layer.

[0107] In the above example, in order to simultaneously decode feature maps at two different scales, the decoder first uses the first decoding convolution upsampling module to fused the coarse-grained visual-to-text feature M output by the feature fusion module of the first level. c Perform convolution and 2× interpolation (the first decoding convolution upsampling module of the decoder) to make its dimension the same as the fine-grained visual to text fusion feature M output by the feature fusion module of the second level f Then, before the second decoding convolution upsampling module, the summation is performed with M f Fusion to obtain M f‘ :

[0108] M f‘ =Lerp(σ(Conv 3×3 (M c )))+σ(Conv 3×3 (M f ))

[0109] Where Lerp(.) represents bilinear interpolation and σ(.) is the activation function. Input M f‘ To the second decoding convolutional upsampling module.

[0110] In this embodiment, the decoder can decode visual-to-text fusion features of different scales, which can improve the accuracy of the density estimation map of the target.

[0111] In a preferred embodiment, the training process of the target counting model includes:

[0112] In the first stage of training, the visual encoder, visual adapter, text encoder, and text adapter are fine-tuned using self-supervised learning. During training, the contrastive learning loss (InfoNCEloss) is used to align the visual features with the text features in the high-dimensional latent space.

[0113] In this embodiment, the visual encoder uses DINOv2, and the image block feature map output by DINOv2 is selected as the visual embedding. The text encoder uses the text encoder of CLIP.

[0114] Constructing a training set Among them, the i-th sample includes a training image A category description text and a binary map In this map, each i Zhongyout i The center of the object being described is set to 1, and other locations are set to 0, i is the sample index. i Sum to count in X i Zhongyout i The number of objects described is as follows, where p and q are the indices of the pixels in the map:

[0115] Count(X i ,t i )=∑ p,q (Y i ) p,q

[0116] Similarly, construct a validation set and test set The training set does not overlap with the validation and test sets:

[0117] In this embodiment, the steps of fine-tuning the text encoder of CLIP are as follows: Figure 5 As shown, we first freeze the network parameters of the existing CLIP text encoder and add multiple learnable hint vectors to the text embedding. Each hint vector is related to the text embedding ε t With the same dimensionality, CLIP's text encoder is trained using a text dataset. During training, the prompt vector interacts with all other vectors in the TransformerBlock, learning a new implicit representation to guide downstream tasks. Finally, features are extracted from the last end vector (EndOfTextToken, where EOT_TOKEN is the maximum value in each sequence) to obtain the final text embedding, completing the fine-tuning of CLIP's text encoder.

[0118] In this embodiment, the steps of fine-tuning the DINOv2 visual encoder are: adding multiple hint vectors to the visual embedding, training the DINOv2 visual encoder using the image set, and the hint vector will participate in the interaction process with all other vectors during training.

[0119] In this embodiment, the first stage of training process is that the visual encoder, visual adapter, text encoder and text adapter form a backbone network, and the backbone network is iteratively trained using the training set until the fine-tuning training stop condition is reached. The image block-text contrast loss is calculated in each training, and the weight of the backbone network is fine-tuned by the image block-text contrast loss. The fine-tuning training stop condition is not limited to the number of training times reaching the preset maximum number of fine-tuning times or the image block-text contrast loss convergence. The trained backbone network is tested on the validation set. Validated on the test set According to the mask value corresponding to the image block in the mapping graph, define and Image patch embedding sets for positive and negative samples, image patch-text contrast loss The calculation formula is as follows:

[0120]

[0121] Among them, w i represents the weight of the i-th positive sample, s(.,.) represents the cosine similarity, and τ=0.07 represents the temperature coefficient. By pulling the text embedding ε t and positive visual embeddings The distance between them pushes away the text embedding ε t Visual Embedding with Negative Samples The distance between them is used to align the image block embedding space with the text embedding space, such as Figure 6 shown.

[0122] In the second phase of training, after fine-tuning the visual feature extraction module and the text feature extraction module, the network parameters of the two are frozen. During the training, the learnable parameters in the multi-level polarity dual attention module and the network parameters of the decoder are updated and adjusted. The second phase of training adopts supervised learning, and the target counting model is iteratively trained using the training set. The binary mapping map Y of each sample is used. i As the true density map of the sample, in each training, for the i-th sample, the density estimation map of the target output by the decoder is used With Mapping Compare pixel by pixel and calculate the mean square error To continue adjusting the parameters of the decoder and the multi-level polarity dual attention module. i By summing pixel by pixel, we can get the estimated number of items of interest.

[0123] The following is an experimental verification of the performance of the target counting model provided in this application:

[0124] Experiments were conducted on the FSC-147, CARPK, and ShanghaiTech datasets. Mean absolute error (MAE) and root mean square error (RMSE) were used as evaluation metrics. The experimental comparison models were compared with several state-of-the-art class-agnostic object counting methods under few-shot, no-reference, and zero-shot conditions, including FamNet, CFOCNet, CounTR, LOCA, RepRPN-C, RCC, Xuetal., and CLIP-Count. The experimental results show that the object counting model (Ours) provided in this application has relatively impressive performance and versatility.

[0125] 1. Comparison of quantitative analysis on FSC-147

[0126] Our object counting model (Ours) is compared with state-of-the-art class-agnostic object counting methods, and the quantitative results are summarized in Table 1. Overall, our object counting model (Ours) outperforms state-of-the-art zero-shot class-agnostic counting methods on FSC-147 and is competitive with no-reference methods. It is worth noting that zero-shot object counting is designed to address more challenging scenarios than no-reference methods because the model needs to understand the correspondence between textual and visual features.

[0127] Table 1 Quantitative analysis results on FSC-147

[0128]

[0129] 2. Quantitative Analysis of CARPK

[0130] The object counting model of this application (Ours) was tested on the CARPK dataset to evaluate its generalization ability across datasets. Specifically, the model was trained on FSC-147 and evaluated directly on the test set of CARPK without any fine-tuning. For a fair comparison, Jiang et al. trained RCC on FSC-147 using the same visual backbone (Vit-B) instead of their proposed FSC-133. As shown in Table 2, the object counting model of this application (Ours) shows a certain performance improvement compared with the representative reference-free counting method RCC and the zero-shot counting method CLIP-Count.

[0131] Table 2 Cross-dataset evaluation on CARPK

[0132]

[0133] Indicates retraining using the same backbone network

[0134] 3. Quantitative analysis at ShanghaiTech

[0135] An existing CLIP-based crowd counting method called CrowdCLIP evaluates the performance of the model across datasets by training a target model for a specific category on one part of the ShanghaiTech dataset and testing it on another part. Similar experiments were conducted to directly evaluate the performance of our target counting model (Ours) and RCC and CLIP-Count on the crowd counting task (i.e., without fine-tuning on any part of the ShanghaiTech dataset). To ensure a fair comparison, the backbone network of RCC was also modified to make it consistent with the target counting model (Ours) of our application. The experimental results are shown in Table 3.

[0136] Table 3 Cross-dataset evaluation on ShanghaiTech

[0137]

[0138]

[0139] Indicates retraining using the same backbone network

[0140] 4. Qualitative analysis and comparison

[0141] On the FSC-147 dataset, we compared our target counting model (Ours) with the most advanced zero-shot counting method CLIP-Count. We found that their performance is similar in sparse scenarios, while our target counting model (Ours) has certain advantages in dense scenarios. Figure 7 Specifically, our target counting model (Ours) produces clearer counting boundaries and can locate high-density regions at the center of targets with high fidelity. In contrast, the density prediction in CLIP-Count may exhibit a non-centralized pattern.

[0142] The present invention also discloses a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned text-guided zero-sample target counting method provided by the present invention. The computer program product should be understood as a software product that mainly implements its solution through a computer program, such as a program product integrated in the cloud or a software library.

[0143] The present invention also discloses an electronic device. In one embodiment, the electronic device includes at least one processor; and a memory connected to the at least one processor; wherein,

[0144] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can execute a text-guided zero-sample target counting method provided by the present invention.

[0145] like Figure 8 FIG2 is a schematic diagram of the structure of an electronic device for implementing a text-guided zero-sample target counting method according to an embodiment of the present invention. The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13. It may also include a computer program stored in the memory 11 and executable on the processor 10, such as a program for implementing a text-guided zero-sample target counting method.

[0146] In some embodiments, the processor 10 may be composed of an integrated circuit, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines. It executes or executes programs or modules stored in the memory 11 (e.g., executing a text-guided zero-sample target counting method), and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0147] The memory 11 includes at least one type of readable storage medium, including flash memory, mobile hard disk, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 can also be an external storage device of the electronic device, such as a plug-in mobile hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. equipped on the electronic device. Furthermore, the memory 11 can also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of a text-guided zero-sample target counting method program, but can also be used to temporarily store data that has been output or is to be output.

[0148] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0149] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, for displaying information processed in the electronic device and for displaying a visual user interface.

[0150] Figure 8 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 8 The structure shown does not limit the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0151] For example, although not shown, the electronic device may further include a power source (such as a battery) for powering various components. Preferably, the power source may be logically connected to at least one processor 10 via a power management device, thereby implementing functions such as charge management, discharge management, and power consumption management through the power management device. The power source may further include any components such as one or more DC or AC power sources, a recharging device, a power failure detection circuit, a power converter or inverter, a power status indicator, etc. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0152] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0153] Furthermore, if the module / unit integrated into the electronic device is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0154] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A text-guided zero-shot target counting method, characterized in that: include: Obtain a query image and target description text, input the query image and target description text into the trained target counting model, and the target counting model outputs a density estimation map of the target; The target counting model includes: A visual feature extraction module, used to obtain visual features of the query image; A text feature extraction module is used to obtain text features of the target description text; A multi-level polarity dual attention module includes multiple cascaded levels of feature fusion modules; wherein the feature fusion module of each level in at least one level performs visual polarity cross-attention processing on the visual-to-text fusion features or visual features output by the feature fusion module of the previous level to obtain visual-to-text fusion features of the current level, and performs text polarity cross-attention processing on the text-to-visual fusion features or text features output by the feature fusion module of the previous level to obtain text-to-visual fusion features of the current level; A decoder decodes the visual-to-text fusion features output by at least one level of feature fusion module to obtain a density estimation map of the target; The feature fusion module of each level in at least one level includes: The visual-to-text polarity cross attention unit performs visual polarity cross attention processing on the visual features or the visual-to-text fusion features output by the feature fusion module of the previous layer to obtain the visual-to-text attention features of this layer; The text-to-visual polarity cross attention unit performs text-to-visual fusion features output by the feature fusion module of the previous level on the text features or the text-to-visual fusion features to obtain the text-to-visual attention features of this level. A first fusion unit performs a first fusion process on the visual-to-text attention feature and the text-to-visual attention feature to obtain a visual-to-text fusion feature; A second fusion unit performs a second fusion process on the visual-to-text attention feature and the text-to-visual attention feature to obtain a text-to-visual fusion feature; The process of the visual-to-text polarity cross attention unit performing visual polarity cross attention processing includes: The visual features or visual self-attention features or the visual-to-text fusion features output by the feature fusion module of the previous layer are used as the visual query vector, and the text features or text self-attention features or the text-to-visual fusion features output by the feature fusion module of the previous layer are used as the visual key vector and the visual value vector; Adjusting the magnitude of the visual query vector and the visual value vector by a first feature scaling factor; Performing positive and negative polarity decomposition and enhancement processing on the amplitude-adjusted visual query vector to obtain a first query positive polarity enhancement feature and a first query negative polarity enhancement feature; performing positive and negative polarity decomposition and enhancement processing on the visual key vector to obtain a first key positive polarity enhancement feature and a first key negative polarity enhancement feature; Calculating a first positive similarity by using the first query positive polarity enhancement feature and the first key positive polarity enhancement feature; calculating a first negative similarity by using the first query negative polarity enhancement feature and the first key negative polarity enhancement feature; Performing activation function processing on the difference between the first positive similarity and the first negative similarity to obtain a first attention weight; Obtain the visual-to-text attention features of this level based on the first attention weight and the amplitude-adjusted visual value vector; The process of the text-to-visual polarity cross attention unit performing text polarity cross attention processing includes: The text features or text self-attention features or the text-to-visual fusion features output by the feature fusion module of the previous layer are used as the text query vector, and the visual features or visual self-attention features or the visual-to-text fusion features output by the feature fusion module of the previous layer are used as the text key vector and text value vector; Performing amplitude adjustment on the text query vector and the text value vector using a second feature scaling factor; Performing positive and negative polarity decomposition and enhancement processing on the amplitude-adjusted text query vector to obtain a second query positive polarity enhancement feature and a second query negative polarity enhancement feature; performing positive and negative polarity decomposition and enhancement processing on the text key vector to obtain a second key positive polarity enhancement feature and a second key negative polarity enhancement feature; Calculating a second positive similarity by using the second query positive polarity enhancement feature and the second key positive polarity enhancement feature; calculating a second negative similarity by using the second query negative polarity enhancement feature and the second key negative polarity enhancement feature; Performing activation function processing on the difference between the second positive similarity and the second negative similarity to obtain a second attention weight; The text-to-visual attention feature of this level is obtained based on the second attention weight and the amplitude-adjusted text value vector.

2. The text-guided zero-shot target counting method according to claim 1, wherein: The feature fusion module of each layer in at least one layer further includes a visual multi-head self-attention unit and a text multi-head self-attention unit; The visual multi-head self-attention unit performs multi-head self-attention processing on the visual features or the visual-to-text fusion features output by the feature fusion module of the previous layer to obtain the visual self-attention features; The visual-to-text polarity cross-attention unit performs visual polarity cross-attention processing on the visual self-attention features to obtain the visual-to-text attention features of this level; The text multi-head self-attention unit performs multi-head self-attention processing on the text features or the text-to-visual fusion features output by the feature fusion module of the previous level to obtain text self-attention features; The text-to-visual polarity cross-attention unit performs text-polarity cross-attention processing on the text self-attention features to obtain the text-to-visual attention features of this level.

3. A text-guided zero-sample target counting method according to claim 1 or 2, characterized in that: A convolution upsampling module is also connected between the feature fusion modules of two adjacent layers. The convolution upsampling module performs convolution and upsampling on the visual-to-text fusion features output by the feature fusion module of the previous layer in the two adjacent layers, and uses the processing results as the input of the feature fusion module of the next layer in the two adjacent layers.

4. A text-guided zero-sample target counting method according to claim 1 or 2, characterized in that: The visual feature extraction module includes: Visual encoder, which processes the query image to obtain visual embedding; Visual adapter, which converts visual embeddings to specified dimensions to obtain visual features; And / or, the text feature extraction module includes: Text encoder, which processes the target description text to obtain text embedding; Text adapter, converts text embedding to specified dimensions to obtain text features.

5. A text-guided zero-sample target counting method according to claim 1 or 2, characterized in that: The decoder includes a plurality of cascaded decoding convolution upsampling modules and an output convolution layer, and further includes one or more adding units, wherein the one or more adding units are respectively connected in series between one or more adjacent pairs of decoding convolution upsampling modules, and each adding unit corresponds to a feature fusion module of a level; The first-level feature fusion module outputs the visual-to-text fusion feature to the input of the first decoding convolutional upsampling module; The addition unit is used to add the output features of the previous decoding convolution upsampling module of the addition unit and the visual to text fusion features output by the feature fusion module of the corresponding level, and input the addition result into the next decoding convolution upsampling module of the addition unit.

6. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of any one of the methods of claims 1 to 5 are implemented.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, A memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor so as to enable the at least one processor to perform a text-guided zero-sample target counting method as described in any one of claims 1 to 5.

Images