Text-guided zero-shot target counting method, device, product and electronic device

Abstract

The present invention belongs to the field of computer image technology and provides a text-guided zero-shot target counting method, device, product and electronic device. The target counting method comprises: inputting a query image into a visual encoder to obtain a visual embedding, and extracting multiple intermediate layer features of the visual encoder; inputting a target category name text into a text encoder to obtain a text embedding; reshaping the visual embedding to obtain a visual patch embedding, and reshaping multiple intermediate layer features to obtain multiple intermediate layer patch embeddings; inputting the text embedding, the visual patch embedding and the multiple intermediate layer patch embeddings into an adaptive multi-view feature fusion module to obtain an adaptive multi-view similarity graph; inputting the adaptive multi-view similarity graph, the text embedding, the multiple intermediate layer features and the visual patch embedding into a decoder to generate a density estimation graph corresponding to the target category; the present invention utilizes the multiple intermediate layer features of the visual encoder to generate a more accurate density estimation graph, thereby enhancing the accuracy of target positioning and counting.

Description

Technical Field

[0001] The present invention relates to the field of computer image technology, and in particular to a text-guided zero-sample target counting method, device, product and electronic equipment. Background Art

[0002] Object counting aims to determine the number of objects in an image. Traditional object counting methods typically rely on datasets for specific categories, such as crowds, agriculture, cells, and cars, and are less effective when dealing with unseen categories of objects.

[0003] To this end, few-shot object counting methods have emerged, which provide visual examples at inference time. Visual examples are usually provided in the form of bounding boxes. This method is easy to deploy in dynamic environments, but it still requires human intervention to provide these visual examples.

[0004] Reference-free object counting methods have also emerged, which are more autonomous and operate without any user input or manual annotation. While this provides flexibility by not requiring visual or textual cues, it does not allow for the specification of objects of interest, which may limit its applicability in some cases.

[0005] Building on the above research, Xu et al. proposed a text-guided object counting method that allows users to specify object categories through text descriptions rather than visual examples. Although this method reduces the need for visual input, it still faces challenges and ignores the value of the intermediate layers of the network. These intermediate layers contain hierarchical features that are crucial for capturing finer image details, which are crucial for improving the performance of downstream tasks. Summary of the Invention

[0006] The present application aims to at least solve the technical problems existing in the prior art and provide a text-guided zero-sample target counting method, device, product and electronic device.

[0007] In the first aspect, the present application provides a text-guided zero-shot target counting method, including: inputting a query image to a visual encoder to obtain a visual embedding, and extracting multiple intermediate layer features of the visual encoder; inputting a target category name text to a text encoder to obtain a text embedding; performing two-dimensional reshaping on the visual embedding to obtain a visual patch embedding, and performing two-dimensional reshaping on multiple intermediate layer features to obtain multiple intermediate layer patch embeddings; inputting text embedding, visual patch embedding and multiple intermediate layer patch embeddings to an adaptive multi-view feature fusion module to obtain an adaptive multi-view similarity graph; inputting an adaptive multi-view similarity graph, text embedding, multiple intermediate layer features and visual patch embedding to a decoder, and the decoder generates a density estimation graph corresponding to the target category.

[0008] In the second aspect, the present application provides a text-guided zero-shot target counting device, which is used to implement the text-guided zero-shot target counting method described in the first aspect of the present application, and is characterized in that it includes: a visual embedding module, which inputs a query image to a visual encoder to obtain a visual embedding, and extracts multiple intermediate layer features of the visual encoder; a text embedding module, which inputs a target category name text to a text encoder to obtain a text embedding; a reshaping module, which performs two-dimensional reshaping on the visual embedding to obtain a visual patch embedding, and performs two-dimensional reshaping on multiple intermediate layer features to obtain multiple intermediate layer patch embeddings; an adaptive multi-view similarity graph acquisition module, which inputs text embedding, visual patch embedding and multiple intermediate layer patch embeddings to an adaptive multi-view feature fusion module to obtain an adaptive multi-view similarity graph; a decoding module, which inputs an adaptive multi-view similarity graph, text embedding, multiple intermediate layer features and visual patch embedding to a decoder, and the decoder generates a density estimation graph corresponding to the target category.

[0009] In a third aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the text-guided zero-sample target counting method described in the first aspect of the present application.

[0010] In a fourth aspect, the present application provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the text-guided zero-sample target counting method described in the first aspect of the present application.

[0011] The beneficial technical effect of the present application is: while using the visual encoder to obtain the visual embedding of the query image, multiple intermediate layer features of the visual encoder are also extracted, and the multiple intermediate layer features are input into the adaptive multi-view feature fusion module together with the visual embedding and text embedding to obtain an adaptive multi-view similarity map, which effectively fuses information from multiple perspectives and ensures better alignment of visual and text embeddings to generate a more accurate density estimation map, thereby enhancing the accuracy of target positioning and counting. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 1 is a flow chart of a text-guided zero-shot target counting method in a preferred embodiment of the present invention;

[0013] Figure 2 This is a network architecture diagram of a zero-sample target counting method in a preferred embodiment of the present invention;

[0014] Figure 3This is a schematic diagram of the internal structure of an adaptive multi-view feature fusion module in a preferred embodiment of the present invention;

[0015] Figure 4 This is a schematic diagram of the internal structure of a decoder in a preferred embodiment of the present invention;

[0016] Figure 5 This is a schematic structural diagram of an electronic device in a preferred embodiment of the present invention;

[0017] Figure 6 This is the experimental verification and comparison result of the zero-sample target counting method provided by the present invention. DETAILED DESCRIPTION

[0018] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0019] In the description of the present invention, it should be understood that the terms "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0020] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal communication between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0021] The execution subject of the text-guided zero-sample target counting method provided by the present invention includes but is not limited to at least one of the electronic devices such as the server and the terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the text-guided zero-sample target counting method can be executed by software or hardware installed on the terminal device or the server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0022] The present invention provides a text-guided zero-shot target counting method. The execution process is shown in the attached Figure 1 The model framework of the text-guided zero-shot target counting method of the present invention is denoted as AMS-Counter. The network framework diagram of AMS-Counter is shown in FIG. Figure 2 In a preferred embodiment, the text-guided zero-shot target counting method includes:

[0023] Step S1: Input the query image into the visual encoder to obtain the visual embedding, which can be expressed as ε v , and extract multiple intermediate layer features of the visual encoder, which can be expressed as l represents the number of extracted intermediate layer features.

[0024] In this embodiment, the visual encoder is an image encoder based on the Transformer architecture, which adopts a stacked structure of the same layer to facilitate the extraction of intermediate layer features. The visual encoder is preferably a DINOv2 visual encoder. DINOv2 is based on the Transformer architecture design and optimizes the self-attention mechanism, which can better capture the long-range dependencies in the query image, thereby extracting more accurate and rich visual features. For example, the query image is represented by I, such as Figure 2 The query image shown is an image of grapes.

[0025] Step S2: Input the target category name text into the text encoder to obtain text embedding, which can be expressed as ε t .

[0026] For example, the target category name text is represented by t, such as Figure 2As shown, t represents "grapes." The text encoder can be an existing encoder such as BERT or XLNet. Preferably, it is a text encoder based on the CLIP model. The CLIP model (Contrastive Language–Image Pre-training) uses contrastive learning to bring semantically similar features of text and images closer together in feature space. It includes an image encoder and a text encoder. This application uses the text encoder to facilitate alignment of visual and textual features.

[0027] Step S3, embedding the visual v Perform two-dimensional reshape to obtain visual patch embedding ε p . For multiple intermediate layer features Perform two-dimensional reshaping to obtain multiple intermediate layer patch embeddings, represented as That is, each intermediate layer feature is reshaped into two dimensions to obtain an intermediate layer patch embedding. The two-dimensional reshape operation mainly reshapes the one-dimensional token sequence into a two-dimensional feature map.

[0028] Step S4, input text embedding ε t , visual patch embedding ε p and multiple intermediate patch embeddings To the adaptive multi-view feature fusion module (abbreviated as AMFFM module) to obtain the adaptive multi-view similarity map (abbreviated as AMS-Map), which can be expressed as S fusion .

[0029] In this embodiment, the adaptive multi-view feature fusion module AMFFM uses contrastive learning to enhance the visual embedding ε v and text embedding ε t The correlation between them, and also introduce multiple intermediate layer patch embedding To enhance the model's ability to locate targets of interest.

[0030] Step S5: Input the adaptive multi-view similarity graph, text embedding, multiple intermediate layer features and visual patch embedding into the decoder (abbreviated as UC-Decoder), and the decoder generates a density estimation map corresponding to the target category, denoted as Y pred For example, Figure 2 As shown, the density estimation map marks a highlight point at the location of each grape.

[0031] The existing object counting method CLIP-Count based on the CLIP model (Contrastive Language–Image Pre-training) solves the challenge of aligning visual and textual features through contrastive learning, enabling it to match images and text in a shared feature space. Although this method has shown potential, it relies on a simple cosine similarity map for feature alignment, which limits its ability to capture more complex relationships between modalities. Therefore, in a preferred embodiment, in order to address the limitations of existing methods that rely solely on cosine similarity to align visual and textual features, multiple intermediate layer features are incorporated to capture complementary and information-rich image representations. The adaptive multi-view feature fusion module AMFFM adopts the following structure. See the attached Figure 3 , AMFFM includes:

[0032] Multi-view branch (i.e., Multi-viewbranch), used for text-based embedding ε t Embedded with multiple middle-layer patches The similarity of the multi-view similarity graph V′ is obtained space .

[0033] Single-view branch, used for text-based embedding ε t and visual patch embedding ε p The similarity of the first similarity graph S is obtained adpt .

[0034] Adaptive Skip Connection (ASC) unit connects the multi-view similarity graph and the first similarity graph S adpt Obtain the combined features, and perform convolution processing on the combined features to obtain the fusion similarity graph S fused , use the learned fusion weight α to weight the first similarity graph S adpt and fusion similarity graph S fused Obtain adaptive multi-view similarity graph S fusion .

[0035] In this embodiment, specifically, the adaptive skip connection unit ASC includes a stacking unit, a first convolution unit and a weighting unit connected in sequence. The stacking unit is used to connect the multi-view similarity graph and the first similarity graph S adpt The first convolution is used to perform convolution processing on the combined features to obtain the fusion similarity graph S fused The processing of the weighted unit can be expressed as:

[0036] S fusion =α×S adpt +(1-α)×Conv([Sadpt ,V′ space ])

[0037] Among them, α represents the fusion weight, which is learned during the AMS-Counter model training and can range from 0 to 1. Conv([S adpt ,V′ space ]) represents the result of the first convolution processing.

[0038] In this embodiment, the adaptive multi-view similarity graph S fusion The information from the single-view branch and the multi-view branch is effectively integrated, enhancing the model's ability to capture object counting-related features.

[0039] In the prior art, the role of frequency domain information is often overlooked. Extracting frequency domain features can capture key structural details that are crucial for accurate image understanding and further improve the accuracy of object positioning and counting. In a preferred embodiment, see the attached Figure 3 , multi-view branch (ie Multi-viewbranch) includes:

[0040] The first similarity calculation unit calculates the text embedding ε t Embedded with multiple middle-layer patches The similarity of is used to obtain a multi-level similarity graph Specifically, calculate the text embedding ε t The similarity (preferably but not limited to cosine similarity) of each intermediate layer patch embedding is used to obtain a hierarchical similarity graph, thus obtaining a multi-level similarity graph.

[0041] Feature extraction unit, multi-level similarity graph of spatial domain Convert to the frequency domain and extract the features of the multi-level similarity graph in the frequency domain to obtain the frequency domain features V freq , the frequency domain feature V freq Convert to spatial domain to obtain multi-view similarity graph V′ space .

[0042] In this embodiment, further preferably, the feature extraction unit includes:

[0043] Unbiased Fast Fourier Convolution (UFFC) is used to transform the multi-level similarity graph Convert to the frequency domain and extract the features of the multi-level similarity graph in the frequency domain to obtain the frequency domain features V freq ;

[0044] Inverse unbiased fast Fourier convolution (inverse UFFC) is used to transform the frequency domain features V freq Convert to spatial domain to obtain multi-view similarity graph V′space .

[0045] In this implementation, unbiased fast Fourier convolution (UFFC) is used to map images from the spatial domain to the frequency domain, capturing a wider receptive field and enhancing global feature extraction. This separates high- and low-frequency signals, improving the processing of image details. This enables more accurate feature extraction in complex scenes, improving recognition accuracy and robustness in object counting tasks.

[0046] In a preferred embodiment, the task-specific adaptive adjustment is performed to create a task-specific first similarity graph S for target counting. adpt Please refer to the attached Figure 3 , single view branches include:

[0047] The second similarity calculation unit is used to calculate the text embedding ε t and visual patch embedding ε p The similarity of the original similarity graph is obtained, which is represented by S orig Specifically, calculate the text embedding ε t and visual patch embedding ε p The cosine similarity of the original similarity graph S is obtained orig .

[0048] Adaptive adjustment unit, using the learned weight matrix W and bias matrix B to adjust the original similarity graph S orig Get the first similarity graph S adpt :

[0049] S adpt =W×S orig +B

[0050] in, They are the weight matrix and bias matrix used for adaptive adjustment, respectively, and are both learned during the AMS-Counter model training.

[0051] In a preferred embodiment, please refer to the attached Figure 4 , N-1 intermediate layer features are extracted from the visual encoder, and the N-1 intermediate layer features and visual patch embeddings are arranged in ascending order of the level in the visual encoder into a visual feature embedding sequence X = {nth visual feature embedding} n＝1,2,...,N Visual patch embedding ε p Corresponding to the Nth visual feature embedding, the first intermediate layer feature extracted is the first visual feature embedding. The decoder includes N decoding layers, where N is a positive integer greater than or equal to 2. Figure 4 The decoder consists of 4 decoding layers. In the decoder:

[0052] The Nth decoding layer includes the Nth crisscross attention module, the Nth reshaping unit, the Nth convolutional layer, the Nth upsampling unit, and the Nth multiplication unit connected in sequence; wherein the Nth crisscross attention module uses the Nth visual feature embedding as the query matrix and the text embedding ε as the query matrix. t It serves as both a key matrix and a value matrix; the Nth layer multiplication unit is used to perform element-wise multiplication on the adaptive multi-view similarity map and the upsampling map of the Nth layer upsampling unit to obtain the Nth layer decoding features.

[0053] Specifically, the N-th layer cross attention module uses the N-th visual feature embedding as the query matrix and the text embedding ε t The Nth layer performs cross-attention processing as both the key matrix and the value matrix, obtaining the Nth layer cross-attention processing result. The Nth layer reshape unit reshapes the Nth layer cross-attention processing result into a two-dimensional patch embedding form, obtaining the Nth layer reshaped embedding. The Nth layer convolution layer convolves the Nth layer reshaped embedding to obtain the Nth layer convolution result. The Nth layer upsampling unit upsamples the Nth layer convolution result by a factor of 2 to obtain the upsampled image.

[0054] The m-th decoding layer includes the m-th crisscross attention module, the m-th reshaping unit, the m-th addition unit, the m-th convolution layer, the m-th upsampling unit and the m-th multiplication unit connected in sequence; wherein, the m-th crisscross attention module takes the m-th visual feature embedding as the query matrix and the text embedding ε t It serves as both a key matrix and a value matrix; the m-th layer addition unit is used to perform element-wise addition of the output features of the m-th layer reshaping unit and the m+1-th layer decoding features (i.e., the output features of the m+1-th layer multiplication unit); the m-th layer multiplication unit is used to perform element-wise multiplication of the adaptive multi-view similarity map and the upsampling map of the m-th layer upsampling unit to obtain the m-th layer decoding features, where m is a positive integer and 1<m<N.

[0055] Specifically, the m-th layer cross attention module uses the m-th visual feature embedding as the query matrix and the text embedding ε t The mth layer performs cross-attention processing as both the key matrix and the value matrix, obtaining the mth layer cross-attention processing result. The mth layer reshape unit reshapes the mth layer cross-attention processing result into a two-dimensional patch embedding form, obtaining the mth layer reshaped embedding. The mth layer addition unit is used to perform element-wise addition of the output features of the mth layer reshape unit and the m+1th layer decoding features (i.e., the output features of the m+1th layer multiplication unit) to obtain the mth layer addition result. The mth layer convolution layer convolves the mth layer addition result to obtain the mth layer convolution result, and the mth layer upsampling unit upsamples the Nth layer convolution result by a factor of 2 to obtain the upsampled image.

[0056] The first decoding layer includes the first crisscross attention module, the first reshaping unit, the first addition unit and the first convolution layer connected in sequence; the first crisscross attention module uses the first visual feature embedding as the query matrix and the text embedding ε as the query matrix. t It serves as both a key matrix and a value matrix. The first-layer addition unit is used to perform element-wise addition of the output features of the first-layer reshape unit and the second-layer decoded features.

[0057] Specifically, the first-layer cross-attention module uses the first visual feature embedding as the query matrix and the text embedding ε t At the same time, it is used as the key matrix and the value matrix for cross attention processing to obtain the first layer cross attention processing result. The first layer reshaping unit reshapes the first layer cross attention processing result into a two-dimensional patch embedding form to obtain the first layer reshaping embedding. The first layer addition unit is used to perform element-wise addition on the output features of the first layer reshaping unit (i.e., the first layer reshaping embedding) and the second layer decoding features (i.e., the output features of the second layer multiplication unit) to obtain the first layer addition result. The first layer convolution layer performs convolution processing on the first layer addition result to obtain a density estimation map, which is represented by Y pred .

[0058] In this embodiment, the crisscross attention module includes multi-head self-attention (MHSA), multi-head crisscross attention (MHCA) and feed-forward network (FFN).

[0059] The present invention also discloses a text-guided zero-sample target counting device for implementing the above-mentioned text-guided zero-sample target counting method. In a preferred embodiment, the device comprises:

[0060] The visual embedding module inputs the query image into the visual encoder to obtain the visual embedding and extracts multiple intermediate layer features of the visual encoder;

[0061] Text embedding module, input the target category name text to the text encoder to obtain text embedding;

[0062] The reshaping module reshapes the visual embedding in two dimensions to obtain the visual patch embedding, and reshapes multiple intermediate layer features in two dimensions to obtain multiple intermediate layer patch embeddings;

[0063] Adaptive multi-view similarity graph acquisition module, which inputs text embedding, visual patch embedding and multiple intermediate layer patch embeddings into the adaptive multi-view feature fusion module to obtain an adaptive multi-view similarity graph;

[0064] The decoding module inputs the adaptive multi-view similarity map, text embedding, multiple intermediate layer features and visual patch embedding into the decoder, and the decoder generates a density estimation map corresponding to the target category.

[0065] In this embodiment, the visual embedding module, the text embedding module, the reshaping module, the adaptive multi-view similarity graph acquisition module and the decoding module correspond one-to-one to steps S1, S2, S3, S4 and S5 of the above-mentioned text-guided zero-shot target counting method, and are not repeated here.

[0066] The text-guided zero-shot target counting method and device provided by the present invention utilizes spatial and frequency domain information in the adaptive multi-view similarity graph AMS generated by the adaptive multi-view feature fusion module (AMFFM). The decoder plays a key role by fusing multi-level features, ensuring better alignment of visual and text embeddings to generate accurate density estimation maps, thereby improving target positioning and counting accuracy.

[0067] The following is an experimental verification of the above text-guided zero-shot target counting method and device:

[0068] 1. Experimental Details

[0069] 1.1 Model Implementation: The AMS-Counter proposed in this paper uses MetaAI pre-trained DINOv2ViT-B / 14 as the visual encoder and OpenAI pre-trained CLIPViT-B / 16 as the text encoder. For fine-tuning the visual encoder, a deep variant of Visual Prompt Fine-tuning (VPT) is used, integrating 40 learnable prompts per layer. For each decoder layer l∈{3,6,9,12}, AMFFM uses the patch embeddings corresponding to the 3rd, 6th, and 9th layers, i.e. as a multi-view similarity graph. The decoder can use the GeLU nonlinear activation function after each convolutional layer. For image preprocessing, all images are resized to 384×384 and augmented. The model is trained on a single NVIDIA RTX4090 using the AdamW optimizer with a batch size of 32 for 200 epochs. For contrastive learning, the InfoNCE loss is used, with a temperature τ set to 0.07.

[0070] 1.2 Dataset: FSC-147, introduced by Ranjan et al. in 2021, is a benchmark dataset for category-independent counting. It contains 6,135 images from 147 different categories, such as food, animals, and vehicles. The dataset is divided into a training set (89 categories), a validation set, and a test set, each containing 29 distinct, non-overlapping categories, meeting the requirements of the TZOC task.

[0071] 1.3 Evaluation Metrics: Consistent with previous class-independent object counting methods, two key evaluation metrics are used: mean absolute error (MAE) and root mean square error (RMSE).

[0072] 2. Quantitative results

[0073] Experimental results highlight AMS-Counter's superior zero-shot counting capabilities, achieving state-of-the-art results on FSC-147, as shown in Table 1. By leveraging an advanced textual hinting mechanism and a more powerful infrastructure, our AMS-Counter surpasses previous zero-shot methods such as VA-Count, improving validation set RMSE by 38.56% and test set RMSE by 40.41%, while also performing well in competition with few-shot and no-reference methods.

[0074] Table 1 Quantitative experiments on the FSC-147 dataset

[0075]

[0076] 3. Ablation studies

[0077] The ablation study in Table 2 examines the role of the adaptive multi-view similarity map (AMS) in contrastive learning and decoding, as well as the impact of the decoder. Replacing the AMS with a cosine similarity map (A1, A2) leads to performance degradation, especially without the decoder. While the decoder improves the results, performance remains suboptimal without the AMS-Map. Incorporating the AMS-Map in contrastive learning (B1) or decoding (B2) shows significant improvements, with the best results achieved when the AMS-Map is used with the decoder in both stages (C1), demonstrating that this combination is crucial for optimal accuracy and generalization.

[0078] Table 2 Ablation experiment results

[0079]

[0080] Figure 6 The qualitative results of the AMS-Counter of the present invention on the FSC-147 dataset are presented, demonstrating its ability to accurately identify and locate targets of interest and generate corresponding density estimation maps. Figure 6 Qualitative comparison of the adaptive multi-view similarity map (column 2) and the cosine similarity map (column 3). Column 1 shows the input image, and column 4 shows the ground truth (GroundTruth). The overlay of the predicted density map on the input image effectively demonstrates AMS-Counter's ability to capture both object distribution and number, demonstrating its robustness and effectiveness.

[0081] This paper proposes AMS-Counter, a novel framework for text-guided object counting that integrates adaptive multi-view similarity graphs (AMS) to improve object localization and counting accuracy. The AMS generated by the Adaptive Multi-view Feature Fusion Module (AMFFM) exploits both spatial and frequency domain information. The decoder plays a key role by fusing multi-level features, ensuring better alignment of visual and textual embeddings to generate accurate density estimation maps. Comprehensive evaluation on the FSC-147 dataset demonstrates state-of-the-art performance, highlighting the robustness of AMS-Counter for open-world object counting and real-world applications.

[0082] The present invention also discloses a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned text-guided zero-sample target counting method provided by the present invention. The computer program product should be understood as a software product that mainly implements its solution through a computer program, such as a program product integrated in the cloud or a software library.

[0083] The present invention also discloses an electronic device. In one embodiment, the electronic device includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the text-guided zero-sample target counting method provided by the present invention.

[0084] like Figure 5 FIG2 is a schematic diagram of an electronic device for implementing a text-guided zero-shot target counting method according to an embodiment of the present invention. The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13. The electronic device may also include a computer program stored in the memory 11 and executable on the processor 10, such as a program for the text-guided zero-shot target counting method.

[0085] In some embodiments, the processor 10 may be composed of an integrated circuit, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines. It executes or runs programs or modules stored in the memory 11 (such as executing a text-guided zero-sample target counting method) and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0086] The memory 11 includes at least one type of readable storage medium, including flash memory, mobile hard disk, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of the electronic device, such as a plug-in mobile hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash memory card (FlashCard), etc. equipped on the electronic device. Furthermore, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can be used not only to store application software and various types of data installed in the electronic device, such as the code of the text-guided zero-sample target counting method program, but also to temporarily store data that has been output or is to be output.

[0087] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0088] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, for displaying information processed in the electronic device and for displaying a visual user interface.

[0089] Figure 5 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 5The structure shown does not limit the electronic device and may include fewer or more components than shown in the figure, or combine some components, or arrange the components differently.

[0090] For example, although not shown, the electronic device may further include a power source (such as a battery) to power various components. Preferably, the power source may be logically connected to at least one processor 10 via a power management device, thereby implementing functions such as charge management, discharge management, and power consumption management through the power management device. The power source may further include any components such as one or more DC or AC power sources, a recharging device, a power failure detection circuit, a power converter or inverter, a power status indicator, etc. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0091] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0092] Furthermore, if the modules / units integrated into an electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Computer-readable storage media can be volatile or non-volatile. For example, computer-readable media can include any entity or device capable of carrying computer program code, recording media, USB flash drives, mobile hard drives, magnetic disks, optical disks, computer memory, and read-only memory (ROM).

[0093] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "example," "specific example," "one implementation," "a preferred implementation," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0094] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A text-guided zero-shot target counting method, characterized in that include: Input the query image into the visual encoder to obtain visual embedding and extract multiple intermediate layer features of the visual encoder; Input the target category name text to the text encoder to obtain text embedding; Perform two-dimensional reshaping on the visual embedding to obtain visual patch embedding, and perform two-dimensional reshaping on multiple intermediate layer features to obtain multiple intermediate layer patch embeddings; Input text embedding, visual patch embedding and multiple intermediate layer patch embeddings into the adaptive multi-view feature fusion module to obtain an adaptive multi-view similarity graph; Input adaptive multi-view similarity map, text embedding, multiple intermediate layer features and visual patch embedding to the decoder, and the decoder generates a density estimation map corresponding to the target category; The adaptive multi-view feature fusion module includes: Multi-view branch, used to obtain multi-view similarity graph based on the similarity between text embedding and multiple intermediate layer patch embeddings; The single-view branch is used to obtain the first similarity graph based on the similarity between text embedding and visual patch embedding; An adaptive skip connection unit connects the multi-view similarity map and the first similarity map to obtain a combined feature, performs convolution processing on the combined feature to obtain a fused similarity map, and uses the learned fusion weight to weight the first similarity map and the fused similarity map to obtain an adaptive multi-view similarity map; N-1 intermediate layer features are extracted from the visual encoder, and the N-1 intermediate layer features and visual patch embeddings are arranged into a visual feature embedding sequence in the order of the levels in the visual encoder from low to high. ; The decoder includes N decoding layers, where N is a positive integer greater than or equal to 2; The N-th decoding layer includes an N-th crisscross attention module, an N-th reshaping unit, an N-th convolutional layer, an N-th upsampling unit, and an N-th multiplication unit, which are connected in sequence. The N-th crisscross attention module uses the N-th visual feature embedding as the query matrix and the text embedding as both the key matrix and the value matrix. The N-th multiplication unit is used to perform element-wise multiplication of the adaptive multi-view similarity graph and the upsampling graph of the N-th upsampling unit to obtain the N-th decoding feature. The m-th decoding layer includes an m-th crisscross attention module, an m-th reshaping unit, an m-th addition unit, an m-th convolutional layer, an m-th upsampling unit, and an m-th multiplication unit connected in sequence; wherein the m-th crisscross attention module uses the m-th visual feature embedding as the query matrix and the text embedding as both the key matrix and the value matrix; the m-th addition unit is used to perform element-wise addition of the output features of the m-th reshaping unit and the m+1-th layer decoding features, and the m-th multiplication unit is used to perform element-wise multiplication of the adaptive multi-view similarity graph and the upsampling graph of the m-th upsampling unit to obtain the m-th layer decoding features, where m is a positive integer and 1<m<N; The first decoding layer includes a first-layer cross-attention module, a first-layer reshaping unit, a first-layer addition unit, and a first-layer convolutional layer connected in sequence; the first-layer cross-attention module uses the first visual feature embedding as the query matrix and the text embedding as both the key matrix and the value matrix; the first-layer addition unit is used to perform element-wise addition of the output features of the first-layer reshaping unit and the second-layer decoding features.

2. The text-guided zero-shot target counting method according to claim 1, wherein: The visual encoder is a DINOv2 visual encoder; And / or, the text encoder is a text encoder of a CLIP model.

3. The text-guided zero-shot target counting method according to claim 2, wherein: The multi-view branch includes: A first similarity calculation unit calculates the similarity between the text embedding and the embedding of multiple intermediate layers of patches to obtain a multi-level similarity graph; The feature extraction unit converts the multi-level similarity graph into the frequency domain, performs feature extraction on the multi-level similarity graph in the frequency domain to obtain frequency domain features, and converts the frequency domain features into the spatial domain to obtain a multi-view similarity graph.

4. The text-guided zero-shot target counting method according to claim 3, wherein: The feature extraction unit includes: Unbiased fast Fourier convolution is used to convert the multi-level similarity map into the frequency domain and perform feature extraction on the multi-level similarity map in the frequency domain to obtain frequency domain features; Inverse unbiased fast Fourier convolution is used to transform frequency domain features into spatial domain to obtain multi-view similarity graph.

5. The text-guided zero-shot target counting method according to claim 1, wherein: The single view branch includes: A second similarity calculation unit is used to calculate the similarity between the text embedding and the visual patch embedding to obtain an original similarity graph; The adaptive adjustment unit adjusts the original similarity map using the learned weight matrix and bias matrix to obtain a first similarity map.

6. A text-guided zero-sample target counting device, for implementing the text-guided zero-sample target counting method according to any one of claims 1 to 5, characterized in that: include: The visual embedding module inputs the query image into the visual encoder to obtain the visual embedding and extracts multiple intermediate layer features of the visual encoder; Text embedding module, input the target category name text to the text encoder to obtain text embedding; The reshaping module reshapes the visual embedding in two dimensions to obtain the visual patch embedding, and reshapes multiple intermediate layer features in two dimensions to obtain multiple intermediate layer patch embeddings; Adaptive multi-view similarity graph acquisition module, which inputs text embedding, visual patch embedding and multiple intermediate layer patch embeddings into the adaptive multi-view feature fusion module to obtain an adaptive multi-view similarity graph; The decoding module inputs the adaptive multi-view similarity map, text embedding, multiple intermediate layer features and visual patch embedding into the decoder, and the decoder generates a density estimation map corresponding to the target category.

7. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

8. An electronic device, characterized in that: The electronic device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor so as to enable the at least one processor to perform the method according to any one of claims 1 to 5.

Images