A method and device for object counting using frequency features to enhance text guidance

Abstract

The present invention belongs to the field of computer image technology and provides a method and device for object counting guided by enhanced text using frequency features. The method comprises: inputting a query image into a visual encoder to obtain visual features; inputting object description text into a text encoder to obtain text embedding; utilizing a fusion network to fuse visual features and text features to obtain fused features; inputting the fused features into a decoder to obtain a density estimation map of the object, wherein the decoder comprises K cascaded adaptive frequency selection modules and an output layer, wherein the kth adaptive frequency selection module comprises a first convolution unit, an adaptive frequency selector and a first upsampling unit; the kth adaptive frequency selector respectively filters the amplitude spectrum and phase spectrum of the output features of the first convolution unit in the frequency domain, and converts the filtering results into the spatial domain. The decoding of the present application dynamically emphasizes relevant frequency components, combines spatial domain and frequency domain features, and accurately locates objects and counts them.

Description

Technical Field

[0001] The present invention relates to the field of computer image technology, and in particular to a method and device for object counting using frequency features to enhance text guidance. Background Art

[0002] Text-guided object counting aims to count specific objects in an image based on text descriptions, outputting a density estimate map that marks a point at the center of each object. Existing text-guided object counting schemes use a visual encoder to extract visual features of the image, a text encoder to obtain the text embedding of the text description, and then fuse the visual features and text embeddings to obtain fused features. The fused features are then input into a decoder to obtain a density estimate map. Existing text-guided object counting schemes primarily operate in the spatial domain, often overlooking the role of frequency domain analysis in object counting. In fact, frequency domain features can provide complementary information and can capture periodic patterns or subtle changes in data that are not obvious in the spatial domain. If the decoder can combine these frequency domain features for decoding processing, it will help improve object counting performance. Summary of the Invention

[0003] The present application aims to at least solve the technical problems existing in the prior art and provide a method and device for object counting using frequency features to enhance text guidance.

[0004] In the first aspect, the present application provides an object counting method using frequency features to enhance text guidance, including: inputting a query image to a visual encoder to obtain visual features; inputting object description text to a text encoder to obtain text embedding; using a fusion network to fuse visual features and text features to obtain fused features; inputting the fused features into a decoder to obtain a density estimation map of the object, the decoder including K cascaded adaptive frequency selection modules and 1 output layer, K and k are both positive integers, k∈[1,K], the kth adaptive frequency selection module including a first convolution unit, an adaptive frequency selector and a first upsampling unit connected in sequence; wherein the adaptive frequency selector of the kth adaptive frequency selection module filters the amplitude spectrum and phase spectrum of the output features of the first convolution unit in the frequency domain, respectively, and converts the filtering results to the spatial domain.

[0005] The above technical solution: the visual features of the query image and the text embedding of the object description text are fused to obtain fused features. The fused features contain rich information from both text and image modalities. The decoder decodes based on the fused features. The decoder includes K cascaded adaptive frequency selection modules and 1 output layer. The adaptive frequency selector filters the amplitude spectrum and phase spectrum of the output features of the first convolution unit in the frequency domain respectively, taking into account the different roles of the amplitude spectrum and phase spectrum in the object counting task, to achieve a more robust and detailed frequency domain feature representation. The relevant frequency components are dynamically emphasized during the decoding process, effectively combining the spatial domain and frequency domain features to achieve precise object positioning and accurate counting.

[0006] In a preferred embodiment of the present application, the adaptive frequency selector of the kth adaptive frequency selection module includes: a transformation unit, which transforms the output features of the first convolution unit in the kth adaptive frequency selection module to the frequency domain to obtain a first frequency domain representation; a decomposition unit, which decomposes the first frequency domain representation into an amplitude spectrum and a phase spectrum; a weighting unit, which performs weighted processing on the amplitude spectrum and the phase spectrum respectively to obtain a weighted amplitude spectrum and a weighted phase spectrum; a synthesis unit, which combines the weighted amplitude spectrum and the weighted phase spectrum to obtain a second frequency domain representation; and an inverse transformation module, which performs spatial domain conversion and residual processing on the second frequency domain representation to obtain the input features of the first upsampling unit.

[0007] The above technical solution: decomposes the first frequency domain representation into an amplitude spectrum and a phase spectrum, and filters the amplitude spectrum and the phase spectrum with different filtering parameters respectively to adaptively emphasize or suppress the characteristic frequency components, and connects the extracted frequency domain detail features (second frequency domain representation) with the spatial domain features (input features of the first upsampling unit) residually through the inverse transform module, effectively combining the spatial domain features and the frequency domain features.

[0008] In a preferred embodiment of the present application, the weighting unit filters the amplitude spectrum using the amplitude weight and the amplitude offset to obtain a weighted amplitude spectrum, and filters the phase spectrum using the phase weight and the phase offset to obtain a weighted phase spectrum.

[0009] The above technical solution: linearly transforms the amplitude spectrum and phase spectrum to facilitate flexible capture of the additive relationship between the amplitude spectrum and the phase spectrum. The amplitude weight, amplitude bias, phase weight and phase bias can be learned during model training and can adaptively emphasize or suppress specific frequency components.

[0010] In a preferred embodiment of the present application, the inverse transformation module includes: an inverse transformation unit, which converts the second frequency domain representation into the spatial domain to obtain a first spatial domain representation; an activation function unit, which performs an activation function on the first spatial domain representation to obtain a second spatial domain representation; and a residual connection unit, which performs a residual connection on the second spatial domain representation and the output features of the first convolution unit of the kth adaptive frequency selection module to obtain the input features of the first upsampling unit of the kth adaptive frequency selection module.

[0011] The above technical solution: introduces nonlinear modeling through the activation function, and uses the residual connection unit to residually connect the second spatial domain representation and the output features of the first convolution unit of the k-th adaptive frequency selection module, ensuring that the model can utilize the adjusted spatial features (the second spatial domain representation) and the original spatial features (the output features of the first convolution unit), which can retain spatial information and promote gradient flow during training.

[0012] In a preferred embodiment of the present application, the visual features include local patch embeddings and global CLS tags; the fusion network includes a first three-stream attention fusion module, which performs cross-attention processing on the local patch embeddings, global CLS tags and text embeddings to obtain a first combined feature, and the fused feature includes the first combined feature.

[0013] The above technical solution: The global CLS tag contains the global context information of the query image. However, existing object counting methods usually ignore the role of the global CLS tag, resulting in poor fusion of the global and local features of the query image, affecting the accuracy of object counting. This application fuses local patch embedding and global CLS tag together as visual features with text embedding, which can capture rich and complementary information across multiple modalities. This fusion enhances the model's ability to accurately locate and count specified objects.

[0014] In a preferred embodiment of the present application, the first three-stream attention fusion module includes: a first cross-attention unit, which performs a cross-attention operation with the local patch embedding as a query and the global CLS tag as a key and value to obtain a first cross-attention feature; a second cross-attention unit, which performs a cross-attention operation with the local patch embedding as a query and the text embedding as a key and value to obtain a second cross-attention feature; a first weighted fusion unit, which performs a weighted fusion of the local patch embedding, the first cross-attention feature and the second cross-attention feature to obtain a first combined feature.

[0015] The above technical solution: uses two cross-attention units, the first cross-attention unit and the second cross-attention unit, to fuse the local patch embedding with the global CLS tag and text embedding respectively, and uses the first weighted fusion unit to balance the contribution of the outputs of the two cross-attention units, so that the first combined feature effectively combines spatial and text information, enhancing multimodal information fusion. This fusion helps to enhance the model's ability to accurately locate and count specified objects.

[0016] In a preferred embodiment of the present application, the fusion network also includes: an upsampling processing module, which performs upsampling processing on the first combined features output by the first three-stream attention fusion module to obtain upsampled combined features; a second three-stream attention fusion module, which performs cross-attention processing on the upsampled combined features, the global CLS tag and the text embedding to obtain a second combined feature, and the fusion feature includes the first combined feature and the second combined feature, and the first combined feature serves as the input feature of the first adaptive frequency selection module; the decoder also includes an addition unit located after the kth adaptive frequency selection module, and the addition unit performs addition processing on the output feature of the kth adaptive frequency selection module and the second combined feature to obtain an added feature, and uses the added feature as the input feature of the k+1th adaptive frequency selection module or the input feature of the output layer.

[0017] The above technical solution: upsamples the first combined features through the upsampling processing module, and further fuses the upsampled combined features, global CLS tags and text embeddings using the second three-stream attention fusion module to obtain information at a different level from the first combined features, which can enrich the feature representation, and add the output features of the kth adaptive frequency selection module and the second combined features through the addition unit, which can enrich the features of the decoding process to improve the model's ability to accurately locate and count specified objects.

[0018] In a preferred embodiment of the present application, the second three-stream attention fusion module includes: a third cross-attention unit, which performs a cross-attention operation with the upsampled combined feature as a query and the global CLS tag as a key and value to obtain a third cross-attention feature; a fourth cross-attention unit, which performs a cross-attention operation with the upsampled combined feature as a query and the text embedding as a key and value to obtain a fourth cross-attention feature; and a second weighted fusion unit, which performs a weighted fusion of the upsampled combined feature, the third cross-attention feature and the fourth cross-attention feature to obtain a second combined feature.

[0019] The above technical solution: uses the third cross-attention unit and the fourth cross-attention unit to fuse the upsampled combined features with the global CLS tag and text embedding respectively, and uses the second weighted fusion unit to balance the contribution of the outputs of the two cross-attention units, so that the second combined features effectively combine spatial and text information, enhance multimodal information fusion, and this fusion helps to enhance the model's ability to accurately locate and count specified objects.

[0020] In a preferred embodiment of the present application, the upsampling processing module includes a cascaded second convolution unit and a second upsampling unit.

[0021] In the second aspect, the present application provides an object counting device that uses frequency features to enhance text guidance, which is used to implement an object counting method that uses frequency features to enhance text guidance in the first aspect, including: a visual feature acquisition module, which inputs a query image to a visual encoder to obtain visual features; a text embedding acquisition module, which inputs an object description text to a text encoder to obtain text embedding; a fusion feature acquisition module, which uses a fusion network to fuse visual features and text features to obtain fusion features; a decoding module, which inputs the fusion features into a decoder to obtain a density estimation map of the object, wherein the decoder includes K cascaded adaptive frequency selection modules and 1 output layer, K and k are both positive integers, k∈[1,K], and the kth adaptive frequency selection module includes a first convolution unit, an adaptive frequency selector and a first upsampling unit connected in sequence; wherein the adaptive frequency selector of the kth adaptive frequency selection module filters the amplitude spectrum and phase spectrum of the output features of the first convolution unit in the frequency domain, respectively, and converts the filtering results to the spatial domain. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 1 is a flow chart of an object counting method in a preferred embodiment of the present invention;

[0023] Figure 2 This is a model framework diagram that the object counting method relies on in a preferred embodiment of the present invention;

[0024] Figure 3 2 is a schematic diagram of the network structure of the first three-stream attention fusion module in a preferred embodiment of the present invention;

[0025] Figure 4 This is the visualization result of the object counting method provided by the present invention on the FSC-147 and CARPK datasets. DETAILED DESCRIPTION

[0026] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0027] In the description of the present invention, it should be understood that the terms "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0028] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal communication between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0029] The execution subject of the object counting method using frequency features to enhance text guidance provided by the present invention includes but is not limited to at least one of the electronic devices such as the server, the terminal, etc. that can be configured to execute the method provided by the embodiment of the present application. In other words, the object counting method using frequency features to enhance text guidance provided by the present invention can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0030] The present invention discloses a method for object counting using frequency features to enhance text guidance. In a preferred embodiment, see Figure 1 ,include:

[0031] Step S1: Input the query image to the visual encoder to obtain visual features.

[0032] In this embodiment, the query image is an image for which object counting is required, and the objects may be living objects or inanimate objects. For example, the query image may be Figure 2 The flying bird image in Figure 4 The visual encoder is preferably, but not limited to, the existing visual Transformer (ViT) model or DINOv2 model. Figure 2 , input the query image Image into the visual encoder It is possible to obtain visual embedding and reshape the visual embedding in two dimensions to obtain a local embedding patch ε p . Exemplarily, the visual features include local embedding patches ε p .

[0033] Step S2: Input the object description text into the text encoder to obtain text embedding.

[0034] In this embodiment, the object description text is a text describing the object to be counted, and may include the object name. For example, Figure 2 The object description text is "A photo of{birds}", Figure 4 The object description text is "eggs", "apples", "cars", etc. The text encoder can select the existing BERT encoder or CLIP (Contrastive Language-Image Pretraining) text encoder. For example, Figure 2 Input the object description text into the text encoder Then get the text embedding ε t .

[0035] Step S3: Using a fusion network to fuse visual features and text features to obtain fusion features.

[0036] For example, the fusion network is not limited to fusing visual features and text features using a direct concatenation method, an element-level fusion method, a cross-attention method, or the like.

[0037] Step S4: Input the fused features into the decoder to obtain the density estimation map Y of the object pred , see Figure 2 , decoder( Figure 2The Decode in the

[15] block includes K cascaded adaptive frequency selection modules and an output layer, where K and k are both positive integers, k∈[1,K]. The kth adaptive frequency selection module (AFS Block) includes a first convolution unit, an adaptive frequency selector (AFS), and a first upsampling unit connected in sequence; wherein the adaptive frequency selector of the kth adaptive frequency selection module filters the amplitude spectrum and phase spectrum of the output features of the first convolution unit in the frequency domain, respectively, and converts the filtering results to the spatial domain. The first upsampling unit is preferably, but not limited to, 2x upsampling. The features are continuously refined through the K cascaded adaptive frequency selection modules to improve the accuracy of the density estimation map.

[0038] In this embodiment, the value range of K can be [1, 10], preferably 4. The structure of each adaptive frequency selection module (AFS Block) is the same. Specifically, the adaptive frequency selector converts the output features of the first convolution unit into the frequency domain and decomposes them into an amplitude spectrum and a phase spectrum. The filtering process is to emphasize and / or suppress the amplitude and phase corresponding to specific frequencies in the amplitude spectrum and the phase spectrum, so that the model can focus on features related to the counting task. The filtering parameters in the filtering process can be learned during model training.

[0039] In a preferred embodiment, see Figure 2 , the adaptive frequency selector AFS of the kth adaptive frequency selection module includes:

[0040] The transformation unit transforms the output feature of the first convolution unit in the kth adaptive frequency selection module (expressed as (the spatial domain feature) is transformed into the frequency domain to obtain the first frequency domain representation Right now:

[0041]

[0042] in, Indicates the conversion of spatial domain features to frequency domain, and is not limited to the discrete Fourier transform method.

[0043] Decomposition unit, the first frequency domain representation Decomposition into amplitude spectrum and phase spectrum The specific decomposition process is:

[0044]

[0045] in, Represents the first frequency domain representation The real part of Represents the first frequency domain representation The imaginary part of . Amplitude spectrum Capture the intensity and phase spectrum of each frequency component Encodes the phase information of each frequency component.

[0046] The weighting unit is used to calculate the amplitude spectrum and phase spectrum Perform weighted processing to obtain weighted amplitude spectrum and weighted phase spectrum Specifically, using the learnable amplitude weight W mag Amplitude spectrum Perform weighted processing and use the learnable phase weight W phase Phase spectrum Perform weighted processing, amplitude weight W mag and phase weight W phase It can be learned during model training to adaptively emphasize or suppress specific frequency components.

[0047] Synthesis unit, merging weighted amplitude spectra and weighted phase spectrum Get the second frequency domain representation The specific combination is:

[0048]

[0049] The inverse transformation module performs spatial domain transformation and residual processing on the second frequency domain representation to obtain the input features of the first upsampling unit.

[0050] In this embodiment, preferably, see Figure 2 , the weighting unit uses the amplitude weight W mag and amplitude offset b mag Amplitude spectrum Perform filtering to obtain weighted amplitude spectrum Using the phase weight W phase and phase offset b phase Phase spectrum Perform filtering to obtain the weighted phase spectrum The specific filtering process is expressed as:

[0051]

[0052] In this embodiment, different weights and biases are used to respectively and phase spectrum Perform weighted processing, amplitude weight W mag , amplitude offset b mag , phase weight W phase and phase offset b phaseBoth can be learned during model training, adaptively selecting frequency components related to the counting task in the amplitude spectrum and phase spectrum respectively, highlighting the different effects of the amplitude spectrum and phase spectrum on the counting task, and achieving more robust and detailed feature representation.

[0053] In this implementation, an adaptive frequency selector (AFS) enhances feature representation by dynamically adjusting frequency content. In computer vision, high frequencies capture fine details, while low frequencies provide broader context. Spatial domain methods often struggle to fully exploit these patterns. The adaptive frequency selector (AFS) addresses this issue by using learnable parameters to adaptively emphasize or suppress specific frequencies, enabling the model to focus on task-relevant features.

[0054] In this embodiment, preferably, see Figure 2 , the inverse transform module includes:

[0055] The inverse transform unit transforms the second frequency domain representation Convert to the spatial domain to obtain the first spatial domain representation Expressed as:

[0056]

[0057] in, The frequency domain representation is converted to the spatial domain, preferably but not limited to using an inverse discrete Fourier transform method.

[0058] Activation function unit, representing the first spatial domain Perform activation function processing to obtain the second spatial domain representation σ(·) is an activation function, preferably but not limited to a GLUE function.

[0059] The residual connection unit represents the second spatial domain and the output features of the first convolution unit of the kth adaptive frequency selection module Perform residual connection to obtain the input features of the first upsampling unit of the kth adaptive frequency selection module

[0060]

[0061] In a preferred embodiment, the visual features are set to include local patch embedding ε, which is a problem that existing object counting methods usually ignore the role of global CLS markers, resulting in poor fusion of global and local features of the query image and affecting the accuracy of object counting. p and the global CLS flag ε cls Global CLS flag ε clsThe CLS (Classification) global embedding of the visual encoder is a special token whose core function is to aggregate the global information of the query image. The CLS embedding gradually integrates the information of all local image patches through the self-attention mechanism of the visual encoder, and finally forms a vector that can represent the overall content of the image.

[0062] In this implementation, see Figure 2 and Figure 3 The fusion network includes a first three-stream attention fusion module, which performs cross-attention processing on the local patch embedding, the global CLS tag and the text embedding to obtain the first combined feature. The fusion feature includes the first combined feature. At this time, the first combined feature is used as the input feature of the first adaptive frequency selection module (AFS Block). The specific structure of the fusion network (Feature Interaction) at this time only includes the attached Figure 2 The solid line part in .

[0063] In this implementation, see Figure 3 ,The first three-stream attention fusion module includes:

[0064] First Cross Attention Unit ( Figure 3 Cross-Attention on the left side of the figure) embeds ε with local patches p For query Q, label ε with global CLS cls Perform cross attention operation on key K and value V to obtain the first cross attention feature

[0065] Second Cross Attention Unit ( Figure 3 Cross-Attention on the right side of the figure) embeds ε with local patches p For query Q, embed ε with text t Perform cross attention operation on key K and value V to obtain the second cross attention feature

[0066] The first weighted fusion unit embeds ε into the local patch p , the first cross attention feature and the second cross-attention feature Perform weighted fusion to obtain the first combined feature ε mix :

[0067]

[0068] Among them, α represents the first fusion weight, which is learned during model training and is used to balance contributions. Its value range is [0,1].

[0069] In a preferred embodiment, in order to enrich the features of the decoding process and improve the model's ability to accurately locate and count specified objects, see the attached Figure 2 The dotted part of the fusion network (Feature Interaction) in the figure also includes:

[0070] The upsampling processing module performs upsampling processing on the first combined feature output by the first three-stream attention fusion module to obtain the upsampled combined feature ε mix-up .

[0071] The second and third stream attention fusion modules are used to combine the upsampled features ε mix-up , global CLS flag ε cls and text embedding ε t Perform cross attention processing to obtain the second combined feature ε' mix , at this time, the fusion feature includes the first combined feature ε mix and the second combined feature ε' mix , the first combined feature ε mix As the input features of the first adaptive frequency selection module.

[0072] See Figure 2 In the embodiment, the decoder further comprises an adding unit located after the k-th adaptive frequency selection module, the adding unit adding the output feature of the k-th adaptive frequency selection module and the second combined feature ε' mix Perform an addition process to obtain added features, and use the added features as input features for the k+1th adaptive frequency selection module (k+1≤K) or the input features of the output layer. Preferably, the addition unit is located between the first adaptive frequency selection module and the second adaptive frequency selection module, so that the added second combined features are continuously refined by subsequent multi-stage adaptive frequency selection modules to improve the accuracy of the density estimation map. When the addition unit is located after the last adaptive frequency selection module, the added features are used as input features for the output layer.

[0073] In this embodiment, the fusion network (Feature Interaction) includes a solid line part and a dotted line part. For details, please refer to Figure 2 The upsampling processing module includes a cascaded second convolution unit and a second upsampling unit, the input end of the second convolution unit is connected to the output end of the first three-stream attention fusion module, and the output end of the second convolution unit is connected to the input end of the second upsampling unit. The second upsampling unit is preferably but not limited to 2x upsampling.

[0074] In this embodiment, preferably, the second three-stream attention fusion module adopts a network structure similar to that of the first three-stream attention fusion module, specifically including:

[0075] The third cross attention unit The above sampling combination features ε mix-up For query Q, label ε with global CLS cls Perform cross attention operation on key K and value V to obtain the third cross attention feature

[0076] Fourth Cross Attention Unit The above sampling combination features ε mix-up For query Q, embed ε with text t Perform cross attention operation on key K and value V to obtain the fourth cross attention feature

[0077] The second weighted fusion unit is used to combine the upsampled features ε mix-up , the third cross attention feature and the fourth cross-attention feature Perform weighted fusion to obtain the second combined feature ε' mix :

[0078]

[0079] Among them, β represents the second fusion weight, which is learned during model training and is used to balance contributions. Its value range is [0,1].

[0080] The present invention also discloses an object counting device using frequency feature enhanced text guidance, which is used to implement the above-mentioned object counting method using frequency feature enhanced text guidance. In a preferred embodiment, the device includes:

[0081] Visual feature acquisition module, inputs the query image to the visual encoder to obtain visual features;

[0082] The text embedding acquisition module inputs the object description text into the text encoder to obtain the text embedding;

[0083] The fusion feature acquisition module uses the fusion network to fuse visual features and text features to obtain fusion features;

[0084] The decoding module inputs the fused features into the decoder to obtain a density estimation map of the object. The decoder includes K cascaded adaptive frequency selection modules and 1 output layer, where K and k are both positive integers, k∈[1,K]. The kth adaptive frequency selection module includes a first convolution unit, an adaptive frequency selector, and a first upsampling unit connected in sequence; wherein the adaptive frequency selector of the kth adaptive frequency selection module filters the amplitude spectrum and phase spectrum of the output features of the first convolution unit in the frequency domain, respectively, and converts the filtering results to the spatial domain.

[0085] In this embodiment, the visual feature acquisition module, the text embedding acquisition module, the fusion feature acquisition module and the decoding module correspond one-to-one to steps S1, S2, S3 and S4 in the above-mentioned object counting method using frequency features to enhance text guidance, and are not repeated here.

[0086] The following describes the training and verification process of the model based on the method for object counting using frequency features to enhance text guidance provided by the present invention.

[0087] 1. Build a model (denoted as FSCNet model) that implements the object counting method guided by frequency features enhancement:

[0088] See Figure 2 The model includes the INOv2 ViTB / 14 pre-trained visual encoder and the Googlebert-base-uncased pre-trained text encoder of the encoding part, freezes the network parameters of the pre-trained visual encoder and the pre-trained text encoder, and the fusion network includes the first three-stream attention fusion module, the upsampling processing module and the second three-stream attention fusion module. Figure 2 The decoder network (Decode) shown.

[0089] 2. Training parameter settings

[0090] In order to balance the importance of each stream in the first three-stream attention fusion module and the second three-stream attention fusion module, the learnable parameters of the first fusion weight α and the second fusion weight β are initialized to 0.5.

[0091] Dataset preparation. To evaluate the counting ability of the model, the FSC-147 dataset was used. This is the first large-scale dataset designed for category-independent counting, containing 6,135 diverse images from 147 categories. To evaluate the generalization ability of the model, the CARPK dataset was used. It contains 1,448 bird's-eye views of parking lots with a total of 89,777 cars. The images in the dataset were resized to 384×384, and data augmentation was applied according to the CLIP-Count protocol for consistent comparison. A point annotation map was pre-annotated for each image in the dataset. The center of each object is marked with a point in the point annotation map, and a corresponding object description text was generated for each image.

[0092] Evaluation Metrics: The mean absolute error (MAE) and root mean square error (RMSE) are used to evaluate model performance. The loss function is constructed by the weighted sum of the mean absolute error (MAE) and the root mean square error (RMSE).

[0093] Training settings: The model is trained using the AdamW optimizer with a learning rate of 1×10 -4 and weight decay is 1×10 -2 A StepLR scheduler is used to reduce the learning rate by a decay factor of 0.33 every 100 epochs. Training is performed on an NVIDIA RTX4090 GPU for 200 epochs with a batch size of 32.

[0094] 3. Model training

[0095] The constructed FSCNet model is trained using the dataset, the point annotation map for each image in the dataset, and the object description text according to the training parameters set above. During training, a loss function is calculated, and model parameters other than the visual encoder and text encoder in the FSCNet model, such as the amplitude weight, amplitude bias, phase weight, phase bias, first fusion weight α, and second fusion weight β, are optimized by gradient descent based on the loss function. Training is terminated when the training stop condition is met, and a trained FSCNet model is obtained. The training stop condition is preferably, but not limited to, reaching a preset maximum number of training times or convergence of the loss function value.

[0096] 4. Comparative verification of the object counting ability of the trained FSCNet model

[0097] Table 1 below shows a quantitative comparison of the zero-shot object counting performance of the FSCNet model trained with this method against state-of-the-art methods on the FSC-147 dataset. FSCNet outperforms previous methods across various evaluation metrics, particularly in validation and test RMSE. Leveraging a context-guided approach and a robust model architecture, FSCNet achieves significant improvements in the zero-shot setting, achieving a 19.75% average MAE improvement and a 39.86% average RMSE improvement over the previous best-performing method, VA-Count.

[0098] Notably, FSCNet achieves performance close to the few-shot method LOCA, which is often considered an upper bound for text-guided methods. This result highlights the effectiveness of FSCNet in handling various counting tasks without requiring additional visual samples, thus narrowing the gap between zero-shot and few-shot methods.

[0099] Table 1 Quantitative comparison results of zero-shot counting performance on the FSC-147 dataset

[0100]

[0101] To evaluate FSCNet's generalization ability on new datasets, experiments were conducted on the CARPK dataset, with quantitative results shown in Table 2. For fair comparison, FSCNet, like previous methods, was trained on the FSC-147 dataset and evaluated directly on the CARPK dataset without any fine-tuning. The experimental results show that FSCNet achieves a MAE of 10.08 and an RMSE of 12.20, significantly outperforming existing object counting methods and demonstrating its strong generalization ability across diverse datasets.

[0102] Table 2 Cross-dataset evaluation on the CARPK dataset

[0103]

[0104]

[0105] The qualitative results on the FSC-147 dataset are visualized by overlaying the predicted density estimate map on the input query image, as shown in Figure 4 (a). These visualizations highlight the ability of the FSCNet model to accurately capture the spatial distribution of objects in different scenes. In addition, Figure 4(b) shows qualitative results of the FSCNet model on the CARPK dataset, demonstrating the effectiveness of the object counting method proposed in this application in localizing and counting objects of various categories, shapes, sizes, and densities. The density estimation plot demonstrates the robustness of the object counting method proposed in this application, effectively handling challenging scenarios such as overlapping objects and variations in object scale and appearance. These results emphasize the versatility of the object counting method proposed in this application across different datasets and object distributions.

[0106] It can be seen that the frequency selective counting network (FSCNet) introduced in the object counting method and device provided by this application solves the key limitations of existing methods by utilizing spatial and frequency domain features. By integrating the three-stream attention fusion module (TSAFM) and the adaptive frequency selector (AFS), FSCNet effectively combines global, local and text features, while dynamically emphasizing the frequency components related to the counting task. Experimental results on the FSC-147 and CARPK benchmarks show that FSCNet not only achieves state-of-the-art accuracy, but also improves robustness and adaptability in different scenarios. These advances highlight the potential of frequency domain analysis and multimodal fusion in improving counting accuracy, and establish FSCNet as a new benchmark for future text-guided object counting research.

[0107] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "example," "specific example," "one implementation," "a preferred implementation," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0108] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A method for object counting using frequency features to enhance text guidance, characterized in that: include: Input the query image to the visual encoder to obtain visual features; Input object description text to text encoder to obtain text embedding; Using fusion network to fuse visual features and text features to obtain fusion features; The fused features are input into a decoder to obtain a density estimation map of the object. The decoder includes K cascaded adaptive frequency selection modules and an output layer, where K and k are both positive integers, k∈[1,K]. The kth adaptive frequency selection module includes a first convolution unit, an adaptive frequency selector, and a first upsampling unit connected in sequence. The adaptive frequency selector of the kth adaptive frequency selection module filters the amplitude spectrum and phase spectrum of the output features of the first convolution unit in the frequency domain, respectively, and converts the filtering results into the spatial domain. The adaptive frequency selector of the kth adaptive frequency selection module includes: a transform unit, transforming the output feature of the first convolution unit in the kth adaptive frequency selection module into the frequency domain to obtain a first frequency domain representation; a decomposition unit, decomposing the first frequency domain representation into an amplitude spectrum and a phase spectrum; A weighting unit performs weighted processing on the amplitude spectrum and the phase spectrum to obtain a weighted amplitude spectrum and a weighted phase spectrum; a synthesis unit, combining the weighted amplitude spectrum and the weighted phase spectrum to obtain a second frequency domain representation; The inverse transformation module performs spatial domain transformation and residual processing on the second frequency domain representation to obtain the input features of the first upsampling unit.

2. The object counting method using frequency feature enhanced text guidance as claimed in claim 1, characterized in that: The weighting unit performs filtering processing on the amplitude spectrum using the amplitude weight and the amplitude offset to obtain a weighted amplitude spectrum, and performs filtering processing on the phase spectrum using the phase weight and the phase offset to obtain a weighted phase spectrum.

3. The object counting method using frequency feature enhanced text guidance as claimed in claim 1, characterized in that: The inverse transformation module includes: an inverse transform unit, converting the second frequency domain representation into the spatial domain to obtain a first spatial domain representation; an activation function unit, performing an activation function on the first spatial domain representation to obtain a second spatial domain representation; The residual connection unit performs a residual connection on the second spatial domain representation and the output features of the first convolution unit of the k-th adaptive frequency selection module to obtain the input features of the first upsampling unit of the k-th adaptive frequency selection module.

4. The object counting method using frequency feature enhanced text guidance according to any one of claims 1 to 3, characterized in that: The visual features include local patch embedding and global CLS labeling; The fusion network includes a first three-stream attention fusion module, which performs cross-attention processing on local patch embedding, global CLS tag and text embedding to obtain a first combined feature, and the fusion feature includes the first combined feature.

5. The object counting method using frequency feature enhanced text guidance as claimed in claim 4, characterized in that: The first three-stream attention fusion module includes: The first cross-attention unit uses the local patch embedding as the query and the global CLS tag as the key and value to perform cross-attention operation to obtain the first cross-attention feature; The second cross-attention unit uses the local patch embedding as the query and the text embedding as the key and value to perform cross-attention operation to obtain the second cross-attention feature; The first weighted fusion unit performs weighted fusion on the local patch embedding, the first cross-attention feature and the second cross-attention feature to obtain a first combined feature.

6. The object counting method using frequency feature enhanced text guidance as claimed in claim 4, characterized in that: The fusion network further includes: An upsampling processing module performs upsampling processing on the first combined feature output by the first three-stream attention fusion module to obtain an upsampled combined feature; The second three-stream attention fusion module performs cross-attention processing on the upsampled combined features, the global CLS tag, and the text embedding to obtain a second combined feature, wherein the fused feature includes the first combined feature and the second combined feature, and the first combined feature is used as the input feature of the first adaptive frequency selection module; The decoder also includes an adding unit located after the kth adaptive frequency selection module, which adds the output features of the kth adaptive frequency selection module and the second combined features to obtain an added feature, and uses the added feature as the input feature of the k+1th adaptive frequency selection module or the input feature of the output layer.

7. The object counting method using frequency feature enhanced text guidance as claimed in claim 6, characterized in that: The second and third stream attention fusion modules include: The third cross attention unit uses the above sampled combined features as the query and the global CLS tag as the key and value to perform cross attention operation to obtain the third cross attention feature; The fourth cross attention unit uses the above sampled combined features as query and the text embedding as key and value to perform cross attention operation to obtain the fourth cross attention feature; The second weighted fusion unit performs weighted fusion on the upsampled combined feature, the third cross-attention feature and the fourth cross-attention feature to obtain the second combined feature.

8. The object counting method using frequency feature enhanced text guidance according to claim 6 or 7, characterized in that: The upsampling processing module includes a cascaded second convolution unit and a second upsampling unit.

9. An object counting device using frequency feature enhanced text guidance, used to implement the object counting method using frequency feature enhanced text guidance according to any one of claims 1 to 8, characterized in that: include: Visual feature acquisition module, inputs the query image to the visual encoder to obtain visual features; The text embedding acquisition module inputs the object description text into the text encoder to obtain the text embedding; The fusion feature acquisition module uses the fusion network to fuse visual features and text features to obtain fusion features; The decoding module inputs the fused features into the decoder to obtain a density estimation map of the object. The decoder includes K cascaded adaptive frequency selection modules and 1 output layer, where K and k are both positive integers, k∈[1,K]. The kth adaptive frequency selection module includes a first convolution unit, an adaptive frequency selector, and a first upsampling unit connected in sequence; wherein the adaptive frequency selector of the kth adaptive frequency selection module filters the amplitude spectrum and phase spectrum of the output features of the first convolution unit in the frequency domain, respectively, and converts the filtering results to the spatial domain.

Images