Visual Text-Based Weakly Supervised Flock Counting Method and Device for Bird Flock Monitoring

Abstract

This invention discloses a visual text-based weakly supervised cluster counting method and apparatus for bird flock monitoring, comprising: Step 1, designing an ordered digit sequence embedded in a text prompt template, generating text features and classifier weights using a frozen text encoder; inputting a bird flock image to an original image encoder to obtain multi-scale visual features; Step 2, fusing the visual features and classifier weights through a learnable feature adapter to generate enhanced visual features; Step 3, fusing the enhanced visual features and text features using the cross-attention mechanism of a multi-scale information interaction module to convey textual semantic information and generate multi-modal fusion features; Step 4, training a model based on multi-modal ranking loss, calculating the similarity matrix between the fusion features and text features, optimizing image patch and sequence matching through loss and fine-tuning the image encoder to obtain an optimized encoder; Step 5, employing a multi-stage filtering strategy in the inference stage: first, removing background image patches using the original encoder, then using the optimized encoder to predict and accumulate the number of birds in the filtered image patches. This invention effectively improves the accuracy of bird flock counting.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of computer vision and wetland ecological monitoring, and in particular to a visual text-based weakly supervised cluster counting method and apparatus for bird flock monitoring. Background Technology

[0002] As key indicator species of wetland ecosystems, wetland birds can directly reflect the environmental quality and biodiversity level of wetland habitats through their numbers and distribution characteristics. Therefore, wetland bird population statistics play an important role in wetland environmental monitoring and management.

[0003] Currently, the mainstream method for wetland bird population statistics is still manual counting, encompassing traditional methods such as sampling point method, transect method, and call survey method. These methods heavily rely on manual field surveys and observer experience, and face many challenges in practical application. On the one hand, harsh field environments (such as complex terrain and extreme weather) severely restrict the statistical work; on the other hand, manual counting is arduous, inefficient, and the results are easily affected by subjective factors, making it difficult to meet the needs of dynamic wetland environmental monitoring.

[0004] With the development of computer technology and 3S technology, image-based bird flock counting methods have emerged. For example, Some researchers used unmanned aerial vehicle (UAV) systems to acquire high-resolution imagery and combined it with multispectral image classification and geographic information system (GIS) technology to achieve spatial analysis and counting of bird populations in seagull breeding grounds. Gottschalk et al., on the other hand, used GIS technology and Landsat satellite remote sensing data to analyze the distribution and population of birds in plains areas. However, these methods rely on high-quality spatial data and are greatly affected by climate and environmental factors, making them less suitable for wetland scenarios with complex terrain and variable weather.

[0005] The rise of deep learning technology has spurred the emergence of numerous deep learning-based bird counting methods, primarily including detection-based and density map-based algorithms. Akcay et al. used a detection algorithm combined with a Faster R-CNN model to achieve automatic bird counting in wild scenes; Wang et al. improved bird counting accuracy by generating density maps through a network incorporating morphological prior knowledge. However, these methods suffer from high requirements for target bounding box annotation and point-level annotation, and the annotation information cannot effectively reflect bird size and posture, thus facing the dual challenges of accuracy and efficiency in practical applications.

[0006] In recent years, visual-language models have made groundbreaking progress in the field of visual representation. Compared with the pre-trained model CLIP, it has demonstrated strong cross-modal transfer capabilities and performed excellently in tasks such as object detection, segmentation, and generation. Subsequent models such as CoOp and CoCoOp have optimized downstream task performance by designing learnable text prompts; Jiang et al. used natural language prompts to guide the model to query the number of targets, while CrowdCLIP reduced the cost of manual annotation by constructing sorted text prompts to match images. These research results highlight the application potential of collaborative training of visual and textual information in specific tasks and provide new technical ideas for tackling the challenge of wetland bird flock counting in complex backgrounds. Summary of the Invention

[0007] The purpose of this invention is to provide a visual text weakly supervised cluster counting method and apparatus for bird flock monitoring. It uses a visual language pre-trained model for bird flock counting and enhances the model's text-image interaction capability by using a visual text feature vector fusion method, thereby alleviating the problems of bird postures being variable, multi-scale, and complex against backgrounds.

[0008] To achieve the above objectives, this invention provides a visual text-based weakly supervised cluster counting method for bird flock monitoring, comprising:

[0009] Step 1: Based on the natural mapping relationship between bird flock size and image patch size, design an ordered sequence of numbers, embed the sequence of numbers into a preset text prompt template, and generate a text feature vector R and classifier weights L using a frozen text encoder. i Where i is the category number in the text prompt template;

[0010] The preprocessed bird flock image m is input into the original image encoder E1 to generate a multi-scale visual feature vector G.

[0011] Step 2, combine the visual feature vector G with the classifier weights L i The input is a learnable feature adapter, which generates an enhanced visual feature vector F through nonlinear mapping and residual connections;

[0012] Step 3: The visual feature vector F and the text feature vector R are fused. The text semantic information of the text feature vector R is transferred to the visual feature vector F at different scales through the cross-attention mechanism of the multi-scale information interaction module to generate multi-modal fusion feature I.

[0013] Step 4: Train the model based on the multimodal ranking loss function, calculate the similarity matrix between the multimodal fusion feature I and the text feature vector R, and optimize the matching between the i-th image patch and the digit sequence through bottom-up loss to obtain the optimized similarity matrix. Fine-tune the original image editor to obtain the optimized image encoder E2.

[0014] Step 5: In the reasoning stage, a multi-stage screening strategy is adopted. In the classification stage, the original image encoder E1 is used to remove redundant background image blocks, and in the counting stage, the optimized image encoder E2 is used to predict the number of the screened image blocks. The total number of birds in the i-th image block is then accumulated.

[0015] The present invention also provides a visual text-based weakly supervised flock counting device for bird flock monitoring, comprising:

[0016] The digital sequence text design module is used to design an ordered sequence of numbers based on the natural mapping relationship between bird flock numbers and image patch size. This sequence is then embedded into a preset text prompt template, and a frozen text encoder generates a text feature vector R and classifier weights L. i Where i is the category number in the text prompt template;

[0017] The visual feature vector generation module is used to input the preprocessed bird flock image m into the original image encoder E1 to generate a multi-scale visual feature vector G.

[0018] Learnable feature adapter, which accepts visual feature vector G and classifier weights L i An enhanced visual feature vector F is generated through nonlinear mapping and residual connections;

[0019] The multi-scale information interaction module is used to fuse visual feature vector F with text feature vector R. Through a cross-attention mechanism, the text semantic information of text feature vector R is transferred to visual feature vector F at different scales to generate multimodal fusion feature I.

[0020] The image encoder optimization module is used to train the model based on the multimodal ranking loss function, calculate the similarity matrix between the multimodal fusion feature I and the text feature vector R, and optimize the matching between the i-th image patch and the digit sequence through bottom-up loss to obtain the optimized similarity matrix. The original image editor is then fine-tuned to obtain the optimized image encoder E2.

[0021] The bird count calculation module employs a multi-stage screening strategy during the inference stage, uses the original image encoder E1 to remove redundant background image blocks during the classification stage, and uses the optimized image encoder E2 to predict the number of filtered image blocks during the counting stage, accumulating the total number of birds in the i-th image block.

[0022] The present invention has the following advantages due to the adoption of the above technical solutions:

[0023] This invention proposes a weakly supervised bird flock counting method based on visual-text fusion, which does not require target annotation. It does not rely on target detection and only utilizes count-level information as a supervision signal, thus reducing annotation costs. A learnable feature adapter is introduced into the visual-language pre-trained model to fuse visual feature vectors with textual information, guiding the model to focus on target features relevant to the counting task. A residual connection mechanism is used to integrate original and new knowledge. Furthermore, a multi-scale information interaction module is designed to introduce textual feature vectors into visual feature vectors at different scales, generating fused features containing rich semantic information. Attached Figure Description

[0024] Figure 1 This is a framework diagram of a visual text-based weakly supervised cluster counting method for bird flock monitoring provided in an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of the structure of the learnable feature adapter provided in an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of the structure of the multi-scale information interaction module provided in the embodiment of the present invention, wherein (a) is a schematic diagram of the multi-scale information interaction module; and (b) is a cross-attention mechanism module.

[0027] Figure 4 This is a flowchart of the multi-stage screening strategy provided in the embodiments of the present invention;

[0028] Figure 5 This is a schematic diagram of the Bird-Count dataset provided in an embodiment of the present invention;

[0029] Figure 6 This is a schematic diagram of the annotation interface provided in an embodiment of the present invention;

[0030] Figure 7 These are experimental results of the Wetland-Bird-Count dataset provided in this embodiment of the invention, wherein (a) is a scene with uneven distribution; (b) is a multi-scale scene; and (c) is a scene with poor lighting.

[0031] Figure 8 This is a loss function curve provided in an embodiment of the present invention. Detailed Implementation

[0032] In the accompanying drawings, the same or similar reference numerals are used to denote the same or similar elements or elements having the same or similar functions. The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0033] In the description of this invention, the terms "center," "longitudinal," "lateral," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting the scope of protection of this invention.

[0034] like Figure 1 As shown, the visual text-based weakly supervised cluster counting method for bird flock monitoring provided in this embodiment of the invention includes:

[0035] Step 1: Based on the natural mapping relationship between bird population size and image patch size (smaller image patches typically correspond to fewer birds, while larger image patches correspond to more birds), design an ordered sequence of numbers. Embed this sequence into a pre-defined text prompt template. Then, generate a text feature vector R and classifier weights L using a frozen text encoder. i Where i is the class number in the text prompt template. The image patch is obtained by cropping the bird flock image m, and the specific method is described in the original image preprocessing section below.

[0036] This embodiment transforms a regression problem into a classification problem by embedding different numbers as category labels into a descriptive text prompt template and matching them with image blocks of different sizes. Based on the natural mapping relationship between bird flock numbers and images, the numbers in the sequence are arranged from smallest to largest, thus ensuring consistent ordering in both the latent space of the text prompt and the visual representation.

[0037] The text prompt template for the counting process is set to: "There are [class] birds in the picture", where the class in the text prompt template is represented by a set of regular numerical sequences, as shown in Equation (1):

[0038] h=[h0,h0+k,...,h0+(N-1)k] (1)

[0039] Where h0, k, and N represent the starting number, counting interval, and total number of categories, respectively.

[0040] This invention selects Transformer as the text encoder, and defines the obtained text prompt feature vector R as (2):

[0041] R = [R1, R2, ..., R] j ,...,R N (2)

[0042] in, R j Let N represent the j-th text feature vector, N represent the total number of text feature vectors, and D represent the dimension of each feature vector.

[0043] Step 1 also inputs the preprocessed bird flock image m into the original image encoder E1 to generate a multi-scale visual feature vector G.

[0044] The preprocessing methods for bird flock images m specifically include:

[0045] bird images As input, it is cropped into a group of image patches with the same center point. Here, H, W, and C represent the height, width, and number of channels of the image, respectively.

[0046] The set of image patches is denoted as {B1, B2, ..., B}. i ,...,B P}, cropped image patch Where P is the number of cropped blocks, 1≤i≤P. These image blocks are nested, following a rule of ascending size, and each block contains the target in varying quantities.

[0047] In this embodiment, a pre-trained ViT / B-16 network is used as the original image encoder E1. The ViT / B-16 network divides the pre-processed bird flock image into multiple 16×16 blocks. After passing through the linear projection layer, a visual feature vector is obtained. Then, the visual feature vector is position-encoded and input into the encoding layer for calculation to obtain the visual feature vector G as shown in (3).

[0048] G = [G1, G2, ..., G N (3)

[0049] in,

[0050] Step 2, combine the visual feature vector G with the classifier weights L i The input is a learnable feature adapter, which generates an enhanced visual feature vector F through nonlinear mapping and residual connections.

[0051] In one embodiment, during the training phase, the text encoder is frozen, generating a text cue feature vector R, which is then encoded into classifier weights L. i 'i' represents the class number in the text prompt template, here indicating N distinct numbers in the digit sequence 'h'. The classifier weights guide the visual text-based weakly supervised cluster counting device for bird flock monitoring to focus on target features relevant to the counting task, thereby improving counting accuracy.

[0052] like Figure 2 As shown, the learnable feature adapter includes an image branch adapter V. f (G) and text branch adapter V t (L).

[0053] Visual feature vector G and classifier weights L i Input to image branch adapter V respectively f (G) and text branch adapter V t A linear transformation is performed in (G), and feature enhancement is achieved through the activation function ReLU and residual connections. The visual and text branch adapters are represented by equations (4) and (5), respectively:

[0054]

[0055] F=[θV f (G) T +(1-θ)G]·[γV t (L) T +(1-γ)L] (6)

[0056] Among them, G T L represents the matrix transpose of the visual feature vector G. T This represents the matrix transpose of the classifier weights L. and Both are learnable parameters, and θ and γ are residual rates representing the degree of residual connectivity.

[0057] Step 3: The visual feature vector F and the text feature vector R are fused. The text semantic information of the text feature vector R is transferred to the visual feature vector F at different scales through the cross-attention mechanism of the multi-scale information interaction module to generate multimodal fusion feature I.

[0058] Bird flocks exhibit multi-scale variations in images, and traditional Transformers lack inductive biases to effectively handle features at different scales, leading to inaccurate flock estimation. To address this, this invention designs a multi-scale information interaction module that combines self-attention and cross-attention mechanisms. Compared to other attention mechanisms, the cross-attention mechanism can transfer textual information between visual feature vectors at different scales, enhancing the network's semantic understanding of images at different scales and thus alleviating the multi-scale problem.

[0059] like Figure 3 As shown, in one embodiment, in step 3, the multi-scale information interaction module includes two similar structures, which are divided into a first layer structure and a second layer structure as follows.

[0060] The first layer consists of a first multi-head self-attention (MHSA) unit, a first multi-head cross-attention (MHCA) unit, a first feedforward neural network, and a convolutional layer with skipping.

[0061] First, the visual feature vector F is input into the first multi-head self-attention unit to obtain the visual feature vector M1.

[0062] Then, both M1 and the text feature vector R are input into the first multi-head cross-attention unit. The first multi-head cross-attention unit uses M1 as the query vector and the text feature vector R as the key and value vector to calculate the attention scores of the key and value vectors of the text feature vector R in the visual feature vector M1 at different scales, thereby obtaining the first fused visual feature vector X1. This can enhance the adaptability and processing ability of the weakly supervised cluster counting network for wetland bird flock monitoring to features at different scales.

[0063] Furthermore, X1 is input into the first feedforward neural network, which is implemented using a multilayer perceptron (MLP) and outputs a feature vector C.

[0064] Finally, C and the low-resolution features J output by the convolutional layer with skip connections are processed through residual connections to obtain the feature vector Y, which is then fed into the second layer structure. The convolutional layers guide the model to capture local features. Residual connections are used to progressively transfer information and gradients within the deep network.

[0065] The second layer consists of a bilinear interpolation unit, a second multi-head self-attention unit, a second multi-head cross-attention unit, and a second feedforward neural network. Y is doubled in resolution via the bilinear interpolation unit to obtain fine-grained image features W. These fine-grained image features W are then processed by the second multi-head self-attention unit to obtain a visual feature vector M2. The second multi-head self-attention unit uses the visual feature vector M2 as the query vector.

[0066] Next, the query vector M2 and the text feature vector R are processed through the second multi-head cross attention. The second multi-head cross attention unit uses the text feature vector R as the key and value vector to calculate the attention score of the key and value vectors of the text feature vector R in M2, and obtains the second fused visual feature vector X2.

[0067] Finally, X2 is input into the second feedforward neural network, which is implemented using a multilayer perceptron (MLP) to obtain high-resolution multimodal fusion features I.

[0068] Step 4: Train the model based on the multimodal ranking loss function, calculate the similarity matrix between the multimodal fusion feature I and the text feature vector R, and optimize the matching between the i-th image patch and the digit sequence through bottom-up loss to obtain the optimized similarity matrix. Fine-tune the original image editor to obtain the optimized image encoder E2.

[0069] By calculating the inner product of the visual feature vector I and the text feature vector R, the similarity matrix score is obtained, and a similarity matrix X is generated, which is defined by equation (7):

[0070] X = I i ·R j =[x i,j (7)

[0071] in, 1≤i≤N, 1≤j≤N, I i Let R represent the visual feature vector of the i-th image patch. j Let x represent the j-th text feature vector. i,j Let represent the inner product of the i-th image patch and the j-th text feature vector.

[0072] To accurately determine the size order of image patches and simultaneously predict the number of bird flocks, this invention uses a multimodal ranking loss function. Based on the diagonal of the similarity matrix X, the multimodal ranking loss function is calculated from bottom to top. The multimodal ranking loss function is Loss(x) i′,i ,x i,j ) is defined as Equation (8):

[0073] Loss(x i′,i ,x i,j ) = max(0, x i′,i -x i,i (8)

[0074] Where, x i,j x i′,i R represents the i-th image patch and the j-th text feature vector, respectively. j The inner product of the i′-th image patch and the j-th text feature vector R j The inner product of the similarity matrix X is given by 1 ≤ i′ ≤ i ≤ N. The multimodal ranking loss ensures that the similarity values ​​on the diagonal of the similarity matrix X are greater than those of other elements. The bottom-up calculation method guarantees that the similarity between the image patch and the corresponding numerically ordered text prompt is higher than that of other mismatched text prompts.

[0075] Step 5: In the reasoning stage, a multi-stage screening strategy is adopted. In the classification stage, the original image encoder E1 is used to remove redundant background image blocks, and in the counting stage, the optimized image encoder E2 is used to predict the number of the screened image blocks. The total number of birds in the i-th image block is then accumulated.

[0076] In one embodiment, step 5, the multi-stage screening strategy includes:

[0077] In the first classification phase, image patches containing birds are filtered out using the first text prompt template of the text encoder;

[0078] In the second classification stage, the second text prompt template of the text encoder is used to filter out image patches containing bird heads;

[0079] The filtered image patches are input into the optimized image encoder for similarity matching, and the total number of predicted birds is obtained by accumulating the predicted number.

[0080] The test set images are cropped into regular Q×Q image patches, and the total number of targets is obtained by summing the predicted counts for each patch. To improve the accuracy of bird flock counting, a multi-stage screening strategy is adopted to achieve coarse-to-fine classification, consisting of a classification stage and a counting stage. The flowchart of the multi-stage screening strategy is as follows: Figure 4 (The multi-stage screening strategy can filter out image patches containing bird heads through two classifications, avoiding the influence of background and blurry image patches on the counting results. In the experiment, the three different text prompts all shared the same text encoder, and all images used the same text prompt template. This means that all text prompts share the same model parameters, and the text embedding vectors can be pre-computed and stored, avoiding the model repeatedly processing text prompts during each inference, thus ensuring the model's running efficiency.)

[0081] In the classification stage, the original CLIP image encoder E1 is used as the visual feature vector extraction network for two classifications. The first text prompt template is set to "The object is [class]", where [class] includes six categories: "bird", "weed", "tree", "lake", "road", and "ground", with the aim of removing redundant background image patches. The second text prompt template is set to "The objects are [class]", where [class] includes six categories: "bird heads", "birdbody", "bird wings", "bird legs", "the neck of birds", and "the shadow of birds", with the aim of filtering out image patches containing bird heads and removing blurry images of other parts of the birds.

[0082] During the counting phase, the first and second text prompt templates are the same as those used in the training phase. The image blocks that have been filtered through the above steps are input into the optimized image encoder E2. The visual feature vectors and text feature vectors are used to calculate the similarity, and finally the number of birds is obtained.

[0083] During the experiment, all three different text prompts shared the same text encoder, and all images used the same text prompt template. This means that all text prompts share the same model parameters, and the text embedding vectors can be pre-computed and stored, avoiding the model repeatedly processing text prompts during each inference, thus ensuring the model's running efficiency.

[0084] This invention also provides a visual text-based weakly supervised flock counting device for bird flock monitoring, comprising a digital sequence text design module, a visual feature vector generation module, a learnable feature adapter, a multi-scale information interaction module, an image encoder optimization module, and a bird flock count calculation module, wherein:

[0085] The digit sequence text design module is used to design an ordered digit sequence based on the natural mapping relationship between bird flock numbers and image patch size. This digit sequence is then embedded into a preset text prompt template, and a frozen text encoder generates a text feature vector R and classifier weights L. i Where i is the category number in the text prompt template;

[0086] The visual feature vector generation module is used to input the preprocessed bird flock image m into the original image encoder E1 to generate a multi-scale visual feature vector G.

[0087] Learnable feature adapter is used to accept visual feature vectors G and classifier weights L i An enhanced visual feature vector F is generated through nonlinear mapping and residual connections;

[0088] The multi-scale information interaction module is used to fuse visual feature vector F with text feature vector R. Through the cross-attention mechanism, the text semantic information of text feature vector R is transferred to visual feature vector F at different scales to generate multimodal fusion feature I.

[0089] The image encoder optimization module is used for model training based on the multimodal ranking loss function. It calculates the similarity matrix between the multimodal fusion feature I and the text feature vector R, and optimizes the matching between the i-th image patch and the digit sequence through bottom-up loss to obtain the optimized similarity matrix. The original image editor is then fine-tuned to obtain the optimized image encoder E2.

[0090] The bird count calculation module is used to employ a multi-stage screening strategy during the inference stage. During the classification stage, the original image encoder E1 is used to remove redundant background image blocks, and during the counting stage, the optimized image encoder E2 is used to predict the number of the screened image blocks. The total number of birds in the i-th image block is then accumulated.

[0091] Figure 1 The model framework introduces a learnable feature adapter to guide the model to focus on target features related to the counting task, and a multi-scale information interaction module is designed to alleviate the multi-scale problem.

[0092] In one embodiment, the learnable feature adapter includes:

[0093] Image Branch Adapter V f (G) and text branch adapter V t (L) performs linear transformations on the visual feature vector and classifier weights respectively, and achieves feature enhancement through the activation function ReLU and residual connections.

[0094]

[0095] Among them, G T L represents the matrix transpose of the visual feature vector G. T This represents the matrix transpose of the classifier weights L. and All parameters are learnable and are continuously updated during training via backpropagation. Newly generated knowledge and original information from the pre-training phase are integrated through residual connections to prevent overfitting in the visual-text weakly supervised cluster counting network for bird flock monitoring. The fused embedding F of the visual and text feature vectors is expressed as Equation (6):

[0096] F=[θV f (G) T +(1-θ)G]·[γV t (L) T +(1-γ)L] (6)

[0097] Where θ and γ are residual rates representing the degree of residual connectivity, both set to 0.4 during the experiment. f (G), V t (L) represent the information obtained after the visual feature vector and text feature vector are encoded, respectively, and the superscript T indicates matrix transpose. i The subscripts represent the classifier weights for different categories of words, and L here represents a general term.

[0098] The following section introduces the creation, experimental setup, and evaluation metrics of the Wetland-Bird-Count dataset, and compares and analyzes other population counting algorithms on the Wetland-Bird-Count dataset.

[0099] (I) Experimental Dataset

[0100] 1. Data Source

[0101] To facilitate research on vision-based bird flock counting, this invention constructs a wetland bird dataset named Wetland-Bird-Count. The data originates from the Yellow River Delta Nature Reserve and includes numerous rare bird species such as the Red-crowned Crane, Oriental White Stork, Whooper Swan, and Chinese Merganser. Scenes depicted include mudflats, skies, rice paddies, and reed meadows—coastal wetland landscapes of the Yellow River Delta. The dataset contains 885 images, comprising 582 training images and 303 test images, totaling 12,362 labeled instances. All images have a resolution of 4000×3000 pixels. All images in this dataset were taken in the wetland reserve, and all subsequent labeled instances have been manually annotated and repeatedly verified. Figure 5 These are images from the dataset, containing different scenes and different species of birds.

[0102] 2. Data labeling

[0103] This invention uses CC Labeler software to annotate bird datasets. The annotation interface is shown below. Figure 6 As shown. The software provides point annotation and bounding box annotation methods. To provide consistent annotation, the following rules are followed when labeling: for bird objects that are fully visible or only partially occluded, their heads or bodies are labeled; for bird objects that are fully occluded or only partially exposed, they are not labeled. The method of this invention only uses count-level annotations as supervision signals and does not utilize their positional information.

[0104] (II) Experimental Design and Evaluation Indicators

[0105] All experiments were conducted on a Linux system (18.04) with an NVIDIA GTX 3090 GPU, using the PyTorch 1.11.3 framework. During the data loading phase, the batch size was set to 16. To conform to the CLIP input assumption, the cropped image size was resized to 224×224. The Q value for the test set was set to 3, and the total number of predicted image patches was obtained by summing them. The training epochs were set to 100, and P was set to 6. Stochastic gradient descent (SGD) was used as the gradient optimization method. The learning rate was set to 1×10⁻⁶. -5 The entire training process takes about 3 hours.

[0106] To evaluate the performance of the method proposed in this invention, mean absolute error (MAE) and mean squared error (MSE) are used as evaluation metrics to assess counting performance. MAE focuses on evaluating counting accuracy, while MSE focuses on evaluating the robustness of the algorithm. Their definitions are as follows:

[0107]

[0108] Where, N C C represents the number of images in the test set. i GT and C i Let represent the true value and the predicted number of the i-th image patch, respectively.

[0109] (III) Comparative Experiment

[0110] To verify the effectiveness of the proposed method, comparative experiments were conducted on the Wetland-Bird-Count dataset. In the comparative experiments, classic methods in the field of crowd counting were selected as the control group, divided into supervised and unsupervised methods, with consistent testing methods to objectively evaluate the effectiveness of the proposed method. The comparative methods selected in the experiments included:

[0111] 1) MCNN: The first to use multi-column convolutional networks to handle multi-scale problems in crowd counting.

[0112] 2) SwitchCNN: An improvement on traditional density estimation methods, it introduces an allocation mechanism to dynamically select convolutional network branches suitable for different regions.

[0113] 3) CSS-CCNN: Requires no manual annotation and uses prior constraints based on population distribution to predict results. CSS-CCNN-R indicates that this method randomly selects a value from a predefined set of levels for each cropped image patch.

[0114] 4) CrowdCLIP: A crowd counting method based on CLIP, which transforms the regression problem of counting into a classification problem by constructing sorting text prompts, reducing manual annotation.

[0115] Table 1 presents the comparative experimental results of different methods on the Wetland-Bird-Count dataset. The results show that the proposed method significantly outperforms other unsupervised methods, with MAE and MSE of only 45.2 and 54.2, respectively. Although the counting accuracy still lags behind supervised crowd counting methods, the MSE surpasses MCNN and SwitchCNN. Crowd counting methods for bird flocks can still be improved. The reason for this is the significant difference in appearance and posture between crowds and birds, making direct training of bird flock counting algorithms ineffective. The algorithm proposed in this invention exhibits better robustness, and from the perspective of reducing annotation costs, the proposed method has a clear advantage.

[0116] Table 1

[0117]

[0118] Figure 7 The results show some of the experimental findings. Sub-image (a) shows a scene with uneven bird distribution, sub-image (b) shows multi-scale phenomena, and sub-image (c) shows an environment with poor lighting in the dataset. In the prediction results, the image patch without numbers in the upper left corner indicates that this image patch was filtered out as background or did not contain bird heads and was not included in the final counting task.

[0119] To further validate the effectiveness of the proposed method, we conducted experiments on three publicly available crowd datasets: QNRF, ShanghaiTech, and UCF_CC_50. We also added three additional crowd counting algorithms—LSC-CNN, CLTR, and TransCrowd—for comparison.

[0120] 1) LSC-CNN: A classic crowd counting method based on object detection, which directly predicts the location and density of crowds.

[0121] 2) CLTR: An end-to-end crowd localization model that introduces the concept of long-distance relationship modeling and uses attention mechanisms to improve the accuracy of crowd counting.

[0122] 3) TransCrowd: A weakly supervised crowd counting model based on Transformer, which reduces the labeling cost by constructing a sorted sequence of numbers to count crowds.

[0123] Table 2

[0124]

[0125] Table 2 presents the comparative experimental results of different algorithms on the QNRF, ShanghaiTech, and UCF_CC_50 datasets, where "improvement" is compared with CrowdCLIP. Numerical experiments show that applying the method of this invention to crowd counting can improve counting accuracy. Although there is a significant gap compared to state-of-the-art crowd counting methods, this work can effectively alleviate the problem of dense crowd counting. The method of this invention shows significant improvements in the MSE metric on three of the datasets, indicating that the proposed method can improve the robustness of the model.

[0126] (iv) Ablation test

[0127] (1) Ablation experiments with different modules

[0128] This group of experiments investigated the impact of the feature adapter and multi-scale information interaction module on the experimental results. To verify the effectiveness of the improved method of this invention, ablation experiments were conducted on different modules on a bird dataset to evaluate the impact of each module on model performance.

[0129] Table 3 shows the ablation experiment results for different modules. Introducing both the learnable feature adapter and the multi-scale information interaction module improved counting accuracy to some extent, demonstrating that these two modules can reduce counting errors to a certain degree. Simultaneously introducing both modules resulted in the lowest MAE and MSE, proving that adding both modules together significantly enhances feature extraction and semantic understanding, thereby improving the accuracy of bird flock counting.

[0130] Table 3

[0131]

[0132] (2) Design methods for different number sequence text prompts

[0133] This experiment explores the impact of different digit sequence text cues on the experimental results. Table 4 shows the experimental results for different digit sequence designs on the Wetland-Bird-Count dataset. For the design of ordered digit sequence text cues, it can be observed that the highest accuracy is achieved when the initial count h0 is 10 and the counting interval k is 15. This indicates that the method of this invention can fine-tune the image encoder using ordered text cues and can study potential ordered representations from a language space with order.

[0134] Table 4

[0135]

[0136]

[0137] The design of digit sequence text can be chosen based on the characteristics of different datasets. For example, in dense crowd datasets, there are more image patches, so a larger initial count and counting interval can be used in the design. Future research could consider selecting appropriate digit sequences based on the characteristics of different datasets.

[0138] (3) Experimental results of fine-tuning different encoders

[0139] This group of experiments investigated the impact of fine-tuning different encoders on the experimental results. The experimental results are shown in Table 5, which shows the results of fine-tuning different encoders.

[0140] Table 5

[0141] ImageEncoder TextEncoder MAE MSE - - 59.0 81.5 √ - 45.2 54.2 - √ 63.4 70.0 √ √ 58.9 71.9

[0142] Using the original CLIP image encoder and text encoder parameters directly, the MAE and MSE are 59.0 and 85.4, respectively. Fine-tuning only the image encoder while freezing the text encoder significantly improves counting accuracy. However, fine-tuning the text encoder results in a noticeable performance decrease. The reason for this might be that fine-tuning the text encoder disrupts the knowledge learned from pre-training priors, failing to properly learn the relationship between bird feature mappings and the numerical ranking language space.

[0143] (4) Experimental results at different counting sites

[0144] Considering the significant occlusion in the images, counting the bird's body parts appears to be more effective from an intuitive perspective. To verify this more accurate method, experiments were conducted on different parts of the birds. Table 6 shows the results of counting the bird's head and body separately. Numerical experiments show that counting the bird's head is more accurate than counting the body, improving MAE by 27.8% and MSE by 12.4%.

[0145] Table 6

[0146] Part MAE MSE birdhead 45.2 54.2 birdbody 58.5 60.0

[0147] Figure 8 The graph shows the change in the loss function during the training phase. The graph shows that as the number of training rounds increases, the model's loss function gradually decreases. This indicates that the model achieves image and text ranking matching, that is, smaller images correspond to fewer texts, and larger images correspond to more texts, proving the effectiveness of the proposed method.

[0148] This invention utilizes count-level annotation information to achieve counting tasks, reducing annotation costs. To alleviate the problems of pose variations and complex backgrounds in bird flock counting tasks, this invention introduces a learnable feature adapter into the visual-language pre-trained model. Visual feature vectors, in conjunction with textual information, guide the model to highlight target features relevant to counting. Furthermore, a multi-scale information interaction module incorporating an attention mechanism is designed to fuse visual feature vectors and textual feature vectors at different scales, thereby enhancing the image-text interaction capability. Numerical experiments show that the counting accuracy of the proposed method is close to that of fully supervised crowd counting methods used for bird flock counting, and surpasses other weakly supervised and unsupervised methods. Future research will explore further applications of visual-language models in the field of target counting and apply the proposed method to small target cluster counting tasks for other targets.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Those skilled in the art should understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual text-based weakly supervised cluster counting method for bird flock monitoring, characterized in that, include: Step 1: Based on the natural mapping relationship between bird flock size and image patch size, design an ordered sequence of numbers, embed the sequence of numbers into a preset text prompt template, and generate a text feature vector R and classifier weights L using a frozen text encoder. i Where i is the category number in the text prompt template; The preprocessed bird flock image m is input into the original image encoder E1 to generate a multi-scale visual feature vector G. Step 2, combine the visual feature vector G with the classifier weights L i The input is a learnable feature adapter, which generates an enhanced visual feature vector F through nonlinear mapping and residual connections; Step 3: The visual feature vector F and the text feature vector R are fused. The text semantic information of the text feature vector R is transferred to the visual feature vector F at different scales through the cross-attention mechanism of the multi-scale information interaction module to generate multi-modal fusion feature I. Step 4: Train the model based on the multimodal ranking loss function, calculate the similarity matrix between the multimodal fusion feature I and the text feature vector R, and optimize the matching between the i-th image patch and the digit sequence through bottom-up loss to obtain the optimized similarity matrix. Fine-tune the original image editor to obtain the optimized image encoder E2. Step 5: In the reasoning stage, a multi-stage screening strategy is adopted. In the classification stage, the original image encoder E1 is used to remove redundant background image blocks, and in the counting stage, the optimized image encoder E2 is used to predict the number of the screened image blocks. The total number of birds in the i-th image block is accumulated. In step 3, the multi-scale information interaction module includes a first-layer structure and a second-layer structure: The first layer structure includes a first multi-head self-attention unit, a first multi-head cross-attention unit, a first feedforward neural network, and a convolutional layer with skipping. The visual feature vector F is input into the first multi-head self-attention unit to obtain the visual feature vector M1. Then, M1 and the text feature vector R are both input into the first multi-head cross-attention unit. The first multi-head cross-attention unit uses M1 as the query vector and the text feature vector R as the key and value vector to calculate the attention scores of the key and value vectors of the text feature vector R in the visual feature vector M1 at different scales, to obtain the first fused visual feature vector X1. Then, X1 is input into the first feedforward neural network to obtain the feature vector C. Finally, C is input into the convolutional layer with skipping to obtain J. C and the low-resolution features J output by the convolutional layer with jumps are processed through residual connections to obtain the feature vector Y, which is then fed into the second layer structure. The second layer structure includes a bilinear interpolation unit, a second multi-head self-attention unit, a second multi-head cross-attention unit, and a second feedforward neural network. Y is doubled in resolution by the bilinear interpolation unit to obtain fine-grained image features W. Then, W is passed through the second multi-head self-attention unit to obtain a visual feature vector M2. The second multi-head self-attention unit uses the visual feature vector M2 as a query vector. The query vector and the text feature vector R are passed through the second multi-head cross-attention unit. The second multi-head cross-attention unit uses the text feature vector R as a key and value vector to calculate the attention score of the key and value vectors of the text feature vector R in M2, thereby obtaining the second fused visual feature vector X2. Finally, X2 is input into the second feedforward neural network to obtain a high-resolution multimodal fused feature I.

2. The visual text-based weakly supervised cluster counting method for bird flock monitoring as described in claim 1, characterized in that, In step 1, the number sequence is generated as follows: Given a starting number h0, a counting interval k, and a total number of categories N, the number sequence h is expressed as equation (1): h=[h0,h0+k,...,h0+(N-1)k] (1).

3. The visual text-based weakly supervised cluster counting method for bird flock monitoring as described in claim 1, characterized in that, In step 2, the learnable feature adapter includes: Image Branch Adapter V f (G) and text branch adapter V t (L) performs linear transformations on the visual feature vector and classifier weights respectively, and achieves feature enhancement through the ReLU activation function and residual connections, as specifically expressed as: F=[θV f (G) T +(1-θ)G]·[γV t (L) T +(1-γ)L] (6) Among them, G T L represents the matrix transpose of the visual feature vector G. T This represents the matrix transpose of the classifier weights L. and Both are learnable parameters, and θ and γ are residual rates representing the degree of residual connectivity.

4. The visual text-based weakly supervised cluster counting method for bird flock monitoring as described in claim 1, characterized in that, In step 4, the multimodal ranking loss function Loss(x) i′,i ,x i,j ) is defined as Equation (8): Loss(x i′,i ,x i,j )=max(0,x i′,i -x i,i ) (8) Where, x i,j x i′,i R represents the i-th image patch and the j-th text feature vector, respectively. j The inner product of the i′-th image patch and the j-th text feature vector R j The inner product of , 1≤i′≤i≤N.

5. The visual text-based weakly supervised cluster counting method for bird flock monitoring as described in claim 1, characterized in that, In step 5, the multi-stage screening strategy includes: In the first classification phase, image patches containing birds are filtered out using the first text prompt template of the text encoder; In the second classification stage, the second text prompt template of the text encoder is used to filter out image patches containing bird heads; The filtered image patches are input into the optimized image encoder for similarity matching, and the total number of predicted birds is obtained by accumulating the predicted number.

6. A visual text-based weakly supervised flock counting device for bird flock monitoring, characterized in that, include: The digital sequence text design module is used to design an ordered sequence of numbers based on the natural mapping relationship between bird flock numbers and image patch size. This sequence is then embedded into a preset text prompt template, and a frozen text encoder generates a text feature vector R and classifier weights L. i Where i is the category number in the text prompt template; The visual feature vector generation module is used to input the preprocessed bird flock image m into the original image encoder E1 to generate a multi-scale visual feature vector G. Learnable feature adapter, which accepts visual feature vector G and classifier weights L i An enhanced visual feature vector F is generated through nonlinear mapping and residual connections; The multi-scale information interaction module is used to fuse visual feature vector F with text feature vector R. Through a cross-attention mechanism, the text semantic information of text feature vector R is transferred to visual feature vector F at different scales to generate multimodal fusion feature I. The image encoder optimization module is used to train the model based on the multimodal ranking loss function, calculate the similarity matrix between the multimodal fusion feature I and the text feature vector R, and optimize the matching between the i-th image patch and the digit sequence through bottom-up loss to obtain the optimized similarity matrix. The original image editor is then fine-tuned to obtain the optimized image encoder E2. The bird count calculation module is used to employ a multi-stage screening strategy in the inference stage, use the original image encoder E1 to remove redundant background image blocks in the classification stage, and use the optimized image encoder E2 to predict the number of the screened image blocks in the counting stage, and accumulate to obtain the total number of birds in the i-th image block. The multi-scale information interaction module includes a first-layer structure and a second-layer structure: The first layer structure includes a first multi-head self-attention unit, a first multi-head cross-attention unit, a first feedforward neural network, and a convolutional layer with skipping. The visual feature vector F is input into the first multi-head self-attention unit to obtain the visual feature vector M1. Then, M1 and the text feature vector R are both input into the first multi-head cross-attention unit. The first multi-head cross-attention unit uses M1 as the query vector and the text feature vector R as the key and value vector to calculate the attention scores of the key and value vectors of the text feature vector R in the visual feature vector M1 at different scales, to obtain the first fused visual feature vector X1. Then, X1 is input into the first feedforward neural network to obtain the feature vector C. Finally, C is input into the convolutional layer with skipping to obtain J. C and the low-resolution features J output by the convolutional layer with jumps are processed through residual connections to obtain the feature vector Y, which is then fed into the second layer structure. The second layer structure includes a bilinear interpolation unit, a second multi-head self-attention unit, a second multi-head cross-attention unit, and a second feedforward neural network. Y is doubled in resolution by the bilinear interpolation unit to obtain fine-grained image features W. Then, W is passed through the second multi-head self-attention unit to obtain a visual feature vector M2. The second multi-head self-attention unit uses the visual feature vector M2 as a query vector. The query vector and the text feature vector R are passed through the second multi-head cross-attention unit. The second multi-head cross-attention unit uses the text feature vector R as a key and value vector to calculate the attention score of the key and value vectors of the text feature vector R in M2, thereby obtaining the second fused visual feature vector X2. Finally, X2 is input into the second feedforward neural network to obtain a high-resolution multimodal fused feature I.

7. The visual text-based weakly supervised flock counting device for bird flock monitoring as described in claim 6, characterized in that, The method for generating a number sequence is as follows: Given a starting number h0, a counting interval k, and a total number of categories N, the number sequence h is expressed as equation (1): h=[h0,h0+k,...,h0+(N-1)k] (1).

8. The visual text-based weakly supervised flock counting device for bird flock monitoring as described in claim 6, characterized in that, Learnable feature adapters include: Image Branch Adapter V f (G) and text branch adapter V t (L) performs linear transformations on the visual feature vector and classifier weights respectively, and achieves feature enhancement through the ReLU activation function and residual connections, as specifically expressed as: F=[θV f (G) T +(1-θ)G]·[γV t (L) T +(1-γ)L] (6) Among them, G T L represents the matrix transpose of the visual feature vector G. T This represents the matrix transpose of the classifier weights L. and Both are learnable parameters, and θ and γ are residual rates representing the degree of residual connectivity.

9. The visual text-based weakly supervised flock counting device for bird flock monitoring as described in claim 6, characterized in that, Multimodal ranking loss function Loss(x) i′,i ,x i,j ) is defined as Equation (8): Loss(x i′,i ,x i,j )=max(0,x i′,i -x i,i ) (8) Where, x i,j x i′,i R represents the i-th image patch and the j-th text feature vector, respectively. j The inner product of the i′-th image patch and the j-th text feature vector R j The inner product, 1≤i′≤i≤N; Multi-stage screening strategies include: In the first classification phase, image patches containing birds are filtered out using the first text prompt template of the text encoder; In the second classification stage, the second text prompt template of the text encoder is used to filter out image patches containing bird heads; The filtered image patches are input into the optimized image encoder for similarity matching, and the total number of predicted birds is obtained by accumulating the predicted number.

Images