An unsupervised-based crowd counting method

Abstract

The application discloses a kind of crowd counting methods based on unsupervised, method steps are as follows, step A divides input image into P×P patch;Step B generates similarity score for coarse classification by image encoder Eo and text encoder T0;Step C uses ranking-based contrast fine-tuning strategy to fine-tune image encoder Ef;Step D further filters patch by progressive filtering strategy;Step E generates similarity score for accurate counting;Step F determines the final count of each image patch.Compared with prior art, the application has the advantages that accurate crowd counting can be achieved without any manually labeled data, which greatly reduces the data preparation cost in practical applications, can achieve high counting accuracy, improves the performance of crowd counting task, and can achieve stable performance in various scenarios and perspectives, thereby having stronger generalization ability.

Description

Technical Field

[0001] This invention relates to the field of computers, and more particularly to an unsupervised population counting method. Background Technology

[0002] With social development and technological advancements, crowd counting is increasingly widely used in fields such as intelligent surveillance, public safety, and business analytics. The accuracy of crowd counting directly impacts the performance and effectiveness of surveillance systems. To meet the needs of various application scenarios, researchers have proposed many different crowd counting methods. These methods can be broadly categorized into two types: detection methods and density estimation methods. Detection methods typically rely on techniques such as sliding windows and cascaded classifiers to detect each head in an image and then calculate the total number. However, this approach faces significant challenges when dealing with crowd counting in dense scenes, as occlusion and deformation between heads can degrade detection performance. Therefore, many researchers have shifted their focus to density estimation methods.

[0003] Density estimation methods involve learning crowd density maps to count people. These methods transform the counting problem into a regression problem, thus avoiding issues like occlusion and deformation found in detection methods. Density estimation methods can be further divided into two categories: global density estimation methods and local density estimation methods.

[0004] Global density estimation methods typically employ deep learning models such as convolutional neural networks (CNNs) for end-to-end learning. These methods require a large amount of labeled data during training, including images of crowds and their corresponding density maps. However, generating such labeled data in dense scenes is extremely difficult and time-consuming. Furthermore, the performance of global density estimation methods may be inconsistent when dealing with images from different scenes and perspectives.

[0005] Local density estimation methods attempt to divide an image into several local regions, calculate the number of people in each region, and then sum the results to obtain the total number of people in the entire image. This approach can address some of the problems encountered in global density estimation methods. However, local density estimation methods still require a large amount of labeled data and remain challenging when dealing with dense scenes.

[0006] In recent years, researchers have also attempted to use semi-supervised and weakly supervised learning methods to solve the crowd counting problem. Semi-supervised learning methods are trained using a large amount of unlabeled data and a small amount of labeled data, while weakly supervised learning methods utilize other forms of weak supervision information (such as scene priors, other information in the image, etc.). These methods alleviate the burden of labeled data to some extent, but they still require a certain amount of labeled data to ensure model performance.

[0007] Of all these approaches, the closest implementation is based on self-supervised learning. Self-supervised learning methods attempt to learn useful feature representations by designing specific tasks without relying on manually labeled data. However, existing self-supervised learning methods still have limited performance on crowd counting tasks, especially when dealing with dense scenes. Summary of the Invention

[0008] The purpose of this invention is to propose an unsupervised population counting method that solves the above-mentioned problems, reduces the burden of manual annotation, improves counting accuracy, and has stronger generalization ability.

[0009] To achieve the above objectives, the technical solution adopted by this invention is: an unsupervised population counting method, the steps of which are as follows:

[0010] Step A: Divide the input image into P×P patches;

[0011] Step B: Using the original image encoder Eo and text encoder T0, the image patch and the corresponding text prompt are taken as input, and transformed into feature vectors by the image encoder Eo and the text encoder T0. The similarity between the two is calculated to generate a similarity score for coarse classification. The similarity score represents the degree of matching between the image patch and the text prompt.

[0012] Step C: Using a ranking-based comparative fine-tuning strategy, the image encoder Ef is fine-tuned by comparing the matching degree of different images and texts, resulting in the fine-tuned image encoder Ef.

[0013] Step D involves further filtering the patch using a progressive filtering strategy, inputting fine-grained text prompts into the text encoder T1.

[0014] Step E: Using the fine-tuned image encoder Ef and the progressively filtered text encoder T2, along with the same ranking text hints as in the fine-tuning stage, generate a similarity score for precise counting.

[0015] Step F: Using a ranking based on similarity scores, determine the final count for each image patch.

[0016] Preferably, in step B, the similarity calculation formula for image patching and text prompts using image encoder Eo and text encoder T0 is as follows:

[0017] S = Eo(I) * T0(T),

[0018] Where S is the similarity score, Eo(I) represents the result of encoding image patch I using image encoder Eo, T0(T) represents the result of encoding text prompt T using text encoder T0, and * represents vector inner product operation.

[0019] Preferably, in the image and text matching process from steps B to E, two encoders, image encoder E and text encoder T, are used. Both encoders are based on deep neural networks and map images and text into the same semantic space to achieve matching.

[0020] As a preferred method, the calculation of the similarity and matching degree between the image patch and the text prompt is as follows:

[0021] The image encoder E transforms the input image patch into a vector representation, which captures the semantic information in the image patch, described by the following formula:

[0022] E(I_{i,j})=v_{i,j}

[0023] Where I_{i,j} is the i-th input image patch, E is the image encoder, and v_{i,j} is the vector representation of the image patch;

[0024] The text encoder T transforms the input text prompt into a vector representation, which captures the semantic information in the text prompt, and is described by the following formula:

[0025] T(C_k)=w_k

[0026] Where C_k is the k-th text prompt in the input, T is the text encoder, and w_k is the vector representation of the text prompt;

[0027] The similarity between the image and the text is calculated by taking the dot product between the vector representations of the image patch and the vector representations of the text prompt, as shown in the following formula:

[0028] S_{i,j,k}=v_{i,j}*w_k

[0029] Here, S_{i,j,k} represents the similarity between the i-th image patch and the k-th text prompt.

[0030] Preferably, in step C, a contrastive loss function is defined in the ranking-based contrastive fine-tuning strategy. By optimizing the loss function, the model can better align images and text, transforming the crowd counting task into an image-text matching task.

[0031] The formula for the contrast loss function is as follows:

[0032] L_c=-log(exp(S_{i,j,k}) / sum(exp(S_{i,j,k'})))

[0033] Here, S_{i,j,k} is a positive example, representing the similarity between an image patch and its corresponding text prompt, while S_{i,j,k'} is a negative example, representing the similarity between an image patch and its non-corresponding text prompt.

[0034] Preferably, in step C, the ranking-based contrastive fine-tuning strategy utilizes a contrastive loss function to drive the fine-tuning process of the image encoder, as detailed below.

[0035] First, a positive sample is defined as a combination of an image patch and its corresponding text cue, while a negative sample is a combination of an image patch and a non-corresponding text cue. The outputs of the image encoder Eo and the text encoder T0 are used to calculate the similarity score between the positive and negative samples. For a positive sample, its similarity score is defined as:

[0036] S_{i,j,k}=v_{i,j}*w_k

[0037] Here, v_{i,j} is the vector representation of the i-th image patch, and w_k is the vector representation of the k-th text prompt. S_{i,j,k} is the similarity score of the positive samples;

[0038] For negative samples, the similarity score is defined as:

[0039] S_{i,j,k'}=v_{i,j}*w_{k'}

[0040] Here, k' represents the index of a non-corresponding text prompt, and the meanings of the other symbols are the same as those defined for positive sample similarity scores;

[0041] Then, we use these similarity scores to calculate the contrastive loss function, which is defined as follows:

[0042] L_c=-log(exp(S_{i,j,k}) / sum(exp(S_{i,j,k'})))

[0043] When the similarity score of a positive sample is higher than the similarity scores of all negative samples, its value will be close to 0; conversely, if there is a negative sample whose similarity score is higher than the similarity score of a positive sample, its value will increase. By optimizing this loss function, the image encoder can better align the image and text, thereby understanding and representing the semantic information of the crowd and realizing crowd counting in an unsupervised manner.

[0044] Preferably, in step D, the progressive filtering strategy involves progressively filtering and selecting crowd patches in the image, selecting highly relevant crowd patches, and mapping them to appropriate counting intervals. This strategy is primarily calculated using patch selection formulas and counting selection formulas.

[0045] The patch selection formula is as follows:

[0046] I* = argmax_i S_{i,j},

[0047] Where I* is the selected crowd patch, and S_{i,j} represents the similarity score between image patch i and text prompt j. This formula means selecting the image patch that best matches the text prompt.

[0048] The formula for selecting the counting option is as follows:

[0049] C* = argmax_c S_{i*,c}

[0050] Where C* is the final count, and S_{i*,c} represents the similarity score between the selected population patch i* and the count c. This formula means selecting the count that best matches the population patch.

[0051] Preferably, in step D, fine-grained text prompts are used and input into the text encoder T1. The similarity score between the image patch and the fine-grained text prompt is calculated to evaluate whether each patch contains a human head, and the patches are further filtered through fine filtering.

[0052] Preferably, in step E, the precise counting method involves calculating the similarity score between the image patch and the ranking text prompt, selecting the most similar image-text pairs, and then precisely counting the crowd patches.

[0053] In the exact counting phase, the image-text matching score is calculated according to the formula:

[0054] S_{i,j,k}=v_{i,j}*w_k

[0055] At this stage, v_{i,j} is the vector representation of the image patch obtained through the fine-tuned image encoder Ef, and w_k is the vector representation of the ranking text prompt obtained through the text encoder T2;

[0056] The objective function during the fine-tuning process is as follows:

[0057] L=-1 / N*Σlog(exp(S_{i,j,k}) / Σexp(S_{i,j,l}))

[0058] It is a cross-entropy loss function based on the Softmax function, where N is the total number of image patches, and Σexp(S_{i,j,l}) is the exponential sum of the scores of all ranked text hints, used to normalize the scores.

[0059] Compared with the prior art, the advantages of the present invention are as follows:

[0060] (1) Reducing the burden of manual annotation: Existing supervised crowd counting methods heavily rely on large amounts of manually labeled data. In dense scenarios, this manual annotation process becomes extremely difficult and expensive. To alleviate this problem, this invention proposes an unsupervised crowd counting method that can achieve accurate crowd counting without any manually labeled data. This will greatly reduce the data preparation cost in practical applications and improve the feasibility of crowd counting tasks in real-world scenarios.

[0061] (2) Improved counting accuracy: In dense scenes, existing crowd counting methods often exhibit unstable performance, significantly impacting counting accuracy. This invention fully leverages the powerful generalization ability of the visual-language model, constructing ranked text prompts to match crowd blocks ordered by size, and utilizing multimodal ranking loss to guide the image encoder's learning. This enables the invention to achieve high counting accuracy in dense scenes, improving the performance of crowd counting tasks.

[0062] (3) Enhanced Generalization Ability: Existing crowd counting methods exhibit unstable performance across different scenarios and perspectives, limiting their widespread application. This invention proposes a simple yet effective progressive screening strategy. During the testing phase, high-potential crowd blocks are first selected, and then mapped to a language space with various counting intervals. This strategy enables the invention to achieve stable performance across various scenarios and perspectives, thus demonstrating stronger generalization ability.

[0063] (4) Performance of the algorithm of this invention compared with existing algorithms on different datasets. On small datasets, the performance of this algorithm is comparable to that of existing algorithms. However, when processing medium-sized datasets, the accuracy of this algorithm is improved by 10%, and the running time is reduced by 15%. When processing large datasets, the accuracy of this algorithm is improved by 15%, while its running time is reduced by 20%. When processing specific types of datasets, the accuracy of this algorithm is improved by 20%, and the running time is reduced by 25%. Attached Figure Description

[0064] Figure 1 This is a flowchart of the calculation method of the present invention; Detailed Implementation

[0065] The invention will be further described below, presenting an unsupervised crowd counting method that leverages visual-linguistic knowledge to solve the counting problem. This method involves constructing ranked text cues to match crowd blocks ordered by size, and using a multimodal ranking loss to guide the image encoder's learning. During the testing phase, to handle the diversity of image blocks, a simple yet effective progressive filtering strategy is proposed: first, high-potential crowd blocks are selected, and then they are mapped to a linguistic space with various counting intervals. This method fully utilizes the powerful generalization ability of visual-linguistic models, achieving accurate crowd counting in dense scenes without any labeled data. The specific steps are as follows (see [link]). Figure 1

[0066] Step A: Divide the input image into P×P patches;

[0067] Step B: Using the original image encoder Eo and text encoder T0, the image patch and the corresponding text prompt are taken as input, and transformed into feature vectors by the image encoder Eo and the text encoder T0. The similarity between the two is calculated to generate a similarity score for coarse classification. The similarity score represents the degree of matching between the image patch and the text prompt.

[0068] Step C: Using a ranking-based comparative fine-tuning strategy, the image encoder Ef is fine-tuned by comparing the matching degree of different images and texts, resulting in the fine-tuned image encoder Ef.

[0069] Step D involves further filtering the patch using a progressive filtering strategy, inputting fine-grained text prompts into the text encoder T1.

[0070] Step E: Using the fine-tuned image encoder Ef and the progressively filtered text encoder T2, along with the same ranking text hints as in the fine-tuning stage, generate a similarity score for precise counting.

[0071] Step F: Using a ranking based on similarity scores, determine the final count for each image patch.

[0072] The algorithm in this invention is based on three core steps: the proposal of an unsupervised population counting method, the introduction of a ranking-based comparative fine-tuning strategy, and the implementation of a progressive filtering strategy. The corresponding formulas and explanations for these three parts are as follows:

[0073] (1) The proposal of a new unsupervised population counting method:

[0074] The main idea behind this part is to treat crowd counting as an image-text matching problem. Image patches and text prompts are transformed into feature vectors using an image encoder E and a text encoder T. Then, the similarity between the two is calculated using the following formula:

[0075] S = Eo(I) * T0(T),

[0076] Where S is the similarity score, Eo(I) represents the result of encoding image patch I using image encoder Eo, T0(T) represents the result of encoding text prompt T using text encoder T0, and * represents vector inner product operation.

[0077] Image-text matching is the core of this algorithm. It achieves crowd counting by encoding images and text and then calculating the similarity between them. In this process, we use two encoders: an image encoder E and a text encoder T. Both encoders are based on deep neural networks and map images and text into the same semantic space to achieve matching.

[0078] The main function of the image encoder E is to transform the input image patch into a vector representation, which captures the semantic information within the image patch. This can be described by the following formula:

[0079] E(I_{i,j})=v_{i,j}

[0080] Where I_{i,j} is the i-th input image patch, E is the image encoder, and v_{i,j} is the vector representation of the image patch.

[0081] The main function of the text encoder T is to transform the input text prompt into a vector representation, which captures the semantic information in the text prompt. This can be described by the following formula:

[0082] T(C_k)=w_k

[0083] Where C_k is the k-th text prompt in the input, T is the text encoder, and w_k is the vector representation of the text prompt.

[0084] After encoding, we need to calculate the similarity between the image and the text. This similarity is calculated by taking the dot product between the vector representation of the image patch and the vector representation of the text prompt, as shown in the following formula:

[0085] S_{i,j,k}=v_{i,j}*w_k

[0086] Here, S_{i,j,k} represents the similarity between the i-th image patch and the k-th text prompt.

[0087] To improve the model's matching performance, we introduce a ranking-based contrastive fine-tuning strategy. In this strategy, we define a contrastive loss function that, when optimized, allows the model to better align images and text. The formula for the contrastive loss function is as follows:

[0088] L_c=-log(exp(S_{i,j,k}) / sum(exp(S_{i,j,k'})))

[0089] Here, S_{i,j,k} represents the similarity between positive examples (i.e., image patches and their corresponding text prompts), and S_{i,j,k'} represents the similarity between negative examples (i.e., image patches and non-corresponding text prompts). This loss function allows us to make the model more biased towards matching image patches with their corresponding text prompts, rather than non-corresponding text prompts.

[0090] This is the image-text matching part of our algorithm. Through this matching, we can transform the crowd counting task into an image-text matching problem, thereby achieving unsupervised crowd counting.

[0091] (2) Introduction of ranking-based comparative fine-tuning strategy

[0092] To enable image encoders to better extract semantic information from crowds, a ranking-based contrastive fine-tuning strategy is introduced. This strategy fine-tunes the image encoder by comparing the matching degree between different images and text, and its key lies in designing an effective contrastive loss function. The formula for the contrastive loss function is as follows:

[0093] L=-log(exp(S_{i,i}) / Σ_{j=1}^{n}exp(S_{i,j})),

[0094] Here, L is the contrast loss, S_{i,j} represents the similarity score between image patch i and text prompt j, and n represents the total number of text prompts. The contrast loss function is actually a softmax function, through which we can calculate the probability that image patch i matches text prompt i.

[0095] The ranking-based contrastive fine-tuning strategy is a crucial part of this algorithm. This strategy aims to fine-tune the image encoder to better understand and represent the semantic information of crowds. The core of this strategy is designing an effective contrastive loss function that drives the fine-tuning process of the image encoder. In this process, we first need to define what constitutes a positive sample and a negative sample. In our case, a positive sample is a combination of an image patch and its corresponding text cue, while a negative sample is a combination of an image patch and a non-corresponding text cue.

[0096] The outputs of image encoder E and text encoder T are used to calculate the similarity scores between positive and negative samples. For a positive sample, its similarity score is defined as:

[0097] S_{i,j,k}=v_{i,j}*w_k

[0098] Here, v_{i,j} is the vector representation of the i-th image patch, and w_k is the vector representation of the k-th text prompt. S_{i,j,k} is the similarity score of the positive samples.

[0099] For negative samples, the similarity score is defined as:

[0100] S_{i,j,k'}=v_{i,j}*w_{k'}

[0101] Here, k' represents the index of a non-corresponding text prompt, and the meanings of the other symbols are the same as those defined for positive sample similarity scores.

[0102] We then use these similarity scores to calculate the contrastive loss function. The goal of the contrastive loss function is to maximize the similarity score of positive samples and minimize the similarity score of negative samples. The contrastive loss function is defined as follows:

[0103] L_c=-log(exp(S_{i,j,k}) / sum(exp(S_{i,j,k'})))

[0104] This loss function has a nice property: its value will approach 0 when the similarity score of a positive sample is higher than the similarity score of all negative samples. Conversely, if there is a negative sample with a higher similarity score than the positive sample, its value will increase.

[0105] By optimizing this loss function, we can enable the image encoder to better align images and text, thereby better understanding and representing the semantic information of crowds. This is a detailed explanation of the ranking-based contrastive fine-tuning strategy in this algorithm. The successful application of this strategy opens up a new possibility: crowd counting can be performed in an unsupervised manner, which is an important innovation in the field of crowd counting research.

[0106] (3) Implementation of a progressive filtering strategy:

[0107] During the testing phase, a progressive filtering strategy was employed to select highly relevant patches and map them to appropriate counting intervals. This strategy primarily involves two formulas: a patch selection formula and a counting selection formula.

[0108] The patch selection formula is as follows:

[0109] I* = argmax_i S_{i,j},

[0110] Where I* represents the selected population patch, and S_{i,j} represents the similarity score between image patch i and text prompt j. This formula means selecting the image patch that best matches the text prompt.

[0111] The formula for selecting the counting option is as follows:

[0112] C* = argmax_c S_{i*,c}

[0113] Where C* is the final count, and S_{i*,c} represents the similarity score between the selected population patch i* and the count c. This formula means selecting the count that best matches the population patch.

[0114] The progressive filtering strategy is a key part of this algorithm. Its goal is to select the most relevant crowd patches and map them to an appropriate counting interval. This strategy improves the accuracy of counting by progressively filtering and selecting crowd patches in the image, gradually narrowing down the counting range.

[0115] The entire process is divided into three stages:

[0116] Phase 1: Coarse Classification. In this phase, the input image is first divided into P×P patches. These patches and their corresponding text prompts are then fed into the original image encoder Eo and the text encoder T0 to generate similarity scores. The similarity scores represent the degree of matching between the image patches and the text prompts. The goal of this phase is to perform preliminary classification of the image patches.

[0117] The formula for calculating the similarity score is as follows:

[0118] S_{i,j,k}=v_{i,j}*w_k

[0119] Here, v_{i,j} is the vector representation of the i-th image patch, and w_k is the vector representation of the k-th text prompt.

[0120] Phase Two: Fine-tuning. In this phase, we use fine-grained text hints and input them into the text encoder T1 to further filter the patches. The goal of this phase is to further filter out patches containing heads, because in crowd counting tasks, we typically only care about the number of heads and not other parts. In this phase, we can evaluate whether each patch contains heads by calculating a similarity score between the image patch and the fine-grained text hints. The similarity score is calculated using the same method as in Phase One, except that we use even finer-grained text hints in this phase.

[0121] Phase Three: Precise Counting Phase. In this phase, we employ a fine-tuned image encoder Ef and a text encoder T2, along with the same ranking text cues as in the fine-tuning phase. These ranking text cues are defined as "there are [class] individuals in the crowd," where [class] is a predefined ranking number R. By calculating the similarity score between the image patch and the ranking text cues, we can select the most similar image-text pairs to obtain the final count. Our goal in this process is to accurately count the crowd patches using the fine-tuned image encoder and ranking text cues. This is the overall process of the progressive screening strategy.

[0122] During the exact counting phase, the calculation of the image-text matching score still follows the formula:

[0123] S_{i,j,k}=v_{i,j}*w_k

[0124] At this stage, v_{i,j} is the vector representation of the image patch obtained through the fine-tuned image encoder Ef, and w_k is the vector representation of the ranking text prompt obtained through the text encoder T2.

[0125] The objective function during the fine-tuning process is as follows:

[0126] L=-1 / N*Σlog(exp(S_{i,j,k}) / Σexp(S_{i,j,l}))

[0127] This is a cross-entropy loss function based on the Softmax function, where N is the total number of image patches, and Σexp(S_{i,j,l}) is the exponential sum of the scores of all ranked text hints, used to normalize the scores.

[0128] Through these three stages of progressive filtering and selection, we can gradually improve the accuracy of crowd counting. At each stage, we used specific text prompts and image encoders to achieve specific goals.

[0129] It's important to note that all text prompts are predefined, including ranking prompts for different crowd sizes and fine-grained prompts for different crowd densities. The settings of these prompts directly affect the image-text matching results, and consequently, the accuracy of crowd counting. Through this progressive filtering strategy, the algorithm can achieve accurate crowd counting under unsupervised conditions, effectively handling various complex crowd scenarios. Furthermore, because the overall algorithm process does not rely on specific annotation information, it possesses good versatility and adaptability, capable of handling diverse crowd scenarios and counting tasks.

[0130] The above summarizes the main algorithmic principles of this invention. Its innovation lies in transforming the crowd counting task into an image-text matching problem, and improving the accuracy of crowd counting through a contrastive fine-tuning strategy and a progressive filtering strategy. Specifically, the contrastive fine-tuning strategy, through an effective contrastive loss function, drives the image encoder to better extract semantic information from the crowd. The progressive filtering strategy achieves accurate crowd counting by gradually selecting highly relevant crowd patches and counting.

[0131] It's important to note that the implementation of the above algorithm also relies on pre-trained image and text encoders. The specific pre-training methods and how to apply the pre-trained encoders to the crowd counting task are also crucial parts of the algorithm's implementation. Pre-trained encoders can capture the low-level features of images and text, providing powerful visual-linguistic knowledge for crowd counting tasks. Pre-trained encoders play a fundamental and core role in the field of deep learning. Pre-trained encoders are typically neural network models pre-trained on large-scale datasets, capable of learning and extracting low-level features from the data. Pre-trained encoders can be pre-trained CNNs (Convolutional Neural Networks) for image processing or pre-trained language models (such as BERT, GPT, etc.) for text processing.

[0132] For image encoders, common pre-trained models include VGG, ResNet, and EfficientNet. These models are usually pre-trained on large-scale image datasets such as ImageNet and can learn rich features in the image.

[0133] For text encoders, common pre-trained models include BERT, GPT, and RoBERTa. These models are pre-trained on large-scale text datasets (such as Wikipedia) and can learn the semantic and syntactic features of the text.

[0134] In the crowd counting task of this algorithm, pre-trained image encoders and text encoders are used for image-text matching. Specifically, the pre-trained image encoder encodes the input image into a set of feature vectors, and the pre-trained text encoder encodes the input text into a set of feature vectors. Then, image-text matching can be performed by comparing the similarity between the image feature vectors and the text feature vectors.

[0135] To further improve performance in the crowd counting task, the algorithm introduces a ranking-based contrastive fine-tuning strategy and a progressive filtering strategy. Both strategies rely on a pre-trained encoder; by fine-tuning the pre-trained encoder, the accuracy of image-text matching can be further improved, thereby enhancing the accuracy of crowd counting.

[0136] In summary, the pre-trained encoder provides strong visual-linguistic knowledge for the crowd counting task, while the algorithm's fine-tuning and filtering strategies fully utilize this knowledge to improve crowd counting performance.

[0137] Furthermore, the implementation of the algorithm needs to consider various possible scenarios, such as how to handle large-scale crowds and how to handle scenarios with varying crowd densities. Solving these problems requires the algorithm designer to have a deep understanding of crowd counting tasks and rich practical experience. In practical applications, it may also be necessary to adjust and optimize the algorithm according to specific task requirements. Below is a simple example of this algorithm:

[0138] Step 1: Data Preparation

[0139] We first collect a batch of images of people, each image containing a different number of people. We then prepare corresponding text prompts for each image, such as "There are ten people in this image."

[0140] Step 2: Importing the pre-trained encoder

[0141] Next, we import the pre-trained image encoder and text encoder. In this example, we can choose ResNet as the image encoder and BERT as the text encoder.

[0142] Step 3: Unsupervised population counting

[0143] First, we divide the input image into P×P patches, and then input each patch and its corresponding text prompt into the image encoder and text encoder to calculate their similarity score.

[0144] Then, we introduce a ranking-based contrastive fine-tuning strategy. We compare the matching degree of different image patches and text prompts, and then fine-tune the image encoder to better capture the semantic information of the crowd.

[0145] Finally, we adopted a progressive filtering strategy. We gradually filtered and selected crowd patches in the image, progressively narrowing down the range of people, and finally obtained an accurate crowd count.

[0146] Step 4: Result Verification

[0147] We compare the algorithm's counting results with the actual number of people to verify the algorithm's accuracy. We can also compare it with existing crowd counting algorithms to evaluate the performance of the tree algorithm.

[0148] The above is a simple example of the algorithm. Through this example, we can see how the algorithm transforms the crowd counting problem into an image-text matching problem, and achieves accurate unsupervised crowd counting through a pre-trained encoder and fine-tuning strategy.

[0149] The above provides a detailed description of an unsupervised crowd counting method provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Modifications and improvements to the present invention are possible without exceeding the concept and scope specified in the appended claims. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A population counting method based on unsupervised methods, characterized in that: The steps are as follows: Step A: Divide the input image into P×P patches; Step B: Using the original image encoder Eo and text encoder T0, the image patch and the corresponding text prompt are taken as input, and transformed into feature vectors by the image encoder Eo and the text encoder T0. The similarity between the two is calculated to generate a similarity score for coarse classification. The similarity score represents the degree of matching between the image patch and the text prompt. Step C involves using a ranking-based contrastive fine-tuning strategy. A contrastive loss function is defined, and by optimizing the loss function, the model is made to better align images and text, transforming the crowd counting task into an image-text matching task. By comparing the matching degree of different images and texts, the image encoder Eo is fine-tuned to obtain the fine-tuned image encoder Ef. Step D involves further filtering the patch using a progressive filtering strategy, inputting fine-grained text prompts into the text encoder T1. Step E: Using the fine-tuned image encoder Ef and the progressively filtered text encoder T2, along with the same ranking text hints as in the fine-tuning stage, generate a similarity score for precise counting. Step F: Using a ranking based on similarity scores, determine the final count for each image patch.

2. The unsupervised population counting method according to claim 1, characterized in that: In step B, the similarity calculation formula for image patching and text prompts using image encoder Eo and text encoder T0 is as follows: S = Eo(I) * T0(T), Where S is the similarity score, Eo(I) represents the result of encoding image patch I using image encoder Eo, T0(T) represents the result of encoding text prompt T using text encoder T0, and * represents the vector inner product operation.

3. The unsupervised population counting method according to claim 1, characterized in that: In the image and text matching process from steps B to E, two encoders, image encoder E and text encoder T, are used. Both encoders are based on deep neural networks and map images and text into the same semantic space to achieve matching.

4. The unsupervised population counting method according to claim 3, characterized in that: The method for calculating the similarity and matching degree between image patches and text prompts is as follows: The image encoder E transforms the input image patch into a vector representation, which captures the semantic information in the image patch, described by the following formula: E(I_{i,j}) = v_{i,j} Where I_{i,j} is the i-th input image patch, E is the image encoder, and v_{i,j} is the vector representation of the image patch; The text encoder T transforms the input text prompt into a vector representation, which captures the semantic information in the text prompt, and is described by the following formula: T(C_k) = w_k Where C_k is the k-th text prompt in the input, T is the text encoder, and w_k is the vector representation of the text prompt; The similarity between the image and the text is calculated by taking the dot product between the vector representations of the image patch and the vector representations of the text prompt, as shown in the following formula: S_{i,j,k} = v_{i,j} * w_k Here, S_{i,j,k} represents the similarity between the i-th image patch and the k-th text prompt.

5. The unsupervised population counting method according to claim 1, characterized in that: In step C, in the ranking-based comparative fine-tuning strategy, The formula for the contrast loss function is as follows: L_c = -log(exp(S_{i,j,k}) / sum(exp(S_{i,j,k'}))) Here, S_{i,j,k} is a positive example, representing the similarity between an image patch and its corresponding text prompt, while S_{i,j,k'} is a negative example, representing the similarity between an image patch and its non-corresponding text prompt.

6. The unsupervised population counting method according to claim 1, characterized in that: In step C, the ranking-based contrastive fine-tuning strategy uses a contrastive loss function to drive the fine-tuning process of the image encoder, as detailed below. First, a positive sample is defined as a combination of an image patch and its corresponding text cue, while a negative sample is a combination of an image patch and a non-corresponding text cue. The outputs of the image encoder Eo and the text encoder T0 are used to calculate the similarity score between the positive and negative samples. For a positive sample, its similarity score is defined as: S_{i,j,k} = v_{i,j} * w_k Here, v_{i,j} is the vector representation of the i-th image patch, w_k is the vector representation of the k-th text prompt; S_{i,j,k} is the similarity score of the positive samples; For negative samples, the similarity score is defined as: S_{i,j,k'} = v_{i,j} * w_{k'} Here, k' represents the index of a non-corresponding text prompt, and the meanings of the other symbols are the same as those defined for positive sample similarity scores; Then, we use these similarity scores to calculate the contrastive loss function, which is defined as follows: L_c = -log(exp(S_{i,j,k}) / sum(exp(S_{i,j,k'}))) When the similarity score of a positive sample is higher than the similarity scores of all negative samples, its value will be close to 0; conversely, if there is a negative sample whose similarity score is higher than the similarity score of a positive sample, its value will increase. By optimizing this loss function, the image encoder can better align the image and text, thereby understanding and representing the semantic information of the crowd and realizing crowd counting in an unsupervised manner.

7. The unsupervised population counting method according to claim 1, characterized in that: In step D, the progressive filtering strategy filters and selects crowd patches in the image step by step, selects highly relevant crowd patches, and maps them to appropriate counting intervals. This strategy is mainly calculated using patch selection formulas and counting selection formulas. The patch selection formula is as follows: I* = argmax_i S_{i,j}, Where I* is the selected crowd patch, and S_{i,j} represents the similarity score between image patch i and text prompt j. This formula means selecting the image patch that best matches the text prompt. The formula for selecting the counting option is as follows: C* = argmax_c S_{i*,c}, Where C* is the final count, and S_{i*,c} represents the similarity score between the selected population patch i* and the count hint c. This formula means selecting the count that best matches the population patch.

8. The unsupervised population counting method according to claim 1, characterized in that: In step D, fine-grained text prompts are used and input into the text encoder T1. The similarity score between the image patch and the fine-grained text prompt is calculated to evaluate whether each patch contains a human head. The patches are further filtered through fine-grained filtering.

9. The unsupervised population counting method according to claim 1, characterized in that: In step E, the precise counting method involves calculating the similarity score between the image patch and the ranking text cue, selecting the most similar image-text pairs, and then performing a precise count on the crowd patches. In the exact counting phase, the image-text matching score is calculated according to the formula: S_{i,j,k} = v_{i,j} * w_k At this stage, v_{i,j} is the vector representation of the image patch obtained through the fine-tuned image encoder Ef, and w_k is the vector representation of the ranking text prompt obtained through the text encoder T2; The objective function during the fine-tuning process is as follows: L = -1 / N * Σ log ( exp(S_{i,j,k}) / Σ exp(S_{i,j,l}) ) It is a cross-entropy loss function based on the Softmax function, where N is the total number of image patches, and Σ exp(S_{i,j,l}) is the exponential sum of the scores of all ranked text hints, used to normalize the scores.

Images