An unsupervised population counting method, device and storage medium

Abstract

This invention discloses an unsupervised crowd counting method, apparatus, and storage medium. The method includes cropping a first input image into image blocks and obtaining coarse-grained text for each image block; inputting the image blocks into a first image encoder and the coarse-grained text into a first text encoder to generate a first similarity matrix; filtering image blocks of a first target category based on the first similarity matrix and a first discriminative category similarity; obtaining fine-grained text for image blocks of a second target category and inputting it into a second text encoder to generate a second similarity matrix; filtering image blocks of a second target category based on the second similarity matrix and a second discriminative category similarity and inputting them into a second image encoder; inputting the counting text into a third text encoder to generate a target similarity matrix; and obtaining the number of people in the image based on the similarity between the target similarity matrix and the counting text. This invention eliminates the need for any manual labeling, significantly reducing annotation costs.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an unsupervised crowd counting method, apparatus, and storage medium. Background Technology

[0002] The primary cause of stampedes is the inability to promptly ascertain the exact number of people, making timely and effective crowd dispersal difficult. Therefore, crowd counting has become a hot research topic in fields such as video surveillance, crowd understanding, and public safety prevention. In the field of computer vision, crowd counting has become an important research direction.

[0003] With the development of deep learning, researchers have implemented various excellent methods for counting dense crowds using deep neural networks. Specifically, existing methods have made significant progress by regressing density maps using convolutional neural networks (CNNs), where the final number of people equals the integral of the predicted density map. However, to train density map-based networks, researchers typically need to annotate a large number of labels, i.e., labeling a point for each head; in other words, such methods rely on expensive annotation costs. For example, the NWPU-Crowd dataset, containing 5,109 images, required 30 annotators to spend 3,000 hours to complete the annotation. To reduce annotation costs, some weakly supervised and semi-supervised methods have been proposed. The former typically uses count-level annotations as supervision, while the latter is trained using a small number of fully labeled images and a large number of unlabeled images. However, weakly supervised and semi-supervised methods still require a considerable amount of annotation cost, especially when annotating images with dense or blurred crowds.

[0004] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0005] The technical problem to be solved by the embodiments of the present invention is how to solve the problem of expensive labeling costs caused by the need for manual labeling of a large number of labels in existing population counting network models.

[0006] The embodiments of the present invention adopt the following technical solutions:

[0007] In a first aspect, the present invention provides an unsupervised population counting method, comprising:

[0008] A dataset of crowd images is obtained, a given first input image in the dataset is cropped into an image block of a first preset size, a first discrimination category of the image block is set, and coarse-grained text corresponding to the image block is obtained according to the first discrimination category;

[0009] The image blocks are input into the first image encoder, and the coarse-grained text is input into the first text encoder to generate a first image-text similarity matrix. Based on the similarity between the first similarity matrix and the first discriminative category, image blocks of the first target category are selected.

[0010] A second distinguishability category for image blocks is defined. Fine-grained text corresponding to image blocks of the first target category is obtained based on the second distinguishability category. The fine-grained text is input into a second text encoder to generate a second image-text similarity matrix. Based on the similarity between the second similarity matrix and the second distinguishability category, image blocks of the second target category are selected.

[0011] The image patch of the second target category is input into the second image encoder, and the counted text is input into the third text encoder to generate the target similarity matrix of image and text. Based on the similarity between the target similarity matrix and the counted text, the number of people in the image is obtained.

[0012] Preferably, the second image encoder is a fine-tuned image encoder, and the fine-tuning training specifically includes:

[0013] The given second input image in the dataset is cropped into a set of square image blocks of a preset number according to different sizes, and the counting text corresponding to the square image block is constructed according to the number of people in each square image block;

[0014] The square image block is input into the first image encoder to generate an image embedding with sorting properties, and the counted text is input into the third text encoder to generate a text embedding with sorting properties.

[0015] Calculate the inner product of the image embedding and the text embedding to obtain the image-text training similarity matrix.

[0016] Preferably, after obtaining the second image encoder through fine-tuning training, the method further includes performing secondary fine-tuning on the second image encoder using a loss function. The secondary fine-tuning specifically includes:

[0017] Obtain the training similarity matrix of the image-text, and calculate the ordinal loss from bottom to top using the main diagonal of the similarity matrix as the reference.

[0018] The image embedding is aligned with the text embedding to obtain a second image encoder after secondary fine-tuning.

[0019] Preferably, before inputting the square image blocks into the first image encoder to generate an image embedding with sorting properties, all square image blocks need to be adjusted to the same second preset size.

[0020] Preferably, the image-text training similarity matrix S = [s i,j ],in Specifically:

[0021]

[0022] in, And 0≤i≤M-1, 0≤j≤N-1; s i,j Let I represent the similarity between the i-th image patch embedding and the j-th text embedding. i and R j ' represents the embedding of the i-th image patch and the embedding of the j-th text, respectively; T represents the matrix transpose operation; C represents the feature dimension; M represents the number of image patches in a set of image patches; and N represents the number of categories of text descriptions used to describe a set of image patches.

[0023] Preferably, the first text encoder, the second text encoder, and the third text encoder share the same network parameters.

[0024] Preferably, the unsupervised population counting method further includes:

[0025] Multiple sample images are collected, and the unsupervised crowd counting method is used to predict the number of people in the multiple sample images to obtain the predicted number of people for each sample image.

[0026] The mean absolute error and mean square error are obtained by comparing the predicted number of people with the actual number of people in all sample images.

[0027] Set a threshold for mean absolute error and a threshold for mean square error. If the mean absolute error and the mean square error are not within the set range, continue to use the unsupervised population counting method for training to optimize the unsupervised population counting method.

[0028] Preferably, the number of the first discrimination category, the number of the first discrimination category, and the preset number are all equal.

[0029] Secondly, based on the unsupervised crowd counting method of the first aspect, the present invention also provides an unsupervised crowd counting device, comprising:

[0030] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the unsupervised population counting method described in the first aspect.

[0031] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the unsupervised population counting method described in the first aspect.

[0032] This invention involves cropping a given first input image into image blocks of a first preset size, obtaining coarse-grained text corresponding to the image blocks based on a first discrimination category, processing the image blocks and coarse-grained text into a first similarity matrix using a first image encoder and a first text encoder, filtering out image blocks of a first target category based on the similarity between the first similarity matrix and the first discrimination category, setting a second discrimination category for the image blocks, obtaining fine-grained text corresponding to the image blocks of a second target category based on the second discrimination category, inputting the fine-grained text into a second text encoder to generate a second similarity matrix, filtering out image blocks of the second target category, processing the image blocks of the second target category using a second image encoder to generate an image-text target similarity matrix, and obtaining the number of people in the image based on the similarity between the target similarity matrix and the count text. This invention only requires corresponding text encoders and image encoders to process the corresponding text and images to obtain the number of people in the image. Compared with existing supervised dense crowd counting methods, this invention does not require any manual labels, greatly reducing the labeling cost. Furthermore, by leveraging the implicit ordinal relationship between the image and text, a similarity matrix is ​​constructed, and the number of people in the image is obtained by using the similarity matrix and the similarity of the text. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart of an unsupervised population counting method provided in an embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram of the test process for an unsupervised population counting method provided in an embodiment of the present invention;

[0036] Figure 3 This is a flowchart of a method for fine-tuning training within an unsupervised population counting method provided in an embodiment of the present invention;

[0037] Figure 4 This is a schematic diagram of the process of fine-tuning training within an unsupervised population counting method provided in an embodiment of the present invention;

[0038] Figure 5 This is a flowchart illustrating an unsupervised population counting method provided by an embodiment of the present invention, which uses mean absolute error and mean square error to measure the corresponding model of the present invention.

[0039] Figure 6 This is a comparison table of the performance of an unsupervised crowd counting method provided in this embodiment of the invention and the CSS-CCNN model;

[0040] Figure 7 This is a schematic diagram of an unsupervised crowd counting device provided in an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0042] In the description of this invention, the terms "inner", "outer", "longitudinal", "lateral", "upper", "lower", "top", "bottom", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and do not require that this invention must be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0043] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0044] Example 1:

[0045] Embodiment 1 of the present invention provides an unsupervised population counting method, such as... Figure 1-2 As shown, it includes:

[0046] Step 201: Obtain a dataset of crowd images, crop a given first input image in the dataset into an image block of a first preset size, set a first discrimination category for the image block, and obtain the coarse-grained text corresponding to the image block according to the first discrimination category.

[0047] In this embodiment of the invention, the acquired crowd images are first compiled into a dataset, and the crowd images in the dataset are sequentially input into the corresponding text encoder and image encoder of this embodiment for testing in a predetermined order. Before inputting into the text encoder and image encoder, the input images need to be processed. When a first input image is given, this embodiment of the invention first crops the first input image to crop a set of square image blocks of the same size (e.g., image blocks of size P×P), then sets a first discrimination category for the image blocks, classifies the cropped directional image blocks according to the set first discrimination category, and obtains the coarse-grained text corresponding to each image block. To facilitate understanding, a specific example will be used below. For instance, a first input image of size 2×2P is cropped into four P×P image blocks A, B, C, and D. The first discrimination category is set to four categories: "crowd," "tree," "light," and "vehicle." First, the content of image block A is identified, and coarse-grained text matching is performed on image block A according to the first discrimination category (for example, the coarse-grained text of image block A is "This image block is a crowd," and the corresponding category is "crowd"). Then, using the same method, coarse-grained text matching is performed on image blocks B (for example, the coarse-grained text of image block A is "This image block is a tree," and the corresponding category is "tree"), C, and D, matching all image blocks in the first input image with the corresponding coarse-grained text. In this embodiment of the invention, the size of the first preset size is set according to the actual situation.

[0048] Step 202: Input the image block into the first image encoder, input the coarse-grained text into the first text encoder, generate the first image-text similarity matrix, and filter out the image block of the first target category based on the similarity between the first similarity matrix and the first discriminative category.

[0049] In this embodiment of the invention, after cropping the first input image into image blocks of the same size and matching each image block with a corresponding coarse-grained text, the image blocks are input into the first image encoder, and the coarse-grained text is input into the first text encoder to obtain the first similarity matrix. Since there is a correspondence between the image blocks and their corresponding coarse-grained texts, the image-text pair with the highest similarity within the first similarity matrix is ​​selected (when processing the similarity between the first similarity matrix and the first discrimination analogy, the first similarity matrix is ​​first sorted according to the similarity magnitude, and then the image blocks are assigned to the text with the highest corresponding similarity score, where the category described in the text corresponds to the category of the image block). Then, the category corresponding to each image block can be determined through the category described in the coarse-grained text. The selected first target category is then used to filter the image blocks within the first input image, filtering out image blocks of the first target category. For ease of understanding, the example of step 201 is used again. For instance, when the selected first target category is "crowd," the image blocks are filtered, and image blocks of the category "crowd" are filtered out based on the coarse-grained text description within the image blocks and input into the next step. Other non-crowd image blocks have their number of people set to 0.

[0050] Step 203: Set a second distinguishability category for the image blocks, obtain the fine-grained text corresponding to the image blocks of the first target category according to the second distinguishability category, input the fine-grained text into the second text encoder to generate a second image-text similarity matrix, and filter out the image blocks of the second target category according to the similarity of the second similarity matrix and the second distinguishability category.

[0051] Continuing with the example in step 201 of this embodiment, the first distinguishing category is "crowd," "tree," "light," and "vehicle." The first target category is "crowd," and the corresponding "crowd" image block may contain different parts of the human body (e.g., head, hands, body, and legs). However, the crowd counting task aims to estimate the number of heads rather than other body parts, and heads are less likely to be occluded compared to other body parts. Therefore, in order to more accurately measure the number of people in an image, this embodiment of the invention sets a second discrimination category, obtains fine-grained text corresponding to the second target category through the second discrimination category, and inputs the fine-grained text into the second text encoder to generate a second image-text similarity matrix. Then, the image-text pair with the highest similarity in the second similarity matrix is ​​selected for matching (when processing the similarity between the second similarity matrix and the second discrimination category, the second similarity matrix is ​​first sorted according to the similarity size, and then the image block is assigned to the text with the highest corresponding similarity score, where the category described by the text corresponds to the category of the image block). Then, the category corresponding to each image block can be known through the category described in the fine-grained text. Then, the image blocks of the first target category are filtered through the second target category to obtain the image blocks of the second target category. To facilitate understanding, we will continue with the example from step 202. For example, the second distinguishing category is "head", "hand", "body" and "leg"; the second target category is "head". The image blocks of the selected first target category (the first target category is "crowd") are matched one by one with the corresponding fine-grained text (for example, if the image is "head" and the corresponding category is "head"), and then the image blocks of the second target category are selected according to the second similarity matrix and the fine-grained text.

[0052] Step 204: Input the image patch of the second target category into the second image encoder and input the counted text into the third text encoder to generate the target similarity matrix of image and text, and obtain the number of people in the image based on the similarity between the target similarity matrix and the counted text.

[0053] After obtaining the image patches of the second target category, the image patches of the second target category are input into the second image, and the corresponding matching counting text is input into the third text encoder to generate an image-text target similarity matrix. The image-text pair with the highest similarity is selected through the target similarity matrix, where the number described by the text pair is the number of people, thereby realizing the counting of people. (When processing the similarity between the target similarity matrix and the counting text, the target similarity matrix is ​​first sorted according to the similarity size, and then the image patch is assigned to the corresponding counting text with the highest similarity score, where the number described by the counting text corresponds to the number of people in the image patch). It should be noted that when using the unsupervised crowd counting method of this embodiment of the invention to count people, the first text encoder, the second text encoder, and the third text encoder share the same network parameters.

[0054] This invention involves cropping a given first input image into image blocks of a first preset size, obtaining coarse-grained text corresponding to the image blocks based on a first discrimination category, processing the image blocks and coarse-grained text into a first similarity matrix using a first image encoder and a first text encoder, filtering out image blocks of a first target category based on the similarity between the first similarity matrix and the first discrimination category, setting a second discrimination category for the image blocks, obtaining fine-grained text corresponding to the image blocks of a second target category based on the second discrimination category, inputting the fine-grained text into a second text encoder to generate a second similarity matrix, filtering out image blocks of the second target category, processing the image blocks of the second target category using a second image encoder to generate an image-text target similarity matrix, and obtaining the number of people in the image based on the similarity between the target similarity matrix and the count text. This invention only requires corresponding text encoders and image encoders to process the corresponding text and images to obtain the number of people in the image. Compared with existing supervised dense crowd counting methods, this invention does not require any manual labels, greatly reducing the labeling cost. Furthermore, by leveraging the implicit ordinal relationship between the image and text, a similarity matrix is ​​constructed, and the number of people in the image is obtained by using the similarity matrix and the similarity of the text.

[0055] To provide a detailed explanation of the complete solution of the embodiments of the present invention, the details of the embodiments of the present invention will be described in detail below. The second image encoder is an image encoder that has been fine-tuned and trained, such as... Figure 3-4 As shown, the fine-tuning training specifically includes:

[0056] Step 301: The given second input image in the dataset is cropped into a set of square image blocks of a preset number according to different sizes, and the counting text corresponding to the square image block is constructed according to the number of people in each square image block.

[0057] In the embodiments of the present invention, in order to distinguish between the fine-tuning training process and the testing process (the testing process refers to the process of predicting the number of people using an image), the first input image and the second input image are used to distinguish between the two. In fact, there is no essential difference between the first input image and the second input image, and both are taken from the dataset of the acquired crowd images. The embodiments of the present invention follow the following rules for cropping the second input image:

[0058] a) For any two cropped image patches ( and 0 ≤ i < j ≤ M - 1), the size of the image patch is smaller than the size of , where i and j are integers, and M represents the total number of image patches.

[0059] b) The image patches from the same image share the same image center.

[0060] When a series of image patches with ordered sizes are successfully obtained, for the same image, the number of people's heads between the multiple generated image patches has an ordinal relationship (that is, the image with a larger image patch contains the image with a smaller image patch, and there is an inclusion relationship among the images in the cropping process, see Figure 4 shown). Comparatively, generally, the larger the image patch, the more or equal number of people's heads it contains. Therefore, the present invention uses text with semantic order to describe the ordinal relationship between different image patches. Specifically, by defining the text as "This picture has [class] people", where [class] represents a set of numbers with an order relationship (that is, a set with an ordinal relationship) R = [R0, R0 + K,..., R0 + (N - 1)K], where R0, K, and N represent the reference quantity benchmark, quantity interval, and number of categories respectively. The embodiments of the present invention set the preset number of the image patches cropped from the second input image according to the actual situation.

[0061] Step 302: Input the square image patch into the first image encoder to generate an image embedding with sorting properties, and input the counting text into the third text encoder to generate a text embedding with sorting properties.

[0062] Among them, in the embodiments of the present invention, before inputting the square image patch into the first image encoder to generate an image embedding with sorting properties, all square image patches need to be adjusted to the same second preset size. The size of the second preset size can be set according to the actual situation and will not be elaborated here. When all the image patches cropped from the second input image are adjusted to the same size and input into the image encoder to generate an image embedding with sorting properties (Imageembedding) I = [I0, I1,..., I M-1 , where C represents the feature dimension, and M represents the number of image patches in a set of image patches. The counted text is then input into a third text encoder to obtain a text embedding R′ = [R′0, R′1, ..., R′] with sorting properties. N-1 ],in Here, C represents the feature dimension, and N represents the number of categories used to describe a set of image patches in textual description. Since sorted images and sorted text are encoded into the same feature space, the images and text can be compared with each other. During the fine-tuning pre-training phase, a large number of crowd image patches and text pairs are used to train the image encoder. By learning from the contrast between images and text, the semantic relationships between them are captured, enabling the image encoder to more effectively perceive crowd semantic information, thereby improving crowd counting accuracy.

[0063] Step 303: Calculate the inner product of the image embedding and the text embedding to obtain the image-text training similarity matrix.

[0064] Specifically, after obtaining the image embedding I and text embedding R′ corresponding to a given second input image, the image-text training similarity matrix is ​​obtained by calculating their inner product. The image-text training similarity matrix S = [s...]. i,j ],in Specifically:

[0065]

[0066] in, And 0≤i≤M-1, 0≤j≤N-1; s i,j Let I represent the similarity between the i-th image patch embedding and the j-th text embedding. i and R′ j Let represent the embedding of the i-th image patch and the embedding of the j-th text description, respectively. T represents the transpose of the matrix, C represents the feature dimension, M represents the number of image patches in a set of image patches, and N represents the number of categories of text descriptions used to describe a set of image patches.

[0067] In this embodiment of the invention, after obtaining the second image encoder through fine-tuning training, the method further includes performing secondary fine-tuning on the second image encoder using a loss function. The secondary fine-tuning specifically includes: obtaining the training similarity matrix of the image-text; using the main diagonal of the similarity matrix as a reference, calculating the ordinal loss from bottom to top using a loss function; and aligning the image embedding with the text embedding to obtain the second image encoder after secondary fine-tuning.

[0068] To preserve the ordinal relationship between image embeddings and text embeddings in the feature space, a multimodal ordinal loss is used to fine-tune the trained image encoder. The ordinal loss is calculated from bottom to top, using the main diagonal of the similarity matrix S as a reference:

[0069] L r =max(0,s) i′,i -s i,i )

[0070] Among them, s i,j Embed I for the i-th image patch corresponding to the second input image i and the i-th text embedding R′ i The inner product, s i′,i To embed I into the i′-th image patch corresponding to the second input image i′ and the i-th text embedding R′ i The inner product of . Where , such that i′<i, realizes the calculation of ordinal loss from bottom to top.

[0071] To ensure that the similarity matrix S includes the main diagonal, the number of the first discriminant categories, the number of the first discriminant categories, and the preset number are usually equal, and M = N is set to ensure that S is a square matrix. During training, the text encoder weights are frozen, i.e., L r The aim is to align image embeddings into a fixed text embedding space with ordinal properties. In this way, since the original text encoder and image encoder are pre-trained with a large number of image-text pairs, the text embeddings in this invention are confined to a well-learned language space, resulting in strong generalization capabilities.

[0072] In addition, after obtaining the encoders (including the first text encoder, the second text encoder, and the third text encoder) and image encoders (including the first image encoder and the second image encoder) corresponding to the unsupervised crowd counting method of this embodiment of the invention, as follows: Figure 5 As shown, the unsupervised population counting method further includes:

[0073] Step 401: Collect multiple sample images and use the unsupervised crowd counting method to predict the number of people in the multiple sample images to obtain the predicted number of people corresponding to each sample image. Based on the predicted number of people corresponding to all sample images and the actual number of people, obtain the mean absolute error and mean square error.

[0074] After training and obtaining trained text encoders (including a first text encoder, a second text encoder, and a third text encoder) and image encoders (including a first image encoder and a second image encoder), the sample images are sequentially input into the corresponding text encoders and image encoders according to an unsupervised crowd counting method and order. This yields a predicted number of people for each sample image. Combined with the actual number of people in the sample images, the obtained text encoders and image encoders are evaluated to determine if they meet practical requirements. This invention uses Mean Absolute Error (MAE) and Mean Squared Error (MSE) to evaluate whether the image encoders and text encoders meet the requirements. The specific formulas for defining MAE and MSE in this invention are as follows:

[0075]

[0076]

[0077] Where K represents the number of images used for testing, y′ represents the predicted number of people, and y represents the actual number of people contained in the image.

[0078] Step 402: Set the mean absolute error threshold and the mean square error threshold. If the mean absolute error and the mean square error are not within the set range, continue to use the unsupervised population counting method for training to optimize the unsupervised population counting method.

[0079] The process of optimizing the unsupervised crowd counting method in this embodiment of the invention actually optimizes the model corresponding to this embodiment, namely the corresponding text encoder (including a first text encoder, a second text encoder, and a third text encoder) and image encoder (including a first image encoder and a second image encoder). After the model corresponding to this embodiment of the invention is optimized, when measuring the performance of the model, this embodiment of the invention needs to set comparison thresholds for mean absolute error (MAE) and mean squared error (MSE) according to actual needs. If the mean absolute error and the mean squared error are not within the set range, the model of this embodiment of the invention continues to be trained until the mean absolute error and the mean squared error measured by the sample images fall within the set range.

[0080] After obtaining the model corresponding to the unsupervised crowd counting method of this invention, a comparison was made between the prediction performance of the model of this invention and the CSS-CCNN model, such as... Figure 6The diagram illustrates the excellent counting capabilities of the model in this embodiment of the invention. It is worth noting that, based on the method proposed in this embodiment, performance results are presented on two common public datasets: ShanghaiTech_partA and UCF-QNRF. The ShanghaiTech_partA dataset contains 482 images, with 300 for training and 182 for testing. These images are primarily crawled from the internet, and the dataset density is relatively moderate, with an average of 501 images per person. The UCF-QNRF dataset contains 1535 images, with 1201 used for training and 334 for testing. This dataset has a very high density, with an average of 815 images per person and a maximum of 12865 images per person.

[0081] This invention involves cropping a given first input image into image blocks of a first preset size, obtaining coarse-grained text corresponding to the image blocks based on a first discrimination category, processing the image blocks and coarse-grained text into a first similarity matrix using a first image encoder and a first text encoder, filtering out image blocks of a first target category based on the similarity between the first similarity matrix and the first discrimination category, setting a second discrimination category for the image blocks, obtaining fine-grained text corresponding to the image blocks of a second target category based on the second discrimination category, inputting the fine-grained text into a second text encoder to generate a second similarity matrix, filtering out image blocks of the second target category, processing the image blocks of the second target category using a second image encoder to generate an image-text target similarity matrix, and obtaining the number of people in the image based on the similarity between the target similarity matrix and the count text. This invention only requires corresponding text encoders and image encoders to process the corresponding text and images to obtain the number of people in the image. Compared with existing supervised dense crowd counting methods, this invention does not require any manual labels, greatly reducing the labeling cost. Furthermore, by leveraging the implicit ordinal relationship between the image and text, a similarity matrix is ​​constructed, and the number of people in the image is obtained by using the similarity matrix and the similarity of the text.

[0082] Example 2:

[0083] like Figure 7 The diagram shown is an architectural schematic of an unsupervised crowd counting device according to an embodiment of the present invention. The unsupervised crowd counting device of this embodiment includes one or more processors 21 and a memory 22. Figure 7 Take a processor 21 as an example.

[0084] Processor 21 and memory 22 can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0085] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the unsupervised crowd counting method in Embodiment 1. The processor 21 executes the unsupervised crowd counting method by running the non-volatile software program and instructions stored in the memory 22.

[0086] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0087] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, they perform the unsupervised crowd counting method described in Embodiment 1 above, for example, the method described above. Figure 1-6 The steps shown.

[0088] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An unsupervised population counting method, characterized in that, include: A dataset of crowd images is obtained, a given first input image in the dataset is cropped into an image block of a first preset size, a first discrimination category of the image block is set, and coarse-grained text corresponding to the image block is obtained according to the first discrimination category; The image blocks are input into the first image encoder, and the coarse-grained text is input into the first text encoder to generate a first image-text similarity matrix. Based on the similarity between the first similarity matrix and the first discriminative category, image blocks of the first target category are selected. A second distinguishability category for image blocks is defined. Fine-grained text corresponding to image blocks of the first target category is obtained based on the second distinguishability category. The fine-grained text is input into a second text encoder to generate a second image-text similarity matrix. Based on the similarity between the second similarity matrix and the second distinguishability category, image blocks of the second target category are selected. The image patch of the second target category is input into the second image encoder, and the counted text is input into the third text encoder to generate the target similarity matrix of image and text. Based on the similarity between the target similarity matrix and the counted text, the number of people in the image is obtained.

2. The unsupervised population counting method according to claim 1, characterized in that, The second image encoder is a fine-tuned image encoder, and the fine-tuning training specifically includes: The given second input image in the dataset is cropped into a set of square image blocks of a preset number according to different sizes, and the counting text corresponding to the square image block is constructed according to the number of people in each square image block; The square image block is input into the first image encoder to generate an image embedding with sorting properties, and the counted text is input into the third text encoder to generate a text embedding with sorting properties. Calculate the inner product of the image embedding and the text embedding to obtain the image-text training similarity matrix.

3. The unsupervised population counting method according to claim 2, characterized in that, After obtaining the second image encoder through fine-tuning training, the method further includes performing secondary fine-tuning on the second image encoder using a loss function. The secondary fine-tuning specifically includes: Obtain the training similarity matrix of the image-text, and calculate the ordinal loss from bottom to top using the main diagonal of the similarity matrix as the reference. The image embedding is aligned with the text embedding to obtain a second image encoder after secondary fine-tuning.

4. The unsupervised population counting method according to claim 2, characterized in that, Before inputting the square image blocks into the first image encoder to generate an image embedding with sorting properties, all square image blocks need to be adjusted to the same second preset size.

5. The unsupervised population counting method according to claim 2, characterized in that, The image-text training similarity matrix S = [s i,j ],in Specifically: in, And 0≤i≤M-1, 0≤j≤N-1; s i,j Let I represent the similarity between the j-th image patch embedding and the j-th text embedding. i and R j ′ Let represent the embedding of the i-th image patch and the embedding of the j-th text patch, respectively. T represents the matrix transpose operation, C represents the feature dimension, M represents the number of image patches in a set of image patches, and N represents the number of categories of text descriptions used to describe a set of image patches.

6. The unsupervised population counting method according to claim 2, characterized in that, The number of the first discrimination category, the number of the second discrimination category, and the preset number are all equal.

7. The unsupervised population counting method according to any one of claims 1 to 6, characterized in that, The unsupervised population counting method also includes: Multiple sample images are collected, and the unsupervised crowd counting method is used to predict the number of people in the multiple sample images to obtain the predicted number of people for each sample image. The mean absolute error and mean square error are obtained by comparing the predicted number of people with the actual number of people in all sample images. Set a threshold for mean absolute error and a threshold for mean square error. If the mean absolute error and the mean square error are not within the set range, continue to use the unsupervised population counting method for training to optimize the unsupervised population counting method.

8. The unsupervised population counting method according to any one of claims 1 to 6, characterized in that, The first text encoder, the second text encoder, and the third text encoder share the same network parameters.

9. An unsupervised crowd counting device, characterized in that, include: At least one processor and at least one memory; The at least one processor and the at least one memory are communicatively connected to each other. The at least one memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the unsupervised population counting method as described in any one of claims 1-8.

10. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, which are executed by one or more processors to perform the unsupervised population counting method according to any one of claims 1-8 of the present invention.

Images