Dense crowd counting method based on language guidance network
By introducing language guidance networks and CLIP visual language models in crowd counts, combined with multi -level quantity reminder algorithms, the labeling of crowd counts in complex scenes is inaccurate and obstructed, and more efficient and accurate crowd counts are achieved.
Patent Information
- Application Number
- CN202411374172.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-29
- Publication Date
- 2025-01-21
AI Technical Summary
The existing crowd counting method has problems of inaccurate marking and occlusion in complex scenes, making it difficult to effectively distinguish the foreground and background of the image.
The intensive crowd count method based on language -guided networks is used to enhance the definition of sample boundaries by migrating the CLIP visual language model and constructing a loss function to enhance the sample boundary definition, and the multi -level number prompt algorithm to achieve fine -grained count.
It improves the accuracy of distinguishing the foreground and background of the image, effectively solves the occlusion problem, and significantly improves the counting accuracy in complex scenes.
Smart Images
Figure CN119339323A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to a dense crowd counting method based on a language guided network. Background Art
[0002] Crowd counting is a key task in computer vision. Its main task is to count the people in a given image containing a crowd. At present, the main method used by the academic community for crowd counting is to regress the density map through a deep convolutional neural network (CNN), that is, to mark the people in the image and form a density map full of marked points. The number of people in the original image can be obtained by summing up the density map. However, there is still a performance gap when using this method. This method has the problem of inaccurate labeling due to redundant feature interference and the inability to label when people are occluded.
[0003] With the development of deep convolutional neural networks in recent years, methods that do not use mainstream regression density maps have emerged. In previous work, some people have borrowed the image segmentation approach to apply image block classification to crowd counting tasks and achieved certain results. However, the block classification method does not perform well in complex scenes (such as the UCF CC 50 dataset) because in complex scenes, the image contains a large amount of redundant features and interference information. Using these features and information to classify the image in blocks will have a great impact on the final classification results. Summary of the invention
[0004] The purpose of the present invention is to provide a dense crowd counting method based on a language guided network, which is conducive to more efficiently and accurately distinguishing the foreground and background of an image, thereby accurately obtaining the specific number of people in the crowd.
[0005] In order to achieve the above object, the technical solution adopted by the present invention is: a method for counting dense crowds based on a language-guided network, characterized in that it includes the following steps:
[0006] Step S1: Convert the dot matrix of the crowd labels to produce a true density map label for highlighting the foreground and background areas;
[0007] Step S2: Migrate the CLIP visual language model to the counting task and construct a loss function for it to enhance the clarity of sample boundaries;
[0008] Step S3: Through the multi-level quantity prompt algorithm, the model training cost is saved without losing the counting accuracy, and fine-grained counting is achieved;
[0009] Step S4: Map the predicted density map to obtain the predicted number of people in the image.
[0010] Furthermore, the step S1 specifically includes the following steps:
[0011] Step S11: Use a Gaussian kernel with a size of 15×15 and a standard deviation of σ=4 Convert the dot map Point(x) to the ground truth density map Ground_Truth density ;
[0012]
[0013] In the formula, x is a pixel point. After the above processing, Gaussian processing of the head of the person in the picture is realized, so that its value obeys Gaussian distribution.
[0014] Step S12: Ground_Truth density Sum the global pixel values to get the real number of people Ground_Truth count :
[0015]
[0016] Where N represents the ground truth density map Ground_Truth density The number of pixels in the image.
[0017] Furthermore, the step S2 specifically includes the following steps:
[0018] Step S21: Migrate the CLIP visual language model to the counting task. The decoder consists of a 3×3 convolutional layer to regress the predicted density map Prediction density ;
[0019] Step S22: Construct a loss function for enhancing the clarity of sample boundaries; First, the original image image is processed by the image encoder to output the image block-level feature embedding feature patch ∈R height×width×depth , where height and width represent the number of image blocks per column and row, respectively, and depth represents the depth of the final output; then, linear projection is used Mapping image block-level feature embedding to text feature embedding feature text Same dimensions:
[0020]
[0021] The image block-level feature embedding is used to divide the positive sample block feature embedding containing the correct target information pos , and the negative sample block feature embedding feature containing incorrect target informationneg ; Define the positive and negative sample block sets as Pos and Neg respectively; According to the characteristics of the positive and negative samples of the image, use the maximum pooling layer to locate the positions of the positive and negative sample blocks; Use Gaussian processing on the dot map of the image to obtain the true value density map, and then apply the maximum pooling layer on the true value density map to obtain the block-level binary mask of the image. The mask is embedded with the image block-level feature feature patch The same size; the image block containing the target information in the definition mask belongs to the positive sample reference block mask pos , the image block that does not contain any target information in the mask belongs to the negative sample reference block mask neg ; Embed binary mask and image block level features into feature patch Multiply to get the positive sample block set Pos and the negative sample block set Neg:
[0022]
[0023] In order to enable the model to clearly distinguish positive and negative samples, increase the contribution of positive samples, and curb the adverse effects of negative samples on the model, a loss function Loss is constructed that can enhance the clarity of sample boundaries. bor :
[0024]
[0025] In the formula, cosSim() represents cosine similarity; the above formula embeds the positive sample block features into Embedding feature with text features text Close the distance and expand the feature embedding of negative sample blocks Embedding feature with text features text To achieve the purpose of aligning the image feature embedding space with the text feature embedding space;
[0026] Step S23: embedding the text hint without indicating quantity from the text feature into the feature text Separation and embedding as text negative sample features A specific number of text prompts are used as text positive samples Construct the final CLIP model counting problem loss optimization function Loss clip :
[0027]
[0028] Furthermore, the step S3 specifically includes the following steps:
[0029] Step S3 1: Each layer of the CLIP text encoder outputs text semantic information, which is inserted into each layer of each image encoder to assist the image encoder in extracting visual features with salient target semantics; meanwhile, learnable visual cue parameters are introduced to fine-tune the image encoder;
[0030] Step S32: After the Patch Embedding operation, the original image image is output as N image block-level feature embedding features patch ∈R height×width×depth , and then freeze the parameters of each ViT encoder layer; introduce an additional set of trainable parameters ρ, and the trainable parameters for the layer are defined as p layer , then the set of trainable parameters ρ input to the layer layer The definition is as follows:
[0031]
[0032] Where cls represents the cls-th learnable visual cue, Represents the clsth trainable parameter;
[0033] At the same time, the text semantic information t output by the text encoder l Add to each learnable visual cue to obtain the final input to the model fine-tuning trainable parameter set θ of the lth layer l :
[0034]
[0035] Furthermore, the specific implementation method of step S4 is:
[0036] Sum the points of the obtained predicted density map to get the predicted number of people in the picture Count image , the formula is as follows:
[0037]
[0038] Compared with the prior art, the present invention has the following beneficial effects:
[0039] 1. The present invention improves the counting convolutional neural network and creatively proposes a method of using natural language to guide auxiliary counting, which strengthens the counting network's attention to the crowd part in the image and improves the counting effect of the network model on the crowd;
[0040] 2. The present invention is different from the traditional counting method based on deep neural network regression density map. The density map counting method is not effective for people who are occluded in the image. The present invention designs a loss function to enhance the clarity of sample boundaries, strengthens the image block-level feature embedding and text feature embedding representation, improves the positioning ability of the CLIP model in counting problems, and effectively solves the occlusion problem.
[0041] 3. Due to the large number of parameters in the CLIP model image encoder, fine-tuning the Transformer usually requires a large amount of data, and end-to-end training on the complex and difficult-to-obtain annotation task of object counting requires high overhead. This paper designs a multi-level quantity prompting algorithm to save resource overhead to the maximum extent and complete the efficient task migration of the CLIP model. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 It is a flow chart of the method implementation of the embodiment of the present invention. DETAILED DESCRIPTION
[0043] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0044] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.
[0045] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0046] like Figure 1 As shown, this embodiment provides a dense crowd counting method based on a language guided network, comprising the following steps:
[0047] Step S1: Convert the dot matrix of the crowd label to produce a true value density map label that can highlight the foreground and background areas;
[0048] Step S2: Migrate the CLIP visual language model to the counting task and construct a loss function that can enhance the clarity of sample boundaries, so that the model can clearly distinguish positive and negative samples, increase the contribution of positive samples, and curb the adverse effects of negative samples on the model;
[0049] Step S3: Through the multi-level quantity prompt algorithm, the model training cost is saved to the maximum extent without losing the counting accuracy, and fine-grained counting is achieved;
[0050] Step S4: Map the predicted density map to obtain the predicted number of people in the image.
[0051] In this embodiment, the specific implementation steps of step S1 are as follows.
[0052] Step S11: Use a Gaussian kernel with a size of 15×15 and a standard deviation of σ=4 Convert the dot map Point(x) to the ground truth density map Ground_Truth density ;
[0053]
[0054] In the formula, x is a pixel point. After being processed by the above formula, Gaussian processing of the head of the person in the picture is achieved, so that its value obeys Gaussian distribution.
[0055] Step S12: Ground_Truth density Sum the global pixel values to get the real number of people Ground_Truth count :
[0056]
[0057] Where N represents the ground truth density map Ground_Truth density The number of pixels in the image.
[0058] In this embodiment, the specific implementation steps of step S2 are as follows.
[0059] Step S21: Migrate the CLIP visual language model to the counting task. The decoder consists of a 3×3 convolutional layer to regress the predicted density map Prediction density .
[0060] Step S22: Construct a loss function that can enhance the clarity of sample boundaries. First, the original image image is processed by the image encoder to output the image block-level feature embedding feature patch ∈R height×width×depth , where height and width are both 14, representing the number of image blocks per column and row, respectively, and depth is 768, representing the depth of the final output; then, linear projection is used Mapping image block-level feature embedding to text feature embedding feature text Same dimensions:
[0061]
[0062] The image block-level feature embedding is divided into the positive sample block feature embedding featurepos containing the correct target information and the negative sample block feature embedding feature containing the wrong target information neg ; Define the positive and negative sample block sets as Pos and Neg respectively; According to the characteristics of the positive and negative samples of the image, use the maximum pooling layer to locate the positions of the positive and negative sample blocks; Use Gaussian processing on the dot map of the image to obtain the true value density map, and then apply the maximum pooling layer on the true value density map to obtain the block-level binary mask of the image. The mask size is 14×14, and the mask contains a total of 196 image blocks, which are embedded with the image block-level feature feature patch The same size; the image block containing the target information in the mask belongs to the positive sample reference block maskpos, and the image block that does not contain any target information in the mask belongs to the negative sample reference block mask neg ; Embed binary mask and image block level features into feature patch By multiplying them, we can get the positive sample block set Pos and the negative sample block set Neg:
[0063]
[0064] In order to enable the model to clearly distinguish positive and negative samples, increase the contribution of positive samples, and curb the adverse effects of negative samples on the model, a loss function Loss is constructed that can enhance the clarity of sample boundaries. bor :
[0065]
[0066] In the formula, cosSim() represents cosine similarity; the above formula embeds the positive sample block features into Embedding feature with text features text Close the distance and expand the feature embedding of negative sample blocks Embedding feature with text features text The loss function mainly focuses on adjusting the distance between image and text feature embeddings. However, it only relies on positive and negative samples of images for distance judgment, but ignores the difference in the degree of guidance of text prompts in target quantity, and cannot fully utilize natural language to guide counting. Therefore, it is necessary to make targeted improvements to the loss function according to the characteristics of text prompts.
[0067] Step S23: After analyzing a large number of existing target counting text prompts, it is found that as the number of targets in the original image gradually increases, the natural language with precise number guidance in the corresponding text prompts will gradually become ambiguous. When the number is too large, the text prompts often do not directly indicate the specific number, but use "a group of" or "many" instead of the specific number, which can easily cause quantitative ambiguity in the model. Therefore, in order to reduce the impact of this part of the text prompts on the model, this type of text prompts that do not accurately indicate the number are embedded from the text feature into the feature text Separation and embedding as text negative sample features A specific number of text prompts are used as text positive samples Construct the final CLIP model counting problem loss optimization function Loss clip :
[0068]
[0069] In this embodiment, the specific implementation steps of step S3 are as follows.
[0070] Step S31: Each layer of the CLIP text encoder outputs text semantic information, which is inserted into each layer of each image encoder to assist the image encoder in extracting visual features that highlight the target semantics; at the same time, a small amount of learnable visual cue parameters are introduced to fine-tune the image encoder.
[0071] Step S32: After the Patch Embedding operation, the original image image is output as N image block-level feature embedding features patch ∈R height×width×depth , where height and width are both 14, indicating the number of image blocks per column and row, depth is 768, indicating the depth of the output, and then each ViT (Visual Transformer) encoder layer parameter is frozen. In the original ViT network model, this part of the image block level feature embedding is only input into the ViT encoder layer together with the class token. The multi-level quantity cue feature learning method proposed in this invention introduces an additional trainable parameter set ρ (cls = 15, representing a total of 15 learnable visual cues), and the trainable parameter for the layer is defined as p layer , then the set of trainable parameters ρ input to the layer layer The definition is as follows:
[0072]
[0073] Where cls represents the cls-th learnable visual cue, Represents the cls-th trainable parameter.
[0074] At the same time, the text semantic information t output by the text encoder l Add to each learnable visual cue to obtain the final input to the model fine-tuning trainable parameter set θ of the lth layer l :
[0075]
[0076] After introducing multiple levels of trainable quantity prompts, the backbone network part does not need to undergo expensive end-to-end training, and the model is fine-tuned with minimal overhead, enhancing the model's generalization ability for complex target group images.
[0077] In this embodiment, the specific implementation method of step S4 is:
[0078] Sum the points of the obtained predicted density map to get the predicted number of people in the picture Count image , the formula is as follows:
[0079]
[0080] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0081] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0082] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0083] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0084] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.
Claims
1. A dense crowd counting method based on a language guided network, characterized in that: The following steps are involved: Step S1: Convert the dot matrix of the crowd labels to produce a true density map label for highlighting the foreground and background areas; Step S2: Migrate the CLIP visual language model to the counting task and construct a loss function for it to enhance the clarity of sample boundaries; Step S3: Through the multi-level quantity prompt algorithm, the model training cost is saved without losing the counting accuracy, and fine-grained counting is achieved; Step S4: Map the predicted density map to obtain the predicted number of people in the image.
2. The method for counting dense crowds based on a language-guided network according to claim 1, characterized in that: The step S1 specifically includes the following steps: Step S11: Use a Gaussian kernel with a size of 15×15 and a standard deviation of σ=4 Convert the dot map Point(x) to the ground truth density map Ground_Truth density ; In the formula, x is a pixel point. After the above processing, Gaussian processing of the head of the person in the picture is realized, so that its value obeys Gaussian distribution. Step S12: Ground_Truth density Sum the global pixel values to get the real number of people Ground_Truth count : Where N represents the ground truth density map Ground_Truth density The number of pixels in the image.
3. The method for counting dense crowds based on a language-guided network according to claim 1, characterized in that: The step S2 specifically includes the following steps: Step S21: Migrate the CLIP visual language model to the counting task. The decoder consists of a 3×3 convolutional layer to regress the predicted density map Prediction density ; Step S22: Construct a loss function for enhancing the clarity of sample boundaries; First, the original image image is processed by the image encoder to output the image block-level feature embedding feature patch ∈R height×width×depth , where height and width represent the number of image blocks per column and row, respectively, and depth represents the depth of the final output; then, linear projection is used Mapping image block-level feature embedding to text feature embedding feature text Same dimensions: The image block-level feature embedding is used to divide the positive sample block feature embedding containing the correct target information pos , and the negative sample block feature embedding feature containing incorrect target information neg ; Define the positive and negative sample block sets as Pos and Neg respectively; According to the characteristics of the positive and negative samples of the image, use the maximum pooling layer to locate the positions of the positive and negative sample blocks; Use Gaussian processing on the dot map of the image to obtain the true value density map, and then apply the maximum pooling layer on the true value density map to obtain the block-level binary mask of the image. The mask is embedded with the image block-level feature feature patch The same size; the image block containing the target information in the definition mask belongs to the positive sample reference block mask pos , the image block that does not contain any target information in the mask belongs to the negative sample reference block mask neg ; Embed binary mask and image block level features into feature patch Multiply to get the positive sample block set Pos and the negative sample block set Neg: In order to enable the model to clearly distinguish positive and negative samples, increase the contribution of positive samples, and curb the adverse effects of negative samples on the model, a loss function Loss is constructed that can enhance the clarity of sample boundaries. bor : In the formula, cosSim( ) represents cosine similarity; the above formula embeds the positive sample block features into Embedding feature with text features text Close the distance and expand the feature embedding of negative sample blocks Embedding feature with text features text To achieve the purpose of aligning the image feature embedding space with the text feature embedding space; Step S23: embedding the text hint without indicating quantity from the text feature into the feature text Separation and embedding as text negative sample features A specific number of text prompts are used as text positive samples Construct the final CLIP model counting problem loss optimization function Loss clip :
4. The method for counting dense crowds based on a language-guided network according to claim 1, characterized in that: The step S3 specifically comprises the following steps: Step S31: Each layer of the CLIP text encoder outputs text semantic information, which is inserted into each layer of each image encoder to assist the image encoder in extracting visual features with prominent target semantics; meanwhile, learnable visual cue parameters are introduced to fine-tune the image encoder; Step S32: After the Patch Embedding operation, the original image image is output as N image block-level feature embedding features patch∈ R height×width×depth , and then freeze the parameters of each ViT encoder layer; introduce an additional set of trainable parameters ρ, and the trainable parameters for the layer are defined as P layer , then the set of trainable parameters ρ input to the layer layer The definition is as follows: Where cls represents the cls-th learnable visual cue, Represents the clsth trainable parameter; At the same time, the text semantic information t output by the text encoder l Add to each learnable visual cue to obtain the final input to the model fine-tuning trainable parameter set θ of the lth layer l :
5. The method for counting dense crowds based on a language-guided network according to claim 1, characterized in that: The specific implementation method of step S4 is: Sum the points of the obtained predicted density map to get the predicted number of people in the picture Count image , the formula is as follows: