Crowd counting method and device, terminal equipment and storage medium
By constructing a crowd counting model based on the DarkNet53 framework and utilizing image adaptive Transformer enhancement and multi-scale weight branching modules, the problem of low detection accuracy of crowd counting models was solved, and the accuracy of crowd counting was improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA MOBILE GROUP JIANGSU
- Filing Date
- 2023-07-05
- Publication Date
- 2026-05-29
AI Technical Summary
The detection and counting accuracy of existing crowd counting models is not high, mainly due to factors such as blurred video quality, crowd occlusion, increased counting difficulty caused by different crowd scales, and low accuracy of predictions based on numerical values and density maps.
A crowd counting model based on the DarkNet53 framework is adopted. By enhancing the image with an adaptive Transformer, adding multi-scale weights and fully connected branch modules, a multi-objective loss function is constructed to improve the accuracy of crowd counting.
By using weighted calculations of density map predictions and head count predictions, the accuracy of population counting is improved, solving the problem of inaccurate counting in existing technologies.
Smart Images
Figure CN116884033B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of public safety technology, and in particular to a crowd counting method, apparatus, terminal device, and storage medium. Background Technology
[0002] With the increase in group gatherings, it is necessary to effectively manage and control these gatherings through crowd counting to avoid congestion or even stampedes.
[0003] Existing technologies typically employ a crowd counting model based on video frame information. By inputting video frames acquired by a camera into the crowd counting model, the number of people within a given time period can be determined in real time. If the number of people exceeds a set threshold, measures can be taken immediately to guide and divert the crowd, ensuring the safety of people's lives and property.
[0004] In conceiving and implementing this application, the inventors discovered at least the following problems: Under real-world conditions, the crowd counting model described above has low detection accuracy. The main reasons include: First, the video is captured by a camera, resulting in relatively blurry image quality. In situations with a large crowd, the movement and occlusion of people make it difficult for the model to extract effective features. Second, crowd features are clearer near the camera, but less so further away, and the different scales of different individuals increase the counting difficulty. Third, existing methods based on numerical prediction and density map prediction have low accuracy. These factors contribute to the low accuracy of crowd counting.
[0005] Therefore, it is necessary to propose a solution to improve the accuracy of population counting.
[0006] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention
[0007] The main objective of this invention is to provide a crowd counting method, apparatus, terminal device, and storage medium, with the aim of improving the accuracy of crowd counting.
[0008] To achieve the above objectives, the present invention provides a crowd counting method, the crowd counting method comprising:
[0009] Obtain the target image;
[0010] The target image is predicted by a pre-trained crowd counting model to obtain density map prediction values and head count prediction values.
[0011] The population count is obtained by weighting the density map prediction value and the head count prediction value.
[0012] Optionally, the crowd counting model includes a multi-scale weighted branch module and a fully connected branch module. Before the step of predicting the target image using the pre-trained crowd counting model to obtain the density map prediction value and the head count prediction value, the model further includes:
[0013] Acquire pre-collected sample data, wherein the sample data includes sample images and corresponding real labels;
[0014] The sample image is enhanced by an adaptive conversion module to obtain a sample feature map;
[0015] The sample feature map is processed by the multi-scale weighted branch module to obtain the sample density map prediction value;
[0016] The predicted value of the sample density map is input into the fully connected branch module for calculation to obtain the predicted value of the sample head count.
[0017] Based on the multi-objective loss function framework, the crowd counting model is trained according to the predicted values of the sample density map, the predicted values of the sample head count, and the corresponding real labels.
[0018] Optionally, the step of enhancing the sample image through the adaptive transformation module to obtain the sample feature map includes:
[0019] The sample image is segmented and vertically combined to obtain a vertically combined image;
[0020] The vertically combined image is processed through a backbone network to obtain a scaled feature map;
[0021] The transformed scale feature map is upsampled to obtain the original scale feature map;
[0022] The original scale feature map is integrated and its position is restored by the adaptive transformation module to obtain the sample feature map.
[0023] Optionally, the sample feature map includes a first-size feature map, a second-size feature map, and a third-size feature map, and the predicted value of the sample density map includes a first density map, a second density map, and a third density map. The step of processing the sample feature map through the multi-scale weighted branch module to obtain the predicted value of the sample density map includes:
[0024] Upsampling is performed on the first size feature map, the second size feature map, and the third size feature map respectively to obtain the first weight matrix, the second weight matrix, and the third weight matrix;
[0025] The first weight matrix, the second weight matrix, and the third weight matrix are concatenated to obtain a three-dimensional weight array, and the three-dimensional weight array is normalized to obtain a fourth weight matrix, a fifth weight matrix, and a sixth weight matrix.
[0026] The first, second, and third size feature maps are downsampled respectively to obtain the fourth, fifth, and sixth size feature maps;
[0027] The first density map, the second density map, and the third density map are obtained by performing dot products on the fourth weight matrix with the fourth size feature map, the fifth weight matrix with the fifth size feature map, and the sixth weight matrix with the sixth size feature map, respectively.
[0028] Optionally, the true labels include density map labels and population count labels. The step of training the population counting model based on the multi-objective loss function framework, according to the sample density map prediction values, the sample head count prediction values, and the corresponding true labels, includes:
[0029] Calculate the density map loss function based on the sample density map prediction values and density map labels;
[0030] Calculate the crowd size loss function based on the predicted head count and crowd size label of the sample;
[0031] By substituting the density map loss function and the crowd size loss function into the multi-objective loss function framework, the multi-objective loss function is calculated.
[0032] The multi-objective loss function is fed back to the crowd counting model to update the network parameters of the crowd counting model. Then, the process of processing the sample feature map through the multi-scale weight branch module to obtain the sample density map prediction value and subsequent steps are performed. This process is repeated to iterate the parameters until the crowd counting model converges, at which point the training is terminated, and the trained crowd counting model is obtained.
[0033] Optionally, the step of calculating the density map loss function based on the sample density map prediction value and the density map label includes:
[0034] Based on the first density map, the second density map, the third density map and the corresponding density map labels, calculate the loss weight and density map error ratio for each density map.
[0035] The density map loss function is calculated based on the loss weights of each density map and the density map error ratio term.
[0036] Optionally, the step of calculating the crowd size loss function based on the predicted sample head count and the crowd size label includes:
[0037] The mean square error and the ratio of population number error are calculated based on the predicted value of the sample head count and the population number label.
[0038] The population loss function is calculated based on the mean square error and the ratio of population number error.
[0039] Furthermore, to achieve the above objectives, the present invention also provides a crowd counting device, the crowd counting device comprising:
[0040] The acquisition module is used to acquire the target image;
[0041] The prediction module is used to predict the target image using a pre-trained crowd counting model to obtain the density map prediction value and the head count prediction value.
[0042] The weighting module is used to perform weighted calculations on the density map prediction value and the head count prediction value to obtain the population count value.
[0043] In addition, to achieve the above objectives, the present invention also provides a terminal device, the terminal device including a memory, a processor, and a crowd counting program stored in the memory and executable on the processor, wherein the crowd counting program, when executed by the processor, implements the steps of the crowd counting method as described above.
[0044] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a crowd counting program, which, when executed by a processor, implements the steps of the crowd counting method as described above.
[0045] This invention proposes a crowd counting method, apparatus, terminal device, and storage medium. The method involves acquiring a target image; predicting the target image using a pre-trained crowd counting model to obtain a density map prediction value and a head count prediction value; and weighting the density map prediction value and the head count prediction value to obtain a crowd count value. By using the head count prediction value as an auxiliary result and weighting it with the density map prediction value, the accuracy of the obtained crowd count value is improved, thereby enhancing the accuracy of crowd counting. Attached Figure Description
[0046] Figure 1 This is a schematic diagram of the functional modules of the terminal device to which the crowd counting device of the present invention belongs;
[0047] Figure 2 This is a flowchart illustrating an exemplary embodiment of the crowd counting method of the present invention;
[0048] Figure 3This is a flowchart illustrating another exemplary embodiment of the crowd counting method of the present invention;
[0049] Figure 4 This is a schematic diagram of the image enhancement process in an embodiment of the present invention;
[0050] Figure 5 for Figure 3 A detailed flowchart of step S003 in the embodiment;
[0051] Figure 6 This is a schematic diagram of the model branching framework in an embodiment of the present invention;
[0052] Figure 7 for Figure 3 A schematic diagram of the specific process of step S005 in the embodiment;
[0053] Figure 8 This is a schematic diagram of the overall process in an embodiment of the present invention.
[0054] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0055] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0056] The main solution of this invention is as follows: acquiring a target image; predicting the target image using a pre-trained crowd counting model to obtain a density map prediction value and a head count prediction value; weighting the density map prediction value and the head count prediction value to obtain a crowd count value; by using the head count prediction value as an auxiliary result and weighting it with the density map prediction value, the accuracy of the obtained crowd count value is improved, thereby improving the accuracy of crowd counting.
[0057] Current technologies typically rely on video frame information to build crowd counting models. These models input video frames captured by cameras to determine the number of people in a given time period. If the crowd exceeds a set threshold, immediate measures can be taken to manage and divert the crowd, ensuring public safety. However, in reality, the accuracy of these crowd counting models is not high. The main reasons are: First, video footage captured by cameras is often blurry, and in large crowds, the movement and occlusion of people make it difficult for the model to extract effective features. Second, crowd features are clearer near the camera, but less so further away, and the varying scales of different individuals increase the difficulty of counting. Third, existing methods based on numerical prediction and density map prediction are not very accurate.
[0058] This invention addresses the problem of inaccurate crowd counting by providing a crowd counting model based on the DarkNet53 framework. The model employs three methods: image adaptive Transformer enhancement, adding multi-scale weights and fully connected branch modules, and reconstructing a multi-objective loss function. This improves the accuracy of crowd counting for newly input images and provides a foundation for subsequent downstream tasks.
[0059] Specifically, refer to Figure 1 , Figure 1 This is a functional module diagram of the terminal device to which the crowd counting device of the present invention belongs. The crowd counting device can be an independent device capable of counting people, separate from the terminal device, and can be implemented on the terminal device in hardware or software form. The terminal device can be a smart mobile terminal with data processing capabilities, such as a mobile phone or tablet computer, or a fixed terminal device or server with data processing capabilities.
[0060] In this embodiment, the terminal device to which the crowd counting device belongs includes at least an output module 110, a processor 120, a memory 130, and a communication module 140.
[0061] The memory 130 stores the operating system and a crowd counting program. The crowd counting device can store information such as the acquired target image, the density map prediction value and the head count prediction value obtained by predicting the target image through a pre-trained crowd counting model, and the crowd count value obtained by weighting the density map prediction value and the head count prediction value in the memory 130. The output module 110 can be a display screen, etc. The communication module 140 can include a WIFI module, a mobile communication module, and a Bluetooth module, etc., and communicates with external devices or servers through the communication module 140.
[0062] When the crowd counting program in memory 130 is executed by the processor, it performs the following steps:
[0063] Obtain the target image;
[0064] The target image is predicted by a pre-trained crowd counting model to obtain density map prediction values and head count prediction values.
[0065] The population count is obtained by weighting the density map prediction value and the head count prediction value.
[0066] Furthermore, when the crowd counting program in memory 130 is executed by the processor, it also performs the following steps:
[0067] Acquire pre-collected sample data, wherein the sample data includes sample images and corresponding real labels;
[0068] The sample image is enhanced by an adaptive conversion module to obtain a sample feature map;
[0069] The sample feature map is processed by the multi-scale weighted branch module to obtain the sample density map prediction value;
[0070] The predicted value of the sample density map is input into the fully connected branch module for calculation to obtain the predicted value of the sample head count.
[0071] Based on the multi-objective loss function framework, the crowd counting model is trained according to the predicted values of the sample density map, the predicted values of the sample head count, and the corresponding real labels.
[0072] Furthermore, when the crowd counting program in memory 130 is executed by the processor, it also performs the following steps:
[0073] The sample image is segmented and vertically combined to obtain a vertically combined image;
[0074] The vertically combined image is processed through a backbone network to obtain a scaled feature map;
[0075] The transformed scale feature map is upsampled to obtain the original scale feature map;
[0076] The original scale feature map is integrated and its position is restored by the adaptive transformation module to obtain the sample feature map.
[0077] Furthermore, when the crowd counting program in memory 130 is executed by the processor, it also performs the following steps:
[0078] Upsampling is performed on the first size feature map, the second size feature map, and the third size feature map respectively to obtain the first weight matrix, the second weight matrix, and the third weight matrix;
[0079] The first weight matrix, the second weight matrix, and the third weight matrix are concatenated to obtain a three-dimensional weight array, and the three-dimensional weight array is normalized to obtain a fourth weight matrix, a fifth weight matrix, and a sixth weight matrix.
[0080] The first, second, and third size feature maps are downsampled respectively to obtain the fourth, fifth, and sixth size feature maps;
[0081] The first density map, the second density map, and the third density map are obtained by performing dot products on the fourth weight matrix with the fourth size feature map, the fifth weight matrix with the fifth size feature map, and the sixth weight matrix with the sixth size feature map, respectively.
[0082] Furthermore, when the crowd counting program in memory 130 is executed by the processor, it also performs the following steps:
[0083] Calculate the density map loss function based on the sample density map prediction values and density map labels;
[0084] Calculate the crowd size loss function based on the predicted head count and crowd size label of the sample;
[0085] By substituting the density map loss function and the crowd size loss function into the multi-objective loss function framework, the multi-objective loss function is calculated.
[0086] The multi-objective loss function is fed back to the crowd counting model to update the network parameters of the crowd counting model. Then, the process of processing the sample feature map through the multi-scale weight branch module to obtain the sample density map prediction value and subsequent steps are performed. This process is repeated to iterate the parameters until the crowd counting model converges, at which point the training is terminated, and the trained crowd counting model is obtained.
[0087] Furthermore, when the crowd counting program in memory 130 is executed by the processor, it also performs the following steps:
[0088] Based on the first density map, the second density map, the third density map and the corresponding density map labels, calculate the loss weight and density map error ratio for each density map.
[0089] The density map loss function is calculated based on the loss weights of each density map and the density map error ratio term.
[0090] Furthermore, when the crowd counting program in memory 130 is executed by the processor, it also performs the following steps:
[0091] The mean square error and the ratio of population number error are calculated based on the predicted value of the sample head count and the population number label.
[0092] The population loss function is calculated based on the mean square error and the ratio of population number error.
[0093] This embodiment, through the above-described scheme, specifically involves acquiring a target image; predicting the target image using a pre-trained crowd counting model to obtain density map prediction values and head count prediction values; and weighting the density map prediction values and head count prediction values to obtain a crowd count value. By using the head count prediction value as an auxiliary result and weighting it with the density map prediction value, the accuracy of the obtained crowd count value is improved, thereby enhancing the accuracy of crowd counting.
[0094] Based on, but not limited to, the terminal device architecture described above, embodiments of the method of the present invention are proposed.
[0095] The subject executing the method in this embodiment can be a crowd counting device or a terminal device, etc. This embodiment takes a crowd counting device as an example.
[0096] Reference Figure 2 , Figure 2 This is a flowchart illustrating an exemplary embodiment of the crowd counting method of the present invention. The crowd counting method includes:
[0097] Step S10: Obtain the target image;
[0098] Specifically, when there are many people in a crowd, due to the shooting angle, people who are closer to the image occupy a larger proportion, while those who are farther away occupy a smaller proportion and are more blurry. In addition, occlusion between people will also result in less information in the extracted features, which will lead to inaccurate crowd counting. To solve the problem of indistinct features caused by occlusion, it is necessary to enhance the local information of the image.
[0099] Optionally, the steps to obtain the target image include:
[0100] Get the original image;
[0101] The original image is enhanced using an adaptive conversion module to obtain the target image.
[0102] Specifically, in this embodiment of the invention, an adaptive Transformer module is introduced to enhance local information of the original image. During the enhancement process of the original image through the adaptive transformation module (i.e., the adaptive Transformer module), the original image is first segmented and vertically combined to obtain a vertically combined image. VGG16 is selected as the backbone network for the vertically combined image. After passing through the backbone network, the feature map of the image is obtained. The feature map is upsampled to restore the image size and number of channels. An image with extracted features, consistent with the original image size, is obtained, and the Transformer module is designed. The module selected is the Encoder module, designed with 12 layers to ensure that each image can integrate information from the other three images, restoring the vertically combined image to its original position, completing the local information enhancement of the image, obtaining the target image, and then using it for prediction.
[0103] Step S20: The target image is predicted using a pre-trained crowd counting model to obtain the density map prediction value and the head count prediction value.
[0104] Furthermore, after obtaining the target image, predictions can be made using a pre-trained crowd counting model. Optionally, in this embodiment of the invention, a multi-scale weighted branch module and a fully connected branch module are added to the backbone network to integrate the advantages of multi-scale methods.
[0105] Optionally, the step of predicting the density map and the number of heads in the target image using a pre-trained crowd counting model includes:
[0106] The target image is processed by the multi-scale weighted branch module to obtain the density map prediction value;
[0107] The density map prediction value is input into the fully connected branch module for calculation to obtain the head number prediction value.
[0108] Specifically, this embodiment of the invention designs three weight branching frameworks, each selecting feature maps of different sizes in the backbone network for several upsampling operations to reconstruct the corresponding weight matrices from the feature maps of each size. These weight matrices are then concatenated to obtain a three-dimensional weight array. A SoftMax operation is performed along each channel to normalize the values of the weight array to between 0 and 1. The array is then decomposed along each channel to obtain new weight matrices, which are then multiplied by their respective feature maps to obtain the corresponding density map prediction values. Optionally, the density map prediction values obtained from the multi-scale weight branching module are input into the fully connected branching module to output the corresponding head number prediction values.
[0109] Step S30: The density map prediction value and the head count prediction value are weighted and calculated to obtain the population count value.
[0110] Furthermore, by predicting the target image using a pre-trained crowd counting model, and obtaining the density map prediction value and the head count prediction value, the density map prediction value and the head count prediction value can be weighted to obtain the crowd count value. In this process, the density map prediction value and the head count prediction value can be assigned corresponding weights according to the actual situation. This embodiment of the invention does not make specific limitations on this.
[0111] In this embodiment, a target image is acquired; a pre-trained crowd counting model is used to predict the target image to obtain a density map prediction value and a head count prediction value; the density map prediction value and the head count prediction value are weighted to obtain a crowd count value. By using the head count prediction value as an auxiliary result and weighting it with the density map prediction value, the accuracy of the obtained crowd count value is improved, thereby improving the accuracy of crowd counting.
[0112] Reference Figure 3 , Figure 3This is a flowchart illustrating another exemplary embodiment of the crowd counting method of the present invention. Based on the above... Figure 2 In the embodiment shown, before the step of predicting the target image using a pre-trained crowd counting model to obtain the density map prediction value and the head count prediction value, the crowd counting method further includes:
[0113] Step S00: Train the crowd counting model. In this embodiment, step S00 is implemented before step S10. In other embodiments, step S00 can also be implemented between step S10 and step S20.
[0114] Compared to the above Figure 2 The embodiment shown also includes a scheme for training the crowd counting model.
[0115] Specifically, the steps for training the crowd counting model may include:
[0116] Step S001: Obtain pre-collected sample data, wherein the sample data includes sample images and corresponding real labels;
[0117] Specifically, in the process of training the crowd counting model, pre-collected sample data can be used, including sample images of different sizes and the corresponding real labels for each sample image. Optionally, the real labels include density map labels and crowd number labels, which are used to calculate the corresponding loss function.
[0118] Step S002: Enhance the sample image using an adaptive conversion module to obtain a sample feature map;
[0119] Reference Figure 4 , Figure 4 This is a schematic diagram of the image enhancement process in an embodiment of the present invention. After obtaining the sample image, the image is divided into four parts, labeled 1, 2, 3, and 4 respectively. Then, the four images are combined vertically, as shown below. Figure 4 The second module in the process involves using VGG16 as the backbone network for vertical image combination. After passing through the backbone, the feature map of the image is obtained, with the number of channels C=1 and the size reduced. Upsampling of the feature map aims to restore the original image size and number of channels. This results in an image with extracted features, identical in size to the original image. The Transformer module is then designed. The chosen module is the Encoder module, designed with 12 layers to ensure that each image integrates information from the other three images, restoring the vertically combined image to its original position and enhancing local information to obtain the sample feature map.
[0120] Step S003: The sample feature map is processed by the multi-scale weighted branch module to obtain the sample density map prediction value;
[0121] The original DarkNet53 outputs feature maps at three scales for object detection. This approach rigidly stipulates that a target can only fall at one scale. However, in reality, scale variations are diverse. Therefore, it's necessary to distribute target detection across different scales and integrate the results from multiple scales to adapt to various subtle scale changes. In this embodiment, the backbone network for crowd counting is DarkNet53. Therefore, to integrate the advantages of multiple scales, a multi-scale weight branch module is added to the original DarkNet53 network. This module generates weights for each scale heatmap, performs a weighted average of the heatmaps, and processes the sample feature maps through the multi-scale weight branch module to obtain the sample density map prediction value, which is then used to calculate the predicted value of the sample head count.
[0122] Step S004: Input the predicted value of the sample density map into the fully connected branch module for calculation to obtain the predicted value of the sample head number;
[0123] Furthermore, the sample feature maps are processed by a multi-scale weighted branch module to obtain the predicted sample density maps, which include each density map. These density maps are then input into the fully connected branch module. Since the last feature map of the backbone network has rich semantic information, the fully connected branch flattens the last feature map of the backbone network and connects it to fully connected layers with 1024, 512, 256, and 1 nodes respectively. The output of the fully connected layer with 1 node represents the number of people in the population.
[0124] Step S005: Based on the multi-objective loss function framework, train the crowd counting model according to the predicted values of the sample density map, the predicted values of the sample head count, and the corresponding real labels.
[0125] Furthermore, after obtaining the sample density map prediction value and the sample head number prediction value through the multi-scale weighted branch module and the fully connected branch module respectively, the crowd counting model can be trained based on the overall framework of the multi-objective loss function, combining the sample density map prediction value, the sample head number prediction value and their corresponding real labels.
[0126] This embodiment, through the above scheme, specifically acquires pre-collected sample data, including sample images and corresponding ground truth labels; enhances the sample images using an adaptive transformation module to obtain sample feature maps; processes the sample feature maps using a multi-scale weighted branch module to obtain sample density map predictions; inputs the sample density map predictions into a fully connected branch module for calculation to obtain sample head count predictions; based on a multi-objective loss function framework, trains the crowd counting model using the sample density map predictions, sample head count predictions, and corresponding ground truth labels; strengthens local image information by constructing an image adaptive Transformer enhancement module; adds weights and fully connected branches based on DarkNet53; integrates multi-scale information; and introduces a numerical prediction module; and improves the accuracy of the crowd counting model through multi-objective loss function reconstruction.
[0127] Reference Figure 5 , Figure 5 for Figure 3 A schematic flowchart illustrating step S003 in this embodiment. This embodiment is based on the above. Figure 3 In the embodiment shown, step S003 includes:
[0128] Step S0031: Upsample the first size feature map, the second size feature map, and the third size feature map respectively to obtain the first weight matrix, the second weight matrix, and the third weight matrix;
[0129] Step S0032: The first weight matrix, the second weight matrix, and the third weight matrix are concatenated to obtain a three-dimensional weight array, and the three-dimensional weight array is normalized to obtain a fourth weight matrix, a fifth weight matrix, and a sixth weight matrix.
[0130] Step S0033: Perform downsampling processing on the first size feature map, the second size feature map and the third size feature map respectively to obtain the fourth size feature map, the fifth size feature map and the sixth size feature map;
[0131] Step S0034: Perform dot product between the fourth weight matrix and the fourth size feature map, the fifth weight matrix and the fifth size feature map, and the sixth weight matrix and the sixth size feature map respectively to obtain the first density map, the second density map, and the third density map.
[0132] Reference Figure 6 , Figure 6 This is a schematic diagram of the model branching framework in an embodiment of the present invention, such as... Figure 6As shown, in this embodiment of the invention, three weight branch frameworks are designed. Feature map F1 in the backbone network is selected, with a size of W / 8*H / 8*256. After three upsampling operations, the feature map is restored to W / 4*H / 4*128, W / 2*H / 2*64, W*H*32, and W*H*1, resulting in weight matrix W1. Feature map F2 in the backbone network is selected, with a size of W / 16*H / 16*512. After four upsampling operations, the feature map is restored to W / 8*H / 8*256, W / 4*H / 4*128, W / 2*H / 2*64, W*H*32, and W*H*1, resulting in weight matrix W2. Select feature map F3 in the backbone network, with a size of W / 32*H / 32*1024. After 5 upsampling, the feature map is restored to W / 16*H / 16*512, W / 8*H / 8*256, W / 4*H / 4*128, W / 2*H / 2*64, W*H*32 and W*H*1, to obtain the weight matrix W3.
[0133] Concatenate W1, W2, and W3 to obtain a three-dimensional weight array. Perform a SoftMax operation on each channel to normalize the values of the weight array to the range of 0-1. Then, decompose the array according to each channel to obtain weight matrices W5, W6, and W7. W5, W6, and W7 are the normalized results of W1, W2, and W3, representing the element weights of F5, F6, and F7, respectively.
[0134] Multiplying W5 and F5 yields a density map with a 32x downsampling representation; multiplying W6 and F6 yields a density map with a 16x downsampling representation; and multiplying W7 and F7 yields a density map with an 8x downsampling representation. These are the predicted sample density maps output by the multi-scale weighted branch module.
[0135] This embodiment, through the above scheme, specifically obtains a first weight matrix, a second weight matrix, and a third weight matrix by upsampling the first, second, and third size feature maps respectively; concatenating the first, second, and third weight matrices to obtain a three-dimensional weight array, and normalizing the three-dimensional weight array to obtain a fourth, fifth, and sixth weight matrix; downsampling the first, second, and third size feature maps respectively to obtain a fourth, fifth, and sixth size feature map; and performing dot products between the fourth weight matrix and the fourth, fifth, and sixth size feature maps respectively to obtain the first, second, and third density maps. A multi-scale weight branch module is added to the original DarkNet53 network. This module generates weights for each scale heatmap and performs a weighted average of the heatmaps, integrating the advantages of multiple scales and improving the accuracy of the crowd counting model prediction.
[0136] Reference Figure 7 , Figure 7 for Figure 3 A schematic diagram of the specific process of step S005 in this embodiment. This embodiment is based on any of the above embodiments. In this embodiment, step S005 includes:
[0137] Step S0051: Calculate the density map loss function based on the predicted sample density map value and the density map label;
[0138] Step S0052: Calculate the population loss function based on the predicted value of the sample head count and the population number label;
[0139] Step S0053: Substitute the density map loss function and the crowd number loss function into the multi-objective loss function framework to calculate the multi-objective loss function;
[0140] Step S0054: The multi-objective loss function is fed back to the crowd counting model to update the network parameters of the crowd counting model. Then, the process of processing the sample feature map through the multi-scale weight branch module to obtain the sample density map prediction value and subsequent steps are returned. This process is repeated to iterate the parameters until the crowd counting model converges, at which point the training is terminated, and the trained crowd counting model is obtained.
[0141] Optionally, the step of calculating the density map loss function based on the sample density map prediction value and the density map label includes:
[0142] Based on the first density map, the second density map, the third density map and the corresponding density map labels, calculate the loss weight and density map error ratio for each density map.
[0143] The density map loss function is calculated based on the loss weights of each density map and the density map error ratio term.
[0144] Optionally, the step of calculating the crowd size loss function based on the predicted sample head count and the crowd size label includes:
[0145] The mean square error and the ratio of population number error are calculated based on the predicted value of the sample head count and the population number label.
[0146] The population loss function is calculated based on the mean square error and the ratio of population number error.
[0147] For one of the samples i, the sample data is available (x i y i1 y i2 ) represents the sample number, where i represents the sample number and x represents the sample number. i Let y represent the i-th sample image. i1 y represents the number of people in the i-th sample image. i1 This represents the density map of the i-th sample image. Section 2.2 shows the network outputting three density maps and a population count. The three predicted density maps can be used... as well as This indicates that the predicted population size is available. express.
[0148] Since the accuracy of existing numerical prediction and density map-based prediction methods is not very high, we consider integrating the results from numerical prediction and density map-based methods to improve the prediction accuracy of the model. The specific steps are as follows.
[0149] Step 1: Design the overall framework of the multi-objective loss function, as shown in Equation (1):
[0150] Loss whole =α1*Loss1+α2*Loss2 (1)
[0151] Where Loss1 represents the density map loss and Loss2 represents the population size loss. α1 represents the weight of the density map loss and α2 represents the weight of the population size loss, with α1 + α2 = 1. This paper uses cross-validation to calculate α1 = 0.7 and α2 = 0.3.
[0152] Step 2 calculates the loss for different density maps. The density map loss consists of four terms. and y i2In crowd counting scenarios, when the crowd scale is large, the density map predicted by 32x downsampling contributes significantly to the loss function; when the crowd scale is medium, the density map predicted by 16x downsampling contributes significantly to the loss function; and when the crowd scale is small, the density map predicted by 8x downsampling contributes significantly to the loss function. To incorporate this information into the loss function, the loss weights for density maps at different scales are calculated, as shown below.
[0153]
[0154]
[0155]
[0156]
[0157] Here, β1, β2, and β3 are the weights of the density map loss at different scales. The larger the mean square loss of the density map, the less accurate the density map prediction at that scale, so its weight should be smaller. The numerators of β1, β2, and β3 are the exponents of the negative of their mean square losses; as the loss increases, the numerators decrease. The denominator is the sum of the three numerators, so the weights decrease as the loss increases.
[0158] Step 3 calculates the crowd size loss. The crowd size loss includes two items: and y i1 The mean square error is used, as shown in formula (6).
[0159]
[0160] Step 4 adds an error ratio term to the two types of losses. The optimization of the network depends on whether the loss function is set reasonably. At present, the population counting uses mean squared error as the loss function. Although it is a feasible solution from the perspective of numerical optimization, this solution also has some drawbacks. First, the mean squared error is a squared term, which will expand or decrease the absolute value of the error; second, the mean squared error only considers the magnitude of the value and does not take into account the relative magnitude of the loss. Based on these two points, this paper adds an error ratio term as a weight to optimize the loss function. The modified Loss1 is shown in formula (7):
[0161]
[0162] The modified Loss2 is shown in formula (8).
[0163]
[0164] For each density map, calculating a ratio function as the weight of the loss can optimize the model parameters from both absolute value and ratio perspectives, thereby further improving the model's performance.
[0165] Furthermore, embodiments of the present invention also propose a crowd counting device, the crowd counting device comprising:
[0166] The acquisition module is used to acquire the target image;
[0167] The prediction module is used to predict the target image using a pre-trained crowd counting model to obtain the density map prediction value and the head count prediction value.
[0168] The weighting module is used to perform weighted calculations on the density map prediction value and the head count prediction value to obtain the population count value.
[0169] Reference Figure 8 , Figure 8 This is a schematic diagram of the overall process in an embodiment of the present invention, such as... Figure 8 As shown, in this embodiment of the invention, a crowd detection model is constructed, and an image adaptive Transformer enhancement module is built to enhance local image information; based on DarkNet53, weights and fully connected branches are added, multi-scale information is fused, and a numerical prediction module is introduced; the multi-objective loss function is reconstructed to improve the model's prediction accuracy.
[0170] After the introduction of the Transformer enhancement module, local images incorporate information from the surrounding parts, enhancing their semantic representation and thus improving the model's accuracy, reducing the model error by 0.85%.
[0171] The original DarkNet53's multi-scale approach is a choice between multiple scales, fixing the prediction results to a single size. This cannot guarantee the model's accuracy at new scales. Therefore, by adding weight branches, the results at multiple scales are weighted, thereby improving the scale flexibility and the model's accuracy. The model's error is reduced by 2.3%.
[0172] The mainstream prediction method is to predict the density map. In this embodiment of the invention, a fully connected branch is added to generate a population numerical prediction as an auxiliary result, which is integrated with the density map result, thereby reducing the model error by 0.32%.
[0173] The single-objective loss function yielded poor prediction results, therefore a multi-objective loss function was reconstructed. First, a framework for the multi-objective loss function was established; second, loss weights for density maps at different scales were calculated; finally, an error ratio term was added to enrich the information of the loss function. The construction of the multi-objective loss function reduced the model's error rate by 4.4%.
[0174] In this embodiment, the effectiveness of features is improved by introducing a Transformer module to enhance local image information; the flexibility of scale is improved by adding weight branches to the backbone network to weight the results at multiple scales, ensuring better prediction results for images at different scales; crowd values are calculated by adding fully connected branches to the backbone network and used as auxiliary predictions to improve prediction capabilities; and the model's prediction ability is enhanced by developing a multi-objective loss framework, assigning different weights to density maps at different scales, and adding an error ratio term.
[0175] The principle and implementation process of crowd counting in this embodiment are explained in the above embodiments and will not be repeated here.
[0176] Furthermore, this embodiment of the invention also proposes a terminal device, which includes a memory, a processor, and a crowd counting program stored in the memory and executable on the processor. When the crowd counting program is executed by the processor, it implements the steps of the crowd counting method described above.
[0177] Since the population counting program uses all the technical solutions of all the aforementioned embodiments when it is executed by the processor, it has at least all the beneficial effects brought about by all the technical solutions of all the aforementioned embodiments, which will not be repeated here.
[0178] Furthermore, embodiments of the present invention also propose a computer-readable storage medium storing a crowd counting program, which, when executed by a processor, implements the steps of the crowd counting method described above.
[0179] Since the population counting program uses all the technical solutions of all the aforementioned embodiments when it is executed by the processor, it has at least all the beneficial effects brought about by all the technical solutions of all the aforementioned embodiments, which will not be repeated here.
[0180] Compared with existing technologies, the crowd counting method, apparatus, terminal device, and storage medium proposed in this invention acquire a target image; predict the target image using a pre-trained crowd counting model to obtain a density map prediction value and a head count prediction value; and perform a weighted calculation on the density map prediction value and the head count prediction value to obtain a crowd count value. By using the head count prediction value as an auxiliary result and weighting it with the density map prediction value, the accuracy of the obtained crowd count value is improved, thereby increasing the accuracy of crowd counting.
[0181] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0182] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0183] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, controlled terminal, or network device, etc.) to execute the methods of each embodiment of this application.
[0184] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.
Claims
1. A method for counting people, characterized in that, The crowd counting method includes the following steps: Obtain the target image; Acquire pre-collected sample data, wherein the sample data includes sample images and corresponding real labels; The sample image is enhanced by an adaptive conversion module to obtain a sample feature map; The sample feature map is processed by a multi-scale weighted branch module to obtain the sample density map prediction value; The predicted value of the sample density map is input into the fully connected branch module for calculation to obtain the predicted value of the sample head count. Based on a multi-objective loss function framework, a crowd counting model is trained using the predicted values of the sample density map, the predicted values of the sample head count, and the corresponding ground truth labels. The ground truth labels include density map labels and crowd count labels. Specifically, this step includes: calculating a density map loss function based on the predicted values of the sample density map and the density map labels; calculating a crowd count loss function based on the predicted values of the sample head count and the crowd count labels; inputting the density map loss function and the crowd count loss function into the multi-objective loss function framework to calculate a multi-objective loss function; feeding the multi-objective loss function back to the crowd counting model to update the network parameters of the crowd counting model, and then returning to execute the step of processing the sample feature map through the multi-scale weight branch module to obtain the predicted values of the sample density map, and subsequent steps. This process is repeated to iterate the parameters until the crowd counting model converges, at which point training is terminated, resulting in the trained crowd counting model. The target image is predicted by a pre-trained crowd counting model to obtain density map prediction values and head count prediction values. The crowd counting model includes a multi-scale weighted branch module and a fully connected branch module. The population count is obtained by weighting the density map prediction value and the head count prediction value.
2. The crowd counting method as described in claim 1, characterized in that, The step of enhancing the sample image through the adaptive transformation module to obtain the sample feature map includes: The sample image is segmented and vertically combined to obtain a vertically combined image; The vertically combined image is processed through a backbone network to obtain a scaled feature map; The transformed scale feature map is upsampled to obtain the original scale feature map; The original scale feature map is integrated and its position is restored by the adaptive transformation module to obtain the sample feature map.
3. The crowd counting method as described in claim 1, characterized in that, The sample feature map includes a first-size feature map, a second-size feature map, and a third-size feature map; the sample density map prediction value includes a first density map, a second density map, and a third density map; the step of processing the sample feature map through a multi-scale weighted branch module to obtain the sample density map prediction value includes: Upsampling is performed on the first size feature map, the second size feature map, and the third size feature map respectively to obtain the first weight matrix, the second weight matrix, and the third weight matrix; The first weight matrix, the second weight matrix, and the third weight matrix are concatenated to obtain a three-dimensional weight array, and the three-dimensional weight array is normalized to obtain a fourth weight matrix, a fifth weight matrix, and a sixth weight matrix. The first, second, and third size feature maps are downsampled respectively to obtain the fourth, fifth, and sixth size feature maps; The first density map, the second density map, and the third density map are obtained by performing dot products on the fourth weight matrix with the fourth size feature map, the fifth weight matrix with the fifth size feature map, and the sixth weight matrix with the sixth size feature map, respectively.
4. The crowd counting method as described in claim 3, characterized in that, The step of calculating the density map loss function based on the sample density map prediction value and density map label includes: Based on the first density map, the second density map, the third density map and the corresponding density map labels, calculate the loss weight and density map error ratio for each density map. The density map loss function is calculated based on the loss weights of each density map and the density map error ratio term.
5. The crowd counting method as described in claim 1, characterized in that, The step of calculating the crowd size loss function based on the predicted sample head count and the crowd size label includes: The mean square error and the ratio of population number error are calculated based on the predicted value of the sample head count and the population number label. The population loss function is calculated based on the mean square error and the ratio of population number error.
6. A crowd counting device, characterized in that, The crowd counting device includes: The acquisition module is used to acquire the target image; The prediction module is used to predict the target image using a pre-trained crowd counting model, obtaining density map predictions and head count predictions. The crowd counting model includes a multi-scale weighted branch module and a fully connected branch module. The prediction module also acquires pre-collected sample data, including sample images and corresponding ground truth labels. An adaptive transformation module enhances the sample images to obtain sample feature maps. The multi-scale weighted branch module processes the sample feature maps to obtain sample density map predictions. The sample density map predictions are input into the fully connected branch module for calculation to obtain sample head count predictions. Based on a multi-objective loss function framework, the prediction module trains the prediction model using the sample density map predictions, sample head count predictions, and corresponding ground truth labels. The crowd counting model includes density map labels and crowd quantity labels. The prediction module is further used to calculate a density map loss function based on the predicted density map value and density map label; calculate a crowd quantity loss function based on the predicted head count value and crowd quantity label; input the density map loss function and crowd quantity loss function into the multi-objective loss function framework to calculate the multi-objective loss function; feed the multi-objective loss function back to the crowd counting model to update the network parameters of the crowd counting model, and return to execute the step of processing the sample feature map through the multi-scale weight branch module to obtain the predicted density map value and subsequent steps. This process is repeated to iterate the parameters until the crowd counting model converges, at which point training is terminated, and the trained crowd counting model is obtained. The weighting module is used to perform weighted calculations on the density map prediction value and the head count prediction value to obtain the population count value.
7. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a crowd counting program stored in the memory and executable on the processor. When the crowd counting program is executed by the processor, it implements the steps of the crowd counting method as described in any one of claims 1-5.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a crowd counting program, which, when executed by a processor, implements the steps of the crowd counting method as described in any one of claims 1-5.