A neural network-based crowd counting method and device
By cropping and processing overlapping regions of sample images, and combining primary and auxiliary tasks to train a neural network model, the problem of poor training performance of crowd counting models in existing technologies is solved, achieving higher prediction accuracy and anti-translation capability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 709TH RESEARCH INSTITUTE CHINA STATE SHIPBUILDING CORP LTD
- Filing Date
- 2023-03-27
- Publication Date
- 2026-05-12
AI Technical Summary
Existing crowd counting network models ignore the implicit logical relationship between the training set and the computational task during training, resulting in poor training performance and a large discrepancy between the predicted number of people and the actual number of people.
By cropping sample images into first and second sub-images with overlapping regions, and training the neural network model in conjunction with the main and auxiliary tasks, the implicit logical relationship between the training set and the counting task is mined. The network parameters are optimized using the stochastic gradient descent algorithm, and a convolutional neural network model based on the VGG-16 skeleton is constructed.
It improves the accuracy of neural network models in predicting population size, enhances the model's resistance to translational changes, is applicable to both labeled and unlabeled datasets, and improves counting performance.
Smart Images
Figure CN116363092B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of deep learning data analysis technology, and in particular to a method and apparatus for counting people based on neural networks. Background Technology
[0002] Crowd counting is a technique for estimating the number and density distribution of crowds in an image or video scene. Mainstream methods employ deep convolutional neural networks, using information such as pedestrian locations and numbers from the training set to learn the mapping relationship from the image to a density map via regression. The total number of people is then obtained by integrating the density map. Existing methods primarily utilize the specific number of people, pedestrian locations, and numbers in the image, directly inputting this information into the network model. The training of the network model is supervised by the number of people using a ground truth density map. After training, the network model is used to predict the number and density of people in new scenes.
[0003] Current methods do not explore the implicit logical relationships between the training set and the computational task, resulting in poor training performance. Consequently, the number of people predicted by the trained network model differs significantly from the actual number, making it difficult to meet the needs of practical applications.
[0004] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention
[0005] The technical problem to be solved by the embodiments of the present invention is how to solve the problem that the training effect of the network model is poor because the implicit logical relationship between the training set and the computation task is ignored during the training process of the existing crowd counting network model.
[0006] The embodiments of the present invention adopt the following technical solutions:
[0007] In a first aspect, the present invention provides a crowd counting method based on a neural network, comprising:
[0008] Acquire multiple sample images;
[0009] The sample image is input into the original neural network model to be trained to obtain the first loss function of the original neural network model;
[0010] The sample image is cropped into a first sub-image and a second sub-image with overlapping regions, and the first sub-image and the second sub-image are respectively input into the original neural network model to obtain the second loss function of the original neural network model;
[0011] The original neural network model is iteratively trained based on the first loss function and the second loss function to minimize the loss of the original neural network model, thereby obtaining the target neural network model, and then crowd counting is performed based on the target neural network model.
[0012] Preferably, the step of inputting the sample image into the original neural network model to be trained to obtain the first loss function of the original neural network model specifically includes:
[0013] The head position information is obtained based on the annotation information of the sample image, and the ground truth density map of the sample image is obtained based on the head position information.
[0014] The original neural network model is used to analyze the sample image to obtain the predicted density information of the sample image;
[0015] The first loss function is obtained based on the truth density map and the predicted density information.
[0016] Preferably, the step of cropping the sample image into a first sub-image and a second sub-image with overlapping regions, and inputting the first sub-image and the second sub-image into the original neural network model respectively to obtain the second loss function of the original neural network model, specifically includes:
[0017] The sample image is cropped into a first sub-image and a second sub-image with overlapping regions;
[0018] The original neural network model is used to analyze the first subgraph and the second subgraph respectively to obtain the common prediction density and population estimate of the overlapping area in different subgraphs. The second loss function is obtained based on the common prediction density and population estimate.
[0019] Preferably, the step of using the original neural network model to analyze the first subgraph and the second subgraph respectively to obtain the common prediction density and population estimate of the overlapping region in different subgraphs, and obtaining the second loss function based on the common prediction density and population estimate, specifically includes:
[0020] The original neural network model is used to analyze the first subgraph and the second subgraph respectively to obtain the first number estimate of the overlapping area in the first subgraph and the second number estimate of the overlapping area in the second subgraph;
[0021] The loss function L is obtained based on the first and second number estimates. c ;
[0022] The original neural network model is used to analyze the first subgraph and the second subgraph respectively to obtain the first common prediction density of the overlapping region in the first subgraph and the second common prediction density of the overlapping region in the second subgraph.
[0023] The loss function L is obtained based on the first common prediction density and the second common prediction density. d1 ;
[0024] For the loss function L c and loss function L d1 The second loss function is obtained by summing the results.
[0025] Preferably, the step of iteratively training the original neural network model according to the first loss function and the second loss function to minimize the loss of the original neural network model and obtain the target neural network model, and then performing crowd counting based on the target neural network model, includes:
[0026] The target loss function is obtained by summing the first loss function and the second loss function;
[0027] The parameters of the original neural network model are optimized using the stochastic gradient descent algorithm.
[0028] Preferably, before inputting the sample image, the first sub-image, and the second sub-image into the original neural network, the method further includes: adjusting the sample image, the first sub-image, and the second sub-image to the same size.
[0029] Preferably, the neural network-based crowd counting method further includes:
[0030] The target neural network model is used to predict the number of people in multiple sample images, and the predicted number of people for each sample image is obtained.
[0031] The mean absolute error and mean square error are obtained by comparing the predicted number of people with the actual number of people in all sample images.
[0032] If the mean absolute error and the mean square error are not within the set range, the neural network model will continue to be trained to optimize the neural network model.
[0033] Preferably, cropping the sample image into a first sub-image and a second sub-image with overlapping regions specifically includes:
[0034] The sample image is cropped using image blocks with side length K to obtain the first sub-image and the second sub-image;
[0035] Wherein, the height of the overlapping region corresponding to the first sub-image and the second sub-image is H and the width is W, and the range of the height H and width W of the overlapping region is:
[0036]
[0037]
[0038] Where K is a random number between 224 and 336.
[0039] Preferably, the original neural network model is constructed based on a convolutional neural network model with VGG-16 as the backbone, and the original neural network model specifically includes:
[0040] The first part is the network front end, which includes ten convolutional layers and four pooling layers for feature extraction from the input image. The network front end removes all the subsequent fully connected layers to make it suitable for output at any resolution.
[0041] The second part is the final segment of the network, which includes three convolutional kernels used to output the predicted density map.
[0042] Secondly, based on the neural network-based crowd counting method of the first aspect, the present invention also provides a neural network-based crowd counting device, comprising:
[0043] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the neural network-based crowd counting method described in the first aspect.
[0044] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the neural network-based crowd counting method described in the first aspect.
[0045] In the training process of the original neural network model, this invention constructs an auxiliary task based on the correspondence between the number of people and density maps within the image, focusing on the consistency of the number of people and density in the overlapping area of the first and second sub-images. The sample image is cropped into a first and second sub-image with overlapping areas. The main task is combined with the auxiliary task to train the network model. That is, the sample image and sub-image are input into the original neural network model for iterative training. The implicit relationship between the training set and the counting task is mined to obtain the target neural network model, thereby improving the accuracy of the network model in predicting the number of people. Attached Figure Description
[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0047] Figure 1 This is a flowchart of a crowd counting method based on a neural network provided in an embodiment of the present invention;
[0048] Figure 2 This is a flowchart detailing step 202 in a neural network-based crowd counting method provided in an embodiment of the present invention.
[0049] Figure 3 This is a schematic diagram of the neural network architecture of a crowd counting method based on a neural network provided in an embodiment of the present invention;
[0050] Figure 4 This is a flowchart of a method for cropping a sample image into sub-images with overlapping regions in a crowd counting method based on a neural network provided in an embodiment of the present invention.
[0051] Figure 5 This is a schematic diagram of density regression of the first and second subgraphs of a crowd counting method based on a neural network provided in an embodiment of the present invention;
[0052] Figure 6 This is a schematic diagram of cropping a sample image into a first sub-image and a second sub-image in a crowd counting method based on a neural network provided in an embodiment of the present invention.
[0053] Figure 7 This is a flowchart detailing step 402 in a neural network-based crowd counting method provided in an embodiment of the present invention.
[0054] Figure 8 This is a flowchart of a method for optimizing a neural network model based on a neural network crowd counting method provided in an embodiment of the present invention;
[0055] Figure 9 This is a flowchart of the auxiliary task loss function calculation for a crowd counting method based on a neural network provided in an embodiment of the present invention;
[0056] Figure 10 This is a schematic diagram of a crowd counting device based on a neural network provided in an embodiment of the present invention. Detailed Implementation
[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0058] In the description of this invention, the terms "inner", "outer", "longitudinal", "lateral", "upper", "lower", "top", "bottom", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and do not require that this invention must be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.
[0059] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0060] Example 1:
[0061] Embodiment 1 of the present invention provides a crowd counting method based on a neural network, such as... Figure 1 As shown, it includes:
[0062] Step 201: Acquire multiple sample images and input the sample images into the original neural network model to be trained to obtain the first loss function of the original neural network model.
[0063] In this embodiment of the invention, the sample images are derived from a dataset. For a given dataset, the dataset contains labeled signals, typically the location information of a person's head in the target scene (i.e., the coordinate information of the head's position). Multiple sample images containing the head's location information are acquired from the dataset, and then these sample images are input into the original neural network model for training, resulting in a first loss function after training.
[0064] Step 202: Crop the sample image into a first sub-image and a second sub-image with overlapping regions, and input the first sub-image and the second sub-image into the original neural network model respectively to obtain the second loss function of the original neural network model.
[0065] In this embodiment of the invention, an auxiliary task is set up based on the principle of translation invariance of the number of people and density maps within an image (i.e., constructing an auxiliary task based on the correspondence between the number of people and density maps within the image, to supervise the consistency of the number of people and density in the overlapping area of the first and second sub-images). The sample image is cropped into a first sub-image and a second sub-image with overlapping areas, and the first and second sub-images are respectively input into the original neural network model for training, thereby obtaining the second loss function. This embodiment of the invention creates an auxiliary task based on the principle of translation invariance of the number of people and density maps within an image (i.e., mining the implicit logical relationship between the first and second sub-images through the overlapping area) to train the original neural network model, thereby improving the accuracy of the network model in predicting the number of people. It is worth noting that before inputting the sample image, the first sub-image, and the second sub-image into the original neural network, this embodiment of the invention further includes: adjusting the sample image, the first sub-image, and the second sub-image to the same size.
[0066] Step 203: Iteratively train the original neural network model according to the first loss function and the second loss function to minimize the loss of the original neural network model and obtain the target neural network model, thereby performing crowd counting based on the target neural network model.
[0067] In this embodiment of the invention, after obtaining the first loss function and the second loss function, the loss function of the entire network model is constructed. That is, the loss function of the entire network is equal to the sum of the first loss function and the second loss function. The training iteration process of the original neural network model in this embodiment of the invention actually involves continuously inputting multiple sample images and corresponding sub-images into the original neural network model, adjusting the parameters within the original neural network model according to the loss function of the entire network, to obtain a target neural network model that meets the requirements, and using the trained target neural network model to count the number of people in a specific scene. The following example illustrates the iterative training process of the original neural network model: Assume three sample images, A, B, and C, are selected and input into the original neural network model in the order of A, B, and C. To train the original neural network, first, sample image A is input into the original neural network model to obtain the first loss function corresponding to sample image A. Then, sample image A is cropped into a first sub-image and a second sub-image corresponding to sample image A, and these first and second sub-images are input into the original neural network model for training. The parameters within the original neural network model are adjusted using the overall network loss function corresponding to sample image A, resulting in a first intermediate neural network model. Next, sample image B is input into the first intermediate neural network model to obtain the sample... The first loss function corresponding to image B is used. Then, the sample image B is cropped into a first sub-image and a second sub-image corresponding to the sample image B. The first and second sub-images corresponding to the sample image B are input into the first intermediate neural network model for training. The parameters in the first intermediate neural network model are adjusted by the loss function of the entire network corresponding to the sample image B to obtain the second intermediate neural network model. Then, the sample image C and the two sub-images corresponding to the sample image C are input into the second intermediate neural network model in the same way. The parameters in the second intermediate neural network model are adjusted to obtain the third intermediate neural network model. The training is carried out in a continuous iterative manner until all sample images are trained and the target neural network model corresponding to the minimum loss is obtained.
[0068] To obtain a high-performance counting model, it is often necessary to learn more useful information from the training set. During network training, most methods utilize ground truth density maps or people-counting supervision. However, if more effective latent information from the dataset, such as the relationships between people in images, can be utilized, the network training effect will be even better. For two images with overlapping regions, the number of people in the overlapping area is equal. Therefore, a high-performance counting network, given two images with a certain degree of overlap, should predict the same number of people in the overlapping region. This indicates that the network can output the same result regardless of the location of the overlapping region patch in the image, demonstrating good resistance to translation changes, i.e., good translation invariance. Based on this thinking, the embodiments of the invention set up an auxiliary task based on the consistency of people in the overlapping region of sub-images. Given an image, two sub-images of the same size with a certain degree of overlap can be randomly cropped. These two sub-images are then input into the network to obtain their respective density maps. The density maps corresponding to the overlapping regions of the sub-images can be used to perform some self-supervised auxiliary tasks, thereby enabling the embodiments of the invention to uncover the latent relationships between the training set and the counting task.
[0069] In the training process of the original neural network model, this invention constructs an auxiliary task based on the correspondence between the number of people and density maps within the image, focusing on the consistency of the number of people and density in the overlapping area of the first and second sub-images. The sample image is cropped into a first and second sub-image with overlapping areas. The main task is combined with the auxiliary task to train the network model. That is, the sample image and sub-image are input into the original neural network model for iterative training. The implicit relationship between the training set and the counting task is mined to obtain the target neural network model, thereby improving the accuracy of the network model in predicting the number of people.
[0070] To illustrate the complete solution of the embodiments of the present invention in detail, the details of the embodiments of the present invention will be described in detail below. The sample image is input into the original neural network model to be trained to obtain the first loss function of the original neural network model, such as... Figure 2 As shown, it specifically includes:
[0071] Step 301: Obtain head position information based on the annotation information of the sample image, and obtain the ground truth density map of the sample image based on the head position information.
[0072] The sample image in this embodiment of the invention is an image containing head position information. A ground truth density map corresponding to the sample image can be generated based on the head position information within the sample image. In this embodiment of the invention, the generation rule for the ground truth density map of the sample image is as follows:
[0073]
[0074] Where N is the total number of people in the sample images, δ(xx) i ) represents the delta function. This represents the Gaussian kernel function normalized to 1, and the Gaussian kernel variance σ. i The value of x is usually set based on the distribution of heads in the images of the dataset. i Let be the pixel coordinates of the i-th person's head.
[0075] Step 302: Analyze the sample image using the original neural network model to obtain the predicted density information of the sample image.
[0076] like Figure 3 The diagram illustrates the original neural network model structure of an embodiment of the present invention. This original neural network model is constructed based on a convolutional neural network model with VGG-16 as its backbone. Specifically, the original neural network model includes: a first part, a network front-end comprising ten convolutional layers and four pooling layers, used for feature extraction from the input image. The network front-end removes all subsequent fully connected layers to accommodate output at any resolution; and a second part, a network tail, comprising three convolutional kernels used to output a predicted density map. Sample images are input into the original neural network model to obtain the corresponding predicted density map, thereby predicting the number of people. It is noteworthy that after the sample image is input into the original neural network model, the size of the obtained predicted density map is 1 / 16 of the input image.
[0077] Step 303: Obtain the first loss function based on the truth density map and the predicted density information.
[0078] In this embodiment of the invention, when the sample image is input into the original neural network model for training, the mean square error (MSE) between the predicted density map and the ground truth density map based on the sample image is used for supervision. Therefore, the loss function for the sample image is:
[0079]
[0080] Where N is the total number of people in the sample images, X i Let Θ be the input sample image, and F(X) represent the set of learnable parameters. i ;Θ) refers to the model's response to image X i The generated predicted density map, and F i That is image X i The corresponding truth density plot.
[0081] The overall network loss function in this embodiment of the invention comprises two parts: a first loss function for the sample image and a second loss function corresponding to the two sub-images cropped from the sample. The second loss function corresponding to the two sub-images is the loss function for the auxiliary task in this embodiment of the invention. The sample image is cropped into a first sub-image and a second sub-image with overlapping regions, and the first and second sub-images are respectively input into the original neural network model to obtain the second loss function of the original neural network model, as follows: Figure 4 and Figure 5 As shown, it specifically includes:
[0082] Step 401: Crop the sample image into a first sub-image and a second sub-image with overlapping regions.
[0083] like Figure 6 As shown, in this embodiment of the invention, multi-scale sampling of the original images in the training set is performed to enhance the incoming data. When the sample image dataset of this embodiment is used to assist in task training, cropping the sample images into a first sub-image and a second sub-image with overlapping regions specifically includes: cropping the sample images using image blocks of side length K to obtain the first sub-image and the second sub-image; wherein the height of the overlapping region corresponding to the first sub-image and the second sub-image is H and the width is W. For this embodiment of the invention, the area of the overlapping region should not be too large or too small (if the overlapping region is too large, there will be fewer areas where the first sub-image and the second sub-image are different, failing to achieve the purpose of information mining; if the overlapping region is too small, the relevant information between the first sub-image and the second sub-image cannot be fully mined). Based on this, the range of the height H and the width W of the overlapping region is: and Where K is a random number between 224 and 336.
[0084] Step 402: The original neural network model is used to analyze the first subgraph and the second subgraph respectively to obtain the common prediction density and population estimate of the overlapping area in different subgraphs, and the second loss function is obtained based on the common prediction density and population estimate.
[0085] In this embodiment of the invention, the second loss function corresponding to the auxiliary task also includes two parts: the first part is the loss function corresponding to the predicted number of people in the subgraph, and the second part is the loss function corresponding to the predicted density map of the subgraph. The process of obtaining the second loss function in step 402 is as follows: Figure 7 As shown, it specifically includes:
[0086] Step 4021: Analyze the first subgraph and the second subgraph using the original neural network model to obtain a first estimate of the number of people in the overlapping area of the first subgraph and a second estimate of the number of people in the overlapping area of the second subgraph. Then, obtain the loss function L based on the first and second estimates of the number of people. c .
[0087] The loss function L corresponding to the predicted number of people in the subgraph of this embodiment of the invention. c for:
[0088]
[0089] Where Θ represents the set of learnable parameters of the network, N is the number of input images, and X... i1 and X i2 The input image X i Two overlapping subgraphs, C1(X) i1 ;Θ) and C2(X i2 ;Θ) represents sub Figure X i1 X i2 The estimated number of people.
[0090] For the neural network model of this invention, regardless of whether it is the first sub-image, the second sub-image, or the sample image, as long as it is input into the neural network model, the predicted density map of the image (sample image, first sub-image, and / or second sub-image) can be obtained. Through the predicted density map, the predicted density and predicted number of people in each region of the image can be determined. For the first and second sub-images, after obtaining the predicted density maps corresponding to the first and second sub-images respectively, since the cropping process reveals the cropped area, the predicted density and predicted number of people in the overlapping region corresponding to the first sub-image can be determined through the predicted density map of the first sub-image, and the predicted density and predicted number of people in the overlapping region corresponding to the second sub-image can be determined through the predicted density map of the second sub-image.
[0091] Step 4022: Analyze the first subgraph and the second subgraph using the original neural network model to obtain the first common prediction density of the overlapping region in the first subgraph and the second common prediction density of the overlapping region in the second subgraph, and obtain the loss function L based on the first common prediction density and the second common prediction density. d1 .
[0092] The loss function L corresponding to the predicted number of people in the subgraph of this embodiment of the invention. d1 for:
[0093]
[0094] Where Θ represents the set of learnable parameters of the network, N is the number of input images, and X...i1 and X i2 The input image X i Two overlapping subgraphs, D1(X) i1 ;Θ) and D2(X i2 ;Θ) respectively represent sub Figure X i1 X i2 The predicted density map.
[0095] Step 4023: Apply the loss function L c and loss function L d1 The second loss function is obtained by summing the results. Therefore, the second loss function corresponding to the subgraph (i.e., the loss function of the auxiliary task) is: L = L c + d1 .
[0096] This invention employs a main task (inputting sample images into the original neural network model) combined with an auxiliary task (inputting first and second sub-images with overlapping regions into the original neural network model) for joint training. The iterative training of the original neural network model based on the first and second loss functions minimizes its loss, resulting in a target neural network model. Crowd counting based on this target neural network model includes: summing the first and second loss functions to obtain a target loss function; and using a stochastic gradient descent algorithm to obtain the minimum value of the target loss function to determine the optimal weights and biases of the neural network, thereby obtaining the target neural network model. The loss function for the entire network is set as: = L d + f As an optional implementation, the initial learning rate for both the primary and secondary counting tasks is set to 10. -6 It decreases by a scaling factor of 0.1, decreasing once every 200 iterations, with a total of 400 iterations for network training.
[0097] In addition, after obtaining the target neural network model of this invention embodiment, it is also necessary to monitor the performance of the target neural network model to measure whether the target neural network model meets the requirements. This invention embodiment uses the mean absolute error (MAE) and mean squared error (MSE) based on the number of people to measure network performance. The neural network-based crowd counting method, such as... Figure 8 and Figure 9 As shown, it also includes:
[0098] Step 501: Use the target neural network model to predict the number of people in multiple sample images, and obtain the predicted number of people for each sample image. Based on the predicted number of people for all sample images and the actual number of people, obtain the mean absolute error and mean square error.
[0099] After obtaining the target neural network model, this invention inputs the sample image into the target neural network model for prediction, obtaining the predicted number of people corresponding to the sample image. Combined with the actual number of people in the sample image, the obtained target neural network is evaluated to determine whether it meets the actual requirements. The specific formulas for the Mean Absolute Error (MAE) and Mean Squared Error (MSE) implemented in this invention are as follows:
[0100]
[0101]
[0102] Where N represents the number of test images, Then z is the predicted number of people in the i-th image. i This represents the actual number of people in the i-th image.
[0103] Step 502: If the mean absolute error and the mean square error are not within the set range, continue training the neural network model to optimize the neural network model.
[0104] In this embodiment of the invention, when evaluating the performance of the target neural network, it is necessary to set comparison thresholds for the mean absolute error (MAE) and mean squared error (MSE) according to actual needs. If the mean absolute error and the mean squared error are not within the set range, the neural network model continues to be trained until the mean absolute error and the mean squared error measured by the sample images fall within the set range.
[0105] In the training process of the original neural network model constructed in this invention, an auxiliary task is constructed based on the correspondence between the number of people and density maps within the image. This task aims to supervise the consistency of the number of people and density in the overlapping areas of the first and second sub-images. The sample image is cropped into first and second sub-images with overlapping areas. The main task is combined with the auxiliary task to train the network model. Specifically, the sample image and sub-images are input into the original neural network model for iterative training. This process uncovers the implicit relationships between the training set and the counting task, resulting in the target neural network model and improving the accuracy of the network model in predicting the number of people. Furthermore, embodiments of this invention can also use only the auxiliary task to optimize the original neural network model to a certain extent. In these embodiments, the auxiliary task does not use labeled data during training (it does not use the coordinates of heads within the sub-images). This allows the crowd counting method of this invention to be applicable not only to labeled data but also to unlabeled data, thereby improving the counting performance of the network model to a certain extent.
[0106] Example 2:
[0107] like Figure 10 The diagram shown is an architectural schematic of a crowd counting device based on a neural network according to an embodiment of the present invention. This embodiment of the crowd counting device based on a neural network includes one or more processors 21 and a memory 22. Figure 10 Take a processor 21 as an example.
[0108] Processor 21 and memory 22 can be connected via a bus or other means. Figure 10 Taking the example of a connection between China and Israel via a bus.
[0109] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the neural network-based crowd counting method in Embodiment 1. The processor 21 executes the neural network-based crowd counting method by running the non-volatile software program and instructions stored in the memory 22.
[0110] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0111] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, they execute the neural network-based crowd counting method described in Embodiment 1 above, for example, executing the method described above. Figure 1-9 The steps shown.
[0112] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A crowd counting method based on neural networks, characterized in that, include: Acquire multiple sample images; The sample image is input into the original neural network model to be trained to obtain the first loss function of the original neural network model; Specifically, this includes: obtaining head position information based on the annotation information of the sample image; obtaining a ground truth density map of the sample image based on the head position information; analyzing the sample image using the original neural network model to obtain predicted density information of the sample image; and obtaining the first loss function based on the ground truth density map and the predicted density information. The sample image is cropped into a first sub-image and a second sub-image with overlapping regions, and the first sub-image and the second sub-image are respectively input into the original neural network model to obtain the second loss function of the original neural network model; specifically, the sample image is cropped into a first sub-image and a second sub-image with overlapping regions; the original neural network model is used to analyze the first sub-image and the second sub-image respectively to obtain the common prediction density and number of people estimate of the overlapping region in different sub-images, and the second loss function is obtained based on the common prediction density and number of people estimate; The original neural network model is iteratively trained based on the first loss function and the second loss function to minimize the loss of the original neural network model, thereby obtaining a target neural network model, and then crowd counting is performed based on the target neural network model; including: summing the first loss function and the second loss function to obtain a target loss function; using the stochastic gradient descent algorithm to obtain the minimum value of the target loss function to determine the optimal weights and optimal biases of the neural network, thereby obtaining the target neural network model.
2. The crowd counting method based on neural networks according to claim 1, characterized in that, The original neural network model is used to analyze the first subgraph and the second subgraph respectively to obtain the common prediction density and population estimate of the overlapping area in different subgraphs. The second loss function is obtained based on the common prediction density and population estimate, specifically including: The original neural network model is used to analyze the first subgraph and the second subgraph respectively to obtain the first number estimate of the overlapping area in the first subgraph and the second number estimate of the overlapping area in the second subgraph; The loss function is obtained based on the first and second number estimates. ; The original neural network model is used to analyze the first subgraph and the second subgraph respectively to obtain the first common prediction density of the overlapping region in the first subgraph and the second common prediction density of the overlapping region in the second subgraph. The loss function is obtained based on the first common prediction density and the second common prediction density. ; For the loss function and loss function The second loss function is obtained by summing the results.
3. The crowd counting method based on neural networks according to claim 1, characterized in that, Before inputting the sample image, the first sub-image, and the second sub-image into the original neural network, the process also includes: adjusting the sample image, the first sub-image, and the second sub-image to the same size.
4. The crowd counting method based on neural networks according to any one of claims 1 to 3, characterized in that, The neural network-based crowd counting method also includes: The target neural network model is used to predict the number of people in multiple sample images, and the predicted number of people for each sample image is obtained. The mean absolute error and mean square error are obtained by comparing the predicted number of people with the actual number of people in all sample images. If the mean absolute error and the mean square error are not within the set range, the neural network model will continue to be trained to optimize the neural network model.
5. The crowd counting method based on neural networks according to any one of claims 1 to 3, characterized in that, The process of cropping the sample image into a first sub-image and a second sub-image with overlapping regions specifically includes: The sample image is cropped using image blocks with side length K to obtain the first sub-image and the second sub-image; Wherein, the height of the overlapping region corresponding to the first sub-image and the second sub-image is H and the width is W, and the range of the height H and width W of the overlapping region is: Where K is a random number between 224 and 336.
6. The crowd counting method based on neural networks according to any one of claims 1 to 3, characterized in that, The original neural network model is constructed based on a convolutional neural network model with VGG-16 as the backbone. Specifically, the original neural network model includes: The first part is the network front end, which includes ten convolutional layers and four pooling layers for feature extraction from the input image. The network front end removes all the subsequent fully connected layers to make it suitable for output at any resolution. The second part is the final segment of the network, which includes three convolutional kernels used to output the predicted density map.
7. A crowd counting device based on a neural network, characterized in that, include: At least one processor and at least one memory; The at least one processor and the at least one memory are communicatively connected to each other. The at least one memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the neural network-based crowd counting method as described in any one of claims 1-6.