Crowd density analysis method and system based on two-path cooperation

Abstract

This invention provides a crowd density analysis method based on dual-path collaboration, comprising: constructing an analysis model, the analysis model including a first generative model, the first generative model taking an image as input and outputting a crowd density map of the image; training the analysis model, comprising: constructing a training set; constructing a first generative adversarial network (GAN) using the first generative model and a first discriminant model, and inputting the training images from the training set into the first GAN; constructing a second GAN using a second generative model and a second discriminant model, randomly segmenting the training image into multiple sub-training images, and inputting these sub-images into the second GAN; optimizing the first generative model using the loss functions of the first and second GANs; and inputting the original image into the trained analysis model to obtain the crowd density map of the original image. This invention also provides a system. This invention uses only the first generative model when analyzing the original image, reducing model overhead and complexity.

Description

Technical Field

[0001] This invention belongs to the field of crowd density technology, specifically relating to a crowd density analysis method and system based on dual-path collaboration. Background Technology

[0002] Estimating crowd density has significant guiding implications for security, urban management, and crowd control. According to relevant literature, there are two main technical approaches: statistical visual analysis methods and deep neural network-based methods.

[0003] Statistical visual analysis methods typically construct detectors to detect pedestrians and estimate crowd density by counting the detected bounding boxes. Specifically, this involves constructing statistical features, such as HoG and SIFT, performing statistical analysis on the image, and identifying regions with specific statistical patterns as valid target areas to detect pedestrians. There are also methods that directly regress the number of people in an image using statistical features. However, these methods have significant limitations: First, statistical visual analysis methods are greatly affected by pixel value distribution, and in the real world, factors such as lighting, occlusion, rain, and fog significantly affect the pixel value distribution of an image, thus impacting detection performance. Second, for extremely dense crowds, this method is not suitable because statistical descriptions tend to be similar in such scenarios. Finally, regressing the number of people in an image through feature mapping cannot consider spatial and density information.

[0004] Deep neural network-based methods use stacked structures such as convolutional neural networks and fully connected neural networks. Through specific learning strategies, they use a large number of matched samples and the principle of error backpropagation to fit a function between the samples and the target result. It is an end-to-end model that does not require much manual construction or integration within the model. Technically, these methods can be divided into three directions: First, regression methods using a combination of basic convolutional neural networks and fully convolutional networks directly regress the number of people in an image. This method cannot obtain the density distribution, and the regression of a three-dimensional image matrix to a one-dimensional numerical value also destroys the original spatial information of the image. Second, context-aware methods use neural networks to learn the context information of an image and divide the image into several levels according to the number of people based on the context information. The range of people corresponding to each level is derived from statistics on the dataset. Although this method can use context information to abstract the image, the determination of the level boundaries is based on sample statistics, which is difficult to determine in actual production. Third, scale-aware methods use convolutional kernels of different sizes to achieve scale awareness in the image. In addition, using different slices can also achieve a similar effect to the context-aware method. Due to the use of scale-aware methods, the model has a large overhead, long computation time, and high consumption of computing resources (memory usage, CPU usage). Summary of the Invention

[0005] To address one or more of the problems existing in the prior art, the present invention provides a crowd density analysis method based on dual-path collaboration, comprising:

[0006] An analysis model is constructed, which includes a first generative model, wherein the first generative model takes an image as input and outputs a crowd density map of the image;

[0007] Training and analyzing the model;

[0008] The original image is input into the trained analysis model to obtain the crowd density map of the original image;

[0009] The steps for training the analysis model include:

[0010] Construct the training set;

[0011] A first generative adversarial network is constructed by using a first generative model and a first discriminative model, and the training images from the training set are input into the first generative adversarial network.

[0012] A second generative adversarial network is constructed using a second generative model and a second discriminative model. The training images are randomly segmented into multiple sub-training images, which are then input into the second generative adversarial network.

[0013] The first generative model is optimized using the loss functions of the first and second generative adversarial networks.

[0014] Optionally, the step of constructing a first generative adversarial network through a first generative model and a first discriminative model, and inputting the training images of the training set into the first generative adversarial network, includes:

[0015] The training image is input into the first generative model, and the first crowd density estimation map is output.

[0016] Obtaining a true crowd density map of the training images, preferably, includes: labeling heads on the training images; and replacing the head labels with a two-dimensional Gaussian distribution with a mean of 1 and a set size.

[0017] Input the first population density estimation map and the real population density map into the first discrimination model, and output the first discrimination error;

[0018] The L2 loss between the first population density estimation map and the actual population density map is used as the first L2 loss error.

[0019] Preferably, it further includes:

[0020] The first estimated crowd density map and the actual crowd density map are input into the first perception network, and the first perception error is output; more preferably, the first perception error is obtained by the following formula:

[0021]

[0022] Among them, L P (G) represents the perceptual error; G is the generative model, T is the number of perceptual features, t is the perceptual feature index, and f is the perceptual error. Generated (t) represents the t-th perceptual feature of the crowd density estimation map, f GroundTruth (t) represents the t-th perceptual feature of the real crowd density map;

[0023] Preferably, the step of inputting the first population density estimation map and the actual population density map into the first discrimination model and outputting the first discrimination error includes:

[0024] Input the first estimated population density map and the actual population density map into the first discrimination model, and obtain the first discrimination error using the following formula:

[0025] L A (G, D) = E x，y～Pdata(x，y) [logD(x, y)] + E x～Pdata(x) [log(1-D(x, G(x)))]

[0026] Among them, L A(G, D) represents the discrimination error; G is the generative model, D is the discriminative model, x is the training image, y is the crowd density estimation map, and Pdata(x, y) is the true distribution of the training image and the crowd density estimation map; E x，y～Pdata(x，y) denoted as the expected value of the distribution function; D(x, y) is the discriminant output of the discriminant model for x and y; G(x) is the output of the generative model; D(x, G(x)) is the discriminant output of the discriminant model for x and G(x);

[0027] Preferably, the step of using the L2 loss of the first population density estimation map and the actual population density map as the first L2 loss error includes:

[0028] The first L2 loss error is obtained using the following formula:

[0029]

[0030] Among them, L E (G) represents the L2 loss; G is the generative model, C is the number of pixels, c is the pixel index, and S... Generated (c) represents the pixel value of the c-th pixel in the crowd density estimation map, S GroundTruth (c) is the pixel value of the c-th pixel in the actual crowd density map.

[0031] Optionally, the step of constructing a second generative adversarial network using a second generative model and a second discriminative model, randomly segmenting the training images into multiple sub-training images, and inputting them into the second generative adversarial network includes:

[0032] The training images are randomly segmented to form multiple sub-training images;

[0033] Multiple batches of sub-training images are fed into the second generative model to generate multiple second crowd density estimation maps.

[0034] The real population density map is divided into multiple sub-real population density maps according to the segmentation of the training image;

[0035] Input the second population density estimation map and the corresponding sub-real population density map into the second discrimination model to obtain the second discrimination error;

[0036] The L2 loss of the second population density estimation map and the corresponding sub-real population density map is used as the second L2 loss error.

[0037] Integrate multiple second-person population density estimation maps to obtain a third-person population density estimation map;

[0038] Obtain the residuals of the third population density estimation map and the first population density estimation map;

[0039] Preferably, it further includes:

[0040] The second population density estimation map and the corresponding sub-true population density image are input into the second perception network, and the second perception error is output.

[0041] Preferably, the step of obtaining the residuals of the third population density estimation map and the first population density estimation map includes:

[0042] The residual is obtained using the following formula:

[0043]

[0044] Among them, L C For the residual, S Large (c) represents the pixel value of the c-th pixel in the first crowd density estimation map, S Small (c) represents the pixel value of the c-th pixel in the third population density map.

[0045] Optionally, the step of optimizing the first generative model using the loss functions of the first and second generative adversarial networks includes:

[0046] A loss constraint model is constructed by weighting multiple factors from the first L2 loss error, the second L2 loss error, the first discrimination error, the second discrimination error, the first perception error, the second perception error, and the residual. Preferably, the loss constraint model is constructed using the following formula:

[0047]

[0048] Among them, L Generator For the total loss, ω e ω p and ω c These represent the weights of L2 loss, perception error, and residual, respectively.

[0049] Optimizing the first generative model using a loss constraint model, preferably, includes:

[0050] Input the training set into the loss-constrained model to obtain the total loss;

[0051] The first generative model that minimizes the total loss is taken as the optimal first generative model.

[0052] Optionally, in the step of constructing a second generative adversarial network through a second generative model and a second discriminative model, randomly segmenting the training image into multiple sub-training images, and inputting them into the second generative adversarial network, the training image is segmented into four sub-training images.

[0053] Optionally, the first sensing network and the second sensing network are VGG-16 networks.

[0054] According to another aspect of the present invention, a crowd density analysis system based on dual-path collaboration is provided, comprising:

[0055] An analysis model building module is used to build an analysis model, which includes a first generation model. The first generation model takes an image as input and outputs a crowd density map of the image.

[0056] The training module trains the analytical model built by the analytical model construction module;

[0057] Image acquisition module, acquires raw images;

[0058] The output module takes the raw images acquired by the image acquisition module and inputs them into the analysis model trained by the training module, and outputs a crowd density map of the raw images.

[0059] The training module includes:

[0060] Training set construction unit: Collect training images to construct the training set;

[0061] The first network construction unit constructs a first generative adversarial network through a first generative model and a first discriminative model;

[0062] The second network construction unit constructs a second generative adversarial network through a second generative model and a second discriminative model;

[0063] The optimization unit inputs the training images from the training set into the first generative adversarial network, randomly segments the training images into multiple sub-training images, inputs them into the second generative adversarial network, and optimizes the first generative model through the loss functions of the first and second generative adversarial networks.

[0064] Optionally, the first generative adversarial network includes a first generative model and a first discriminative model. The input of the first generative model is a training image, and the output is a first crowd density estimation map. The input of the first discriminative model is a real crowd density map and the first crowd density estimation map, and the output is a first discrimination error. Preferably, the first network construction unit further includes a first L2 loss error acquisition unit, which uses the L2 loss of the first crowd density estimation map and the real crowd density map as the first L2 loss error. Preferably, the first network construction unit further includes a first perceptual network. The input of the first perceptual network is the first crowd density estimation map and the real crowd density map, and the output is a first perceptual error of the perceptual features of the first crowd density estimation map and the real crowd density map.

[0065] Optionally, the optimization unit includes a first segmentation unit and a second segmentation unit, and the second generative adversarial network includes a second generative model and a second discriminative model. The first segmentation unit randomly segments the training image into multiple sub-training images; the second generative model takes the sub-training images as input and outputs a second crowd density estimation map corresponding to the sub-training images; the second segmentation unit segments the real crowd density map into multiple sub-real crowd density maps according to the same segmentation as the first segmentation unit; the second discriminative model takes the second crowd density estimation map generated by the second generative model and the corresponding sub-real crowd density maps of the second segmentation unit as input and outputs a second discriminative error; preferably, the second network construction unit also includes... The system includes a second L2 loss error acquisition unit, which uses the L2 loss of the second population density estimation map and the sub-real population density map as the second L2 loss error. Preferably, the second network construction unit further includes a second perceptual network, whose input is the second population density estimation map and the sub-real population density map, and whose output is the second perceptual error of the perceptual features of the second population density estimation map and the sub-real population density map. Preferably, the second network construction unit further includes an integration unit, and the optimization unit further includes a residual acquisition unit. The integration unit integrates multiple second population density estimation maps to obtain a third population density estimation map, and the residual acquisition unit obtains the residual between the third population density estimation map and the first population density estimation map.

[0066] Optionally, the optimization unit further includes an error acquisition unit and a combination unit. The error acquisition unit acquires multiple errors from the following: a first L2 loss error, a second L2 loss error, a first discrimination error, a second discrimination error, a first perception error, a second perception error, and a residual. The combination unit performs a weighted combination of the multiple errors acquired by the error acquisition unit.

[0067] This invention introduces multi-scale feature representation through random segmentation, and achieves consistency constraint through the loss of the first and second generative adversarial networks, thereby improving the accuracy of crowd density estimation by optimizing the consistency of multi-scale and crowd density. The first generative model of the analysis model is optimized by the first and second generative adversarial networks. In the application process, only the first generative model is used, which has a short computation time, low computational resource consumption, and fast analysis speed. Attached Figure Description

[0068] Figure 1 This is a schematic diagram of the crowd density analysis system based on dual-path collaboration described in this invention;

[0069] Figure 2 This is a schematic diagram of the structural block diagram of the crowd density analysis system based on dual-path collaboration described in this invention;

[0070] Figure 3This is a schematic diagram of the flowchart of the population density analysis method based on dual-path collaboration described in this invention. Detailed Implementation

[0071] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0072] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0073] Figure 1 This is a schematic diagram of the crowd density analysis system based on dual-path collaboration described in this invention. Figure 2 This is a schematic diagram of the structural block diagram of the crowd density analysis system based on dual-path collaboration described in this invention, as shown below. Figure 1 and 2 As shown, the crowd density analysis system based on dual-path collaboration includes...

[0074] Analysis model construction module 1 constructs an analysis model, which includes a first generation model G1. The first generation model takes an image as input and outputs a crowd density map of the image.

[0075] Training Module 2: Training the analysis model constructed by the analysis model construction module;

[0076] Image acquisition module 3 acquires the original image;

[0077] Output module 4 takes the original images acquired by the image acquisition module and inputs them into the analysis model trained by the training module, and outputs a crowd density map of the original images.

[0078] In one embodiment, the training module includes:

[0079] Training set construction unit 21 collects training images to construct the training set;

[0080] The first network construction unit 22 constructs a first generative adversarial network through a first generative model and a first discriminative model;

[0081] The second network construction unit 23 constructs a second generative adversarial network through a second generative model and a second discriminative model;

[0082] The optimization unit 24 inputs the training images from the training set into the first generative adversarial network, randomly segments the training images into multiple sub-training images, inputs them into the second generative adversarial network, and optimizes the first generative model through the loss functions of the first and second generative adversarial networks.

[0083] In one embodiment, the first generative adversarial network includes a first generative model G1 and a first discriminative model D1. The input of the first generative model is a training image, and the output is a first crowd density estimation map E1. The input of the first discriminative model is a real crowd density map GT1 and the first crowd density estimation map E1, and the output is a first discrimination error.

[0084] Preferably, the first network construction unit further includes a first L2 loss error acquisition unit 221, which uses the L2 loss of the first crowd density estimation map and the real crowd density map as the first L2 loss error.

[0085] Preferably, the first network construction unit further includes a first perception network V1, wherein the input of the first perception network is a first crowd density estimation map and a real crowd density map, and the output is a first perception error of the perception features of the first crowd density estimation map and the real crowd density map.

[0086] In one embodiment, the optimization unit includes a first segmentation unit 241 and a second segmentation unit 242, and the second generative adversarial network includes a second generative model G2 and a second discriminative model D2. The first segmentation unit randomly segments the training image into multiple sub-training images; the second generative model takes the sub-training images as input and outputs a second crowd density estimation map E2 corresponding to the sub-training images; the second segmentation unit segments the real crowd density map into multiple sub-real crowd density maps GT2 according to the same segmentation as the first segmentation unit; the second discriminative model takes the second crowd density estimation map generated by the second generative model and the corresponding sub-real crowd density maps of the second segmentation unit as input and outputs a second discriminative error.

[0087] Preferably, the second network construction unit further includes a second L2 loss error acquisition unit 231, which uses the L2 loss of the second population density estimation map and the sub-real population density map as the second L2 loss error.

[0088] Preferably, the second network construction unit further includes a second perception network V2, the input of which is a second crowd density estimation map and a sub-real crowd density map, and the output is a second perception error of the perception features of the second crowd density estimation map and the sub-real crowd density map.

[0089] Preferably, the second network construction unit further includes an integration unit 232, and the optimization unit further includes a residual acquisition unit 243. The integration unit integrates multiple second population density estimation maps to obtain a third population density estimation map E3; the residual acquisition unit obtains the residual LC between the third population density estimation map and the first population density estimation map.

[0090] In one embodiment, the optimization unit further includes an error acquisition unit 244 and a combination unit 245. The error acquisition unit acquires multiple of the following: a first L2 loss error, a second L2 loss error, a first discrimination error, a second discrimination error, a first perception error, a second perception error, and a residual. The combination unit performs a weighted combination of the multiple errors acquired by the error acquisition unit.

[0091] In a preferred embodiment, the training module includes a first branch B1, a second branch B2, and an optimization unit. The first branch consists of a first generative adversarial network (GAN) and a first perceptual network V1. The first GAN consists of a first generative model G1 and a first discriminative model D1. The second branch consists of a second GAN and a second perceptual network V2. The second GAN consists of a second generative model G2, a second discriminative model D2, and an integration unit. The input of the first branch is the training image, and the output is a first crowd density estimation map E1 corresponding to the training image. The input of the second branch is the same as the input of the first branch model. Figure 1 The first and second branches are randomly divided into four parts, and the output is the third population density estimation map E3 corresponding to the four sub-training images. The optimization unit takes the first population density estimation map E1 output from the first branch and the third population density estimation map E3 obtained by stitching the four sub-branches of the second population density estimation map E2 from the second branch as input, and outputs the difference (second norm) between the first and third population density estimation maps as the residual LC. The loss of the first branch, the loss of the second branch and the residual are combined in a weighted polynomial manner to jointly optimize the first and second branches.

[0092] Figure 3 This is a schematic diagram of the flowchart of the crowd density analysis method based on dual-path collaboration described in this invention, as shown below. Figure 3 As shown, the crowd density analysis method based on dual-path collaboration includes:

[0093] Step S1: Construct an analysis model, which includes a first generation model. The first generation model takes an image as input and outputs a crowd density map of the image.

[0094] Step S2: Train and analyze the model;

[0095] Step S3: Input the original image into the trained analysis model to obtain the crowd density map of the original image.

[0096] In one embodiment, step S2 includes:

[0097] Construct the training set;

[0098] A first generative adversarial network is constructed by using a first generative model and a first discriminative model, and the training images from the training set are input into the first generative adversarial network.

[0099] A second generative adversarial network is constructed using a second generative model and a second discriminative model. The training images are randomly segmented into multiple sub-training images, which are then input into the second generative adversarial network.

[0100] The first generative model is optimized using the loss functions of the first and second generative adversarial networks.

[0101] In one embodiment, the step of constructing a first generative adversarial network using a first generative model and a first discriminative model, and inputting the training images from the training set into the first generative adversarial network, includes:

[0102] The training image is input into the first generative model, and the first crowd density estimation map is output.

[0103] Obtaining a true crowd density map of the training images, preferably, includes: labeling heads on the training images; and replacing the head labels with a two-dimensional Gaussian distribution with a mean of 1 and a set size.

[0104] Input the first population density estimation map and the real population density map into the first discrimination model, and output the first discrimination error;

[0105] The L2 loss between the first population density estimation map and the actual population density map is used as the first L2 loss error.

[0106] Preferably, it further includes:

[0107] The first estimated crowd density map and the actual crowd density map are input into the first perception network, and the first perception error is output; more preferably, the first perception error is obtained by the following formula:

[0108]

[0109] Among them, L P (G) represents the perceptual error; G is the generative model, T is the number of perceptual features, t is the perceptual feature index, and f is the perceptual error. Generated (t) represents the t-th perceptual feature of the crowd density estimation map, f GroundTruth (t) represents the t-th perceptual feature of the real population density map.

[0110] In one embodiment, the step of inputting the first population density estimation map and the actual population density map into the first discrimination model and outputting the first discrimination error includes:

[0111] Input the first estimated population density map and the actual population density map into the first discrimination model, and obtain the first discrimination error using the following formula:

[0112] L A (G, D) = E x，y～Pdata(x，y) [logD(x, y)] + E x～Pdata(x) [log(1-D(x, G(x)))]

[0113] Among them, L A (G, D) represents the discrimination error; G is the generative model, D is the discriminative model, x is the training image, y is the crowd density estimation map, and Pdata(x, y) is the true distribution of the training image and the crowd density estimation map; E x，y～Pdata(x，y) denoted as the expected value of the distribution function; D(x, y) is the discriminant output of the discriminant model for x and y; G(x) is the output of the generative model; and D(x, G(x)) is the discriminant output of the discriminant model for x and G(x).

[0114] In one embodiment, the step of using the L2 loss of the first crowd density estimation map and the actual crowd density map as the first L2 loss error includes:

[0115] The first L2 loss error is obtained using the following formula:

[0116]

[0117] Among them, L E (G) represents the L2 loss; G is the generative model, C is the number of pixels, c is the pixel index, and S... Generated (c) represents the pixel value of the c-th pixel in the crowd density estimation map, S GroundTruth (c) is the pixel value of the c-th pixel in the actual crowd density map.

[0118] In one embodiment, the step of constructing a second generative adversarial network using a second generative model and a second discriminative model, randomly segmenting the training image into multiple sub-training images, and inputting them into the second generative adversarial network includes:

[0119] The training images are randomly segmented to form multiple sub-training images;

[0120] Multiple batches of sub-training images are fed into the second generative model to generate multiple second crowd density estimation maps.

[0121] The real population density map is divided into multiple sub-real population density maps according to the segmentation of the training image;

[0122] Input the second population density estimation map and the corresponding sub-real population density map into the second discrimination model to obtain the second discrimination error;

[0123] The L2 loss of the second population density estimation map and the corresponding sub-real population density map is used as the second L2 loss error.

[0124] Integrate multiple second-person population density estimation maps to obtain a third-person population density estimation map;

[0125] Obtain the residuals of the third population density estimation map and the first population density estimation map.

[0126] Preferably, it further includes:

[0127] The second population density estimation map and the corresponding sub-true population density image are input into the second perception network, and the second perception error is output.

[0128] In one embodiment, the step of obtaining the residuals of the third population density estimation map and the first population density estimation map includes:

[0129] The residual is obtained using the following formula:

[0130]

[0131] Among them, L C For the residual, S Large (c) represents the pixel value of the c-th pixel in the first crowd density estimation map, S Small (c) represents the pixel value of the c-th pixel in the third population density map.

[0132] In one embodiment, the step of optimizing the first generative model using the loss functions of the first generative adversarial network and the second generative adversarial network includes:

[0133] A loss constraint model is constructed by weighting multiple factors from the first L2 loss error, the second L2 loss error, the first discrimination error, the second discrimination error, the first perception error, the second perception error, and the residual. Preferably, the loss constraint model is constructed using the following formula:

[0134]

[0135] Among them, L Generator For the total loss, ω e ω p and ω c These represent the weights of L2 loss, perception error, and residual, respectively.

[0136] Optimizing the first generative model using a loss constraint model, preferably, includes:

[0137] Input the training set into the loss-constrained model to obtain the total loss;

[0138] The first generative model that minimizes the total loss is taken as the optimal first generative model.

[0139] In one embodiment, in the step of constructing a second generative adversarial network through a second generative model and a second discriminative model, randomly segmenting the training image into multiple sub-training images, and inputting them into the second generative adversarial network, the training image is segmented into four sub-training images.

[0140] In the above embodiments, the first sensing network and the second sensing network are VGG-16 networks.

[0141] In the above embodiments, the crowd density map maps people from the original image to a density map using a specific mapping method, so that the integral of the density map can represent the number of people in the original image. The intensity distribution of the density map represents the distribution of the crowd in the original image. For example, replacing the locations of people in the original image with a two-dimensional Gaussian distribution with a mean of 1 and a variance of 0 generates a heat map, which represents the distribution of the crowd in the image. Integrating this heat map can represent the number of people in the corresponding image. The number of people calculated in this way is the estimated value of the crowd density, i.e., the crowd density estimate.

[0142] In a preferred embodiment, the crowd density analysis method based on dual-path collaboration described in this invention is implemented through a first branch B1 and a second branch B2. The input to B1 is a complete training image with RGB channel order. First, it passes through a first generator model G1. The generator's destructured structure consists of a set of encoding networks with scales decreasing from large to small and a set of decoding networks with scales increasing from small to large (U-Net). The first generator generates a first crowd density estimation map E1, forming a first L2 loss. Then, E1 and the real crowd density map GT1 enter a first discriminator model D1, which forms a first discrimination error. Furthermore, E1 and the real crowd density map GT1 simultaneously enter a VGG-16 network to obtain perceptual features. The MSE of these perceptual features is calculated as the first perceptual error. These three error terms—the first L2 loss, the first discrimination error, and the first perceptual error—together constitute the loss term of B1.

[0143] The second branch, B2, is similar to B1. However, B2's input is not the complete training image but randomly segmented sub-training images. The second generative model G2 and the second discriminative model D2 used in B2 are independent models from the first generative model G1 and the first discriminative model D1 used in B1, with different model parameters. The randomly segmented images are resized to 1 / 4 of the original image and fed into the second generative model G2 in batches of size 4 (the four images can be the same or different sizes), generating the second crowd density estimation map E2, forming the second L2 loss. Then, E2 and the sub-real crowd density map GT2 are fed into the second discriminative model D2, forming the second discrimination error. Furthermore, E2 and the sub-real crowd density map GT2 are simultaneously fed into the VGG-16 network to obtain perceptual features, and the MSE of these perceptual features is calculated as the second perceptual error. These three error terms—the second L2 loss, the second discrimination error, and the second perceptual error—together constitute the loss term of B2.

[0144] The fourth second crowd density estimation map E2 is integrated to obtain the third crowd density estimation map E3. The difference between E1 and E3 is used to obtain the residual. Since E1 and E3 are both crowd density estimates of training images, the residual obtained by the difference between the two should be a crowd density map of the same size with all zeros. In this application, the residual is directly used to represent the L2 norm of E1 and E3 as the consistency constraint error term.

[0145] This invention has 7 error terms, namely the first discrimination error L of B1. A1 First perception error L P1 First L2 loss error L E1 The second discrimination error L of B2 A2 Second sensing error L P2 Second L2 loss error L E2 Consistency loss term (residual) of the two branches L C .

[0146] During the training phase, B1 and B2 are trained collaboratively. B1 includes adversarial generative loss (error loss), L2 loss, and perceptual loss. B2 follows the same process. The residuals between the two density maps generated by the generative models of B1 and B2 are added to the overall loss function as a consistency constraint. All these loss terms together constitute the overall loss function during training, and the collaborative optimization of B1 and B2 can be achieved through backpropagation. During the testing and real-world application phases, only the first generative model of the B1 branch is operational.

[0147] The present invention does not limit the number of random segments, but too many segments will increase the amount of computation and also result in too few people. If too few people are involved, the detection effect will be the same as before. The size of the segmented images will be different, resulting in multiple scales. If too few segments are involved, the accuracy of scale perception will be reduced.

[0148] To illustrate the beneficial effects of this invention, MSE and MAE metrics were used. Comparisons were made on the ShangHaiTech dataset and the UCF_CC_50 dataset using various existing methods and the method of this application, wherein:

[0149]

[0150]

[0151] Where Count represents the true label value, Count(^) represents the predicted value, and N represents the number of pixels in the crowd density map. Specifically:

[0152] The results on the ShangHaiTech dataset are compared in Table 1 below:

[0153] Table 1

[0154]

[0155] The results on the UCF_CC_50 dataset are compared in Table 2 below:

[0156] Table 2

[0157] method MAE MSE [1] 467.0 798.5 [2] 377.6 509.1 [3] 318.1 439.2 [4] 295.8 320.9 This invention 270.3 317.6

[0158] In the two embodiments described above:

[0159] Method [1] adopts the method in "Cong Zhang, Hongsheng Li, Xiaogang Wang, and Xiaokang Yang. Cross-scene crowd counting via deep convolutional neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, pages 833–841, 2015", which proposes a deep convolutional neural network (CNN) for crowd counting. It is trained alternately with two related learning objectives: crowd density and crowd counting. This proposed switchable learning method can obtain better local optima for both objectives without unifying the density map with the crowd counting.

[0160] Method [2] adopts the method in "Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, and Yi Ma. Single-image crowd counting via multi-column convolutional neural network. In Computer Vision and Pattern Recognition, pages 589–597, 2016", which proposes MCNN to allow input images to have arbitrary size or resolution. By using filters with receptive fields of different sizes, the features learned by each column of CNN can adapt to changes in the size of people / heads due to perspective effects or image resolution. Multi-scale is used, which increases the overhead of the model.

[0161] Method [3] adopts the method in "Deepak Babu Sam, Shiv Surya, and R. Venkatesh Babu. Switching convolutional neural network for crowd counting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4031–4039, 2017", switching convolutional neural networks to improve the accuracy and localization of crowd count prediction by utilizing changes in crowd density within the image. Based on the crowd counting prediction quality of the CNN established during training, blocks from the grid in the crowd scene are distributed to independent CNN regressors. The independent CNN regressors are designed with different receptive fields, and a switching classifier is trained to relay the crowd scene blocks to the best CNN regressor. A hand-crafted density evaluation metric is introduced in the switching structure, increasing the model parameter tuning overhead. The branching of the distribution structure increases the model's resource and computational overhead, and cannot universally cope with real-world scenarios.

[0162] Method [4] adopts the method in "Vishwanath A. Sindagi and Vishal M. Patel. Generating high-quality crowd density maps using contextual pyramid CNNs. In international conference on computer vision, pages 1879–1888, 2017", switching the convolutional neural network to the contextual pyramid CNN (CP-CNN) method. It generates high-quality crowd density and count estimates by explicitly combining the global and local contextual information of the crowd image. Compared with the manual setting of boundaries in [3], a classifier is used here to classify the crowd density first, but it still cannot universally cope with the actual scene.

[0163] This invention introduces multi-scale feature representation through random segmentation, and achieves consistency constraint through the loss of a first generative adversarial network and a second generative adversarial network, thereby realizing multi-scale and crowd density consistency optimization.

[0164] Existing methods do not take into account the natural internal consistency of multi-scale and crowd density. This method uses two generative adversarial networks and consistency constraints. By randomly splitting one branch, it introduces feature representations at multiple scales, realizes the optimization of multi-scale and crowd density consistency, and improves the accuracy of crowd density estimation.

[0165] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A population density analysis method based on dual-path collaboration, characterized in that, include: An analysis model is constructed, which includes a first generative model, wherein the first generative model takes an image as input and outputs a crowd density map of the image; Training and analyzing the model; The original image is input into the trained analysis model to obtain the crowd density map of the original image; The steps for training the analysis model include: Construct the training set; A first generative adversarial network is constructed by using a first generative model and a first discriminative model, and the training images from the training set are input into the first generative adversarial network. A second generative adversarial network is constructed using a second generative model and a second discriminative model. The training images are randomly segmented into multiple sub-training images, which are then input into the second generative adversarial network. The first generative model is optimized using the loss functions of the first and second generative adversarial networks. The step of constructing a first generative adversarial network through a first generative model and a first discriminative model, and inputting the training images of the training set into the first generative adversarial network, includes: The training image is input into the first generative model, and the first crowd density estimation map is output. Obtain the true crowd density map of the training images; Input the first population density estimation map and the real population density map into the first discrimination model, and output the first discrimination error; The L2 loss between the first estimated population density map and the actual population density map is used as the first L2 loss error, which is obtained by the following formula: in, For L2 loss; To generate models, Number of pixels For pixel index, For the population density estimation map, the first The pixel value of each pixel. For the true population density map The pixel value of each pixel; The step of constructing a second generative adversarial network using a second generative model and a second discriminative model, randomly segmenting the training image into multiple sub-training images, and inputting them into the second generative adversarial network includes: The training images are randomly segmented to form multiple sub-training images; Multiple batches of sub-training images are fed into the second generative model to generate multiple second crowd density estimation maps. The real population density map is divided into multiple sub-real population density maps according to the segmentation of the training image; Input the second population density estimation map and the corresponding sub-real population density map into the second discrimination model to obtain the second discrimination error; The L2 loss of the second population density estimation map and the corresponding sub-real population density map is used as the second L2 loss error. Integrate multiple second-person population density estimation maps to obtain a third-person population density estimation map; Obtain the residuals of the third population density estimation map and the first population density estimation map, and use the residuals as the L2 norm of the third population density estimation map and the first population density estimation map, as the consistency constraint error term; The step of optimizing the first generative model using the loss functions of the first and second generative adversarial networks includes: A loss constraint model is constructed by weighting the first L2 loss error, the second L2 loss error, the first discrimination error, the second discrimination error, and the residual. Optimize the first generative model using a loss constraint model; The step of inputting the original image into the trained analysis model to obtain the crowd density map of the original image includes: The original image is input into the first generative model after training to obtain the crowd density map of the original image.

2. The crowd density analysis method based on dual-path collaboration according to claim 1, characterized in that, The steps for obtaining the true crowd density map of the training images include: labeling human heads on the training images; and replacing the human head labels with a two-dimensional Gaussian distribution with a mean of 1 and a set size.

3. The crowd density analysis method based on dual-path collaboration according to claim 1, characterized in that, The step of constructing a first generative adversarial network using a first generative model and a first discriminative model, and inputting the training images from the training set into the first generative adversarial network, further includes: The first population density estimation map and the actual population density map are input into the first perception network, and the first perception error is output.

4. The crowd density analysis method based on dual-path collaboration according to claim 3, characterized in that, The first perceptual error is obtained using the following formula: in, For perceived error; To generate models, Let t be the number of perceptual features and t be the index of the perceptual feature. For the population density estimation map, the first A perceptual feature, For the true population density map A perceptual feature.

5. The crowd density analysis method based on dual-path collaboration according to claim 1, characterized in that, The first discrimination error is obtained by the following formula: in, To determine the error; To generate models, To discriminate the model, For training images, For population density estimation map, The true distribution of data for training images and crowd density estimation maps; This represents the expected value of the distribution function. To determine the discriminant output of the model for x and y; For the output of the generation model; The discriminant output of G(x) is for the model to discriminate x.

6. The crowd density analysis method based on dual-path collaboration according to claim 3, characterized in that, The step of constructing a second generative adversarial network using a second generative model and a second discriminative model, randomly segmenting the training images into multiple sub-training images, and inputting them into the second generative adversarial network further includes: The second population density estimation map and the corresponding sub-true population density image are input into the second perception network, and the second perception error is output.

7. The crowd density analysis method based on dual-path collaboration according to claim 1, characterized in that, The steps for obtaining the residuals of the third population density estimation map and the first population density estimation map include: The residual is obtained using the following formula: in, For residuals, The first population density estimation map The pixel value of each pixel. For the third population density map The pixel value of each pixel.

8. The crowd density analysis method based on dual-path collaboration according to claim 6, characterized in that, The step of optimizing the first generative model using the loss functions of the first and second generative adversarial networks includes: A loss constraint model is constructed by weighting multiple combinations of the first L2 loss error, the second L2 loss error, the first discrimination error, the second discrimination error, the first perception error, the second perception error, and the residual. The first generative model is optimized using a loss constraint model.

9. The crowd density analysis method based on dual-path collaboration according to claim 8, characterized in that, The loss constraint model is constructed using the following formula: in, For the total loss, , and These represent the weights of L2 loss, perception error, and residual, respectively.

10. The crowd density analysis method based on dual-path collaboration according to claim 8, characterized in that, The steps of optimizing the first generative model using the loss constraint model include: Input the training set into the loss-constrained model to obtain the total loss; The first generative model that minimizes the total loss is taken as the optimal first generative model.

11. The crowd density analysis method based on dual-path collaboration according to claim 1, characterized in that, In the step of constructing a second generative adversarial network through a second generative model and a second discriminative model, the training image is randomly segmented into multiple sub-training images and input into the second generative adversarial network, in which the training image is segmented into four sub-training images.

12. The crowd density analysis method based on dual-path collaboration according to claim 6, characterized in that, The first and second sensing networks are VGG-16 networks.

13. A crowd density analysis system based on dual-path collaboration, characterized in that, include: An analysis model building module is used to build an analysis model, which includes a first generation model. The first generation model takes an image as input and outputs a crowd density map of the image. The training module trains the analytical model built by the analytical model construction module; Image acquisition module, acquires raw images; The output module takes the raw images acquired by the image acquisition module and inputs them into the analysis model trained by the training module, and outputs a crowd density map of the raw images. The training module includes: Training set construction unit: Collect training images to construct the training set; The first network construction unit constructs a first generative adversarial network through a first generative model and a first discriminative model; The second network construction unit constructs a second generative adversarial network through a second generative model and a second discriminative model; The optimization unit inputs the training images from the training set into the first generative adversarial network, randomly segments the training images into multiple sub-training images, inputs them into the second generative adversarial network, and optimizes the first generative model through the loss functions of the first and second generative adversarial networks. The input module inputs the original image into the trained first generative model to obtain the crowd density map of the original image; The first generative adversarial network includes a first generative model and a first discriminative model. The input of the first generative model is a training image, and the output is a first crowd density estimation map. The input of the first discriminative model is a real crowd density map and the first crowd density estimation map, and the output is a first discrimination error. The first network construction unit also includes a first L2 loss error acquisition unit, which uses the L2 loss of the first crowd density estimation map and the real crowd density map as the first L2 loss error. The optimization unit includes a first segmentation unit and a second segmentation unit. The second generative adversarial network includes a second generative model and a second discriminative model. The first segmentation unit randomly segments the training image into multiple sub-training images. The second generative model takes the sub-training images as input and outputs a second crowd density estimation map corresponding to the sub-training images. The second segmentation unit segments the real crowd density map into multiple sub-real crowd density maps according to the same segmentation as the first segmentation unit. The second discriminative model takes the second crowd density estimation map generated by the second generative model and the corresponding sub-real crowd density maps of the second segmentation unit as input and outputs a second discriminative error. The second network construction unit also includes a second L2 loss error acquisition unit, which uses the L2 loss of the second crowd density estimation map and the sub-real crowd density maps as the second L2 loss error. The second network construction unit also includes an integration unit, and the optimization unit also includes a residual acquisition unit. The integration unit integrates multiple second crowd density estimation maps to obtain a third crowd density estimation map. The residual acquisition unit obtains the residual between the third crowd density estimation map and the first crowd density estimation map.

14. The crowd density analysis system based on dual-path collaboration according to claim 13, characterized in that, The first network construction unit further includes a first perception network, the input of which is a first crowd density estimation map and a real crowd density map, and the output is a first perception error of the perception features of the first crowd density estimation map and the real crowd density map.

15. The crowd density analysis system based on dual-path collaboration according to claim 14, characterized in that, The second network construction unit further includes a second perception network. The input of the second perception network is a second crowd density estimation map and a sub-real crowd density map, and the output is a second perception error of the perception features of the second crowd density estimation map and the sub-real crowd density map.

16. The crowd density analysis system based on dual-path collaboration according to claim 15, characterized in that, The optimization unit further includes an error acquisition unit and a combination unit. The error acquisition unit acquires multiple errors from the following: first L2 loss error, second L2 loss error, first discrimination error, second discrimination error, first perception error, second perception error, and residual. The combination unit performs a weighted combination of the multiple errors acquired by the error acquisition unit.

Images