Method for estimating number of people in crowded scene and electronic equipment

Abstract

The invention discloses a method for estimating the number of people in a crowded scene and electronic equipment. The method comprises the following steps: S1, constructing a number of people density training data set; s2, constructing a people number density estimation network; s3, obtaining a trained population density estimation network; and S4, inputting the monitoring picture into the trained population density estimation network to obtain a population density estimation result. According to the method, a deep learning method is adopted, a foreground and a background can be well distinguished, especially when a portrait poster exists in a picture, false detection can be avoided, the number information and position information of passengers can be rapidly and accurately obtained in a region of interest given by a client, and the safety of the passengers can be improved through timely adjustment of safety personnel. Potential safety hazards caused by the fact that the number of passengers in the interested area exceeds the threshold value are avoided.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to a method and electronic equipment for estimating the number of people in a crowded scene. Background Art

[0002] With the development of urban transportation, more and more people choose subway as a means of travel. Generally, during peak hours in the morning and evening or during holidays, there is a large flow of passengers in subway stations, which is prone to unsafe incidents such as crowding. However, subway stations currently use security personnel to maintain on-site order. Due to the limited number of security personnel and their energy, it is impossible to quickly and accurately judge the degree of crowding on site, resulting in poor passenger experience and certain safety hazards. Therefore, it is necessary to realize real-time feedback on the degree of crowding of passengers in subway stations, so as to issue crowding warnings to passengers and adjust the number of security personnel in time.

[0003] Currently known public patents mainly provide the following two solutions for counting the density of people in subway stations: The detector-based method uses a general detector, such as the YOLO series, FASTER-RCNN, etc., to detect targets in the area of ​​interest, and then counts the number of people in the area based on the detection results. With the help of efficient detectors, this method can quickly and accurately give the number of people and location information in the area; the regression-based method, as described in the invention application CN104992223A, directly regresses the number of people in the image, so as to quickly obtain the approximate number of people in the area; the method based on the Gaussian kernel to generate a density map, as described in the invention application CN108876774A, is used for number counting. This patent uses a density map generated based on Gaussian blur, which can avoid the problem of detector performance degradation due to severe occlusion, and can quickly give the number of people in the area; the method based on the distance transformation function to generate a density map, as described in the invention application CN111062280, can quickly and accurately predict the number of people and target locations in the area.

[0004] The problems with the above methods are as follows: although the detection-based method can quickly and accurately obtain the number of people in the area, in crowded scenes, the accuracy of the detector drops rapidly due to severe occlusion of the target; the regression-based method has low accuracy and cannot accurately reflect the location information of people in the area; the density map-based method uses a density map generated by Gaussian blur, so the Gaussian kernel size is not easy to determine, and the Gaussian kernel affects the accuracy of the generated density map, which in turn affects the prediction accuracy. In addition, the single-scale prediction used in this method cannot effectively extract information from the image because the scale of the head distribution in the picture varies greatly due to the perspective of the camera in crowded scenes; the method based on the distance transformation function to generate a density map has the disadvantage that it cannot accurately distinguish between the foreground and the background, especially when there is a portrait poster in the background, there will be a problem of false detection, such as Figure 7-2 As shown in the figure, in addition, this method simply fuses the features output by different convolutional layers of the vgg16 layer, resulting in low utilization of feature information in different layers. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a method and electronic device for estimating the number of people in a crowded scene. The purpose of the present invention is achieved through the following technical solutions: A method for estimating the number of people in a crowded scene comprises the following steps: S1. Construct a population density training data set; S2. Build a population density estimation network and set the total loss function of each layer network : (1) in, represents the loss function of the pyramid; represents the background perception loss function; (2) ; ; in, Represents the label of the head detection box with i horizontal columns and j vertical rows; Indicates if it is a real person's head detection frame; Indicates the situation where the head detection frame is not a real person; F It is the density map output by the population density estimation network. represents the transpose of F; S3, train the population density estimation network until Converge and obtain a trained population density estimation network; S4. Input the monitoring image into the trained population density estimation network to obtain the population density estimation result.

[0006] Further improvements, (3); Among them, G k represents the true density map; F k represents the predicted density map obtained by the k-th layer network prediction of the population density estimation network, L ssim is the structural similarity loss function; K is the total number of layers of the population density estimation network; k represents the number of layers of the network.

[0007] A further improvement is that each layer of the population density estimation network includes an encoder and a decoder; the encoder receives information and encodes it and transmits it to the decoder for decoding, and the decoded information is used as the input of the encoder of the next level network; the encoder and the decoder both use two convolutions, and the output of each convolution corresponds to a linear activation unit.

[0008] For further improvement, K=5; the two convolutions of the encoder and decoder of the 4th layer network are 7*7 convolutions with a stride of 1, and the two convolutions of the remaining encoders and decoders are 3*3 convolutions with a stride of 1.

[0009] As a further improvement, in step S3, the population density estimation network is trained by a coarse-to-fine training method.

[0010] For further improvement, the steps of the coarse-to-fine training method are as follows: first, the labels in the crowd density training data set are used as supervision information, and the original images and corresponding labels are downsampled in turn to obtain images with a resolution of 1 / 2, 1 / 4, 1 / 8, and 1 / 16 of the original image and their corresponding supervision information; then the L0 layer network of the crowd density estimation network is first trained, corresponding to the image with an input resolution of 1 / 16 of the original image and its supervision information E 0 , the pyramid loss function is calculated according to formula (3), and the background perception loss is calculated according to formula (2). At this time, k=5. The L0 layer network is trained to converge to the L0 layer network, and the weight weight0 of the L0 layer network and the density map E0 output by the L0 layer network are obtained; then the L0 layer network loads the weight weight0 of the L0 layer network, and then the density map E0 is upsampled to obtain the density map E0' with a size of 1 / 8 of the original image, and then E0' and the supervision information E0 of the image with a size of 1 / 8 of the original image are upsampled. 1 Add them together to get E10, and then use formula (3) to calculate the pyramid loss function, at this time k = 4, and use formula (2) to calculate the background perception loss function. Train the L1 layer until the L1 layer network converges, get the weight weight1, and get the density map E1 output by the L1 layer; similarly, train the L2, L3, and L4 layer networks in turn, and finally get the trained population density estimation network.

[0011] As a further improvement, the density map is obtained by the following formula: (4); is a density map; I(x,y) is the Euclidean distance between x and y; x represents the x-axis coordinate in the head coordinate information, y represents the y-axis coordinate in the head coordinate information, and C is 1 to avoid division by zero.

[0012] As a further improvement, in step S4, the output of the L4 layer is operated using the maximum pooling operation to return all local maxima, i.e., candidate head points. If the maximum value of all local maxima is less than 0.1, it is considered that there is no person in the image and no statistics are performed. If it is greater than 0.1, a threshold T=100 / 255*the maximum value of all local maxima is set, and values ​​less than T are treated as 0 for filtering, and values ​​greater than or equal to T are treated as 1 and the corresponding head target box is obtained, and the size information of the head is obtained through the head target box.

[0013] A further improvement is to issue an alarm when it is detected that the population density is greater than a preset value.

[0014] As a further improvement, the population density training data sets are all resampled and normalized into samples with a size of 512*512 pixels.

[0015] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method described above.

[0016] The beneficial effects of the present invention are: 1) Adopting a coarse-to-fine training method, fully utilizing the information of feature maps of different scales, so that the detection results are more accurate; 2) In view of the problem that existing methods cannot accurately distinguish foreground and background, a background-aware loss is proposed to alleviate the problem of the model learning wrong information and effectively avoid the problem of detecting people in posters as real people. 3) The present invention adopts a deep learning method, which can distinguish the foreground and background very well. Especially when there is a portrait poster in the picture, the method of the present invention can avoid false detection and realize the rapid and accurate acquisition of passenger number information and location information in the customer's given area of ​​interest. Through timely adjustment by security personnel, the number of passengers in the area of ​​interest can be avoided from exceeding the threshold, thereby causing safety hazards. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present invention is further described with reference to the accompanying drawings, but the contents in the accompanying drawings do not constitute any limitation to the present invention.

[0018] Figure 1 is a flow chart of the present invention; Figure 2 It is a training process diagram of the present invention; Figure 3 This is a single-frame image alarm flow chart; Figure 4 This is the flow chart of the density estimation module; Figure 5-1 The original image for people counting; Figure 5-2 Pseudo label map generated for the original image; Figure 5-3 is the generated density map; Figure 6 It is the alarm logic flow chart; Figure 7-1 For portrait posters; Figure 7-2 This is a false detection image of a portrait poster; Figure 8 It is a schematic diagram of the network structure; Fig. 9 The structure of each layer of decoder and each layer of encoder. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0021] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0022] In this application, the word "exemplary" is used to mean "serving as an example, illustration, or description". Any embodiment described in this application as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments. The following description is given to enable any technician in the field to implement and use the present application. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present application can be implemented without using these specific details. In other instances, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present application with unnecessary details. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0023] In this invention patent, a method for estimating the number of people in a crowded scene is proposed to solve the problems of the prior art. Figure 1 and Figure 2 shown.

[0024] First, the present invention adopts the following improved method: The present invention sets a pyramid-based loss function to alleviate the problem of lack of semantic information in traditional MSE loss, as shown in formula (1); 3) The patent of the present invention also uses a distance transformation function to generate a density map, as shown in formula (2).

[0025] (1); Among them, G k represents the true density map; F k represents the predicted density map obtained by the k-th layer network prediction of the population density estimation network, Lssim is the structural similarity loss function; K is the total number of layers of the population density estimation network; k represents the number of layers of the network; the value of k is 1, 2, 3, 4, 5.

[0026] (2); is a density map; I(x,y) is the Euclidean distance between x and y; x represents the x-axis coordinate in the head coordinate information, y represents the y-axis coordinate in the head coordinate information, and C is 1 to avoid division by zero.

[0027] In view of the problem that existing methods cannot accurately distinguish foreground and background, the present invention proposes background-aware loss to alleviate the problem that the model learns erroneous information, as shown in formula (5).

[0028] (3); (4); (5); in, Represents the label of the head detection box with i horizontal columns and j vertical rows; Indicates if it is a real person's head detection frame; Indicates the case of a head detection frame that is not a real person; F is the density map output by the population density estimation network, represents the transpose of F.

[0029] And the total loss function : (6).

[0030] The specific steps are as follows: 1) Training Data First, the head coordinate information is obtained through manual annotation. Based on the head coordinate information, the density map is generated by formula (2), that is, the label of the training data; First, the labels in the crowd density training dataset are used as supervision information, and the original images and corresponding labels are downsampled in turn to obtain images with a resolution of 1 / 2, 1 / 4, 1 / 8, and 1 / 16 of the original image and their corresponding supervision information; then the L0 layer network of the crowd density estimation network is first trained, corresponding to the image with an input resolution of 1 / 16 of the original image and its supervision information E 0 , the pyramid loss function is calculated according to formula (1), and the background perception loss is calculated according to formula (5). At this time, k=5. The L0 layer network is trained until the L0 layer network converges, and the weight weight0 of the L0 layer network and the density map E0 output by the L0 layer network are obtained; then the L0 layer network loads the weight weight0 of the L0 layer network, and then the density map E0 is upsampled to obtain the density map E0' with a size of 1 / 8 of the original image, and then E0' and the supervision information E0 of the image with a size of 1 / 8 of the original image are upsampled. 1 Add them together to get E10, and then use formula (1) to calculate the pyramid loss function, at this time k = 4, and use formula (5) to calculate the background perception loss function. Train the L1 layer until the L1 layer network converges, get the weight weight1, and get the density map E1 of the L1 layer output.

[0031] Then the L0 layer network loads the L0 layer network weight weight0, the L1 layer network loads the L1 layer network weight weight1, and then upsamples the density map E1 to obtain a density map E1' with a size of 1 / 4 of the original image, and then E1' and the supervision information E of the image with a size of 1 / 4 of the original image are combined. 2Add them together to get E21, and then use formula (1) to calculate the pyramid loss function, at this time k = 3, and use formula (5) to calculate the background perception loss function. Train the L2 layer until the L2 layer network converges, get the weight weight2, and get the density map E2 of the L2 layer output.

[0032] Then the L0 layer network loads the L0 layer network weight weight0, the L1 layer network loads the L1 layer network weight weight1, and the L2 layer network loads the L2 layer network weight weight2, and then the density map E2 is upsampled to obtain a density map E2' with a size of 1 / 2 of the original image, and then E2' and the supervision information E of the image with a size of 1 / 2 of the original image are combined. 3 Add them together to get E32, and then use formula (1) to calculate the pyramid loss function, at this time k = 2, and use formula (5) to calculate the background perception loss function. Train the L3 layer until the L3 layer converges, get the weight weight3, and get the density map E3 output by the L3 layer.

[0033] Then the L0 layer network loads the L0 layer network weight weight0, the L1 layer network loads the L1 layer network weight weight1, the L2 layer network loads the L2 layer network weight weight2, and the L3 layer network loads the L3 layer network weight weight3, and then the density map E3 is upsampled to obtain the density map E3' of the same size as the original image, and then E3' and the supervision information E of the image of the same size as the original image are added. 4 Add them together to get E43, then use formula (1) to calculate the pyramid loss function, at this time k = 1, and use formula (5) to calculate the background perception loss function. Train the L4 layer until the L4 layer converges, get the weight weight4, get the density map E4 output by the L4 layer, and finally get the trained population density estimation network. The schematic diagram of the network structure is shown in Figure 8 shown.

[0034] The above density map is obtained by using the distance transformation function when constructing the data set.

[0035] In fact, the encoder of each layer of the density estimation network proposed in the patent of this invention can adopt the corresponding convolution structure in the backbone network as the encoder of this density estimation network, such as Resent, Vgg, Densenet, etc., and then add the corresponding decoder.

[0036] Among them, the ResNet residual network is mainly composed of residual blocks. Before the residual network was proposed, the network structure could not be very deep. In VGG, the convolutional network reached 19 layers, and in GoogLeNet, the network reached 22 layers. As the number of network layers increases, the network degrades: as the number of network layers increases, the training set loss gradually decreases, and then tends to saturation. When you increase the network depth, the training set loss will increase instead. After the introduction of residual blocks, the network can reach a very deep depth, and the network effect will also improve. That is, ResNet is a deep convolutional neural network that solves the gradient vanishing problem in deep networks by introducing residual connections. In ResNet, each convolutional layer does not directly map the input to the output, but adds the input to the output, that is, through a "shortcut" connection across several layers. This design makes the model easier to optimize and can still achieve good performance in deep networks with more than 100 layers.

[0037] VGG network is a common very deep convolutional network for large-scale image recognition. VGG was proposed by the Visual Geometry Group of Oxford. This network is related to the work at ILSVRC 2014. The main work is to prove that increasing the depth of the network can affect the final performance of the network to a certain extent. VGG has two structures, namely VGG16 and VGG19. There is no essential difference between the two, but the network depth is different. One improvement of VGG16 over AlexNet is to use several consecutive 3x3 convolution kernels instead of the larger convolution kernels (11x11, 7x7, 5x5) in AlexNet. For a given receptive field (the local size of the input image related to the output), using stacked small convolution kernels is better than using large convolution kernels, because multiple layers of nonlinear layers can increase the network depth to ensure learning of more complex patterns, and the cost is relatively small (fewer parameters). In simple terms, in VGG, three 3x3 convolution kernels are used to replace the 7x7 convolution kernel, and two 3x3 convolution kernels are used to replace the 5*5 convolution kernel. The main purpose of this is to increase the depth of the network while ensuring the same receptive field, and to a certain extent improve the effect of the neural network. The structure of the VGG network is very consistent, and 3x3 convolution and 2x2 max pooling are used from beginning to end.

[0038] DenseNet refers to Densely connected convolutional networks. Its advantages mainly include effectively alleviating gradient vanishing, more efficient feature transfer, less computation, fewer parameters, and better performance than ResNet. Its disadvantage is mainly the large memory usage. DenseNet network is similar to Resnet and GoogleNet, both of which are networks designed to solve the problem of gradient vanishing in deep networks. Resnet starts from the depth direction and can train deeper CNN networks by establishing "short-circuit connections" or shortcuts between the previous layer and the next layer.

[0039] The structure of the encoder and decoder used in the present invention is as follows: Fig. 9 As shown in the figure, each layer of encoder and decoder consists of a residual block. It is worth noting that except for the L4 layer of the encoder, which uses two 7*7 convolutions with a step size of 1, in order to increase the receptive field of the network, the remaining layers all use two 3*3 convolutions with a step size of 1. Each convolution is activated by a linear activation unit. In each layer of the encoder, except for the L0 layer, the convolution kernel is , convolution with a step size of 2 is used to downsample and change the channel information; after each decoder layer except the L4 layer, transposed convolution is used to restore the spatial dimension of the feature map.

[0040] Reasoning process: 1) Data input The present invention targets scenes such as waiting areas and platforms, and the data used are surveillance camera data of waiting areas and platforms. To ensure the normal operation of the algorithm, it is necessary to ensure that the data format in the input algorithm is correct, including real-time video streams, offline video streams, and surveillance pictures.

[0041] The user also needs to set the area of ​​interest. This patent supports users to set the area of ​​interest by drawing a line on the video screen with the mouse. The algorithm will calculate the number of people in the area of ​​interest and compare it with the set threshold. When the number of people in the area of ​​interest exceeds the threshold for a period of time, an alarm will be generated. The model will output an alarm picture, including a density map of the crowd distribution, a specific crowd head size map, and number information.

[0042] 2) Data preprocessing Resample and extract frames of the data input by the user. First, resample the size of the data to a size that meets the algorithm requirements. In this patent, the data input by the user is resampled to a size of 512*512 to reduce the memory and video memory usage.

[0043] 3) Density Estimation Network The trained density estimation network can be used for inference. The density estimation network process is as follows: Figure 4 As shown, the post-processing module is used to extract the coordinates of the head from the generated density map and perform head counting. Specifically, this patent uses the output of the L4 layer of the maximum pooling operation to return the local maximum value, sets a threshold T, uses T to filter out values ​​that do not meet the requirements, and treats the values ​​that do not meet the threshold T as 0, and the values ​​that meet the threshold T as 1. According to the head coordinate information obtained by post-processing, the specific position information of the head can be obtained. Based on the head coordinate information, the corresponding target box, that is, the detailed size information of the head, can be generated. For the input data, the user first sets the area of ​​interest, and the model only extracts features for the area of ​​interest, and then outputs a density map. The number of people in the area of ​​interest is obtained by processing the density map, and the specific distribution of the number of people and the size of the head are also output. The effect diagram is shown as follows Figure 5-1 and Figure 5-2 shown.

[0044] 4) Alarm When the number of people in the area of ​​interest exceeds the set threshold, the server alerts the staff, who can view the distribution of people and the degree of crowding in the area in real time. The overall alarm logic flow chart of the present invention is as follows: Figure 6 shown.

[0045] Structural Similarity Loss Function: SSIM loss function is an indicator used for image quality assessment, which can measure the similarity between two images. The full name of SSIM loss function is Structural Similarity Index, which is an image quality assessment method based on human visual perception. The calculation method of SSIM loss function is to evaluate the similarity between two images by comparing their brightness, contrast and structural similarity. It is a well-known loss function, so it will not be described in detail.

[0046] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for estimating the number of people in a crowded scene, characterized by: The steps include: S1. Construct a population density training data set; S2. Build a population density estimation network and set the total loss function of each layer network : (1) in, represents the loss function of the pyramid; represents the background perception loss function; （2） ； ； in, Represents the label of the head detection box with i horizontal columns and j vertical rows; Indicates if it is a real person's head detection frame; Indicates the case of a head detection frame that is not a real person; F It is the density map output by the population density estimation network, indicating The transpose of S3, train the population density estimation network until Converge and obtain a trained population density estimation network; S4. Input the monitoring image into the trained population density estimation network to obtain the population density estimation result.

2. The method for estimating the number of people in a crowded scene according to claim 1, characterized in that: (3) Among them, G k represents the true density map; F k represents the predicted density map obtained by the k-th layer network prediction of the population density estimation network, L ssim is the structural similarity loss function; K is the total number of layers of the population density estimation network; k represents the number of layers of the network.

3. The method for estimating the number of people in a crowded scene according to claim 1, characterized in that: Each layer of the population density estimation network includes an encoder and a decoder; the encoder receives information and transmits it to the decoder for decoding after encoding, and the decoded information is used as the input of the encoder of the next level network; the encoder and the decoder both use two convolutions, and the output of each convolution corresponds to a linear activation unit.

4. The method for estimating the number of people in a crowded scene according to claim 2, characterized in that: K=5; the two convolutions of the encoder and decoder of the 4th layer network are 7*7 convolutions with a stride of 1, and the two convolutions of the remaining encoders and decoders are 3*3 convolutions with a stride of 1.

5. The method for estimating the number of people in a crowded scene as claimed in claim 4, characterized in that: In step S3, the population density estimation network is trained by a coarse-to-fine training method.

6. The method for estimating the number of people in a crowded scene according to claim 5, characterized in that: In step S4, use The output of the L4 layer is max pooled and all local maxima, i.e., candidate head points, are returned. If the maximum value among all local maxima is less than 0.1, it is considered that there is no person in the image and no statistics are performed. If it is greater than 0.1, a threshold T is set and values ​​less than T are treated as 0 for filtering, and values ​​greater than or equal to T are treated as 1 and the corresponding head target box is obtained. The size information of the head is obtained through the head target box.

7. The method for estimating the number of people in a crowded scene according to claim 6, characterized in that: T = 100 / 255*the maximum of all local maxima.

8. The method for estimating the number of people in a crowded scene according to claim 1, characterized in that: When the population density is detected to be greater than the preset value, an alarm will be triggered.

9. The method for estimating the number of people in a crowded scene according to claim 1, characterized in that: The population density training data sets are all resampled and normalized to samples with a size of 512*512 pixels.

10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 9.

Images