A crowd gathering warning method based on Transformer technology

Abstract

The present invention provides a crowd gathering warning method based on the transformer technology in the field of image processing technology, including the following steps: Step S10, obtaining a large number of crowd images and annotating each of the crowd images; Step S20, converting each of the annotated crowd images into a crowd density map; Step S30, preprocessing each of the crowd density maps to obtain a training image set; Step S40, creating a crowd counting model based on the transformer and training the crowd counting model using the training image set; Step S50, using the trained crowd counting model to count the video collected by the camera, and then warning of crowd gathering. The advantages of the present invention are: greatly improving the accuracy and efficiency of crowd gathering recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a crowd gathering warning method based on transformer technology. Background Art

[0002] With the progress of technology, cameras have become popular, and monitoring through cameras has become a common technical means. In some scenarios, it is necessary to count the number of people gathering at the scene through the videos captured by cameras. When there are too many people gathering, warnings or evacuations need to be carried out for safety reasons.

[0003] Traditional crowd counting mainly calculates the number of people in a picture through object detectors, such as anchor base and anchor free; however, object detectors are limited by the lower limit of detecting the smallest object, and are only applicable to sparse crowd counting. For example, when there are fewer pedestrians in the picture and the pedestrians are more obvious, the recognition and calculation effect is better. However, when the pedestrian targets in the picture are small, the crowd density is high, and the occlusion between people is serious, traditional object detectors cannot effectively count, and traditional object detectors need to label the bounding boxes for each person, and labeling the bounding boxes for each person in a dense crowd is expensive and laborious. Therefore, traditional object detectors can no longer meet the current needs.

[0004] Therefore, how to provide a crowd gathering warning method based on transformer technology to improve the accuracy and efficiency of crowd gathering recognition has become an urgent technical problem to be solved. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a crowd gathering warning method based on transformer technology to improve the accuracy and efficiency of crowd gathering recognition.

[0006] The present invention is implemented as follows: A crowd gathering warning method based on transformer technology includes the following steps:

[0007] Step S10: Obtain a large number of crowd images and label each of the crowd images;

[0008] Step S20: Convert each of the labeled crowd images into a crowd density map;

[0009] Step S30: Preprocess each of the crowd density maps to obtain a training image set;

[0010] Step S40: Create a crowd counting model based on transformer and train the crowd counting model using the training image set;

[0011] Step S50: Use the trained crowd counting model to count the videos collected by the camera, and then alarm for crowd gathering.

[0012] Further, the specific steps of step S10 are as follows:

[0013] Obtain a large number of crowd images, and label each person's head in each crowd image through the coordinate points of one pixel.

[0014] Further, the specific steps of step S20 are as follows:

[0015] Use the Gaussian kernel function to replace the pixel values corresponding to the labeled coordinate points in each crowd image with the weighted average of the pixel values corresponding to the surrounding coordinate points, and then convert them into crowd density maps.

[0016] Further, in step S20, the expression function corresponding to the crowd density map is:

[0017]

[0018] Among them, F(x) represents the crowd density map; represents the crowd image with the coordinate points of N labeled people's heads; represents the Gaussian kernel function; σ i represents the head size coefficient; β represents the Gaussian parameter; represents the coordinate point x of the person's head i to the average distance of the surrounding k adjacent people's heads, and δ() represents the Dirac function, and

[0019] Further, the specific steps of step S30 are as follows:

[0020] Perform random cropping and flipping operations on each crowd density map to expand the data volume, and then uniformly set the resolution of each crowd density map to 256×256 to complete the preprocessing of each crowd density map to obtain a training dataset.

[0021] Further, in step S40, the crowd counting model consists of an encoder, a bottleneck layer, and a decoder; the bottleneck layer consists of two consecutive Swin Transformer Blocks; the encoder and the decoder adopt a skip connection method;

[0022] The encoder is used to perform downsampling operations on images, the bottleneck layer is used to extract features of images, and the decoder is used to perform upsampling operations on images.

[0023] Further, the Swin Transformer Block consists of a window multi-head self-attention layer and a shifted window multi-head self-attention layer, and the two adjacent Swin Transformer Blocks alternately use the window multi-head self-attention layer and the shifted window multi-head self-attention layer.

[0024] Further, in the step S40, the activation function of the crowd counting model adopts the Swish function.

[0025] Further, in the step S40, the loss function of the crowd counting model adopts the MSE loss function plus the MSSIM loss function:

[0026] L = L MSE + L MSSIM ;

[0027] wherein, L represents the total loss of the crowd counting model; L MSE represents the MSE loss; L MSSIM represents the MSSIM loss.

[0028] Further, the step S50 is specifically as follows:

[0029] Obtain the video collected by the camera, extract the image frames of the video in units of 1 second, input each of the image frames into the trained crowd counting model to obtain a predicted crowd density map, perform integral summation on the predicted crowd density map to count the number of people in the image frame, and determine whether the number of people exceeds a set threshold. If so, generate an alarm for crowd gathering; if not, continue to count the number of people in the next image frame.

[0030] The advantages of the present invention are as follows:

[0031] 1. By using the coordinate points of a pixel to label each person's head in each crowd image, converting the labeled crowd images into a crowd density map and performing preprocessing to obtain a training dataset, and then using the training dataset to train the crowd counting model created based on the transformer, finally, the trained crowd counting model can be used to count the number of people; since the labeling is performed using coordinate points, the limitation of detecting the smallest target of the traditional object detector is overcome, and there is no need to label the bounding box for each person, which is applicable to the situation where the pedestrian target is small, the crowd density is high, and the occlusion between people is serious, and finally, the accuracy and efficiency of crowd gathering recognition are greatly improved.

[0032] 2. By adopting the Swin Transformer Block in the crowd counting model to control the calculation area of self-attention within a window-based area, the calculation amount of the crowd counting model is greatly reduced, and the efficiency of crowd gathering recognition is further improved.

[0033] 3. By using a crowd counting model created based on transformers to replace the traditional convolutional neural network (CNN), the drawback of insufficient global information feature extraction by CNN is overcome, thereby greatly improving the accuracy of crowd gathering recognition.

[0034] 4. By replacing the activation function of the crowd counting model from the traditional ReLU function with the Swish function, and the Swish function has the characteristics of no upper bound, having a lower bound, being smooth, and non-monotonic, thereby greatly improving the feature extraction effect of the deep network in the crowd counting model.

[0035] 5. By preprocessing the crowd density map through random cropping and flipping operations, that is, performing data augmentation, and then training the crowd counting model with the augmented data, the generalization ability and robustness of the crowd counting model are greatly improved.

[0036] 6. By adding the MSSIM loss function to the loss function of the crowd counting model, which is used to compare the similarity between two images in three dimensions of brightness, contrast, and structure, and can better reflect the judgment of the human visual system on the similarity between two images. Brief Description of the Drawings

[0037] The present invention will be further described below with reference to the drawings in conjunction with the embodiments.

[0038] Figure 1 is a flowchart of a crowd gathering warning method based on transformer technology of the present invention.

[0039] Figure 2 is a schematic diagram of the image segmentation effect of W-MSA of the present invention.

[0040] Figure 3 is one of the schematic diagrams of the image segmentation effect of SW-MSA of the present invention.

[0041] Figure 4 is another schematic diagram of the image segmentation effect of SW-MSA of the present invention.

[0042] Figure 5 is a schematic diagram of the structure of the crowd counting model of the present invention.

[0043] Figure 6 is a schematic diagram of the structure of the Swin Transformer Block of the present invention.

[0044] Figure 7 is a comparison schematic diagram of the crowd image and the crowd density map of the present invention.

[0045] Figure 8It is a comparison schematic diagram of crowd prediction of the present invention. Detailed implementation manners

[0046] The technical solution in the embodiment of the present application has the following general idea: annotate each head in each crowd image through the coordinate points of a pixel, convert the annotated crowd images into crowd density maps and perform preprocessing to obtain a training set of images, and then use the training set of images to train a crowd counting model created based on the transformer. Finally, use the trained crowd counting model to count the crowd, overcoming the limitation of detecting the smallest target and eliminating the need to annotate bounding boxes, so as to improve the accuracy and efficiency of crowd gathering recognition.

[0047] Please refer to Figures 1 to 8 As shown, a preferred embodiment of a crowd gathering warning method based on the transformer technology of the present invention includes the following steps:

[0048] Step S10: Obtain a large number of crowd images and annotate each of the crowd images.

[0049] Step S20: Convert each of the annotated crowd images into a crowd density map (Density map).

[0050] Step S30: Perform preprocessing on each of the crowd density maps to obtain a training set of images.

[0051] Step S40: Create a crowd counting model based on the transformer of natural language processing and use the training set of images to train the crowd counting model.

[0052] In the task of crowd counting, it is necessary to generate a high-resolution density map with the same size as the original image, and then integrate on the density map to calculate the number of people in the image. Therefore, downsample the crowd image to obtain a low-resolution feature map, and then restore it from the low-resolution feature map to a high-resolution density map, so as to achieve multi-scale feature extraction in this way.

[0053] The crowd counting model adopts an encoder-decoder network structure similar to U-net, removes the CNN encoding, and replaces all the structures of U-net with Swin Transformer.

[0054] Step S50: Use the trained crowd counting model to count the video collected by the camera, and then give a warning for crowd gathering.

[0055] The specific content of step S10 is as follows:

[0056] Obtain a large number of crowd images and respectively annotate each head in each of the crowd images through the coordinate points of a pixel.

[0057] Since there are situations such as body overlap and occlusion in crowd counting in dense scenes, only the coordinate points are used to label and save the human heads.

[0058] The specific steps of step S20 are as follows:

[0059] Using the Gaussian kernel function, the pixel values corresponding to the coordinate points marked in each of the crowd images are replaced with the weighted average of the pixel values corresponding to the surrounding coordinate points, and then converted into a crowd density map.

[0060] In a real image, each human head has a certain size, corresponding to a small area in the image. It is obviously unreasonable to represent a human head with a single coordinate point in the annotation file. Therefore, the Gaussian kernel function is used to replace the pixel value of this coordinate point with the weighted average of the pixel values of its surrounding points. The weights of the surrounding pixel points add up to 1. In this way, it does not affect the total number of human heads in the generated crowd density map and can more realistically reflect the position characteristics of each human head in space.

[0061] In step S20, the expression function corresponding to the crowd density map is:

[0062]

[0063] Among them, F(x) represents the crowd density map; represents the crowd image with the coordinate points of N marked human heads; represents the Gaussian kernel function; σ i represents the human head size coefficient; β represents the Gaussian parameter, and its value is 0.3; represents the coordinate point x of the human head i to the average distance of the surrounding k adjacent human heads, and δ() represents the Dirac function, and the value of k is 4.

[0064] Since it is necessary to consider the influence of perspective distortion on the size of the human head, that is, when taking pictures, the farther an object is from the camera lens, the smaller it appears in the image, and the closer a human head is to the lens, the larger it is, and the occupied spatial pixel points are different. Therefore, when using the Gaussian kernel function, different blur radii need to be set according to the size of different human heads, that is, the parameter σ is determined according to the size of each human head in the image i . However, in actual situations, it is impossible to accurately obtain the size of each head, and it is difficult to find the relationship between the head size and the crowd density map. Therefore, the average distance between k adjacent human heads and this human head is used as the parameter σ of the Gaussian kernel function i , which can better represent the characteristic information of the human head size.

[0065] The specific steps of step S30 are as follows:

[0066] Randomly crop and flip each of the population density maps to increase the data volume, and then uniformly set the resolution of each of the population density maps to 256×256 to complete the preprocessing of each of the population density maps and obtain a training map set.

[0067] In the step S40, the population counting model is composed of an encoder, a bottleneck layer, and a decoder; the bottleneck layer is composed of two consecutive Swin Transformer Blocks; the encoder and the decoder adopt a skip connection method similar to Unet;

[0068] The encoder is used to perform downsampling on the image, the bottleneck layer is used to extract the features of the image, and the decoder is used to perform upsampling on the image.

[0069] The encoder first performs patch partition on the input image. Each patch has a size of 4×4, and the input dimension is H / 4×W / 4×48. After linear embedding and two Swin Transformer Blocks, the feature map size is H / 4×W / 4×C. Then, downsampling is performed through patch merging, and after two more Swin Transformer Blocks, the feature map size becomes H / 16×W / 16×2C. Finally, multiple similar downsampling operations are performed to complete the operation of the encoder. Each time, the encoder reduces the number of patches by a factor of 2 and then increases the number of feature dimensions by a factor of 2.

[0070] The bottleneck layer is composed of two consecutive Swin Transformer Blocks. To prevent the network from being too deep to converge, only two blocks are used. In the bottleneck layer, the feature size remains unchanged at H / 32×W / 32×8C.

[0071] The decoder mainly realizes upsampling through patch expanding. As a completely symmetric network structure, the decoder also performs upsampling by expanding by a factor of 2 each time. The core module is composed of a Swin Transformer block and patch expanding.

[0072] The Swin Transformer Block consists of a window multi-head self-attention layer (W-MSA) and a shifted-window multi-head self-attention layer (SW-MSA). The two adjacent Swin Transformer Blocks alternately use the window multi-head self-attention layer and the shifted-window multi-head self-attention layer.

[0073] The Swin Transformer Block uses window self-attention, which reduces the computational complexity. Assuming an image size of h×w, and each window contains M×M pixels, the computational complexity of the original Transformer self-attention is: (h×w) 2 , while in Swin Transformer, window self-attention is used, and its computational complexity is the window computational complexity multiplied by the number of windows. The number of windows = h×w / M 2 , and the window computational complexity = (M 2 ) 2 , so the Swin Transformer self-attention computational complexity is (h×w / M 2 ) × (M 2 ) 2 = M 2 × (h×w). The computational complexity is reduced from a quadratic relationship with the number of image pixels to a linear relationship with the square of the window size. To ensure the connection between non-overlapping windows, the self-attention after window offset is recalculated using the SW-MSA method.

[0074] W-MSA performs Self-Attention calculation (self-attention calculation) on the window size. The Windows division used is as Figure 2 shown, where the image is divided into non-overlapping Windows. However, this approach will make W-MSA lack connections between different windows, restricting the receptive field of the model. The modeling ability of the network obtained by using W-MSA alone is very poor because each window is treated as an independent region for calculation, ignoring the necessity of interaction between windows. Based on this motivation, SW-MSA is used.

[0075] The position of SW-MSA is connected after the W-MSA layer. Therefore, as long as a window splitting method different from W-MSA is provided, cross-window communication can be achieved. SW-MSA shifts each cycle of the image up and left by half the size of the window. This operation can be implemented through the roll function of pytorch, as Figure 3 、 4 shown. After the window of SW-MSA is split, a shifted area is introduced in the last row and the last column of the shifted image. The attention mechanism that calculates the similarity pixel by pixel is not applicable. It is necessary to calculate the self-attention of the similarity of the areas of the same color in a window. For the attention calculation of the similarity of different color areas, relative position encoding and mask can be used to make the attention between them tend to 0.

[0076] In the step S40, the activation function of the crowd counting model adopts the Swish function.

[0077] The formula of the Swish function is: f(x) = x × sigmoid(x).

[0078] In the step S40, the loss function of the crowd counting model adopts the MSE loss function plus the MSSIM loss function:

[0079] L = L MSE + L MSSIM ;

[0080] where, L represents the total loss of the crowd counting model; L MSE represents the MSE loss; L MSSIM represents the MSSIM loss.

[0081]

[0082] L MSSIM = 1 - MSSIM(G, P);

[0083] where, G(i, j) represents the labeled crowd density map, and P(i, j) represents the predicted crowd density map;

[0084] The present invention uses SSIM (structural similarity) to judge the similarity between two images, and the value range is [0, 1]. The higher the value of SSIM, the more similar the two images are. The formula is as follows:

[0085]

[0086] where, means that G has N pixel points, and the pixel value of each pixel point is G i , μ G is the average brightness of this image; Represents the contrast of an image, that is, the degree of intensity change in the image, which is the standard deviation of pixel values; Represents the covariance of two images; C1 = (K1L) 2 , where K1 is a constant much smaller than 1, preferably taking the value of 0.01, L is the grayscale dynamic range, which is determined by the data type of the image. If the data type is uint8, then L = 255; C2 = (K2L) 2 , where K2 is preferably taken as 0.03; SSIM(G, P) can compare the similarity of two images in terms of brightness, contrast, and structure.

[0087] In specific implementation, instead of calculating the mean, variance, and covariance of the entire image to calculate SSIM, the image is divided into blocks, and the SSIM of each window is calculated by sliding the window (stride = 1) and then averaged; when actually calculating the mean and variance of each window, a Gaussian convolution kernel with a variance of 1.5 is used for weighted averaging, and the sliding window size is 20×20.

[0088] If pixel G i corresponds to the Gaussian kernel weight ω i , then the formulas for the weighted mean, variance, and covariance of a window are as follows:

[0089]

[0090]

[0091]

[0092] Suppose there are M windows in the entire image, then the formula for MSSIM is:

[0093]

[0094] Since the loss based on MSE is not sufficient to express the intuitive perception of the human visual system for pictures. For example, sometimes two images only differ in brightness, but the MSE Loss between the two images is very different, while the MSE Loss of a very blurred image and a very clear image may be relatively small. Therefore, MSSIMLoss is added while using MES Loss.

[0095] The specific content of step S50 is as follows:

[0096] Obtain the video captured by the camera, extract the image frames of the video in units of 1 second, input each of the image frames into the trained crowd counting model to obtain a predicted crowd density map, perform integral summation on the predicted crowd density map to count the number of people in the image frame, and determine whether the number of people exceeds a set threshold. If so, generate an alarm for crowd gathering; if not, continue to count the number of people in the next image frame.

[0097] In summary, the advantages of the present invention are as follows:

[0098] 1. Mark each person's head in each crowd image through the coordinate points of a pixel, convert the marked crowd images into a crowd density map and perform preprocessing to obtain a training dataset, and then use the training dataset to train the crowd counting model created based on the transformer. Finally, the trained crowd counting model can be used for crowd counting; since coordinate points are used for marking, the limitation of detecting the smallest target by traditional object detectors is overcome, and there is no need to mark the bounding box for each person, which is applicable to the situation where pedestrian targets are small, the crowd density is high, and the occlusion between people is serious. Ultimately, the accuracy and efficiency of crowd gathering recognition are greatly improved.

[0099] 2. By adopting the Swin Transformer Block in the crowd counting model to control the calculation area of self-attention within a window-based area, the calculation amount of the crowd counting model is greatly reduced, and the efficiency of crowd gathering recognition is further improved.

[0100] 3. By using the crowd counting model created based on the transformer to replace the traditional convolutional neural network (CNN), the disadvantage of insufficient global information feature extraction by CNN is overcome, and thus the accuracy of crowd gathering recognition is greatly improved.

[0101] 4. By replacing the activation function of the crowd counting model from the traditional ReLU function with the Swish function, and the Swish function has the characteristics of no upper bound, having a lower bound, being smooth, and non-monotonic, thus greatly improving the feature extraction effect of the deep network in the crowd counting model.

[0102] 5. By performing preprocessing on the crowd density map through random cropping and flipping operations, that is, data augmentation, and then using the augmented data to train the crowd counting model, the generalization ability and robustness of the crowd counting model are greatly improved.

[0103] 6. By adding the MSSIM loss function to the loss function of the crowd counting model, which is used to compare the similarity between two images in three dimensions of brightness, contrast, and structure, it can better reflect the judgment of the human visual system on the similarity between two images.

[0104] Although the specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments we described are illustrative only and not used to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered by the scope protected by the claims of the present invention.

Claims

1. A method for crowd gathering warning based on transformer technology, characterized in that: It includes the following steps: Step S10: Obtain a large number of crowd images, and label each person's head in each of the crowd images with the coordinate points of a pixel; Step S20: Use the Gaussian kernel function to replace the pixel values corresponding to the labeled coordinate points in each of the crowd images with the weighted average of the pixel values corresponding to the surrounding coordinate points, and then convert them into crowd density maps; The expression function corresponding to the crowd density map is: Among them, F(x) represents the crowd density map; represents the crowd image with the coordinate points of N labeled human heads; represents the Gaussian kernel function; σ i represents the human head size coefficient; β represents the Gaussian parameter; represents the coordinate point x of the human head i to the average distance of k adjacent human heads around, and δ() represents the Dirac function, and Step S30: Perform random cropping and flipping operations on each of the crowd density maps to expand the data volume, and then uniformly set the resolution of each of the crowd density maps to 256×256 to complete the preprocessing of each of the crowd density maps to obtain a training dataset; Step S40: Create a crowd counting model based on the transformer, and use the training dataset to train the crowd counting model; The crowd counting model consists of an encoder, a bottleneck layer, and a decoder; the bottleneck layer consists of two consecutive Swin Transformer Blocks; the encoder and the decoder adopt a skip connection method; The encoder is used to perform downsampling operations on the image, the bottleneck layer is used to extract the features of the image, and the decoder is used to perform upsampling operations on the image; The Swin Transformer Block consists of a window multi-head self-attention layer and a shifted window multi-head self-attention layer, and the two adjacent Swin Transformer Blocks alternately use the window multi-head self-attention layer and the shifted window multi-head self-attention layer; The activation function of the crowd counting model adopts the Swish function; The loss function of the crowd counting model adopts the MSE loss function plus the MSSIM loss function: L = L MSE + L MSSIM ; Among them, L represents the total loss of the crowd counting model; L MSE represents the MSE loss; L MSSIM represents the MSSIM loss; Step S50: Obtain the video collected by the camera, extract the image frames of the video in units of 1 second, input each of the image frames into the trained crowd counting model to obtain a predicted crowd density map, perform integral summation on the predicted crowd density map to count the number of people in the image frame, and determine whether the number of people exceeds the set threshold. If so, generate an alarm for crowd gathering; if not, continue to count the number of people in the next image frame.

Images