A method, system and product for counting attributes of crowd behavior

Abstract

The application relates to a crowd behavior attribute counting method, system and product, and belongs to the field of computer vision.The counting method comprises the following steps: acquiring a complex crowd image; constructing a multi-scale feature fusion attention mechanism crowd behavior attribute counting neural network trunk; training the multi-scale feature fusion attention mechanism crowd behavior attribute counting neural network trunk based on the complex crowd image; acquiring a complex crowd image to be counted; and inputting the complex crowd image to be counted into the trained counting neural network to obtain a counting result.The above scheme in the application is suitable for dividing the behavior attributes of large-scale high-density crowds and detecting the number of crowds, and effectively improves the accuracy of detecting the number of specific crowds.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method, system, and product for counting crowd behavior attributes. Background Technology

[0002] With the rapid pace of urbanization in China, crowd counting has received increasing attention due to its practical applications in public surveillance, traffic control, and ensuring public safety. Crowd behavior attribute counting, which categorizes people according to different behavioral attributes, can further expand the applications of crowd counting. For example, segmenting crowds at bus stops into "waiting crowds" and "passing crowds" can be used to determine the departure intervals of public transportation. Crowd behavior attribute counting can also be applied to private surveillance, allowing business owners to track the number of customers with different behavioral attributes in their stores to gain better business insights. Furthermore, in highly crowded scenarios, identifying individuals engaging in violence or robbery can help relevant personnel respond promptly to emergencies, ensuring the safety of people's lives and property.

[0003] Currently, there are two main methods for distinguishing crowd behavior attributes through semantic segmentation: 1) CRF is a graph model widely used in semantic segmentation to improve accuracy. It models the relationships between pixels using probabilistic inference, assuming similar pixels should have similar labels. However, this algorithm has high training costs and complexity. 2) GCN is a generalization of CNN to graph structures, used to complete semantic segmentation using contextual information. However, this method suffers from performance degradation when there are few labels. Therefore, both algorithms have certain limitations. OpenPose is a bottom-up detection algorithm with high detection accuracy. Based on the shortcomings of the above two algorithms, this invention proposes a crowd behavior attribute counting method based on a multi-scale feature fusion attention mechanism. It classifies the crowd using OpenPose, then calculates the number of people in each category, and finally summarizes the results. Summary of the Invention

[0004] The purpose of this invention is to provide a method, system, and product for counting the behavioral attributes of a population, thereby improving the accuracy of detecting the number of a specific population.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] In a first aspect, the present invention provides a method for counting crowd behavioral attributes, the counting method comprising:

[0007] Acquiring images of complex crowds;

[0008] Construct a counting neural network backbone for crowd behavior attributes based on a multi-scale feature fusion attention mechanism;

[0009] The backbone of the counting neural network for crowd behavior attributes based on the complex crowd image is trained using the multi-scale feature fusion attention mechanism.

[0010] Obtain images of complex crowds to be counted;

[0011] The complex crowd image to be counted is input into a trained counting neural network to obtain the counting result.

[0012] Optionally, the method may further include preprocessing the complex crowd image between the steps of "acquiring complex crowd images" and "constructing the backbone of a counting neural network for crowd behavior attributes based on a multi-scale feature fusion attention mechanism".

[0013] Optionally, preprocessing the complex crowd image specifically includes the following steps:

[0014] The complex crowd images are sorted and categorized, and a required number of images are randomly selected as the training set, with the remainder used as the test set.

[0015] The training set images are flipped, cropped, and mirrored to expand the dataset;

[0016] The expanded dataset is binarized, data augmented, and normalized to remove irrelevant information from the images.

[0017] Optionally, the backbone of the counting neural network for the crowd behavior attributes of the multi-scale feature fusion attention mechanism specifically includes:

[0018] The module includes VGG16, multi-branch feature extraction, segmentation map refinement, OpenPose network, and U-Net semantic segmentation.

[0019] The VGG16 module, multi-branch feature extraction module, refined segmentation map module, and OpenPose network module are connected in sequence; the U-Net semantic segmentation module is connected to the OpenPose network module.

[0020] Optionally, the multi-branch feature extraction module includes:

[0021] The system consists of a first convolutional layer, a first upsampling layer, a first attention module, a second convolutional layer, a second upsampling layer, a second attention module, a third convolutional layer, a third upsampling layer, a third attention module, a fourth convolutional layer, a fourth upsampling layer, a fourth attention module, a global receptive field module, and a fifth convolutional layer.

[0022] The first convolutional layer, the first upsampling layer, and the first attention module are connected in sequence;

[0023] The second convolutional layer, the second upsampling layer, and the second attention module are connected in sequence;

[0024] The third convolutional layer, the third upsampling layer, and the third attention module are connected in sequence;

[0025] The fourth convolutional layer, the fourth upsampling layer, and the fourth attention module are connected in sequence;

[0026] The first attention module, the second attention module, the third attention module, and the fourth attention module are all connected to the global receptive domain module;

[0027] The global receptive field module is also connected to the fifth convolutional layer.

[0028] Optionally, the kernel size of the first convolutional layer is 1, the number of input channels is 512, the number of output network layers is 128, the dilation coefficient is 1, the padding number is 0, and it is followed by an LRelu activation function.

[0029] The second convolutional layer has a kernel size of 3, 512 input channels, 128 output network layers, a dilation factor of 2, and 2 padding elements, followed by an LRelu activation function.

[0030] The third convolutional layer has a kernel size of 3, 512 input channels, 128 output network layers, a dilation factor of 3, and 3 padding elements, followed by an LRelu activation function.

[0031] The fourth convolutional layer has a kernel size of 3, 512 input channels, 128 output network layers, a dilation factor of 6, 6 padding elements, and is followed by an LRelu activation function.

[0032] Secondly, based on the method described above in this invention, this invention further provides a counting system for population behavior attributes, the counting system comprising:

[0033] The first image acquisition module is used to acquire images of complex crowds;

[0034] The counting neural network backbone construction module is used to construct the counting neural network backbone of crowd behavior attributes based on multi-scale feature fusion attention mechanism;

[0035] The training module is used to train the backbone of the counting neural network for crowd behavior attributes based on the complex crowd image;

[0036] The second image acquisition module is used to acquire images of complex crowds to be counted;

[0037] The prediction module is used to input the complex crowd image to be counted into a trained counting neural network to obtain the counting result.

[0038] Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to cause the electronic device to perform the above-described method for counting crowd behavior attributes.

[0039] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for counting crowd behavior attributes.

[0040] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0041] The above-mentioned method of the present invention improves the structure of the classic convolutional neural network by replacing the simple convolutional network layer with a multi-scale feature fusion attention mechanism module, and using the Adam optimizer to optimize the initial weight threshold of the neural network, thereby accelerating the convergence speed of the network, getting closer to the optimal parameters of the network, and enhancing the network's ability to extract different features.

[0042] In addition, based on the feature extraction of the first 10 layers of VGG16, this invention further enhances the feature information of different regions through a multi-scale feature fusion attention mechanism, improves the network's attention to dense crowds, overcomes the problem of detailed feature extraction in dense areas, and further refines density prediction through semantic segmentation branches to finally obtain the number of people in different behavioral groups. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of the method for counting human behavioral attributes according to the present invention;

[0045] Figure 2 This is a schematic diagram of the backbone of the counting neural network of the present invention;

[0046] Figure 3 This is a schematic diagram of the structure of the counting system for human behavior attributes of the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] The purpose of this invention is to provide a method, system, and product for counting the behavioral attributes of a population, thereby improving the accuracy of detecting the number of a specific population.

[0049] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] Figure 1 This is a flowchart of the method for counting human behavioral attributes according to the present invention, as shown below. Figure 1 As shown, the method in this invention includes:

[0051] S1: Obtain images of complex crowds.

[0052] Specifically, the following steps are included:

[0053] S1.1: Obtain multiple publicly available datasets in the field of computer vision from the internet.

[0054] S1.2: Organize and classify each dataset separately, and randomly select the required number of images as the training set, and use the rest as the test set.

[0055] S1.3: Flip, crop, and mirror the training set images to augment the dataset.

[0056] S1.4: Perform binarization, data augmentation, normalization preprocessing on the processed dataset and remove irrelevant information from the image.

[0057] S2: Construct the backbone of a counting neural network for crowd behavior attributes using a multi-scale feature fusion attention mechanism.

[0058] The main components of this counting neural network include:

[0059] The module includes VGG16, multi-branch feature extraction, segmentation map refinement, OpenPose network, and U-Net semantic segmentation.

[0060] The VGG16 module, multi-branch feature extraction module, refined segmentation map module, and OpenPose network module are connected in sequence; the U-Net semantic segmentation module is connected to the OpenPose network module.

[0061] Specifically, in this invention, the first 10 layers of VGG16 are selected as feature extraction layers, and the kernel size is 3.

[0062] Specifically, the multi-branch feature extraction module includes:

[0063] The system consists of a first convolutional layer, a first upsampling layer, a first attention module, a second convolutional layer, a second upsampling layer, a second attention module, a third convolutional layer, a third upsampling layer, a third attention module, a fourth convolutional layer, a fourth upsampling layer, a fourth attention module, a global receptive field module, and a fifth convolutional layer.

[0064] The first convolutional layer, the first upsampling layer, and the first attention module are connected in sequence;

[0065] The second convolutional layer, the second upsampling layer, and the second attention module are connected in sequence;

[0066] The third convolutional layer, the third upsampling layer, and the third attention module are connected in sequence;

[0067] The fourth convolutional layer, the fourth upsampling layer, and the fourth attention module are connected sequentially.

[0068] The first attention module, the second attention module, the third attention module, and the fourth attention module are all connected to the global receptive domain module;

[0069] The global receptive field module is also connected to the fifth convolutional layer.

[0070] The first convolutional layer has a kernel size of 1, 512 input channels, 128 output network layers, a dilation coefficient of 1, and 0 padding. It is followed by an LRelu activation function, nearest neighbor interpolation upsampling, and the upsampling factor is twice that of the original image.

[0071] The second convolutional layer has a kernel size of 3, 512 input channels, 128 output network layers, a dilation coefficient of 2, and 2 padding elements. It is followed by an LRelu activation function, nearest neighbor interpolation upsampling, and the upsampling factor is twice that of the original image.

[0072] The third convolutional layer has a kernel size of 3, 512 input channels, 128 output network layers, a dilation coefficient of 3, and 3 padding elements. It is followed by an LRelu activation function, nearest neighbor interpolation upsampling, and the upsampling factor is twice that of the original image.

[0073] The fourth convolutional layer has a kernel size of 3, 512 input channels, 128 output network layers, a dilation coefficient of 6, and 6 padding elements. It is followed by an LRelu activation function, nearest neighbor interpolation upsampling, and the upsampling factor is twice that of the original image.

[0074] Finally, after connecting the above modules, a fifth convolutional layer with a kernel size of 1 is passed through it, followed by an LRelu function to output the predicted density map.

[0075] Specifically, the enhanced attention module (AT) passes the four branches of features through a Global Average Pooling process, outputting 1×1×C data. This data is then processed through two fully connected layers, and finally, the Sigmoid function is used to constrain the result to the range [0, 1]. This value is used as the scale and multiplied by the C channels of the first fully connected layer of the input image, serving as the input data for the next stage. The principle behind this structure is to enhance important features and weaken less important features by controlling the scale, thereby making the extracted features more targeted. The first fully connected layer compresses the C channels into C / r channels, reducing computational cost. The compression ratio r is set to 16, and the LRelu function is used to improve computational speed and accelerate network training. The second fully connected layer restores the C channels and then uses the Sigmoid function to constrain the result to the range [0, 1].

[0076] Specifically, the Global Receptive Field (GRF) module first passes the input features through a max-pooling layer (kernel size 3, stride 2) and an average-pooling layer (Avgpool, kernel size 3, stride 2). Then, the output features of the two modules are concatenated, and the result is constrained to the range [0, 1] using the sigmoid function. This branch with global receptive field information is then applied to the other four multi-scale branches to assist in multi-scale feature learning.

[0077] Here, a weight parameter is added to each feature map in this way to improve accuracy.

[0078] Specifically, the refinement image segmentation module is generated by stitching together the overall density map generated by the predicted density map branch with the image semantically segmented by the U-Net network. The refined segmentation image is then input into the OpenPose network module.

[0079] In this invention, the semantic segmentation branch extracts the first 10 layers of U-Net, and an LRelu activation function is added after each convolutional layer. The kernel size is set to 5, and the number of channels per layer is 16. MaxPooling (kernel = 2).

[0080] The OpenPose network module is a widely recognized tool for pose recognition. OpenPose is an open-source library based on convolutional neural networks and supervised learning, written using the Caffe framework. It can track facial expressions, torso, limbs, and even fingers, and is suitable for both single and multi-person applications, while exhibiting good robustness. It can be considered the world's first real-time multi-person 2D pose estimation based on deep learning, a milestone in human-computer interaction, providing a high-quality information dimension for machines to understand humans.

[0081] S3: Train the backbone of the counting neural network for crowd behavior attributes based on the complex crowd image and the multi-scale feature fusion attention mechanism.

[0082] Specifically, it includes:

[0083] S3.1 Loss function and parameter settings used during training: The loss function is MSE (mean squared error), the Adam optimizer is used, the batch size is set to 5, the learning rate is 0.00001, and the epoch is set to 3000.

[0084] S3.2: Input the Gaussianized graph into the neural network for training.

[0085] S3.4: Load the trained network parameters and test the evaluation functions mae and mse using the test set to estimate the performance of the neural network.

[0086] S4: Obtain the complex crowd image to be counted.

[0087] Specifically, the crowd image is processed to a size smaller than 1024×1024 pixels to obtain the processed image.

[0088] S5: Input the complex crowd image to be counted into the trained counting neural network to obtain the counting result.

[0089] Specifically, the processed image is input into the tested network to obtain the predicted number of people, y.

[0090] The specific processing details are described below:

[0091] The image to be predicted is input into VGG16 for shallow feature extraction. The feature extraction results are then input into four branches: the first, second, third, and fourth convolutional layers. The outputs of these layers are then sequentially input into the first, second, third, and fourth upsampling layers, with each upsampling layer being twice the size of the original image. The outputs of these upsampling layers are then sequentially input into the first, second, third, and fourth attention modules. Finally, the outputs of these attention modules are multiplied sequentially by the global receptive field module, and then fused through the fifth convolutional layer to obtain the overall density map.

[0092] The image to be predicted is input into the U-Net semantic segmentation module to obtain the segmented image.

[0093] The overall density map and the segmented image are stitched together to obtain a refined segmentation map.

[0094] The refined segmentation map is input into the OpenPose network module for pose estimation to obtain the final semantic segmentation map.

[0095] The output of the prediction density map module is concatenated with the output of the final semantic segmentation map module to generate the final refined density map. The final population behavior attribute density map is obtained by multiplying the elements in the semantic segmentation map with the elements in the refined density map one by one.

[0096] Figure 3 This is a schematic diagram of the structure of the counting system for human behavior attributes of the present invention, as shown below. Figure 3 As shown, the system in this invention includes:

[0097] The first image acquisition module 201 is used to acquire images of complex crowds;

[0098] The counting neural network backbone construction module 202 is used to construct the counting neural network backbone of the crowd behavior attributes based on the multi-scale feature fusion attention mechanism;

[0099] Training module 203 is used to train the backbone of the counting neural network for crowd behavior attributes of the multi-scale feature fusion attention mechanism based on the complex crowd image;

[0100] The second image acquisition module 204 is used to acquire images of complex crowds to be counted;

[0101] The prediction module 205 is used to input the complex crowd image to be counted into the trained counting neural network to obtain the counting result.

[0102] In addition, the present invention also provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the above-described method for counting crowd behavior attributes.

[0103] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for counting crowd behavior attributes.

[0104] In summary, this invention can be used to count the number of people with potential different behaviors among different groups of people.

[0105] This invention improves the structure of classic convolutional neural networks by replacing simple convolutional network layers with a custom attention module through a feature extraction module. It also uses the Adam optimizer to optimize the initial weight thresholds of the neural network, thereby accelerating the network's convergence speed, bringing it closer to the network's optimal parameters, and enhancing the network's ability to extract different features.

[0106] Based on the feature extraction of the first 10 layers of VGG16, this invention further enhances the feature information of different regions by using a multi-scale feature fusion attention module, thereby improving the network's attention to dense crowds.

[0107] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0108] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for counting behavioral attributes of a population, characterized in that, The counting method includes: Acquiring images of complex crowds; Construct a counting neural network backbone for crowd behavior attributes based on a multi-scale feature fusion attention mechanism; The backbone of the counting neural network for crowd behavior attributes based on the complex crowd image is trained using the multi-scale feature fusion attention mechanism. Obtain images of complex crowds to be counted; The complex crowd image to be counted is input into a trained counting neural network to obtain the counting result; The core of the counting neural network for the crowd behavior attributes of the multi-scale feature fusion attention mechanism specifically includes: The module includes VGG16, multi-branch feature extraction, segmentation map refinement, OpenPose network, and U-Net semantic segmentation. The VGG16 module, multi-branch feature extraction module, refined segmentation map module, and OpenPose network module are connected in sequence; the U-Net semantic segmentation module is connected to the OpenPose network module. The multi-branch feature extraction module includes: The system consists of a first convolutional layer, a first upsampling layer, a first attention module, a second convolutional layer, a second upsampling layer, a second attention module, a third convolutional layer, a third upsampling layer, a third attention module, a fourth convolutional layer, a fourth upsampling layer, a fourth attention module, a global receptive field module, and a fifth convolutional layer. The first convolutional layer, the first upsampling layer, and the first attention module are connected in sequence; The second convolutional layer, the second upsampling layer, and the second attention module are connected in sequence; The third convolutional layer, the third upsampling layer, and the third attention module are connected in sequence; The fourth convolutional layer, the fourth upsampling layer, and the fourth attention module are connected in sequence. The first attention module, the second attention module, the third attention module, and the fourth attention module are all connected to the global receptive domain module; The global receptive field module is also connected to the fifth convolutional layer.

2. The method for counting crowd behavior attributes according to claim 1, characterized in that, The method further includes, between the steps of "acquiring complex crowd images" and "constructing the backbone of a counting neural network for crowd behavior attributes based on a multi-scale feature fusion attention mechanism", the following: preprocessing the complex crowd images.

3. The method for counting crowd behavior attributes according to claim 2, characterized in that, The preprocessing of the complex crowd image specifically includes the following steps: The complex crowd images are sorted and categorized, and a required number of images are randomly selected as the training set, with the remainder used as the test set. The training set images are flipped, cropped, and mirrored to expand the dataset; The expanded dataset is binarized, data augmented, and normalized to remove irrelevant information from the images.

4. The method for counting crowd behavior attributes according to claim 1, characterized in that, The first convolutional layer has a kernel size of 1, 512 input channels, 128 output network layers, a dilation coefficient of 1, and 0 padding, followed by an LRelu activation function. The second convolutional layer has a kernel size of 3, 512 input channels, 128 output network layers, a dilation factor of 2, and 2 padding elements, followed by an LRelu activation function. The third convolutional layer has a kernel size of 3, 512 input channels, 128 output network layers, a dilation factor of 3, and 3 padding elements, followed by an LRelu activation function. The fourth convolutional layer has a kernel size of 3, 512 input channels, 128 output network layers, a dilation factor of 6, 6 padding elements, and is followed by an LRelu activation function.

5. The method for counting crowd behavior attributes according to claim 1, characterized in that, When training the counting neural network backbone of the crowd behavior attributes of the multi-scale feature fusion attention mechanism, the loss function used is MSE mean squared error, the Adam optimizer is used, the batch size is set to 5, the learning rate is 0.00001, and the epoch is set to 3000.

6. A counting system for crowd behavior attributes, characterized in that, The counting system includes: The first image acquisition module is used to acquire images of complex crowds; The counting neural network backbone construction module is used to construct the counting neural network backbone of crowd behavior attributes based on multi-scale feature fusion attention mechanism; The training module is used to train the backbone of the counting neural network for crowd behavior attributes based on the complex crowd image; The second image acquisition module is used to acquire images of complex crowds to be counted; The prediction module is used to input the complex crowd image to be counted into a trained counting neural network to obtain the counting result; The core of the counting neural network for the crowd behavior attributes of the multi-scale feature fusion attention mechanism specifically includes: The module includes VGG16, multi-branch feature extraction, segmentation map refinement, OpenPose network, and U-Net semantic segmentation. The VGG16 module, multi-branch feature extraction module, refined segmentation map module, and OpenPose network module are connected in sequence; the U-Net semantic segmentation module is connected to the OpenPose network module. The multi-branch feature extraction module includes: The system consists of a first convolutional layer, a first upsampling layer, a first attention module, a second convolutional layer, a second upsampling layer, a second attention module, a third convolutional layer, a third upsampling layer, a third attention module, a fourth convolutional layer, a fourth upsampling layer, a fourth attention module, a global receptive field module, and a fifth convolutional layer. The first convolutional layer, the first upsampling layer, and the first attention module are connected in sequence; The second convolutional layer, the second upsampling layer, and the second attention module are connected in sequence; The third convolutional layer, the third upsampling layer, and the third attention module are connected in sequence; The fourth convolutional layer, the fourth upsampling layer, and the fourth attention module are connected in sequence. The first attention module, the second attention module, the third attention module, and the fourth attention module are all connected to the global receptive domain module; The global receptive field module is also connected to the fifth convolutional layer.

7. An electronic device, characterized in that, It includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the counting method of crowd behavior attributes as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method for counting crowd behavior attributes as described in any one of claims 1-5.

Images