Crowd density statistics method, device, equipment and medium based on key point detection

Abstract

An embodiment of the present invention discloses a method, device, equipment and medium for crowd density statistics based on key point detection. The method includes: obtaining image data; processing the image data to obtain a sample map; inputting the sample map into a crowd density statistics model for head probability prediction to obtain a head probability map; and calculating the number of people according to the head probability map. The present invention not only relies on head key points, but also relies on key points of other parts of the human body for recognition, increases the model supervision information to improve the representation ability of the human body density feature map, accelerates the model training convergence speed, facilitates crowd density prediction, and solves the problem of missing person statistics due to partial occlusion of human body parts. In addition, the correct rate of the model for crowd density map prediction is improved through multi-feature fusion and attention mechanism.

Description

Technical Field

[0001] The present invention relates to the technical field of data statistics, and more specifically to a crowd density statistics method, device, equipment and medium based on key point detection. Background Art

[0002] With the development of society, crowd counting or statistics is a research hotspot and difficulty in the current industrial and academic fields, and it has important application value in real life. There are the following several methods for crowd density statistics currently.

[0003] The first one is to perform feature fusion of information with different receptive fields through dilated convolution and original convolution, as well as feature fusion of information with different receptive fields, and fuse different hierarchical semantic information of feature maps at different resolutions, so as to generate a crowd density map with higher quality. This patent uses a large number of dilated convolutions, which results in loss of continuous information in the picture features, has a great impact on density statistics, and ignores the spatial information that the crowd density is larger near and smaller far away.

[0004] The second one is to use the AlexNet network to divide the crowd picture dataset into two categories: density and sparsity, and send them into the corresponding feature extraction networks according to the differences in the density features of the two types of images. For density images, an attention mechanism method is used for personnel density statistics, and for sparse crowd density, a dilated convolution method is used for personnel density statistics. This patent first needs to analyze the personnel density of the picture and needs to judge whether the personnel density in the picture is sparse or dense, and then conduct density analysis on it. This patent cannot directly judge the crowd density and needs to judge its density sparsity and density first before it can be judged, and the network is bloated and cannot adapt to different degrees of personnel density classification.

[0005] The third one is a method for crowd density and quantity estimation based on a convolutional neural network. This method only uses a convolutional neural network without using a multi-scale method, cannot be compatible with the feature information of targets of different scales, has a poor model prediction effect, and the recognition result is inaccurate.

[0006] The fourth one is to detect the number of human heads through an object detection network and judge the crowd density according to the number of human heads. This method of judging the crowd density based on object detection of human heads will have missed detections for human body occlusion and human head occlusion, and at the same time, the detection recall rate for distant dense small targets is low, which will ultimately lead to inaccurate crowd density evaluation. Summary of the Invention

[0007] The purpose of the present invention is to overcome the deficiencies of the prior art, and provide a crowd density statistics method, device, equipment and medium based on key point detection, which can effectively solve the problem of missing person count caused by partial occlusion of human body parts, and has a higher correct rate of crowd density prediction.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] In a first aspect, a crowd density statistics method based on key point detection includes:

[0010] Obtain image data;

[0011] Process the image data to obtain a sample map;

[0012] Input the sample map into a crowd density statistics model for head probability prediction to obtain a head probability map;

[0013] Calculate the number of people based on the head probability map.

[0014] A further technical solution thereof is: when inputting the sample map into the crowd density statistics model for head probability prediction to obtain a head probability map, the processing method of the crowd density statistics model includes:

[0015] Input the sample map into a stacked hourglass network to obtain a first processed feature and a second processed feature;

[0016] The first processed feature and the second processed feature are associated through the key points in open pose to obtain human key point features;

[0017] Upsample the first processed feature to obtain an upsampled feature;

[0018] Perform convolution processing on the second processed feature to obtain a first convolution feature;

[0019] Concatenate and merge the upsampled feature and the convolution feature to obtain a first merged feature;

[0020] Process the merged feature through the attention mechanism of CBAM to obtain an attention feature;

[0021] Perform self-calibrated convolution processing on the human key point features to obtain self-supervised features;

[0022] Perform convolution processing on the attention feature to obtain a second convolution feature;

[0023] Multiply and calculate the second convolution feature and the self-supervised feature to obtain a third processed feature;

[0024] Perform dilated convolution processing on the sample map to obtain a fourth processed feature;

[0025] Perform Channel-wise Cross Attentation processing on the fourth processing feature to obtain the fifth processing feature;

[0026] Concatenate the fifth processing feature and the third processing feature to obtain the second concatenated feature;

[0027] Input the second concatenated feature into the upsampling combination module for upsampling processing to obtain the sixth processing feature, the seventh processing feature, and the eighth processing feature;

[0028] Normalize the eighth processing feature through the sigmoid function to obtain the crowd density map;

[0029] Set each pixel value of the crowd density map to be between 0 and 1 to obtain the head probability map.

[0030] Its further technical solution is: The self-calibrated convolution processing of the human key point features to obtain the self-supervised features includes:

[0031] Slice the human key point features through the channels of self-calibrated convolution into the first sliced feature and the second sliced feature;

[0032] Perform average pooling processing on the first sliced feature to obtain the pooled feature;

[0033] Perform convolution and upsampling processing on the pooled feature to obtain the ninth processing feature;

[0034] Input the calculation result obtained by adding the ninth processing feature and the first sliced feature into the sigmoid activation function processing to obtain the tenth processing feature;

[0035] Multiply the calculation result of the convolution processing of the tenth processing feature and the first sliced feature to obtain the eleventh processing feature;

[0036] Concatenate the results obtained by performing convolution processing on the eleventh processing feature and the second sliced feature respectively to obtain the self-supervised feature.

[0037] Its further technical solution is: The input of the second concatenated feature into the upsampling combination module for upsampling processing to obtain the sixth processing feature, the seventh processing feature, and the eighth processing feature, and the upsampling combination module is respectively composed of a transposed convolution, a relu activation function, a dilated convolution, and an unpooling.

[0038] Its further technical solution is: The number of feature channels of the sixth processing feature, the seventh processing feature, and the eighth processing feature is all 1.

[0039] Its further technical solution is that the sizes of the sixth processing feature, the seventh processing feature, and the eighth processing feature are enlarged to the size of the sample image through a difference.

[0040] Its further technical solution is that calculating the number of people according to the head probability map includes:

[0041] Accumulating and summing the probability densities of each pixel on the head probability map and then taking the integer calculation to obtain the number of people in the image.

[0042] In a second aspect, a crowd density statistics device based on key point detection includes an acquisition unit, a processing unit, a prediction unit, and a calculation unit;

[0043] The acquisition unit is used to acquire image data;

[0044] The processing unit is used to process the image data to obtain a sample image;

[0045] The prediction unit is used to input the sample image into a crowd density statistics model for head probability prediction to obtain a head probability map;

[0046] The calculation unit is used to calculate the number of people according to the head probability map.

[0047] In a third aspect, a computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the crowd density statistics steps based on key point detection as described above.

[0048] In a fourth aspect, a computer-readable storage medium stores a computer program, and the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the crowd density statistics steps based on key point detection as described above.

[0049] The beneficial effects of the present invention compared with the prior art are as follows: The present invention not only relies on head key points but also relies on key points of other parts of the human body for recognition, increases the model supervision information to improve the representation ability of the human density feature map, accelerates the model training convergence speed, facilitates crowd density prediction, and solves the problem of missing person counting due to partial occlusion of human body parts. In addition, the prediction accuracy of the model for the crowd density map is improved through multi-feature fusion and attention mechanism.

[0050] The above description is only an overview of the technical solution of the present invention. In order to understand the technical means of the present invention more clearly, it can be implemented according to the content of the specification. And in order to make the above and other objects, features and advantages of the present invention more obvious and understandable, the following preferred embodiments are specifically described in detail as follows. Brief Description of the Drawings

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0052] Figure 1 Schematic diagram of the application scenario of the crowd density statistics method based on key point detection provided by a specific embodiment of the present invention;

[0053] Figure 2 Flowchart of the crowd density statistics method based on key point detection provided by a specific embodiment of the present invention;

[0054] Figure 3 Schematic block diagram of the crowd density statistics device based on key point detection provided by a specific embodiment of the present invention;

[0055] Figure 4 Schematic block diagram of a computer device provided by a specific embodiment of the present invention. Detailed Description of the Embodiments

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0057] It should be understood that when used in this specification and the appended claims, the terms "comprises" and "comprising" indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their groups.

[0058] It should also be understood that the terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0059] It should also be further understood that the term "and / or" used in the specification and appended claims of the present invention refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0060] Please refer to Figure 1 and Figure 2 , Figure 1 which is a schematic diagram of the application scenario of the crowd density statistics method based on key point detection provided by an embodiment of the present invention; Figure 2 which is a flowchart of the crowd density statistics method based on key point detection provided by a specific embodiment of the present invention. The crowd density statistics method based on key point detection is applied to a server and is executed by an application software installed in the server.

[0061] As Figure 2 shown, the crowd density statistics method based on key point detection includes the following steps S10 - S40.

[0062] S10. Obtain image data.

[0063] In this embodiment, video data (i.e., image data) of the crowd in the subway car is collected through a monitoring device in the subway car. The monitoring device can be a common one on the market, and the present application does not make any limitations in this regard. In order to collect the video data of each car, monitoring devices can be installed in each car, and the video data collected by the monitoring devices installed in each car can be aggregated into the subway data background through wired or wireless means. By accessing the data background, the video data situation of each car can be queried.

[0064] S20. Process the image data to obtain a sample image.

[0065] In one embodiment, step S20 specifically includes the following steps: S201 - S202.

[0066] S201. Segment the image data to obtain segmented image data.

[0067] In this embodiment, since the subway needs to stop at different stations, the situation of people getting on or off after each car stops at each station will basically change. Therefore, the image data can be segmented according to the way of each car corresponding to each station, and the segmented image data corresponding to each car at each station can be obtained.

[0068] S202. Select a frame of picture from the segmented image data as the sample image.

[0069] In this embodiment, since the segmented image data includes multiple frames of pictures, a frame of picture can be selected from the segmented image data as the sample image Iimage Perform head probability prediction.

[0070] S30. Input the sample image into the crowd density statistical model to perform head probability prediction, so as to obtain a head probability map.

[0071] In one embodiment, the processing method of the crowd density statistical model specifically includes the following steps: S301 - S315.

[0072] S301. Input the sample image into the stacked hourglass network to obtain a first processed feature and a second processed feature.

[0073] In this embodiment, by inputting Figure I image into the stacked hourglass network for processing, feature F h1 and feature F h2 .

[0074] The stacked hourglass network is composed of multiple hourglass networks connected in series. Compared with a single network, the connected stacked hourglass network mainly reuses the whole - body joint information to improve the recognition accuracy of individual joints.

[0075] S302. The first processed feature and the second processed feature are associated through the key points in open pose to obtain human key - point features.

[0076] In this embodiment, feature F h1 and feature F h2 obtain PAF (Part Affinity Fields) through the key points in open pose, and here the human key - point features can be directly predicted.

[0077] Open pose is a bottom - up algorithm, an open - source library developed based on convolutional neural network and supervised learning with Caffe as the framework. It can achieve pose estimation of human actions, facial expressions, finger movements, etc. It is applicable to single - person and multi - person scenarios and has excellent robustness.

[0078] S303. Upsample the first processed feature to obtain an upsampled feature.

[0079] In this embodiment, upsample feature F h1 to obtain feature F u1 .

[0080] S304. Convolve the second processed feature to obtain a first convolutional feature.

[0081] In this embodiment, convolve feature F h2 to obtain feature Fu2 。

[0082] S305. Concatenate the upsampled feature and the convolutional feature to obtain a first concatenated feature.

[0083] In this embodiment, the feature F u1 and the feature F u2 are concatenated to obtain the feature F c 。

[0084] S306. Process the concatenated feature through the attention mechanism of CBAM to obtain an attention feature.

[0085] In this embodiment, the feature F c is processed through the attention mechanism of CBAM to obtain the feature F cm 。

[0086] S307. Perform self-calibrated convolution processing on the human key point feature to obtain a self-supervised feature.

[0087] In this embodiment, the PAF is processed through the self-supervised mechanism self-calibrated convolution (self-correction convolution) to obtain the feature F cc 。

[0088] The advantages of self-correction convolution are reflected in three aspects:

[0089] First, each spatial position is allowed to not only adaptively regard the information environment around it as an embedding from the latent space, as a scalar in the response from the original scale space, but also model the inter-channel dependencies. Therefore, the field of view of the convolutional layer with self-calibration can be effectively expanded.

[0090] Second, the self-calibration operation does not collect global context, but only considers the context around each spatial position, thus avoiding some contaminated information from irrelevant regions to a certain extent.

[0091] Third, the self-calibration operation encodes multi-scale information, which is urgently needed for tasks related to object detection.

[0092] In one embodiment, step S307 specifically includes the following steps: S3071 - S3076.

[0093] S3071. Split the human key point feature through the channels of self-calibrated convolution into a first split feature and a second split feature.

[0094] In this embodiment, self - calibrated convolution splits the PAF into feature X1 and feature X2 through channels.

[0095] S3072. Perform average pooling on the first split feature to obtain a pooled feature.

[0096] In this embodiment, average pooling is performed on X1.

[0097] S3073. Perform convolution and upsampling on the pooled feature to obtain a ninth processed feature.

[0098] In this embodiment, after performing average pooling on X1, convolution and upsampling are performed to obtain feature X'1.

[0099] S3074. Add the ninth processed feature and the first split feature and input the calculation result into the sigmoid activation function for processing to obtain a tenth processed feature.

[0100] In this embodiment, the result obtained by adding feature X'1 and X1 is input into the sigmoid activation function.

[0101] S3075. Multiply the result of convolution processing of the tenth processed feature and the first split feature to obtain an eleventh processed feature.

[0102] In this embodiment, the result obtained by adding feature X'1 and X1 is input into the sigmoid activation function for processing, and the result is multiplied by the result of convolution of X1 to obtain Y'1.

[0103] S3076. Concatenate the results obtained by performing convolution processing on the eleventh processed feature and the second split feature respectively to obtain a self - supervised feature.

[0104] In this embodiment, convolution processing is respectively performed on Y'1 and X2, and the results of the two are concatenated to obtain feature F cc 。

[0105] S308. Perform convolution processing on the attention feature to obtain a second convolution feature.

[0106] In this embodiment, feature F cm is subjected to convolution processing.

[0107] S309. Multiply the second convolution feature and the self - supervised feature to obtain a third processed feature.

[0108] In this embodiment, the result feature F after convolution processing of feature F cm is...cc Multiply the features to obtain Feature F acm 。

[0109] S310. Perform dilated convolution on the sample image to obtain a fourth processed feature.

[0110] In this embodiment, perform dilated convolution on the sample image I image to obtain Feature F di 。

[0111] S311. Perform Channel-wise Cross Attentation on the fourth processed feature to obtain a fifth processed feature.

[0112] In this embodiment, perform Channel-wise Cross Attentation on Feature F di to obtain Feature F cwca 。

[0113] For the channel-wise attention mechanism, this attention mechanism learns the connection between each feature in the multi-layer 3D-feature map and the hidden state, that is, introducing attention in the CNN instead of simply using the output of the CNN part.

[0114] Based on the channel-wise attention mechanism, it can be regarded as a process of selecting relevant semantic features according to the context semantics.

[0115] The core of Channel-wise convolution lies in the sparsification of the input and output connections. Each output is only connected to a part of the input. Conceptually, it is different from grouped convolution. Instead of strictly distinguishing the input, it samples multiple relevant inputs with a certain stride for output (sliding in the channel dimension), which can reduce the number of parameters and ensure a certain degree of information flow between channels.

[0116] S312. Concatenate and merge the fifth processed feature with the third processed feature to obtain a second merged feature.

[0117] In this embodiment, concatenate and merge Feature F cwca with Feature F acm to obtain Feature F t 。

[0118] S313. Input the second merged feature into the upsampling combination module for upsampling to obtain a sixth processed feature, a seventh processed feature, and an eighth processed feature.

[0119] In this embodiment, the feature F t is input into the upsampling combination module for upsampling processing, and the features F uc1 , feature F uc2 and feature F uc3 are obtained respectively.

[0120] In this embodiment, the upsampling combination module is composed of a transposed convolution, a relu activation function, a dilated convolution, and an anti-pooling respectively. The number of feature channels of the features F uc1 , feature F uc2 and feature F uc3 is all 1. The sizes of the features F uc1 , feature F uc2 and feature F uc3 are enlarged to the size of the sample image I image by difference.

[0121] S314. Normalize the eighth processed feature through the sigmoid function to obtain the crowd density map.

[0122] In this embodiment, finally, the feature F uc3 is normalized through the sigmoid function to obtain the output crowd density map F out .

[0123] S315. Set each pixel value of the crowd density map to be between 0 and 1 to obtain the head probability map.

[0124] In this embodiment, by controlling each pixel value of the crowd density map F out to be between 0 and 1, the head probability map can be obtained.

[0125] S40. Calculate the number of people according to the head probability map.

[0126] In one embodiment, step S40 specifically includes the following steps:

[0127] S401. Accumulate and sum the probability densities of each pixel on the head probability map and then perform rounding calculation to obtain the number of people in the image.

[0128] In this embodiment, according to the head probability map, the number of heads in the image can be known. Summing up all the head numbers, the number of people in the image is obtained.

[0129] In addition, the loss functions used in the crowd density statistical model are the key point detection loss function, the crowd density segmentation function, and the total loss function. Among them, the key point detection loss Loss kpThe functions are the key point confidence heat map loss function and the key point close connection field loss function respectively. The crowd density segmentation function is the L2 loss function, and the L2 loss function is as follows:

[0130]

[0131] Among them, B is the batch size, represents the crowd density annotation map, represents the crowd density prediction map. For F uc2 ,F uc3 and F out calculate their respective loss functions with the true crowd density label respectively. This is done to better accelerate the model training speed and obtain Loss crowd density -1, Loss crowd density-2 and Loss crowd density-3 respectively. The total crowd density loss is Loss crowd density-total = Loss crowd density-1 + Loss crowd density-2 + Loss crowd density-3 . is the regularization term of the loss function.

[0132] The total loss function is:

[0133] Loss total = αLoss kp + βLoss crowd density ;

[0134] Among them, α is 0.2 and β is 0.8.

[0135] The crowd density annotation map of the crowd density statistical model is generated using the Gaussian method. Based on the key points of each head, a Gaussian probability distribution is generated as follows:

[0136]

[0137]

[0138] β is a hyperparameter of 0.3. Here, represents the average distance between the m heads closest to the i-th head and that head. δ is the impulse function, and the integral sum within the range is 1. is the Gaussian kernel function, and finally the crowd density annotation map is generated.

[0139] The present invention not only relies on head key points, but also relies on key points of other parts of the human body for recognition, increases the model supervision information to improve the representation ability of the human body density feature map, accelerates the model training convergence speed, facilitates crowd density prediction, and solves the problem of missing people counting due to partial occlusion of human body parts. In addition, the accuracy of the model for crowd density map prediction is improved through multi-feature fusion and attention mechanism.

[0140] Figure 3 FIG. 4 is a schematic block diagram of a crowd density statistics device 100 based on key point detection provided by an embodiment of the present invention. Corresponding to the above-mentioned crowd density statistics method based on key point detection, a specific embodiment of the present invention further provides a crowd density statistics device 100 based on key point detection. The crowd density statistics device 100 based on key point detection includes units and modules for executing the above-mentioned crowd density statistics method based on key point detection, and the device can be configured in a server.

[0141] As Figure 3 shown, the crowd density statistics device 100 based on key point detection includes an acquisition unit 110, a processing unit 120, a prediction unit 130, and a calculation unit 140.

[0142] The acquisition unit 110 is used to acquire image data.

[0143] In this embodiment, video data (i.e., image data) of the crowd in the subway carriage is collected through a monitoring device in the subway carriage. The monitoring device can be a common one on the market, and the present application does not make any limitation on this. In order to collect the video data of each carriage, a monitoring device can be installed in each carriage, and the video data collected by the monitoring device installed in each carriage can be summarized into the subway data background through wired or wireless means. The video data situation of each carriage can be queried by accessing the data background.

[0144] The processing unit 120 is used to process the image data to obtain a sample map.

[0145] In one embodiment, the processing unit 120 includes a splitting module and a selection module.

[0146] The splitting module is used to split the image data to obtain split image data.

[0147] In this embodiment, since the subway needs to stop at different stations, the situation of people getting on or off after each carriage stops at each station will basically change. Therefore, the image data can be split in the way of each carriage corresponding to each station, and the split image data of each carriage corresponding to each station can be obtained.

[0148] Selection module, configured to select one frame of picture from the segmented image data as a sample picture.

[0149] In this embodiment, since the segmented image data includes multiple frames of pictures, one frame of picture can be selected from the segmented image data as sample picture I image Perform head probability prediction.

[0150] Prediction unit 130, configured to input the sample picture into the crowd density statistical model to perform head probability prediction, so as to obtain a head probability map.

[0151] In one embodiment, the prediction unit 130 includes a first processing module, an association processing module, a second processing module, a third processing module, a first merging module, a fourth processing module, a fifth processing module, a sixth processing module, a first calculation module, a seventh processing module, an eighth processing module, a second merging module, a ninth processing module, a normalization processing module, and a setting module.

[0152] The first processing module is configured to input the sample picture into the stacked hourglass network to obtain a first processing feature and a second processing feature.

[0153] In this embodiment, by inputting Figure I image into the stacked hourglass network for processing, feature F h1 and feature F h2 .

[0154] The stacked hourglass network is composed of multiple hourglass networks connected in series. Compared with a single network, the connected stacked hourglass network mainly reuses the whole body joint information to improve the recognition accuracy of a single joint.

[0155] The association processing module is configured to associate the first processing feature and the second processing feature through the association between key points in open pose to obtain human key point features.

[0156] In this embodiment, feature F h1 and feature F h2 obtain PAF (Part Affinity Fields) through the key points in open pose, and here the human key point features can be directly predicted.

[0157] Open pose is a bottom-up algorithm, an open-source library developed based on convolutional neural network and supervised learning with Caffe as the framework. It can realize pose estimation of human actions, facial expressions, finger movements, etc. It is applicable to single and multiple people and has excellent robustness.

[0158] The second processing module is used to upsample the first processing feature to obtain an upsampled feature.

[0159] In this embodiment, the feature F h1 is upsampled to obtain the feature F u1 .

[0160] The third processing module is used to perform convolution processing on the second processing feature to obtain a first convolution feature.

[0161] In this embodiment, the feature F h2 is convolved to obtain the feature F u2 .

[0162] The first merging module is used to concatenate and merge the upsampled feature and the convolution feature to obtain a first merged feature.

[0163] In this embodiment, the feature F u1 is concatenated and merged with the feature F u2 to obtain the feature F c .

[0164] The fourth processing module is used to process the merged feature through the attention mechanism of CBAM to obtain an attention feature.

[0165] In this embodiment, the feature F c is processed through the attention mechanism of CBAM to obtain the feature F cm .

[0166] The fifth processing module is used to perform self-calibrated convolution processing on the human key point feature to obtain a self-supervised feature.

[0167] In this embodiment, the PAF is processed by the self-supervised mechanism self-calibrated convolution (self-correction convolution) to obtain the feature F cc .

[0168] The advantages of self-correction convolution are reflected in three aspects:

[0169] In the first aspect, it allows each spatial position to not only adaptively regard the surrounding information environment as an embedding from the latent space as a scalar in the response from the original scale space, but also model the inter-channel dependencies. Therefore, the field of view of the convolutional layer with self-calibration can be effectively expanded.

[0170] In the second aspect, the self-calibration operation does not collect global context, but only considers the context around each spatial position, thus avoiding some contaminated information from irrelevant regions to a certain extent.

[0171] In the third aspect, the self-calibration operation encodes multi-scale information, which is urgently needed for tasks related to object detection.

[0172] In one embodiment, the fifth processing module includes a slicing sub-module, a first processing sub-module, a second processing sub-module, a third processing sub-module, a fourth processing sub-module, and a fifth processing sub-module.

[0173] The slicing sub-module is used to slice the human key point features into a first sliced feature and a second sliced feature through the channels of self-calibrated convolution.

[0174] In this embodiment, self-calibrated convolution slices the PAF into feature X1 and feature X2 through channels.

[0175] The first processing sub-module is used to perform average pooling on the first sliced feature to obtain a pooled feature.

[0176] In this embodiment, average pooling is performed on X1.

[0177] The second processing sub-module is used to perform convolution and upsampling on the pooled feature to obtain a ninth processed feature.

[0178] In this embodiment, after performing average pooling on X1, convolution and upsampling are performed to obtain feature X'1.

[0179] The third processing sub-module is used to add the ninth processed feature and the first sliced feature, input the calculation result into the sigmoid activation function for processing, and obtain a tenth processed feature.

[0180] In this embodiment, the result obtained by adding feature X'1 and X1 is input into the sigmoid activation function.

[0181] The fourth processing sub-module is used to multiply the result of convolving the tenth processed feature and the first sliced feature to obtain an eleventh processed feature.

[0182] In this embodiment, the result obtained by adding feature X'1 and X1, input into the sigmoid activation function for processing, is multiplied by the result of convolving X1 to obtain Y'1.

[0183] The fifth processing sub-module is used to concatenate the results obtained by respectively performing convolution on the eleventh processed feature and the second sliced feature to obtain a self-supervised feature.

[0184] In this embodiment, convolution processing is respectively performed on Y'1 and X2, and the results of the two are combined by concatenation to obtain feature F. cc .

[0185] The sixth processing module is used to perform convolution processing on the attention feature to obtain a second convolution feature.

[0186] In this embodiment, feature F cm is subjected to convolution processing.

[0187] The first calculation module is used to multiply the second convolution feature by the self-supervised feature to obtain a third processed feature.

[0188] In this embodiment, the result feature F cm after convolution processing of feature F cc features are multiplied to obtain feature F acm .

[0189] The seventh processing module is used to perform dilated convolution processing on the sample image to obtain a fourth processed feature.

[0190] In this embodiment, dilated convolution processing is performed on sample image I image to obtain feature F di .

[0191] The eighth processing module is used to perform Channel-wise Cross Attentation processing on the fourth processed feature to obtain a fifth processed feature.

[0192] In this embodiment, Channel-wise Cross Attentation processing is performed on feature F di to obtain feature F cwca .

[0193] For the channel-wise attention mechanism, this attention mechanism learns the connection between each feature in the multi-layer 3D-feature map and the hidden state, that is, introducing attention in the CNN, rather than simply using the output of the CNN part.

[0194] Based on the channel-wise attention mechanism, it can be regarded as a process of selecting relevant semantic features according to the context semantics.

[0195] The core of channel-wise convolution lies in the sparsification of the input and output connections. Each output is only connected to a part of the input, which conceptually differs from grouped convolution. Instead of strictly distinguishing the input, it samples multiple relevant inputs with a certain stride for output (sliding in the channel dimension), which can reduce the number of parameters and ensure a certain degree of information flow between channels.

[0196] The second merging module is used to concatenate and merge the fifth processed feature and the third processed feature to obtain the second merged feature.

[0197] In this embodiment, the feature F cwca and the feature F acm are concatenated and merged to obtain the feature F t .

[0198] The ninth processing module is used to input the second merged feature into the upsampling combination module for upsampling processing to obtain the sixth processed feature, the seventh processed feature, and the eighth processed feature.

[0199] In this embodiment, the feature F t is input into the upsampling combination module for upsampling processing, and the features F uc1 , the feature F uc2 and the feature F uc3 are obtained respectively.

[0200] In this embodiment, the upsampling combination module is composed of a transposed convolution, a relu activation function, a dilated convolution, and an anti-pooling respectively. The number of feature channels of the features F uc1 , the feature F uc2 and the feature F uc3 is all 1. The sizes of the features F uc1 , the feature F uc2 and the feature F uc3 are enlarged to the size of the sample image I image through difference.

[0201] The normalization processing module is used to normalize the eighth processed feature through the sigmoid function to obtain the crowd density map.

[0202] In this embodiment, finally, the feature F uc3 is normalized through the sigmoid function to obtain the output crowd density map F out .

[0203] The setting module is used to set each pixel value of the crowd density map to be between 0 and 1 to obtain the head probability map.

[0204] In this embodiment, control the crowd density map F outBy setting each pixel value between 0 and 1, the head probability map can be obtained.

[0205] The calculation unit 140 is configured to calculate the number of people in the crowd based on the head probability map.

[0206] In one embodiment, the calculation unit 140 includes a summation operation module, which is used to sum up the probability density of each pixel on the head probability map and then perform rounding calculation to obtain the number of people in the figure.

[0207] In this embodiment, according to the head probability map, the number of heads in the figure can be known. By summing up all the head numbers, the number of people in the figure is obtained.

[0208] In addition, the loss functions used in the crowd density statistical model are the key point detection loss function and the crowd density segmentation function. Among them, the key point detection loss Loss kp functions are the key point confidence heat map loss function and the key point close connection field loss function respectively. The crowd density segmentation function is the L2 loss function, and the L2 loss function is as follows:

[0209]

[0210] where B is the batch size, represents the crowd density annotation map, represents the crowd density prediction map. For F uc2 ,F uc3 and F out calculate the loss functions with the true crowd density label respectively. This is to better accelerate the model training speed and obtain Loss crowd density-1 ,Loss crowd density-2 and Loss crowd density-3 respectively. The total crowd density loss is Loss crowd density-total =Loss crowd density-1 +Loss crowd density-2 +Loss crowd density-3 , is the regularization term of the loss function.

[0211] The total loss function is:

[0212] Loss total =αLoss kp +βLoss crowd density ;

[0213] where α is 0.2 and β is 0.8.

[0214] The crowd density annotation map of the crowd density statistical model is generated using the Gaussian method, generating a Gaussian probability distribution based on the key points of each head, as follows:

[0215]

[0216]

[0217] β is a hyperparameter of 0.3, where represents the average distance between the m heads closest to the i-th head and that head, δ is the impulse function, and the integral sum within the range is 1. is the Gaussian kernel function, and finally the crowd density annotation map is generated.

[0218] The present invention not only relies on the head key points, but also relies on the key points of other parts of the human body for recognition, increases the model supervision information to improve the representation ability of the human density feature map, accelerates the model training convergence speed, facilitates crowd density prediction, and solves the problem of missing person count due to partial occlusion of human body parts. In addition, the correct rate of the model for crowd density map prediction is improved through multi-feature fusion and attention mechanism.

[0219] The above-mentioned crowd density statistical device based on key point detection can be implemented in the form of a computer program, and this computer program can run on a computer device as Figure 4 shown.

[0220] Please refer to Figure 4 , Figure 4 which is a schematic block diagram of a computer device provided by an embodiment of the present application. The computer device 500 can be a server. Among them, the server can be an independent server or a server cluster composed of multiple servers.

[0221] As Figure 4 shown, this computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the crowd density statistical method based on key point detection as described above.

[0222] The computer device 700 can be a terminal or a server. The computer device 700 includes a processor 720, a memory, and a network interface 750 connected through a system bus 710. Among them, the memory can include a non-volatile storage medium 730 and an internal memory 740.

[0223] The non-volatile storage medium 730 can store an operating system 731 and a computer program 732. When the computer program 732 is executed, it can cause the processor 720 to execute any crowd density statistical method based on key point detection.

[0224] The processor 720 is used to provide computing and control capabilities to support the operation of the entire computer device 700.

[0225] The internal memory 740 provides an environment for the operation of the computer program 732 in the non-volatile storage medium 730. When the computer program 732 is executed by the processor 720, the processor 720 can execute any one of the crowd density statistical methods based on key point detection.

[0226] The network interface 750 is used for network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that Figure 4 The structure shown in is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device 700 to which the solution of this application is applied. Specifically, the computer device 700 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements. Among them, the processor 720 is used to run the program code stored in the memory to implement the following steps:

[0227] Obtain image data;

[0228] Process the image data to obtain a sample map;

[0229] Input the sample map into the crowd density statistical model for head probability prediction to obtain a head probability map;

[0230] Calculate the number of people according to the head probability map.

[0231] In an embodiment, when inputting the sample map into the crowd density statistical model for head probability prediction to obtain a head probability map, the processing method of the crowd density statistical model includes:

[0232] Input the sample map into the stacked hourglass network to obtain a first processing feature and a second processing feature;

[0233] The first processing feature and the second processing feature are associated through the key points in the open pose to obtain human key point features;

[0234] Upsample the first processing feature to obtain an upsampled feature;

[0235] Perform convolution processing on the second processing feature to obtain a first convolution feature;

[0236] Concatenate and combine the upsampled feature and the convolution feature to obtain a first combined feature;

[0237] Process the combined feature through the attention mechanism of CBAM to obtain an attention feature;

[0238] Perform self - calibrated convolution on the human key - point features to obtain self - supervised features;

[0239] Perform convolution on the attention features to obtain second - convolution features;

[0240] Multiply the second - convolution features and the self - supervised features to obtain third - processed features;

[0241] Perform dilated convolution on the sample image to obtain fourth - processed features;

[0242] Perform Channel - wise Cross Attentation on the fourth - processed features to obtain fifth - processed features;

[0243] Concatenate the fifth - processed features and the third - processed features to obtain second - concatenated features;

[0244] Input the second - concatenated features into an up - sampling combination module for up - sampling to obtain sixth - processed features, seventh - processed features, and eighth - processed features;

[0245] Normalize the eighth - processed features through a sigmoid function to obtain a crowd density map;

[0246] Set each pixel value of the crowd density map to be between 0 and 1 to obtain a head probability map.

[0247] In one embodiment, the performing self - calibrated convolution on the human key - point features to obtain self - supervised features includes:

[0248] Slice the human key - point features through the channels of self - calibrated convolution into first - sliced features and second - sliced features;

[0249] Perform average pooling on the first - sliced features to obtain pooled features;

[0250] Perform convolution and up - sampling on the pooled features to obtain ninth - processed features;

[0251] Input the calculation result obtained by adding the ninth - processed features and the first - sliced features into a sigmoid activation function for processing to obtain tenth - processed features;

[0252] Multiply the result of convolving the tenth - processed features and the first - sliced features to obtain eleventh - processed features;

[0253] The results obtained by performing convolution processing on the eleventh processing feature and the second segmentation feature respectively are merged through concatenation to obtain self-supervised features.

[0254] In one embodiment, the second merged feature is input into an upsampling combination module for upsampling processing to obtain a sixth processing feature, a seventh processing feature, and an eighth processing feature. The upsampling combination module is respectively composed of a transposed convolution, a relu activation function, a dilated convolution, and an anti-pooling.

[0255] In one embodiment, the number of feature channels of the sixth processing feature, the seventh processing feature, and the eighth processing feature is all 1.

[0256] In one embodiment, the sizes of the sixth processing feature, the seventh processing feature, and the eighth processing feature are enlarged to the size of the sample image through difference.

[0257] In one embodiment, calculating the number of people according to the head probability map includes:

[0258] Accumulate and sum the probability densities of each pixel on the head probability map and then round up the calculation to obtain the number of people in the image.

[0259] It should be understood that in the embodiments of the present application, the processor 720 may be a central processing unit (CPU), and this processor 720 may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0260] Those skilled in the art can understand that Figure 4 the structure of the computer device 700 shown in does not constitute a limitation on the computer device 700, and it may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0261] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium may be a non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the method for crowd density statistics based on key point detection disclosed in the embodiments of the present invention.

[0262] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the above-described devices, apparatuses, and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein. Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0263] In several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, or units with the same function can be aggregated into one unit. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed couplings or direct couplings or communication connections to each other can be indirect couplings or communication connections through some interfaces, devices, or units, or can be electrical, mechanical, or other forms of connection.

[0264] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiments of the present invention.

[0265] In addition, the functional units in each embodiment of the present invention can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-integrated units can be implemented in the form of hardware or in the form of software functional units.

[0266] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROMs), magnetic disks, or optical discs.

[0267] As described above, the foregoing are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for crowd density statistics based on key point detection, characterized in that, Including: Obtain image data; Process the image data to obtain a sample image; Input the sample image into a crowd density statistical model for head probability prediction to obtain a head probability map; Calculate the number of people according to the head probability map; The step of inputting the sample image into a crowd density statistical model for head probability prediction to obtain a head probability map, and the processing method of the crowd density statistical model includes: Input the sample image into a stacked hourglass network to obtain a first processed feature and a second processed feature; The first processed feature and the second processed feature are associated through the key points in open pose to obtain human key point features; Upsample the first processed feature to obtain an upsampled feature; Perform convolution processing on the second processed feature to obtain a first convolutional feature; Concatenate and merge the upsampled feature and the convolutional feature to obtain a first merged feature; Process the merged feature through the attention mechanism of CBAM to obtain an attention feature; Perform self-calibrated convolution processing on the human key point features to obtain self-supervised features; Perform convolution processing on the attention feature to obtain a second convolutional feature; Multiply the second convolutional feature and the self-supervised feature to obtain a third processed feature; Perform dilated convolution processing on the sample image to obtain a fourth processed feature; Perform Channel-wise Cross Attentation processing on the fourth processed feature to obtain a fifth processed feature; Concatenate and merge the fifth processed feature and the third processed feature to obtain a second merged feature; Input the second merged feature into an upsampling combination module for upsampling processing to obtain a sixth processed feature, a seventh processed feature, and an eighth processed feature; Normalize the eighth processed feature through a sigmoid function to obtain a crowd density map; Set each pixel value of the crowd density map to be between 0 and 1 to obtain a head probability map.

2. The method for crowd density statistics based on key point detection according to claim 1, wherein, The step of performing self-calibrated convolution processing on the human key point features to obtain self-supervised features includes: Slice the human key point features through the channels of self-calibrated convolution into a first sliced feature and a second sliced feature; Perform average pooling processing on the first sliced feature to obtain a pooled feature; Perform convolution and upsampling processing on the pooled feature to obtain a ninth processed feature; Input the calculation result obtained by adding the ninth processed feature and the first sliced feature into a sigmoid activation function for processing to obtain a tenth processed feature; Multiply the result of convolution processing of the tenth processed feature and the first sliced feature; Concatenate and merge the results obtained by performing convolution processing on the eleventh processed feature and the second sliced feature respectively to obtain self-supervised features.

3. The method for crowd density statistics based on key point detection according to claim 1, wherein The second combined feature is input into the upsampling combination module for upsampling processing to obtain a sixth processed feature, a seventh processed feature, and an eighth processed feature. The upsampling combination module is respectively composed of a transposed convolution, a relu activation function, a dilated convolution, and an anti-pooling.

4. The method for crowd density statistics based on key point detection according to claim 3, wherein The number of feature channels of the sixth processed feature, the seventh processed feature, and the eighth processed feature is all 1.

5. The method for crowd density statistics based on key point detection according to claim 4, wherein The sizes of the sixth processed feature, the seventh processed feature, and the eighth processed feature are enlarged to the size of the sample map through a difference.

6. The method for crowd density statistics based on key point detection according to claim 1, wherein The calculating the number of people according to the head probability map includes: The probability density of each pixel on the head probability map is accumulated and summed and then rounded to calculate the number of people in the map.

7. A crowd density statistical device based on key point detection, when operating, executes the crowd density statistical method based on key point detection according to any one of claims 1-6, characterized in that It includes an acquisition unit, a processing unit, a prediction unit, and a calculation unit; The acquisition unit is used to acquire image data; The processing unit is used to process the image data to obtain a sample map; The prediction unit is used to input the sample map into the crowd density statistical model for head probability prediction to obtain a head probability map; The calculation unit is used to calculate the number of people according to the head probability map.

8. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the crowd density statistical steps based on key point detection described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program includes program instructions. When the program instructions are executed by the processor, the processor executes the crowd density statistical steps based on key point detection described in any one of claims 1 to 6.

Images