Counting method, counting device, and storage medium

Abstract

The present disclosure relates to a counting method, a counting device and a storage medium. The counting method comprises: obtaining an image comprising an object to be counted; performing object feature extraction on the object to be counted in the image to obtain different scale feature maps of the object to be counted, and fusing the different scale feature maps; performing up-sampling processing on the fused feature maps by using a depth separable convolution model and a sub-pixel convolution model to obtain a density map representing the pixel distribution of the object to be counted in the image; and taking the sum of the pixel values in the density map as the number of the object to be counted. The present disclosure performs up-sampling processing on the fused feature maps by using the depth separable convolution model and the sub-pixel convolution model to reduce the calculation amount when counting the object to be counted, so that the counting method can be applied to a mobile terminal.

Description

Technical Field

[0001] This disclosure relates to the field of machine learning, and more particularly to a counting method, counting device, and storage medium. Background Technology

[0002] During holidays, large crowds often move through public places such as tourist attractions, subway stations, train stations, and shopping malls. These dense crowds can easily lead to stampedes and other incidents, endangering public safety and personal property. Furthermore, with highly contagious viruses, exceeding a certain population density threshold in enclosed spaces can accelerate virus transmission. Therefore, collecting images of the locations to be counted and using these images to count and estimate the density of people within those locations facilitates the implementation of appropriate crowd control and evacuation measures.

[0003] With the development of deep learning, deep learning-based crowd counting methods can now operate 24 / 7, offering greater real-time performance and efficiency than manual monitoring. However, the deep learning-based counting models are complex, consume large amounts of memory, and are not suitable for mobile devices with limited computing power. Summary of the Invention

[0004] To overcome the problems existing in the related technologies, this disclosure provides a counting method, a counting device, and a storage medium.

[0005] According to a first aspect of the present disclosure, a counting method is provided, comprising: acquiring an image including objects to be counted; extracting object features from the objects to be counted in the image to obtain feature maps of the objects to be counted at different scales, and fusing the feature maps of different scales; upsampling the fused feature maps using a depthwise separable convolution model and a subpixel convolution model to obtain a density map, the density map representing the pixel distribution of the objects to be counted in the image; and using the sum of the pixel values ​​in the density map as the number of the objects to be counted.

[0006] In one embodiment, the step of extracting object features from the object to be counted in the image to obtain feature maps of the object to be counted at different scales includes: extracting object features from the object to be counted in the image sequentially using a convolutional layer and a bottleneck layer to obtain feature maps of the object to be counted at different scales.

[0007] In one embodiment, object features are extracted from the object to be counted in the image sequentially using convolutional layers and bottleneck layers to obtain different scale features of the object to be counted. This includes: performing a first object feature extraction on the object to be counted in the image using convolutional layers to obtain a first feature map of the object to be counted; and iteratively extracting object features from the first feature map sequentially using bottleneck modules in multiple bottleneck layers to obtain different scale feature maps of the object to be counted. The number of bottleneck modules stacked in each bottleneck layer is different.

[0008] In one embodiment, the number of bottleneck modules in multiple bottleneck layers that sequentially extract object features from the first feature map increases progressively according to a first value. The step size of the first bottleneck module in the first type of bottleneck layer in the multiple bottleneck layers is always the first step size. The first type of bottleneck layer refers to the other bottleneck layers in the multiple bottleneck layers except for the first and last bottleneck layers in which feature extraction is performed. The step size of the remaining bottleneck modules in the multiple bottleneck layers except for the first bottleneck module in the first type of bottleneck layer is always the second step size. The first step size and the second step size are different.

[0009] In one implementation, object feature extraction is performed using the first bottleneck module in the first type of bottleneck layer, including:

[0010] In response to performing the Nth object feature extraction using the first bottleneck module in the Nth bottleneck layer, the second feature map is convolved to obtain the third feature map, and the second feature map is max-pooled to obtain the fourth feature map. The second feature map is obtained by performing the N-1th object feature extraction using the (N-1)th bottleneck layer, where N is a positive integer greater than or equal to 2. The third feature map is depthwise convolutionally separated, and the fourth feature map is channel-expanded so that the number of channels in the depthwise convolutionally separated third feature map is the same as the number of channels in the channel-expanded fourth feature map. The depthwise convolutionally separated third feature map and the channel-expanded fourth feature map are added by channel to obtain the feature map extracted by the first bottleneck module in the Nth bottleneck layer.

[0011] In one embodiment, fusing feature maps of different scales includes: performing depthwise convolution separation on the feature map obtained from the last feature extraction to obtain a fifth feature map; concatenating the fifth feature map with a portion of the feature map extracted from the first bottleneck layer, and then concatenating it with the feature map obtained from the first feature extraction to obtain a fused feature map; the portion of the feature map extracted from the first bottleneck layer includes feature maps extracted from non-adjacent iterations of the last feature extraction.

[0012] According to a second aspect of the present disclosure, a counting device is provided, comprising:

[0013] An acquisition unit is used to acquire an image including objects to be counted; a feature extraction unit is used to extract object features from the objects to be counted in the image, obtain feature maps of the objects to be counted at different scales, and fuse the feature maps of different scales; a counting unit is used to upsample the fused feature maps using a depthwise separable convolution model and a sub-pixel convolution model to obtain a density map, wherein the density map represents the pixel distribution of the objects to be counted in the image; and the sum of the pixel values ​​in the density map is used as the number of objects to be counted.

[0014] In one embodiment, the feature extraction unit is used to: extract object features from the object to be counted in the image by using a convolutional layer and a bottleneck layer in sequence, so as to obtain different scale features of the object to be counted.

[0015] In one embodiment, the feature extraction unit is used to: perform a first object feature extraction on the object to be counted in the image using a convolutional layer to obtain a first feature map of the object to be counted; and use bottleneck modules in multiple bottleneck layers to iteratively extract object features from the first feature map in sequence to obtain feature maps of the object to be counted at different scales, wherein the number of bottleneck modules stacked in each of the multiple bottleneck layers is different.

[0016] In one embodiment, the number of bottleneck modules in multiple bottleneck layers that sequentially extract object features from the first feature map increases progressively according to a first value. The step size of the first bottleneck module in the first type of bottleneck layer in the multiple bottleneck layers is always the first step size. The first type of bottleneck layer refers to the other bottleneck layers in the multiple bottleneck layers except for the first and last bottleneck layers in which feature extraction is performed. The step size of the remaining bottleneck modules in the multiple bottleneck layers except for the first bottleneck module in the first type of bottleneck layer is always the second step size. The first step size and the second step size are different.

[0017] In one embodiment, the feature extraction unit is configured to: in response to performing the Nth object feature extraction using the first bottleneck module in the Nth bottleneck layer, perform convolution processing on the second feature map to obtain a third feature map, and perform max pooling processing on the second feature map to obtain a fourth feature map, wherein the second feature map is obtained by performing the N-1th object feature extraction using the (N-1)th bottleneck layer, and N is a positive integer greater than or equal to 2; perform depthwise convolution separation on the third feature map, and perform channel expansion on the fourth feature map so that the number of channels of the depthwise convolution-separated third feature map is the same as the number of channels of the channel-expanded fourth feature map; and add the depthwise convolution-separated third feature map and the channel-expanded fourth feature map by channel to obtain the feature map extracted by the first bottleneck module in the Nth bottleneck layer.

[0018] In one embodiment, the feature extraction unit is used to: perform depthwise convolution separation on the feature map obtained from the last feature extraction to obtain a fifth feature map; concatenate the fifth feature map with a portion of the feature map extracted from the first bottleneck layer, and then concatenate it with the feature map obtained from the first feature extraction to obtain a fused feature map; the portion of the feature map extracted from the first bottleneck layer includes feature maps extracted from non-adjacent features of the last feature extraction.

[0019] According to a third aspect of the present disclosure, a counting device is provided, comprising:

[0020] A processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the counting method described in any embodiment of the first aspect.

[0021] According to a fourth aspect of the present disclosure, a storage medium is provided that stores instructions which, when executed by a processor of a mobile terminal, enable the mobile terminal to perform the counting method described in any embodiment of the first aspect.

[0022] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: acquiring an image including objects to be counted; extracting object features from the objects to be counted in the image to obtain feature maps of the objects to be counted at different scales; fusing the feature maps of different scales to improve the accuracy of object feature extraction; upsampling the fused feature maps using a depthwise separable convolution model and a subpixel convolution model to obtain a density map; upsampling the fused feature maps using a depthwise separable convolution model and a subpixel convolution model to reduce the computational load when counting the objects to be counted, making the counting method applicable to mobile terminals; obtaining the number of objects to be counted through the pixels representing the objects to be counted in the density map; and using the density map to count the objects to be counted to avoid severe occlusion between the objects to be counted.

[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0025] Figure 1 This is a flowchart illustrating a counting method according to an exemplary embodiment.

[0026] Figure 2 This is a schematic diagram illustrating a model structure for counting according to an exemplary embodiment.

[0027] Figure 3-1 This is a schematic diagram of a bottleneck module with a step size of 1, according to an exemplary embodiment.

[0028] Figure 3-2 This is a schematic diagram of a bottleneck module with a step size of 2, according to an exemplary embodiment.

[0029] Figure 4 This is a schematic diagram illustrating the structure of a depth-separable convolutional model and a subpixel convolutional model according to an exemplary embodiment.

[0030] Figure 5 This is a schematic diagram illustrating feature fusion in a model according to an exemplary embodiment.

[0031] Figure 6 This is a flowchart illustrating object feature extraction according to an exemplary embodiment.

[0032] Figure 7 This is a flowchart illustrating object feature extraction using the first bottleneck module in a first type of bottleneck layer, according to an exemplary embodiment.

[0033] Figure 8 This is a flowchart illustrating the fusion of feature maps at different scales according to an exemplary embodiment.

[0034] Figure 9 This is a flowchart illustrating counting using a model for counting, according to an exemplary embodiment.

[0035] Figure 10 This is a block diagram of a counting device according to an exemplary embodiment.

[0036] Figure 11 This is a block diagram illustrating a counting apparatus according to an exemplary embodiment. Detailed Implementation

[0037] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure.

[0038] In some applications, it's necessary to count the number of people based on captured images and then implement different processing measures accordingly. For example, during holidays, in public places such as train stations, tourist attractions, and shopping malls, it's necessary to count the number of people in public places to avoid stampedes caused by overcrowding. Based on the count, appropriate crowd control and evacuation measures are then implemented.

[0039] Related technologies have achieved real-time and efficient crowd counting based on deep learning. However, crowd counting algorithms in these technologies may have at least one of the following problems.

[0040] (1) When the crowd is too dense, it is difficult to effectively detect each person due to severe occlusion between people.

[0041] (2) The population has multi-scale problems and deformation problems, requiring the algorithm to be able to adapt to them.

[0042] (3) Not applicable to mobile terminals.

[0043] This can be understood as follows: current counting algorithms mostly have complex structures, a large number of parameters, and high computational demands. Therefore, related technologies deploy counting algorithms in the cloud. During actual counting, the terminal uploads the captured images to the cloud. The cloud then counts the number of people in the images. The cloud sends the calculated results back to the terminal. Transmitting images from the terminal to the cloud requires significant bandwidth. In locations with poor network connectivity, network issues can easily prevent the terminal from uploading images to the cloud in a timely manner, or from receiving the counting results from the cloud, thus hindering the rapid acquisition of counting results.

[0044] Based on this, this disclosure provides a counting method that acquires an image including a crowd, extracts features from the people in the image, and obtains feature maps at different scales. Fusing these feature maps at different scales preserves the features of people in the image to the greatest extent possible. The fused feature maps are then upsampled using a depthwise separable convolution model and a subpixel convolution model to obtain a density map. The sum of the pixel values ​​in the density map is used as the number of people. Counting the objects to be counted using the density map avoids severe occlusion between objects. In other words, this disclosure uses a density map-based approach for crowd counting and builds a lightweight counting model based on depthwise separable convolution and multi-scale feature fusion. Compared to counting models in related technologies, this reduces the number of parameters, enabling real-time deployment on mobile terminals. For example, the technical method provided in this disclosure can be applied to intelligent monitoring equipment in public places with high population mobility and density, such as tourist attractions, subways, shopping malls, and squares, to perform crowd counting and density analysis within the monitored area, enabling managers to conduct timely and effective crowd control to prevent public safety issues.

[0045] It should be noted that the counting method provided in this disclosure is not only applicable to the number of people in a statistical image. It is also applicable to cell counting, vehicle counting and vehicle density analysis, and biological asset inventory, etc. Therefore, this disclosure uses the objects to be counted to represent people, cells, and vehicles, etc.

[0046] The following embodiments will illustrate a counting method provided in this disclosure in conjunction with the accompanying drawings.

[0047] Figure 1 This is a flowchart illustrating a counting method according to an exemplary embodiment. Figure 1 As shown, the counting method used in a mobile terminal includes the following steps.

[0048] In step S11, an image including the objects to be counted is acquired.

[0049] In this embodiment, a mobile terminal is placed in the location where counting is required, and images are captured by a camera device on the mobile terminal. The captured images include objects to be counted. These objects can be people, animals, cells, vehicles, etc. The mobile terminal can be a smartphone, a smart wearable device, etc. For example, the mobile terminal is placed in the feeding room of a farm, and images of the animals in the feeding room are acquired through the mobile terminal. The animals in the images are counted using the counting method provided in this disclosure to obtain the number of animals in the feeding room. When applying the counting method provided in this disclosure to count different objects, the neural network model involved in the calculation method needs to be trained in advance using images of the objects to be counted.

[0050] In step S12, object features are extracted from the objects to be counted in the image to obtain feature maps of the objects to be counted at different scales, and the feature maps at different scales are fused together.

[0051] This disclosure employs convolutions of varying lengths to extract object features from the image to be counted. However, using only convolutions of varying lengths would result in a large number of parameters during object extraction, increasing the computational load for counting the objects. To reduce computational load and enable the counting method to be applied to mobile terminals, this disclosure uses a bottleneck layer to extract object features. Feature maps of different scales are extracted through convolution and the bottleneck layer. Fusing these feature maps at different scales preserves more of the object features of the objects to be counted in the image.

[0052] In step S13, the fused feature map is upsampled using a depthwise separable convolution model and a subpixel convolution model to obtain a density map.

[0053] In this disclosure, a depthwise separable convolutional model can be used to extract features from the fused feature map. The extracted features are then input into a sub-pixel convolutional model. The extracted features are grouped by channel, and the grouped features are sequentially concatenated point by point to obtain a density map. The density map represents the pixel distribution of the object to be counted in the image. Using a depthwise separable convolutional model and a sub-pixel convolutional model to upsample the fused feature map allows for more learnable parameters compared to bilinear interpolation upsampling. The density map obtained by upsampling the fused feature map using a depthwise separable convolutional model and a sub-pixel convolutional model retains more details and reduces the loss of details in the density map.

[0054] In step S14, the sum of the pixel values ​​in the density map is used as the number of objects to be counted.

[0055] The total number of objects to be counted in the image can be obtained by directly summing the pixel values ​​on the density map. Using the density map to count objects can avoid the problem of severe occlusion between objects and can also reflect the distribution of objects in the region of interest.

[0056] This disclosure employs a depthwise separable convolutional model to lightweight the counting model, enabling it to maintain accuracy while minimizing parameters and computational cost, thus allowing for real-time operation on mobile devices. The following embodiments of this disclosure will illustrate the model structure for counting with reference to the accompanying drawings.

[0057] Figure 2 This is a schematic diagram illustrating a model structure for counting, according to an exemplary embodiment. Figure 2 As shown, the model structure provided in this disclosure can be divided into a feature extraction module and a density map regression module. The feature extraction module extracts object features from the input image. The density map regression module outputs the predicted density map through upsampling and multi-feature fusion. Figure 2In this code, "Conv3" represents a standard 3×3 convolution, and "stride=2" indicates a stride of 2. "BottleNecks*" represents a bottleneck layer, which is composed of several stacked bottleneck modules (BottleNecks). "*" indicates the number of BottleNecks, and "stride=2" indicates the stride of the first BottleNeck; the others default to 1. For example, BottleNecks2 represents a bottleneck layer generated by stacking 2 bottleneck modules. BottleNecks4 represents a bottleneck layer generated by stacking 4 bottleneck modules. "Conv1" represents a 1×1 standard convolution with a stride of 1. "DWS Conv3" represents a 3×3 depthwise separable convolution with a stride of 1. "Sub-pixel Conv3 2X" represents a sub-pixel convolution model, where upsampling operations are learned by upsampling parameters by a factor of 2 using a sub-pixel convolution model, employing a 3×3 depthwise separable convolution. In this embodiment of the disclosure, each type of convolution is followed by a normalization layer and a Rectified Linear Unit (ReLU) or ReLU6 activation function by default.

[0058] This disclosure will be explained in conjunction with the accompanying drawings. Figure 2 The various parts of the model used for counting are explained.

[0059] Figure 3-1 This is a schematic diagram of a bottleneck module with a step size of 1, according to an exemplary embodiment. Figure 3-2 This is a schematic diagram illustrating the structure of a bottleneck module with a step size of 2 according to an exemplary embodiment. Figure 3-1 As shown, when using BottleNeck with stride=1 for object feature extraction, the input includes a feature map of the objects to be counted, and Conv1 is used for object feature extraction. The feature map obtained from object feature extraction in Conv1 is input into DWS Conv3. The initial input feature map including the objects to be counted is fused with the feature map obtained from object feature extraction in DWS Conv3, and the output is the feature map obtained from object feature extraction by the bottleneck module with a stride of 1.

[0060] This disclosure adds a max-pooling operation to BottleNeck with stride=2, enabling residual learning for BottleNeck with stride=2. Figure 3-2In this code, "Max Pool" indicates max pooling; "Stride=2" indicates a stride of 2, equivalent to downsampling the input features by a factor of 2. "Channel Pad" indicates channel padding, which pads the channel dimension with zeros to match the number of channels in the "DWS Conv3" output, allowing for summation. Figure 3-2 As shown, when using BottleNeck with stride=2 for object feature extraction, the input includes a feature map containing the objects to be counted. Object features are extracted from this feature map using two methods, and the extracted features are then fused to obtain the feature map for object feature extraction using BottleNeck with stride=2. For example, in the first method, Conv1 is used to extract object features from the feature map containing the objects to be counted. The feature map obtained from Conv1 is then used for further object feature extraction using DWS Conv3 with stride=2. In the second method, Max Pooling with stride=2 is used to pool the feature map containing the objects to be counted. Channel Pad is used to expand the channels of the pooled feature map so that the number of channels in the pooled feature map is the same as the feature map obtained using DWS Conv3 in the first method. The feature map extracted in the first method and the feature map extracted in the second method are then fused to obtain the feature map output by BottleNeck with stride=2.

[0061] Figure 4 This is a schematic diagram illustrating the structure of a depthwise separable convolution model and a subpixel convolution model according to an exemplary embodiment. For example... Figure 4 As shown, in this embodiment of the disclosure, the sub-pixel convolution model (Sub-pixel Conv) is improved using the depthwise separable convolution model (DWS Conv3) to obtain Sub-pixel Conv3 2X. Sub-pixel Conv3 2X enables upsampling processing with an upsampling factor of 2. Figure 4 In this context, W is short for width, representing the width of the feature map. H is short for height, representing the height of the feature map. Using... Figure 4The Sub-pixel Conv3 2X upsampling process shown includes: extracting object features from a 4-channel feature map using DWS Conv3 to obtain a 16-channel feature map. The 16-channel feature map is then grouped according to its channels, and the grouped feature maps are sequentially concatenated point-by-point to obtain a density map with an upsampling factor of 2. In this embodiment, by using a parameter-learnable upsampling operation to upsample low-resolution features, the loss of detail in the density map can be reduced, increasing the accuracy of counting objects based on the density map.

[0062] Figure 5 This is a schematic diagram illustrating feature fusion in a model according to an exemplary embodiment. For example... Figure 5 As shown, fusing features at different levels and scales can effectively improve the performance of models used for counting. This embodiment employs channel concatenation for feature fusion. Compared to feature fusion using an additive approach, this method retains more dimensional feature information. Figure 5 In this context, deep features refer to features obtained through multiple object feature extractions. Shallow features, compared to deep features, require fewer object feature extractions. Sub-pixel Conv3 is used for object feature extraction from deep features, while Conv1 is used for object feature extraction from shallow features. The feature map obtained from object feature extraction using Sub-pixel Conv3 is concatenated with the feature map obtained from object feature extraction using Conv1. The concatenated feature map is then used again for object feature extraction using Conv1, and the resulting feature map is used as the fused feature map. This embodiment fuses shallow and deep features, improving the accuracy of object feature extraction. Figure 2 As can be seen, this disclosure performs feature fusion at two different scales, resulting in a sparser model structure and less memory usage during model execution.

[0063] As can be seen from the above description of the counting model in the embodiments, the counting model provided in this disclosure uses convolutions with kernel sizes of 3×3 and 1×1, and employs a depthwise separable convolution model for object feature extraction. Under the same input size, the counting model of this disclosure requires less computation than related technologies, and has a deeper model, resulting in higher accuracy. This disclosure only requires a density map to count the objects to be counted, which consumes less memory and has a faster training speed compared to dense blocks.

[0064] In this disclosure, the steps for counting using a counting model are described below through embodiments.

[0065] In this embodiment of the disclosure, an image including the object to be counted is obtained, and the object features of the object to be counted in the image are extracted sequentially using a convolutional layer and a bottleneck layer to obtain the different scale features of the object to be counted.

[0066] Convolutional layers are used to extract object features from the objects to be counted in the image, obtaining the first-scale features. A bottleneck layer is then used to further extract object features from the first-scale features, yielding the second-scale features. This process allows for the acquisition of features at different scales. Using features at different scales ensures the extraction of more dimensional features, improving counting accuracy. The use of a bottleneck layer with fewer parameters for object feature extraction prepares the method for deployment on mobile devices.

[0067] In one implementation, Figure 6 This is a flowchart illustrating object feature extraction according to an exemplary embodiment. For example... Figure 6 As shown, convolutional layers and bottleneck layers are used sequentially to extract object features from the objects to be counted in the image, obtaining features of the objects at different scales. The process includes the following steps.

[0068] In step S21, the convolutional layer is used to perform the first object feature extraction on the object to be counted in the image to obtain the first feature map of the object to be counted.

[0069] In step S22, the bottleneck modules in multiple bottleneck layers are used to iteratively extract the object features from the first feature map in turn, so as to obtain feature maps of the object to be counted at different scales. The number of bottleneck modules stacked in each bottleneck layer is different.

[0070] In this embodiment of the disclosure, the number of bottleneck modules included in the multiple bottleneck layers that sequentially perform object feature iterative extraction on the first feature map increases progressively according to a first value. The step size of the first bottleneck module in the first type of bottleneck layer in the multiple bottleneck layers is always the first step size. The first type of bottleneck layer refers to the other bottleneck layers in the multiple bottleneck layers except for the bottleneck layers in the first and last feature extractions. The step size of the other bottleneck modules in the multiple bottleneck layers except for the first bottleneck module in the first type of bottleneck layer is always the second step size. The first step size and the second step size are different.

[0071] In this embodiment, the counting model includes multiple bottleneck layers, each containing bottleneck modules. The number of bottleneck modules varies across different bottleneck layers. Starting from the first bottleneck layer, the number of bottleneck modules in each subsequent bottleneck layer is greater than the number of bottleneck modules in the preceding adjacent bottleneck layer. Features at different scales can be extracted through different bottleneck modules within a bottleneck layer. To enable learnable parameters for the bottleneck modules, this embodiment improves upon bottleneck modules with different step lengths. Taking a first step length of 2 and a second step length of 1 as an example, a bottleneck layer containing a bottleneck module with a step length of 2 is considered a first-type bottleneck layer. In the first-type bottleneck layer, the first bottleneck module has a step length of 2, and all other bottleneck modules in the first-type bottleneck layer have a step length of 1.

[0072] Figure 7 This is a flowchart illustrating object feature extraction using the first bottleneck module in a first type of bottleneck layer, according to an exemplary embodiment. For example... Figure 7 As shown, object feature extraction is performed using the first bottleneck module in the first type of bottleneck layer, including the following steps.

[0073] In step S31, in response to performing the Nth object feature extraction using the first bottleneck module in the Nth bottleneck layer, the second feature map is convolved to obtain the third feature map, and the second feature map is max-pooled to obtain the fourth feature map.

[0074] In this embodiment of the disclosure, the second feature map is obtained by performing the N-1th object feature extraction using the N-1th bottleneck layer, where N is a positive integer greater than or equal to 2.

[0075] In step S32, the third feature map is subjected to depthwise convolution separation, and the fourth feature map is subjected to channel expansion, so that the number of channels in the third feature map after depthwise convolution separation is the same as the number of channels in the fourth feature map after channel expansion.

[0076] In step S33, the third feature map after depthwise convolution is added to the fourth feature map after channel expansion, and the feature map is obtained by channel addition, to obtain the feature map extracted by the first bottleneck module in the Nth bottleneck layer.

[0077] The above-described process of extracting object features using the first bottleneck module in the first type of bottleneck layer has already been discussed in the context of... Figure 3-2 The relevant section has already been described, so it will not be repeated here.

[0078] Figure 8 This is a flowchart illustrating the fusion of feature maps at different scales according to an exemplary embodiment. For example... Figure 8 As shown, the fusion of feature maps at different scales includes the following steps.

[0079] In step S41, the feature map obtained from the last feature extraction is subjected to depthwise convolution separation to obtain the fifth feature map.

[0080] In step S42, the fifth feature map is concatenated with the partial feature map extracted from the first bottleneck layer, and then concatenated with the feature map obtained from the first feature extraction to obtain the fused feature map.

[0081] In this embodiment, the feature maps extracted from the first bottleneck layer include feature maps extracted in non-adjacent iterations to the last feature extraction. The process of fusing feature maps of different scales can be found in [reference needed]. Figure 2 as well as Figure 5 The explanation of feature fusion in this paper will not be repeated here.

[0082] In one embodiment, combined with Figure 9 This describes the process of counting using the counting model provided in this disclosure. Figure 9 This is a flowchart illustrating counting using a model for counting, according to an exemplary embodiment. Figure 9 As shown, counting using a model for counting includes the following steps.

[0083] In step S51, data processing occurs. This can be understood as adding labels to the image containing the objects to be counted. Taking counting people in an image as an example, when creating labels, a marker is made at the center pixel of each person's head in the image, and then a two-dimensional Gaussian kernel is placed at each marker to generate a density map.

[0084] The formula for calculating the Gaussian kernel generation density map is:

[0085]

[0086] In the formula, N represents the total number of people in the image, and x i Let δ(xx) represent the i-th pixel in the image. i ) indicates that there is a head marker at the i-th pixel position, G σ (x) represents a Gaussian kernel, and σ is the standard deviation of the Gaussian kernel.

[0087] In step S52, the model is trained. In this embodiment, the parameters of the model used for counting are set using the parameter values ​​in Table 1. Table 1 is the model parameter setting table.

[0088] Table 1

[0089]

[0090]

[0091] In Table 1, "p1" and "p2" are... Figure 2 The 1×1 convolution is shown at the position indicated. "Concat(*)" means the output of "*" is concatenated with the output of the previous layer along the channel dimension. t represents the dilation factor of the dilated convolution in the BottleNeck. c represents the number of output channels of the module. n represents the number of module repetitions. For example, "BottleNecks" indicates the number of BottleNecks. s represents the total stride of all convolutions in the module.

[0092] The training process for the above model consists of two parts: first, floating-point training, and then quantization-aware training.

[0093] Before starting floating-point training, for the feature extraction module, weight parameters from MobileNetV2 are loaded; for the density map regression module, convolution parameters are randomly initialized from a Gaussian distribution.

[0094] In step S53, quantization-aware training is performed. Before starting quantization-aware training, the parameter weights are loaded with weight values ​​from floating-point training. This disclosure uses quantization-aware training to quantize and compress the model, reducing the model size by 1 / 4, making the model embeddable on mobile terminals. This disclosure uses the following formula to calculate the model error during training:

[0095]

[0096] In the formula, Θ represents the trainable parameter weights in the model, N represents the batch size for each training batch, and X... i The image input to the model, F(X) i ;Θ) represents the input image X i The density map predicted by the model, F i For the input image X i The corresponding density map label.

[0097] In step S54, the mobile terminal is embedded.

[0098] The trained counting model is embedded into a mobile terminal. The mobile terminal receives an image containing the objects to be counted and inputs it into the model to obtain a predicted density map. The values ​​at each pixel location in the predicted density map are accumulated to obtain the predicted number of objects to be counted.

[0099] The counting model provided in this disclosure employs a depthwise separable convolutional model, reducing the computational cost during counting and enabling faster operation when embedded in mobile terminals. Furthermore, a lightweight, learnable upsampling operation is introduced to improve the accuracy of the density map, ensuring counting accuracy. This disclosure achieves both reduced computational cost and high counting accuracy.

[0100] Based on the same concept, embodiments of this disclosure also provide a counting device.

[0101] It is understood that the counting device provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0102] Figure 10 This is a block diagram illustrating a counting device according to an exemplary embodiment. (Refer to...) Figure 10 The counting device 100 includes an acquisition unit 101, a feature extraction unit 102, and a counting unit 103.

[0103] The acquisition unit 101 is used to acquire an image containing the objects to be counted; the feature extraction unit 102 is used to extract the object features of the objects to be counted in the image, obtain feature maps of the objects to be counted at different scales, and fuse the feature maps of different scales; the counting unit 103 is used to upsample the fused feature maps using a depthwise separable convolution model and a sub-pixel convolution model to obtain a density map, which represents the pixel distribution of the objects to be counted in the image; the sum of the pixel values ​​in the density map is used as the number of objects to be counted.

[0104] In one embodiment, the feature extraction unit 102 is used to: extract object features from the objects to be counted in the image by using convolutional layers and bottleneck layers in sequence, so as to obtain different scale features of the objects to be counted.

[0105] In one embodiment, the feature extraction unit 102 is used to: perform a first object feature extraction on the object to be counted in the image using a convolutional layer to obtain a first feature map of the object to be counted; and use the bottleneck modules in multiple bottleneck layers to iteratively extract object features from the first feature map in sequence to obtain feature maps of the object to be counted at different scales, wherein the number of bottleneck modules stacked in each bottleneck layer is different.

[0106] In one embodiment, the number of bottleneck modules in multiple bottleneck layers that sequentially extract object features from the first feature map increases progressively according to a first value. The step size of the first bottleneck module in the first type of bottleneck layer in the multiple bottleneck layers is always the first step size. The first type of bottleneck layer refers to the other bottleneck layers in the multiple bottleneck layers except for the first and last bottleneck layers in which feature extraction is performed. The step size of the other bottleneck modules in the multiple bottleneck layers except for the first bottleneck module in the first type of bottleneck layer is always the second step size. The first step size and the second step size are different.

[0107] In one embodiment, the feature extraction unit 102 is configured to: in response to performing the Nth object feature extraction using the first bottleneck module in the Nth bottleneck layer, perform convolution processing on the second feature map to obtain a third feature map, and perform max pooling processing on the second feature map to obtain a fourth feature map, wherein the second feature map is obtained by performing the N-1th object feature extraction using the (N-1)th bottleneck layer, and N is a positive integer greater than or equal to 2; perform depthwise convolution separation on the third feature map, and perform channel expansion on the fourth feature map, so that the number of channels of the depthwise convolution-separated third feature map is the same as the number of channels of the channel-expanded fourth feature map; and add the depthwise convolution-separated third feature map and the channel-expanded fourth feature map by channel to obtain the feature map extracted by the first bottleneck module in the Nth bottleneck layer.

[0108] In one embodiment, the feature extraction unit 102 is used to: perform depthwise convolution separation on the feature map obtained from the last feature extraction to obtain a fifth feature map; concatenate the fifth feature map with a portion of the feature map extracted from the first bottleneck layer, and then concatenate it with the feature map obtained from the first feature extraction to obtain a fused feature map; the portion of the feature map extracted from the first bottleneck layer includes feature maps extracted from non-adjacent features of the last feature extraction.

[0109] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0110] Figure 11 This is a block diagram illustrating a counting device according to an exemplary embodiment. For example, device 200 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0111] Reference Figure 11 The device 200 may include one or more of the following components: processing component 202, memory 204, power component 206, multimedia component 208, audio component 210, input / output (I / O) interface 212, sensor component 214, and communication component 216.

[0112] Processing component 202 typically controls the overall operation of device 200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 202 may include one or more modules to facilitate interaction between processing component 202 and other components. For example, processing component 202 may include a multimedia module to facilitate interaction between multimedia component 208 and processing component 202.

[0113] Memory 204 is configured to store various types of data to support the operation of device 200. Examples of such data include instructions for any application or method operating on device 200, contact data, phonebook data, messages, pictures, videos, etc. Memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0114] The power supply component 206 provides power to the various components of the device 200. The power supply component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 200.

[0115] Multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 208 includes a front-facing camera and / or a rear-facing camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0116] Audio component 210 is configured to output and / or input audio signals. For example, audio component 210 includes a microphone (MIC) configured to receive external audio signals when device 200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 204 or transmitted via communication component 216. In some embodiments, audio component 210 also includes a speaker for outputting audio signals.

[0117] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0118] Sensor assembly 214 includes one or more sensors for providing status assessments of various aspects of device 200. For example, sensor assembly 214 may detect the on / off state of device 200, the relative positioning of components such as the display and keypad of device 200, changes in the position of device 200 or a component of device 200, the presence or absence of user contact with device 200, the orientation or acceleration / deceleration of device 200, and temperature changes of device 200. Sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0119] Communication component 216 is configured to facilitate wired or wireless communication between device 200 and other devices. Device 200 can access wireless networks based on communication standards, such as WiFi, 4G, or 5G, or combinations thereof. In one exemplary embodiment, communication component 216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 216 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0120] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0121] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, which can be executed by a processor 220 of the device 200 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0122] It is understood that in this disclosure, "multiple" refers to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0123] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0124] It can be further understood that, unless otherwise specified, "connection" includes both direct connections where no other components exist between the two parties and indirect connections where other components exist between them.

[0125] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0126] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following scope of claims.

[0127] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A counting method, characterized by, include: Get the image containing the objects to be counted; The first feature map of the object to be counted is obtained by using a convolutional layer to perform the first object feature extraction on the object to be counted in the image. By using the bottleneck modules in multiple bottleneck layers, the object features are iteratively extracted from the first feature map in turn to obtain feature maps of the object to be counted at different scales. The feature map obtained from the last feature extraction is then separated by depthwise convolution to obtain the fifth feature map. The number of bottleneck modules stacked in each of the multiple bottleneck layers is different. After concatenating the fifth feature map with the partial feature map extracted from the first type of bottleneck layer, it is concatenated with the feature map obtained from the first feature extraction to obtain the fused feature map. The step size of the first bottleneck module in the first type of bottleneck layer among the multiple bottleneck layers is the first step size. The first type of bottleneck layer refers to the other bottleneck layers among the multiple bottleneck layers except for the bottleneck layers where feature extraction is performed for the first and last time. The partial feature map extracted from the first type of bottleneck layer includes the feature map extracted in a non-adjacent time to the last feature extraction. Feature extraction is performed on the fused feature map using a depthwise separable convolutional model deployed on a mobile terminal. The extracted features are then input into a sub-pixel convolutional model deployed on the mobile terminal. The extracted features are grouped according to channels, and the grouped features are sequentially concatenated point by point to obtain a density map. The density map represents the pixel distribution of the object to be counted in the image. The sum of the pixel values ​​in the density map is used as the number of objects to be counted.

2. The counting method according to claim 1, characterized in that, The number of bottleneck modules in the multiple bottleneck layers that iteratively extract object features from the first feature map increases progressively according to the first value. In the plurality of bottleneck layers, except for the first bottleneck module in the first type of bottleneck layer, the step size of the remaining bottleneck modules is the second step size. The first step length and the second step length are different.

3. The counting method of claim 2, wherein, Object feature extraction is performed using the first bottleneck module in the first type of bottleneck layer, including: In response to performing the Nth object feature extraction using the first bottleneck module in the Nth bottleneck layer, the second feature map is convolved to obtain the third feature map, and the second feature map is max-pooled to obtain the fourth feature map. The second feature map is obtained by performing the N-1th object feature extraction using the (N-1)th bottleneck layer, where N is a positive integer greater than or equal to 2. The third feature map is subjected to depthwise convolution separation, and the fourth feature map is subjected to channel expansion, so that the number of channels in the third feature map after depthwise convolution separation is the same as the number of channels in the fourth feature map after channel expansion. The third feature map after depthwise convolution is added to the fourth feature map after channel expansion, and the result is the feature map extracted from the first bottleneck module in the Nth bottleneck layer.

4. A counting device, characterized by include: The acquisition unit is used to acquire an image including the objects to be counted; The feature extraction unit is used to perform the first object feature extraction on the object to be counted in the image using a convolutional layer, so as to obtain the first feature map of the object to be counted. By using the bottleneck modules in multiple bottleneck layers, the object features are iteratively extracted from the first feature map in turn to obtain feature maps of the object to be counted at different scales. The feature map obtained from the last feature extraction is then separated by depthwise convolution to obtain the fifth feature map. The number of bottleneck modules stacked in each of the multiple bottleneck layers is different. The fifth feature map is concatenated with the partial feature map extracted from the first type of bottleneck layer, and then concatenated with the feature map obtained from the first feature extraction to obtain the fused feature map. The partial feature map extracted from the first type of bottleneck layer includes feature maps extracted in non-adjacent steps to the last feature extraction. The step size of the first bottleneck module in the first type of bottleneck layer among the multiple bottleneck layers is the first step size. The first type of bottleneck layer refers to the other bottleneck layers among the multiple bottleneck layers except for the first and last bottleneck layers where feature extraction is performed. The counting unit is used to extract features from the fused feature map using a depthwise separable convolutional model deployed on the mobile terminal. The extracted features are input into a sub-pixel convolutional model deployed on the mobile terminal. The extracted features are grouped according to channels, and the grouped features are sequentially concatenated point by point to obtain a density map. The density map represents the pixel distribution of the object to be counted in the image. The sum of the pixel values ​​in the density map is used as the number of the object to be counted.

5. A counting device, characterized by include: processor; Memory used to store processor-executable instructions; The processor is configured to execute the counting method according to any one of claims 1 to 3.

6. A storage medium, characterized by The storage medium stores instructions that, when executed by the processor of the mobile terminal, enable the mobile terminal to perform the counting method according to any one of claims 1 to 3.

Images