Training method of target counting model, target counting method and electronic device

Abstract

The present application relates to the technical field of image processing, and particularly relates to a target counting model training method, a target counting method and an electronic device, the training method comprising: obtaining a sample image and a density map label of the sample image; inputting the sample image into a pre-design counting model to obtain a predicted density map; performing loss calculation based on the predicted density map and the density map label to update parameters of the pre-design counting model and determine a target counting model. In the pre-design counting model, a convolution module comprises a linear mapping unit and a convolution unit, and by performing convolution processing only on the remaining part, the data processing amount of convolution can be reduced. Meanwhile, in order to make up for the problem of reduced feature extraction capability caused by the convolution module, global context feature extraction is performed by a global information extraction module and is applied to a target feature map, which can improve the accuracy of the obtained predicted density map, thereby improving the accuracy of the counting result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a target counting model training method, a target counting method and an electronic device. BACKGROUND

[0002] With the development of image processing technology, counting algorithms based on computer vision have become a popular research direction, and target counting algorithms have been widely used in industrial manufacturing and other fields. Obtaining the number of targets in a fixed area has become an indispensable part of many computer vision tasks.

[0003] For scenes such as crossroads, classrooms and factories where there are dense crowds or dense products, it is often necessary to count the number of target objects to guide the implementation of relevant measures for public safety and production management. If manual counting is used, the efficiency is low and the accuracy cannot be guaranteed, so most attempts use image processing methods to solve this problem.

[0004] Existing target counting methods are mainly based on trained counting models, that is, the counting model is used to extract the shallow features of the image, and the algorithm used is generally a traditional image processing algorithm. This shallow learning algorithm generally includes image preprocessing, feature extraction and target feature classification steps, and the core step is feature extraction. Representative related algorithms include Harr wavelet, gradient direction histogram HOG and scale invariant feature transform SIFT. However, these algorithms have obvious disadvantages. On the one hand, experienced experts need to manually adjust parameters for specific targets, and on the other hand, the robustness of shallow features is not high, and the anti-interference performance for light blur and other disturbances is poor, resulting in inaccurate counting results of the counting model. SUMMARY

[0005] Therefore, the embodiments of the present application provide a target counting model training method, a target counting method and an electronic device to solve the problem of low accuracy of the target counting model.

[0006] According to a first aspect, the embodiments of the present application provide a target counting model training method, comprising:

[0007] obtaining a sample image and a density map label of the sample image;

[0008] input the sample image into a pre-design counting model to obtain a predicted density map, the pre-design counting model comprising a convolution module, a processing module, and a global information extraction module, the convolution module being configured to extract features of the sample image to obtain target features, the target features comprising mapping features obtained by mapping a preset part of the sample image by a linear mapping unit in the convolution module and convolution features obtained by convolving a remaining part of the sample image other than the preset part by a convolution unit in the convolution module, the processing module being configured to process the target features to obtain a target feature map, and the global information extraction module being configured to extract global context features based on the target feature map and apply the global context features to the target feature map to obtain the predicted density map.

[0009] loss calculation is performed based on the predicted density map and the density map label to update parameters of the pre-design counting model to determine a target counting model.

[0010] The training method of the target counting model provided in the embodiments of the present application can reduce the data processing amount of convolution by mapping the preset part of the sample image and only convolving the remaining part in the convolution module comprising the linear mapping unit and the convolution unit, can improve the accuracy of the obtained predicted density map by extracting global context features by the global information extraction module and applying the global context features to the target feature map to make up for the reduction of feature extraction capability caused by the convolution module, and can improve the accuracy of the obtained target counting model by combining the density map label for loss calculation.

[0011] In some embodiments, the preset part is half of the sample image, and the convolution module is configured to extract features of the sample image to obtain target features, comprising:

[0012] linearly mapping half of the sample image by the linear mapping unit in the convolution module to obtain the mapping features;

[0013] convolving the other half of the sample image by the convolution unit in the convolution module to obtain the convolution features;

[0014] fusing the mapping features and the convolution features to obtain the target features.

[0015] The training method of the target counting model provided in the embodiments of the present application adopts linear mapping for half of the sample image and convolution for the other half, so that the calculation amount of the convolution module is greatly optimized.

[0016] In some embodiments, the extracting global context features from the second target feature by using the global information extraction module and acting on the second target feature to obtain the predicted density map comprises:

[0017] reconstructing the shape of the target feature map and expanding the dimension of the target feature map by using a first branch of the global information extraction module to obtain a first feature;

[0018] extracting global context features from the target feature map by using a second branch of the global information extraction module to obtain a second feature;

[0019] fusing the first feature and the second feature to obtain a third feature;

[0020] adding the third feature to the target feature map to obtain the predicted density map.

[0021] The training method of the target counting model provided by the embodiment of the application improves the reliability of the obtained predicted density map by aggregating global context features to the features at each position of the target feature map through global pooling and superposition operations of the global information extraction module.

[0022] In some embodiments, the loss calculation based on the predicted density map and the density map label is performed to update the parameters of the pre-designed density model to determine the target counting model, comprising:

[0023] obtaining an attention map label of the sample image;

[0024] performing attention processing on the predicted density map to obtain an attention map;

[0025] performing loss calculation based on the attention map and the attention map label to obtain a first loss value;

[0026] fusing the attention map and the predicted density map to obtain a fused predicted density map;

[0027] performing loss calculation based on the fused predicted density map and the density map label to obtain a second loss value;

[0028] fusing the first loss value and the second loss value to obtain a target loss value;

[0029] updating the parameters of the pre-designed density model based on the target loss value to determine the target counting model.

[0030] The training method of the target counting model provided in the embodiment of the present application can obtain an attention map by performing attention processing on the predicted density map, and the attention map is used to predict and distinguish foreground and background, so that the target and the background can be accurately distinguished, and therefore the attention map and the predicted density map are fused to assist in locating the target, and the accuracy of the prediction result of the target counting model obtained by training is further improved.

[0031] In some embodiments, the inputting the sample image into the pre-designed counting model to obtain a predicted density map further includes:

[0032] The processing result of the global information extraction module is up-sampled by using an up-sampling layer in the pre-designed counting model to determine the predicted density map.

[0033] The training method of the target counting model provided in the embodiment of the present application uses the up-sampling layer to ensure that the down-sampling multiple of the entire target counting model is small enough to reduce the information loss caused by the down-sampling, because the density map method is very sensitive to the size of the feature map.

[0034] In some embodiments, the obtaining of the sample image and the density map label of the sample image includes:

[0035] The sample image and the coordinates of each sample target in the sample image are obtained.

[0036] The density map label of the sample image is obtained by performing Gaussian kernel processing based on the coordinates of each sample target.

[0037] The training method of the target counting model provided in the embodiment of the present application uses the coordinates of each sample target to perform Gaussian kernel processing, because the Gaussian kernel is a linear kernel, and therefore the efficiency of generating the density map label of the sample image online can be ensured.

[0038] According to a second aspect, the embodiment of the present application further provides a target counting method, including:

[0039] Obtaining a to-be-processed image;

[0040] The to-be-processed image is input into a target counting model to obtain a target density map, and the number of targets in the to-be-processed image is determined by summing the target density map, and the target counting model is trained according to the training method of the target counting model of the first aspect or any one of the first aspect.

[0041] The target counting method provided in the embodiment of the present application can obtain an accurate target density map by processing the to-be-processed image on the basis of a reliable target counting model, and therefore the accuracy of the number of targets determined in the to-be-processed image can be ensured.

[0042] In some embodiments, the inputting the to-be-processed image into the target counting model obtains a target density map, and the number of targets in the to-be-processed image is determined by summing the target density map, comprising:

[0043] The to-be-processed image is divided to obtain a preset number of to-be-processed sub-images;

[0044] The preset number of to-be-processed sub-images are input into the target counting model to obtain target sub-density maps corresponding to the to-be-processed sub-images;

[0045] Each target sub-density map is summed to obtain a sum result, and the sum results are added to determine the number of targets in the to-be-processed image.

[0046] The target counting method provided by the embodiment of the application can better count the number of small targets in the to-be-processed image, and further improve the accuracy of the counting result.

[0047] According to a third aspect, the embodiment of the application further provides a training device of a target counting model, comprising:

[0048] The first acquisition module is configured to acquire a sample image and a density map label of the sample image;

[0049] The counting processing module is configured to input the sample image into a pre-design counting model to obtain a predicted density map, wherein the pre-design counting model comprises a convolution module, a processing module and a global information extraction module, the convolution module is configured to extract features of the sample image to obtain target features, the target features comprise mapping features obtained by mapping a preset part of the sample image by a linear mapping unit in the convolution module and convolution features obtained by convolving a remaining part of the sample image except the preset part by a convolution unit in the convolution module, the processing module is configured to process the target features to obtain a target feature map, and the global information extraction module is configured to extract global context features based on the target feature map and act on the target feature map to obtain the predicted density map;

[0050] The update module is configured to perform loss calculation based on the predicted density map and the density map label to update parameters of the pre-design counting model and determine a target counting model.

[0051] According to a fourth aspect, the embodiments of the present application further provide a target counting device, comprising:

[0052] a second obtaining module, configured to obtain a to-be-processed image;

[0053] a counting module, configured to input the to-be-processed image into a target counting model to obtain a target density map, and sum the target density map to determine the number of targets in the to-be-processed image, wherein the target counting model is obtained by training according to the training method of the target counting model of any one of the first aspect.

[0054] According to a fifth aspect, the embodiments of the present application provide an electronic device, comprising a memory and a processor, which are in communication connection with each other, the memory stores computer instructions, and the processor executes the computer instructions to execute the training method of the target counting model in the first aspect or any one of the embodiments of the first aspect, or executes the target counting method in the second aspect or any one of the embodiments of the second aspect.

[0055] According to a sixth aspect, the embodiments of the present application provide a computer readable storage medium, which stores computer instructions for causing the computer to execute the training method of the target counting model in the first aspect or any one of the embodiments of the first aspect, or execute the target counting method in the second aspect or any one of the embodiments of the second aspect.

[0056] It should be noted that the corresponding beneficial effects of the training device of the target counting model, the target counting device, the electronic device and the computer readable storage medium provided by the embodiments of the present application are described above in the description of the training method of the target counting model or the target counting method, and will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0058] Figure 1 is a flowchart of the training method of the target counting model according to the embodiments of the present application;

[0059] Figure 2 is a flowchart of the training method of the target counting model according to the embodiments of the present application;

[0060] Figure 3 is a structural schematic diagram of a convolution module according to an embodiment of the present application;

[0061] Figure 4 is a structural schematic diagram of a global information extraction module according to an embodiment of the present application;

[0062] Figure 5 is a structural schematic diagram of a pre-designed counting model training according to an embodiment of the present application;

[0063] Figure 6 is a flowchart of a target counting method according to an embodiment of the present application;

[0064] Figure 7 is a structural block diagram of a training device of a target counting model according to an embodiment of the present application;

[0065] Figure 8 is a structural block diagram of a target counting device according to an embodiment of the present application;

[0066] Figure 9 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0067] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in a clear and complete manner with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0068] The target counting method provided by the embodiments of the present application is a density map based method to obtain the number of targets. It should be noted that the target counting model in the embodiments of the present application can be used to count personnel, vehicles, annual inspection labels and other targets, and the specific counting target depends on the category of the target in the sample image used to train the pre-designed counting model. The specific counting target is not limited here, and can be set according to actual needs.

[0069] According to the embodiments of the present application, a target counting model training method and a target counting method are provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0070] In this embodiment, a target counting model training method is provided, which can be used in electronic devices such as computers, servers, etc.Figure 1 is a flowchart of a training method of a target counting model according to an embodiment of the present application, as shown in the figure, the flow includes the following steps: Figure 1

[0071] S11, obtain a sample image and a density map label of the sample image.

[0072] The sample image is a picture collected under different scenes, different illuminations and different angles, or a picture collected and processed to expand the sample, etc. The acquisition method of the sample image is not limited here, and is set according to actual needs.

[0073] The density map label of the sample image can be generated online after obtaining the sample image, for example, using an impact function, a static Gaussian kernel or a dynamic Gaussian kernel, etc. which is not limited here.

[0074] S12, input the sample image into a pre-designed density model to obtain a predicted density map.

[0075] The pre-designed density model includes a convolution module, a processing module and a global information extraction module. The convolution module is used to extract features of the sample image to obtain target features. The target features include mapping features obtained by mapping a preset part of the sample image by a linear mapping unit in the convolution module and convolution features obtained by convolving the remaining part of the sample image except the preset part by a convolution unit in the convolution module. The processing module is used to process the target features to obtain a target feature map. The global information extraction module is used to extract global context features based on the target feature map and act on the target feature map to obtain a predicted density map.

[0076] The input of the pre-designed density model is an image, and the output is a predicted density map of the image. The pre-designed density model can be established based on an image classification model, for example, Inception-V3, Inception-V2, etc. The pre-designed density model is obtained by adjusting the image classification model.

[0077] Specifically, the pre-designed density model includes a convolution module, a processing module and a global information extraction module. The convolution module is used to extract features of the sample image to obtain target features. The convolution module includes a linear mapping unit and a convolution unit. The linear mapping unit is used to linearly map a preset part of the sample image to obtain mapping features, and the convolution unit is used to convolve the part of the sample image except the preset part to obtain convolution features. Therefore, the target features include the mapping features and the convolution features.

[0078] ​The linear mapping can be understood as copying a preset part of the sample image between the input and the output, wherein the preset part of the sample image is not particularly the image itself, but a preset part of an input vector formed by each channel of the sample image. For example, the input sample image is represented as a 50*50*50 matrix, the preset part can be a 20*50*50 matrix, and the remaining part is a 30*50*50 matrix. Of course, the specific size of the preset part is set according to actual needs, and it is not limited here.

[0079] The processing module is used to process the target feature to obtain a target feature map. The processing module is a module for other processing functions, including but not limited to an Inception module and the like. The specific structure depends on the structure of the image classification model on which the preset number module depends, and it is not limited here.

[0080] The input of the global information extraction module is the output of the processing module, that is, the target feature map. The global information extraction module is used to extract global context features based on the target feature map, and to act the extracted features on the target feature map to obtain a predicted density map.

[0081] S13, loss calculation is performed based on the predicted density map and the density map label to update the parameters of the preset number model, and a target counting model is determined.

[0082] The parameters of the preset number model are fixed after multiple rounds of iteration, and the target counting model is determined. When performing loss calculation, the loss function used is selected according to actual needs, and it is not limited here.

[0083] It should be noted that the training end condition can be setting a target iteration number, when the training number reaches the target iteration number, it means that the training is completed; or setting a loss threshold, if the calculated loss value is less than the loss threshold, it means that the training is completed. The target iteration number and the loss threshold are set according to actual needs, and it is not limited here.

[0084] The training method of the target counting model provided in the embodiment can reduce the data processing amount of convolution by performing mapping processing on a preset part of a sample image and performing convolution processing only on the remaining part. Meanwhile, in order to make up for the problem of reduced feature extraction capability caused by the convolution module, global context feature extraction is performed by using a global information extraction module and is applied to a target feature map, so as to improve the accuracy of the obtained prediction density map. On this basis, loss calculation is performed in combination with a density map label, and the accuracy of the obtained target counting model is improved.

[0085] In the embodiment, a training method of a target counting model is provided, which can be used in electronic devices such as computers and servers. Figure 2 The training method of the target counting model according to the embodiment of the application is shown in a flowchart as shown in Figure 2 The flowchart includes the following steps:

[0086] S21, obtaining a sample image and a density map label of the sample image.

[0087] In some embodiments, the above S21 includes:

[0088] (1) obtaining a sample image and coordinates of each sample target in the sample image.

[0089] (2) performing Gaussian kernel processing based on the coordinates of each sample target to obtain a density map label of the sample image.

[0090] The sample image is obtained, and the label file corresponding to each sample image contains the coordinates of each target. The density map label is generated online by using the Gaussian kernel based on the coordinates of each sample target. For example, a 15x15 Gaussian kernel is used.

[0091] The sample image can be obtained by first collecting pictures of different scenes, different illuminations and different angles to obtain original pictures, and then labeling the original pictures to determine the coordinates of each sample target in the original pictures. In order to preserve the original information of the sample target, a fixed size picture is obtained by randomly cutting the original picture from the original picture.

[0092] The coordinates of each sample target are used for Gaussian kernel processing. Since the Gaussian kernel is a linear kernel, the efficiency of online generation of the density map label of the sample image can be ensured.

[0093] In some embodiments, since the density map is sensitive to the change of the target shape, the input aspect ratio during use is ensured to be consistent with that during training. During training, if the length and width of the collected image are not fixed, it has a greater impact on the final density map result. Therefore, in order to ensure that the aspect ratio of the sample image used for training is unchanged, a certain expansion to the preset aspect ratio of the collected image is required to obtain a sample image, wherein the preset ratio can be 1:1, etc. The specific expansion rule is as follows: expand outward along the shorter side down or to the right until the height and width are equal, if the length of the expanded side is less than half of the input size, then expand both sides down and to the right to half of the input size, and finally, in order to ensure the center symmetry, shift the target bounding box left or right, and ensure that it does not exceed the image boundary. The reason for such expansion processing is to ensure that the target object is as possible as in the middle of the image, avoiding the statistical loss caused by the target object existing in the edge part, and improving the accuracy of the subsequent target counting model obtained by training.

[0094] S22, inputting the sample image into the pre-designed counting model to obtain a predicted density map.

[0095] The pre-designed counting model includes a convolution module, a processing module, and a global information extraction module. The convolution module is used to extract features of the sample image to obtain target features. The target features include mapping features obtained by mapping a preset part of the sample image by a linear mapping unit in the convolution module, and convolution features obtained by convolving the remaining part of the sample image except the preset part by a convolution unit in the convolution module. The processing module is used to process the target features to obtain a target feature map. The global information extraction module is used to extract global context features based on the target feature map and act on the target feature map to obtain a predicted density map.

[0096] If the pre-designed counting model is constructed based on the Inception-V3 model, all inception modules in the Inception-V3 model are retained, the ordinary convolution operation in the Inception-V3 model is replaced by the convolution module (including the linear mapping unit and the convolution unit), and the last two layers of the maximum pooling layer are replaced by the global information extraction module to better extract global features.

[0097] Specifically, the preset part is half of the sample image. Based on this, S22 includes:

[0098] S221, linearly mapping half of the sample image by a linear mapping unit in the convolution module to obtain mapping features.

[0099] S222, convolving the other half of the sample image using a convolution unit in the convolution module to obtain a convolution feature.

[0100] S223, fusing the mapping feature and the convolution feature to obtain a target feature.

[0101] As shown in Figure 3 , for the input sample image, one half is linearly mapped to obtain a mapping feature, and the other half is convolved to obtain a convolution feature. For example, the shape of the input sample image is 50*50*50, then 25*50*50 of them are input to the linear mapping unit to obtain the mapping feature, and the other 25*50*50 are input to the convolution unit to obtain the convolution feature.

[0102] The fusion manner of the mapping feature and the convolution feature includes but is not limited to splicing or adding, etc. After the fusion processing, the target feature is obtained.

[0103] S224, processing the target feature using the processing module to obtain a target feature map.

[0104] The description of the processing module is described above and will not be repeated here. That is, after the processing of the processing module, the target feature map is obtained.

[0105] S225, using the global information extraction module to extract the global context feature of the target feature map and act on the target feature map to obtain a predicted density map.

[0106] In some embodiments, as shown in Figure 4 , the above S225 includes:

[0107] (1) using the first branch of the global information extraction module to perform shape reconstruction and dimension expansion on the target feature map to obtain a first feature.

[0108] (2) using the second branch of the global information extraction module to perform global context feature extraction on the target feature map to obtain a second feature.

[0109] (3) fusing the first feature and the second feature to obtain a third feature.

[0110] (4) adding the third feature and the target feature map to obtain a predicted density map.

[0111] As shown in Figure 4The Input is a target feature map, the first branch is View and Unsqueeze, that is, View reconstructs the shape of the target feature map from CxHxW to CxHW, and Unsqueese expands CxHW by one dimension to reconstruct 1xCxHW from CxHW; the second branch is the branch where softmax is located, and the second branch is used for global context feature extraction on the target feature map to obtain a second feature. It should be noted that the global pooling in the second branch is not limited to softmax, and other ways can also be used, and no limitation is made here, and the specific setting can be made according to actual needs.

[0112] It should be noted that, in addition to the manner shown in Figure 4 , the first branch can also use a convolution processing manner; the softmax in the second branch is used to realize global pooling, and in addition to using softmax, global maximum pooling or global maximum pooling implementation and the like can also be used.

[0113] The first feature is obtained through the processing of the first branch, and the second feature is obtained through the processing of the second branch. The purpose of the processing of the first branch is to be able to perform matrix calculation with the second feature obtained in the second branch, and then to convert the shape of the second feature. After the first feature and the second feature are fused, a third feature is obtained. At this time, the third feature is the global context feature, and then the third feature is added to the target feature map to realize the aggregation of the global context feature to the feature at each position on the target feature map. After the processing of the global information extraction module, the prediction density map can be obtained.

[0114] The global information extraction module aggregates the global context feature to the feature at each position on the target feature map through global pooling and superposition operation, which improves the reliability of the obtained prediction density map.

[0115] In some embodiments, as Figure 5 shown, the S22 further includes: using an up-sampling layer in the pre-designed density model to up-sample the processing result of the global information extraction module to determine the prediction density map. As Figure 5 shown, an up-sampling layer is further connected after the output of the global information extraction module, which is used to up-sample the processing result of the global information extraction module to determine the prediction density map. At this time, the pre-designed density model is constructed based on the Inception-V3 model, all inception modules are retained, the ordinary convolution operation in the inception module is replaced by the convolution module described above, the last two maximum pooling layers are replaced by the global information extraction module to better extract global features, and an up-sampling layer, for example, a 2-fold up-sampling layer, is added at the end. The final model has a down-sampling multiple of 4.

[0116] Since the density map method is sensitive to the size of the feature map, an up-sampling layer is used to ensure that the down-sampling multiple of the entire target counting model is small enough to reduce the information loss caused by down-sampling.

[0117] S23, loss calculation based on the predicted density map and the density map label to update the parameters of the pre-designed counting model to determine the target counting model.

[0118] In some embodiments, as shown in Figure 5 The above S23 includes:

[0119] (1) Obtain the attention map label of the sample image.

[0120] (2) Perform attention processing on the predicted density map to obtain an attention map.

[0121] (3) Perform loss calculation based on the attention map and the attention map label to obtain a first loss value.

[0122] (4) Fuse the attention map and the predicted density map to obtain a fused predicted density map.

[0123] (5) Perform loss calculation based on the fused predicted density map and the density map label to obtain a second loss value.

[0124] (6) Fuse the first loss value and the second loss value to obtain a target loss value.

[0125] (7) Update the parameters of the pre-designed counting model based on the target loss value to determine the target counting model.

[0126] As shown in Figure 5 The output result of the up-sampling layer is referred to as the predicted density map. An additional path is introduced from the output of the up-sampling layer for attention processing to obtain an attention map. The attention map and the predicted density map are fused to obtain a fused predicted density map. Based on this, when calculating the loss value, two loss values are involved, i.e., a first loss value corresponding to the attention map and a second loss value corresponding to the fused predicted density map.

[0127] Specifically, the attention map label of the sample image is obtained. The determination method of the attention map label is similar to the determination method of the density map label described above, and both are generated online using a Gaussian kernel. As described above, a 15x15 Gaussian kernel is used to generate the density map label online; accordingly, a 25x25 Gaussian kernel is used to generate the attention map label online. Since attention only needs to distinguish between foreground and background and does not need to be very specific, the size of the Gaussian kernel is less demanding. In an embodiment, the attention map label and the density map label are generated simultaneously using different Gaussian kernels.

[0128] After obtaining the first loss value corresponding to the attention map and the second loss value corresponding to the density map respectively, the first loss value and the second loss value are fused to obtain a target loss value. The fusion manner includes but is not limited to weighted fusion, and is specifically set according to actual needs.

[0129] On the basis of the determined target loss value, the parameters of the pre-designed counting model are updated. After multiple rounds of iteration, the parameters of the pre-designed counting model are finally fixed, and a target counting model is determined.

[0130] As shown in the embodiment shown in Figure 5 The loss during training is divided into two parts: the mean square error loss is used for the density map part to better regress the number of pedestrians; for the attention part, considering that its main function is to distinguish foreground and background, the accuracy requirement is not so high, so the binary cross entropy loss is used. Among them, due to the fusion of the attention map and the predicted density map, the loss of the attention part also assists in guiding the training of the backbone model to some extent, and the model result converges to the extreme point better and faster.

[0131] The attention map is obtained by attention processing on the predicted density map, which is used to predict and distinguish foreground and background, so as to accurately distinguish the target and the background, and therefore the attention map and the predicted density map are fused to assist in positioning the target, further improving the accuracy of the prediction result of the target counting model obtained by training.

[0132] The training method of the target counting model provided in the embodiment uses linear mapping for half of the sample images and convolution for the other half, so that the computational amount of the convolution module is greatly optimized; at the same time, a global information extraction module is introduced to improve the feature extraction capability and ensure the reliability of the predicted density map.

[0133] In the embodiment, a target counting method is provided, which can be used in electronic devices such as computers, servers, etc. Figure 6 The flowchart of the target counting method according to the embodiment of the present application is shown in Figure 6 The flowchart of the target counting method according to the embodiment of the present application is shown in

[0134] S31, obtaining an image to be processed.

[0135] The source of the image to be processed can be obtained from an image acquisition device by an electronic device, or obtained from a third-party device, or obtained in other ways, which is not limited here and can be set according to actual needs.

[0136] S32, input the to-be-processed image into the target counting model to obtain a target density map, and sum the target density map to determine the number of targets in the to-be-processed image.

[0137] The target counting model is obtained by training according to the training method of the target counting model, and specific structural details are described above and will not be repeated here.

[0138] The target counting model is used to process the to-be-processed image to obtain a target density map. Summing the target density map can determine the number of targets in the to-be-processed image.

[0139] The target counting method provided in this embodiment can obtain an accurate target density map by processing the to-be-processed image based on a reliable target counting model, thereby ensuring the accuracy of the determined number of targets in the to-be-processed image.

[0140] In some embodiments, S32 includes:

[0141] (1) dividing the to-be-processed image to obtain a preset number of to-be-processed sub-images.

[0142] (2) inputting the preset number of to-be-processed sub-images into the target counting model to obtain target sub-density maps corresponding to the to-be-processed sub-images.

[0143] (3) summing each target sub-density map to obtain a sum result, and adding the sum results to determine the number of targets in the to-be-processed image.

[0144] In order to better count the number of small targets, the to-be-processed image is first divided to obtain a preset number of to-be-processed sub-images, and then the preset number of to-be-processed sub-images are sent into the target counting model in parallel for inference to obtain a preset number of target sub-density maps. The sum result is obtained by summing each target sub-density map, and the sum result is the number of targets in the target sub-density map. Finally, the counting result of the whole image is obtained by adding the number of targets in each target sub-density map to obtain the number of targets in the to-be-processed image.

[0145] The division of the to-be-processed image includes but is not limited to equal division, for example, being divided into 2 to-be-processed sub-images on average, or being divided into 4 to-be-processed sub-images on average, and the like. The specific division is set according to actual needs.

[0146] Further, the preset number of to-be-processed sub-images can be input or input in parallel into the target counting model to obtain target sub-density maps corresponding to the to-be-processed sub-images.

[0147] For small targets in the image to be processed, the proportion of the small targets in the image to be processed is small, and based on this, a preset number of to-be-processed sub-images are obtained by dividing the image to be processed. Then, the proportion of the small targets in the to-be-processed sub-image is equivalent to the proportion of the small targets in the entire to-be-processed image, which will be larger. Therefore, the to-be-processed sub-image is input into the target counting model for counting, so that the number of small targets can be better counted, and the accuracy of the counting result is further improved.

[0148] The target counting model provided in the embodiments of the present application is applied to various scenes that need to count the number, including but not limited to classroom number counting, factory part counting, and annual inspection mark counting, etc. Under the premise of ensuring sufficient training data, good results can be obtained, and the entire model does not need complex post-processing, only needs to perform simple summation, and the overall model structure is simple and convenient to deploy.

[0149] In the embodiments, a target counting model training device and a target counting device are also provided. The device is used to implement the above embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware is also possible and is contemplated.

[0150] The embodiments provide a target counting model training device, as shown in Figure 7 The device includes:

[0151] The first acquisition module 41 is configured to acquire a sample image and a density map label of the sample image.

[0152] The counting processing module 42 is configured to input the sample image into a pre-design counting model to obtain a predicted density map. The pre-design counting model includes a convolution module, a processing module, and a global information extraction module. The convolution module is configured to perform feature extraction on the sample image to obtain target features. The target features include mapping features obtained by mapping a preset part of the sample image by a linear mapping unit in the convolution module and convolution features obtained by convolving a remaining part of the sample image except the preset part by a convolution unit in the convolution module. The processing module is configured to process the target features to obtain a target feature map. The global information extraction module is configured to extract global context features based on the target feature map and act on the target feature map to obtain the predicted density map.

[0153] The updating module 43 is configured to perform loss calculation based on the predicted density map and the density map label to update parameters of the pre-design counting model and determine a target counting model.

[0154] In some embodiments, the preset part is half of the sample image, and the target feature determination module comprises:

[0155] a first processing unit configured to perform linear mapping on the half of the sample image by using a linear mapping unit in the convolution module to obtain the mapping feature;

[0156] a first processing unit configured to perform linear mapping on the half of the sample image by using a linear mapping unit in the convolution module to obtain the mapping feature;

[0157] a first fusion unit configured to fuse the mapping feature and the convolution feature to obtain the target feature.

[0158] In some embodiments, the counting processing module comprises:

[0159] a first sub-processing unit configured to perform shape reconstruction and dimension expansion on the target feature map by using a first branch of the global information extraction module to obtain a first feature;

[0160] a second sub-processing unit configured to perform global context feature extraction on the target feature map by using a second branch of the global information extraction module to obtain a second feature;

[0161] a fusion sub-unit configured to fuse the first feature and the second feature to obtain a third feature;

[0162] an addition sub-unit configured to add the third feature and the target feature map to obtain the predicted density map.

[0163] In some embodiments, the updating module 43 comprises:

[0164] a first acquisition unit configured to acquire an attention map label of the sample image;

[0165] an attention unit configured to perform attention processing on the predicted density map to obtain an attention map;

[0166] a first loss calculation unit configured to perform loss calculation based on the attention map and the attention map label to obtain a first loss value;

[0167] a second fusion unit configured to fuse the attention map and the predicted density map to obtain a fused predicted density map;

[0168] a second loss calculation unit configured to perform loss calculation based on the fused predicted density map and the density map label to obtain a second loss value;

[0169] a third fusion unit, configured to fuse the first loss value and the second loss value to obtain a target loss value;

[0170] an updating unit, configured to update parameters of the pre-designed counting model based on the target loss value to determine the target counting model.

[0171] In some embodiments, the counting processing module further includes:

[0172] an up-sampling unit, configured to up-sample the processing result of the global information extraction module by using an up-sampling layer in the pre-designed counting model to determine the prediction density map.

[0173] In some embodiments, the first obtaining module includes:

[0174] a second obtaining unit, configured to obtain the sample image and coordinates of each sample target in the sample image;

[0175] a Gaussian kernel processing unit, configured to perform Gaussian kernel processing based on the coordinates of the each sample target to obtain a density map label of the sample image.

[0176] The embodiment provides a target counting device, as shown in the accompanying drawings, which includes: Figure 8

[0177] a second obtaining module 51, configured to obtain a to-be-processed image;

[0178] a counting module 52, configured to input the to-be-processed image into a target counting model to obtain a target density map, and sum the target density map to determine a number of targets in the to-be-processed image, wherein the target counting model is obtained by training according to the training method of the target counting model.

[0179] In some embodiments, the counting module 52 includes:

[0180] a dividing unit, configured to divide the to-be-processed image to obtain a preset number of to-be-processed sub-images;

[0181] a counting unit, configured to input the preset number of to-be-processed sub-images into the target counting model to obtain target sub-density maps corresponding to the to-be-processed sub-images;

[0182] a determining unit, configured to sum each target sub-density map to obtain a sum result, and add the sum results to determine the number of targets in the to-be-processed image.

[0183] ​In this embodiment, the training device and the target counting device of the target counting model are presented in the form of functional units. Here, a unit refers to an ASIC circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0184] Further functional descriptions of the above modules are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0185] This invention also provides an electronic device having the above-described features. Figure 7 The training device for the target counting model shown, or Figure 8 The target counting device shown.

[0186] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device provided in an optional embodiment of the present invention, such as... Figure 9 As shown, the electronic device may include: at least one processor 61, such as a CPU (Central Processing Unit), at least one communication interface 63, memory 64, and at least one communication bus 62. The communication bus 62 is used to enable communication between these components. The communication interface 63 may include a display screen or a keyboard; optionally, the communication interface 63 may also include a standard wired interface or a wireless interface. The memory 64 may be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 64 may also be at least one storage device located remotely from the aforementioned processor 61. The processor 61 may be combined with... Figure 7 or Figure 8 The described apparatus has an application program stored in memory 64, and the processor 61 calls the program code stored in memory 64 to perform any of the above method steps.

[0187] The communication bus 62 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 62 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0188] The memory 64 can include a volatile memory, such as a random-access memory (RAM), and / or a non-volatile memory, such as a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD). The memory 64 can also include a combination of the above-mentioned types of memories.

[0189] The processor 61 can be a central processing unit (CPU), a network processor (NP), or a combination thereof.

[0190] The processor 61 can further include a hardware chip. The hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0191] Optionally, the memory 64 is further configured to store program instructions. The processor 61 can invoke the program instructions to implement the training method of the target counting model as shown in any of the embodiments of the present application, or the target counting method as shown in any of the embodiments of the present application.

[0192] The embodiment of the present application further provides a non-transitory computer storage medium, which stores computer executable instructions, and the computer executable instructions can execute the training method of the target counting model or the target counting method in any method embodiment described above. The storage medium can be a magnetic disc, an optical disc, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory, a Hard Disk Drive (HDD) or a Solid-State Drive (SSD), etc. The storage medium can also include a combination of the above-mentioned types of memories.

[0193] Although the embodiments of the present application are described in conjunction with the drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. A training method for a target counting model, characterized in that, The method comprises the following steps: obtaining a sample image and a density map label of the sample image; inputting the sample image into a pre-designed counting model to obtain a predicted density map, wherein the pre-designed counting model comprises a convolution module, a processing module and a global information extraction module, the convolution module is used for feature extraction of the sample image to obtain a target feature, the target feature comprises a mapping feature obtained by mapping a preset part of the sample image by a linear mapping unit in the convolution module and a convolution feature obtained by convolving a remaining part of the sample image except the preset part by a convolution unit in the convolution module, the processing module is used for processing the target feature to obtain a target feature map, and the global information extraction module is used for extracting a global context feature based on the target feature map and acting on the target feature map to obtain the predicted density map; loss calculation is performed based on the predicted density map and the density map label to update parameters of the pre-designed counting model and determine a target counting model; wherein the loss calculation based on the predicted density map and the density map label to update the parameters of the pre-designed counting model and determine the target counting model comprises: obtaining an attention map label of the sample image; attention processing is performed on the predicted density map to obtain an attention map; loss calculation is performed based on the attention map and the attention map label to obtain a first loss value; the attention map and the predicted density map are fused to obtain a fused predicted density map; loss calculation is performed based on the fused predicted density map and the density map label to obtain a second loss value; the first loss value and the second loss value are fused to obtain a target loss value; the parameters of the pre-designed counting model are updated based on the target loss value to determine the target counting model.

2. The method of claim 1, wherein, The preset part is half of the sample image, and the convolution module is used for feature extraction of the sample image to obtain a target feature, which comprises: linear mapping of half of the sample image by a linear mapping unit in the convolution module to obtain the mapping feature; convolution of the other half of the sample image by a convolution unit in the convolution module to obtain the convolution feature; fusion of the mapping feature and the convolution feature to obtain the target feature.

3. The method of claim 1, wherein, The global context feature is extracted based on the target feature map and acts on the target feature map to obtain the predicted density map, which comprises: shape reconstruction and dimension expansion of the target feature map by a first branch of the global information extraction module to obtain a first feature; global context feature extraction of the target feature map by a second branch of the global information extraction module to obtain a second feature; fusion of the first feature and the second feature to obtain a third feature; addition of the third feature and the target feature map to obtain the predicted density map.

4. The method of claim 1, wherein, The sample image is input into the pre-designed counting model to obtain the predicted density map, which further comprises: The up-sampling layer in the pre-design model is used to up-sample the processing result of the global information extraction module to determine the prediction density map.

5. The method according to any one of claims 1-4, characterized in that, The sample image and the density map label of the sample image include: Obtaining the sample image and the coordinates of each sample target in the sample image; Based on the coordinates of each sample target, Gaussian kernel processing is performed to obtain the density map label of the sample image.

6. A target counting method characterized by, It includes: Obtaining a to-be-processed image; The to-be-processed image is input into the target counting model to obtain a target density map, and the target density map is summed to determine the number of targets in the to-be-processed image, and the target counting model is obtained by training according to the training method of the target counting model in any one of claims 1-5.

7. The method of claim 6, wherein, The to-be-processed image is input into the target counting model to obtain a target density map, and the target density map is summed to determine the number of targets in the to-be-processed image, and the target counting model is obtained by training according to the training method of the target counting model in any one of claims 1-5. It includes: Dividing the to-be-processed image to obtain a preset number of to-be-processed sub-images; The preset number of to-be-processed sub-images are input into the target counting model to obtain target sub-density maps corresponding to the to-be-processed sub-images; 8. An electronic device, comprising: Each target sub-density map is summed to obtain a sum result, and the sum results are added to determine the number of targets in the to-be-processed image. It includes:

9. A computer-readable storage medium, characterized in that, A memory and a processor, which are connected in communication with each other, the memory stores computer instructions, and the processor executes the computer instructions to execute the training method of the target counting model in any one of claims 1-5, or execute the target counting method in claim 6 or 7. The computer readable storage medium stores computer instructions for causing a computer to execute the training method of the target counting model in any one of claims 1-5, or execute the target counting method in claim 6 or 7.

Images