Target counting method and device, electronic device, and storage medium

Abstract

The application provides a target counting method and device, electronic equipment and a computer readable storage medium. The method comprises: inputting a to-be-processed image into a target counting network constructed based on U-Net which has been trained; performing encoding processing on the to-be-processed image by an encoder of the target counting network; performing decoding processing on a plurality of image features obtained by encoding by a decoder of the target counting network to obtain a density map of the to-be-processed image; wherein the target counting network is trained as a generator in a generative adversarial mechanism; and performing integral summation on the density map to obtain a total number of targets in the to-be-processed image. In the application, the target counting network based on U-Net is introduced into the target counting task, the target counting network is trained through the generative adversarial mechanism, the target counting network can accurately calculate the density map of the to-be-processed image, and the total number of targets in the to-be-processed image can be obtained through integral summation of the density map.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a target counting method and device, electronic equipment and computer readable storage medium. BACKGROUND

[0002] Most alloy materials will experience a solidification process during the melting and casting process. During the solidification process, the liquid alloy gradually precipitates solid phase grains, and as the temperature drops, the grains continuously nucleate, grow and coarsen in different forms, gradually forming dendritic structure with complex morphology and complex spatial structure. At the same time, complex intermetallic compounds, secondary precipitates, cracks, shrinkage holes and other structural defects are formed in the metal solidification process. The microstructure formed by the metal in the solidification process has a decisive role on the final performance of the alloy material. At present, X-ray imaging technology can be used to collect synchrotron radiation images of alloy samples at each stage of the solidification process, so as to study the microstructure in the solidification process according to the images. Grain counting is one of the research contents. SUMMARY

[0003] The purpose of the embodiments of the present application is to provide a target counting method and device, electronic equipment and computer readable storage medium, which are used to obtain a density map by encoding and decoding processing an image based on a target counting network trained under a generative adversarial mechanism, and determine the total number of targets in the image based on the density map.

[0004] In one aspect, the present application provides a target counting method, comprising:

[0005] inputting a to-be-processed image into a trained target counting network based on U-Net construction, performing encoding processing on the to-be-processed image by an encoder of the target counting network, performing decoding processing on a plurality of image features obtained by encoding by a decoder of the target counting network, and obtaining a density map of the to-be-processed image; wherein the target counting network is trained as a generator in a generative adversarial mechanism;

[0006] integrating and summing the density map to obtain the total number of targets in the to-be-processed image.

[0007] In one embodiment, the encoder includes a plurality of multi-scale down-sampling modules.

[0008] The encoding processing by the encoder of the target counting network comprises:

[0009] performing step-by-step encoding processing on the to-be-processed image by the encoder to obtain a plurality of levels of image features; wherein in the encoding processing, the multi-scale down-sampling modules are used to perform multi-scale convolution calculation and aggregation on the image features of the previous level to obtain transition image features used to encode the image features of the next level.

[0010] In an embodiment, the multi-scale down-sampling module comprises a first branch, a second branch and a third branch, the first branch comprises a max-pooling layer and a 1*1 convolution layer, the second branch comprises a 1*1 convolution layer, a 3*3 deformable convolution layer and a 3*3 convolution layer, and the third branch comprises a 1*1 convolution layer, a 3*3 deformable convolution layer and two consecutive 3*3 convolution layers.

[0011] The multi-scale convolution calculation and aggregation of the image features of the previous stage by the multi-scale down-sampling module obtain a transition image feature used for encoding the image features of the next stage, comprising:

[0012] The first sub-transition image feature is calculated from the image features of the previous stage by the first branch;

[0013] The second sub-transition image feature is calculated from the image features of the previous stage by the second branch;

[0014] The third sub-transition image feature is calculated from the image features of the previous stage by the third branch;

[0015] The first sub-transition image feature, the second sub-transition image feature and the third sub-transition image feature are fused into the transition image feature.

[0016] In an embodiment, the target counting network is trained in the following manner:

[0017] A sample image in a sample data set is input into a generator of a generative adversarial network to obtain a hypothetical density map output by the generator;

[0018] Based on the sample image, the hypothetical density map corresponding to the sample image and the label density map carried by the sample image, a discriminator of the generative adversarial network is trained;

[0019] After training the discriminator, the generator is trained based on the sample image and the hypothetical density map;

[0020] The above process is repeated until the generative adversarial network converges, and the generator of the generative adversarial network is taken as a target counting network.

[0021] In an embodiment, the discriminator comprises a first sub-discriminator and a second sub-discriminator;

[0022] The training of the discriminator of the generative adversarial network based on the sample image, the hypothetical density map corresponding to the sample image and the label density map carried by the sample image comprises:

[0023] constructing a sample image-hypothetical density map pair, a sample image-label density map pair based on the sample image, the hypothetical density map corresponding to the sample image and the label density map carried by the sample image;

[0024] constructing a local image-local hypothetical density map pair, a local image-local label density map pair based on the local position information carried by the sample image, the sample image, the hypothetical density map corresponding to the sample image and the label density map carried by the sample image;

[0025] training the first sub-discriminator based on the sample image-hypothetical density map pair and the sample image-label density map pair;

[0026] training the second sub-discriminator based on the local image-local hypothetical density map pair and the local image-local label density map pair.

[0027] In an embodiment, after training the discriminator, the generator is trained based on the sample image and the hypothetical density map, comprising:

[0028] After training the first sub-discriminator and the second sub-discriminator, the generator is trained based on the sample image-hypothetical density map pair and the local image-local hypothetical density map pair.

[0029] In an embodiment, before training the target counting network, the method further comprises:

[0030] obtaining a plurality of sample images and target position information annotated for each sample image; wherein the target position information comprises target center position and target radius information;

[0031] For each sample image, the target position information in the sample image is calculated by a Gaussian kernel function that is adaptive based on the target radius information, to obtain a label density map of the sample image;

[0032] constructing the sample data set based on a plurality of sample images carrying label density maps.

[0033] On the other hand, the present application also provides a target counting device, comprising:

[0034] The first computing module is configured to input an image to be processed into a target counting network constructed based on U-Net which has been trained, encode and process the image to be processed by an encoder of the target counting network, and decode and process a plurality of image features obtained by encoding by a decoder of the target counting network to obtain a density map of the image to be processed; wherein the target counting network is trained as a generator in a generative adversarial mechanism;

[0035] The second calculation module is configured to integrate and sum the density map to obtain a total number of the targets in the image to be processed.

[0036] Further, the present application also provides an electronic device, which comprises:

[0037] a processor;

[0038] a memory for storing processor-executable instructions;

[0039] The processor is configured to execute the target counting method.

[0040] In addition, the present application also provides a computer-readable storage medium, which stores a computer program executable by a processor to complete the target counting method.

[0041] In the present application, the U-Net-based target counting network is introduced into the target counting task, and the target counting network is trained through the generative adversarial mechanism, so that the target counting network can accurately calculate the density map of the image to be processed, and the total number of targets in the image to be processed can be obtained through the integration and summation of the density map. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly introduced.

[0043] Figure 1 The application scenario of the target counting method provided by an embodiment of the present application is shown in the figure;

[0044] Figure 2 The structure of the electronic device provided by an embodiment of the present application is shown in the figure;

[0045] Figure 3 The flowchart of the target counting method provided by an embodiment of the present application is shown in the figure;

[0046] Figure 4 The schematic diagram of the image to be processed and the density map provided by an embodiment of the present application is shown in the figure;

[0047] Figure 5 The structure of the multi-scale down-sampling module provided by an embodiment of the present application is shown in the figure;

[0048] Figure 6 The flowchart of the multi-scale coding process provided by an embodiment of the present application is shown in the figure;

[0049] Figure 7 The structure of the target counting network provided by an embodiment of the present application is shown in the figure;

[0050] Figure 8 A schematic diagram of the training process of a target counting network provided in an embodiment of this application;

[0051] Figure 9 This is a schematic diagram of a dual-channel discriminator provided in an embodiment of this application;

[0052] Figure 10 This is a schematic diagram of the training process of a discriminator provided in one embodiment of this application;

[0053] Figure 11 A flowchart illustrating a method for constructing a sample dataset according to an embodiment of this application;

[0054] Figure 12 This is a schematic diagram of density map comparison results provided in an embodiment of this application;

[0055] Figure 13 A block diagram of a target counting device provided in an embodiment of this application. Detailed Implementation

[0056] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0057] Similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0058] Figure 1 This is a schematic diagram illustrating an application scenario of the target counting method provided in the embodiments of this application. For example... Figure 1 As shown, the application scenario includes a client 20 and a server 30. The client 20 can be a user terminal such as a host, mobile phone, or tablet computer, used to send images to be processed to the server 30 for performing target counting tasks. The server 30 can be a server, server cluster, or cloud computing center, which can obtain the images sent by the client 20 and perform target counting tasks on the images to obtain the total number of targets in the images to be processed.

[0059] like Figure 2 As shown, this embodiment provides an electronic device 1, including: at least one processor 11 and a memory 12. Figure 2 Taking a processor 11 as an example, the processor 11 and memory 12 are connected via a bus 10. The memory 12 stores instructions that can be executed by the processor 11. The instructions are executed by the processor 11 to enable the electronic device 1 to perform all or part of the process of the method in the following embodiments. In one embodiment, the electronic device 1 may be the server 30 described above, used to execute the target counting method.

[0060] The memory 12 can be implemented by any type of volatile or nonvolatile storage devices or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0061] The application also provides a computer readable storage medium, the storage medium stores a computer program, and the computer program can be executed by the processor 11 to complete the target counting method provided by the application.

[0062] Referring to Figure 3 , a flowchart of the target counting method provided by an embodiment of the application is shown in Figure 3 , the method can include the following steps 310-320.

[0063] Step 310: input the to-be-processed image into the trained target counting network to obtain a density map of the to-be-processed image output by the target counting network. The to-be-processed image is input into the trained target counting network based on U-Net, the encoder of the target counting network is used for encoding processing, the decoder of the target counting network is used for decoding processing on the multiple image features obtained by encoding, and a density map of the to-be-processed image is obtained; wherein the target counting network is trained as a generator in a generative adversarial mechanism.

[0064] The to-be-processed image is an image that needs to be subjected to target counting, and the to-be-processed image contains a plurality of targets of the same category. For example, the to-be-processed image can be a cell tissue image, and the target to be counted in the to-be-processed image is a cell; the to-be-processed image can be a synchrotron radiation image (which can be referred to as a "grain image") collected by X-ray imaging technology during the metal solidification process, and the target to be counted in the to-be-processed image is a grain.

[0065] The density map is used to show the distribution of the targets in the to-be-processed image, and the density map contains a plurality of density cores, each density core corresponding to a target in the to-be-processed image.

[0066] Referring to Figure 4Fig. 1 shows a schematic diagram of a to-be-processed image and a density map provided by an embodiment of the present application, where the left side is a to-be-processed image, which is a grain image containing multiple grains, and the right side is a density map corresponding to the grain image. The larger the grain in the grain image, the more dispersed the corresponding density core in the density map and the darker the center. The smaller the grain in the grain image, the more concentrated the corresponding density core in the density map and the brighter the center. Figure 4

[0067] The target counting network based on U-Net includes an encoder and a decoder, which are used to encode and decode the to-be-processed image and output the density map corresponding to the to-be-processed image. The applicable scenario of U-Net is a generation task with the same input and output image size, such as image segmentation, reconstruction or repair, and other pixel-level image translation tasks. At the same time, the encoding and decoding results of U-Net itself can integrate information at each scale, and gradually downsample and upsample, which is better for the reconstruction of details. In addition, the target counting network is trained as a generator under the generation of the adversarial mechanism. Through adversarial game training, the accuracy of the target counting network is higher, and it can output an accurate density map.

[0068] The server inputs the to-be-processed image into the target counting network, encodes the to-be-processed image through the encoder to obtain multiple image features (which can be feature maps), and transmits the multiple image features to the decoder through the way of skip connection. Then, the multiple image features can be decoded by the decoder to obtain the density map of the to-be-processed image.

[0069] Step 320: Integrate and sum the density map to obtain the total number of targets in the to-be-processed image.

[0070] After obtaining the density map of the to-be-processed image, the server can integrate and sum the pixel values of the density map, and the sum result can indicate the total number of targets in the to-be-processed image.

[0071] By the above measures, the U-Net structure is introduced into the target counting task, and the density map generation scheme is used to approximate the coverage of the Gaussian density core corresponding to each target to the area where the target center is located. The complete density map reflects the distribution of targets in the to-be-processed image. In a certain sense, this approximate coverage is close to the semantic segmentation of the target. The difference lies in that the semantic segmentation only requires a binary segmentation result, while the pixel values of each pixel in the density map are not necessarily 0 or 1, but can be a value between 0 and 1.

[0072] ​By combining the U-Net encoding-decoding structure with the target counting task, the target and the background in the image to be processed can be effectively distinguished, the difference between the two is learned from the semantic level, the learning focus is placed on the target part rather than the background area, the influence of the background change on the image of the same type of target is reduced, the robustness of the target counting network is improved, more detailed information is introduced through the aggregation of the encoding-decoding features, the loss of important details is minimized on the premise of helping the density map to recover to the original resolution, and therefore the density map that can be used for accurate counting can be obtained.

[0073] In an embodiment, the encoder comprises a plurality of multi-scale down-sampling modules, which can comprise a plurality of branches, different branches can contain convolution layers of different scales, and the multi-scale down-sampling modules can aggregate image features at multiple scales.

[0074] In an embodiment, the server performs step 310, and when encoding the image to be processed through the encoder, the image to be processed can be encoded through the encoder level by level to obtain image features at multiple levels. The encoder can comprise a plurality of network layers, and the server encodes the image to be processed through the encoder, encodes the image to be processed into image features at the first level through the first network layer, encodes the image features at the first level through the second network layer to obtain image features at the second level, encodes the image features at the second level through the third network layer to obtain image features at the third level, and so on.

[0075] In the encoding process, the server can perform multi-scale convolution calculation on the image features at the previous level through the multi-scale down-sampling module. The server can calculate the image features at the previous level through a plurality of branches of the multi-scale down-sampling module to obtain the results output by the plurality of branches. The server can aggregate the results output by the plurality of branches to obtain a transition image feature used to encode the image features at the next level. Here, in the aggregation process, the server can perform convolution calculation on the results output by the plurality of branches through a 1*1 convolution layer, and concatenate the calculation results in the channel dimension to obtain the transition image feature.

[0076] The subsequent server can input the transition image feature into the next network layer to obtain the image features at the next level through encoding.

[0077] Through the above measures, the multi-scale down-sampling module can improve the extraction capability of the target counting network for multi-scale features, thereby improving the representation capability of the target counting network for small-scale and medium-scale targets in the image to be processed, and reducing over-detection (i.e., identifying a medium-scale target as a plurality of small-scale target groups) and missing detection.

[0078] In an embodiment, referring toFigure 5 A structural diagram of a multi-scale down-sampling module provided by an embodiment of the present application is shown in FIG. 2. As shown in FIG. 2, the multi-scale down-sampling module includes a first branch, a second branch, and a third branch. The first branch includes a max-pooling layer and a 1*1 convolutional layer; the second branch includes a 1*1 convolutional layer, a 3*3 deformable convolutional layer, and a 3*3 convolutional layer, with a receptive field of 5*5 under a normal convolution kernel setting, and the receptive field can be an irregular region of about 5*5 considering the introduction of the deformable convolution; and the third branch includes a 1*1 convolutional layer, a 3*3 deformable convolutional layer, and two consecutive 3*3 convolutional layers. Figure 5

[0079] Referring to FIG. 3, a flowchart of a multi-scale encoding process provided by an embodiment of the present application is shown in FIG. 3. As shown in FIG. 3, the process of encoding a transition image feature through a multi-scale down-sampling module can include steps 610-640. Figure 6 Figure 6

[0080] Step 610: Calculate a first sub-transition image feature from an image feature of a previous stage through the first branch.

[0081] The first sub-transition image feature is an image feature output by the first branch.

[0082] The server can obtain the first sub-transition image feature by performing global pooling processing on the image feature of the previous stage through the first branch and then performing convolution calculation.

[0083] Step 620: Calculate a second sub-transition image feature from the image feature of the previous stage through the second branch.

[0084] The second sub-transition image feature is an image feature output by the second branch.

[0085] The server can obtain the second sub-transition image feature by performing multiple convolution calculations on the image feature of the previous stage through the second branch.

[0086] Step 630: Calculate a third sub-transition image feature from the image feature of the previous stage through the third branch.

[0087] The third sub-transition image feature is an image feature output by the third branch.

[0088] The server can obtain the third sub-transition image feature by performing multiple convolution calculations on the image feature of the previous stage through the third branch.

[0089] Step 640: Fuse the first sub-transition image feature, the second sub-transition image feature, and the third sub-transition image feature into a transition image feature.

[0090] ​​​After obtaining the first sub-transition image feature, the second sub-transition image feature and the third sub-transition image feature, the server can perform convolution calculation on the first sub-transition image feature, the second sub-transition image feature and the third sub-transition image feature respectively through a 1*1 convolution layer, and splices the calculation results in the channel dimension, thereby obtaining the transition image feature.

[0091] The 1*1 convolution layer in the three branches is used for dimension reduction processing in the channel dimension to reduce the amount of subsequent calculation. The convolution layer of the first branch is used for dimension reduction processing under the current receptive field to obtain the first sub-transition image feature. The step length of the last convolution layer of the second branch and the third branch can be 2 to achieve the purpose of down-sampling.

[0092] The receptive field of the internal convolution kernel of the multi-scale down-sampling module contains three scales from large to small. The channel number of the input image feature and the output transition image feature can remain consistent. The transition image feature is obtained by extracting features at different scales through each branch and aggregating them together, which is used for learning the representation of different scale features in the next stage. The second branch and the third branch introduce a deformable convolution layer, which can learn additional offsets from the image feature to fine-tune the spatial sampling position in the module, prompting the target counting network to learn more attention-worthy regions and improving the representation ability of target features. Different targets can have different collection positions and receptive fields. This multi-branch multi-scale deformable down-sampling structure can maximize the extraction ability of the network for multi-scale features, realize adaptive extraction of features in more attention-worthy regions, and thus improve the representation ability of small and medium-sized targets in the image to be processed.

[0093] Referring to Figure 7 , a structural diagram of a target counting network provided by an embodiment of the present application is shown in Figure 7 , the target counting network is constructed based on U-Net and includes an encoder and a decoder. The multi-scale down-sampling module is included between every two network layers of the encoder, the multi-scale down-sampling module includes multiple branches, and there is a branch including a deformable convolution layer. The encoder of the target counting network encodes the image to be processed to obtain multiple levels of image features, and transmits the image features to each network layer of the decoder through a skip connection, and obtains a density map of the image to be processed through decoding processing.

[0094] In an embodiment, the target counting network can be trained before the target counting method is executed. Referring to Figure 8 , a training flowchart of a target counting network provided by an embodiment of the present application is shown in Figure 8 , the method can include the following steps 810-840.

[0095] Step 810: inputting a sample image in a sample data set into a generator of the generative adversarial network to obtain a hypothetical density map output by the generator.

[0096] The sample data set includes a plurality of sample images, and the sample images contain the same type of target as the to-be-processed image. Each sample image carries a label density map, which can indicate the actual distribution of the target in the sample image.

[0097] The generative adversarial network includes a generator and a discriminator, and good output can be obtained through mutual game learning.

[0098] In this scheme, the generator can be constructed based on a U-Net. In an embodiment, the encoder of the generator constructed based on the U-Net includes a plurality of multi-scale down-sampling modules. In an embodiment, the down-sampling module of the encoder of the generator constructed based on the U-Net can include a deformable convolution layer.

[0099] In this scheme, the discriminator can be constructed based on a PatchGAN, and a BN (Batch Normalization) layer in the PatchGAN network is replaced by an SN (Spectral Normalization) layer, thereby obtaining the discriminator.

[0100] The hypothetical density map is a density map calculated by the generator for the sample image in the training process.

[0101] Step 820: training the discriminator of the generative adversarial network based on the sample image, the hypothetical density map corresponding to the sample image, and the label density map carried by the sample image.

[0102] The server can construct a sample image-hypothetical density map pair and a sample image-label density map pair based on the sample image, the hypothetical density map corresponding to the sample image, and the label density map carried by the sample image.

[0103] The server can input the sample image-hypothetical density map pair and the sample image-label density map pair into the discriminator to obtain a first discrimination result corresponding to the hypothetical density map and a second discrimination result corresponding to the label density map output by the discriminator. Here, the discrimination result can be a true-false matrix with the same width and height as the hypothetical density map or the label density map, and each element in the true-false matrix represents a true-false judgment value of a pixel at the same position on the density map. The true-false judgment value can be between 0 and 1, and the closer to 1, the more true (the higher the possibility of belonging to the target) the pixel at the corresponding position on the density map is, and vice versa.

[0104] The server can evaluate the first discrimination result and the second discrimination result according to a preset loss function, and adjust the network parameters of the discriminator according to the function value of the loss function, so as to realize the training of the discriminator. Here, the loss function can be an LSGAN (Least Squares GAN) function, which can be represented by the following formula (1):

[0105] L cGAN (G,D)=E x,y [(D(x,y)-1) 2 ]+E x,z [(D(x,G(x,z))+1) 2 ] (1)

[0106] Wherein, x represents a sample image, y represents a label density map, D(x,y) represents a discrimination result of the label density map, G(x,z) represents an imaginary density map, and D(x,G(x,z)) represents a discrimination result of the imaginary density map; E represents a mean value calculation process.

[0107] Step 830: After training the discriminator, the generator is trained based on the sample image and the imaginary density map.

[0108] After training the discriminator, the server can input the sample image-imaginary density map pair into the trained discriminator to obtain a third discrimination result corresponding to the imaginary density map output by the discriminator.

[0109] The server can evaluate the third discrimination result according to a preset loss function, and adjust the network parameters of the generator according to the function value of the loss function, so as to realize the training of the generator. Here, the loss function can be represented by the following formula (2):

[0110] L(G,D)=λ r L r (G)+E x,z [(D(x,G(x,z))-1) 2 ] (2)

[0111] Wherein, λ r is the weight of the reconstruction loss, which can be pre-configured based on experience, and the weight is 200 in an example; L r (G) represents the reconstruction loss; x represents a sample image; G(x,z) represents an imaginary density map; D(x,G(x,z)) represents a discrimination result of the imaginary density map; E represents a mean value calculation process.

[0112] The reconstruction loss can be represented by the following formula (3):

[0113]

[0114] Here, E denotes a mean calculation process; y denotes a label density map; and G(x, z) denotes a hypothetical density map.

[0115] After training the generator, the learning rate can be updated.

[0116] Step 840: Repeat the above process until the generative adversarial network converges, and the generator of the generative adversarial network is used as the target counting network.

[0117] After the generator is trained, the process can be returned to step 810, and the sample image is input into the trained generator to obtain the hypothetical density map of the sample image. The process of steps 810 to 830 can be iterated multiple times. When all sample images in the sample data set are iterated and trained, it can be considered that the generative adversarial network converges. At this time, the generator can be used as the target counting network.

[0118] Through the above measures, under the generative adversarial mechanism, through mutual game learning, the target counting network capable of accurately outputting the density map can be obtained.

[0119] In an embodiment, the discriminator of the generative adversarial network can be a two-way discriminator. One way is a global discriminator, which focuses on the true or false judgment of the overall density map. The other way is a local discriminator, which focuses on the true or false judgment of the local area of the density map. For example, the local area can be a pre-specified area containing the most targets, which occupies one quarter of the size of the density map.

[0120] Referring to Figure 9 , a schematic diagram of the two-way discriminator provided by an embodiment of the present application is shown in Figure 9 , the discriminator includes a first sub-discriminator and a second sub-discriminator. The first sub-discriminator is a global discriminator, and the second sub-discriminator is a local discriminator. The first sub-discriminator outputs a discrimination result for the label density map and the hypothetical density map corresponding to the sample image. The second sub-discriminator outputs a discrimination result for the local label density map and the local hypothetical density map corresponding to the local area of the sample image.

[0121] Referring to Figure 10 , a schematic diagram of the training process of the discriminator provided by an embodiment of the present application is shown in Figure 10 , when training the discriminator, the server can perform the following steps 821-824.

[0122] Step 821: Based on the sample image, the hypothetical density map corresponding to the sample image, and the label density map carried by the sample image, a sample image-hypothetical density map pair and a sample image-label density map pair are constructed.

[0123] Step 822: based on the local position information carried by the sample image, the sample image, the hypothetical density map corresponding to the sample image, and the label density map carried by the sample image, a local image-local hypothetical density map pair and a local image-local label density map pair are constructed.

[0124] The server can construct a sample image-hypothetical density map pair and a sample image-label density map pair according to the sample image, the hypothetical density map of the sample image, and the label density map.

[0125] The local position information carried by the sample image is pre-labeled, which is used to indicate the local area with the most grains in the sample image. The server can crop a local image from the sample image, a local hypothetical density map from the hypothetical density map, and a local label density map from the label density map according to the local position information, and construct a local image-local hypothetical density map pair and a local image-local label density map pair.

[0126] Step 823: based on the sample image-hypothetical density map pair and the sample image-label density map pair, the first sub-discriminator is trained.

[0127] The server can input the sample image-hypothetical density map pair and the sample image-label density map pair into the first sub-discriminator to obtain a fourth discrimination result corresponding to the hypothetical density map and a fifth discrimination result corresponding to the label density map output by the first sub-discriminator.

[0128] Step 824: based on the local image-local hypothetical density map pair and the local image-local label density map pair, the second sub-discriminator is trained.

[0129] The server can input the local image-local hypothetical density map pair and the local image-local label density map pair into the second sub-discriminator to obtain a sixth discrimination result corresponding to the local hypothetical density map and a seventh discrimination result corresponding to the local label density map.

[0130] The server can evaluate the fourth discrimination result and the fifth discrimination result according to the loss function shown in the above formula (1), and evaluate the sixth discrimination result and the seventh discrimination result according to the loss function shown in the above formula (1), and adjust the network parameters of the first sub-discriminator and the second sub-discriminator according to the function values of the two loss functions, thereby realizing the training of the first sub-discriminator and the second sub-discriminator.

[0131] In an embodiment, after the first sub-discriminator and the second sub-discriminator are trained, the server can train the generator based on the sample image-hypothetical density map pair and the local image-local hypothetical density map pair.

[0132] The server can input the sample image-hypothetical density map pair into the trained first sub-discriminator to obtain an eighth discrimination result corresponding to the hypothetical density map. The server can input the local image-local hypothetical density map pair into the trained second sub-discriminator to obtain a ninth discrimination result corresponding to the local hypothetical density map.

[0133] The server can evaluate the eighth discrimination result and the ninth discrimination result according to the loss function indicated by the above formula (2), and adjust the network parameters of the generator according to the function value of the loss function, so as to realize the training of the generator.

[0134] Through the game training of the generator in the generative adversarial network by the double-path discriminator, the attention of the network to the local area with high target density in the sample image can be improved, so as to assist the generator to more accurately extract features and generate density maps for density estimation of the target dense area.

[0135] In an embodiment, referring to Figure 11 The flowchart of the method for constructing a sample data set provided by an embodiment of the present application is shown in Figure 11 The method can include the following steps 910-930.

[0136] Step 910: Obtain a plurality of sample images and target position information labeled for each sample image; wherein the target position information includes target center position and target radius information.

[0137] The target position information can include target center position and target radius information of a plurality of targets, where the target center position indicates the position of the center of the target in the sample image, which can generally be the coordinate position of a pixel in the sample image, and the target radius information is a pre-labeled target radius value.

[0138] Step 920: For each sample image, calculate the target position information in the sample image by using a Gaussian kernel function that is adaptive based on the target radius information to obtain a label density map of the sample image.

[0139] Here, the Gaussian kernel function that is adaptive based on the target radius information can be represented by the following formula (4):

[0140]

[0141] where p represents a pixel in the sample image; P n is a set of all positions in the sample image where the target is labeled; represents a normalized two-dimensional Gaussian kernel, m represents the brightness mean of the Gaussian kernel, represents an adaptive variance calculated by the target radius information corresponding to the p point, and I represents a unit matrix, This represents an isotropic covariance matrix generated based on adaptive variance.

[0142] here, It can be calculated using formula (5):

[0143]

[0144] Where r represents the target radius information corresponding to point p; λ s , λ σ , λ offset Both μ and λ are preset values; for example, λ s =4,λ σ =0.125, μ=12, λ offset =8.5.

[0145] Step 930: Construct a sample dataset based on multiple sample images carrying label density maps.

[0146] After generating a corresponding label density map for each sample image, the server can construct a sample dataset based on multiple sample images carrying label density maps.

[0147] By introducing target radius information as an auxiliary tool to generate the density map, the label density map carried by the sample image becomes more accurate and has less noise, which helps with subsequent training.

[0148] The target counting algorithm proposed in this application combines the ideas of generative adversarial mechanism, U-Net encoding and decoding structure, multi-scale feature aggregation and dual-path discriminator. It uses an end-to-end training framework to solve the problem of density map quality deterioration caused by density map missing distortion and target cluster identification errors in the target counting task. At the same time, it overcomes the multi-scale problem and background interference caused by target scale changes in the image to be processed (e.g., the grains in the grain image gradually increase in size and the grain size and shape have large differences).

[0149] See Figure 12 This is a schematic diagram of density map comparison results provided in an embodiment of this application, as shown below. Figure 12 As shown, Figure a is the grain image that serves as the image to be processed; Figure b is the density map output by the target counting network in this application; Figure c is the density map output by the original U-Net network; and Figure d is the density map obtained by other machine learning schemes.

[0150] Although the other two methods can also roughly reflect the distribution of the grains, they are greatly affected by the background, the density kernel shape is often distorted, and there is a serious over-detection phenomenon in the grain clustering area. In contrast, the density map generated by the target counting network of the present application is more robust, and the details are closer to the actual distribution. Benefiting from the coding structure, the MSD module and the dual-channel discriminator auxiliary supervision, the target counting network can not only effectively improve the multi-scale feature extraction problem, but also effectively aggregate the context information to improve the discrimination ability of the grains and the noise. At the same time, the generation of the generated adversarial mechanism makes the density kernel in the generated density map more clear and smooth, and the distribution is closer to the actual grain distribution. In addition, it can be seen from the figure that the over-detection and missed detection phenomenon exists in the comparative method, while the present scheme can normally detect, and can normally reflect the actual density level of the corresponding area.

[0151] Figure 13 A target counting device according to an embodiment of the present application, as shown in Figure 13 The device can include:

[0152] A first calculation module 1310 is configured to input a to-be-processed image into a target counting network based on U-Net, encode the to-be-processed image through an encoder of the target counting network, decode a plurality of image features obtained by the encoding through a decoder of the target counting network, and obtain a density map of the to-be-processed image. The target counting network is trained as a generator in a generative adversarial mechanism.

[0153] A second calculation module 1320 is configured to integrate and sum the density map to obtain a total number of targets in the to-be-processed image.

[0154] The implementation process of the functions and roles of each module in the above device is specifically described in the implementation process of the corresponding steps in the above target counting method, and will not be repeated here.

[0155] In several embodiments provided in the present application, the disclosed apparatus and method can also be implemented in other manners. The embodiments described above are only schematic. For example, the flowcharts and block diagrams in the embodiments of the present application illustrate the possible implementation ways of the apparatus, methods and computer program products according to the present application. In this regard, the flowcharts and block diagrams in the embodiments of the present application can represent a possible implementation way of the device, methods and computer program products according to the present application. In some alternative implementations, the functions noted in the flowcharts or block diagrams can occur in a sequence different from that noted in the flowcharts or block diagrams. For example, two sequentially numbered steps in the flowcharts or block diagrams can actually be executed concurrently or in the reverse order. Depending on the implementation, the functions can be executed in a different order. It should also be noted that each block in the flowcharts or block diagrams and combinations of blocks in the flowcharts or block diagrams can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0156] In addition, each functional module in the embodiments of the present application can be integrated together to form a separate part, or each module can exist independently, or two or more modules can be integrated to form a separate part.

[0157] If the functions are implemented in the form of software function modules and sold or used as an independent product, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or partly or the part of the technical solutions that make contributions to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk.

Claims

1. A target counting method characterized by comprising: The method comprises the following steps: inputting a sample image in a sample data set into a generator of a generative adversarial network to obtain a hypothetical density map output by the generator; training a discriminator of the generative adversarial network based on the sample image, the hypothetical density map corresponding to the sample image, and a label density map carried by the sample image; after training the discriminator, training the generator based on the sample image and the hypothetical density map; repeating the above process until the generative adversarial network converges, and taking the generator of the generative adversarial network as a target counting network; wherein the discriminator comprises a first sub-discriminator and a second sub-discriminator; the training of the discriminator of the generative adversarial network based on the sample image, the hypothetical density map corresponding to the sample image, and the label density map carried by the sample image comprises: constructing a sample image-hypothetical density map pair and a sample image-label density map pair based on the sample image, the hypothetical density map corresponding to the sample image, and the label density map carried by the sample image; constructing a local image-local hypothetical density map pair and a local image-local label density map pair based on the local position information carried by the sample image, the sample image, the hypothetical density map corresponding to the sample image, and the label density map carried by the sample image; training the first sub-discriminator based on the sample image-hypothetical density map pair and the sample image-label density map pair; training the second sub-discriminator based on the local image-local hypothetical density map pair and the local image-local label density map pair; inputting a to-be-processed image into the trained target counting network based on U-Net, performing encoding processing on the to-be-processed image by an encoder of the target counting network, performing decoding processing on a plurality of image features obtained by encoding by a decoder of the target counting network to obtain a density map of the to-be-processed image; wherein the target counting network is trained as a generator in a generative adversarial mechanism; integrating and summing the density map to obtain a total number of targets in the to-be-processed image; wherein before training the target counting network, the method further comprises: obtaining a plurality of sample images and target position information annotated for each sample image; wherein the target position information comprises target center position information and target radius information; for each sample image, calculating the target position information in the sample image by a Gaussian kernel function that is adaptive based on the target radius information to obtain a label density map of the sample image; constructing the sample data set based on a plurality of sample images carrying label density maps.

2. The method of claim 1, wherein, The encoder comprises a plurality of multi-scale down-sampling modules; the encoding processing by the encoder of the target counting network comprises: performing step-by-step encoding processing on the to-be-processed image by the encoder to obtain a plurality of levels of image features; wherein in the encoding processing, the multi-scale down-sampling modules are used to perform multi-scale convolution calculation and aggregation on the image features of the previous level to obtain transition image features used for encoding the image features of the next level.

3. The method of claim 2, wherein, The multi-scale downsampling module comprises a first branch, a second branch and a third branch, the first branch comprises a max-pooling layer and a 1*1 convolution layer, the second branch comprises a 1*1 convolution layer, a 3*3 deformable convolution layer and a 3*3 convolution layer, and the third branch comprises a 1*1 convolution layer, a 3*3 deformable convolution layer and two consecutive 3*3 convolution layers; The multi-scale convolution calculation and aggregation of the image features of the previous stage through the multi-scale downsampling module obtain a transition image feature used for encoding the image features of the next stage, comprising: calculating a first sub-transition image feature from the image features of the previous stage through the first branch; calculating a second sub-transition image feature from the image features of the previous stage through the second branch; calculating a third sub-transition image feature from the image features of the previous stage through the third branch; fusing the first sub-transition image feature, the second sub-transition image feature and the third sub-transition image feature into the transition image feature.

4. The method of claim 1, wherein, After training the discriminator, the generator is trained based on the sample image and the hypothetical density map, comprising: After training the first sub-discriminator and the second sub-discriminator, the generator is trained based on the sample image-hypothetical density map pair and the local image-local hypothetical density map pair.

5. A target counting device, characterized by, comprising: The first calculation module is configured to input a to-be-processed image into a target counting network constructed based on a U-Net, perform encoding processing on the to-be-processed image through an encoder of the target counting network, perform decoding processing on a plurality of image features obtained through the encoding through a decoder of the target counting network, and obtain a density map of the to-be-processed image; wherein the target counting network is trained as a generator in a generative adversarial mechanism. The target counting network is trained in the following manner: input a sample image in a sample data set into a generator of a generative adversarial network, and obtain a hypothetical density map output by the generator; train a discriminator of the generative adversarial network based on the sample image, the hypothetical density map corresponding to the sample image and a label density map carried by the sample image; train the generator based on the sample image and the hypothetical density map after training the discriminator; repeat the above process until the generative adversarial network converges, and take the generator of the generative adversarial network as a target counting network; The discriminator comprises a first sub-discriminator and a second sub-discriminator. The training of the discriminator of the generative adversarial network based on the sample image, the hypothetical density map corresponding to the sample image and the label density map carried by the sample image comprises: construct a sample image-hypothetical density map pair and a sample image-label density map pair based on the sample image, the hypothetical density map corresponding to the sample image and the label density map carried by the sample image. constructing a local image-local hypothetical density map pair and a local image-local label density map pair based on the local position information carried by the sample image, the sample image, the hypothetical density map corresponding to the sample image, and the label density map carried by the sample image; training the first sub-discriminator based on the sample image-hypothetical density map pair and the sample image-label density map pair; training the second sub-discriminator based on the local image-local hypothetical density map pair and the local image-local label density map pair; Before training the target counting network, the method further comprises: obtaining a plurality of sample images and target position information annotated for each sample image, wherein the target position information comprises target center position information and target radius information; for each sample image, calculating the target position information in the sample image by using a Gaussian kernel function that is adaptive based on the target radius information to obtain a label density map of the sample image; constructing the sample data set based on a plurality of sample images carrying label density maps; a second calculation module configured to integrate and sum the density map to obtain a total number of targets in the image to be processed.

6. An electronic device, comprising: The electronic device comprises: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform the target counting method of any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that can be executed by the processor to complete the target counting method of any one of claims 1-4.

Images