A weakly supervised object counting neural network training method and counting method

Abstract

The present invention provides a weakly supervised object counting neural network training method and counting method, and relates to the field of weakly supervised learning technology. The training method comprises the steps of: S1, using training sample images to perform incomplete supervision training on the initial neural network to obtain an incompletely supervised neural network; S2, using the number of recognized objects in the training sample images to perform inaccurate supervision training on the incompletely supervised neural network to obtain an object counting neural network; compared with simple incomplete supervision and inaccurate supervision, more supervision information is input into the neural network, and under the premise of ensuring counting accuracy, the labeling workload of a single image is reduced, and it is not easy to miss detection and has high recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of weakly supervised learning, and more specifically, to an object counting neural network training method and a counting method based on weak supervision. Background Art

[0002] Neural networks have achieved good results in the application of image recognition. Neural networks can be used to identify objects in images and automatically count the recognized objects in images. The neural network training method in the prior art is usually one of the following three methods: fully supervised training, incomplete supervision training or inexact supervision training.

[0003] Fully supervised training uses fully supervised information to train neural networks, which can achieve relatively high accuracy. However, in order to obtain fully supervised information, a large amount of manual labeling is required, which wastes manpower. Incompletely supervised training uses incompletely supervised information to train neural networks. Incompletely supervised information only needs to label some objects in the image, which can reduce the amount of labeling. However, since the labels of unlabeled areas in the image cannot be determined, it is easy to cause missed detections and false detections. Inaccurately supervised training uses inaccurately supervised information to train neural networks. Inaccurately supervised information only coarsely labels the image, such as only labeling the number of objects in the image. The amount of labeling is also low, but the neural network cannot determine the position of the object, making model training difficult, which limits the potential room for improvement. The existing inaccurately supervised training method uses a pre-trained model to generate an initial detection frame to determine the object position, which relies on the pre-trained model's prior knowledge of the detected object. The pre-trained model is trained using a large data set. Secondly, the object category of the inaccurately supervised training cannot be determined, and the candidate frame generated by the pre-trained model on the new category of objects is not accurate enough.

[0004] A crowd counting method based on semi-supervised manifold embedding in the prior art only labels part of the samples and uses training sample images with only partial labels to implement semi-supervised training on the neural network to reduce the human resources and time spent on labeling. However, this technical solution only uses incomplete supervision to train the neural network, which is prone to missed detections and false detections, and the detection accuracy is not ideal. Summary of the invention

[0005] In order to overcome the above technical problems, the present invention provides a weakly supervised object counting neural network training method and counting method with a small amount of annotation and less chance of missing detection.

[0006] The technical solution of the present invention is as follows:

[0007] A method for training a neural network for object counting based on weak supervision, comprising the steps of:

[0008] S1. Perform incompletely supervised training on the initial neural network using the training sample images to obtain an incompletely supervised neural network;

[0009] S2. Perform imprecise supervision training on the incompletely supervised neural network using the number of recognized objects in the training sample images to obtain an object counting neural network.

[0010] This technical solution proposes a method for training a neural network for object counting based on weak supervision. The neural network combines incomplete supervision training and inaccurate supervision training during the training process. Compared with the neural network trained with complete supervision information, it can effectively reduce the amount of annotation and save labor costs. Compared with the neural network trained with only incomplete supervision, it can effectively improve the accuracy of the detection frame. Compared with the conventional inaccurate supervision training using the semantic information of the identified object, this solution uses the quantity information of the identified object for inaccurate supervision training, which can achieve better counting effects. This solution first performs incomplete supervision training on the neural network, which is targeted for objects of a specific category, improves the quality of the initial detection frame, and obtains a more accurate candidate area with a higher recall rate. Compared with simple incomplete supervision and inaccurate supervision, more supervision information is input into the neural network. Under the premise of ensuring the counting accuracy, it reduces the workload of labeling a single picture, is not easy to miss detection, and has high recognition accuracy.

[0011] Furthermore, the method of incomplete supervision training in step S1 is:

[0012] S11, obtaining training sample images and an initial neural network;

[0013] S12, using the initial neural network to generate a first heat map for the training sample image, and generating a first heat map true value for the marked box area in the first heat map;

[0014] S13. cyclically calculate a first loss value using the first heat map and the first heat map true value, and update the initial neural network weights according to the first loss value until the network can detect all labeled marking boxes, thereby obtaining an incompletely supervised neural network.

[0015] Furthermore, in step S12, the method for calculating the true value of the first heat map is:

[0016]

[0017] (x,y)∈{x,y∈R:x i1 ≤x≤x i2 and y i1 ≤y≤y i2}

[0018] Among them, I tiis the true value of the first heat map, and the coordinate of the upper left corner of the i-th annotation box is (x i1 ,y i1 ), the coordinates of the lower right corner are (x i2 ,y i2 ), height is h i , width is w i ,

[0019] In step S13, the calculation method of the first loss value is:

[0020]

[0021] Among them I pi ,I ti are the predicted value of the first heat map in the corresponding box i and the true value of the first heat map respectively; k is the number of annotated boxes of the training sample image.

[0022] Furthermore, the method of the imprecise supervised training in step S2 is:

[0023] S21, obtaining an incompletely supervised neural network that has completed incompletely supervised training, and generating a second heat map for the training sample image according to the number of recognized objects in the training sample image by the incompletely supervised neural network;

[0024] S22, top-hat filtering is performed on the second heat map and then thresholding is performed to obtain a series of initial patches, represented as B = {b e , e=1, 2, 3,..., m};

[0025] S23, calculate the area and mean of the initial patches, sort all the initial patches according to the area and mean of the initial patches, and divide the heat map patches into two categories in descending order: the first n patches are the first patch set B f = {b f , f = 1, 2, 3, ..., n}, where n is the number of objects to be identified in the current training sample image, and the mn patches after sorting are the second patch set B g = {b g , g=n+1, n+2,..., m};

[0026] S24. Calculate B g The false detection loss and calculate B f and B g Gap loss;

[0027] S25, the loss function calculates the second loss value using the first loss value, the false detection loss and the gap loss;

[0028] S26, determining whether the specified number of iteration steps has been reached, if not, updating the weight of the incompletely supervised neural network according to the second loss value, and then returning to step S21 to iteratively train the incompletely supervised neural network; if the number of iteration steps has been reached, stopping the iteration and saving the neural network as an object counting neural network;

[0029] In step S23, the number of recognized objects in the current training sample image is obtained by coarse-grained labeling of the training samples.

[0030] In the above technical solution, the second heat map is first subjected to top-hat filtering and then thresholding. During training, since the values ​​of the heat map gradually increase from small to large, but most of them are fragmented, small, and uneven, if the thresholding is used directly to extract the region, the region with lower values ​​is easy to be lost. Top-hat filtering first can avoid losing these regions.

[0031] Furthermore, the calculation method of the top hat filter in step S22 is:

[0032] Wherein, e is a structural element, and the structural element is a circle; I is a second heat map.

[0033] In the above technical solution, the structural element is a circle. Since the shape and direction of the object in the picture are uncertain, it is inappropriate to use a rectangle, ellipse or other directional graphics as a structural element. Therefore, using an isotropic circle as a structural element obtains better performance.

[0034] Furthermore, in step S24, B is calculated g The method of false detection loss is:

[0035]

[0036] Among them, b g is the first patch set B g Plaques in

[0037] Step S24: Calculate B f and B g The gap loss method is:

[0038]

[0039] Furthermore, the loss function in step S25 is:

[0040] L(P, T, n) = αL reg (P, T)+βL gap (B f , B g )+γL fp (B g )

[0041] Among them, α, β, and γ are the coefficients of the three loss terms in the loss function.

[0042] An object counting method based on weak supervision inputs an image to be identified into an object counting neural network, and the object counting neural network outputs the number of identified objects in the image to be identified.

[0043] Furthermore, the identification objects in the image to be identified and the training sample image are objects of the same type.

[0044] Furthermore, the initial neural network in step S1 is an FCN neural network or a U-Net neural network.

[0045] This technical solution proposes a training method and counting method for object counting neural network based on weak supervision. Compared with the prior art, the beneficial effects of the technical solution of the present invention are as follows: the neural network combines incomplete supervision training and inaccurate supervision training during the training process. Compared with the neural network trained with complete supervision information, it can effectively reduce the amount of annotation and save labor costs. Compared with the neural network trained with only incomplete supervision, it can effectively improve the accuracy of the detection frame; compared with the conventional inaccurate supervision training using the semantic information of the identified object, this solution uses the quantity information of the identified object for inaccurate supervision training, which can achieve better counting effect; this solution first performs incomplete supervision training on the neural network, which is targeted for objects of a specific category, improves the quality of the initial detection frame, and obtains a more accurate candidate area with a higher recall rate. Compared with simple incomplete supervision and inaccurate supervision, more supervision information is input into the neural network. Under the premise of ensuring the counting accuracy, the annotation workload of a single image is reduced, and it is not easy to miss detection and the recognition accuracy is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of the steps of the object counting neural network training method based on weak supervision of the present invention;

[0047] Figure 2 Flowchart for training neural network;

[0048] Figure 3 Schematic diagram of the annotation of training sample images;

[0049] Figure 4 This is a schematic diagram of the second heat map patch. DETAILED DESCRIPTION

[0050] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0051] Example 1

[0052] A weakly supervised object counting neural network training method Figure 1 As shown, the steps include:

[0053] S1, using the training sample images to perform incomplete supervision training on the initial neural network to obtain an incompletely supervised neural network;

[0054] S2. Perform imprecise supervision training on the incompletely supervised neural network using the number of recognized objects in the training sample images to obtain an object counting neural network.

[0055] In this embodiment, a method for training a neural network for counting objects based on weak supervision is used. The neural network combines incomplete supervision training and inaccurate supervision training during the training process. Compared with the neural network trained with complete supervision information, the amount of annotation can be effectively reduced to save labor costs. Compared with the neural network trained with only incomplete supervision, the accuracy of the detection frame can be effectively improved. Compared with the conventional inaccurate supervision training using the semantic information of the identified object, this scheme uses the quantity information of the identified object for inaccurate supervision training, which can achieve a better counting effect. This scheme first performs incomplete supervision training on the neural network, which is targeted for objects of a specific category, improves the quality of the initial detection frame, and obtains a more accurate candidate area with a higher recall rate. Compared with simple incomplete supervision and inaccurate supervision, more supervision information is input into the neural network. Under the premise of ensuring the counting accuracy, the annotation workload of a single image is reduced, and the recognition accuracy is high and not easy to miss.

[0056] Example 2

[0057] This embodiment provides a method for training a neural network for object counting based on weak supervision. The flowchart is as follows: Figure 2 As shown, on the basis of Example 1, in this embodiment, the training sample images need to be incompletely labeled and coarse-grained labeled before the initial neural network training. The incomplete labeling and coarse-grained labeling are used for incomplete supervised training and coarse-grained supervised training respectively. The incomplete labeling means that for each training sample image, only part of the recognition object in the training sample image is marked with an annotation box; the initial neural network in step S1 is an FCN neural network or a U-Net neural network.

[0058] The coarse-grained labeling means that only the number of recognized objects in each training sample image is labeled.

[0059] When the training sample images are subjected to the incomplete labeling, the number of labeling boxes in each training sample image is 1-3.

[0060] The diagram of the annotation of the training sample image is as follows: Figure 3 As shown, Figure 3 The upper left picture is fully labeled, that is, every identified object in the picture is labeled; Figure 3 The upper right picture is incompletely marked, that is, some of the identified objects in the picture are marked; Figure 3 The lower left image is a coarse-grained label, which only marks the number of objects identified in the image; Figure 3 The lower right picture shows incomplete labeling and coarse-grained labeling, that is, some of the recognized objects in the image are marked, and the number of recognized objects in the image is marked.

[0061] The method for obtaining training sample images is as follows: shooting a video of the object to be counted, then extracting each frame, and performing incomplete labeling and coarse-grained labeling on each frame.

[0062] The method of incomplete supervision training described in step S1 is:

[0063] S11, obtaining training sample images and an initial neural network;

[0064] S12, using the initial neural network to generate a first heat map for the training sample image, and generating a first heat map true value for the marked box area in the first heat map;

[0065] S13. cyclically calculate a first loss value using the first heat map and the first heat map true value, and update the initial neural network weights according to the first loss value until the network can detect all labeled marking boxes, thereby obtaining an incompletely supervised neural network.

[0066] In step S12, the calculation method of the true value of the first heat map is:

[0067]

[0068] (x, y)∈{x, y∈R:x i1 ≤x≤x i2 and y i1 ≤y≤y i2}

[0069] Among them, I ti is the true value of the first heat map, and the coordinate of the upper left corner of the i-th annotation box is (x i1 ,y i1 ), the coordinates of the lower right corner are (x i2 ,y i2 ), height is h i , width is w i ,

[0070] In step S13, the calculation method of the first loss value is:

[0071]

[0072] Among them I pi ,I ti are the predicted value of the first heat map in the corresponding box i and the true value of the first heat map respectively; k is the number of annotation boxes in the training sample image.

[0073] The method of imprecise supervision training described in step S2 is:

[0074] S21, obtaining an incompletely supervised neural network that has completed incompletely supervised training, wherein the incompletely supervised neural network generates a second heat map for the training sample image;

[0075] S22, top-hat filtering is performed on the second heat map and then thresholding is performed to obtain a series of initial patches, represented as B = {b e , e=1, 2, 3,..., m};

[0076] S23, calculate the area and mean of the initial patches, sort all the initial patches according to the area and mean of the initial patches, and divide the heat map patches into two categories in descending order: the first n patches are the first patch set B f = {b f , f = 1, 2, 3, ..., n}, where n is the number of objects to be identified in the current training sample image, and the mn patches after sorting are the second patch set B g = {b g , g=n+1, n+2,..., m};

[0077] S24. Calculate B g The false detection loss and calculate B f and B g Gap loss;

[0078] S25, the loss function calculates the second loss value using the first loss value, the false detection loss and the gap loss;

[0079] S26, determining whether the specified number of iteration steps has been reached, if not, updating the weight of the incompletely supervised neural network according to the second loss value, and then returning to step S21 to iteratively train the incompletely supervised neural network; if the number of iteration steps has been reached, stopping the iteration and saving the neural network as an object counting neural network;

[0080] In step S23, the number of recognized objects in the current training sample image is obtained by coarse-grained labeling of the training samples.

[0081] Example 3

[0082] On the basis of Example 2, in this embodiment, the calculation method of the top hat filter in step S22 is:

[0083] Where e is a structural element, which is a circle; I is the second heat map. Since the shape and direction of the object in the image are uncertain, it is inappropriate to use a directional figure such as a rectangle or an ellipse as a structural element, so an isotropic circle is used as the structural element to obtain better performance.

[0084] In step S24, B is calculated g The method of false detection loss is:

[0085]

[0086] Among them, b g is the first patch set B g Plaques in

[0087] Step S24: Calculate B f and B g The gap loss method is:

[0088]

[0089] The loss function in step S25 is:

[0090] L(P, T, n) = αL reg (P, T)+βL gap (B f , B g )+γL fp (B g )

[0091] Among them, α, β, and γ are the coefficients of the three loss terms in the loss function.

[0092] This embodiment performs incomplete supervision training and inaccurate supervision training on the neural network to obtain an object counting neural network, and uses the object counting neural network to identify the number of objects. The neural network combines incomplete labeling information and coarse-grained labeling information during the training process. Compared with the neural network trained with complete supervision information, it can effectively reduce the amount of labeling and save labor costs. Compared with the neural network trained with only incomplete supervision, it can effectively improve the accuracy of the detection frame. Compared with the neural network trained with only inaccurate supervision, this solution first performs incomplete supervision training on the neural network, which is targeted for objects of a specific category, improves the quality of the initial detection frame, and obtains a more accurate candidate area with a higher recall rate. Compared with simple incomplete supervision and inaccurate supervision, more supervision information is input into the neural network. Under the premise of ensuring the counting accuracy, the labeling workload of a single image is reduced, and it is not easy to miss detection and has high recognition accuracy.

[0093] Example 4

[0094] A method for counting objects based on weak supervision, wherein an image to be identified is input into an object counting neural network, and the object counting neural network outputs the number of identified objects in the image to be identified. The identified objects in the image to be identified and the training sample image are of the same type of objects. The method for obtaining the image to be identified is: shooting a video of the object to be counted, and then extracting each frame as the image to be identified. If the image to be identified is a side view image, each frame of the image to be identified needs to be incompletely labeled and coarse-grainedly labeled.

Claims

1. A method for training a neural network for object counting based on weak supervision, characterized in that: Includes steps: S1, using the training sample images to perform incomplete supervision training on the initial neural network to obtain an incompletely supervised neural network; During training, a first heat map is generated, and a first loss value is calculated using the first heat map; S2, using the number of recognized objects in the training sample images to perform imprecise supervision training on the incompletely supervised neural network to obtain an object counting neural network; The method of imprecise supervision training described in step S2 is: S21, obtaining an incompletely supervised neural network that has completed incompletely supervised training, wherein the incompletely supervised neural network generates a second heat map for the training sample image; S22, top-hat filtering is performed on the second heat map and then thresholding is performed to obtain a series of initial patches, represented as B = {b e ,e=1,2,3,…,m}; S23, calculate the area and mean of the initial patches, sort all the initial patches according to the area and mean of the initial patches, and divide the heat map patches into two categories in descending order: the first n patches are the first patch set B f = {b f ,f=1,2,3,…,n}, where n is the number of recognized objects in the current training sample image, and the mn patches after sorting are the second patch set B g = {b g ,g=n+1,n+2,…,m}; S24. Calculate B g The false detection loss and calculate B f and B g Gap loss; S25, the loss function calculates the second loss value using the first loss value, the false detection loss and the gap loss; S26, determining whether the specified number of iteration steps has been reached, if not, updating the weight of the incompletely supervised neural network according to the second loss value, and then returning to step S21 to iteratively train the incompletely supervised neural network; if the number of iteration steps has been reached, stopping the iteration and saving the neural network as an object counting neural network; Wherein, the number of recognized objects in the current training sample image in step S23 is obtained by coarse-grained marking of the training samples; In step S24, B is calculated g The method of false detection loss is: Among them, b g is the first patch set B g Plaques in Step S24: Calculate B f and B g The gap loss method is:

2. The method for training a weakly supervised object counting neural network according to claim 1, characterized in that: The method of incomplete supervision training described in step S1 is: S11, obtaining training sample images and an initial neural network; S12, using the initial neural network to generate a first heat map for the training sample image, and generating a first heat map true value for the marked box area in the first heat map; S13. cyclically calculate a first loss value using the first heat map and the first heat map true value, and update the initial neural network weights according to the first loss value until the network can detect all labeled marking boxes, thereby obtaining an incompletely supervised neural network.

3. The method for training a weakly supervised object counting neural network according to claim 2, characterized in that: In step S12, the calculation method of the true value of the first heat map is: (x,y)∈{x,y∈R:x i1 ≤x≤x i2 and y i1 ≤y≤y i2 } Among them, I ti is the true value of the first heat map, and the coordinate of the upper left corner of the i-th annotation box is (x i1 ,y i1 ), the coordinates of the lower right corner are (x i2 ,y i2 ), height is h i , width is w i , In step S13, the calculation method of the first loss value is: Among them I pi ,I ti are the predicted value of the first heat map in the corresponding box i and the true value of the first heat map respectively; k is the number of annotation boxes in the training sample image.

4. The method for training a weakly supervised object counting neural network according to claim 3, characterized in that: The calculation method of the top hat filter in step S22 is: Wherein e is a structural element, and the structural element is a circle; I is the second heat map.

5. The method for training a weakly supervised object counting neural network according to claim 4, characterized in that: The loss function in step S25 is: K(P,T,n)=αL reg (P,T)+βL gap (B f ,B g )+γL fp (B g ) Among them, α, β, and γ are the coefficients of the three loss terms in the loss function.

6. A method for counting objects based on weak supervision, characterized in that: The object counting neural network obtained by the training method described in any one of claims 1 to 5 is used to count objects in the picture, specifically: the image to be identified is input into the object counting neural network, and the object counting neural network outputs the number of identified objects in the image to be identified.

7. The method for counting objects based on weak supervision according to claim 6, characterized in that: The identification objects in the image to be identified and the training sample image are objects of the same type.

8. The method for counting objects based on weak supervision according to claim 7, characterized in that: The initial neural network in step S1 is an FCN neural network or a U-Net neural network.

Images