A weakly supervised target localization method based on entropy consistency

Abstract

The invention discloses a weakly supervised target positioning method based on entropy consistency, comprising: a training phase: step 1, inputting a training set picture containing category annotations into a neural network, generating four-layer feature maps, and calculating classification losses respectively; step 2, obtaining a fused feature map from the four-layer feature map, and calculating the classification loss; step 3, respectively generating an entropy feature map and a fused entropy feature map from the four-layer feature map and the fused feature map; step 4, respectively calculating foreground-background separation losses from the fused entropy feature map and the four-layer entropy feature map; step 5, respectively calculating entropy consistency losses from the four-layer entropy feature map and the fused entropy feature map; step 6, updating the neural network using classification losses, separation losses, and entropy consistency losses; a testing phase: step 7, inputting an unlabeled picture into a trained network, obtaining a fused entropy feature map and a fused feature map; step 8, obtaining picture categories through the fused feature map; and step 9, obtaining the position of a target through the fused entropy feature map.

Description

Technical Field

[0001] The invention relates to a weakly supervised positioning method, in particular to a weakly supervised target positioning method based on entropy consistency. Background Art

[0002] Weakly supervised object localization aims to locate objects in a scene using only image-level labels. It has attracted increasing attention in the research community due to its wide application in industrial scenarios. The current mainstream methods for weakly supervised object localization mainly include the following two categories:

[0003] 1. Class Activation Mapping-based Method: This method uses the class activation map generated by the network as the basis and extracts the target coordinates through feature enhancement methods. Methods based on feature erasure (Has) and methods based on feature similarity (SPA) both belong to the category of this method.

[0004] 2. Foreground learning-based method: This method separates the positioning problem from the classification problem and directly obtains the coordinates of the foreground area extraction of the image. Common methods include pseudo-supervision-based methods (PSOL) and geometric constraint-based methods (GC-Net).

[0005] Among them, the method based on class activation mapping is relatively simple, but because it is based on category information, it tends to obtain the most discriminative local features of the object, while target positioning requires more global features;

[0006] The foreground learning-based method decouples target localization and classification, but requires multiple stages of learning and cannot train the entire model end-to-end at one time. Summary of the invention

[0007] Purpose of the invention: The technical problem to be solved by the present invention is to provide a weakly supervised target positioning method based on entropy consistency in view of the shortcomings of the prior art.

[0008] In order to solve the above technical problems, the present invention discloses a weakly supervised target positioning method based on entropy consistency, comprising the following steps:

[0009] Training phase:

[0010] Step 1: Input the training set images including images and category annotations into the convolutional neural network to generate four layers of feature maps, and calculate the classification loss respectively;

[0011] Step 2: Fuse the four-layer feature maps obtained in step 1 to obtain a fused feature map and calculate the classification loss.

[0012] Step 3, generating four layers of entropy feature maps and one layer of fused entropy feature map respectively from the four layers of feature maps obtained in step 1 and the fused feature map obtained in step 2;

[0013] Step 4, the one-layer fusion entropy feature map and the four-layer entropy feature map respectively calculate the foreground-background separation loss;

[0014] Step 5, calculating the entropy consistency loss of the four layers of entropy feature maps obtained in step 3 and the one layer of fusion entropy feature map respectively;

[0015] Step 6: Update the convolutional neural network using the classification loss obtained in steps 1 and 2 and the foreground-background separation loss and entropy consistency loss obtained in steps 4 and 5.

[0016] Testing phase:

[0017] Step 7, input the unlabeled image into the convolutional neural network trained in the training phase to obtain a fused entropy feature map and a fused feature map;

[0018] Step 8, obtaining the image category through the fusion feature map obtained in step 7;

[0019] Step 9: Obtain the position of the target in the image through the fused entropy feature map obtained in step 7.

[0020] The method for generating a four-layer feature map in step 1 of the present invention includes:

[0021] The convolutional neural network VGG16 is used, and the picture is input into the convolutional neural network to obtain feature maps with downsampling ratios of 4, 8, 16, and 16 as four-layer feature maps. The number of channels of the four-layer feature maps is changed to C0 through the convolution layer, where C0 is the number of categories.

[0022] The method for obtaining the fusion feature map described in step 2 of the present invention includes:

[0023] Let feature map X m , where 1≤m≤4, represents the mth of the four feature maps, whose size is H m ×W m ×C0, where H m is the height of the mth feature map, W m is the width of the mth feature map;

[0024] Perform bilinear interpolation to unify the size of the four feature maps into H f ×W f ×C0, where H f is the maximum height of the four feature maps, W f is the largest width among the four feature maps;

[0025] The four feature maps of uniform size are added together to obtain the fused feature map X f , size is H f ×W f ×C0.

[0026] The method for generating an entropy feature map in step 3 of the present invention includes:

[0027] For a feature map X of size H×W×C0, where H is the height of the feature map and W is the width of the feature map, an entropy feature map Y of size H×W×1 is generated. The process includes:

[0028] Let each element on the feature map X be x i,j,k , where 1≤i≤H,1≤j≤W,1≤k≤C0;

[0029] set up:

[0030]

[0031] Among them, S i,j,k is to calculate Y i,j The intermediate variable is:

[0032]

[0033] Among them, Y i,j is the value at position (i, j) in the entropy feature map Y; finally, the entropy feature map Y is obtained.

[0034] The method for generating a layer of fused entropy feature map in step 3 of the present invention comprises: for the four layers of entropy feature maps obtained in step 3, the fused feature map obtaining method described in step 2 is used to obtain a layer of fused entropy feature map.

[0035] The method for calculating the foreground-background separation loss in step 4 of the present invention includes:

[0036] Suppose each element of the entropy feature map Y of size H×W×1 is y i,j , where 1≤i≤H,1≤j≤W;

[0037] Get the foreground background loss L fb for:

[0038]

[0039] According to the obtained foreground background separation loss L fb , update the parameters in the convolutional neural network through the backpropagation method (see Werbos PJ. Backpropagation through time: what it does and how to do it [J]. Proceedings of the IEEE, 1990, 78 (10): 1550-1560.).

[0040] The method for calculating entropy consistency loss in step 5 of the present invention includes:

[0041] Let entropy feature map Y m , where 1≤m≤4, represents the mth of the four entropy feature maps, whose size is H m ×W m ×1, each element is y i,j,m ;

[0042] Assume that the fusion entropy feature map Y f Size H f ×W f ×1, each element is y i,j ; The entropy feature map Y m Upsample and fuse entropy feature map Y f Consistent size;

[0043] Get the entropy consistency loss L c for:

[0044]

[0045] According to the obtained entropy consistency loss L c , the parameters in the convolutional neural network are updated through the back-propagation method.

[0046] The method for calculating the classification loss described in step 1 and step 2 of the present invention includes:

[0047] For a feature map X of size H×W×C0, each element on the feature map Z obtained after the convolution layer and the global average pooling operation is z k , where 1≤k≤C0;

[0048] Let the vector G of size C0 be G, and each element of G be g k , the element corresponding to the true category is 1, and the other positions are zero, and the confidence q is calculated k :

[0049]

[0050] Its classification loss L cls for:

[0051]

[0052] According to the classification loss L cls , update the parameters in the convolutional neural network through the back-propagation method;

[0053] The global average pooling operation method includes:

[0054] The global average pooling layer performs a pooling operation on the feature maps extracted by the convolutional layer, that is, it calculates the average value of each feature map in the spatial domain. For a feature map X of size H×W×C, let each element on the feature map be x i,j,k , where 1≤i≤H,1≤j≤W,1≤k≤C, C is the number of feature map channels;

[0055] Then the feature map after the global average pooling operation becomes a feature map Z of size C, and each element is z k , where 1≤k≤C; then:

[0056]

[0057] The method for obtaining the image category by fusing the feature map in step 8 of the present invention includes:

[0058] For a size of H f ×W f ×C0 fusion feature map X f , each element on the feature map Z obtained after the global average pooling operation is z k , where 1≤k≤C0; calculate:

[0059]

[0060] Among them, the largest q k The corresponding serial number is used as the category k = argmax k (q k ).

[0061] The method for obtaining the position of the target in the image by entropy fusion feature map in step 9 of the present invention includes:

[0062] A threshold t is set in advance, the coordinates of the values ​​greater than the threshold in the entropy fusion feature map are taken out, and the maximum and minimum values ​​in the x-axis coordinates and the y-axis coordinates are taken out to obtain the bounding rectangle of the target.

[0063] Beneficial effects:

[0064] The present invention provides a weakly supervised target positioning method based on entropy consistency. The method introduces entropy as class-independent information and makes full use of shallow features. By constraining the entropy consistency between layers, shallow location information is better mined. Compared with the method based on class activation mapping, the method described in the present invention is not based on category information and can obtain more comprehensive features; compared with the method based on foreground learning, the method described in the present invention can train the entire network end-to-end without multi-stage training. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0066] Figure 1 It is a schematic diagram of the process of the present invention.

[0067] Figure 2 The figure is a schematic diagram of target positioning using the method of the present invention. DETAILED DESCRIPTION

[0068] like Figure 1 As shown, a weakly supervised target positioning method based on entropy consistency includes the following steps:

[0069] Training phase:

[0070] Step 1: Input the training set images including images and category annotations into the convolutional neural network to generate four layers of feature maps, and calculate the classification loss respectively;

[0071] Among them, the method of generating a four-layer feature map includes:

[0072] The convolutional neural network VGG16 is used, and the picture is input into the convolutional neural network to obtain feature maps with downsampling ratios of 4, 8, 16, and 16 as four-layer feature maps. The number of channels of the four-layer feature maps is changed to C0 through the convolution layer, where C0 is the number of categories.

[0073] Step 2: Fuse the four-layer feature maps obtained in step 1 to obtain a fused feature map and calculate the classification loss.

[0074] The method for obtaining the fused feature map includes:

[0075] Let feature map X m , where 1≤m≤4, represents the mth of the four feature maps, whose size is H m ×W m ×C0, where H m is the height of the mth feature map, W m is the width of the mth feature map;

[0076] Perform bilinear interpolation to unify the size of the four feature maps into H f ×W f ×C0, where H f is the maximum height of the four feature maps, W f is the largest width among the four feature maps;

[0077] The four feature maps of uniform size are added together to obtain the fused feature map X f , size is H f ×W f ×C0.

[0078] The method for calculating the classification loss described in steps 1 and 2 includes:

[0079] For a feature map X of size H×W×C0, each element on the feature map Z obtained after the convolution layer and the global average pooling operation is z k , where 1≤k≤C0;

[0080] Let the vector G of size C0 be G, and each element of G be g k , the element corresponding to the true category is 1, and the other positions are zero, and the confidence q is calculated k :

[0081]

[0082] Its classification loss L cls for:

[0083]

[0084] According to the classification loss L cls , update the parameters in the convolutional neural network through the back-propagation method;

[0085] The global average pooling operation method includes:

[0086] The global average pooling layer performs a pooling operation on the feature maps extracted by the convolutional layer, that is, it calculates the average value of each feature map in the spatial domain. For a feature map X of size H×W×C, let each element on the feature map be x i,j,k , where 1≤i≤H,1≤j≤W,1≤k≤C, C is the number of feature map channels;

[0087] Then the feature map after the global average pooling operation becomes a feature map Z of size C, and each element is z k , where 1≤k≤C; then:

[0088]

[0089] Step 3, generating four layers of entropy feature maps and one layer of fused entropy feature map respectively from the four layers of feature maps obtained in step 1 and the fused feature map obtained in step 2;

[0090] Among them, the method for generating an entropy feature map includes:

[0091] For a feature map X of size H×W×C0, where H is the height of the feature map and W is the width of the feature map, an entropy feature map Y of size H×W×1 is generated. The process includes:

[0092] Let each element on the feature map X be x i,j,k, where 1≤i≤H,1≤j≤W,1≤k≤C0;

[0093] set up:

[0094]

[0095] Among them, S i,j,k is to calculate Y i,j The intermediate variable is:

[0096]

[0097] Among them, Y i,j is the value at position (i, j) in the entropy feature map Y; finally, the entropy feature map Y is obtained.

[0098] The method for generating a layer of fused entropy feature map includes: for the four layers of entropy feature maps obtained in step 3, using the fused feature map obtaining method described in step 2 to obtain a layer of fused entropy feature map.

[0099] Step 4, the one-layer fusion entropy feature map and the four-layer entropy feature map respectively calculate the foreground-background separation loss;

[0100] Among them, the method for calculating the foreground background separation loss includes:

[0101] Suppose each element of the entropy feature map Y of size H×W×1 is y i,j , where 1≤i≤H,1≤j≤W;

[0102] Get the foreground background loss L fb for:

[0103]

[0104] According to the obtained foreground background separation loss L fb , update the parameters in the convolutional neural network through the backpropagation method (see Werbos PJ. Backpropagation through time: what it does and how to do it [J]. Proceedings of the IEEE, 1990, 78 (10): 1550-1560.).

[0105] Step 5, calculate the entropy consistency loss of the four layers of entropy feature maps obtained in step 3 and the one layer of fusion entropy feature map respectively;

[0106] Among them, the method for calculating entropy consistency loss includes:

[0107] Let entropy feature map Y m , where 1≤m≤4, represents the mth of the four entropy feature maps, whose size is Hm ×W m ×1, each element is y i,j,m ;

[0108] Assume that the fusion entropy feature map Y f Size H f ×W f ×1, each element is y i,j ; The entropy feature map Y m Upsample and fuse entropy feature map Y f Consistent size;

[0109] Get the entropy consistency loss L c for:

[0110]

[0111] According to the obtained entropy consistency loss L c , the parameters in the convolutional neural network are updated through the back-propagation method.

[0112] Step 6: Update the convolutional neural network using the classification loss obtained in steps 1 and 2 and the foreground-background separation loss and entropy consistency loss obtained in steps 4 and 5.

[0113] Testing phase:

[0114] Step 7, input the unlabeled image into the convolutional neural network trained in the training phase to obtain a fused entropy feature map and a fused feature map;

[0115] Step 8, obtaining the image category through the fusion feature map obtained in step 7, the method includes:

[0116] For a size of H f ×W f ×C0 fusion feature map X f , each element on the feature map Z obtained after the global average pooling operation is z k , where 1≤k≤C0; calculate:

[0117]

[0118] Among them, the largest q k The corresponding serial number is used as the category k = argmax k (q k ).

[0119] Step 9, obtaining the position of the target in the image through the fusion entropy feature map obtained in step 7, the method includes:

[0120] A threshold t is set in advance, the coordinates of the values ​​greater than the threshold in the entropy fusion feature map are taken out, and the maximum and minimum values ​​in the x-axis coordinates and the y-axis coordinates are taken out to obtain the bounding rectangle of the target.

[0121] like Figure 2 As shown, the solid arrows represent the process of the training phase, and the dotted arrows represent the process of the testing phase:

[0122] During the training phase, the image and the corresponding category are input into the convolutional neural network to obtain four layers of feature maps X1, X2, X3, and X4, and the fusion feature map X is obtained through the four layers of feature maps. f , calculate the classification loss and update the network parameters respectively.

[0123] The four-layer fusion feature map and the one-layer fusion feature map are respectively generated into four-layer entropy feature maps Y1, Y2, Y3, Y4 and a layer of fusion entropy feature map Y f The four layers of entropy feature maps calculate the foreground and background loss and the entropy consistency loss to update the network parameters. f Calculate the foreground background loss and update the network parameters.

[0124] In the test phase, only the image is input into the convolutional neural network to obtain the fusion entropy feature map Y f And the fusion feature map X f , and get the category and bounding box respectively.

[0125] The present invention provides a method and idea for a weakly supervised target positioning method based on entropy consistency. There are many methods and approaches to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A weakly supervised target localization method based on entropy consistency, characterized in that: The following steps are involved: Training phase: Step 1: Input the training set images including images and category annotations into the convolutional neural network to generate four layers of feature maps, and calculate the classification loss respectively; Step 2: Fuse the four-layer feature maps obtained in step 1 to obtain a fused feature map and calculate the classification loss. Step 3, generating four layers of entropy feature maps and one layer of fused entropy feature map respectively from the four layers of feature maps obtained in step 1 and the fused feature map obtained in step 2; Step 4, the one-layer fusion entropy feature map and the four-layer entropy feature map respectively calculate the foreground-background separation loss; Step 5, calculating the entropy consistency loss of the four layers of entropy feature maps obtained in step 3 and the one layer of fusion entropy feature map respectively; Step 6: Update the convolutional neural network using the classification loss obtained in steps 1 and 2 and the foreground-background separation loss and entropy consistency loss obtained in steps 4 and 5. Testing phase: Step 7, input the unlabeled image into the convolutional neural network trained in the training phase to obtain a fused entropy feature map and a fused feature map; Step 8, obtaining the image category through the fusion feature map obtained in step 7; Step 9, obtaining the position of the target in the image through the fusion entropy feature map obtained in step 7; The method for generating the entropy feature map in step 3 includes: For a feature map X of size H×W×C0, where H is the height of the feature map and W is the width of the feature map, an entropy feature map Y of size H×W×1 is generated. The process includes: Let each element on the feature map X be x i,j,k , where 1≤i≤H,1≤h≤W,1≤k≤C0; set up: Among them, S i,j,k is to calculate Y i,j The intermediate variable is: Among them, Y i,j is the value at position (i, j) in the entropy feature map Y; finally, the entropy feature map Y is obtained; The method for calculating entropy consistency loss in step 5 includes: Let entropy feature map Y m , where 1≤m≤4, represents the mth of the four entropy feature maps, whose size is H m ×W m ×1, each element is y i,j,m ; Assume that the fusion entropy feature map Y f Size H f ×W f ×1, each element is y i,j ; The entropy feature map Y m Upsample and fuse entropy feature map Y f Consistent size; Get the entropy consistency loss L c for: According to the obtained entropy consistency loss L c , update the parameters in the convolutional neural network through the back-propagation method; The method for calculating the classification loss described in steps 1 and 2 includes: For a feature map X of size H×W×C0, each element on the feature map Z obtained after the convolution layer and the global average pooling operation is z k , where 1≤k≤C0; Let the vector G of size C0 be G, and each element of G be g k , the element corresponding to the true category is 1, and the other positions are zero, and the confidence q is calculated k : Its classification loss L cls for: According to the classification loss L cls , update the parameters in the convolutional neural network through the back-propagation method; The global average pooling operation method includes: The global average pooling layer performs a pooling operation on the feature maps extracted by the convolutional layer, that is, it calculates the average value of each feature map in the spatial domain. For a feature map X of size H×W×C, let each element on the feature map be x i,j,k , where 1≤i≤H,1≤j≤W,1≤k≤C, C is the number of feature map channels; Then the feature map after the global average pooling operation becomes a feature map Z of size C, and each element is z k , where 1≤k≤C; then:

2. According to the entropy consistency-based weakly supervised target positioning method of claim 1, it is characterized in that: The method for generating a four-layer feature map in step 1 includes: The convolutional neural network VGG16 is used, and the picture is input into the convolutional neural network to obtain feature maps with downsampling ratios of 4, 8, 16, and 16 as four-layer feature maps. The number of channels of the four-layer feature maps is changed to C0 through the convolution layer, where C0 is the number of categories.

3. A weakly supervised target positioning method based on entropy consistency according to claim 2, characterized in that: The method for obtaining the fusion feature map described in step 2 includes: Let feature map X m , where 1≤m≤4, represents the mth feature map in the four feature maps, and its size is H m ×W m ×C0, where H m is the height of the mth feature map, W m is the width of the mth feature map; Perform bilinear interpolation to unify the size of the four feature maps into H f ×W f ×C0, where H f is the maximum height of the four feature maps, W f is the largest width among the four feature maps; The four feature maps of uniform size are added together to obtain the fused feature map X f , size is H f ×W f ×C0.

4. The method for weakly supervised target positioning based on entropy consistency according to claim 3 is characterized in that: The method for generating a layer of fused entropy feature map in step 3 includes: for the four layers of entropy feature maps obtained in step 3, the fused feature map obtaining method described in step 2 is used to obtain a layer of fused entropy feature map.

5. The method for weakly supervised target positioning based on entropy consistency according to claim 4 is characterized in that: The method for calculating the foreground-background separation loss described in step 4 includes: Suppose each element of the entropy feature map Y of size H×W×1 is y i,j , where 1≤i≤H,1≤j≤W; Get the foreground background loss L fb for: According to the obtained foreground background separation loss L fb , the parameters in the convolutional neural network are updated through the back-propagation method.

6. The method for weakly supervised target positioning based on entropy consistency according to claim 5, characterized in that: The method for obtaining the image category by fusing the feature map in step 8 includes: For a size of H f ×W f ×C0 fusion feature map X f , each element on the feature map Z obtained after the global average pooling operation is z k , where 1≤k≤C0; calculate: Among them, the largest q k The corresponding serial number is used as the category k = argmax k (q k ).

7. The method for weakly supervised target positioning based on entropy consistency according to claim 6, characterized in that: The method for obtaining the position of the target in the image by fusing the feature map with entropy in step 9 includes: A threshold t is set in advance, the coordinates of the values ​​greater than the threshold in the entropy fusion feature map are taken out, and the maximum and minimum values ​​in the x-axis coordinates and the y-axis coordinates are taken out to obtain the bounding rectangle of the target.

Images