Self-supervised and interpolation-free weakly supervised object localization method, medium and device

Abstract

The application discloses a kind of weakly supervised object positioning method based on self-supervision and without interpolation, medium and equipment, belong to image object positioning field.The method includes the following steps: obtaining a classification network with pooling layer and a positioning network without pooling layer;Target image is respectively input into the classification network and positioning network, the image prediction class probability distribution is output by classification network, and the image prediction class probability distribution is multiplied with the image feature output by the last convolution block in positioning network, to obtain the same size heat map as target image, finally, the object position in image is segmented by threshold method using pixel heat value.The application completes accurate classification of image in classification network, while retaining all image information in positioning network, to complete accurate object positioning.Moreover, positioning network can generate high-definition heat map with the same size as original image, effectively ensuring the definition of image heat map and the accuracy of positioning result.

Description

Technical Field

[0001] This invention belongs to the field of image object localization, and particularly relates to a self-supervised and interpolation-free weakly supervised object localization method, medium and device. Background Technology

[0002] In recent years, object localization techniques based on deep convolutional neural networks have reached a new level. However, this performance improvement comes at the cost of extensive fine-grained manual annotation. To alleviate this heavy burden, weakly supervised object localization methods that rely solely on image-level labels have been proposed.

[0003] Existing weakly supervised object localization models are essentially image classification networks, which have two main drawbacks in weakly supervised object localization tasks:

[0004] First, to achieve accurate image classification, classification networks continuously compress image features using pooling layers during data processing, ultimately retaining only the features of the semantically richest local regions of the object to achieve accurate image classification. Therefore, in image localization, weakly supervised object localization models often only locate the most sensitive regions of the object, ignoring other areas.

[0005] Secondly, because classification networks use pooling layers, the generated image heatmaps are much smaller than the original images. To locate objects in the original images, weakly supervised object localization models need to use interpolation methods to enlarge the image heatmaps to the original image size, which degrades the clarity of the image heatmaps and thus affects the localization results. Summary of the Invention

[0006] The purpose of this invention is to solve the problem of low positioning accuracy in the prior art and to provide a weakly supervised object positioning method based on self-supervision and without interpolation.

[0007] This invention is specifically implemented through the following technical solutions:

[0008] In a first aspect, the present invention provides a weakly supervised object localization method based on self-supervision and without interpolation, which includes the following steps:

[0009] S1: Obtain a pre-trained classification network and a localization network, both networks having the same number of one-to-one corresponding convolutional blocks; in the classification network, all convolutional blocks except the last one have pooling layers, and the supervision data during its training is the image category; in the localization network, the convolutional blocks do not have pooling layers, and the supervision data during its training is the convolutional layer features output by the corresponding convolutional blocks in the classification network.

[0010] S2: Input the target image into the classification network and the localization network respectively. The classification network outputs the image prediction category probability distribution. The image prediction category probability distribution is multiplied with the image features output by the last convolutional block in the localization network to obtain a heatmap with the same size as the target image. Finally, the position of the object in the image is segmented by using the pixel heat value through the thresholding method to achieve localization.

[0011] As a preferred embodiment of the first aspect, the classification network is composed of k convolutional blocks with pooling layers, one convolutional block without pooling layers, and a global average pooling layer cascaded in sequence, where k ≥ 5; after each convolutional block with pooling layers, the width and height of the generated image feature map are reduced by half, while after passing through the convolutional block without pooling layers, the width and height of the generated image feature map remain unchanged.

[0012] As a preferred embodiment of the first aspect, the localization network is composed of k+1 concatenated convolutional blocks without pooling layers, and the network output is the result of pixel heat value segmentation of the heat map, and k≥5; after passing through the convolutional blocks without pooling layers, the width and height of the generated image feature map remain unchanged.

[0013] As a preferred embodiment of the first aspect above, the classification network and the localization network are jointly trained iteratively in advance, and the training samples are in the form of (I,Y) data, where I and Y represent the training image and the true category label of the training image, respectively.

[0014] In each iteration, the training image I is input into the classification network. First, the k+1 convolutional blocks in the classification network generate corresponding k+1 image feature maps {F1, F2, ..., F...}. k+1 The image feature map F is output by the last convolutional block. k+1 The image prediction category probability distribution P is obtained through a global average pooling layer. On the other hand, the training image I is input into the localization network, and k+1 corresponding image feature maps are generated by k+1 convolutional blocks. The classification loss in the form of cross-entropy is calculated based on the true class label Y of the training images and the predicted class probability distribution P of the images, and based on the image feature maps {F1,F2,…,F…} k+1} and image feature maps Calculate the KL divergence loss, and use the sum of the classification loss and the KL divergence loss as the total loss to update the network parameters of the classification network and the localization network using gradient descent.

[0015] As a preferred embodiment of the first aspect above, the total loss is in the functional form of:

[0016]

[0017] Where C represents the number of image categories, P i and Y i These are the i-th elements of P and Y, respectively; KL() represents the KL divergence function, Pool i () indicates that a total of i pooling operations have been performed.

[0018] As a preferred embodiment of the first aspect, each convolutional block of the classification network and the localization network has 2 to 3 convolutional layers, and each convolutional layer has an activation function.

[0019] As a preferred embodiment of the first aspect above, when segmenting the object position in the image using the threshold method, the heat value of each pixel in the heat map is first calculated, and the object instance region is determined according to the preset threshold T. If the heat value of a pixel is greater than the threshold T, it is classified as an object instance region; otherwise, it is classified as background. Preferably, T is 0.2 times the maximum heat value of the pixel in the heat map H.

[0020] As a preferred embodiment of the first aspect, after determining the object instance region of all pixels in the image, the result of object localization is output as a minimum bounding box that can enclose all pixels belonging to the object instance region.

[0021] In a second aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a weakly supervised object localization method based on self-supervision and without interpolation as described in any of the solutions of the first aspect above.

[0022] Thirdly, the present invention provides a computer electronic device, which includes a memory and a processor;

[0023] The memory is used to store computer programs;

[0024] The processor is configured to, when executing the computer program, implement the weakly supervised object localization method based on self-supervision and without interpolation as described in any of the solutions of the first aspect above.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1) This invention proposes a weakly supervised object localization method based on self-supervised learning and without the need for interpolation algorithms. Based on self-supervised learning, this invention can accurately classify images in the classification network CLS while retaining all image information in the localization network to achieve accurate object localization.

[0027] 2) This invention can generate a high-definition heat map H with the same size as the original image, effectively avoiding the use of interpolation algorithms in traditional weakly supervised object localization methods to enlarge the heat map to the size of the original image, thereby damaging the clarity of the image heat map and the localization results. Attached Figure Description

[0028] Figure 1 This is a flowchart of a weakly supervised object localization method based on self-supervision and without interpolation.

[0029] Figure 2 This is a network structure diagram of a weakly supervised object localization method based on self-supervision and no interpolation.

[0030] Figure 3 The images show the interpolation method and the method of this invention in terms of image heatmaps and location bounding boxes. Detailed Implementation

[0031] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0032] This invention addresses a fundamental task in computer vision: weakly supervised object localization. In this task, given an image, accurate classification and object localization within the image are required. The specific implementation process of this invention is described in detail below.

[0033] See Figure 1 As shown, in a preferred embodiment of the present invention, a weakly supervised object localization method based on self-supervised learning and without the need for interpolation algorithms is provided, the basic steps of which are as follows:

[0034] S1: Obtain a pre-trained classification network CLS and a localization network LOC, both networks having the same number of one-to-one corresponding convolutional blocks. In the classification network CLS, all convolutional blocks except the last one have pooling layers, and its training supervision data is the image category; while in the localization network LOC, all convolutional blocks do not have pooling layers, and its training supervision data is the convolutional layer features output by the corresponding convolutional blocks in the classification network CLS.

[0035] S2: Input the target image into the classification network CLS and the localization network LOC respectively. The classification network CLS outputs the image prediction category probability distribution, and multiplies the image prediction category probability distribution with the image features output by the last convolutional block in the localization network LOC to obtain a heat map with the same size as the target image. Finally, the object position in the image is segmented by using the pixel heat value through the thresholding method to achieve localization.

[0036] In the implementation of the above method, the core classification network CLS and the localization network LOC together constitute the model framework for image classification and object localization. This model framework is as follows: Figure 2 As shown: First, the classification network CLS with pooling layers is used to obtain the category P of the object in the image; second, the localization network LOC without pooling layers is used to obtain a high-resolution heatmap H of the same size as the original image; finally, the heat value of the pixels is used to determine the final object bounding box.

[0037] exist Figure 2 In the model framework shown, the classification network CLS has pooling layers. Specifically, the classification network CLS consists of k convolutional blocks with pooling layers, one convolutional block without pooling layers, and a global average pooling layer cascaded sequentially. Each time the image feature map passes through a convolutional block with a pooling layer, its width and height are halved, while the width and height of the image feature map generated by the convolutional block without pooling layers remain unchanged. Let the k+1 image features output by the k+1 convolutional blocks in the classification network CLS be denoted as {F1, F2, ..., F...}. k+1}, then the size relationship of their corresponding feature maps can be expressed by the formula:

[0038]

[0039] w(F k+1 )=w(F k ),h(F k+1 )=h(F k ), i = k

[0040] Where w() represents the width of the feature map and h() represents the height of the feature map.

[0041] The localization network (LOC) consists of k+1 cascaded convolutional blocks without pooling layers, and its output is the result of pixel heatmap segmentation. The width and height of the image feature map generated by the convolutional blocks without pooling layers remain unchanged. Let the k+1 image features output by the k+1 convolutional blocks in the LOC be denoted as... The size relationship of their corresponding feature maps can be expressed by the formula:

[0042]

[0043] It should also be noted that, in principle, the number of convolutional blocks in the classification network CLS and the localization network LOC can be optimized according to the actual situation, as long as k≥5. In the embodiment of this invention, k is optimized to 5. Furthermore, each convolutional block in the classification and localization networks has 2 to 3 convolutional layers, and each convolutional layer has an activation function. The specific number of layers needs to be adjusted according to the actual situation. In the embodiment of this invention, in the six convolutional blocks of the classification network CLS, the 1st, 2nd, and 6th convolutional blocks use 2 convolutional layers, and the 3rd, 4th, and 5th convolutional blocks use 3 convolutional layers. The kernel size is (3, 3), the stride is 1, and the activation function is LeakyReLU. Similarly, in the six convolutional blocks of the localization network LOC, the 1st, 2nd, and 6th convolutional blocks use 2 convolutional layers, and the 3rd, 4th, and 5th convolutional blocks use 3 convolutional layers. The kernel size is (3, 3), the stride is 1, and the activation function is LeakyReLU.

[0044] The above Figure 2 The weakly supervised object localization framework shown requires simultaneous pre-training and optimization of the classification network CLS and the localization network LOC when actually used for object localization. In the embodiments of this invention, when the classification network and the localization network are jointly trained iteratively, the training samples used are in the form of (I,Y) data, where I and Y represent the training image and the true class label of the training image, respectively. The training of the classification network CLS and the localization network LOC is performed simultaneously. In each iteration, the training image I in the training sample (I,Y) needs to be input into both the classification network CLS and the localization network LOC. The parameters of both are updated by calculating the loss value. The training process of each iteration is described in detail below.

[0045] In each iteration, the training image I is input into the classification network. First, the k+1 convolutional blocks in the classification network generate corresponding k+1 image feature maps {F1, F2, ..., F...}. k+1 The image feature map F is output by the last convolutional block. k+1 The image prediction category probability distribution P is obtained through a global average pooling layer. On the other hand, the training image I is input into the localization network, and k+1 corresponding image feature maps are generated by k+1 convolutional blocks. In this embodiment, the optimization objective is to ensure both high-accuracy image classification and high-accuracy object localization. Therefore, the loss of the weakly supervised object localization model of this invention is... It can be divided into two parts: one part is the classification loss in the form of cross-entropy calculated based on the true class label Y of the training image and the probability distribution P of the predicted class of the image; the other part is based on the image feature map {F1,F2,…,F…} k+1} and image feature maps The calculated KL divergence loss is used in this invention to update the network parameters of the classification network and the localization network by gradient descent after the forward propagation of one iteration. The sum of the classification loss and the KL divergence loss is used as the total loss.

[0046] In an embodiment of the present invention, the total loss formed by adding the classification loss and the KL divergence loss described above is... The function form can be expressed as follows:

[0047]

[0048] Where C represents the number of image categories, P i and Y i These are the i-th elements of P and Y, respectively; KL() represents the KL divergence function, Pool i () indicates that a total of i pooling operations were performed, F i This represents the output feature map of the i-th convolutional block in the classification network CLS. This represents the output feature map of the i-th convolutional block of the Localization Network (LOC).

[0049] Through the above joint training steps, a high-performance classification network CLS and localization network LOC can be trained. By inputting the detection image, it can output accurate image classification and accurate image heatmap.

[0050] In the model framework of this invention, the Localization Network (LOC) without pooling layers can obtain a high-resolution heatmap H of the same size as the original image. Based on this high-resolution heatmap H, a thresholding method can be used to identify whether each pixel belongs to an object instance region, and then the final object bounding box can be determined using the pixel's heat value. In an embodiment of this invention, when segmenting the object position in the image using the above-mentioned thresholding method, the specific method is as follows: First, calculate the heat value of each pixel in the heatmap. Then, determine the object instance region according to a preset threshold T. If the heat value of a pixel is greater than the threshold T, it is classified as an object instance region; otherwise, it is classified as background. It should be noted that the specific value of the above-mentioned threshold T needs to be optimized according to the actual dataset. In an embodiment of this invention, T can be optimized to 0.2 times the maximum heat value of the pixel in the heatmap H.

[0051] It is important to note that the predicted category output by the classification network CLS It is actually a probability distribution, which is compared with the feature map output by the last convolutional block of the localization network. Multiplication yields the heatmap of the detected image. The formula is expressed as follows:

[0052]

[0053] In another embodiment of the present invention, in order to more clearly and intuitively display the object localization result, after determining the object instance region of all pixels in the image, a minimum bounding box that can enclose all pixels belonging to the object instance region can be used as the object localization result for output.

[0054] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides an electronic device corresponding to the self-supervised and interpolation-free weakly supervised object localization method provided in the above embodiments, which includes a memory and a processor;

[0055] The memory is used to store computer programs;

[0056] The processor is configured to implement, when executing the computer program, the weakly supervised object localization method based on self-supervision and without interpolation described in any of the foregoing embodiments.

[0057] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0058] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the self-supervised and interpolation-free weakly supervised object localization method provided in the above embodiments. The storage medium stores a computer program that, when executed by a processor, can realize the self-supervised and interpolation-free weakly supervised object localization method described in any of the foregoing embodiments.

[0059] Specifically, in the computer-readable storage medium of the two embodiments described above, the stored computer program is executed by a processor, which can perform the aforementioned steps S1 to S2.

[0060] It is understood that the aforementioned storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage media may also be various media capable of storing program code, such as USB flash drives, external hard drives, magnetic disks, or optical discs.

[0061] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0062] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0063] The present invention will now demonstrate the application effect of the weakly supervised object localization method based on self-supervised learning and without interpolation algorithm described in S1 to S2 of the above embodiments on a specific dataset through a specific example, so as to facilitate understanding of the essence of the present invention.

[0064] Example

[0065] This embodiment uses the publicly available bird dataset (CUB-200-2011). The CUB-200-2011 dataset is an extension of the Caltech-UCSD Birds 200 (CUB-200) dataset, containing a collection of 200 bird species, primarily used for exploring genus classification. CUB-200-2011 includes 5994 images in the training set and 5794 images in the test set. The training set only has image category labels, while the test set has image category labels and localization bounding boxes for bird instances. The specific implementation steps in this embodiment are as described in S1-S2 above, where the classification network CLS and the localization network LOC are trained simultaneously for 100 epochs before performance evaluation.

[0066] To objectively evaluate the performance of this algorithm, this embodiment uses two evaluation metrics widely used in weakly supervised object localization tasks: accuracy and maximum bounding box accuracy (MaxBoxAccV2). Accuracy includes classification accuracy (Top-1 Cls), localization accuracy (Top-1 Loc), and class-independent localization accuracy (GT-known). Classification accuracy (Top-1 Cls) represents the correctness of the highest prediction score; localization accuracy (Top-1 Loc) represents the accuracy of evaluating both class prediction and bounding box localization; and class-independent localization accuracy (GT-known) represents the accuracy of localization prediction without considering classification results. Maximum bounding box accuracy (MaxBoxAccV2) provides a more comprehensive evaluation by averaging the performance over three intersection-union (IoU) thresholds (e.g., 0.3, 0.5, 0.7), addressing the impact of different segmentation thresholds on model localization performance to meet varying localization accuracy requirements.

[0067] like Figure 2 The diagram shows the network model framework structure in this embodiment. (a) shows the network structure of the classification network CLS, and (b) shows the network structure of the localization network LOC. The red arrows in (a) represent the supervision data for the classification network CLS, and the red arrows between (a) and (b) represent the output features of the classification network CLS used as training supervision data for the localization network LOC. Therefore, it can be foreseen that when the classification network CLS is trained to a high classification performance, the localization network LOC can also obtain accurate image features to achieve precise object localization.

[0068] Finally, this embodiment also compares the performance differences between traditional interpolation methods and the method of this invention. The comparison of the effects of traditional interpolation methods and the trained model used in this embodiment for weakly supervised object localization is as follows: Figure 3 As shown, this embodiment randomly selected several bird images, and then our method's model accurately marked the specific locations within the objects.

[0069] In addition, the experimental results obtained in this embodiment are shown in Table 1. The results show that although this method is simple, it still has high accuracy and maximum bounding box accuracy (MaxBoxAccV2).

[0070] Table 1

[0071] 73.7 48.5 64.7 31.5

[0072] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A weakly supervised object localization method based on self-supervision and without interpolation, characterized in that, Includes the following steps: S1: Obtain a pre-trained classification network and a localization network, both networks having the same number of one-to-one corresponding convolutional blocks; in the classification network, all convolutional blocks except the last one have pooling layers, and the supervision data during its training is the image category; in the localization network, the convolutional blocks do not have pooling layers, and the supervision data during its training is the convolutional layer features output by the corresponding convolutional blocks in the classification network. S2: Input the target image into the classification network and the localization network respectively. The classification network outputs the image prediction category probability distribution. The image prediction category probability distribution is multiplied with the image features output by the last convolutional block in the localization network to obtain a heatmap with the same size as the target image. Finally, the position of the object in the image is segmented by using the pixel heat value through the thresholding method to achieve localization.

2. The weakly supervised object localization method based on self-supervision and without interpolation as described in claim 1, characterized in that, The classification network consists of k convolutional blocks with pooling layers, one convolutional block without pooling layers, and a global average pooling layer cascaded in sequence, where k ≥ 5. Each time the image feature map is passed through a convolutional block with pooling layers, the width and height of the generated image feature map are reduced by half, while the width and height of the generated image feature map remain unchanged after passing through a convolutional block without pooling layers.

3. The weakly supervised object localization method based on self-supervision and without interpolation as described in claim 2, characterized in that, The localization network is composed of k+1 concatenated convolutional blocks without pooling layers, and the network output is the result of pixel heat value segmentation of the heat map, and k≥5; after passing through the convolutional blocks without pooling layers, the width and height of the generated image feature map remain unchanged.

4. The weakly supervised object localization method based on self-supervision and without interpolation as described in claim 3, characterized in that, The classification network and the localization network are jointly trained iteratively in advance, and the training samples are in the form of (I,Y) data, where I and Y represent the training image and the true class label of the training image, respectively. In each iteration, the training image I is input into the classification network. First, the k+1 convolutional blocks in the classification network generate corresponding k+1 image feature maps {F1, F2, ..., F...}. k+1 The image feature map F is output by the last convolutional block. k+1 The image prediction category probability distribution P is obtained through a global average pooling layer. On the other hand, the training image I is input into the localization network, and k+1 corresponding image feature maps are generated by k+1 convolutional blocks. The classification loss in the form of cross-entropy is calculated based on the true class label Y of the training images and the predicted class probability distribution P of the images, and based on the image feature maps {F1,F2,…,F…} k+1 } and image feature maps Calculate the KL divergence loss, and use the sum of the classification loss and the KL divergence loss as the total loss to update the network parameters of the classification network and the localization network using gradient descent.

5. The weakly supervised object localization method based on self-supervision and without interpolation as described in claim 4, characterized in that, The total loss is in the functional form of: Where C represents the number of image categories, P i and Y i Let i be the i-th element of P and Y, respectively; KL() represents the KL divergence function, Pool i () indicates that a total of i pooling operations have been performed.

6. The weakly supervised object localization method based on self-supervision and without interpolation as described in claim 1, characterized in that, Each convolutional block of the classification network and the localization network has 2 to 3 convolutional layers, and each convolutional layer has an activation function.

7. The weakly supervised object localization method based on self-supervision and without interpolation as described in claim 1, characterized in that, When segmenting the object location in an image using the threshold method, the heat value of each pixel in the heat map is first calculated. The object instance region is determined according to a preset threshold T. If the heat value of a pixel is greater than the threshold T, it is classified as an object instance region; otherwise, it is classified as background. Preferably, T is 0.2 times the maximum heat value of the pixel in the heat map H.

8. The weakly supervised object localization method based on self-supervision and without interpolation as described in claim 7, characterized in that, After determining the object instance region for all pixels in the image, the result of object localization is output as the smallest bounding box that can enclose all pixels belonging to the object instance region.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the weakly supervised object localization method based on self-supervision and without interpolation as described in any one of claims 1 to 8.

10. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the weakly supervised object localization method based on self-supervision and without interpolation as described in any one of claims 1 to 8.

Images