Weakly supervised object localization method and device based on shallow feature background suppression

Abstract

The application discloses a weakly supervised target positioning method and device based on shallow feature background suppression, and the method is as follows: acquiring an image data set; constructing a target positioning classification network based on a neural network, including a classifier, a generator, and an evaluator; the classifier generates a shallow feature map and initializes the generator to obtain a foreground activation map; the foreground activation map is point multiplied with the shallow feature map, input into the evaluator to obtain a classification result and calculate the foreground cross-entropy loss of the classification result and an original label; a PCA method is used to analyze the background principal components of each image in the training data set, and a background activation map is obtained through clustering and input into the evaluator to calculate a background classification loss; a foreground-background adversarial loss is introduced and a Transformer is used to assist the generator in learning, so that a total loss function is obtained; and a test data set is input to obtain a positioning result. The method uses shallow feature information to suppress the background to train the target positioning network, and the classification and positioning tasks are decoupled, so that the limitation of the classification on the positioning effect is avoided.

Description

Technical Field

[0001] This invention belongs to the technical field of weakly supervised target localization in computer vision, specifically relating to a weakly supervised target localization method and apparatus based on shallow feature background suppression. Background Technology

[0002] Weakly supervised target localization is a sub-problem of target localization. Compared with fully supervised target localization (given object bounding box and category), weakly supervised localization only needs to use more readily available annotation information as supervision signals, such as image-level labels, to achieve better localization results. However, how to detect the position information of the target in the image to the maximum extent using only category label information has become a major challenge in current research.

[0003] Existing technologies employ a localization learning method based on class activation maps for target localization. In the paper "A-fast-rcnn: Hard positive generation via adversary for object detection" published by Wang et al., a method for indirect target localization based on classification was proposed. This method generates an object localization map, or class activation map (CAM), by multiplying the weights of the last fully connected layer by the feature map in the classification network. This map reflects the target location information that the network focuses on based on the classification score. However, because the localization learning method based on class activation maps relies on activation maps obtained from class label information, it tends to mine the most discriminative regions. This causes the classification network to assign greater weight to these regions while ignoring other regions that are less discriminative. Another existing object localization method—the erasure learning method—belongs to the multi-stage approach. Wei et al., in their paper "Object region mining with adversarial erasing: A simple classification to semantic segmentation approach," proposed a stepwise learning method. This method first finds the most prominent features, then generates corresponding attention maps as masks to erase the target region, and finally inputs the erased feature map into the network to force the network to detect other parts of the target. Zhang et al., in their paper "Adversarial complementary learning for weakly supervised object localization," introduced an iterative erasure method to reduce background noise and gradually find features through multiple iterations. Unlike the stepwise erasure learning method, Adversarial Complementary Learning (ACoL), after erasing the most critical regions on the original feature map, utilizes two parallel classifiers to learn the original feature map and the erased feature map respectively. Based on the attention-based dropout layer (ADL), it randomly erases the most discriminative regions in the forward propagation, thereby reducing computational and parameter overhead. Furthermore, Zhang et al. proposed a three-branch erasure method in their paper "Inter-imagecommunication for weakly supervised localization," which utilizes two self-erasure strategies to enhance feature parsing and suppress background noise after multiple erasures. Although these multi-stage erasure methods have achieved promising performance, progressive erasure leads to the introduction of background noise by adversarial learning and consumes a large amount of memory resources. They are also limited by the random loss of information regions, resulting in classification degradation. Simultaneously, they ignore adversarial erasure between different images and different unerased / erased results, making it difficult for them to complement each other for comprehensive object mining.Since sample correlation is an effective method for mining reliable localization results, researchers have proposed an image pair feature correlation target localization learning method. The paper "Mining cross-image semantics for weakly supervised semanticsegmentation" proposes a random and globally consistent localization method based on image pair communication to maintain feature proximity of the same class in a high-level feature space. The paper "Show, match and segment: Joint weakly supervised learning of semantic matching and object co-segmentation" has introduced a joint attention mechanism aimed at understanding object patterns for cross-image common semantic mining within paired images. Xie et al. proposed joint segmentation in the paper "Online refinement of low-level feature based activation map for weakly supervised object localization" to use a geometric model to determine the dense correspondence between two images. However, the above methods only mine the correspondence in image pairs and ignore class-level feature correlations throughout the training process, resulting in low target localization performance. Another target localization method uses a semantic information activation map generator. It designs a two-stage weakly supervised target localization framework. In the first stage, it proposes an activation map generator based on low-level features to explore the underlying information in the low-level features. It mainly consists of an image classifier and an activation map generator with a classification head (referred to as the generator). The generator is directly integrated into the shallow layer of the image classifier, and online activation map generation based on low-level features is performed under the joint supervision of the two classification losses. However, in most cases, the pixel values ​​in the activation map tend to linger near the boundary region, and the final target localization effect may also spread into the background region, resulting in a low localization effect. Summary of the Invention

[0004] The main objective of this invention is to overcome the shortcomings and deficiencies of existing technologies and provide a weakly supervised target localization method and apparatus based on shallow feature background suppression. This method designs a classifier and generator network framework. The classifier obtains shallow feature information to ensure correct identification of the target's category information. The generator uses the shallow features of the classification network to train the target localization by suppressing the background, thus decoupling the classification and localization tasks and avoiding the limitation of classification on the localization effect. At the same time, foreground-background adversarial loss is introduced and Transformer is used to assist the generator learning to obtain more accurate mask pseudo-labels, ensuring that the background is close to the overall background and suppressing background noise, thereby achieving good target localization and classification results.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] On the one hand, this invention provides a weakly supervised target localization method based on shallow feature background suppression, comprising the following steps:

[0007] Obtain the image dataset and divide it into a training dataset and a test dataset;

[0008] A target localization and classification network is constructed based on a neural network; the target localization and classification network includes a classifier, a generator, and an evaluator;

[0009] The training dataset is input into the classifier to generate a shallow feature map;

[0010] The generator is initialized using shallow feature maps to obtain a foreground activation map;

[0011] The foreground activation map is multiplied by the shallow feature map and input into the evaluator for training to obtain the classification result. The foreground cross-entropy loss of the classification result and the original labels of the training dataset is calculated.

[0012] The PCA method is used to perform background principal component analysis on each image in the training dataset. The background activation map is obtained by clustering and input into the evaluator to calculate the background classification loss.

[0013] We introduce foreground-background adversarial loss and use a Transformer-assisted generator to learn the total loss function of the object localization and classification network.

[0014] Input the test dataset into the target localization classification network to obtain the localization results.

[0015] As a preferred technical solution, the generation of the shallow feature map specifically includes:

[0016] For image I in the training dataset, the classifier network structure is F1, and the corresponding network parameters are W1. The features after the first convolutional block are selected as shallow features, resulting in the shallow feature map f:

[0017] f = F1(I, W1)

[0018] Wherein, the size of the shallow feature map f is b×c×h×w, where b is the number of images trained by one GPU at a time, c is the number of channels of the shallow feature map, and h×w is the size of the shallow feature map.

[0019] As a preferred technical solution, the foreground activation map is obtained as follows:

[0020] The shallow feature f is input into the generator G to generate the foreground activation map a:

[0021] a = G(f, Wg)

[0022] The generator network parameter is Wg, and the size of the foreground activation map a is b×1×h×w.

[0023] As a preferred technical solution, the classification result obtained is specifically as follows:

[0024] The evaluator is a pre-trained classifier used to evaluate the effect of the activation map; the evaluator network structure is F2, the feature extraction parameters of the evaluator network are W2, and the classification parameters of the fully connected layer of the evaluator network are W3.

[0025] The foreground activation map and the shallow feature map are multiplied by a dot product, and then input into the evaluator for training to obtain the classification result:

[0026] f2 = F2(a×f, W2)

[0027]

[0028] Where a is the foreground activation map, f is the shallow feature map, and f2 is the feature extracted by the evaluator. The classification results are used by the evaluator; Liner() is a fully connected layer.

[0029] The calculation of the foreground classification loss of the classification result and the original labels of the training dataset is as follows:

[0030] Classification results Calculate the foreground cross-entropy loss using the original labels y of the training dataset:

[0031]

[0032] As a preferred technical solution, obtaining the background activation map specifically involves:

[0033] The PCA method is introduced to perform background principal component analysis on each image in the training dataset, calculating the average feature value x for each image, and obtaining the covariance matrix Cov(x) of this value. Simultaneously, the eigenvectors {ξ1,…,ξd} of the covariance matrix are obtained. The direction with the largest feature, ξ1, is selected to obtain the background region in each feature map.

[0034] b = ξ1(f2-x)

[0035] fb = F2(b×f, W2)

[0036]

[0037] Where b is the background activation map of each image in the training dataset, and fb is the background feature. The classification result is obtained based on background features;

[0038] The background regions in each feature map are clustered to obtain background activation maps, which are then used as new categories.

[0039] Since the foreground and background share the classification weights of the evaluator, the background activation map is input into the evaluator to calculate the background classification loss:

[0040]

[0041] As a preferred technical solution, the introduction of foreground-background adversarial loss specifically includes:

[0042] To ensure the background closely approximates the overall background, a foreground-background adversarial loss is introduced:

[0043] L area =-(logM) f +logM b )

[0044] M f ={a>γ}

[0045] M b ={a<γ}

[0046] γ is a hyperparameter. When the value in the foreground activation map a is greater than γ, it indicates that the classifier has trained on the foreground region. f M is the foreground region in foreground activation map a that is greater than the hyperparameter γ. b It is the background region where the foreground activation map a is less than the hyperparameter γ.

[0047] As a preferred technical solution, the use of Transformer-assisted generator learning specifically includes:

[0048] After the classifier generates shallow feature maps, it obtains the tensors (C, H, W) corresponding to each image in the training dataset.

[0049] By using a 1x1 convolution, the size of the shallow feature map is reduced to obtain the tensor (d, H, W) corresponding to the shallow feature map, where d < 0. <C;

[0050] The tensors corresponding to the shallow feature maps are compressed into (d, HW) to obtain d vector sequences, which are then input into the Transformer.

[0051] The Transformer outputs a vector sequence Q, which is then passed through multiple CNN blocks, alternating between convolutional layers and a 2x upsampling operation, to restore the original resolution of the corresponding images in the training set, resulting in a Mask matrix.

[0052] The segmentation loss is obtained by using a Mask matrix supervised generator for learning.

[0053]

[0054] Where a is the foreground activation map obtained by the generator, and m is the Mask matrix generated by the Transformer calculated using the threshold method. When the value is higher than the threshold, m = 1, otherwise m = 0.

[0055] As a preferred technical solution, the method for obtaining the total loss function of the target localization and classification network is as follows:

[0056] During the classifier training phase, the foreground cross-entropy loss L is utilized. f Train classifier F;

[0057] When initializing the parameters in the generator G, the foreground-background adversarial loss L is utilized. area Constrain the foreground and background ranges, and use the background classification loss L. b Learning background characteristics;

[0058] Using segmentation loss L seg The constraint generator learns more foreground region information and suppresses background noise under the supervision of the Mask pseudo-label, resulting in the total loss function of the target localization and classification network:

[0059] L = L f +L b +L area +L seg .

[0060] On the other hand, the present invention also provides a weakly supervised target localization system based on shallow feature background suppression, which is applied to the aforementioned weakly supervised target localization method based on shallow feature background suppression. The system includes a dataset acquisition module, a model building module, a shallow feature generation module, a foreground acquisition module, a foreground loss calculation module, a background loss calculation module, a loss introduction module, and a result output module.

[0061] The dataset acquisition module is used to acquire the image dataset and divide it into a training dataset and a test dataset;

[0062] The model building module constructs a target localization and classification network based on a neural network; the target localization and classification network includes a classifier, a generator, and an evaluator; the generator includes an encoder-decoder, convolutional blocks, and a normalization layer;

[0063] The shallow feature generation module is used to input the training dataset into the classifier and generate a shallow feature map;

[0064] The foreground acquisition module is used to initialize the generator using a shallow feature map to obtain a foreground activation map;

[0065] The foreground loss calculation module performs a dot product between the foreground activation map and the shallow feature map, inputs it into the evaluator for training, obtains the classification result, and calculates the foreground cross-entropy loss between the classification result and the original labels of the training dataset.

[0066] The background loss calculation module uses the PCA method to perform background principal component analysis on each image in the training dataset, obtains background activation maps through clustering, and inputs them into the evaluator to calculate the background classification loss.

[0067] The loss introduction module obtains the total loss function of the target localization and classification network by introducing foreground-background adversarial loss and learning it using a Transformer-assisted generator.

[0068] The result output module inputs the test dataset into the target localization classification network to obtain the localization result.

[0069] In another aspect, the present invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements the aforementioned weakly supervised target localization method based on shallow feature background suppression.

[0070] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0071] 1. Weakly supervised localization uses an independent generator instead of the traditional CAM map, which largely decouples localization from classification. Compared with traditional weakly supervised target localization based on target classification weights and feature map dot product, this invention better avoids the limitation of clustering in a small part of the target due to classification, and achieves better localization and classification results.

[0072] 2. A background classification constraint is introduced. Principal components of the background are extracted using PCA to obtain a background activation map, which serves as a new category, forcing the network to learn the background information for that category label. Existing technologies divide the background classification prediction result by the foreground classification to suppress background classification information, but this has limitations. Specifically, if the background contains only a very small portion of the object, and this portion has no effect on classification, it will still be defaulted to the background region, unlike the background category added in this invention.

[0073] 3. Introducing foreground-background adversarial loss: During training, background generation is fully considered. The model learns from both the foreground and background simultaneously, accelerating the convergence process and shortening the training time. Existing weakly supervised generator techniques introduce foreground area loss. Since there is a lot of shallow information, the purpose of introducing foreground area loss is to make the foreground region as small as possible so that it can be extracted from the current coarse information. However, excessive introduction of foreground area loss can lead to the detected target region becoming smaller and smaller. Therefore, the foreground-background adversarial loss introduced in this invention makes both the background and foreground as large as possible, ultimately achieving a balance.

[0074] 4. Introducing Transformer to train coarse Masks yields more accurate Mask pseudo-labels, supervising the generator's learning and improving network performance and localization / classification results. Previously, coarse Masks were often directly used as pseudo-labels to supervise encoder-decoder learning, which resulted in network training fluctuations and low segmentation accuracy due to the inaccuracy of the labels. Attached Figure Description

[0075] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0076] Figure 1 These are rough diagrams of the target localization and classification network and the traditional network architecture based on CNN in this embodiment of the invention.

[0077] Figure 2 This is a schematic diagram of the target localization and classification network in an embodiment of the present invention;

[0078] Figure 3 This is a comparison chart of experimental results on the CUB-200-2011 dataset in this embodiment of the invention;

[0079] Figure 4 This is a comparison chart of visualization results on the CUB-200-2011 dataset in an embodiment of the present invention;

[0080] Figure 5 This is a schematic diagram of the structure of a weakly supervised target localization system based on shallow feature background suppression in an embodiment of the present invention;

[0081] Figure 6 This is a schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Detailed Implementation

[0082] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0083] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0084] like Figure 1 As shown, traditional localization and classification networks based on CNNs rely on activation maps obtained from category label information to extract the most discriminative regions. Because the classification network assigns greater weight to these regions and ignores other regions with less discriminative information, the localization performance is limited.

[0085] This embodiment provides a weakly supervised target localization method based on shallow feature background suppression. It decouples classification and localization tasks, uses a classification network structure to obtain shallow feature information, and designs a classifier and a generator. The classifier ensures the correct identification of the target's category information, while the generator uses the shallow information features of the classification network to train the target localization by suppressing the background. This avoids the limitation of classification on localization performance and achieves good localization and classification results.

[0086] like Figure 2As shown in the figure, this embodiment provides a weakly supervised target localization method based on shallow feature background suppression, which includes the following steps:

[0087] S1. Obtain the image dataset and divide it into a training dataset and a test dataset;

[0088] S2. Construct a target localization and classification network based on a neural network, which includes a classifier, a generator, and an evaluator;

[0089] S3. Input the training dataset into the classifier to generate a shallow feature map, specifically:

[0090] For image I in the training dataset, the classifier network structure is F1, and the corresponding network parameters are W1. The features after the first convolutional block are selected as shallow features, resulting in the shallow feature map f:

[0091] f = F1(I, W1)

[0092] Wherein, the size of the shallow feature map f is b×c×h×w, where b is the number of images trained by one GPU at a time, c is the number of channels of the shallow feature map, and h×w is the size of the shallow feature map.

[0093] In this embodiment, the classifier output uses the output features of the first convolutional block in the ResNet50 network structure as shallow features.

[0094] S4. Initialize the generator using shallow feature maps to obtain the foreground activation map, specifically:

[0095] To better initialize the generator and ensure its parameters are closely aligned with the classification information, shallow features f are input into the generator G to generate a foreground activation map a:

[0096] a = G(f, Wg)

[0097] The generator network parameter is Wg, and the size of the foreground activation map a is b×1×h×w.

[0098] In this embodiment, the generator includes a codec, a convolutional block, and a normalization layer.

[0099] S5. Perform a dot product between the foreground activation map and the shallow feature map, input the result into the evaluator for training, obtain the classification result, and calculate the foreground cross-entropy loss between the classification result and the original labels of the training dataset. Specifically:

[0100] The evaluator is a pre-trained classifier used to evaluate the performance of the activation map; the network structure of the evaluator is F2, the corresponding feature extractor parameters are W2, and the corresponding fully connected layer classification parameters are W3.

[0101] The foreground activation map and the shallow feature map are multiplied by a dot product, and then input into the evaluator for training to obtain the classification result:

[0102] f2 = F2(a×f, W2)

[0103]

[0104] Where a is the foreground activation map, f is the shallow feature map, and f2 is the feature extracted by the evaluator. The classification result is given by the evaluator, and Liner() is a fully connected layer.

[0105] The foreground activation map 'a' is multiplied by the shallow feature map, which highlights the target's location information while suppressing background information. This is because the foreground activation map makes the target region's information closer to 1, while making the background region's value closer to 0, thus being more beneficial for classification. In this embodiment, the evaluator can be a network model such as VGG16 or Inception-V3.

[0106] Next, the cross-entropy loss function is used to analyze the classification results. Calculate the foreground cross-entropy loss using the original labels y of the training dataset:

[0107]

[0108] Classifiers and generators can learn under the guidance of category information. The classifier learns the salient information of the target in the image, and the generator activates the salient region. At the same time, by using the underlying network features, that is, feature maps containing more global information, the network can learn more holistic regional information of the target.

[0109] S6. Use PCA to perform background principal component analysis on each image in the training dataset, obtain background activation maps through clustering, and input them into the evaluator to calculate the background classification loss. Specifically:

[0110] While the generator learns the foreground region, it also learns information about the background region. Therefore, the PCA method is introduced to perform principal component analysis on each image in the training dataset, calculating the average value of each image feature as x, and obtaining the covariance matrix Cov(x) of this value. Simultaneously, the eigenvectors {ξ1,…,ξd} of the covariance matrix are obtained. The feature direction ξ1 with the largest value is selected, which represents the most similar background information feature within the same feature set. The background region in each feature map is thus obtained as follows:

[0111] b = ξ1(f2-x)

[0112] fb = F2(b×f, W2)

[0113]

[0114] Where b is the background activation map of each image in the training dataset, and fb is the background feature. The classification result is obtained based on background features;

[0115] At the same time, the background regions in each feature map are clustered to obtain background activation maps, which are used as new categories to form background classes. For example, if there are 200 categories in the original dataset, the background classes will be added to form a total of 201 categories.

[0116] Since the foreground and background share the evaluator's classification weights, the network is forced to learn the background class using background information. Therefore, the background activation map is input into the evaluator to calculate the background classification loss.

[0117]

[0118] S7. Introduce foreground-background adversarial loss and use a Transformer-assisted generator to learn the total loss function of the object localization and classification network, specifically:

[0119] During the classifier's activation of the target region, in order to ensure that the background is close to the overall background, that is, to maximize the background area of ​​both sides, a foreground-background adversarial loss is introduced:

[0120] L area =-(logM) f +logM b )

[0121] M f ={a>γ}

[0122] M b ={a<γ}

[0123] γ is a hyperparameter. When the value in the foreground activation map a is greater than γ, it indicates that the classifier has trained on the foreground region. f M is the foreground region in foreground activation map a that is greater than the hyperparameter γ. b The foreground activation map a is the background region where the hyperparameter γ is smaller than the foreground activation map a; by forcing the loss function to be as small as possible, the area of ​​the two regions is made as large as possible.

[0124] Furthermore, since the CAM features generated by the evaluator are often coarse and overly concentrated in small target regions, and may also contain background noise, inaccurate mask pseudo-labels are obtained. Therefore, this paper proposes to optimize the CAM image using a Transformer to obtain a more accurate mask image, which is then used as a supervision signal for the generator to optimize its learning. Specifically:

[0125] After the classifier generates shallow feature maps, it obtains the tensors (C, H, W) corresponding to each image in the training dataset.

[0126] By using a 1x1 convolution, the size of the shallow feature map is reduced to obtain the tensor (d, H, W) corresponding to the shallow feature map, where d < 0. <C;

[0127] The tensor corresponding to the shallow feature map is squeezed into (d, HW), and d vector sequences are input into the Transformer.

[0128] The Transformer outputs a vector sequence Q, which is then passed through multiple CNN blocks, alternating between convolutional layers and a 2x upsampling operation, to restore the original resolution of the corresponding images in the training set, resulting in a Mask matrix.

[0129] The segmentation loss is obtained by using a Mask matrix supervised generator for learning.

[0130]

[0131] Where a is the foreground activation map obtained by the generator, and m is the Mask matrix generated by the Transformer calculated using the threshold method. When the value is higher than the threshold, m = 1, otherwise m = 0.

[0132] Network training is primarily driven by four types of loss functions. During the classifier training phase, the foreground cross-entropy loss L is utilized. f Train the classifier F, while initializing the parameters in the generator G, and utilize the foreground-background adversarial loss L. area Constrain the foreground and background ranges, and use the background classification loss L. b Learning background features; using segmentation loss L seg The constraint generator learns more foreground region information and suppresses background noise under the supervision of the Mask pseudo-label. Therefore, the total loss function of the object localization and classification network is expressed as:

[0133] L = L f +L b +L area +L seg .

[0134] S8. Input the test dataset into the target localization classification network to obtain the localization results.

[0135] During the testing phase, an input image is processed by a classifier network to obtain a shallow feature map, which is then fed into a generator to generate an activation map. Regions with values ​​greater than a certain threshold (e.g., 0.5) are defined as target regions, thus obtaining the target's coordinates. The activation map is multiplied by the original image and input into an evaluator to obtain the target's classification result. Specific experimental results are shown below. Figure 3 As shown in Figure 4.

[0136] This test was conducted on the CUB-200-2011 dataset, training both the VGG16 network and Inception-V3. Figure 3 As can be seen from the data, the method proposed in this embodiment achieved an accuracy of 65.20% on the Top-1 Loc metric.

[0137] Depend on Figure 4 The visualization results show that traditional CAM often focuses on the most prominent feature regions of an object, while the improved EIL method is insensitive to object boundary information and easily generalizes to the background region, failing to capture the overall target region effectively. This invention proposes a weakly supervised target localization method based on shallow feature background suppression. Experimental results demonstrate that by suppressing background information while preserving foreground information, and utilizing the learning of an encoder-decoder generator, the object's boundaries can be captured, thus better detecting the overall target region and achieving excellent localization results.

[0138] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0139] Based on the same idea as the weakly supervised target localization method based on shallow feature background suppression in the above embodiments, the present invention also provides a weakly supervised target localization system based on shallow feature background suppression. This system can be used to execute the aforementioned weakly supervised target localization method based on shallow feature background suppression. For ease of explanation, the structural diagram of the embodiment of the weakly supervised target localization system based on shallow feature background suppression only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0140] like Figure 5 As shown, another embodiment of the present invention provides a weakly supervised target localization system based on shallow feature background suppression, applied to the aforementioned weakly supervised target localization method based on shallow feature background suppression, and includes the following modules:

[0141] The dataset acquisition module is used to acquire image datasets and divide them into training datasets and test datasets;

[0142] The model building module constructs a target localization and classification network based on a neural network, which includes a classifier, a generator, and an evaluator.

[0143] The shallow feature generation module is used to input the training dataset into the classifier and generate shallow feature maps;

[0144] The foreground acquisition module is used to initialize the generator with shallow feature maps to obtain the foreground activation map;

[0145] The foreground loss calculation module performs a dot product between the foreground activation map and the shallow feature map, inputs it into the evaluator for training, obtains the classification result, and calculates the foreground cross-entropy loss between the classification result and the original labels of the training dataset.

[0146] The background loss calculation module uses the PCA method to perform background principal component analysis on each image in the training dataset, obtains background activation maps through clustering, and inputs them into the evaluator to calculate the background classification loss.

[0147] The loss introduction module obtains the total loss function of the object localization and classification network by introducing foreground-background adversarial loss and learning it using a Transformer-assisted generator.

[0148] The results output module inputs the test dataset into the target localization classification network to obtain the localization results.

[0149] It should be noted that the weakly supervised target localization system based on shallow feature background suppression of the present invention corresponds one-to-one with the weakly supervised target localization method based on shallow feature background suppression of the present invention. The technical features and beneficial effects described in the embodiments of the weakly supervised target localization method based on shallow feature background suppression described above are applicable to the embodiments of the weakly supervised target localization system based on shallow feature background suppression. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0150] Furthermore, in the above embodiments of the weakly supervised target localization system based on shallow feature background suppression, the logical division of each program module is only an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or software implementation convenience. That is, the internal structure of the weakly supervised target localization system based on shallow feature background suppression is divided into different program modules to complete all or part of the functions described above.

[0151] like Figure 6 As shown, in one embodiment, a computer-readable storage medium is provided, storing a program in a memory. When the program is executed by a processor, it implements the weakly supervised target localization method based on shallow feature background suppression, specifically:

[0152] Obtain the image dataset and divide it into a training dataset and a test dataset;

[0153] A target localization and classification network is constructed based on a neural network; the target localization and classification network includes a classifier, a generator, and an evaluator;

[0154] The training dataset is input into the classifier to generate a shallow feature map;

[0155] The generator is initialized using shallow feature maps to obtain a foreground activation map;

[0156] The foreground activation map is multiplied by the shallow feature map and input into the evaluator for training to obtain the classification result. The foreground cross-entropy loss of the classification result and the original labels of the training dataset is calculated.

[0157] The PCA method is used to perform background principal component analysis on each image in the training dataset. The background activation map is obtained by clustering and input into the evaluator to calculate the background classification loss.

[0158] We introduce foreground-background adversarial loss and use a Transformer-assisted generator to learn the total loss function of the object localization and classification network.

[0159] Input the test dataset into the target localization classification network to obtain the localization results.

[0160] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0161] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A weakly supervised target localization method based on shallow feature background suppression, characterized in that, Includes the following steps: Obtain the image dataset and divide it into a training dataset and a test dataset; Construct a target localization and classification network based on neural networks; The target localization and classification network includes a classifier, a generator, and an evaluator; The training dataset is input into the classifier to generate a shallow feature map; The generator is initialized using shallow feature maps to obtain a foreground activation map; The foreground activation map is multiplied by the shallow feature map and input into the evaluator for training to obtain the classification result. The foreground cross-entropy loss of the classification result and the original labels of the training dataset is calculated. The PCA method is used to perform background principal component analysis on each image in the training dataset. The background activation map is obtained by clustering and input into the evaluator to calculate the background classification loss. The obtained background activation map is specifically as follows: The PCA method is introduced to perform background principal component analysis on each image in the training dataset, and the average value of each image feature is calculated. x Calculate the covariance matrix of this value. Cov(x) Simultaneously, the eigenvectors {k1,…,kd} of the covariance matrix are obtained. The largest eigendirection k1 is selected, and the background region in each feature map is obtained as follows: b = 𝝃1(𝑓2−x) ， 𝑓b = 𝐹2 (b×𝑓, 𝑊2) ， 𝑦̂b = Liner (fb, 𝑊3) ， in, b To train the background activation map for each image in the dataset, f This is a shallow feature map. f2 Features extracted for the evaluator W2 For the feature extraction parameters of the evaluator network, W3 The classification parameters are the fully connected layers of the evaluator network. Liner () represents a fully connected layer. 𝑓b As background features, 𝑦̂b The classification result is obtained based on background features; The background regions in each feature map are clustered to obtain background activation maps, which are then used as new categories. Since the foreground and background share the classification weights of the evaluator, the background activation map is input into the evaluator to calculate the background classification loss: 𝐿 b = −l𝑜𝑔(𝑦[𝑦̂ b ]) ; A foreground-background adversarial loss is introduced and learned using a Transformer-assisted generator to obtain the total loss function of the object localization and classification network; the foreground-background adversarial loss is: 𝐿 𝑎𝑟𝑒𝑎 = −(log𝑀 𝑓 + log𝑀 𝑏 ), 𝑀 𝑓 = { a > 𝛾}, 𝑀 𝑏 = { a < 𝛾}, 𝛾 is a hyperparameter, which is the foreground activation map. a When the value in is greater than 𝛾, it indicates that the classifier has been trained to identify the foreground region, 𝑀 𝑓 It is a foreground activation map a The foreground region is larger than the hyperparameter , 𝑏 It is a foreground activation map a Background region smaller than hyperparameter 𝛾; Input the test dataset into the target localization classification network to obtain the localization results.

2. The weakly supervised target localization method based on shallow feature background suppression according to claim 1, characterized in that, The generation of the shallow feature map specifically involves: For the images in the training dataset I The classifier network structure is as follows: F1 The corresponding network parameters are W1 The features after the first convolutional block are selected as shallow features to obtain the shallow feature map. f : f = F1(I,W1) ， Among them, shallow feature map f The size is b×c×h×w,b The number of images trained on a single GPU at one time. c The number of channels representing shallow features. h×w It is the size of the shallow feature map.

3. The weakly supervised target localization method based on shallow feature background suppression according to claim 2, characterized in that, The obtained foreground activation map is specifically as follows: shallow features f Input generator G In the process, a foreground activation map is generated. a : a = G(f,Wg) ， The generator network parameters are as follows: Wg Foreground activation map a The size is b×1×h×w .

4. The weakly supervised target localization method based on shallow feature background suppression according to claim 3, characterized in that, The specific classification result obtained is as follows: The evaluator is a pre-trained classifier used to evaluate the effectiveness of the activation map; the evaluator network structure is as follows: F2 The feature extraction parameters of the evaluator network are: W2 The classification parameters of the fully connected layer of the evaluator network are: W3 ; The foreground activation map and the shallow feature map are multiplied by a dot product, and then input into the evaluator for training to obtain the classification result: f2 = F2(a × f,W2) ， 𝑦̂ = Liner(f2,W3) , in, a Foreground activation map, f This is a shallow feature map. f2 Here, represents the features extracted by the evaluator, and 𝑦̂ represents the classification result of the evaluator. Liner () represents a fully connected layer; The calculation of the foreground classification loss of the classification result and the original labels of the training dataset is as follows: The classification result 𝑦̂ and the original labels of the training dataset y Calculate the foreground cross-entropy loss: 𝐿 f = −l𝑜𝑔(𝑦[𝑦̂]) 。 5. The weakly supervised target localization method based on shallow feature background suppression according to claim 1, characterized in that, The use of Transformer-assisted generator learning specifically refers to: After the classifier generates shallow feature maps, it obtains the tensors corresponding to each image in the training dataset. (C,H,W) ; By using a 1x1 convolution, the size of the shallow feature map is reduced to obtain the tensor corresponding to the shallow feature map. (d,H,W), in d< <C ; Compress the tensor corresponding to the shallow feature map to (d,HW) ,get d A sequence of vectors is input into the Transformer; Transformer output vector sequence Q And by using multiple CNN blocks, alternating between convolutional layers and double upsampling operations, the original resolution of the corresponding images in the training set is restored, resulting in a Mask matrix; The segmentation loss is obtained by using a Mask matrix supervised generator for learning. ， in, a It is the foreground activation map obtained by the generator. m It is calculated using a threshold method on the Mask matrix generated by the Transformer. When it exceeds the threshold, m=1 ,otherwise m=0 .

6. The weakly supervised target localization method based on shallow feature background suppression according to claim 5, characterized in that, The total loss function of the target localization and classification network is specifically as follows: During the classifier training phase, the foreground cross-entropy loss λ is utilized. f Training classifier F ; Initialize generator G When considering the parameters, utilize the foreground-background adversarial loss F. 𝑎𝑟𝑒𝑎 Constrain the foreground and background ranges, and use background classification loss. 𝐿 b Learning background characteristics; Using segmentation loss 𝐿 seg The constraint generator learns more foreground region information and suppresses background noise under the supervision of the Mask pseudo-label, resulting in the total loss function of the target localization and classification network: 𝐿 = 𝐿 f + 𝐿 b + 𝐿 𝑎𝑟𝑒𝑎 + 𝐿 seg 。 7. A weakly supervised target localization system based on shallow feature background suppression, characterized in that, The weakly supervised target localization method based on shallow feature background suppression, applied to any one of claims 1-6, includes a dataset acquisition module, a model building module, a shallow feature generation module, a foreground acquisition module, a foreground loss calculation module, a background loss calculation module, a loss introduction module, and a result output module. The dataset acquisition module is used to acquire the image dataset and divide it into a training dataset and a test dataset; The model building module constructs a target localization and classification network based on a neural network; the target localization and classification network includes a classifier, a generator, and an evaluator; the generator includes an encoder-decoder, convolutional blocks, and a normalization layer; The shallow feature generation module is used to input the training dataset into the classifier and generate a shallow feature map; The foreground acquisition module is used to initialize the generator using a shallow feature map to obtain a foreground activation map; The foreground loss calculation module performs a dot product between the foreground activation map and the shallow feature map, inputs it into the evaluator for training, obtains the classification result, and calculates the foreground cross-entropy loss between the classification result and the original labels of the training dataset. The background loss calculation module uses the PCA method to perform background principal component analysis on each image in the training dataset, obtains background activation maps through clustering, and inputs them into the evaluator to calculate the background classification loss. The loss introduction module obtains the total loss function of the target localization and classification network by introducing foreground-background adversarial loss and learning it using a Transformer-assisted generator. The result output module inputs the test dataset into the target localization classification network to obtain the localization result.

8. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the weakly supervised target localization method based on shallow feature background suppression as described in any one of claims 1-6.

Images