Target positioning method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a target positioning method and device, electronic equipment and a storage medium. The method comprises the following steps: generating a first classification network for identifying category information of a target in an image; generating a positioning map for expressing initial positioning information of the target through the first classification network; determining a positioning map used for expressing initial positioning information of the target as a first pseudo label, and generating a first segmentation network used for segmenting a target area and a background area of the target through the first pseudo label; converting the first segmentation network into a second classification network; and generating a positioning map for expressing the updated positioning information of the target through the second classification network, and remarkably improving the accuracy of target positioning through an iterative enhanced learning process under the condition of not increasing extra marking cost.

Description

Technical Field

[0001] The present invention relates to the field of target positioning technology, and in particular to a target positioning method, a target positioning device, an electronic device and a computer-readable storage medium. Background Art

[0002] Target positioning is the core task of computer vision. Deep learning has greatly promoted the development of target positioning, but it relies on a large amount of accurately labeled data, which not only consumes a lot of time and human resources, but also limits the application of these algorithms on unlabeled data.

[0003] Therefore, how to achieve target positioning in images without relying on a large amount of accurately labeled data is a research hotspot in the field of computer vision. Summary of the invention

[0004] The embodiments of the present invention provide a target positioning method, device, electronic device and computer-readable storage medium to overcome the above problems or at least partially solve the above problems.

[0005] The embodiment of the present invention discloses a target positioning method, including:

[0006] A first classification network is generated for identifying category information of an object in an image;

[0007] Generating, by the first classification network, a positioning map for expressing initial positioning information of the target;

[0008] Determine a positioning map for expressing initial positioning information of the target as a first pseudo label, and generate a first segmentation network for segmenting a target area and a background area of ​​the target through the first pseudo label;

[0009] Converting the first segmentation network into a second classification network;

[0010] A positioning map for expressing updated positioning information of the target is generated by the second classification network.

[0011] Optionally, it also includes:

[0012] A positioning map for expressing the updated positioning information of the target is determined as a second pseudo label, and the second pseudo label is used for a second segmentation network for segmenting a target area and a background area of ​​the target.

[0013] Optionally, the step of generating a first classification network for identifying category information of an object in an image comprises:

[0014] Obtaining a training set and performing a preprocessing operation on the training set;

[0015] determining a first loss function for the first classification network;

[0016] Based on the first loss function, a first classification network is constructed using the preprocessed training set.

[0017] Optionally, the first classification network includes a plurality of convolutional layers, the plurality of convolutional layers are arranged in sequence, and the step of generating a positioning map for expressing initial positioning information of the target through the first classification network includes:

[0018] Get the output result of the last convolutional layer;

[0019] determining a category weight for the output result;

[0020] The output results are weighted and summed based on the category weights to generate a positioning map for expressing initial positioning information of the target.

[0021] Optionally, the step of generating a first segmentation network for segmenting a target area and a background area of ​​the target by using the first pseudo label includes:

[0022] Calculate the cross entropy between the segmentation map and the first pseudo label;

[0023] Determining a second loss function of the first segmentation network based on the cross entropy;

[0024] A first segmentation network for segmenting a target area and a background area of ​​the target is generated based on the second loss function.

[0025] Optionally, the step of converting the first segmentation network into a second classification network includes:

[0026] Determining a segmentation head of the first segmentation network;

[0027] Determine the classification global pooling layer and classification fully connected layer for classification tasks;

[0028] Remove the segmentation head, and add the classification global pooling layer and the classification fully connected layer to generate an initial classification network;

[0029] The weights of the encoder part of the initial segmentation network are fixed to generate a second classification network.

[0030] Optionally, the global pooling layer is added as follows:

[0031] The average value and the maximum value of each channel feature map of the global pooling layer are calculated, and the global pooling layer is added based on the average value and the maximum value.

[0032] The embodiment of the present invention further discloses a target positioning device, comprising:

[0033] A first classification network generation module, used to generate a first classification network for identifying category information of an object in an image;

[0034] A first positioning map generating module, used to generate a positioning map for expressing initial positioning information of the target through the first classification network;

[0035] A first segmentation network generation module, used to determine a positioning map used to express initial positioning information of the target as a first pseudo label, and generate a first segmentation network used to segment a target area and a background area of ​​the target through the first pseudo label;

[0036] A network conversion module, used to convert the first segmentation network into a second classification network;

[0037] The second positioning map generating module is used to generate a positioning map for expressing the updated positioning information of the target through the second classification network.

[0038] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0039] The memory is used to store computer programs;

[0040] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0041] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0042] The embodiments of the present invention include the following advantages:

[0043] In an embodiment of the present invention, a first classification network is generated for identifying category information of a target in an image; a positioning map for expressing initial positioning information of the target is generated through the first classification network; the positioning map for expressing the initial positioning information of the target is determined as a first pseudo label, and a first segmentation network for segmenting a target area and a background area of ​​the target is generated through the first pseudo label; the first segmentation network is converted into a second classification network; and a positioning map for expressing updated positioning information of the target is generated through the second classification network, thereby achieving the following beneficial effects:

[0044] Classification-guided positioning: Generate initial positioning information through the classification network.

[0045] Segmentation refinement positioning: Use the segmentation network to generate pixel-level segmentation results to improve positioning accuracy.

[0046] Iterative optimization: By converting the segmentation network into a classification network, iterative optimization is continuously performed to improve positioning accuracy.

[0047] This iterative optimization approach can effectively utilize the complementarity of classification and segmentation tasks and gradually improve the accuracy of target positioning.

[0048] The accuracy of target positioning can be significantly improved through an iterative enhanced learning process without adding additional annotation costs. By utilizing the complementarity between the classification network and the segmentation network, the present invention realizes deep mining and optimization of image features, so that the final target positioning is not only accurate, but also has good robustness for targets in various complex backgrounds. In addition, by applying the model conversion after the segmentation network optimization to the classification task, this technology can effectively utilize the sensitivity of the segmentation network to details, and further enhance the accuracy and reliability of the class activation map generated by the classification network. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flowchart of a target positioning method provided in an embodiment of the present invention;

[0050] Figure 2 is a flow chart of another target positioning method provided in an embodiment of the present invention;

[0051] Figure 3 is a flow chart of another target positioning method provided in an embodiment of the present invention;

[0052] Figure 4 is a flowchart of another target positioning method provided in an embodiment of the present invention;

[0053] Figure 5 is a structural block diagram of a target positioning device provided in an embodiment of the present invention;

[0054] Figure 6 is a hardware structure block diagram of an electronic device provided in an embodiment of the present invention;

[0055] Figure 7 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] CAM (Class Activation Mapping) technology. It was first proposed in the paper "Learning Deep Features for Discriminative Localization" and is indeed an important technology in the field of weakly supervised localization.

[0058] How CAM works:

[0059] The core idea of ​​CAM is to generate a heat map of the same size as the input image by weighted summing the outputs of the last convolutional layer of the neural network. This heat map can intuitively show which areas of the image the model focuses on when classifying.

[0060] Specifically, the implementation steps of CAM are as follows:

[0061] Get the output of the last convolutional layer: For each category, each channel of the last convolutional layer produces a feature map.

[0062] Calculate the weight of the category: Use the weights of the fully connected layer to calculate the weight of each category. These weights reflect the importance of different feature maps for the category.

[0063] Weighted summation: Multiply each feature map by the corresponding weight, and then perform weighted summation on all feature maps to obtain an activation map for a category.

[0064] Visualization: By overlaying the resulting activation map onto the original image, we can see the areas that the model focuses on.

[0065] Advantages of CAM:

[0066] Intuitiveness: CAM can intuitively display the model's focus areas, helping us understand the model's decision-making process.

[0067] Universality: CAM can be applied to various CNN models without the need for additional training of the models.

[0068] Effectiveness: CAM has shown good results in many tasks, such as object localization, image interpretation, etc.

[0069] Limitations of CAM

[0070] Reliance on Global Average Pooling: The original design of CAM relies on a global average pooling layer, which limits its application scope.

[0071] Limited utilization of shallow features: CAM mainly focuses on the output of the last convolutional layer and does not make full use of shallow features.

[0072] The localization effect for small targets may not be ideal: For small targets, the heat map generated by CAM may not be detailed enough.

[0073] CAM is a very useful technique that can help us better understand the decision-making process of deep neural networks. Although CAM has some limitations, its performance will be further improved as research continues.

[0074] In practical applications, object localization is a core task of computer vision. Deep learning has greatly promoted the development of object localization, but it relies on a large amount of accurately labeled data, which limits its application scope. Weakly supervised object localization (WSOL) provides a more cost-effective solution by only using image-level labels to locate objects.

[0075] The main challenges faced by WSOL and existing solutions:

[0076] Incomplete localization: Classification models often over-focus on salient areas of the image, resulting in incomplete localization. Existing methods such as data augmentation and erasing techniques attempt to address this problem.

[0077] Target and background confusion: Especially for similar categories, the classifier can easily mistake the background for the target.

[0078] Limitations of the two-stage approach: Although the two-stage approach improves localization accuracy, category-independent labels may introduce additional errors.

[0079] The key technical issues to be solved are how to train the classification model so that it can cover the target area more comprehensively when generating class activation maps; how to effectively distinguish between the target and the background, especially when the target and the background have similar features; and how to improve the generalization ability of the positioning network under weak supervision conditions to avoid misidentifying non-target salient objects as targets.

[0080] WSOL has broad application prospects in the field of target positioning, but there are still some key technical problems to be solved. To address these problems, it is necessary to further explore more effective model design, training strategies and evaluation indicators. "

[0081] The embodiment of the present invention proposes a weakly supervised target localization method based on improved class activation mapping and image reconstruction by combining multi-task learning, feature enhancement and iterative optimization strategies. This technology aims to achieve a target localization system that can work robustly in various complex backgrounds by integrating classification, localization and image reconstruction tasks.

[0082] By introducing the image reconstruction task as an auxiliary, not only the network's ability to represent the target is enhanced, but also the model's ability to capture fine-grained features is significantly improved through the iterative feedback mechanism, effectively solving the problem that traditional methods are easily confused when dealing with visually similar objects.

[0083] In addition, the embodiment of the present invention also adopts an innovative iterative enhancement strategy, which further improves the positioning accuracy by continuously optimizing the positioning map, thereby significantly improving the accuracy and robustness of weakly supervised target positioning while ensuring the classification performance.

[0084] Based on the weakly supervised target localization task process, a classification network is first trained to generate pseudo labels. When training the classification network, an image classification network is trained in parallel on the basis of the same backbone network, so that the network can better capture the visual features of the entire image, rather than some salient features that only serve classification. Specifically, when training the classification network, a certain amount of noise and occlusion are randomly added to the image to train an image reconstruction network. The mean square error (MSE) loss function is used during training to encourage the image reconstructed by the network to be as similar as possible to the original image at the pixel level. In this way, the image reconstruction task serves as an auxiliary to help the main network understand the image content more comprehensively, so that it has richer background information and detailed features when performing classification and localization tasks.

[0085] Through the above method, the embodiment of the present invention proposes an iterative enhancement method combining image reconstruction and multi-task learning to address the common challenges in weakly supervised object localization (WSOL) tasks, which effectively improves the model's ability to recognize and locate targets in complex scenes.

[0086] Reference Figure 1 , shows a step flow chart of a target positioning method provided in an embodiment of the present invention, which may specifically include the following steps:

[0087] Step 101, generating a first classification network for identifying category information of an object in an image;

[0088] Step 102, generating a positioning map for expressing initial positioning information of the target through the first classification network;

[0089] Step 103, determining a positioning map used to express initial positioning information of the target as a first pseudo label, and generating a first segmentation network used to segment a target area and a background area of ​​the target through the first pseudo label;

[0090] Step 104, converting the first segmentation network into a second classification network;

[0091] Step 105: Generate a positioning graph for expressing updated positioning information of the target through the second classification network.

[0092] An embodiment of the present invention can generate a first classification network for identifying category information of targets in an image, with the aim of training a basic classification model for identifying target categories of targets in images, providing an initial classification model for subsequent positioning tasks, and laying the foundation for extracting semantic information of the target.

[0093] In practical applications, Class Activation Mapping (CAM) is used to visualize the area that a neural network focuses on when classifying an image. Simply put, it generates a heat map through calculation. The hotter the area on the heat map, the more the network pays attention to this area, and the greater the contribution of this area to the classification result.

[0094] Specific steps are broken down:

[0095] 1. Global Average Pooling (GAP):

[0096] After the last convolutional layer of the first classification network, there is usually a global average pooling layer. The function of this layer is to average the responses of each feature channel to obtain a feature vector of fixed size.

[0097] The benefits of doing this are:

[0098] Reduced the number of parameters.

[0099] Enhanced the generalization ability of the model.

[0100] It provides the global importance information of each channel feature.

[0101] 2. Fully connected layer weights:

[0102] The fully connected layer connects the output of the global average pooling layer and the final classification result.

[0103] The weight of the fully connected layer can be regarded as the contribution of each feature channel to the final classification result.

[0104] 3. Generate CAM:

[0105] Multiply the weights of the fully connected layer by the feature map after global average pooling: the output of each feature channel is multiplied by the corresponding weight to obtain the weighted feature map.

[0106] Add up all weighted feature maps: we get a heat map of the same size as the input image, i.e. CAM.

[0107] The value of the heat map: indicates the contribution of the position to the final classification result. The larger the value, the more the network pays attention to this area.

[0108] The embodiment of the present invention may also generate a positioning map for expressing the initial positioning information of the target through the first classification network, with the aim of generating a rough target positioning map, and adjusting the weight of the classification network based on the positioning map.

[0109] For the convenience of distinction, the positioning map used to express the initial positioning information of the target is called the initial positioning map.

[0110] In a specific implementation, the embodiment of the present invention can use methods such as class activation mapping (CAM) to generate a localization map by weighted summing the outputs of the last convolutional layer of the classification network. The localization map indicates where the target may appear in the image. Based on the localization map, the weights of the classification network can be fine-tuned to enhance the network's attention to the target area, thereby providing an initial positioning reference for subsequent segmentation tasks. At the same time, the sensitivity of the classification network to the target area is improved through weight adjustment.

[0111] The embodiment of the present invention can also determine the positioning map used to express the initial positioning information of the target as a first pseudo label, and generate a first segmentation network for segmenting the target area and the background area of ​​the target through the first pseudo label, with the purpose of training a segmentation model to segment the image into a target area and a background area.

[0112] In a specific implementation, the positioning map generated in step 102 can be used as the first pseudo label, and the training of the segmentation network can be supervised based on the first pseudo label. The output of the segmentation network is a pixel-level segmentation map, where each pixel represents a target or background, thereby generating a more accurate target positioning result and providing more accurate supervision information for subsequent iterations.

[0113] In the embodiment of the present invention, the first segmentation network may also be converted into a second classification network, with the purpose of migrating the feature extraction capability of the segmentation network to the classification network to enhance the positioning capability of the classification network.

[0114] Exemplarily, the feature extraction part of the segmentation network can be retained, the classification head can be replaced, and the weights of the segmentation network can be loaded as initialization, thereby enhancing the feature extraction capability of the classification network, so that the classification network can better capture the fine-grained features of the target.

[0115] In the embodiment of the present invention, a positioning map for expressing updated positioning information of the target is generated by the second classification network, with the aim of generating a more accurate positioning map based on an improved classification network.

[0116] Exemplarily, a method similar to step 102 may be used to generate a new positioning map, and this new positioning map may be a positioning map for expressing updated positioning information of the target, thereby generating more accurate target positioning information and providing more accurate supervision information for subsequent iterations.

[0117] For the sake of distinction, the positioning graph used to express the updated positioning information of the target is referred to as the target positioning graph below.

[0118] In an embodiment of the present invention, a first classification network is generated for identifying category information of a target in an image; a positioning map for expressing initial positioning information of the target is generated through the first classification network; the positioning map for expressing the initial positioning information of the target is determined as a first pseudo label, and a first segmentation network for segmenting a target area and a background area of ​​the target is generated through the first pseudo label; the first segmentation network is converted into a second classification network; and a positioning map for expressing updated positioning information of the target is generated through the second classification network, thereby achieving the following beneficial effects:

[0119] Classification-guided positioning: Generate initial positioning information through the classification network.

[0120] Segmentation refinement positioning: Use the segmentation network to generate pixel-level segmentation results to improve positioning accuracy.

[0121] Iterative optimization: By converting the segmentation network into a classification network, iterative optimization is continuously performed to improve positioning accuracy.

[0122] This iterative optimization approach can effectively utilize the complementarity of classification and segmentation tasks and gradually improve the accuracy of target positioning.

[0123] The accuracy of target positioning can be significantly improved through an iterative enhanced learning process without adding additional annotation costs. By utilizing the complementarity between the classification network and the segmentation network, the present invention realizes deep mining and optimization of image features, so that the final target positioning is not only accurate, but also has good robustness for targets in various complex backgrounds. In addition, by applying the model conversion after the segmentation network optimization to the classification task, this technology can effectively utilize the sensitivity of the segmentation network to details, and further enhance the accuracy and reliability of the class activation map generated by the classification network.

[0124] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.

[0125] Optionally, it also includes:

[0126] A positioning map for expressing the updated positioning information of the target is determined as a second pseudo label, and the second pseudo label is used for a second segmentation network for segmenting a target area and a background area of ​​the target.

[0127] In practical applications, through continuous iteration, the segmentation results of the previous round are used to guide the next round of segmentation, thereby gradually improving the segmentation accuracy and thus improving the accuracy of target positioning.

[0128] Step Breakdown:

[0129] Generate a more precise positioning map:

[0130] The generation process of the second classification network is to adjust the weight of the classification network based on the initial positioning map, so the second classification network has a stronger ability to classify the target, so the accuracy of the target positioning map output by the second classification network is higher than that of the previous initial positioning map. Therefore, if the target positioning map is used as a pseudo-label, this heat map is more accurate than the CAM (initial positioning map) generated in step 102 because it incorporates the fine-grained information of the segmentation network.

[0131] The second pseudo label can be to use the target positioning map as a pseudo label and the generated target positioning map as new training data, wherein the area with higher heat is regarded as the target area and the area with lower heat is regarded as the background area.

[0132] The purpose of training the second segmentation network is to use the more accurate second pseudo-label to train a new segmentation network that can better distinguish the target area from the background area and generate more accurate segmentation results. After generating the new second segmentation network, it can be iteratively optimized and the above process can be repeated to convert the newly generated segmentation network into a new classification network. Through continuous iteration, the segmentation accuracy will continue to improve, the classification results will become more and more accurate, and the positioning will become more and more precise.

[0133] The purpose of the second segmentation network is to:

[0134] Refine the segmentation results: Perform more fine-grained optimization to address the deficiencies in the previous round of segmentation results.

[0135] Improve positioning accuracy: Through more precise segmentation, a more accurate target positioning frame can be obtained.

[0136] Provide more reliable input for subsequent tasks: More accurate segmentation results can provide more reliable input for subsequent tasks such as target tracking and image segmentation.

[0137] In the embodiment of the present invention, the positioning map used to express the updated positioning information of the target is determined as the second pseudo label, and the second pseudo label is used for the second segmentation network to segment the target area and the background area of ​​the target, thereby achieving the following beneficial effects:

[0138] Self-reinforcement learning: The model can learn from its own output and continuously improve its performance.

[0139] Improve segmentation accuracy: Through iterative optimization, the segmentation results become more and more accurate.

[0140] Improved localization accuracy: More accurate segmentation directly leads to more accurate localization.

[0141] Enhance model robustness: The model is more adaptable to different images.

[0142] By converting the segmentation task into a self-enhanced learning process, the performance of the model is continuously improved. By using the output of the previous round as the input of the next round, the model can gradually learn more complex features, thereby improving the accuracy of segmentation and positioning.

[0143] Optionally, the step of generating a first classification network for identifying category information of an object in an image comprises:

[0144] Obtaining a training set and performing a preprocessing operation on the training set;

[0145] determining a first loss function for the first classification network;

[0146] Based on the first loss function, a first classification network is constructed using the preprocessed training set.

[0147] Exemplarily, a first classification network for identifying category information of an object in an image may be generated in the following manner.

[0148] 1. Data preparation:

[0149] Dataset collection:

[0150] Collect a large amount of image data containing target categories and ensure the diversity of the data, including different perspectives, lighting conditions, scales, etc.

[0151] Preprocess the image, such as resizing and normalizing.

[0152] Label annotation:

[0153] Label each image with a category to identify the target category contained in each image.

[0154] This can be done using manual annotation or in conjunction with image annotation tools.

[0155] 2. Select the network architecture of the first classification network:

[0156] Choose the right network:

[0157] Choose an appropriate network architecture based on task complexity, dataset size, and computing resources.

[0158] Commonly used network architectures are:

[0159] Shallow networks: LeNet, AlexNet, etc., suitable for small data sets or simple tasks.

[0160] Deep networks: VGG, ResNet, Inception, etc., have stronger feature extraction capabilities.

[0161] Specialized networks: such as EfficientNet and MobileNet designed for specific tasks.

[0162] 3. Build the first classification network:

[0163] Define the input layer:

[0164] The size of the input layer is consistent with the size of the image, usually a three-dimensional tensor, representing the width, height, and number of channels of the image.

[0165] Construct a convolutional layer:

[0166] Multiple convolutional layers are used to extract local features of the image.

[0167] Different kernel sizes, strides, and padding can be used.

[0168] Add a pooling layer:

[0169] The pooling layer is used to reduce the size of the feature map, reduce the number of parameters, and improve generalization ability.

[0170] Fully connected layer:

[0171] The fully connected layer is used to map the high-dimensional feature vector to the category space.

[0172] The number of neurons in the output layer is equal to the number of target categories.

[0173] 4. Select the loss function of the first classification network:

[0174] Cross entropy loss function: Commonly used in multi-classification problems, it can effectively measure the difference between the predicted probability distribution and the true label.

[0175] Other loss functions: Depending on the specific task and data set, you can consider using other loss functions, such as FocalLoss.

[0176] 5. Select the optimizer for the first classification network:

[0177] Choose the appropriate optimizer:

[0178] Commonly used optimizers include SGD, Adam, RMSprop, etc.

[0179] Different optimizers have different convergence speeds and stabilities.

[0180] 6. Train the first classification network:

[0181] Setting hyperparameters:

[0182] Hyperparameters such as learning rate, batch size, and epoch have an important impact on the model training effect.

[0183] Training process:

[0184] The training data is input into the network, the loss is calculated, and the network parameters are updated according to the loss function.

[0185] Strategies such as learning rate decay and early stopping can be used to prevent overfitting.

[0186] 7. Model Evaluation:

[0187] Evaluation indicators: Use accuracy, recall, F1-score and other indicators to evaluate the performance of the model.

[0188] Validation set: Divide the dataset into training set and validation set, and evaluate the generalization ability of the model on the validation set.

[0189] In the embodiment of the present invention, generating the first classification network is a relatively standard deep learning process. By carefully selecting the network architecture, optimizer and loss function, and conducting sufficient training and evaluation, a good classification model can be obtained, laying the foundation for subsequent positioning tasks.

[0190] Optionally, the first classification network includes a plurality of convolutional layers, the plurality of convolutional layers are arranged in sequence, and the step of generating a positioning map for expressing initial positioning information of the target through the first classification network includes:

[0191] Get the output result of the last convolutional layer;

[0192] determining a category weight for the output result;

[0193] The output results are weighted and summed based on the category weights to generate a positioning map for expressing initial positioning information of the target.

[0194] In practical applications, Class Activation Mapping (CAM) is a technique that generates a heat map of the same size as the input image by weighted summing the outputs of the last convolutional layer of the neural network. This heat map can intuitively show which areas of the image the model focuses on when classifying.

[0195] In a specific implementation, the initial positioning map can be generated through the following steps.

[0196] Get the output of the last convolutional layer of the first classification network:

[0197] In the first classification network, find the last convolutional layer (usually the last layer before the global average pooling layer).

[0198] Get the feature map output by this layer.

[0199] Calculate the weights of the categories:

[0200] Get the weights of the fully connected layer. These weights reflect the importance of different feature maps for the category.

[0201] Weighted sum:

[0202] Multiply each feature map by the corresponding weight, and then perform weighted summation on all feature maps to obtain an activation map for a category.

[0203] Normalization:

[0204] The obtained CAM is normalized, for example, using a ReLU activation function to set negative values ​​to 0, and performing L1 or L2 normalization.

[0205] Visualization:

[0206] By overlaying the obtained CAM on the original image, you can see the area that the model focuses on.

[0207] By obtaining the output result of the last convolutional layer; determining the class weight for the output result; and weighted summing the output result based on the class weight to generate a positioning map for expressing the initial positioning information of the target, the trained classification network can be used to generate a class activation map reflecting the model's focus area. CAM provides us with an intuitive way to understand the model's decision-making process and helps improve the model's performance.

[0208] Optionally, the step of generating a first segmentation network for segmenting a target area and a background area of ​​the target by using the first pseudo label includes:

[0209] Calculate the cross entropy between the segmentation map and the first pseudo label;

[0210] Determining a second loss function of the first segmentation network based on the cross entropy;

[0211] A first segmentation network for segmenting a target area and a background area of ​​the target is generated based on the second loss function.

[0212] The generated initial positioning map is used as the first pseudo label to train a segmentation network. The purpose is to speed up the model. The bright parts in the positioning map are most likely the target, and the dark parts are most likely the background. The model is allowed to learn this correspondence and generate more accurate segmentation results.

[0213] Exemplarily, the first segmentation network may be generated in the following manner.

[0214] The specific sub-steps are as follows:

[0215] Generate the first segmentation network:

[0216] Select network architecture:

[0217] U-Net: A classic semantic segmentation network, where the encoder extracts features and the decoder restores spatial information.

[0218] FPN: Feature Pyramid Network, which can extract features at different scales.

[0219] DeepLabv3+: Combines Atrous Spatial Pyramid Pooling (ASPP) and encoder-decoder structure to capture multi-scale information.

[0220] Selection basis: Choose the appropriate network architecture based on factors such as dataset size, image resolution, and hardware resources.

[0221] Construct the loss function:

[0222] Cross entropy loss: Commonly used in pixel-level segmentation tasks, it calculates the cross entropy between the predicted segmentation map and the true label (ie, pseudo label).

[0223] Dice Loss: Measures the similarity between the predicted segmentation map and the true label, and works better for unbalanced data sets.

[0224] Focal Loss: To address the class imbalance problem, higher weights are given to difficult samples.

[0225] Combined loss: Multiple loss functions can be combined to better balance errors in different aspects.

[0226] Prepare training data:

[0227] Input data: original image.

[0228] Label: The localization map generated in step S2.

[0229] Data enhancement: Random cropping, flipping, rotation and other data enhancements can be performed to increase data diversity.

[0230] Training process:

[0231] Initialization: Randomly initialize the parameters of the segmentation network.

[0232] Forward propagation: Input the input image into the network and get the predicted segmentation map.

[0233] Calculate loss: Calculate the loss between the predicted segmentation map and the pseudo label.

[0234] Back propagation: Update the network parameters according to the loss function.

[0235] Iterative training: Repeat the above process until the model converges or reaches the preset number of iterations.

[0236] In an embodiment of the present invention, by calculating the cross entropy between the segmentation map and the first pseudo-label; determining the second loss function of the first segmentation network based on the cross entropy; and generating a first segmentation network for segmenting the target area and the background area of ​​the target based on the second loss function, a positioning map can be used as a pseudo-label to train a segmentation network that can better segment the target area.

[0237] Optionally, the step of converting the first segmentation network into a second classification network includes:

[0238] Determining a segmentation head of the first segmentation network;

[0239] Determine the classification global pooling layer and classification fully connected layer for classification tasks;

[0240] Remove the segmentation head, and add the classification global pooling layer and the classification fully connected layer to generate an initial classification network;

[0241] The weights of the encoder part of the initial segmentation network are fixed to generate a second classification network.

[0242] In practical applications, a network trained for segmentation tasks is adapted to classification tasks by modifying some of its structures. The purpose is that the segmentation network has learned rich image features that are also useful for classification tasks.

[0243] Exemplarily, the first segmentation network can be converted into the second classification network in the following manner.

[0244] Remove the split header:

[0245] Segmentation head: The last part of the segmentation network, usually multiple 1x1 convolutional layers, is used to map the feature map to the class probability of each pixel.

[0246] Reason: The output of the segmentation task is a pixel-level class probability map, while the output of the classification task is a class probability vector. Therefore, the output form of the segmentation head does not match that of the classification task.

[0247] Action: Simply remove the split head from the network.

[0248] Add a global pooling layer:

[0249] Function: Convert the feature map into a fixed-size feature vector to prepare for the fully connected layer.

[0250] Position: A global average pooling layer or a global maximum pooling layer is usually added at the position of the removed segmentation head.

[0251] Global average pooling: Calculate the average value of each channel feature map to obtain a C-dimensional vector, where C is the number of channels.

[0252] Global maximum pooling: Calculate the maximum value of each channel feature map to obtain a C-dimensional vector.

[0253] Add a fully connected layer:

[0254] Function: Map the output of the global pooling layer to the category space.

[0255] Number of neurons: The number of neurons in the output layer is equal to the number of target categories.

[0256] Activation function: Softmax activation function is usually used to convert the output into probability distribution.

[0257] Fixed part weights:

[0258] Fixed feature extraction part: The weights of the encoder part (i.e., feature extraction part) of the segmentation network are fixed and no longer participate in training.

[0259] Reason: This part of the weights has been trained for the segmentation task and contains rich image features, which can be well transferred to the classification task.

[0260] Fine-tune the fully connected layer: Only train the weights of the newly added fully connected layer.

[0261] Optionally, the global pooling layer is added as follows:

[0262] The average value and the maximum value of each channel feature map of the global pooling layer are calculated, and the global pooling layer is added based on the average value and the maximum value.

[0263] Through the above steps, a network trained for segmentation tasks can be converted into a classification network. This process makes full use of the features that the segmentation network has learned, thereby improving the performance of the new classification network.

[0264] In order to enable those skilled in the art to better understand the embodiment of the present invention, an example is used below to illustrate the embodiment of the present invention.

[0265] refer to Figure 2 , Figure 2 is a flow chart of another target positioning method provided in an embodiment of the present invention;

[0266] First, the implementation of the present invention can train a first classification network for generating pseudo labels. When training the classification network, a first segmentation network for segmenting the target area and the background area of ​​the target is trained in parallel on the basis of the same backbone network, so that the network can better capture the visual features of the entire image, rather than only the salient features that serve the classification. Specifically, when training the classification network, a certain amount of noise and occlusion are randomly added to the image to train an image reconstruction network. The mean square error (MSE) loss function is used during training to encourage the image reconstructed by the network to be as similar as possible to the original image at the pixel level. In this way, the image reconstruction task serves as an auxiliary to help the main network understand the image content more comprehensively, so that it has richer background information and detailed features when performing classification and positioning tasks.

[0267] refer to Figure 3 , Figure 3 is a flow chart of another target positioning method provided in an embodiment of the present invention;

[0268] An iteratively enhanced weakly supervised learning method is introduced to iterate the network's positioning capability multiple times by alternately replacing the output heads of the same backbone network corresponding to different characters.

[0269] Specifically, the present invention proposes an innovative iterative enhanced target localization method. The method effectively improves the accuracy and robustness of target localization through a series of carefully designed steps. The core of the present invention is to apply the segmentation network optimized model to the classification task, thereby achieving a more accurate Class Activation Map (CAM) generation, and the specific steps are as follows.

[0270] Step S1, classification network training: First, in combination with the image reconstruction task mentioned above, a first classification network is trained, which is designed to identify category information in the image.

[0271] Step S2, localization map generation: Based on the trained first classification network, combined with the global average pooling technique, the weight of the last fully connected layer is multiplied back to the feature map after the pooling layer to generate a CAM. This map indicates where the target may appear in the image.

[0272] Step S3, segmentation network training: Use the CAM generated in step S2 as pseudo labels to train a first segmentation network. The task of the first segmentation network is to separate the target area from the background area in the image, so as to obtain more accurate target positioning information.

[0273] Step S4, conversion from segmentation network to classification network: This step is an important part of this technology. The trained first segmentation network is converted into a new second classification network. This conversion includes removing the segmentation head of the first segmentation network and replacing it with a head suitable for the classification task. In addition, the second classification network loads and fixes the weights of the first segmentation network as the basis for its feature extraction.

[0274] The weights in S2 and S4 are not exactly the same, but there is a close connection.

[0275] The weights in S2 are the parameters that connect the fully connected layer to the last convolutional layer. They reflect the contribution of different feature channels to the final classification result, that is, which features are more important for identifying the current category. When generating CAM, these weights are used as weighting coefficients of the feature map.

[0276] The weights in S4 mainly come from the fine-tuned segmentation network. Although the classification head is replaced, the feature extraction part (i.e., most of the weights) are shared. After training on the segmentation task, these weights have a deeper understanding of the characteristics of the target area.

[0277] The relationship between the two:

[0278] The weights of S2 provide initialization for S4: when S4 is converted, it will load the weights trained by S2 as the initial values, which helps to speed up the training of S4 and retain the useful information learned by S2.

[0279] The weights of S4 are fine-tuned: After training on the segmentation task, the weights in S4 are adjusted to a certain extent to better suit the segmentation task. This makes the CAM generated by S4 reflect the target area more accurately.

[0280] The reason why the weight of S2 is not exactly the same as the weight of S4 is:

[0281] Different tasks: Although there are overlaps between classification and segmentation tasks, they focus on different aspects. Classification tasks focus more on global semantic information, while segmentation tasks require more fine-grained local features.

[0282] Different training objectives: The loss functions of the two tasks are different, resulting in different weight update directions.

[0283] The weights in S2 and S4 are related but different. The weights in S2 provide a good initialization for S4, and after fine-tuning the weights in S4 for the segmentation task, the positioning of the target area is more accurate. This inheritance and optimization of weights is the key to the entire iterative process.

[0284] Step S5, generating a more accurate positioning map: using the second classification network after feature enhancement, generating a more accurate positioning map. This step uses the same or improved technology as step S2 to extract more detailed target location information.

[0285] Further training the segmentation network: Using the more accurate positioning map generated in step S5 as a pseudo label, a new second segmentation network is trained to further optimize the segmentation accuracy of the target and improve the accuracy of positioning.

[0286] refer to Figure 4 , Figure 4 is a flowchart of another target positioning method provided in an embodiment of the present invention;

[0287] After multiple iterations, the positioning map generated by the above network, namely the class activation map, is thresholded and pixels larger than the foreground threshold are regarded as target areas. The maximum connected area technology and the minimum positioning box technology are combined to generate the target positioning result.

[0288] Through the above-mentioned method, the accuracy of target positioning can be significantly improved through the iterative enhanced learning process without increasing the additional annotation cost. By utilizing the complementarity between the classification network and the segmentation network, the present invention realizes the deep mining and optimization of image features, so that the final target positioning is not only accurate, but also has good robustness for targets under various complex backgrounds. In addition, by applying the model conversion after the segmentation network optimization to the classification task, this technology can effectively utilize the sensitivity of the segmentation network to details, and further enhance the accuracy and reliability of the class activation map generated by the classification network.

[0289] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0290] Reference Figure 5 , shows a structural block diagram of a target positioning device provided in an embodiment of the present invention, which may specifically include the following modules:

[0291] A first classification network generation module 501, used to generate a first classification network for identifying category information of an object in an image;

[0292] A first positioning map generating module 502, configured to generate a positioning map for expressing initial positioning information of the target through the first classification network;

[0293] A first segmentation network generation module 503 is used to determine the positioning map used to express the initial positioning information of the target as a first pseudo label, and generate a first segmentation network used to segment the target area and the background area of ​​the target through the first pseudo label;

[0294] A network conversion module 504, configured to convert the first segmentation network into a second classification network;

[0295] The second positioning map generating module 505 is used to generate a positioning map for expressing the updated positioning information of the target through the second classification network.

[0296] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0297] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0298] Memory 603, used for storing computer programs;

[0299] The processor 601 is used to implement the target positioning method described in any one of the above embodiments when executing the program stored in the memory 603:

[0300] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0301] The communication interface is used for communication between the above terminal and other devices.

[0302] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0303] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0304] like Figure 7 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 701 is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the target positioning method described in the above embodiment.

[0305] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

[0306] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0307] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0308] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0309] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0310] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0311] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0312] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A target positioning method, characterized in that: include: A first classification network is generated for identifying category information of an object in an image; Generating, by the first classification network, a positioning map for expressing initial positioning information of the target; Determine a positioning map for expressing initial positioning information of the target as a first pseudo label, and generate a first segmentation network for segmenting a target area and a background area of ​​the target through the first pseudo label; Converting the first segmentation network into a second classification network; A positioning map for expressing updated positioning information of the target is generated by the second classification network.

2. The method according to claim 1, characterized in that Also includes: A positioning map for expressing the updated positioning information of the target is determined as a second pseudo label, and the second pseudo label is used for a second segmentation network for segmenting a target area and a background area of ​​the target.

3. The method according to claim 1, characterized in that The step of generating a first classification network for identifying category information of an object in an image comprises: Obtaining a training set and performing a preprocessing operation on the training set; determining a first loss function for the first classification network; Based on the first loss function, a first classification network is constructed using the preprocessed training set.

4. The method according to claim 1, characterized in that: The first classification network includes a plurality of convolutional layers, and the plurality of convolutional layers are arranged in sequence. The step of generating a positioning map for expressing initial positioning information of the target through the first classification network includes: Get the output result of the last convolutional layer; determining a category weight for the output result; The output results are weighted and summed based on the category weights to generate a positioning map for expressing initial positioning information of the target.

5. The method according to claim 4, characterized in that The step of generating a first segmentation network for segmenting the target area and the background area of ​​the target by using the first pseudo label comprises: Calculate the cross entropy between the segmentation map and the first pseudo label; Determining a second loss function of the first segmentation network based on the cross entropy; A first segmentation network for segmenting a target area and a background area of ​​the target is generated based on the second loss function.

6. The method according to claim 1, characterized in that The step of converting the first segmentation network into a second classification network comprises: Determining a segmentation head of the first segmentation network; Determine the classification global pooling layer and classification fully connected layer for classification tasks; Remove the segmentation head, and add the classification global pooling layer and the classification fully connected layer to generate an initial classification network; The weights of the encoder part of the initial segmentation network are fixed to generate a second classification network.

7. The method according to claim 6, characterized in that The global pooling layer is added as follows: The average value and the maximum value of each channel feature map of the global pooling layer are calculated, and the global pooling layer is added based on the average value and the maximum value.

8. A target positioning device, characterized in that: include: A first classification network is generated for identifying category information of an object in an image; Generating, by the first classification network, a positioning map for expressing initial positioning information of the target; Determine a positioning map for expressing initial positioning information of the target as a first pseudo label, and generate a first segmentation network for segmenting a target area and a background area of ​​the target through the first pseudo label; Converting the first segmentation network into a second classification network; A positioning map for expressing updated positioning information of the target is generated by the second classification network.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7.

Images