Area perception open set identification method based on causal class activation mapping

Abstract

The invention discloses a causal class activation mapping-based region perception open set identification method, which comprises the following steps of: performing closed set training through a convolutional neural network by taking a training image of a known class as input, and generating class activation mapping reflecting causal contribution of a local region to a classification result based on region ablation analysis; the method comprises the following steps: in a training stage, extracting a region perception feature vector of a training image by utilizing a saliency map, and constructing a region prototype corresponding to a known category; in a test stage, region perception feature vectors are extracted from a test image, a region consistency score is calculated with the region prototype constructed in the training stage, and fusion judgment is carried out in combination with a global uncertainty score based on an energy model, so that known category identification and unknown category rejection are realized. According to the method, causal class activation mapping is introduced into an open set identification judgment process, so that the capability of distinguishing context similar unknown classes can be enhanced, and a regional interpretable basis is provided.

Description

A Region-Aware Open Set Recognition Method Based on Causal Class Activation Mapping Technical Field

[0001] This invention belongs to the field of open set recognition in pattern recognition, specifically relating to a region-aware open set recognition method based on causal class activation mapping. Background Technology

[0002] With the development of deep learning technology, neural network-based image recognition methods have been widely used in fields such as autonomous driving and medical image analysis. While these models can achieve high recognition accuracy under known category conditions, in real-world applications, input data often contains unknown category samples that were not present during training, thus placing higher demands on the reliability and security of the models.

[0003] To address these challenges, researchers have proposed Open Set Recognition (OSR) technology, which aims to effectively detect and reject samples of unknown categories while ensuring the accuracy of known category recognition. Related research has developed into several technical branches, including methods based on probabilistic calibration, generative and contrastive learning, prototype-based methods, and energy model-based methods. Existing open set recognition methods typically make judgments based on the distribution characteristics of model outputs or feature space representations, and their decision-making process mainly relies on global feature information extracted from the entire image. However, in the process of constructing global features, foreground targets and background context information are often co-encoded. When unknown categories are similar to known categories in overall appearance or scene structure, and the distinction is mainly based on differences in local regions, existing methods struggle to accurately characterize this key discriminative information, easily leading to misclassification.

[0004] On the other hand, to enhance the interpretability of the model's decision-making process, researchers have proposed Class Activation Mapping (CAM) technology to generate saliency maps reflecting the regions of interest to the model. Existing CAM methods typically estimate regional importance based on gradient information, feature responses, or feature perturbations, which can reveal the spatial locations of interest to the model to some extent. However, these methods mainly rely on indirect indicators to infer regional importance, making it difficult to directly measure the actual impact of changes in specific regions on the model's prediction results. Their saliency results mainly reflect the distribution of the model's response to input features, making it difficult to directly characterize the actual impact of local regional changes on the classification decision.

[0005] In open set recognition scenarios, if the direct contribution of local regions to classification decisions cannot be clearly characterized, class activation mapping is difficult to introduce as an effective basis for discrimination into the recognition process, and open set recognition methods still mainly rely on global features for judgment. Current technologies lack a technical solution that can effectively incorporate region-level class activation results into the open set recognition discrimination mechanism, making it difficult to effectively improve the detection capability of unknown categories when local differences dominate. Summary of the Invention

[0006] This invention provides a region-aware open set recognition method based on causal class activation mapping, aiming to solve the core limitation of existing open set recognition methods, which are sensitive to local anomalies of known categories due to their reliance on global features.

[0007] To achieve the objectives of this invention, the proposed technical solution is: a region-aware open set recognition method based on causal class activation mapping, comprising the following steps:

[0008] Step 1, Closed-set training phase: Train the convolutional neural network using training images of known categories, optimize the network parameters by minimizing the classification loss function, and obtain a convolutional neural network that can map the input image into a feature map and output the category prediction result.

[0009] Step 2, Region-Aware Feature Extraction Stage: Based on the convolutional neural network, an ablation operation is performed on local spatial regions in the feature map. Based on the change in the corresponding category prediction results before and after ablation, a class activation map reflecting the influence of local spatial regions on the category prediction results is generated, and its output is a saliency map. Then, based on the saliency map, weighted pooling is performed on the feature map to extract a one-dimensional region-aware feature vector for each image. Finally, the region-aware feature vectors corresponding to all training images of the same known category are aggregated to construct the region prototype corresponding to that known category.

[0010] Step 3, Dual-Score Inference Stage: First, acquire the test image and generate a saliency map and region-aware feature vector corresponding to the test image using the convolutional neural network; then, calculate the region consistency score between the region-aware feature vector and the region prototype; simultaneously, calculate the global uncertainty score based on the category prediction result of the test image; then, fuse the region consistency score and the global uncertainty score; finally, determine whether the test image belongs to a known class or an unknown class based on the comparison result of the fused score and a preset threshold.

[0011] Furthermore, in step two above, the step of generating the saliency map includes:

[0012] 2.1. Input the training image into the feature extraction model and perform forward propagation to obtain the final feature map and target category prediction value;

[0013] 2.2 Define a set of grid regions covering all spatial locations of the feature map, perform an ablation operation on each grid region, and obtain the ablated feature map by setting the features within the grid region to zero or to a preset reference value;

[0014] 2.3 Input the ablated feature map into the subsequent network layer to obtain the ablated category prediction value;

[0015] 2.4 Calculate the change in the predicted value of the target category before and after ablation, and use it as the importance score for each grid region;

[0016] 2.5. Combining the feature channel gradient weights with the importance scores, the final saliency map is generated using the Rectified Linear Unit (ReLU).

[0017] Furthermore, in step two above, the region prototype is obtained by aggregating the region-aware feature vectors of all training images corresponding to the known category. The aggregation method includes calculating the centroid of the region-aware feature vector.

[0018] Furthermore, in step three above, the regional consistency score is used to quantify the similarity deviation between the region-aware feature vector and the region prototype by calculating a similarity metric or distance metric between them; the global uncertainty score is calculated based on an energy model, which determines the degree of uncertainty of the test image in the known category space based on the category prediction results of the test image.

[0019] Furthermore, in step three above, the fusion process adopts a weighted fusion method, which obtains a fusion score by assigning weights to the regional consistency score and the global uncertainty score respectively and then summing them; when the fusion score is greater than a preset threshold, the test image is determined to be of the unknown class, otherwise it is determined to be of the predicted known class.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] 1. This invention applies ablation intervention to local regions in the feature space and quantifies the importance of these regions based on the changes in prediction results before and after ablation, generating a saliency map with causal significance. This alleviates the localization instability problem caused by traditional class activation mapping methods that rely on indirect estimation using gradients or feature responses, and facilitates the acquisition of more discriminative local region information. This invention introduces this type of region information into the open set identification and discrimination process, successfully providing a region analysis method capable of characterizing the influence of local regions on classification decisions.

[0022] 2. The present invention extracts region-aware features based on the saliency map generated by the causal class activation mapping, constructs the region prototype corresponding to the category, and then introduces the local region discrimination information into the discrimination process of open set recognition, so that the model can simultaneously consider local region consistency and global uncertainty information when making judgments on known and unknown categories.

[0023] 3. This invention uses the fusion of regional consistency score and global uncertainty score to make open set recognition decisions, which helps to alleviate the problem of unknown class images with similar context being misclassified as known class images, and enhances the discrimination between known and unknown classes.

[0024] 4. This invention introduces regional saliency results as a criterion during the identification process, enabling the model's decision-making process to have clear spatial interpretability, making it suitable for application scenarios with high requirements for identification reliability and interpretability. Detailed Implementation

[0025] The present invention will be further described in detail below with reference to specific embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0026] The design concept of the technical solution proposed in this invention is as follows:

[0027] First, during the closed-set training phase, the convolutional neural network is trained under supervision using training images of known categories. The network parameters are optimized by minimizing the classification loss function, enabling the convolutional neural network to learn to extract discriminative features that distinguish different known categories. After training, the network parameters are frozen to provide a stable feature mapping foundation for subsequent region-aware feature extraction and inference.

[0028] Then, in the region-aware feature extraction stage, to address the issue of traditional class activation mapping being more about correlation than causation, region ablation analysis is introduced to quantify the causal contribution of local regions. Specifically, based on the trained network, forward propagation is performed on each training image to obtain its corresponding feature map. By systematically ablating local spatial regions on the feature map and observing the change in the model's prediction confidence for the target class before and after ablation, the causal importance of each region to the classification decision is directly calculated, thereby generating a causal class activation map (i.e., a saliency map). Furthermore, this saliency map is used as a spatial attention weight to perform weighted pooling on the original feature map, thereby extracting a one-dimensional region-aware feature vector focused on the discriminative region. Finally, by calculating the centroid of the region-aware feature vectors of all training images under the same known class, a region prototype representing the discriminative region pattern of that class is constructed.

[0029] Finally, in the dual-score inference stage, the same region ablation and saliency generation methods as in the training stage are used on the test image to extract the region-aware feature vector. The region prototypes constructed and fixed in the training stage are invoked to calculate the region consistency score of the test image in the local discriminative regions. Simultaneously, a global uncertainty score is calculated based on the category prediction results of the test image. By fusing the region consistency score and the global uncertainty score, a comprehensive judgment score is obtained. Finally, by comparing this fused score with a preset threshold, a robust determination of whether the test image belongs to a known or unknown category is achieved: a score higher than the threshold indicates a significant anomaly and is classified as an unknown class; otherwise, it is classified as a predicted known class.

[0030] Example: This example aims to illustrate the specific implementation of the present invention in detail. A region-aware open set recognition method based on causal class activation mapping constructs feature extraction capabilities through closed set training, generates class-specific region prototypes through region-aware processing, and completes category determination through dual-fraction reasoning, forming a complete "training-modeling-reasoning" process. Specifically, it includes the following steps:

[0031] Step 1, Closed-Set Training Phase: The convolutional neural network is trained using training images of known categories. The network parameters are optimized by minimizing the classification loss function to obtain a convolutional neural network capable of mapping input images to feature maps and outputting category prediction results. This includes the following sub-steps:

[0032] 1.1 First, construct a training dataset of known classes for closed-set training. Let the training dataset be... Where x i Let y represent the i-th input training image. i This represents the corresponding category label, and y i ∈C known C known Given a known set of categories, with k categories, the training images serve as the input data for the convolutional neural network.

[0033] 1.2 A convolutional neural network is used as the backbone network, which includes a convolutional feature extraction layer, a global average pooling layer, and a fully connected classification layer connected in sequence. The convolutional feature extraction layer is used to process the input training image x. i Stepwise mapping to feature map A∈R C×H×W It then outputs the predicted logit vector z for the corresponding category through a fully connected classification layer. i ∈R k ;

[0034] 1.3. Based on the logit vector output by the above network, the convolutional neural network is trained using a classification loss function. By calculating the loss between the logit vector and the true class label, and minimizing this loss function, the parameters of the convolutional neural network are optimized, enabling the convolutional neural network to learn to extract discriminative features that can distinguish known classes. In this embodiment, the classification loss function is cross-entropy loss;

[0035] 1.4 After training is completed, the parameters of the trained convolutional neural network are frozen for subsequent region-aware feature extraction and inference processes.

[0036] Step 2, Region-Aware Feature Extraction Stage: Based on the trained convolutional neural network, region ablation analysis is performed on the feature maps of the training images to generate causal class activation maps and construct region prototypes. The specific steps are as follows:

[0037] 2.1 For each training image x i The feature map A is then input into the convolutional feature extraction layer to obtain the corresponding feature map A. The feature map A is then input into a global average pooling layer to converge the spatial dimensions, resulting in a one-dimensional feature vector. This one-dimensional feature vector is then input into a fully connected classification layer to output the target class logit value z. c ;

[0038] 2.2 For feature map A, define a set of grid regions R = {R1, R2, ..., R...} covering all spatial locations in its spatial dimension. L}; sequentially process each grid region R S Perform an ablation operation, set the feature values ​​in the region to zero or to a preset baseline value, and obtain the ablated feature map A / R. S ;

[0039] 2.3. Convert the ablation feature map A / R S Repeat step 2.1 to obtain the target category logit value after ablation.

[0040] 2.4 Calculate the change in the target class logit value before and after ablation to obtain the importance score for each grid region. in This is the average of the original logit values ​​for all classes except the target class c;

[0041] 2.5 Calculate the gradient weights of the network feature channels. And combined with the regional importance score α s A saliency plot is generated using a linear rectified function.

[0042] 2.6. The significance plot M cNormalization is performed, and the feature map A is used as an attention weight for weighted pooling. Through weighted pooling, highly salient features in feature map A are enhanced, while less salient features are weakened, ultimately yielding a one-dimensional region-aware feature vector f for each image. region =∑ i,j M(i,j)·A(:,i,j), where A(:,i,j) is the feature vector of the spatial position (i,j);

[0043] 2.7 For all training images of the same known class c, collect all region-aware feature vectors obtained in step 2.6, and calculate the centroid of this set of vectors as the region prototype of that class: Where N c Let be the number of training images for class c. Let be the region-aware feature vector of the i-th training image. This prototype embodies the most representative local discriminative features of this category.

[0044] Step 3, Double Fraction Reasoning Stage:

[0045] 3.1. The test image to be identified, x... test Input the convolutional feature extraction layer trained in step 1.3, and obtain the corresponding feature map A through forward propagation. test And the logit vector z output by the fully connected classification layer corresponding to all known classes c. test,c The predicted known category of the test image is determined based on the maximum logit value.

[0046] 3.2. For predicting known categories Generate feature map A corresponding to the test image. test Using the same regional ablation causal analysis method as in step two, a significance map corresponding to the predicted category is generated. Then, using the same weighted pooling method as in step 2.6, the region-aware feature vector f of the test image is extracted. rehion,test ;

[0047] 3.3 Calculation of f using cosine distance rehion,test The similarity deviation from the regional prototype yields the regional consistency score S. region Then, the region consistency score is normalized to eliminate the influence of different feature scales on the score amplitude, ensuring that the score is comparable when fused with the global uncertainty score in the subsequent process. The normalized region consistency score is used for subsequent fusion and open set determination. The normalized region consistency score is used to quantify the degree of deviation between the test image and the predicted category in the local discrimination region. The larger the score, the lower the consistency between the local features and the corresponding category region prototype.

[0048] Simultaneously, based on the test image category prediction logit vector z obtained in step 3.1 test,c The global uncertainty score of the test image is calculated using an energy model: Where T is the temperature parameter;

[0049] The global uncertainty score is then normalized to ensure its numerical scale aligns with the region consistency score, thus preventing a single score from dominating the judgment result due to numerical scale differences during subsequent fusion. The normalized global uncertainty score is used for subsequent fusion and open set determination. The normalized global uncertainty score reflects the degree of deviation of the test image from the known category distribution in the overall feature space; a higher score indicates a higher probability that the test image belongs to an unknown category.

[0050] 3.4 Fusion Process:

[0051] Set the equilibrium hyperparameter λ and use the fusion formula S final =λ·S global +(1-λ)·S region The normalized regional consistency score and global uncertainty score from step 3.3 are then fused to obtain the final fused score S. final ; and based on the final fusion score S final Compared with a preset threshold τ, the open set of the test image is determined: if the score is higher than the threshold, the image is determined to be of unknown class; if it is lower than the threshold, the image is determined to be of predicted known class. At the same time, the saliency map generated in step 3.2 is output as a regional visualization basis for decision-making.

[0052] The following simulation experiments exemplify the application of the method of the present invention in open set recognition scenarios. It should be noted that the simulation experiments described below are only used to illustrate the feasibility of the technical solution described in this invention and its application effect in open set recognition tasks.

[0053] 1. Simulation conditions

[0054] The hardware platform for the simulation experiment in this embodiment is:

[0055] Processor: Intel(R) Core(TM) i9-10980XE, clock speed 3.00GHz, memory 256GB.

[0056] The software platform for the simulation experiment in this embodiment is: Ubuntu 20.04 operating system and Python 3.10.

[0057] 2. Dataset and Experiment Setup:

[0058] To illustrate the application of the method of the present invention in open set recognition tasks, simulation experiments were conducted on multiple public datasets, including CIFAR10, CIFAR+10, CIFAR+50, and TinyImageNet datasets.

[0059] In the experimental setup, different datasets were divided into known and unknown categories according to preset rules, where:

[0060] Known and unknown class classification: In the CIFAR10 dataset, 6 classes are randomly selected as known classes, and the remaining 4 classes are considered unknown classes. For CIFAR+10, 4 classes are randomly selected from CIFAR10 as known classes, and 10 classes are randomly selected from CIFAR100 as unknown classes. For CIFAR+50, 4 classes are randomly selected from CIFAR10 as known classes, and 50 classes are randomly selected from CIFAR100 as unknown classes. In the TinyImageNet dataset, 20 classes are randomly selected as known classes, and the remaining 180 classes are considered unknown classes.

[0061] Under the same backbone network structure and training parameters, the method of this invention, along with SoftMax, GCPL, RPL, and RAD methods, were applied to the above datasets to obtain the corresponding open set recognition results.

[0062] Evaluation indicators:

[0063] (1) The area under the receiver operating characteristic curve (AUROC) was used to evaluate the model's ability to distinguish unknown samples.

[0064] (2) The open-set classification rate (OSCR) is used to comprehensively evaluate the overall recognition performance of the model for known and unknown classes.

[0065] 3. Simulation Result Analysis

[0066] Table 1 shows exemplary results of the AUROC and OSCR metrics for each method on different datasets.

[0067] Table 1

[0068]

[0069] As can be seen from the experimental data in Table 1, the region-aware open set recognition method based on causal class activation mapping proposed in this invention improves the values ​​of both AUROC and OSCR compared with existing mainstream methods. The results are used to illustrate the practical application effect of the method of this invention in the open set recognition task.

[0070] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A region-aware open set recognition method based on causal class activation mapping, characterized in that: The process includes the following steps: Step 1, Closed-set training phase: The convolutional neural network is trained using training images of known categories. The network parameters are optimized by minimizing the classification loss function to obtain a convolutional neural network capable of mapping input images to feature maps and outputting category prediction results. Step 2, Region-aware feature extraction phase: Based on the convolutional neural network, ablation operations are performed on local spatial regions in the feature map. Based on the changes in the corresponding category prediction results before and after ablation, a class activation map reflecting the degree of influence of local spatial regions on the category prediction results is generated, and its output is a saliency map. Then, based on the saliency map, weighted pooling is performed on the feature map to extract a one-dimensional region-aware feature map for each image. First, the region-aware feature vectors are known; then, the region-aware feature vectors corresponding to all training images of the same known category are aggregated to construct the region prototype corresponding to the known category; Step 3, Dual-Score Inference Stage: First, the test image is acquired, and the convolutional neural network generates the saliency map and region-aware feature vector corresponding to the test image; then, the region consistency score between the region-aware feature vector and the region prototype is calculated; simultaneously, the global uncertainty score is calculated based on the category prediction result of the test image; then, the region consistency score and the global uncertainty score are fused; finally, based on the comparison result of the fused score and the preset threshold, the test image is determined to be of a known class or an unknown class.

2. The region-aware open set recognition method based on causal class activation mapping according to claim 1, characterized in that: Step two, the step of generating the saliency map, includes: 2.

1. Inputting the training image into the feature extraction model for forward propagation to obtain the final feature map and target category prediction value; 2.

2. Defining a set of grid regions covering all spatial locations of the feature map, performing an ablation operation on each grid region, and obtaining the ablated feature map by setting the features within the grid region to zero or to a preset baseline value; 2.

3. Inputting the ablated feature map into subsequent network layers to obtain the ablated category prediction value; 2.

4. Calculating the change in the target category prediction value before and after ablation, as the importance score for each grid region; 2.

5. Combining the feature channel gradient weights with the importance score, generating the final saliency map using the Rectified Linear Unit (ReLU) function.

3. The region-aware open set recognition method based on causal class activation mapping according to claim 1, characterized in that: In step two, the region prototype is obtained by aggregating the region-aware feature vectors of all training images corresponding to the known category. The aggregation method includes calculating the centroid of the region-aware feature vector.

4. The region-aware open set recognition method based on causal class activation mapping according to claim 1, characterized in that: In step three, the regional consistency score is used to quantify the similarity deviation between the region-aware feature vector and the region prototype by calculating the similarity measure or distance measure between them; the global uncertainty score is calculated based on the energy model, which determines the degree of uncertainty of the test image in the known category space according to the category prediction result of the test image.

5. The region-aware open set recognition method based on causal class activation mapping according to claim 1, characterized in that: In step three, the fusion process adopts a weighted fusion method, which obtains the fusion score by assigning weights to the regional consistency score and the global uncertainty score respectively and then summing them; when the fusion score is greater than a preset threshold, the test image is determined to be of the unknown class, otherwise it is determined to be of the predicted known class.

Images