An interactive semantic perception self-learning framework and an interpretable visual recognition method

Abstract

The application discloses an interactive semantic perception self-learning framework and an interpretable visual recognition method, and relates to the technical field of computer vision. The interactive semantic perception self-learning framework comprises a teacher module and a student module. The student module is used for visual recognition, and the teacher module is used for semantic guidance on the student module. The student module inputs calculated slice-feature pairs to the teacher module, and the teacher module outputs calculated semantic-rich slices to the student module. The teacher module comprises a first encoder, a category concept library and a similarity comparison submodule. The student module comprises a slice library, a second encoder and a feature selection submodule. The application can accurately capture features of different granularities, has excellent generalization and accuracy, enhances the interpretability of the alignment of abstract semantic concepts and specific image regions, and is compatible with different network structures.

Description

Technical Field

[0001] This invention relates to the field of interpretable visual recognition technology in computers, and more specifically, to an interactive semantic-aware self-learning framework and an interpretable visual recognition method. Background Technology

[0002] Explainable visual recognition aims to generate interpretable and transparent feature representations, thereby enhancing the interpretability and traceability of visual recognition models. Currently, research in explainable visual recognition mainly focuses on two directions: self-explainable models and post-analysis methods. Self-explainable models are transparent and interpretable, allowing for the extraction of concepts generated by the visual recognition model. Post-analysis methods improve the interpretability of the model by analyzing its output; some related research focuses on using high-level convolutional layers to learn concentrated semantic regions of image areas, such as ProtoPNet and ProtoPFormer.

[0003] ProtoPNet is a discriminative method that utilizes category-specific prototypes based on convolutional neural networks (CNNs) to accurately perceive and recognize objects. ProtoPNet performs visual interpretation by comparing input images with selected category-specific prototypes and calculating similarity, but this method is highly dependent on the selection of category prototypes, and the selection of category prototypes is relatively coarse-grained. ProtoPFormer extends the ViT architecture using a prototype-based approach to achieve interpretable visual recognition. However, due to the lack of human guidance during model training, its interpretability in aligning abstract semantic concepts with specific image regions remains limited, and the extraction of image regions is relatively coarse-grained. Furthermore, existing interpretable methods have limited applicability, generally only supporting CNNs or ViTs. Summary of the Invention

[0004] To overcome the shortcomings of existing visual recognition models, such as the lack of human guidance during training, limited interpretability in aligning abstract semantic concepts with specific image regions, and the relatively limited applicability of existing interpretable methods, this invention provides an interactive semantic perception self-learning framework and an interpretable visual recognition method.

[0005] The primary objective of this invention is to solve the aforementioned technical problems. The technical solution of this invention is as follows:

[0006] The first aspect of this invention proposes an interactive semantic perception self-learning framework, comprising a teacher module and a student module. The student module is used for visual recognition, and the teacher module is used to provide semantic guidance to the student module. The student module inputs the calculated slice-feature pairs into the teacher module, and the teacher module outputs the calculated semantically rich slices to the student module. The teacher module includes a first encoder, a category concept library, and a similarity comparison submodule. The encoder is used to extract image features; the category concept library is used to store category concepts; and the similarity comparison submodule is used to generate semantically rich slices, i.e., semantic guidance. The student module includes a slice library, a second encoder, and a feature selection submodule. The slice library is used to store image slices for classification; and the feature selection submodule is used to select features from the image slice features generated by the encoder.

[0007] Furthermore, the encoder is a pre-trained feature extraction network, which can be CNN, ViT, or Swin-Transformer. The second encoder of the student module is the same encoder as the first encoder of the teacher module, with the same network structure, and the teacher module and the student module can share a single encoder.

[0008] The category concept library stores a global concept feature for each image category, i.e., a category concept. Its structure is a dictionary structure, including keys and values. The key is the category, and the value is the global feature vector corresponding to the category.

[0009] A second aspect of this invention proposes an interpretable visual recognition method based on an interactive semantic-aware self-learning framework, comprising the following steps:

[0010] S1: After the image is sliced, it is input into the teacher module and student module respectively, and the teacher module and student module are initialized;

[0011] S2: The student module performs feature selection on the segmented image to generate segment-feature pairs;

[0012] S3: Use shard-feature pairs to query the corresponding index in the teacher module and generate patch-specific feature responses;

[0013] S4: The teacher module compares the feature responses with index pairs to identify and generate semantically rich slices;

[0014] S5: Update the teacher and student modules using semantically rich slices;

[0015] S6: Use the updated student module to classify images.

[0016] Furthermore, in step S1, the image is segmented and input into the teacher module and student module respectively, and the teacher module and student module are initialized. The specific process is as follows:

[0017] The input image is segmented into pieces of a preset size and then forwarded to the student module and the teacher module. The student module stores the received pieces in the piece library, and the teacher module inputs the received pieces into the encoder for feature extraction and uses the obtained features to update the category concept library.

[0018] Furthermore, in step S2, the student module performs feature selection on the segmented image to generate segment-feature pairs. The specific process is as follows:

[0019] The student module inputs the slices from the slice library into the encoder in the student module for feature extraction, and then inputs the obtained features into the feature selection submodule to generate slice-feature pairs;

[0020] First, the attention weight matrix is ​​obtained based on the attention mechanism.

[0021]

[0022] Where x i It is the original input sample, w ps f represents the parameters in the encoder. ps This represents the process of feature selection from the original input sample, where N is the number of segments in the original input sample. This indicates that during the patch selection process, patches are indexed using i and j, and ps stands for patch selection;

[0023] Secondly, identify the top-k elements with the highest attention weight, 1≤k≤N; finally, select features from the input image corresponding to the row and column indices of the top k values ​​in the attention matrix.

[0024] Furthermore, step S3, which involves using shard-feature pairs to query the corresponding index in the teacher module and generate patch-specific feature responses, is described in the following process:

[0025] (1) Use the shard-feature pair to query the corresponding index in the category concept library.

[0026] The teacher module maintains a category concept base matrix. Each line These are the feature vectors obtained after the corresponding category passes through the teacher module:

[0027]

[0028] Where c(x) i ) represents sample x iIn the matrix Category index in y i It is a one-hot column vector representing x i Category;

[0029] (2) Generating feature responses using piecewise feature pairs

[0030] The teacher module is configured to know some of the student's parameters; the teacher module accepts selected features from the feature selection submodule. As input, and generate feature responses

[0031]

[0032] w te These are the learning parameters for the teacher module, f t () represents the characteristic response generation process.

[0033] Furthermore, in step S4, the teacher module compares the feature responses with the index pairs to identify and generate semantically rich slices. The specific process is as follows:

[0034] (1) Similarity comparison

[0035] Construct a matrix PS with b rows and N columns, where b is the input batch size and N is the number of patches in each sample. Calculate similarity using cosine similarity:

[0036]

[0037] Among them, Ps ij This represents the feature of the j-th patch of the i-th sample in the input batch. Corresponding category concept patch features The similarity, i.e., similar patch features, It is a characteristic response The j-th element in It is an eigenvector The j-th element;

[0038] (2) Semantic patch optimization

[0039] Further select from each sample Similarity patch features as semantic patches:

[0040]

[0041]

[0042]

[0043] in This represents the new sample after semantic patching optimization. This represents the sample before semantic patching optimization. Indicates the preceding The patch feature with the highest similarity This represents the semantic patch features after selection. This represents the normalized semantic patch features;

[0044] (3) Sample optimization

[0045] Construct a sample similarity vector S, which is a b-dimensional vector, where each element represents the current sample feature in the category concept library. Category representation of current sample features The similarity between them, where b is the batch size of the input samples, and then the first... The features of the most similar samples Sample; After the above optimizations, the teacher module generated a sample with semantic patches. Selected useful samples, i.e., semantically rich slices.

[0046] Furthermore, step S5 involves updating the teacher and student modules using semantically rich slices, and the specific process is as follows:

[0047] (1) Update the category concept library using semantically rich slices.

[0048] With semantic patch Update the category concept library according to the momentum mechanism:

[0049]

[0050] in Represents the category concept library matrix, where α is a hyperparameter that balances the weights of the current samples and the corresponding category concept representations in the maintained category concept library when updating the category concept library;

[0051] (2) Update the shard library using semantically rich shards.

[0052] Furthermore, step S6 involves using the updated student module for image classification, and the specific process is as follows:

[0053] Use semantically rich slices as input data for the student module to predict image labels:

[0054]

[0055] Among them, w s y' represents the learning parameters of the student module, f represents the patch selection process, and y' represents the predicted label.

[0056] Furthermore, in the process of using semantically rich slices as input data for the student module to predict image labels, a contrastive semantic loss is introduced, the specific process of which is as follows:

[0057] The new loss function can be expressed as:

[0058]

[0059] Where β is the weighting factor; It is the original cross-entropy loss. It is the contrastive learning loss; defined as:

[0060]

[0061] Where z i and z j This is the processed image with completed patch selection, Sim(z) i ,z j ) is z i and z j The cosine similarity, γ represents (z i ,z j The threshold for similarity, y i y j These are the labels corresponding to the two contrasting samples in the contrast loss.

[0062] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0063] The interactive semantic perception self-learning framework of this invention includes a teacher module and a student module. The student module is used for visual recognition, and the teacher module is used to provide semantic guidance to the student module. The student module inputs the calculated slice-feature pairs into the teacher module, and the teacher module outputs the calculated semantically rich slices to the student module. By drawing on the two-way communication mechanism of human cognitive structure, the interactive learning method is conducive to interpretable visual recognition. The teacher module includes an encoder, a category concept library, and a similarity comparison submodule. The student module includes a slice library, an encoder, and a feature selection submodule. The encoder can be CNN, ViT, or Swin-Transformer, which has good compatibility and scalability. The similarity comparison submodule performs fine-grained optimization of semantic patches, which can obtain patches with more semantic concept information, which will help improve the interpretability of the framework. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of an interactive semantic awareness self-learning framework provided in this embodiment.

[0065] Figure 2This embodiment provides a flowchart of an interpretable visual recognition method based on an interactive semantic awareness self-learning framework.

[0066] Figure 3 The experimental results of an interpretable visual recognition method based on an interactive semantic awareness self-learning framework provided in this embodiment are shown in the figure. Detailed Implementation

[0067] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0068] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0069] Example 1

[0070] like Figure 1 As shown, the first aspect of this invention proposes an interactive semantic perception self-learning framework, which includes a teacher module and a student module. The student module is used for visual recognition, and the teacher module is used to provide semantic guidance to the student module. The student module inputs the calculated slice-feature pairs into the teacher module, and the teacher module outputs the calculated semantically rich slices to the student module. The teacher module includes a first encoder, a category concept library, and a similarity comparison submodule. The encoder is used to extract features from the image. The category concept library is used to store category concepts. The similarity comparison submodule is used to generate semantically rich slices, i.e., semantic guidance. The student module includes a slice library, a second encoder, and a feature selection submodule. The slice library is used to store image slices for classification. The feature selection submodule is used to select features from the image slice features generated by the encoder.

[0071] More specifically, the encoder is a pre-trained feature extraction network, which can be CNN, ViT, or Swin-Transformer. The second encoder of the student module is the same encoder as the first encoder of the teacher module, with the same network structure, and the teacher module and the student module can share the same encoder.

[0072] The category concept library stores a global concept feature for each image category, i.e., a category concept. Its structure is a dictionary structure, including keys and values. The key is the category, and the value is the global feature vector corresponding to the category.

[0073] like Figure 2 As shown, the second aspect of this invention proposes an interpretable visual recognition method based on an interactive semantic-aware self-learning framework, comprising the following steps:

[0074] S1: After the image is sliced, it is input into the teacher module and the student module respectively, and the teacher module and student module are initialized.

[0075] The input image is segmented into 16×16 pieces, which are then forwarded to the student module and the teacher module. The student module stores the received pieces in the piece library, while the teacher module inputs the received pieces into the encoder for feature extraction and uses the obtained features to update the category concept library.

[0076] S2: The student module performs feature selection on the segmented image to generate segment-feature pairs.

[0077] The student module inputs the slices from the slice library into the encoder in the student module for feature extraction, and then inputs the obtained features into the feature selection submodule to generate slice-feature pairs;

[0078] The interactive semantic-aware self-learning framework is compatible with isotropic and pyramid architectures. Using the SwinTransformer as an example, it employs a pyramid architecture. The SwinTransformer uses a self-attention mechanism computed within a local window. The window is configured to uniformly divide the image in a non-overlapping manner. The process of coarsely selecting image patches from the semantic set using the SwinTransformer involves several steps. First, the attention weight matrix is ​​obtained based on the SwinTransformer's attention mechanism.

[0079]

[0080] Where x i It is the original input sample, w ps f represents the parameter in the Swing Transformer. ps This represents the process of feature selection from the original input sample, where N is the number of segments in the original input sample. This indicates that during the patch selection process, patches are indexed using i and j, and ps stands for patch selection;

[0081] Secondly, identify the top-k elements with the highest attention weight, 1≤k≤N; finally, select features from the input image corresponding to the row and column indices of the top k values ​​in the attention matrix.

[0082] S3: Use shard-feature pairs to query the corresponding index in the teacher module and generate patch-specific feature responses;

[0083] (1) Use the shard-feature pair to query the corresponding index in the category concept library.

[0084] The teacher module maintains a category concept base matrix. Each line These are the feature vectors obtained after the corresponding category passes through the teacher module:

[0085]

[0086] Where c(x) i ) represents sample x i In the matrix Category index in y i It is a one-hot column vector representing x i Category;

[0087] (2) Generating feature responses using piecewise feature pairs

[0088] The teacher module is configured to know some of the student's parameters; the teacher module accepts selected features from the feature selection submodule. As input, and generate feature responses

[0089]

[0090] w te These are the learning parameters for the teacher module, f t () represents the characteristic response generation process.

[0091] S4: The teacher module compares the feature responses with index pairs to identify and generate semantically rich slices;

[0092] The similarity comparison submodule within the teacher module performs fine-grained optimization of semantic patches to obtain patches with more semantic conceptual information, which helps improve the interpretability of the framework. The similarity comparison submodule within the teacher module can also select more important samples for sample optimization.

[0093] (1) Similarity comparison

[0094] Construct a matrix PS with b rows and N columns, where b is the input batch size and N is the number of patches in each sample. Calculate similarity using cosine similarity:

[0095]

[0096] Among them, PS ij This represents the feature of the j-th patch of the i-th sample in the input batch. Corresponding category concept patch features The similarity, i.e., similar patch features, It is a characteristic response The j-th element in It is an eigenvector The j-th element;

[0097] (2) Semantic patch optimization

[0098] Further select from each sample Similarity patch features as semantic patches:

[0099]

[0100]

[0101]

[0102] in This represents the new sample after semantic patching optimization. This represents the sample before semantic patching optimization. Indicates the preceding The patch feature with the highest similarity This represents the selected semantic patch features. This represents the normalized semantic patch features;

[0103] (3) Sample optimization

[0104] The sample optimization process is similar to semantic patching optimization. A sample similarity vector S is constructed, which is a b-dimensional vector, where each element represents the current sample feature in the category concept base. Category representation of current sample features The similarity between them, where b is the batch size of the input samples, and then the first... The features of the most similar samples Sample; After the above optimizations, the teacher module generated a sample with semantic patches. Selected useful samples, i.e., semantically rich slices.

[0105] S5: Update the teacher and student modules using semantically rich slices;

[0106] (1) Update the category concept library using semantically rich slices.

[0107] With semantic patch Update the category concept library according to the momentum mechanism:

[0108]

[0109] in Represents the category concept library matrix, where α is a hyperparameter that balances the weights of the current samples and the corresponding category concept representations in the maintained category concept library when updating the category concept library;

[0110] (2) Update the shard library using semantically rich shards.

[0111] S6: Use the updated student module to classify images.

[0112] More specifically, strong interpretability requires strong discriminative ability as its foundation. Specifically, we introduce contrastive learning loss to achieve better feature extraction from coarse to fine granularity, as follows:

[0113] The new loss function can be expressed as:

[0114]

[0115] Where β is the weighting factor; It is the original cross-entropy loss. It is the contrastive learning loss; defined as:

[0116]

[0117] Where z i and z j This is the processed image with completed patch selection, Sim(z) i ,z j ) is z i and z j The cosine similarity, γ represents (z i ,z j The threshold for similarity, y i y j These are the labels corresponding to the two contrasting samples in the contrast loss.

[0118] Finally, semantically rich slices are used as input data for the student module to predict image labels:

[0119]

[0120] Among them, w s These are the learning parameters for the student module, f s This represents the patch selection process, where y' represents the predicted label. The experimental results are shown in the figure below. Figure 3 As shown.

[0121] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. An interpretable visual recognition method based on an interactive semantic-aware self-learning system, characterized in that, Includes the following steps: S1: After the image is sliced, it is input into the teacher module and student module respectively, and the teacher module and student module are initialized; S2: The student module performs feature selection on the segmented image to generate segment-feature pairs; S3: Use shard-feature pairs to query the corresponding index in the teacher module and generate patch-specific feature responses; S4: The teacher module compares the feature responses with the index pairs to identify and generate semantically rich slices. The specific process is as follows: (1) Similarity comparison Construct a matrix ,have lines and Column, in which This is the input batch size. This represents the number of segments in the original input sample. Cosine similarity is used to calculate the similarity. in, This represents the feature of the j-th patch of the i-th sample in the input batch. Corresponding category concept patch features The similarity, i.e., similar patch features, It is a characteristic response The j-th element in It is an eigenvector The j-th element; (2) Semantic patch optimization Further select from each sample Similarity patch features as semantic patches: in This represents the new sample after semantic patching optimization. The top-value represents the sample before semantic patching optimization. Indicates the preceding The patch feature with the highest similarity This represents the selected semantic patch features. This represents the normalized semantic patch features; (3) Sample optimization Construct sample similarity vector It is a A dimensional vector, where each element represents a feature of the current sample in the category concept library. Category representation of current sample features The similarity between them, of which Enter the batch size of the samples, then select the previous option. The features of the most similar samples Sample; After the above optimizations, the teacher module generated a sample with semantic patches. Selected useful samples, i.e., semantically rich slices; S5: Update the teacher and student modules using semantically rich slices; S6: Use the updated student module to classify images.

2. The interpretable visual recognition method based on an interactive semantic perception self-learning system according to claim 1, characterized in that, Step S1 involves segmenting the image and inputting it into the teacher module and student module respectively, and initializing the teacher module and student module. The specific process is as follows: The input image is segmented into pieces of a preset size and then forwarded to the student module and the teacher module. The student module stores the received pieces in the piece library, and the teacher module inputs the received pieces into the encoder for feature extraction and uses the obtained features to update the category concept library.

3. The interpretable visual recognition method based on an interactive semantic perception self-learning system according to claim 1, characterized in that, Step S2 describes the student module performing feature selection on the segmented image to generate segment-feature pairs. The specific process is as follows: The student module inputs the slices from the slice library into the encoder in the student module for feature extraction, and then inputs the obtained features into the feature selection submodule to generate slice-feature pairs; First, the attention weight matrix is ​​obtained based on the attention mechanism. : in It is the original input sample. Represents the parameters in the encoder. This represents the process of feature selection from the original input sample, where N is the number of segments in the original input sample. This indicates that during the patch selection process, patches are indexed using i and j, and ps stands for patch selection; Secondly, identify the one with the highest attention weight. element, Finally, select the image corresponding to the first element in the attention matrix from the input image. Characteristics of row and column indexes for each value.

4. The interpretable visual recognition method based on an interactive semantic perception self-learning system according to claim 1, characterized in that, Step S3 involves querying the corresponding index in the teacher module using shard-feature pairs and generating a patch-specific feature response. The specific process is as follows: (1) Use the shard-feature pair to query the corresponding index in the category concept library. The teacher module maintains a category concept base matrix. Each line These are the feature vectors obtained after the corresponding category passes through the teacher module: in Indicates sample In the matrix Category index in It is a one-hot column vector, representing Category; (2) Using the fragment-feature pair to generate feature responses The teacher module is configured to know some of the student's parameters; the teacher module accepts selected features from the feature selection submodule. As input, and generate feature responses : These are the learning parameters for the teacher module. () represents the characteristic response generation process.

5. The interpretable visual recognition method based on an interactive semantic perception self-learning system according to claim 1, characterized in that, Step S5 involves updating the teacher and student modules using semantically rich slices. The specific process is as follows: (1) Update the category concept library using semantically rich slices With semantic patch Update the category concept library according to the momentum mechanism: in Represents the category concept library matrix, It is a hyperparameter that balances the weights of the current sample and the corresponding category concept representations in the maintained category concept library when updating the category concept library; (2) Update the shard library using semantically rich shards.

6. The interpretable visual recognition method based on an interactive semantic-aware self-learning system according to claim 1, characterized in that, Step S6 describes using the updated student module to classify images. The specific process is as follows: Use semantically rich slices as input data for the student module to predict image labels: in, These are the learning parameters for the student module. This indicates the patch selection process. Labels indicating predictions.

7. The interpretable visual recognition method based on an interactive semantic perception self-learning system according to claim 6, characterized in that, In the process of using semantically rich slices as input data for the student module to predict image labels, a contrastive semantic loss is introduced, the specific process of which is as follows: The new loss function is expressed as: Where β is the weighting factor; It is the original cross-entropy loss. It is the contrastive learning loss; defined as: in and This is a processed image with completed patch selection. yes and cosine similarity, express The similarity threshold, , is the label corresponding to the two contrasting samples in the contrastive loss, and N is the number of segments in the original input sample.

8. An interactive semantic-aware self-learning system, said system being used in the interpretable visual recognition method based on an interactive semantic-aware self-learning system as described in any one of claims 1 to 7, characterized in that, The interactive semantic-aware self-learning system includes a teacher module and a student module. The student module is used for visual recognition, and the teacher module is used to provide semantic guidance to the student module. The student module inputs the calculated slice-feature pairs into the teacher module, and the teacher module outputs the calculated semantically rich slices to the student module. The teacher module includes a first encoder, a category concept library, and a similarity comparison submodule. The first encoder is used to extract features from the image. The category concept library is used to store category concepts. The similarity comparison submodule is used to generate semantically rich slices, i.e., semantic guidance. The student module includes a slice library, a second encoder, and a feature selection submodule. The image segment library is used to store image segments for classification; the feature selection submodule is used to select features from the image segment features generated by the second encoder.

9. An interactive semantic-aware self-learning system according to claim 8, characterized in that, The second encoder of the student module is the same encoder as the first encoder of the teacher module, with the same network structure. The teacher module and the student module share the same encoder, which is a pre-trained feature extraction network, such as CNN, ViT and Swin-Transformer. The category concept library stores a global concept feature for each image category, i.e., a category concept. Its structure is a dictionary structure, including keys and values. The key is the category, and the value is the global feature vector corresponding to the category.

Images