Semantic interactive meta-learning network and learning method based on patch matching metric

Abstract

The application discloses a semantic interaction meta-learning network and a learning method based on patch matching degree, and the learning network comprises the following: a feature extractor receives a support set image and a query image, and outputs patch embedding; a task-adaptive patch embedding update module receives the patch embedding, and outputs task-adaptive patch embedding of the support set image; a channel semantic interaction module receives the task-adaptive patch embedding of the support set image, and outputs semantic patch embedding of the support set image; a matching degree module receives the semantic patch embedding of the support set image and patch embedding of the query image, calculates patch-level similarity between the query image and the support set image of each category; and a classifier calculates classification probability and classification cross-entropy loss of the query image. The application solves the problem of supervision collapse in the application of the existing meta-learning to the small sample image classification scene and the problem that semantic knowledge is not effectively utilized, and alleviates the problem of low small sample image classification accuracy.

Description

Technical Field

[0001] This invention belongs to the technical field of meta-learning, specifically relating to a semantic interaction meta-learning network and learning method based on patch matching metric. Background Technology

[0002] With the widespread application of large-scale datasets and the rapid development of deep convolutional architectures, supervised learning plays a crucial role in various fields such as computer vision, speech recognition, and machine translation. However, in areas lacking large-scale available data, such as anomaly detection and medical image processing, it is difficult to obtain sufficient training samples to support supervised learning methods, leading to a decline in model performance. In contrast, few-shot learning (FSL) aims to simulate the human ability to learn from limited data, while meta-learning is a method for addressing the few-shot learning problem. Meta-learning extracts general knowledge or patterns by learning on a large number of different tasks, enabling models to quickly adapt and learn when faced with new tasks.

[0003] Metric meta-learning methods learn similarity metrics between samples from a broad task space, thereby learning to distinguish the categories of input images and quickly adapting to few-shot classification tasks without fine-tuning. Prototype networks are a classic metric-based meta-learning architecture used to solve the few-shot learning problem. The core idea of ​​prototype networks is to classify by learning class prototypes. Subsequent methods such as DynamicFSL build upon prototype networks, combining pre-training methods to form a few-shot learning approach that combines pre-training and prototype fine-tuning. However, most existing metric meta-learning methods lack effective utilization of local image information and additional semantic knowledge. First, meta-learning training is often supervised training. Since features useful for the meta-training task may be unhelpful or even harmful to the target task, this leads to the supervision collapse problem in few-shot scenarios—losing information that is unnecessary for training specific tasks but essential for new tasks. Second, although utilizing additional semantic knowledge can help supplement the understanding of new classes, unreliable semantic knowledge sources and unreasonable combinations of semantics and vision can lead to problems in the effective utilization of semantic knowledge, thus impairing few-shot image classification performance. Summary of the Invention

[0004] The purpose of this invention is to provide a semantic interaction meta-learning network and learning method based on patch matching metric to solve the supervision collapse problem and the problem of ineffective utilization of semantic knowledge in existing meta-learning applications in few-sample image classification scenarios, thereby alleviating the problem of low accuracy in few-sample image classification.

[0005] The first technical solution adopted in this invention is a semantic interaction meta-learning network based on patch matching metric, comprising: The feature extractor receives a support set image and a query image, and outputs patch embeddings for the support set image and the query image. The task-adaptive patch embedding update module receives patch embeddings for the support set image, updates node features, and outputs task-adaptive patch embeddings for the support set image. The channel semantic interaction module receives task-adaptive patch embeddings for the supported image set and outputs semantic patch embeddings for the supported image set. The matching metric module receives the semantic patch embeddings of the support set images and the patch embeddings of the query image, and calculates the patch-level similarity between the query image and the support set images for each category. The classifier receives the patch-level similarity and calculates the classification probability and classification cross-entropy loss of the query image.

[0006] The second technical solution adopted in this invention is a semantic interaction meta-learning method based on patch matching metric, characterized by the following steps: Step 1: Pre-train the initial feature extractor parameters using a self-supervised learning method based on mask image modeling, and then share the pre-trained initial feature extractor parameters to the feature extractor in the meta-training stage. Step 2: Input the support set image and the query image into the feature extractor, divide each image into U patch blocks evenly, and finally output the feature descriptor of each patch block, that is, the patch embedding of the support set image and the patch embedding of the query image. Step 3: Construct a patch embedding semantic graph using the patch embeddings of the supported set images as nodes, and input the patch embedding semantic graph into the task-adaptive patch embedding update module to update the node features. The updated node features are the task-adaptive patch embeddings. Step 4: Embed the task-adaptive patch into the input channel semantic interaction module to obtain the semantic patch embedding of the support set image; Step 5: Input the patch embedding of the query image obtained in Step 2 and the semantic patch embedding of the support set image obtained in Step 4 into the matching metric module, and use the matching metric module to calculate the patch-level similarity between the query image and the support set image of each category. Step 6: Based on the patch-level similarity, use a classifier to obtain the classification probability of the query image, calculate the classification cross-entropy loss, and then update the parameters of the semantic interaction meta-learning network based on the patch matching metric.

[0007] Furthermore, the specific process of step 3 is as follows: 3.1) Define the node information of the patch embedding semantic graph: Graph node set One of the tasks involves supporting a set of images. Zhang, the total number of nodes in the corresponding graph is indivual; 3.2) Define the edges of the patch embedding semantic graph: The set of edges connecting nodes is , Constructing an adjacency matrix using set E ; By setting the boundary value range to [0, 2], we obtain the adjacency matrix. The value of each element in the formula is as follows: For adjacency matrix Normalization is performed to obtain the normalized adjacency matrix. ; 3.3) Update the patch embedding semantic graph: The initial features of the graph nodes are embedded using patching that supports a set of images. To represent; the patch embeddings of all images in the support set for a task are concatenated into a support set patch embedding matrix. Each row of the patch embedding matrix corresponds to the initial feature of each node in the patch embedding semantic graph, forming the graph node initial feature matrix. ; After performing B propagations on the support set patch embedding matrix, the final task-adaptive patch embedding matrix is ​​obtained. , Update the steps for the graph convolutional network.

[0008] Furthermore, the specific process of step 4 is as follows: 4.1) The text encoder of the CLIP model is used as the semantic extractor 6, and the input class label is expanded with a text template, which is "A photo of a {class name}". The class label c is used by the semantic extractor 6 to generate a semantic cue feature vector. ; 4.2) Embed each patch of the image supporting set category c. With semantic cue feature vector Features are concatenated along the channel dimension in a serial manner to obtain concatenated features. , ; The concatenated features are input into the two-layer MLP module 7, which adjusts the channel dimensions and enhances the semantics of the visual features to obtain a modulation vector after the interaction of semantic knowledge and visual feature channels. ; 4.3) Add the modulation vector to the corresponding patch embedding to adjust the visual features of each channel, resulting in the final semantic patch embedding as follows: The semantic patch embedding matrix that supports class C is: Each column of the semantic patch embedding matrix corresponds to a semantic patch embedding. .

[0009] Furthermore, the specific process of step 5 is as follows: Calculate the patch-level similarity matrix of the query image with respect to category c according to formula (6). : For a given query image q, the patch embedding matrix, To support semantic patch embedding matrices of class C; Patch-level similarity matrix The element in the m-th row and n-th column is represented as This reflects the distance between the m-th patch embedding of the query image and the n-th patch embedding of the support set category c; the patch-level similarity matrix. One row in the table represents the semantic similarity between the corresponding patch embedding in the query image and all patch embeddings in the supporting class; For the m-th patch embedding of the query image q, from the patch embedding matrix Find the L most similar patch embeddings in the m-th row of the query image q, and sum the similarity values ​​of these L patch embeddings with the patch embeddings of the query image q to obtain the similarity between the m-th patch embedding of the query image q and the supporting class c. ; Finally, the patch-level similarity between the query image q and the supporting class c is calculated using formula (7). : The patch-level similarity between the query image q and the supporting class c is obtained by summing the similarity of all patch blocks of the query image q.

[0010] The third technical solution adopted in this invention: an electronic device, including a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program to implement a semantic interaction meta-learning method based on patch matching metrics.

[0011] The third technical solution adopted in this invention is: a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements a semantic interaction meta-learning method based on patch matching metric.

[0012] The beneficial effects of this invention are as follows: This invention employs two measures to mitigate the supervised learning collapse problem, enabling effective utilization of local and contextual information in images. Firstly, it uses self-supervised training instead of supervised pre-training, which is more conducive to learning general knowledge in images. Secondly, it proposes a patch matching metric strategy that no longer uses category prototypes as metric references, avoiding the loss of local information. Furthermore, this invention uses a class-label-assisted channel semantic interaction strategy and a more advanced semantic extractor to optimize the semantic knowledge source and the combination of semantic knowledge with visual features. Ultimately, this improves the generalization ability of the meta-learning network and increases the classification accuracy of small sample images.

[0013] Unlike most meta-metric learning methods that use overall image features for matching and measurement, directly using class embeddings as encoded image features of the input image and then inputting these features into the classification layer for prediction, this invention fully utilizes patch embeddings output by the Transformer series of networks, while introducing semantic knowledge assisted by class labels. A self-supervised method based on masked image reconstruction is used for pre-training to obtain a more generalized feature extractor; to address the supervision collapse caused by local feature overload, an image local meta-metric learning method based on patch embeddings is constructed; the channel semantic interaction module further utilizes semantic knowledge to supplement visual features and designs a channel semantic interaction strategy adapted to patch embeddings, guiding visual features to be more discriminative. Self-supervised pre-training and patch matching metrics complement each other. The full utilization of patch blocks makes patch matching metrics more advantageous than general global-level metrics, and the channel semantic interaction strategy achieves the fusion and interaction of semantic knowledge and visual features in a more robust form, ultimately improving the performance of the trained meta-learning model on few-sample image classification tasks. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the architecture of the semantic interaction meta-learning network based on patch matching metric of the present invention; Figure 2 This is a schematic diagram of the architecture of the task-adaptive patch embedding update module of the present invention; Figure 3 This is a schematic diagram of the architecture of the channel semantic interaction module of the present invention.

[0015] In the diagram, 1. Feature extractor, 2. Task-adaptive patch embedding update module, 3. Channel semantic interaction module, 4. Matching metric module, 5. Classifier, 6. Semantic extractor, 7. Two-layer MLP module. Detailed Implementation

[0016] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0017] This invention provides a semantic interaction meta-learning network based on patch matching metrics, such as... Figure 1 As shown, it includes a feature extractor 1, a task-adaptive patch embedding update module 2, a channel semantic interaction module 3, a matching metric module 4, and a classifier 5.

[0018] Feature extractor 1 receives a support set image and a query image, and outputs patch embeddings of the support set image and the query image. Task-adaptive patch embedding update module 2 receives patch embeddings of the support set image, updates node features, and outputs task-adaptive patch embeddings of the support set image. Channel semantic interaction module 3 receives task-adaptive patch embeddings for supporting image sets and outputs semantic patch embeddings for supporting image sets. The matching metric module 4 receives the semantic patch embeddings of the support set images and the patch embeddings of the query image, and calculates the patch-level similarity between the query image and the support set images for each category. Classifier 5 receives the patch-level similarity and calculates the classification probability and classification cross-entropy loss of the query image.

[0019] Figure 1 This paper presents the framework of PatSiML (Patch Matchingmetric Based Semantic Interaction Meta-Learning), a method based on patch matching metrics. The implementation of the PatSiML algorithm on a 3-way 2-shot few-sample image classification task is simulated, assuming that each image is extracted into 9 patch embeddings by feature extractor 1. The PatSiML algorithm generally follows a pre-training, meta-training, and meta-testing process.

[0020] In the pre-training phase, unlike general meta-learning algorithms, the PatSiML algorithm uses a self-supervised learning method based on mask image modeling to train the feature extractor. The entire network is divided into two paths: one path uses a teacher network acting as a labeler to obtain augmented features, and the other path uses a student network with the feature extractor as the backbone to perform mask image modeling on the mask image. The training objective is to minimize the distillation loss between augmented and reconstructed features. The weights of the feature extractor after self-supervised pre-training are retained and imported into the feature extractor in the meta-training phase.

[0021] During the meta-training phase, the support set samples are processed by a feature extractor to obtain all patch embeddings for each category. A patch embedding semantic graph is constructed using the patch embeddings of the support set images as nodes, and the node features are updated through a graph convolutional network to obtain task-adaptive patch embeddings. The task-adaptive patch embeddings are then input to a channel semantic interaction module, where semantic cue features of the category labels are fused with visual features in the channel dimension to provide semantic supplementation and guidance. Next, a matching metric module calculates the patch-level similarity between the query set image and each category in the support set, thus achieving a patch-level matching metric for the images. Finally, a classifier is used to predict the results and calculate the classification cross-entropy loss to update the entire network model.

[0022] The meta-testing phase corresponds to the implementation process of the downstream few-sample image classification task. The process is similar to the meta-training phase, requiring no network updates; the classifier directly outputs the prediction results.

[0023] like Figure 2 The schematic diagram of the task-adaptive patch embedding update module shows that when updating the patch embedding semantic graph, the normalized adjacency matrix and initial node features are input into the task-adaptive patch embedding update module 2. The main network of the task-adaptive patch embedding update module 2 consists of two graph convolutional layers (GCN1, GCN2) and an intermediate activation function and dropout layer. The initial features are propagated twice through the two graph convolutional layers, and the updated features are then added to the initial features to obtain the output features. In this task-adaptive patch embedding update module 2, the patch embedding is updated specifically for the task based on the patch embedding semantic graph, while maintaining the graph edge information unchanged.

[0024] like Figure 3 The architecture diagram of the channel semantic interaction module is shown below: First, the class label generates a semantic cue feature vector through the semantic extractor 6. Next, the semantic cue feature vector is concatenated with the visual context features extracted from all patch embeddings along the channel dimension, and the extracted semantic cue feature vector is added to each patch embedding. Then, the concatenated features are input into the two-layer MLP module 7, which adjusts the visual features along the channel dimension and enhances their semantics. Finally, the modulated features are added to the patch embedding without semantic modulation to obtain the final semantic patch embedding. Through the class label-assisted channel semantic interaction mechanism, semantic knowledge and visual features can be fused and interacted. By adjusting the visual features channel by channel through the text information in the class label, the discriminative power of the features is improved, promoting the task adaptability of the meta-learning model and thus improving the performance of few-sample image classification.

[0025] This invention also provides a semantic interaction meta-learning method based on patch matching metrics. The method, based on a semantic interaction meta-learning network using patch matching metrics, includes the following steps: Step 1: Pre-train the initial feature extractor parameters using a self-supervised learning method based on mask image modeling, and then share the pre-trained initial feature extractor parameters to the feature extractor in the meta-training stage.

[0026] The core of both the target network and the feature extractor shares the same network structure, employing the Transformer series of networks. Masked Image Modeling (MIM) involves randomly masking regions of the image to be pre-trained. The target network is then required to reconstruct the regional features of the original image based on the masked image, thus achieving pre-training of the image data. This invention utilizes the iBOT algorithm.

[0027] Step 2: Input the support set image and query image into the feature extractor. Divide each image evenly into U patch blocks, and finally output the feature descriptor for each patch block, i.e., the patch embedding of the support set image and the patch embedding of the query image. Transformer series networks generally output two parts: category embedding and patch embedding. Here, we only use the patch embedding.

[0028] Step 3: Construct a patch embedding semantic graph using the patch embeddings of the supported set images as nodes, and input the patch embedding semantic graph into the task-adaptive patch embedding update module 2 to update the node features. The updated node features are the task-adaptive patch embeddings.

[0029] Step 4: Embed the task-adaptive patch into the input channel semantic interaction module 3. The semantic prompt features of the category label are fused and interacted with the visual features in the channel dimension to supplement and guide semantics, thereby obtaining the semantic patch embedding of the support set image.

[0030] Step 5: Input the patch embedding of the query image obtained in Step 2 and the semantic patch embedding of the support set image obtained in Step 4 into the matching metric module 4, and calculate the patch-level similarity between the query image and the support set image of each category through the matching metric module 4.

[0031] Step 6: Based on the patch-level similarity, classifier 5 is used to obtain the classification probability of the query image, and the classification cross-entropy loss is calculated. Then, the parameters of the semantic interaction meta-learning network based on the patch matching metric are updated. The network parameters include feature extractor parameters, task-adaptive patch embedding update parameters, and channel semantic interaction module parameters.

[0032] In some embodiments, step 3 is performed as follows: 3.1) Define the node information of the patch embedding semantic graph: Suppose the features of an image are composed of U patch embeddings. A patch embedding of a support set image is treated as a node in the patch embedding semantic graph. A task contains support set images... Zhang, the total number of nodes in the corresponding graph is If there are , then there is a graph node set. .

[0033] 3.2) Define the edges of the patch embedding semantic graph: The set of edges connecting nodes is The boundary value between two nodes The cosine similarity is used for calculation, and the specific formula is as follows: In the formula: , — Representing nodes respectively Category, node Category; —True / false indicator function, if If true, output 1; otherwise, output 0.

[0034] Constructing an adjacency matrix using set E ; Since the node edge values ​​constructed by formula (1) range from [-1, 1], to avoid network training errors caused by negative values ​​and to ensure the stability and convergence of the network, the edge value range is set to [0, 2], resulting in the adjacency matrix. The value of each element in the formula is as follows: To eliminate the influence of node degree (i.e., the number of neighboring nodes) on feature propagation, ensure that the degree of each node does not introduce numerical bias during feature propagation, and guarantee the stability of the numerical range, it is necessary to adjust the adjacency matrix. Normalization is performed using random and symmetric normalization methods to obtain a normalized adjacency matrix. .

[0035] 3.3) Update the patch embedding semantic graph: The initial features of the graph nodes are embedded using patch embedding that supports a set of images. To represent; the patch embeddings of all images in the support set for a task are concatenated into a support set patch embedding matrix. Each row of the patch embedding matrix corresponds to the initial feature of each node in the patch embedding semantic graph, forming the graph node initial feature matrix. ; The patch embedding is propagated in the graph based on the following formula. After B propagations of the support set patch embedding matrix, the final task-adaptive patch embedding matrix is ​​obtained. , In the formula: To update the step count for graph convolutional networks, this invention sets... ; Embed update parameters for task-adaptive patches. For activation function, For the first The patch embedding matrix after the layer graph convolutional layer is updated.

[0036] In some embodiments, such as Figure 3 As shown, the specific process of step 4 is as follows: 4.1) The text encoder of the CLIP model is used as the semantic extractor 6, and the input class label is expanded with a text template, which is "A photo of a {class name}". The class label c is used by the semantic extractor 6 to generate a semantic cue feature vector. ; 4.2) Embed each patch of the image supporting set category c. With semantic cue feature vector Features are concatenated along the channel dimension in a serial manner to obtain concatenated features. , ; The concatenated features are input into the two-layer MLP module 7, which adjusts the channel dimensions and enhances the semantics of the visual features to obtain a modulation vector after the interaction of semantic knowledge and visual feature channels. ; 4.3) Add the modulation vector to the corresponding patch embedding to adjust the visual features of each channel, resulting in the final semantic patch embedding as follows: The semantic patch embedding matrix that supports class C is: Each column of the semantic patch embedding matrix corresponds to a semantic patch embedding. .

[0037] In some embodiments, step 5 is performed as follows: Given the patch embedding matrix of the query image q Supports semantic patch embedding matrices of class C. Then we can obtain the patch-level similarity matrix of the query image with respect to category c. The calculation formula is as follows: Patch-level similarity matrix The element in the m-th row and n-th column is represented as This reflects the distance between the m-th patch embedding of the query image and the n-th patch embedding of the support set class c. A row in the patch-level similarity matrix represents the semantic similarity between the corresponding patch embedding in the query image and all patch embeddings in the support class.

[0038] For the m-th patch embedding of the query image q, from the patch embedding matrix Find the L most similar patch embeddings in the m-th row of the query image q, and sum the similarity values ​​of these L patch embeddings with the patch embeddings of the query image q to obtain the similarity between the m-th patch embedding of the query image q and the supporting class c. Finally, the patch-level similarity between the query image q and the supporting class c is calculated. The formula is as follows: According to formula (7), the patch-level similarity between the query image q and the supporting class c is obtained by summing the similarities of all patch blocks of the query image. The patch-level similarity represents the distance between the query image and the supporting class. The greater the distance, the higher the similarity, and the more likely the query image is to be predicted as the supporting class.

[0039] Example The proposed semantic interaction meta-learning method based on patch matching metric (PatSiML) is experimentally compared with current state-of-the-art and classic techniques in the field of few-sample image classification.

[0040] DynamicFSL is a classic metric learning method. DeepEMD-Bert uses the BERT language model to learn class name embeddings, and is a metric learning method that utilizes semantic knowledge. The prototype network (Swin) replaces the backbone of the classic prototype network method with Swin-Tiny, and undergoes the same self-supervised pre-training process as the meta-learning method of this invention. FewTURE, like this invention, uses the iBOT algorithm for self-supervised pre-training, and is a method for matching metrics using local image patch information.

[0041] Small sample sizes were set as 5way-1shot (5W1S) and 5way-5shot (5W5S), using the Mini-ImageNet dataset. Two different Transformer backbone network architectures were used on the PatSiML algorithm: Vit-Small and Swin-Tiny as the backbone networks. Table 1 shows the experimental results on the Mini-ImageNet dataset. Bold figures represent the optimal algorithm. The number of parameters refers to the space complexity of the feature extractor. The values ​​in the 5W1S and 5W5S columns are the accuracy and 95% confidence interval, expressed as percentages.

[0042] As shown in the table, the PatSiML algorithm exhibits optimal performance on both 1-shot and 5-shot tasks. The PatSiML-Swin algorithm of this invention outperforms the state-of-the-art algorithm FewTURE, which uses the same self-supervised pre-training, by 1.48%. Under the same backbone network Swin-Tiny, PatSiML-Swin achieves higher accuracy than ProtoNet-Swin with its global metric, and also surpasses FewTURE, which uses a local metric, demonstrating the superiority of the patch-level matching metric strategy used in this invention.

[0043] Table 1. Experimental results of each algorithm on the Mini-ImageNet dataset.

Claims

1. A semantic interaction meta-learning method based on patch matching metric, characterized in that, Includes the following steps: Step 1: Use the self-supervised learning method based on mask image modeling to pre-train the initial feature extractor parameters, and then share the pre-trained initial feature extractor parameters to the feature extractor in the meta-training stage (1). Step 2: Input the support set image and the query image into the feature extractor (1), divide each image into U patch blocks evenly, and finally output the feature descriptor of each patch block, that is, the patch embedding of the support set image and the patch embedding of the query image. Step 3: Construct a patch embedding semantic graph using the patch embedding of the supported set image as nodes, and input the patch embedding semantic graph into the task-adaptive patch embedding update module (2) to update the node features. The updated node features are the task-adaptive patch embedding. Among them, the main network of the task-adaptive patch embedding update module (2) consists of two graph convolutional layers GCN1 and GCN2, as well as the activation function and Dropout layer in between; Step 4: Embed the task-adaptive patch into the input channel semantic interaction module (3) to obtain the semantic patch embedding of the support set image; The architecture of the channel semantic interaction module (3) includes: First, the class label generates a semantic cue feature vector through the semantic extractor (6); then, the semantic cue feature vector is concatenated with the visual context features extracted from all patch embeddings in the channel dimension, and the extracted semantic cue feature vector is added to each patch embedding; then the concatenated features are input into the two-layer MLP module (7), and the two-layer MLP module (7) adjusts the visual features in the channel dimension and enhances the semantics; finally, the modulation features are added to the patch embedding without semantic modulation to obtain the final semantic patch embedding. Step 5: Input the patch embedding of the query image obtained in Step 2 and the semantic patch embedding of the support set image obtained in Step 4 into the matching metric module (4), and calculate the patch-level similarity between the query image and the support set image of each category through the matching metric module (4); Step 6: Based on the patch-level similarity, use classifier (5) to obtain the classification probability of the query image, calculate the classification cross-entropy loss, and then update the parameters of the semantic interaction meta-learning network based on patch matching metric.

2. The semantic interaction meta-learning method based on patch matching metric as described in claim 1, characterized in that, The specific process of step 3 is as follows: 3.1) Define the node information of the patch embedding semantic graph: Graph node set One of the tasks involves supporting a set of images. Zhang, the total number of nodes in the corresponding graph is indivual; 3.2) Define the edges of the patch embedding semantic graph: The set of edges connecting nodes is The boundary value between two nodes; Constructing an adjacency matrix using set E By setting the boundary value range to [0, 2], we obtain the adjacency matrix. The value of each element in the formula is as follows: For adjacency matrix Normalization is performed to obtain the normalized adjacency matrix. ; 3.3) Update the patch embedding semantic graph: The initial features of the graph nodes are embedded using patching that supports a set of images. To represent; the patch embeddings of all images in the support set for a task are concatenated into a support set patch embedding matrix. Each row of the patch embedding matrix corresponds to the initial feature of each node in the patch embedding semantic graph, forming the graph node initial feature matrix. ; After performing B propagations on the support set patch embedding matrix, the final task-adaptive patch embedding matrix is ​​obtained. , Update the steps for the graph convolutional network.

3. The semantic interaction meta-learning method based on patch matching metric as described in claim 2, characterized in that, The specific process of step 4 is as follows: 4.1) The text encoder of the CLIP model is used as the semantic extractor 6, and the input class label is expanded with a text template, which is "A photo of a {class name}". The class label c is used by the semantic extractor 6 to generate a semantic cue feature vector. ; 4.2) Embed each patch of the image supporting set category c. With semantic cue feature vector Features are concatenated along the channel dimension in a serial manner to obtain concatenated features. , ; The concatenated features are input into the two-layer MLP module 7, which adjusts the channel dimensions and enhances the semantics of the visual features to obtain a modulation vector after the interaction of semantic knowledge and visual feature channels. ; 4.3) Add the modulation vector to the corresponding patch embedding to adjust the visual features of each channel, resulting in the final semantic patch embedding as follows: The semantic patch embedding matrix that supports class C is: Each column of the semantic patch embedding matrix corresponds to a semantic patch embedding. .

4. The semantic interaction meta-learning method based on patch matching metric as described in claim 3, characterized in that, The specific process of step 5 is as follows: Calculate the patch-level similarity matrix of the query image with respect to category c according to formula (6). : For a given query image q, the patch embedding matrix, To support semantic patch embedding matrices of class C; Patch-level similarity matrix The element in the m-th row and n-th column is represented as This reflects the distance between the m-th patch embedding of the query image and the n-th patch embedding of the support set category c; the patch-level similarity matrix. One row in the table represents the semantic similarity between the corresponding patch embedding in the query image and all patch embeddings in the supporting class; For the m-th patch embedding of the query image q, from the patch embedding matrix Find the L most similar patch embeddings in the m-th row of the query image q, and sum the similarity values ​​of these L patch embeddings with the patch embeddings of the query image q to obtain the similarity between the m-th patch embedding of the query image q and the supporting class c. ; Finally, the patch-level similarity between the query image q and the supporting class c is calculated using formula (7). : The patch-level similarity between the query image q and the supporting class c is obtained by summing the similarity of all patch blocks of the query image q.

5. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is used to execute the computer program to implement the semantic interaction meta-learning method based on patch matching metric as described in any one of claims 1-4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the semantic interaction meta-learning method based on patch matching metric as described in any one of claims 1-4.

Images