Small-Sample Object Recognition Method Based on Two-Stage Causal Intervention

Abstract

The present invention discloses a few-shot object recognition method based on two-stage causal intervention, including: Step 1, constructing and using a causal intervention method to pre-train a few-shot object recognition deep network model; Step 2, using the trained object recognition deep network model to extract confounding factors and object features; Step 3, causally intervening on the object features of the support set; Step 4, calculating and correcting the distance matrix and optimizing the network model; Step 5, obtaining the final few-shot object recognition network model. The present invention optimizes the model pre-training process of the transfer-based few-shot object recognition method by using causal intervention, weakens the spurious association between the confounding factors and the object features, and fully considers the alignment relationship between the confounding factors and the object features under different distributions, performs counterfactual on the object features of the base domain, facilitates the distance measurement of object features in different domains, and improves the classification accuracy and robustness of object recognition under few-shot conditions.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent interpretation of remote sensing images, and particularly relates to a small-sample target recognition method based on two-stage causal intervention. Background Art

[0002] With the rapid development of machine learning technology, convolutional neural networks (CNNs) have become extremely popular and indispensable architectures in target classification, outperforming traditional methods by a large margin. Their data-driven mechanism avoids the cumbersome and complex design of feature extractors. In addition, methods such as local connection, weight sharing, pooling, and multi-layer stacking have promoted deeper networks and hierarchical feature representations, achieving significant progress in solving target feature extraction and classification problems.

[0003] However, training deep CNNs requires a large number of high-quality training samples to prevent the model from overfitting during inference. In practice, the collection of image data and sample labeling can be quite expensive and difficult, making it difficult for researchers to obtain large-scale training samples. Some target categories may have only a few or dozens of labeled samples. In this case, the performance of deep CNNs drops significantly, and the problem of small-sample target recognition emerges.

[0004] Generally, small-sample target recognition methods can be divided into two categories: meta-learning methods and transfer learning methods. Meta-learning methods include two stages: meta-training and meta-testing. In either stage, training samples are combined into a batch in the form of tasks. The model is trained on the support set and then inferred on the query set. Meta-learning methods can be further subdivided into optimization-based meta-learning and metric-based meta-learning. Optimization learning methods aim to obtain a model with strong generalization ability through the meta-training stage, use a small number of labeled support set samples for model fine-tuning in the meta-testing stage, and then classify the unlabeled query set samples. Metric learning methods map samples into a high-dimensional space and classify them by measuring the distances between the feature vectors of different samples in the high-dimensional space using common distance metric formulas or learnable distance metric methods. Metric learning methods are more effective than optimization learning methods in target recognition. However, for image target recognition, factors such as strong background scattering interference lead to large variations in the features of the same type of target. In this case, it is difficult to generalize the features of the training set to the test set, and the representation vectors of the same category in the feature space exhibit different structures. Metric learning methods rely heavily on the similarity between unlabeled samples and labeled samples, and their performance drops severely with inaccurate representation of sample relationships.

[0005] Recently, some transfer learning methods with small samples have achieved more accurate classification results than complex meta-learning algorithms. Transfer learning first performs multi-class pre-training on base class data, and then fine-tunes on scarce new class data. Some methods add contrastive learning and self-supervised distillation to the pre-training stage to obtain better pre-trained models, and their classification performance surpasses a large number of metric-based meta-learning methods.

[0006] However, these pre-training methods ignore which features the model should learn and which features it should not learn during the pre-training process. Therefore, when the new domain distribution is very different from the base domain distribution, the pre-trained model is affected by the false associations established by the base domain confusion factors, and the recognition effect is seriously reduced under the influence of the new domain confusion factors. These methods have not yet effectively eliminated the false associations caused by the confusion factors, and improving the robustness of small sample target recognition is an important issue that needs to be solved urgently. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a small sample target recognition method based on two-stage causal intervention in response to the deficiencies in the above-mentioned prior art. The method has a simple structure, a reasonable design, and optimizes the small sample target recognition model. The present invention utilizes causal intervention to optimize the model pre-training process of the transfer-based small sample target recognition method, weakens the false correlation between the confusion factor and the target feature, and fully considers the alignment relationship between the confusion factor and the target feature under different distributions. The base domain target features are counterfactually measured, which is convenient for the distance measurement of target features in different domains, and improves the classification accuracy and robustness of target recognition under small sample conditions.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is: a small sample target recognition method based on two-stage causal intervention, characterized in that it includes the following steps:

[0009] Step 1: Build a deep network model for image target recognition and use the basic category dataset to pre-train the network model;

[0010] Step 101, using variational backdoor correction to change the objective function of model training;

[0011] Step 102: Obtain data input and model the confusion factor C in the data;

[0012] Step 103: Use pseudo-variational posterior probability to approximate the distribution of the background

[0013] Step 104: Based on the data input and the confounding factor, use the probability distribution after the causal intervention Train and optimize deep network models;

[0014] Step 2: Use the pre-trained object recognition deep network to extract the confounding factors and object features of the support set and the query set respectively. The expression of the confounding factor features of the support set and the query set is The expression of the object feature is The expression of the object feature corrected by the confounding factor is

[0015] Step 3: Conduct causal intervention on the object features of the support set data:

[0016] Step 301: Obtain the confounding factors C s and C q of the support set and the query set from Step 2, and generate a new confounding factor C q ′ based on the confounding factor of the support set;

[0017] Step 302: Conduct counterfactual on the object features of the support set using the new confounding factor C q ′, which can be expressed as c i ~C q ′;

[0018] Step 4: Calculate and correct the distance matrix, optimize the network model after obtaining the classification result:

[0019] Step 401: Calculate the N class prototypes after counterfactual where M represents that there are M feature vectors in the i-th class. Use the cosine distance to measure the distance between the N class prototypes cls i and the K object features F of the query set. Here, <·,·> represents the vector inner product, and a distance matrix D1 with dimensions N×K can be obtained;

[0020] Step 402: Calculate the N class prototypes without counterfactual where M represents that there are M feature vectors in the i-th class. Use the cosine distance to measure the distance between the N class prototypes cls i and the object features F c of the K query sets without being corrected by the confounding factor. Here, ·,·> represents the vector inner product, and a distance matrix D2 with dimensions N×K can be obtained;

[0021] Step 403: Obtain the finally corrected distance matrix D out =λD1+(1 - λ)D2, and then obtain the object recognition result of the query set; out

[0022] Step 404: Calculate the cross-entropy loss according to the object recognition result, and update and optimize the parameters of the deep network model.

[0023] Step Five: Obtain the final small-sample target recognition network model.

[0024] The above-mentioned small-sample target recognition method based on two-stage causal intervention is characterized in that the specific method of step 101 includes:

[0025] Step 1011: Construct a causal graph as: C→X, C→Y, X→Y, where C represents a confounding factor, X represents the input image, and Y represents the prediction result of the image;

[0026] Step 1012: Cut off the association of C→X through the do operator, and obtain the model output after causal intervention through the backdoor correction formula where

[0027] Step 1013: Use variational inference to introduce a new distribution Q(C|X) as the probability estimate of the latent variable when a given image is input, and the objective function is changed to the evidence lower bound of the model output, that is

[0028] The above-mentioned small-sample target recognition method based on two-stage causal intervention is characterized in that the specific method of step 102 includes:

[0029] Step 1021: Randomly initialize the feature mapping matrix W i and the scaling matrix a i ;

[0030] Step 1022: Calculate the association score s i between the input image X and the confounding factor c i = a i ·Tanh(W i ·X);

[0031] Step 1023: Obtain

[0032] The above-mentioned small-sample target recognition method based on two-stage causal intervention is characterized in that the specific method of step 103 includes:

[0033] Step 1031: Randomly generate R pseudo-image samples X′;

[0034] Step 1032: Repeat step 1022 and step 1023 to obtain The approximate estimation formula of the background distribution is

[0035] The above-mentioned small-sample target recognition method based on two-stage causal intervention is characterized in that the expression of the model output after causal intervention in step 104 is where φ irepresents the encoder, θ i represents the parameters of the encoder, c i Obtained from Q(C|X) in step 102.

[0036] The above-mentioned small sample target recognition method based on two-stage causal intervention is characterized in that: the specific method of generating the confusion factor in step 301 includes:

[0037] Step 3011: Set the confusion factor generation network g(·) and initialize its parameters;

[0038] Step 3012: Input support set confusion factor C s Get the newly generated confusion factor C q ′=g(C s );

[0039] Step 3013, calculate C q With C q The KL divergence D of KL {C q ||C q ′} to optimize the parameters of the confusion factor generation network.

[0040] Compared with the prior art, the present invention has the following advantages:

[0041] 1. The invention has a simple structure, reasonable design, and is easy to implement and operate.

[0042] 2. The present invention utilizes causal intervention to change the objective function of the pre-training stage, weakens the spurious correlation between the confusion factor and the target feature, and obtains an optimized target recognition deep network model, thereby preliminarily improving the overall robustness of the optimized target recognition deep network model to small sample data sets under different distributions, and providing a basis for the counterfactuals of the target features.

[0043] 3. The present invention fully considers the alignment relationship between the confusion factor and the target feature under different distributions, generates the confusion factor of the new domain d2 based on the confusion factor under the base domain d1, performs counterfactuals on the target features of the base domain, facilitates the distance measurement of target features in different domains, and improves the classification accuracy and robustness of target recognition under small sample conditions.

[0044] In summary, the present invention has a simple structure and a reasonable design, optimizes the model pre-training process of the transfer-type small sample target recognition method, weakens the false correlation between the confusion factor and the target feature, and fully considers the alignment relationship between the confusion factor and the target feature under different distributions, performs counterfactuals on the base domain target feature, facilitates the distance measurement of target features in different domains, and improves the classification accuracy and robustness of target recognition under small sample conditions. The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is the flowchart of the method of the present invention. Detailed implementation manners

[0046] The method of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments of the present invention.

[0047] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0048] It should be noted that the terms used herein are only for describing specific implementation manners and are not intended to limit the exemplary implementation manners according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they specify the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0049] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the implementation manners of the present application described herein can be implemented in an order different from those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0050] For the sake of description, spatial relative terms such as "above...", "over...", "on the upper surface of...", "above" etc. may be used herein to describe the spatial positional relationship between a device or feature shown in the figure and other devices or features. It should be understood that the spatial relative terms are intended to include different orientations in use or operation in addition to the orientation of the device shown in the figure. For example, if the device in the figure is inverted, the device described as "above other devices or structures" or "over other devices or structures" will then be positioned as "below other devices or structures" or "under other devices or structures". Thus, the exemplary term "above..." may include both the orientations of "above..." and "below...". The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and corresponding explanations will be made for the spatial relative descriptions used herein.

[0051] As Figure 1 shown, the small - sample object recognition method based on two - stage causal intervention of the present invention includes the following steps:

[0052] Step 1: Construct an image object recognition deep network model and pre - train the network model using a basic category dataset;

[0053] Step 101: Use variational backdoor correction to change the objective function of model training;

[0054] Step 102: Obtain data input and model the confounding factor C in the data;

[0055] Step 103: Use the pseudo - variational posterior probability to approximate the distribution of the background

[0056] Step 104: According to the data input and the confounding factor, use the probability distribution after causal intervention to train and optimize the deep network model;

[0057] Step 2: Use the pre - trained object recognition deep network to extract the confounding factors and object features of the support set and the query set respectively: The expression of the confounding factor features of the support set and the query set is The expression of the object feature is The expression of the object feature corrected by the confounding factor is

[0058] Step 3: Conduct causal intervention on the object features of the support set data:

[0059] Step 301: Obtain the confounding factors C s and C q of the support set and the query set from Step 2, and generate a new confounding factor C q ′ based on the confounding factor of the support set;

[0060] Step 302: Use the new confounding factor C q ′ to conduct counterfactual on the object features of the support set, which can be expressed as c i ~C q ′;

[0061] Step 4: Calculate and correct the distance matrix, and optimize the network model after obtaining the classification result:

[0062] Step 401: Calculate the N class prototypes after counterfactual where M represents that there are M feature vectors in the i - th class, and use the cosine distance to measure the N class prototypes cls iThe distance between the target feature F of K query sets, where <·,·> represents the vector inner product, and a distance matrix D1 of dimension N×K can be obtained;

[0063] Step 402, calculate the N class prototypes without counterfactuals where M represents that there are M feature vectors in the i-th class, and use the cosine distance to measure the N class prototypes cls i and the target feature F of the K query sets without being corrected by the confounding factor c The distance between them, where <·,·> represents the vector inner product, and a distance matrix D2 of dimension N×K can be obtained;

[0064] Step 403, according to the formula D out =λD1+(1 - λ)D2 to obtain the finally corrected distance matrix D out , and then obtain the target recognition result of the query set;

[0065] Step 404, calculate the cross-entropy loss according to the target recognition result, and update and optimize the parameters of the deep network model.

[0066] Step Five: Obtain the final few-shot target recognition network model.

[0067] The above few-shot target recognition method based on two-stage causal intervention is characterized in that: the specific method of step 101 includes:

[0068] Step 1011, construct a causal graph as: C→X, C→Y, X→Y, where C represents the confounding factor, X represents the input picture, and Y represents the prediction result of the picture;

[0069] Step 1012, cut off the association of C→X through the do operator, and obtain the model output after causal intervention through the backdoor correction formula where

[0070] Step 1013, use variational inference to introduce a new distribution Q(C|X) as the probability estimate of the latent variable when a given picture is input, and the objective function is changed to the evidence lower bound of the model output, that is

[0071] The above few-shot target recognition method based on two-stage causal intervention is characterized in that: the specific method of step 102 includes:

[0072] Step 1021, randomly initialize the feature mapping matrix W i and the scaling matrix a i ;

[0073] Step 1022, calculate the input picture X and the confounding factor ci The associated score s i = a i · Tanh(W i · X);

[0074] Step 1023. Obtain according to the associated score

[0075] The above-mentioned small-sample target recognition method based on two-stage causal intervention is characterized in that: the specific method of step 103 includes:

[0076] Step 1031. Randomly generate R pseudo-image samples X';

[0077] Step 1032. Repeat step 1022 and step 1023 to obtain The approximate estimation formula of the background distribution is

[0078] The above-mentioned small-sample target recognition method based on two-stage causal intervention is characterized in that: the model output expression after causal intervention in step 104 is where φ i represents the encoder, θ i represents the parameters of the encoder, and c i is obtained from Q(C|X) in step 102.

[0079] The above-mentioned small-sample target recognition method based on two-stage causal intervention is characterized in that: the specific method for generating the confounding factor in step 301 includes:

[0080] Step 3011. Set the confounding factor generation network g(·) and initialize its parameters;

[0081] Step 3012. Input the support set confounding factor C s to obtain the newly generated confounding factor C q ' = g(C s );

[0082] Step 3013. Calculate the KL divergence D q between C q and C KL '{C q ||C q '} to optimize the parameters of the confounding factor generation network.

[0083] The above are only the embodiments of the present invention, and do not impose any limitations on the present invention. Any simple modifications, changes, and equivalent structural changes made to the above embodiments according to the technical essence of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A small-sample target recognition method based on two-stage causal intervention, characterized in that: It includes the following steps: Step 1: Construct a deep network model for image target recognition and pre-train the network model using a basic category dataset; Step 101. Use variational backdoor correction to change the objective function of model training: Construct a causal graph, C→X, C→Y, X→Y, where C represents the confounding factor, X represents the input image, and Y represents the prediction result of the image; Cut off the association of C→X through the do operator, and obtain the model output after causal intervention through the backdoor correction formula where Use variational inference to introduce a new distribution Q(C|X) as the probability estimate of the latent variable given the image input, and the objective function is changed to the evidence lower bound of the model output, that is Step 102: Obtain data input and model the confounding factor C in the data; Step 103: Obtain the distribution of the confounding factor using the pseudo-variational posterior probability Step 104: Train and optimize the deep network model using the probability distribution P θ (Y|do(X=x)) based on the data input and the confounding factor; Step 2. Use the pre-trained object recognition deep network to extract the confusion factors and object features of the support set and the query set respectively: The feature expression of the confusion factors of the support set and the query set is The object feature expression is The object feature expression corrected by the confusion factor is Step 3: Conduct causal intervention on the target features of the support set data: Step 301: Obtain the confusion factors C of the support set and the query set from Step 2 s and C q , and generate a new confusion factor C q ′; Step 302: Use the new confounding factor C q ′ to perform counterfactual on the target features of the support set, and the counterfactual support set target features can be obtained as Step 4: Calculate and correct the distance matrix, and optimize the network model after obtaining the classification result: Step 401: Calculate the N class prototypes after counterfactual where M represents that there are M feature vectors in the i-th class, and the cosine distance is used to measure the distance between the N class prototypes cls i and the K query set target features F. Here, <·,·〉 represents the vector inner product, and a distance matrix D1 of dimension N×K can be obtained; Step 402: Calculate N category prototypes without counterfactuals where M represents that there are M feature vectors in the i-th category, and use the cosine distance to measure the N category prototypes after counterfactuals and the target feature F of the K query sets without the confounding factor correction c The distance between them, where <·,·> represents the vector inner product, and a distance matrix D2 of dimension N×K can be obtained; Step 403: According to the formula D out = λ·D1+(1 - λ)·D2, obtain the finally corrected distance matrix D out , and then obtain the target recognition result of the query set; Step 404: Calculate the cross-entropy loss based on the target recognition result and update and optimize the parameters of the deep network model; Step 5: Obtain the final few-shot target recognition network model.

2. The small-sample target recognition method based on two-stage causal intervention according to claim 1, characterized in that: The specific method of Step 102 includes: Step 1021: Randomly initialize the feature mapping matrix W i and the scaling matrix a i ; Step 1022, calculate the correlation score s between the input image X and the confusion factor c i ; i = a i · Tanh(W i · X); Step 1023, obtain according to the association score 3. The small-sample target recognition method based on two-stage causal intervention according to claim 1, wherein: The specific method of Step 103 includes: Step 1031: Randomly generate R pseudo-image samples x'; Step 1032. Repeat Step 1022 and Step 1023 to obtain Q(C|X = x′). The approximate estimation formula for the confounding factor distribution is 4. The small-sample target recognition method based on two-stage causal intervention according to claim 1, characterized in that: The model output expression after causal intervention in step 104 is where φ i represents the encoder, θ i represents the parameters of the encoder, and c i is obtained from Q(C|X) in step 102.

5. The small-sample target recognition method based on two-stage causal intervention according to claim 1, wherein: The specific method for generating the confounding factor in Step 301 includes: Step 3011: Set the confounding factor generation network g(·) and initialize its parameters; Step 3012, input the support set confusion factor C s Obtain the newly generated confusion factor C q ′ = g(C s ); Step 3013, calculate C q and C q ′s KL divergence D KL (C q , C q ′) to optimize the parameters of the confusion factor generation network.

Images