Feature enhancement method, device, equipment and storage medium for image recognition

Abstract

The present invention discloses a feature enhancement method, device, equipment and storage medium for cross-domain generalized image recognition, wherein the method includes: constructing a triplet input image; extracting causal features and non-causal features from the triplet input image; encoding the causal features and non-causal features to obtain two feature-level enhancement meta-knowledge; generating a feature-level implicit enhancement strategy based on the two feature-level enhancement meta-knowledge; using causal features, non-causal features and the feature-level implicit enhancement strategy to generate enhanced causal features and enhanced non-causal features; using a classifier to identify causal features, non-causal features, enhanced causal features, and enhanced non-causal features respectively, and combining the classification results and a preset loss function to reversely update the feature extractor, encoder, enhancer and classifier. The present invention effectively expands the source domain and learns cross-domain invariant causal features by learning two feature-level enhancement meta-knowledge and causal feature intervention, thereby improving the single-source domain generalization performance of the model.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a feature enhancement method, apparatus, device and storage medium for image recognition. Background Art

[0002] With the development of artificial intelligence technology, deep learning models have been widely used in the field of image classification. However, for the problem of single-source domain generalization that is more in line with real-world scenarios, how to reduce the dependence on image-level enhancement strategies and quantities and learn more stable cross-domain invariant causal features remains a more challenging and underexplored task. The design of deep learning models is based on a strong distribution assumption, that is, the training samples and test samples come from independent and identically distributed datasets. However, due to the inevitable domain shift between the source (training) domain and the target (testing) domain in real scenarios, when the model is directly applied to unseen scenarios, it often leads to a significant decrease in generalization performance. Single-source domain generalization aims to improve the generalization performance of models trained only in a single source domain on other unseen target domains, and its practical application value is increasingly attracting attention.

[0003] Existing work on single-source domain generalization improves the generalization performance of models through data augmentation and learning domain-invariant representations. The goal of data augmentation is to generate diverse samples through various image-level augmentation techniques to expand the coverage of the source domain and improve the cross-domain performance of the model. However, good extension to unknown domains depends on the careful design of image-level augmentation methods. Although recent studies have achieved better performance, the generalization ability of these models is closely related to the method and amount of image-level augmentation. Most of these strategies are based on perturbations of the entire image, which may lead to changes in the semantic information of the image itself during the process of learning cross-domain invariant features. In addition, domain-invariant features may establish spurious connections between data and labels because the model does not clearly distinguish causal and non-causal features between domains. Non-causal discriminative features in the training data replace inherent semantic features as decision variables for the prediction task, resulting in poor single-source domain generalization performance. Summary of the Invention

[0004] In view of this, the present application provides a feature enhancement method, apparatus, device and storage medium for image recognition to solve the problem of poor single-source domain generalization performance of existing image recognition algorithms.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: to provide a feature enhancement method applied to an image recognition model, characterized in that it includes: constructing a triplet input image, the triplet input image includes a first image and a second image corresponding to the same real label, and a third image generated by amplifying the first image based on a preset image-level enhancement strategy; inputting the triplet input image into a feature extractor to extract causal features and non-causal features; inputting the causal features and non-causal features into an encoder for encoding to obtain a first type of meta-knowledge representing the feature conversion information between the first image and the second image and a second type of meta-knowledge representing the feature conversion information between the first image and the third image; generating a feature-level implicit enhancement strategy based on the first type of meta-knowledge and the second type of meta-knowledge; inputting the causal features and non-causal features corresponding to the first image, as well as the feature-level implicit enhancement strategy into an enhancer to generate enhanced causal features and enhanced non-causal features; using a classifier to perform feature category identification on causal features, non-causal features, enhanced causal features, and enhanced non-causal features, respectively, to obtain classification results, and combining the classification results with a preset loss function to reversely update the feature extractor, encoder, enhancer, and classifier.

[0006] As a further improvement of the present application, the triple input image is input into a feature extractor to extract causal features and non-causal features, including: inputting the triple input image into a feature extractor to extract initial latent features; explicitly decomposing the initial latent features into a causal feature group and a non-causal feature group, the causal feature group including a first causal feature, a second causal feature, and a third causal feature corresponding to the first image, the second image, and the third image in sequence, and the non-causal feature group including a first non-causal feature, a second non-causal feature, and a third non-causal feature corresponding to the first image, the second image, and the third image in sequence.

[0007] As a further improvement of the present application, the causal features and the non-causal features are input into the encoder for encoding to obtain a first type of meta-knowledge representing the feature conversion information between the first image and the second image and a second type of meta-knowledge representing the feature conversion information between the first image and the third image, including: the first causal feature and the second causal feature are spliced ​​and input into the first encoder to obtain the first meta-knowledge, and the first non-causal feature and the second non-causal feature are spliced ​​and input into the first encoder to obtain the second meta-knowledge, the first meta-knowledge and the second meta-knowledge belong to the first type of meta-knowledge, and the first causal feature and the third causal feature are spliced ​​and input into the second encoder to obtain the third meta-knowledge, and the first non-causal feature and the third non-causal feature are spliced ​​and input into the second encoder to obtain the fourth meta-knowledge, the third meta-knowledge and the fourth meta-knowledge belong to the second type of meta-knowledge.

[0008] As a further improvement of the present application, generating a feature-level implicit enhancement strategy based on the first type of meta-knowledge and the second type of meta-knowledge includes: generating a first feature-level implicit enhancement strategy based on the first meta-knowledge in a preset manner, generating a second feature-level implicit enhancement strategy based on the second meta-knowledge in a preset manner, generating a third feature-level implicit enhancement strategy based on the third meta-knowledge in a preset manner, and generating a fourth feature-level implicit enhancement strategy based on the fourth meta-knowledge in a preset manner; the preset manner is expressed as: z t,ag =μ t,ag +∈∑ t,ag ;z t,ap =μ t,ap +∈∑ t,ap Among them, E ag and E ap The encoders corresponding to two feature-level enhanced meta-knowledge, and f t g Represents the initial coding features corresponding to the first image, the second image, and the third image, t∈{c,b}, represents the splicing operation, z t,ag and z t,ap Both represent feature-level implicit enhancement strategies, t∈{c,b}, c represents causal features, b represents non-causal features, μ t,ag , represents the first type of meta-knowledge, μ t,ap , represents the second type of meta-knowledge, ∈~N(0,1) represents the random sampling coefficient, and ∑=diag(σ) represents the scale parameter.

[0009] As a further improvement of the present application, the causal features and non-causal features corresponding to the first image, as well as the feature-level implicit enhancement strategy are input into the enhancer to generate enhanced causal features and enhanced non-causal features, including: splicing the first causal feature with the first feature-level implicit enhancement strategy and inputting it into the enhancer to obtain a first enhanced causal feature, and splicing the first non-causal feature with the second feature-level implicit enhancement strategy and inputting it into the enhancer to obtain a first enhanced non-causal feature, and splicing the first causal feature with the third feature-level implicit enhancement strategy and inputting it into the enhancer to obtain a second enhanced causal feature, and splicing the first non-causal feature with the fourth feature-level implicit enhancement strategy and inputting it into the enhancer to obtain a second enhanced non-causal feature.

[0010] As a further improvement of the present application, a classifier is used to identify the feature categories of causal features, non-causal features, enhanced causal features, and enhanced non-causal features, respectively, to obtain classification results, and the feature extractor, encoder, enhancer, and classifier are reversely updated in combination with the classification results and the preset loss function, including: marking the first causal feature, the second causal feature, the third causal feature, the first enhanced causal feature, and the second enhanced causal feature with causal feature labels, and marking the first non-causal feature, the second non-causal feature, the third non-causal feature, the first enhanced non-causal feature, and the second enhanced non-causal feature with non-causal feature labels; confusing the first causal feature, the second causal feature, the third causal feature, the first enhanced causal feature, the second enhanced causal feature, the first non-causal feature, the second non-causal feature, the third non-causal feature, the first enhanced non-causal feature, and the second enhanced non-causal feature, and randomly inputting them into the classifier in sequence to obtain the classification result corresponding to each feature; and reversely updating the feature extractor, encoder, enhancer, and classifier according to the classification result, the label corresponding to each feature, and the preset loss function.

[0011] As a further improvement of the present application, the preset loss function includes a first loss function representing the decoupling between causal features and non-causal features, a second loss function representing the independence of causal features and non-causal features, a third loss function representing the enhancement of the diversity of non-causal features while ensuring the enhanced consistency of causal features, and a fourth loss function representing the promotion of correct feature classification and the guarantee of the consistency of feature classification probability distribution;

[0012] The first loss function is expressed as:

[0013]

[0014]

[0015]

[0016] in, represents the first loss function, represents the loss of decoupling between causal and non-causal features, represents the loss of decoupling between enhanced causal features and enhanced non-causal features, a, p, g represent the first image, the second image, and the third image respectively, F represents the feature extractor, H represents the classifier, represents the category label of the triplet input image, Represents causal characteristics, Represents non-causal features, KL() represents Kullback-Leibler divergence, which is used to compare the distribution of non-causal features output by the classifier with the uniform distribution y uniform Alignment, represents the enhanced causal feature, Indicates the enhancement of non-causal features.

[0017] The second loss function is expressed as:

[0018]

[0019]

[0020]

[0021]

[0022]

[0023] in, represents the second loss function, represents the loss of independence between causal and non-causal features, represents the loss of enhancing the independence between causal features and non-causal features, C v∈{a,p,g} Used to measure the correlation between causal features and non-causal features, C w∈{p,g} Used to measure the correlation between enhanced causal features and enhanced non-causal features.

[0024] The third loss function is expressed as:

[0025]

[0026] Among them, δ>0 is a small boundary variable, represents the third loss function, A represents the enhancer, and d() represents the distance between the initial feature and the enhanced feature;

[0027] The fourth loss function is expressed as:

[0028]

[0029]

[0030]

[0031] Ω={F,E av ,E ag ,a,M,H};

[0032] in, represents the fourth loss function, Ω is the intermediate parameter, represents the set of causal features and enhanced causal features, represents the set of non-causal features and enhanced non-causal features, Ω represents the feature extractor F, the first encoder E av , the second encoder Eag , a collection of enhancer A, linear layer M, and classifier H.

[0033] In order to solve the above technical problems, another technical solution adopted in the present application is: to provide a feature enhancement device applied to an image recognition model, which includes: a construction module for constructing a triple input image, the triple input image includes a first image and a second image corresponding to the same real label, and a third image generated by amplifying the first image based on a preset image-level enhancement strategy; an extraction module for inputting the triple input image into a feature extractor to extract causal features and non-causal features; an encoding module for inputting causal features and non-causal features into an encoder for encoding to obtain a first type of meta-knowledge representing feature conversion information between the first image and the second image. and a second type of meta-knowledge that characterizes the feature conversion information between the first image and the third image; a strategy generation module for generating a feature-level implicit enhancement strategy based on the first type of meta-knowledge and the second type of meta-knowledge; an enhanced feature generation module for inputting the causal features and non-causal features corresponding to the first image, as well as the feature-level implicit enhancement strategy, into the enhancer to generate enhanced causal features and enhanced non-causal features; a classification module for using a classifier to perform feature category identification on causal features, non-causal features, enhanced causal features, and enhanced non-causal features, respectively, to obtain classification results, and to reversely update the feature extractor, encoder, enhancer, and classifier based on the classification results and a preset loss function.

[0034] To solve the above technical problems, another technical solution adopted in this application is: to provide a computer device, which includes a processor and a memory coupled to the processor, wherein program instructions are stored in the memory, and when the program instructions are executed by the processor, the processor executes the steps of the feature enhancement method for image recognition as described in any one of the above items.

[0035] In order to solve the above technical problems, another technical solution adopted in this application is: providing a storage medium storing program instructions capable of implementing any of the above-mentioned feature enhancement methods for image recognition.

[0036] The beneficial effects of the present application are: the feature enhancement method for image recognition of the present application utilizes the first image as source data, generates a third image as image-level enhancement data based on the first image based on a preset image-level enhancement strategy, utilizes the source data and the image-level enhancement data, and learns two types of feature-level enhancement meta-knowledge at the feature level to generate more diverse feature-level implicit enhancement strategies, reduces dependence on the preset image-level enhancement strategy, expands the source domain distribution, and achieves more stable cross-domain invariant causal feature learning, thereby improving the model's cross-domain generalization ability, and also further improves the single-source domain generalization performance by mixing causal features, non-causal features, enhanced causal features, and enhanced non-causal features and classifying them, so as to utilize causal intervention to learn more stable domain-invariant causal features. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 1 is a flow chart of a feature enhancement method for image recognition according to an embodiment of the present invention;

[0038] Figure 2 Schematic diagram of a model framework structure of a feature enhancement method for image recognition according to an embodiment of the present invention;

[0039] Figure 3 1 is a schematic diagram of functional modules of a feature enhancement device for image recognition according to an embodiment of the present invention;

[0040] Figure 4 is a schematic structural diagram of a computer device according to an embodiment of the present invention;

[0041] Figure 5 It is a schematic structural diagram of a storage medium according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0043] The terms "first," "second," and "third" in this application are used only for descriptive purposes and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of such features. In the description of this application, "multiple" means at least two, for example, two, three, etc., unless otherwise specifically defined. All directional indications in the embodiments of this application (such as up, down, left, right, front, back...) are only used to explain the relative positional relationship, movement, etc. between the components under a specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications also change accordingly. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products, or devices.

[0044] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0045] Figure 1 It is a flow chart of a feature enhancement method for image recognition according to an embodiment of the present invention. It should be noted that the method of the present invention is not limited to the method of the embodiment of the present invention if the results are substantially the same. Figure 1 The process sequence shown is limited. Figure 1 As shown, the feature enhancement method for image recognition includes the steps of:

[0046] Step S101: constructing a triplet input image, where the triplet input image includes a first image and a second image corresponding to the same true label, and a third image generated by amplifying the first image based on a preset image-level enhancement strategy.

[0047] Specifically, in this embodiment, given the source domain A small batch of triplets is generated in each training iteration, each triplet is of the form in is the anchor point sample, i.e. the first image, It's from D s The sample, and has the same category, i.e. the second image. Indicates that the preset image level enhancement strategy T is used to Amplify the sample to obtain a third image. For the sake of convenience, v is defined as an element in the set {a, p, g}, that is, v∈{a, p, g}. For the overall model framework, please refer to Figure 2 shown.

[0048] Step S102: Input the triplet input image into a feature extractor to extract causal features and non-causal features.

[0049] Specifically, see Figure 2 In this embodiment, a feature extractor F is constructed to extract features from the triplet input image to obtain causal features and non-causal features. The causal features are related to the true labels, while the non-causal features are used to encode other trivial patterns or deviations.

[0050] Furthermore, step S102 includes:

[0051] 1. Input the triplet input image into the feature extractor to extract the initial latent features.

[0052] 2. The initial latent features are explicitly decomposed into a causal feature group and a non-causal feature group. The causal feature group includes the first causal feature, the second causal feature, and the third causal feature corresponding to the first image, the second image, and the third image in sequence. The non-causal feature group includes the first non-causal feature, the second non-causal feature, and the third non-causal feature corresponding to the first image, the second image, and the third image in sequence.

[0053] Specifically, this embodiment extracts the initial latent features through the feature extractor F. and explicitly decompose it into causal features and non-causal features Two parts. Please refer to 2. The initial causal features include the first causal features corresponding to the first image, the second image, and the third image in sequence. Second causal characteristic The third causal characteristic The non-causal feature group includes the first non-causal feature corresponding to the first image, the second image, and the third image in sequence. Second non-causal characteristic The third non-causal characteristic

[0054] Step S103: Input the causal features and the non-causal features into the encoder for encoding to obtain a first type of meta-knowledge representing feature conversion information between the first image and the second image and a second type of meta-knowledge representing feature conversion information between the first image and the third image.

[0055] It should be noted that in order to expand the potential feature space of the source domain and learn domain-independent causal features, specifically, in this embodiment, two kinds of meta-knowledge about feature-level enhancement are learned: and Feature conversion information between and The feature transformation information between and is obtained from the feature-level enhancement meta-knowledge. Feature-level enhancement meta-knowledge represents nonlinear transformations that can enhance feature diversity, thereby expanding the source distribution and enabling more accurate learning of causal features. These two types of meta-knowledge are regarded as multivariate Gaussian distributions. With the help of these distributions, more diverse feature-level implicit enhancement strategies can be generated, rather than just being limited to the preset image-level enhancement strategy.

[0056] Furthermore, step S103 specifically includes: splicing the first causal feature and the second causal feature and inputting them into the first encoder to obtain first meta-knowledge, and splicing the first non-causal feature and the second non-causal feature and inputting them into the first encoder to obtain second meta-knowledge, the first meta-knowledge and the second meta-knowledge belong to the first category of meta-knowledge, and splicing the first causal feature and the third causal feature and inputting them into the second encoder to obtain third meta-knowledge, and splicing the first non-causal feature and the third non-causal feature and inputting them into the second encoder to obtain fourth meta-knowledge, the third meta-knowledge and the fourth meta-knowledge belong to the second category of meta-knowledge.

[0057] Specifically, see Figure 2 , the model includes a first encoder E ah and the second encoder E ap , used to generate the first type of meta-knowledge and the second type of meta-knowledge, which also includes a shared enhancer for generating enhanced features by combining feature-level enhanced meta-knowledge with initial features. Specifically, in this embodiment, it is set t∈{c,b}, and Through the splicing operation and input to E ag , we get the first type of meta-knowledge, and and Through the splicing operation and input to E ap , the second type of meta-knowledge is obtained, and the output is expressed as meta-knowledge about feature-level enhancement, is represented as first-class meta-knowledge, and Expressed as the second type of meta-knowledge:

[0058]

[0059]

[0060] in, Represents a feature concatenation operation.

[0061] Step S104: Generate a feature-level implicit enhancement strategy based on the first type of meta-knowledge and the second type of meta-knowledge.

[0062] Specifically, this embodiment utilizes feature-level implicit enhancement meta-knowledge to generate diverse feature-level implicit enhancement strategies through sampling.

[0063] Furthermore, step S104 specifically includes: generating a first feature-level implicit enhancement strategy based on the first meta-knowledge in a preset manner, generating a second feature-level implicit enhancement strategy based on the second meta-knowledge in a preset manner, generating a third feature-level implicit enhancement strategy based on the third meta-knowledge in a preset manner, and generating a fourth feature-level implicit enhancement strategy based on the fourth meta-knowledge in a preset manner.

[0064] Specifically, this embodiment uses the reparameterization technique to perform back propagation calculation of the feature-level implicit enhancement strategy, and the preset method is expressed as:

[0065] Z t,ag =μ t,ag +∈∑ t,ag ;

[0066] z t,ap =μ t,ap +∈∑ t,ap ;

[0067] Among them, z t,ag and z t,ap Both represent feature-level implicit enhancement strategies, t∈{c,b}, c represents causal features, b represents non-causal features, ∈~N(0,1) represents random sampling coefficients, and ∑=diag(σ) represents scale parameters.

[0068] Step S105: inputting the causal features and non-causal features corresponding to the first image, as well as the feature-level implicit enhancement strategy, into the enhancer to generate enhanced causal features and enhanced non-causal features.

[0069] Specifically, after obtaining the feature-level implicit enhancement strategy and inputting it into the enhancer, the initial features (i.e., the causal features and non-causal features corresponding to the first image) and the feature-level implicit enhancement strategy generated by random sampling are input into the enhancer to generate two types of enhanced features: enhanced causal features and enhanced non-causal features. The details are as follows:

[0070]

[0071]

[0072] Furthermore, step S105 specifically includes:

[0073] The first causal feature is spliced ​​with the first feature-level implicit enhancement strategy and input into the enhancer to obtain a first enhanced causal feature, and the first non-causal feature is spliced ​​with the second feature-level implicit enhancement strategy and input into the enhancer to obtain a first enhanced non-causal feature, and the first causal feature is spliced ​​with the third feature-level implicit enhancement strategy and input into the enhancer to obtain a second enhanced causal feature, and the first non-causal feature is spliced ​​with the fourth feature-level implicit enhancement strategy and input into the enhancer to obtain a second enhanced non-causal feature.

[0074] Step S106: Use the classifier to identify the feature categories of causal features, non-causal features, enhanced causal features, and enhanced non-causal features respectively to obtain classification results, and combine the classification results and the preset loss function to reversely update the feature extractor, encoder, enhancer and classifier.

[0075] Furthermore, step S106 specifically includes:

[0076] 1. Label the first causal feature, the second causal feature, the third causal feature, the first enhanced causal feature, and the second enhanced causal feature with causal feature labels, and label the first non-causal feature, the second non-causal feature, the third non-causal feature, the first enhanced non-causal feature, and the second enhanced non-causal feature with non-causal feature labels;

[0077] 2. The first causal feature, the second causal feature, the third causal feature, the first enhanced causal feature, the second enhanced causal feature, the first non-causal feature, the second non-causal feature, the third non-causal feature, the first enhanced non-causal feature, and the second enhanced non-causal feature are mixed and randomly input into the classifier in sequence to obtain the classification result corresponding to each feature;

[0078] 3. Update the feature extractor, encoder, enhancer, and classifier in reverse based on the classification results, the label corresponding to each feature, and the preset loss function.

[0079] Specifically, to obtain sufficient distribution diversity and learn better domain-invariant causal features, see Figure 2 In this embodiment, a dimensionality reduction linear layer M is defined, combined with a classifier H, to classify the "intervened features" that are a combination of fixed causal features and differently augmented variable non-causal features. The "intervened features" are obtained by mixing causal features, non-causal features, enhanced causal features, and enhanced non-causal features.

[0080] Furthermore, the preset loss function includes a first loss function representing the decoupling between causal features and non-causal features, a second loss function representing the independence of causal features and non-causal features, a third loss function representing the enhancement of the diversity of non-causal features while ensuring the enhanced consistency of causal features, and a fourth loss function representing the promotion of correct feature classification and the guarantee of the consistency of feature classification probability distribution;

[0081] The first loss function is expressed as:

[0082]

[0083]

[0084]

[0085] in, represents the first loss function, represents the loss of decoupling between causal and non-causal features, represents the loss of decoupling between enhanced causal features and enhanced non-causal features, a, p, g represent the first image, the second image, and the third image respectively, F represents the feature extractor, H represents the classifier, represents the category label of the triplet input image, Represents causal characteristics, Represents non-causal features, KL() represents Kullback-Leibler divergence, which is used to compare the distribution of non-causal features output by the classifier with the uniform distribution y uniform Alignment, represents the enhanced causal feature, Indicates the enhancement of non-causal features.

[0086] The second loss function is expressed as:

[0087]

[0088]

[0089]

[0090]

[0091]

[0092] in, represents the second loss function, represents the loss of independence between causal and non-causal features, represents the loss of enhancing the independence between causal features and non-causal features, C v∈{a,p,g}Used to measure the correlation between causal features and non-causal features, C w∈{p,g} Used to measure the correlation between enhanced causal features and enhanced non-causal features.

[0093] The third loss function is expressed as:

[0094]

[0095] Among them, δ>0 is a small boundary variable, represents the third loss function, A represents the enhancer, and d() represents the distance between the initial feature and the enhanced feature;

[0096] The fourth loss function is expressed as:

[0097]

[0098]

[0099]

[0100] Ω={F,E av ,E ag ,A,M,H};

[0101] in, represents the fourth loss function, represents the set of causal features and enhanced causal features, represents the set of non-causal features and enhanced non-causal features, Ω represents the feature extractor F, the first encoder E av , the second encoder E ag , a collection of enhancer A, linear layer M, and classifier H.

[0102] The feature enhancement method for image recognition of this embodiment uses the first image as source data, generates a third image as image-level enhancement data based on the first image based on a preset image-level enhancement strategy, and uses the source data and image-level enhancement data to learn two types of feature-level enhancement meta-knowledge at the feature level, generate more diverse feature-level implicit enhancement strategies, reduce dependence on the preset image-level enhancement strategy, expand the source domain distribution, and achieve more stable cross-domain invariant causal feature learning, thereby improving the model's cross-domain generalization ability, and also further improves the single-source domain generalization performance by mixing causal features, non-causal features, enhanced causal features, and enhanced non-causal features and classifying them, so as to utilize causal intervention to learn more stable domain-invariant causal features.

[0103] Figure 3 FIG is a schematic diagram of the functional modules of the feature enhancement device for image recognition according to an embodiment of the present invention. Figure 3As shown, the feature enhancement device 20 for image recognition includes: a construction module 21, an extraction module 22, an encoding module 23, a strategy generation module 24, an enhanced feature generation module 25, and a classification module 26.

[0104] A construction module 21 is configured to construct a triplet input image, where the triplet input image includes a first image and a second image corresponding to the same true label, and a third image generated by amplifying the first image based on a preset image-level enhancement strategy;

[0105] An extraction module 22 is used to input the triplet input image into a feature extractor to extract causal features and non-causal features;

[0106] An encoding module 23 is configured to input the causal features and the non-causal features into an encoder for encoding to obtain first-category meta-knowledge representing feature conversion information between the first image and the second image and second-category meta-knowledge representing feature conversion information between the first image and the third image;

[0107] A strategy generation module 24, configured to generate a feature-level implicit enhancement strategy based on the first type of meta-knowledge and the second type of meta-knowledge;

[0108] an enhanced feature generation module 25 for inputting the causal features and non-causal features corresponding to the first image, and the feature-level implicit enhancement strategy, into the enhancer to generate enhanced causal features and enhanced non-causal features;

[0109] The classification module 26 is used to use the classifier to identify the feature categories of causal features, non-causal features, enhanced causal features, and enhanced non-causal features respectively, obtain classification results, and combine the classification results and the preset loss function to reversely update the feature extractor, encoder, enhancer and classifier.

[0110] Optionally, the extraction module 22 performs an operation of inputting the triple input image into the feature extractor to extract causal features and non-causal features, specifically including: inputting the triple input image into the feature extractor to extract initial latent features; explicitly decomposing the initial latent features into a causal feature group and a non-causal feature group, the causal feature group including the first causal feature, the second causal feature, and the third causal feature corresponding to the first image, the second image, and the third image in sequence, and the non-causal feature group including the first non-causal feature, the second non-causal feature, and the third non-causal feature corresponding to the first image, the second image, and the third image in sequence.

[0111] Optionally, the encoding module 23 performs an operation of inputting the causal features and the non-causal features into the encoder for encoding to obtain a first type of meta-knowledge representing the feature conversion information between the first image and the second image and a second type of meta-knowledge representing the feature conversion information between the first image and the third image, specifically including: splicing the first causal feature and the second causal feature and inputting them into the first encoder to obtain first meta-knowledge, and splicing the first non-causal feature and the second non-causal feature and inputting them into the first encoder to obtain second meta-knowledge, the first meta-knowledge and the second meta-knowledge belong to the first type of meta-knowledge, and splicing the first causal feature and the third causal feature and inputting them into the second encoder to obtain third meta-knowledge, and splicing the first non-causal feature and the third non-causal feature and inputting them into the second encoder to obtain fourth meta-knowledge, the third meta-knowledge and the fourth meta-knowledge belong to the second type of meta-knowledge.

[0112] Optionally, the strategy generation module 24 performs an operation of generating a feature-level implicit enhancement strategy based on the first type of meta-knowledge and the second type of meta-knowledge, specifically including: generating a first feature-level implicit enhancement strategy based on the first meta-knowledge in a preset manner, generating a second feature-level implicit enhancement strategy based on the second meta-knowledge in a preset manner, generating a third feature-level implicit enhancement strategy based on the third meta-knowledge in a preset manner, and generating a fourth feature-level implicit enhancement strategy based on the fourth meta-knowledge in a preset manner; the preset manner is expressed as: z t,ag =μ t,ag +∈∑ t,ag ;z t,ap =μ t,ap +∈∑ t,ap Among them, E ag and E ap The encoders corresponding to two feature-level enhanced meta-knowledge, and Represents the initial coding features corresponding to the first image, the second image, and the third image, t∈{c,b}, represents the splicing operation, z t,ag and z t,ap Both represent feature-level implicit enhancement strategies, t∈{c,b}, c represents causal features, b represents non-causal features, μ t,ag , represents the first type of meta-knowledge, μ t,ap , represents the second type of meta-knowledge, ∈~N(0,1) represents the random sampling coefficient, and ∑=diag(σ) represents the scale parameter.

[0113] Optionally, the enhanced feature generation module 25 performs an operation of inputting the causal features and non-causal features corresponding to the first image, and the feature-level implicit enhancement strategy into the enhancer to generate enhanced causal features and enhanced non-causal features, including: splicing the first causal feature with the first feature-level implicit enhancement strategy and inputting them into the enhancer to obtain a first enhanced causal feature, and splicing the first non-causal feature with the second feature-level implicit enhancement strategy and inputting them into the enhancer to obtain a first enhanced non-causal feature, and splicing the first causal feature with the third feature-level implicit enhancement strategy and inputting them into the enhancer to obtain a second enhanced causal feature, and splicing the first non-causal feature with the fourth feature-level implicit enhancement strategy and inputting them into the enhancer to obtain a second enhanced non-causal feature.

[0114] Optionally, the classification module 26 uses a classifier to perform feature category identification on the causal feature, the non-causal feature, the enhanced causal feature, and the enhanced non-causal feature, respectively, to obtain a classification result, and combines the classification result and the preset loss function to reversely update the operation of the feature extractor, encoder, enhancer and classifier, including: marking the first causal feature, the second causal feature, the third causal feature, the first enhanced causal feature, and the second enhanced causal feature with a causal feature label, and marking the first non-causal feature, the second non-causal feature, the third non-causal feature, the first enhanced non-causal feature, and the second enhanced non-causal feature with a non-causal feature label; confusing the first causal feature, the second causal feature, the third causal feature, the first enhanced causal feature, the second enhanced causal feature, the first non-causal feature, the second non-causal feature, the third non-causal feature, the first enhanced non-causal feature, and the second enhanced non-causal feature, and randomly inputting them into the classifier in sequence to obtain the classification result corresponding to each feature; and reversely updating the feature extractor, encoder, enhancer and classifier according to the classification result, the label corresponding to each feature and the preset loss function.

[0115] Optionally, the preset loss function includes a first loss function representing the decoupling between causal features and non-causal features, a second loss function representing the independence of causal features and non-causal features, a third loss function representing the enhancement of the diversity of non-causal features while ensuring the enhanced consistency of causal features, and a fourth loss function representing the promotion of correct feature classification and the guarantee of consistency of feature classification probability distribution;

[0116] The first loss function is expressed as:

[0117]

[0118]

[0119]

[0120] in, represents the first loss function, represents the loss of decoupling between causal and non-causal features, represents the loss of decoupling between enhanced causal features and enhanced non-causal features, a, p, g represent the first image, the second image, and the third image respectively, F represents the feature extractor, H represents the classifier, represents the category label of the triplet input image, Represents causal characteristics, Represents non-causal features, KL() represents Kullback-Leibler divergence, which is used to compare the distribution of non-causal features output by the classifier with the uniform distribution y uniform Alignment, represents the enhanced causal feature, Indicates the enhancement of non-causal features.

[0121] The second loss function is expressed as:

[0122]

[0123]

[0124]

[0125]

[0126]

[0127] in, represents the second loss function, represents the loss of independence between causal and non-causal features, represents the loss of enhancing the independence between causal features and non-causal features, C v∈{a,p,g} Used to measure the correlation between causal features and non-causal features, C w∈{p,g} Used to measure the correlation between enhanced causal features and enhanced non-causal features.

[0128] The third loss function is expressed as:

[0129]

[0130] Among them, δ>0 is a small boundary variable, represents the third loss function, A represents the enhancer, and d() represents the distance between the initial feature and the enhanced feature;

[0131] The fourth loss function is expressed as:

[0132]

[0133]

[0134]

[0135] Ω={F,E av ,E ag ,A,M,H};

[0136] in, represents the fourth loss function, represents the set of causal features and enhanced causal features, represents the set of non-causal features and enhanced non-causal features, Ω represents the feature extractor F, the first encoder E av , the second encoder E ag , a collection of enhancer A, linear layer M, and classifier H.

[0137] For other details about the technical solutions for implementing each module in the feature enhancement device for image recognition in the above embodiment, please refer to the description of the feature enhancement method for image recognition in the above embodiment, which will not be repeated here.

[0138] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similarities between the various embodiments can be referred to in conjunction with each other. For device embodiments, since they are generally similar to method embodiments, their description is relatively simple, and for relevant details, reference can be made to the description of the method embodiments.

[0139] See also Figure 4 , Figure 4 FIG. 1 is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Figure 4 As shown, the computer device 30 includes a processor 31 and a memory 32 coupled to the processor 31. The memory 32 stores program instructions. When the program instructions are executed by the processor 31, the processor 31 executes the steps of the feature enhancement method for image recognition described in any of the above embodiments.

[0140] The processor 31 may also be referred to as a resource (Central Processing Unit). The processor 31 may be an integrated circuit chip having signal processing capabilities. The processor 31 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor or any conventional processor.

[0141] See Figure 5 , Figure 5 Schematic diagram of the structure of the storage medium of an embodiment of the present invention. The storage medium of an embodiment of the present invention stores program instructions 41 that can implement the above-mentioned feature enhancement method for image recognition, wherein the program instructions 41 can be stored in the above-mentioned storage medium in the form of a software product, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or a computer device such as a computer, a server, a mobile phone, or a tablet.

[0142] In the several embodiments provided in this application, it should be understood that the disclosed computer devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0143] In addition, the functional units in the various embodiments of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units. The above is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A feature enhancement method for image recognition, characterized in that: It includes: Constructing a triplet input image, the triplet input image including a first image and a second image corresponding to the same true label, and a third image generated by augmenting the first image based on a preset image-level enhancement strategy; Inputting the triplet input image into a feature extractor to extract causal features and non-causal features; Inputting the causal features and the non-causal features into an encoder for encoding to obtain first-category meta-knowledge representing feature conversion information between the first image and the second image and second-category meta-knowledge representing feature conversion information between the first image and the third image; generating a feature-level implicit enhancement strategy based on the first type of meta-knowledge and the second type of meta-knowledge; Inputting the causal features and non-causal features corresponding to the first image and the feature-level implicit enhancement strategy into an enhancer to generate enhanced causal features and enhanced non-causal features; A classifier is used to identify the feature categories of causal features, non-causal features, enhanced causal features, and enhanced non-causal features respectively to obtain classification results, and the feature extractor, the encoder, the enhancer, and the classifier are reversely updated based on the classification results and a preset loss function.

2. The feature enhancement method for image recognition according to claim 1, characterized in that: Inputting the triplet input image into a feature extractor to extract causal features and non-causal features includes: Inputting the triplet input image into the feature extractor to extract initial latent features; The initial latent features are explicitly decomposed into a causal feature group and a non-causal feature group, wherein the causal feature group includes a first causal feature, a second causal feature, and a third causal feature corresponding to the first image, the second image, and the third image in sequence, and the non-causal feature group includes a first non-causal feature, a second non-causal feature, and a third non-causal feature corresponding to the first image, the second image, and the third image in sequence.

3. The feature enhancement method for image recognition according to claim 2, characterized in that: Inputting the causal features and the non-causal features into an encoder for encoding to obtain first-category meta-knowledge representing feature conversion information between the first image and the second image and second-category meta-knowledge representing feature conversion information between the first image and the third image, includes: The first causal feature and the second causal feature are spliced ​​and input into the first encoder to obtain first meta-knowledge, and the first non-causal feature and the second non-causal feature are spliced ​​and input into the first encoder to obtain second meta-knowledge, and the first meta-knowledge and the second meta-knowledge belong to the first category of meta-knowledge, and the first causal feature and the third causal feature are spliced ​​and input into the second encoder to obtain third meta-knowledge, and the first non-causal feature and the third non-causal feature are spliced ​​and input into the second encoder to obtain fourth meta-knowledge, and the third meta-knowledge and the fourth meta-knowledge belong to the second category of meta-knowledge.

4. The feature enhancement method for image recognition according to claim 3, characterized in that: The generating of a feature-level implicit enhancement strategy based on the first type of meta-knowledge and the second type of meta-knowledge includes: Generating a first feature-level implicit enhancement strategy based on the first meta-knowledge in a preset manner, generating a second feature-level implicit enhancement strategy based on the second meta-knowledge in a preset manner, generating a third feature-level implicit enhancement strategy based on the third meta-knowledge in a preset manner, and generating a fourth feature-level implicit enhancement strategy based on the fourth meta-knowledge in a preset manner; The preset method is expressed as: With t,ag =μ t,ag +∈∑ t,ag ; With t,ap =μ t,ap +∈∑ t,ap ; Among them, E ag and E ap The encoders corresponding to two feature-level enhanced meta-knowledge, and Represents the initial coding features corresponding to the first image, the second image, and the third image, t∈{c,b}, represents the splicing operation, z t,ag and z t,ap Both represent feature-level implicit enhancement strategies, t∈{c,b}, c represents causal features, b represents non-causal features, μ t,ag , represents the first type of meta-knowledge, μ t,ap , represents the second type of meta-knowledge, ∈~N(0,1) represents the random sampling coefficient, and ∑=diag(σ) represents the scale parameter.

5. The feature enhancement method for image recognition according to claim 4, characterized in that: The step of inputting the causal features and the non-causal features corresponding to the first image and the feature-level implicit enhancement strategy into the enhancer to generate enhanced causal features and enhanced non-causal features includes: The first causal feature is spliced ​​with the first feature-level implicit enhancement strategy and input into the enhancer to obtain a first enhanced causal feature, and the first non-causal feature is spliced ​​with the second feature-level implicit enhancement strategy and input into the enhancer to obtain a first enhanced non-causal feature, and the first causal feature is spliced ​​with the third feature-level implicit enhancement strategy and input into the enhancer to obtain a second enhanced causal feature, and the first non-causal feature is spliced ​​with the fourth feature-level implicit enhancement strategy and input into the enhancer to obtain a second enhanced non-causal feature.

6. The feature enhancement method for image recognition according to claim 5, characterized in that: The method comprises: using a classifier to identify the causal features, the non-causal features, the enhanced causal features, and the enhanced non-causal features, respectively, to obtain classification results, and combining the classification results with a preset loss function to reversely update the feature extractor, the encoder, the enhancer, and the classifier, including: Marking the first causal feature, the second causal feature, the third causal feature, the first enhanced causal feature, and the second enhanced causal feature with a causal feature label, and marking the first non-causal feature, the second non-causal feature, the third non-causal feature, the first enhanced non-causal feature, and the second enhanced non-causal feature with a non-causal feature label; The first causal feature, the second causal feature, the third causal feature, the first enhanced causal feature, the second enhanced causal feature, the first non-causal feature, the second non-causal feature, the third non-causal feature, the first enhanced non-causal feature, and the second enhanced non-causal feature are mixed up and randomly input into the classifier in sequence to obtain a classification result corresponding to each feature; The feature extractor, the encoder, the enhancer and the classifier are reversely updated according to the classification result, the label corresponding to each feature and the preset loss function.

7. The feature enhancement method for image recognition according to claim 6, characterized in that: The preset loss function includes a first loss function representing the decoupling between causal features and non-causal features, a second loss function representing the independence of causal features and non-causal features, a third loss function representing the enhancement of the diversity of non-causal features while ensuring the enhanced consistency of causal features, and a fourth loss function representing the promotion of correct feature classification and the guarantee of the consistency of feature classification probability distribution; The first loss function is expressed as: in, represents the first loss function, represents the loss of decoupling between causal and non-causal features, represents the loss of decoupling between enhanced causal features and enhanced non-causal features, a, p, g represent the first image, the second image, and the third image respectively, F represents the feature extractor, H represents the classifier, represents the category label of the triplet input image, Represents causal characteristics, Represents non-causal features, KL() represents Kullback-Leibler divergence, which is used to compare the distribution of non-causal features output by the classifier with the uniform distribution y uniform Alignment, represents the enhanced causal feature, represents the enhancement of non-causal features; The second loss function is expressed as: in, represents the second loss function, represents the loss of independence between causal and non-causal features, represents the loss of enhancing the independence between causal features and non-causal features, C v∈{a,p,g} Used to measure the correlation between causal features and non-causal features, C w∈{p,g} Used to measure the correlation between enhanced causal features and enhanced non-causal features; The third loss function is expressed as: Among them, δ>0 is a small boundary variable, represents the third loss function, A represents the enhancer, and d() represents the distance between the initial feature and the enhanced feature; The fourth loss function is expressed as: Ω={F,E av ,E ag ,A,M,H}; in, represents the fourth loss function, represents the set of causal features and enhanced causal features, represents the set of non-causal features and enhanced non-causal features, Ω represents the feature extractor F, the first encoder E av , the second encoder E ag , a collection of enhancer A, linear layer M, and classifier H.

8. A feature enhancement device for image recognition, characterized in that: It includes: A construction module is configured to construct a triplet input image, wherein the triplet input image includes a first image and a second image corresponding to the same true label, and a third image generated by amplifying the first image based on a preset image-level enhancement strategy; An extraction module, configured to input the triplet input image into a feature extractor to extract causal features and non-causal features; an encoding module, configured to input the causal features and the non-causal features into an encoder for encoding, thereby obtaining first-category meta-knowledge representing feature conversion information between the first image and the second image, and second-category meta-knowledge representing feature conversion information between the first image and the third image; a strategy generation module, configured to generate a feature-level implicit enhancement strategy based on the first category meta-knowledge and the second category meta-knowledge; an enhanced feature generation module, configured to input the causal features and non-causal features corresponding to the first image and the feature-level implicit enhancement strategy into an enhancer to generate enhanced causal features and enhanced non-causal features; The classification module is used to use a classifier to identify the feature categories of causal features, non-causal features, enhanced causal features, and enhanced non-causal features, obtain classification results, and reversely update the feature extractor, the encoder, the enhancer, and the classifier based on the classification results and a preset loss function.

9. A computer device, characterized in that: The computer device includes a processor and a memory coupled to the processor, wherein program instructions are stored in the memory. When the program instructions are executed by the processor, the processor performs the steps of the feature enhancement method for image recognition as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The device stores program instructions capable of implementing the feature enhancement method for image recognition according to any one of claims 1 to 7.

Images