Image classification method and device based on causal reasoning

Abstract

The embodiment of the present application provides an image classification method and device based on causal reasoning, wherein the method comprises: obtaining image data to be classified; inputting the image data into a trained image classification model; and obtaining the image classification result output by the trained image classification model. Through the present application, the problem of low image classification accuracy is solved, thereby achieving the effect of improving image classification accuracy.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and in particular, to a method and device for image classification based on causal reasoning. Background Art

[0002] In the related art, traditional image classification algorithms are often affected by misleading environmental factors when facing complex and changeable actual environmental conditions, which leads to a decrease in the accuracy of image classification. Therefore, there is a problem of low image classification accuracy.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a method and device for image classification based on causal reasoning, so as to at least solve the problem of low efficiency of image classification based on causal reasoning in the related art.

[0005] According to an embodiment of the present application, a causal reasoning-based image classification method is provided, comprising: obtaining image data to be classified; inputting the image data into a trained image classification model, wherein the trained image classification model is an image classification model trained using a sample support set and a sample query set, the image classification model is a neural network model that performs image classification through causal reasoning, the sample support set is a sample set constructed using a plurality of first image data, the sample query set is a sample set constructed using a plurality of first image data and a plurality of second image data, the plurality of first image data are a plurality of image data located in a first environment obtained after classifying a plurality of image sample data, the plurality of second image data are a plurality of image data located in a second environment obtained after classifying the plurality of image sample data, the image data category in the sample support set is the same as the image data category in the sample query set; obtaining an image classification result output by the trained image classification model, wherein the image classification result is used to indicate the image data category to which the image data belongs.

[0006] According to another embodiment of the present application, there is provided an image classification device based on causal reasoning, comprising:

[0007] An image acquisition unit is used to acquire image data to be classified; an input unit is used to input the above image data into a trained image classification model, wherein the above trained image classification model is an image classification model trained using a sample support set and a sample query set, and the above image classification model is a neural network model that performs image classification through causal reasoning, the above sample support set is a sample set constructed using multiple first image data, the above sample query set is a sample set constructed using multiple first image data and multiple second image data, the above multiple first image data are multiple image data located in a first environment after classifying multiple image sample data, the above multiple second image data are multiple image data located in a second environment after classifying the above multiple image sample data, and the image data category in the above sample support set is the same as the image data category in the above sample query set; a first acquisition unit is used to acquire an image classification result output by the above trained image classification model, wherein the above image classification result is used to indicate the image data category to which the above image data belongs.

[0008] According to another embodiment of the present application, a method for training an image classification model based on causal reasoning is provided, including: acquiring multiple image sample data; classifying the multiple image sample data to obtain multiple first image data located in a first environment, and multiple second image data located in a second environment; using the multiple first image data to construct a sample support set; using the multiple first image data and the multiple second image data to construct a sample query set, wherein the image data category in the sample support set is the same as the image data category in the sample query set; training an initial image classification model through the sample support set and the sample query set to obtain a trained image classification model, wherein the image classification model is a neural network model that performs image classification through causal reasoning.

[0009] According to another aspect of the present application, there is provided an image classification model training device based on causal reasoning, comprising: a second acquisition unit, used to acquire multiple image sample data; a classification unit, used to classify the above multiple image sample data to obtain multiple first image data located in a first environment, and multiple second image data located in a second environment; a first construction unit, used to construct a sample support set using the above multiple first image data; a second construction unit, used to construct a sample query set using the above multiple first image data and the above multiple second image data, wherein the image data category in the above sample support set is the same as the image data category in the above sample query set; a training unit, used to train an initial image classification model through the above sample support set and the above sample query set to obtain a trained image classification model, wherein the above image classification model is a neural network model that performs image classification through causal reasoning.

[0010] As an optional scheme, the above-mentioned training unit includes: a first updating module, used to update the first parameter of the feature extraction module in the above-mentioned initial image classification model through the above-mentioned sample support set until the first convergence condition is met, and an updated feature extraction module is obtained, wherein the above-mentioned feature extraction module is used to capture and represent causal features; a second updating module, used to update the second parameter of the image classification module in the above-mentioned initial image classification model through the above-mentioned sample query set until the second convergence condition is met, and an updated image classification module is obtained, wherein the above-mentioned image classification module is used to perform image classification using the above-mentioned causal features.

[0011] As an optional solution, the first updating module includes: a first acquisition submodule, used to obtain a current feature extractor represented by a first subparameter, and a current causal classifier represented by a second subparameter, wherein the first parameter includes the first subparameter and the second subparameter, and the feature extraction module includes the feature extractor and the causal classifier; a first extraction submodule, used to use the current feature extractor to perform feature extraction on the image data in the sample support set to obtain a current feature vector, wherein the feature vector includes causal features; a first classification submodule, used to use the current causal classifier to classify the current feature vector to obtain a current prediction result; a first measurement submodule A block is used to measure the difference between the current prediction result and the data label carried by the image data in the sample support set through the current first loss function; a second acquisition submodule is used to obtain the updated first sub-parameter and the updated second sub-parameter when the current first loss function does not meet the first convergence condition, and use the feature extractor represented by the updated first sub-parameter as the current feature extractor, and use the causal classifier represented by the updated second sub-parameter as the current causal classifier; a second extraction submodule is used to determine the current feature extraction module as the updated feature extraction module when the current first loss function meets the first convergence condition.

[0012] As an optional solution, the above-mentioned second update module includes: a third acquisition submodule, used to obtain the sample feature vector obtained after the above-mentioned updated feature extraction module performs feature extraction on the image data in the above-mentioned sample query set; an input submodule, used to input the above-mentioned sample feature vector into the current image classification module to obtain the current classification result output by the above-mentioned current image classification module, wherein the above-mentioned image classification module is obtained by the representation of the above-mentioned second parameter; a second measurement submodule, used to measure the difference between the current classification result and the data label carried by the image data in the above-mentioned sample query set through the current second loss function; a fourth acquisition submodule, used to obtain the updated second parameter when the above-mentioned current second loss function does not meet the above-mentioned second convergence condition, and use the image classification module represented by the above-mentioned updated second parameter as the above-mentioned current image classification module; a determination submodule, used to determine the above-mentioned current image classification module as the above-mentioned updated image classification module when the above-mentioned current second loss function meets the above-mentioned second convergence condition.

[0013] As an optional scheme, the above-mentioned classification unit includes: an acquisition module, used to obtain multiple confusion features corresponding to each image sample data among the above-mentioned multiple image sample data, wherein the confusion feature is a feature that affects the generalization ability of the above-mentioned image classification model; a training module, used to use the above-mentioned multiple confusion features to train a deviation classifier, wherein the above-mentioned deviation classifier is used to identify and classify deviations caused by the above-mentioned confusion features; a deviation classification module, used to use the above-mentioned deviation classifier to classify the above-mentioned multiple image sample data to obtain the above-mentioned multiple first image data, and the above-mentioned multiple second image data.

[0014] As an optional scheme, the above-mentioned deviation classification module includes: an optimization submodule, which is used to use the above-mentioned deviation classifier to optimize the data distribution in the initial partitioning matrix to obtain the optimized partitioning matrix, wherein the above-mentioned partitioning matrix is ​​used to guide the partitioning of data into different environments, and the above-mentioned initial partitioning matrix is ​​a randomly generated matrix that conforms to the normal distribution; a second classification submodule, which is used to use the above-mentioned optimized partitioning matrix to classify the above-mentioned multiple image sample data to obtain the above-mentioned multiple first image data, and the above-mentioned multiple second image data.

[0015] As an optional scheme, the above-mentioned optimization submodule includes: a first acquisition subunit, used to obtain the empirical risk minimization loss and the gradient norm penalty, wherein the above-mentioned empirical risk minimization loss is used to reflect the performance of the above-mentioned deviation classifier on each sample data set, and the above-mentioned gradient norm penalty is used to measure the optimality of the above-mentioned deviation classifier on each of the above-mentioned sample data sets; an adjustment subunit, used to adjust the data distribution in the above-mentioned initial partitioning matrix until the sum of the above-mentioned empirical risk minimization loss and the above-mentioned gradient norm penalty is maximized, thereby obtaining the above-mentioned optimized partitioning matrix.

[0016] As an optional scheme, the above-mentioned optimization submodule includes: a second acquisition subunit, used to acquire a first function and a second function, wherein the above-mentioned first function is used to focus on the quantitative imbalance of different data partition intervals, and the above-mentioned second function is used to focus on maintaining the minimum adjustment during the data partitioning process; a setting subunit, used to use the above-mentioned first function and the above-mentioned second function as a balance loss function in the process of optimizing the data distribution in the above-mentioned initial partitioning matrix; a constraint subunit, used to constrain the optimization process of the data distribution in the above-mentioned initial partitioning matrix through the above-mentioned balance loss function.

[0017] As an optional solution, the above-mentioned training unit includes: an iterative training module, which is used to iteratively train the initial image classification model through the above-mentioned sample support set and the above-mentioned sample query set to obtain the optimal round image classification model; a testing module, which is used to test the above-mentioned optimal round image classification model using the image data in the sample test set to obtain a test result, wherein the above-mentioned test result is used to measure whether the above-mentioned optimal round image classification model is used as the above-mentioned trained image classification model.

[0018] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when running.

[0019] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0020] According to another embodiment of the present application, a computer program product is provided, including a computer program, wherein when the computer program is configured to be executed by a processor, the steps in any one of the above method embodiments are performed.

[0021] Through the present application, image data to be classified is obtained; the image data is input into a trained image classification model, wherein the trained image classification model is an image classification model trained using a sample support set and a sample query set, the image classification model is a neural network model that performs image classification through causal reasoning, the sample support set is a sample set constructed using a plurality of first image data, the sample query set is a sample set constructed using a plurality of first image data and a plurality of second image data, the plurality of first image data is a plurality of image data located in a first environment after classifying a plurality of image sample data, the plurality of second image data is a plurality of image data located in a second environment after classifying a plurality of image sample data, the image data category in the sample support set is the same as the image data category in the sample query set; and an image classification result output by the trained image classification model is obtained, wherein the image classification result is used to indicate the image data category to which the image data belongs.

[0022] Specifically, after obtaining the image data to be classified, the image data is input into the image classification model, and the image classification model is obtained by a sample support set and a sample query set, and the sample support set is obtained by multiple first images, and the sample query set is obtained by multiple first image data and a second image, and the second image is obtained after classification, and the environment of the first image and the second image are inconsistent, but the data category is the same. Therefore, when classifying images in different environments, the image data model performs more stably, can quickly adapt to the data features in the new environment, and reduce the impact of environmental features on image classification. Since the impact of different environmental features of the support set and the query set on graphic classification can be solved, the problem of low accuracy of image classification based on causal reasoning can be solved, thereby achieving the technical effect of improving the accuracy of image classification based on causal reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a schematic diagram of an application environment of an image classification method based on causal reasoning according to an embodiment of the present application;

[0024] Figure 2 is a flow chart of an image classification method based on causal reasoning according to an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of an image classification model training method based on causal reasoning according to an embodiment of the present application;

[0026] Figure 4 is a schematic diagram of an image classification model training method based on causal reasoning according to an embodiment of the present application;

[0027] Figure 5 It is a structural block diagram of an image classification device based on causal reasoning in an embodiment of the present application.

[0028] Figure 6 It is a structural block diagram of an image classification model training device based on causal reasoning in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0031] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 11 is a hardware structure block diagram of a server device of an image classification method based on causal reasoning in an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned server device may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0032] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the image classification method based on causal reasoning in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0033] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0034] In this embodiment, a causal reasoning-based image classification method is provided. Figure 2 is a flowchart of an image classification method based on causal reasoning according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0035] Step S202, obtaining image data to be classified;

[0036] In an optional embodiment, the image data to be classified may refer to, but is not limited to, an input image for which the category needs to be determined by an image classification model.

[0037] Step S204, inputting the image data into a trained image classification model, wherein the trained image classification model is an image classification model trained using a sample support set and a sample query set, the image classification model is a neural network model that performs image classification through causal reasoning, the sample support set is a sample set constructed using a plurality of first image data, the sample query set is a sample set constructed using a plurality of first image data and a plurality of second image data, the plurality of first image data is a plurality of image data located in a first environment obtained after classification of a plurality of image sample data, the plurality of second image data is a plurality of image data located in a second environment obtained after classification of a plurality of image sample data, and the category of image data in the sample support set is the same as the category of image data in the sample query set;

[0038] In an optional embodiment, the image classification model may be, but is not limited to, a model that has the ability to accurately classify image data, and may be, but is not limited to, a model that can identify features in the image that have a direct causal impact on the classification result.

[0039] In an optional embodiment, the sample support set may be, but is not limited to, a data set used to help an image classification model quickly learn and adapt to new tasks, may be, but is not limited to, consisting of multiple image data from a specific environment, such as a first environment, and may be, but is not limited to, containing core information and invariant features of a specific category.

[0040] In an optional embodiment, the sample query set may be, but is not limited to, a sample data set used to evaluate the learning effect and generalization ability of an image classification model, and may be, but is not limited to, used to test the robustness and generalization performance of an image classification model in the face of environmental changes.

[0041] In an optional embodiment, the image data is input into a trained image classification model, wherein the trained image classification model is an image classification model trained using a sample support set and a sample query set, and the image classification model is a neural network model that performs image classification through causal reasoning, which can be understood, but not limited to, as classification performed by inputting the image data to be classified into an image classification model trained through meta-learning and causal reasoning, and the training of the model depends on the sample support set and the sample query set, wherein the sample support set is used for model learning core features, the sample query set is used to evaluate the generalization ability of the image classification model, and causal reasoning ensures that the model can distinguish and utilize causal relationships, rather than superficial associations, to improve classification performance on out-of-distribution data.

[0042] It should be noted that the first image data and the second data in this embodiment are only for illustration. In fact, when training the model, the data used are multiple image data. That is, in addition to the first image data and the second image data, there may also be third image data, fourth image data and other data, which are not limited here.

[0043] Step S206, obtaining an image classification result output by the trained image classification model, wherein the image classification result is used to indicate the image data category to which the image data belongs.

[0044] In an optional embodiment, the image classification result may be, but is not limited to, a prediction result outputted after a trained image classification model processes input image data to be classified, indicating that the image belongs to a specific category.

[0045] It should be noted that in order to improve the classification ability of the model on data with different distributions. First of all, the image data to be classified is the input of the whole process. These data may contain changes and confusing features in various environments. Next, the trained image classification model is used to classify these images. The model has the ability to accurately classify out-of-distribution data through the training of sample support sets and sample query sets, combined with the guidance of causal reasoning. Secondly, the sample support set and sample query set come from the image data of the first environment and the second environment after environmental division, respectively, which enables the image classification model to make accurate classification decisions under different environmental conditions, improves the ability of the image classification model to process out-of-distribution data, and thus improves the accuracy of image classification.

[0046] Through the above steps, image data to be classified is obtained; the image data is input into a trained image classification model, wherein the trained image classification model is an image classification model trained using a sample support set and a sample query set, the image classification model is a neural network model that performs image classification through causal reasoning, the sample support set is a sample set constructed using a plurality of first image data, the sample query set is a sample set constructed using a plurality of first image data and a plurality of second image data, the plurality of first image data is a plurality of image data located in a first environment after classifying a plurality of image sample data, the plurality of second image data is a plurality of image data located in a second environment after classifying a plurality of image sample data, the image data category in the sample support set is the same as the image data category in the sample query set; the image classification result output by the trained image classification model is obtained, wherein the image classification result is used to indicate the image data category to which the image data belongs.

[0047] Specifically, after obtaining the image data to be classified, the image data is input into the image classification model, and the image classification model is obtained by a sample support set and a sample query set, and the sample support set is obtained by multiple first images, and the sample query set is obtained by multiple first image data and multiple second image data, and the second image is obtained after classification, and the environment of the first image is inconsistent with that of the second image, but the data category is the same. Therefore, when classifying images in different environments, the image data model performs more stably, can quickly adapt to the data features in the new environment, and reduce the impact of environmental features on image classification. The technical purpose of being able to solve the impact of different environmental features of the support set and the query set on graphic classification is achieved, thereby achieving the technical effect of improving the accuracy of image classification based on causal reasoning.

[0048] The execution subject of the above steps may be a server, a terminal, etc., but is not limited thereto.

[0049] In this embodiment, a method for training an image classification model based on causal reasoning is also provided. The following is a flow chart of the method for training an image classification model based on causal reasoning according to an embodiment of the present application. The flow chart includes the following steps:

[0050] Step 1, obtaining multiple image sample data;

[0051] Step 2, classifying the plurality of image sample data to obtain a plurality of first image data located in the first environment and a plurality of second image data located in the second environment;

[0052] Step 3, constructing a sample support set using a plurality of first image data;

[0053] Step 4, constructing a sample query set using the plurality of first image data and the plurality of second image data, wherein the image data category in the sample support set is the same as the image data category in the sample query set;

[0054] Step 5: Train the initial image classification model through the sample support set and the sample query set to obtain a trained image classification model, wherein the image classification model is a neural network model that performs image classification through causal reasoning.

[0055] In an optional embodiment, the image sample data may refer to, but is not limited to, images used for training and testing image classification models, and may come from, but is not limited to, different environments and conditions.

[0056] In an optional embodiment, the first environment and the second environment may be, but are not limited to, two different environment categories separated from a plurality of image sample data by a specific division method.

[0057] It should be noted that multiple image sample data for training and testing are obtained from various data sources, and these image sample data cover various environments and conditions. Then, by classifying the original image sample data, the image sample data is divided into the first image data and the second image data representing different environmental features, which provides a data basis for constructing the sample support set and the sample query set to enhance the generalization ability and robustness of the image classification model. Then, the sample support set is constructed using the image data in the first environment, which provides a relatively stable environment for the image classification model to learn the core causal features; and the sample query set is constructed using the image data of the first image data and the second image data, which can be used to test the generalization ability of the image classification model. Finally, the image classification model is iteratively trained according to the sample support set and the sample query set. During the training process, based on the framework of meta-learning, through preliminary learning on the sample support set and evaluation of the generalization ability on the sample query set, combined with causal reasoning, the image classification model can accurately classify out-of-distribution data, so as to improve the efficiency of the image classification model in classifying out-of-distribution data.

[0058] It should be noted that the first image data and the second data in this embodiment are only for illustration. In fact, when training the model, the data used are multiple image data. That is, in addition to the first image data and the second image data, there may also be third image data, fourth image data and other data, which are not limited here.

[0059] Through the embodiment of the present application, multiple image sample data are obtained; the multiple image sample data are classified to obtain multiple first image data located in the first environment and multiple second image data located in the second environment; a sample support set is constructed using the multiple first image data; a sample query set is constructed using the multiple first image data and the multiple second image data, wherein the image data category in the sample support set is the same as the image data category in the sample query set; the initial image classification model is trained through the sample support set and the sample query set to obtain a trained image classification model, wherein the image classification model is a neural network model that performs image classification through causal reasoning. Thus, the technical purpose of enabling the image classification model to accurately classify out-of-distribution data is achieved, thereby achieving the technical effect of improving the efficiency of the image classification model in classifying out-of-distribution data.

[0060] As an optional solution, the initial image classification model is trained through the sample support set and the sample query set to obtain a trained image classification model, including:

[0061] S1-1, updating the first parameter of the feature extraction module in the initial image classification model through the sample support set until the first convergence condition is met, thereby obtaining an updated feature extraction module, wherein the feature extraction module is used to capture and represent causal features;

[0062] S1-2, updating the second parameter of the image classification module in the initial image classification model through the sample query set until the second convergence condition is met, thereby obtaining an updated image classification module, wherein the image classification module is used to classify images using causal features.

[0063] In an optional embodiment, the feature extraction module may be, but is not limited to, a module for capturing and representing causal features.

[0064] In an optional embodiment, the first parameter may be, but is not limited to, a trainable parameter of a feature extraction module, and the second parameter may be, but is not limited to, a trainable parameter of an image classification module.

[0065] In an optional embodiment, the first convergence condition may be, but is not limited to, a standard for determining whether the model has achieved a preset learning objective during the training of the feature extraction module, and may be, but is not limited to, be related to the degree of decrease of the loss function and the number of training rounds.

[0066] In an optional embodiment, after iterative training, when the model reaches the first convergence condition, the feature extraction module is updated to a more optimal state, and can more accurately extract causal features that are invariant across environments from the input image, thereby improving the accuracy and robustness of the image classification model in performing image classification.

[0067] It should be noted that the second convergence condition can be, but is not limited to, used to evaluate the performance of the image classification module on sample queries. Unlike the first convergence condition, the second convergence condition focuses more on the generalization ability of the image classification model on out-of-distribution data. When the model reaches the second convergence condition, it indicates that the image classification module can accurately classify images in different environments.

[0068] On the whole, the first parameter of the feature extraction module is updated through iterative training on the sample support set to capture and represent the causal features that are invariant across environments, ensuring that the model can extract core features that are beneficial to classification from the image. When the feature extraction module reaches the first convergence condition, it means that the feature extraction module can effectively capture the causal features. Then, the image classification model updates the second parameter of the image classification module through training on the sample query set until the second convergence condition is met. The sample query set contains more out-of-distribution images with environmental changes and confusing features, which can verify and improve the generalization ability of the image classification model when processing out-of-distribution data and improve the accuracy of the image classification module when processing out-of-distribution images.

[0069] Through the embodiment of the present application, the first parameter of the feature extraction module in the initial image classification model is updated through the sample support set until the first convergence condition is met, and an updated feature extraction module is obtained, wherein the feature extraction module is used to capture and represent causal features; the second parameter of the image classification module in the initial image classification model is updated through the sample query set until the second convergence condition is met, and an updated image classification module is obtained, wherein the image classification module is used to classify images using causal features. Thus, the technical purpose of being able to verify and improve the generalization ability of the image classification model when processing out-of-distribution data is achieved, and then the technical effect of improving the accuracy of the image classification module when processing out-of-distribution images is achieved.

[0070] As an optional solution, the first parameter of the feature extraction module in the initial image classification model is updated through the sample support set until the first convergence condition is met to obtain an updated feature extraction module, including:

[0071] Perform the following steps until you get an updated feature extraction module:

[0072] S2-1, obtaining a current feature extractor represented by a first sub-parameter and a current causal classifier represented by a second sub-parameter, wherein the first parameter includes a first sub-parameter and a second sub-parameter, and the feature extraction module includes a feature extractor and a causal classifier;

[0073] S2-2, using the current feature extractor, extracting features from the image data in the sample support set to obtain a current feature vector, wherein the feature vector includes a causal feature;

[0074] S2-3, using the current causal classifier, classify the current feature vector to obtain the current prediction result;

[0075] S2-4, using the current first loss function, measures the difference between the current prediction result and the data label carried by the image data in the sample support set;

[0076] S2-5, when the current first loss function does not satisfy the first convergence condition, obtaining the updated first sub-parameter and the updated second sub-parameter, and using the feature extractor represented by the updated first sub-parameter as the current feature extractor, and using the causal classifier represented by the updated second sub-parameter as the current causal classifier;

[0077] S2-6, when the current first loss function satisfies the first convergence condition, the current feature extraction module is determined as the updated feature extraction module.

[0078] In an optional embodiment, the first sub-parameter may be, but is not limited to, used to represent the current feature extractor, and the second sub-parameter may be, but is not limited to, representing the current causal classifier.

[0079] In an optional embodiment, the feature vector may be, but is not limited to, a vector extracted from image data by a feature extractor and used to represent image special effects, and may include, but is not limited to, causal features.

[0080] In an optional embodiment, the first loss function may be, but is not limited to, a function for measuring the difference between the current prediction result and the actual label of the image data in the sample support set. The image classification model may, but is not limited to, learn and optimize the first sub-parameter and the second sub-parameter by minimizing the first loss function to make the prediction result more accurate, thereby improving the accuracy of the image classification model in image classification.

[0081] It should be noted that the feature extractor and causal classifier in the current state of the image classification model, as well as their corresponding first sub-parameters and second sub-parameters, are obtained. Next, the feature extractor represented by the current first sub-parameter is used to extract features from the image data in the sample support set to obtain feature vectors, and then the feature vectors are classified by the causal classifier represented by the second sub-parameter to obtain preliminary prediction results. Then, the first loss function is used to measure the difference between the current prediction result and the actual label. If the first loss function does not meet the convergence condition, the first sub-parameter and the second sub-parameter are updated, and then the steps of feature extraction and causal classification are repeated according to the updated parameters until the first loss function meets the convergence condition. Finally, when the first loss function meets the convergence condition, it means that the feature extractor can extract key features from out-of-distribution data, that is, through iterative training, the generalization ability of the image classification model is improved, and then the accuracy of the image classification model can be improved.

[0082] Through the embodiment of the present application, the first sub-parameter is used to represent the current feature extractor, and the second sub-parameter is used to represent the current causal classifier, wherein the first parameter includes the first sub-parameter and the second sub-parameter, and the feature extraction module includes the feature extractor and the causal classifier; the current feature extractor is used to extract the image data in the sample support set to obtain the current feature vector, wherein the feature vector includes the causal feature; the current causal classifier is used to classify the current feature vector to obtain the current prediction result; the current first loss function is used to measure the difference between the current prediction result and the data label carried by the image data in the sample support set; when the current first loss function does not meet the first convergence condition, the updated first sub-parameter and the updated second sub-parameter are obtained, and the feature extractor represented by the updated first sub-parameter is used as the current feature extractor, and the causal classifier represented by the updated second sub-parameter is used as the current causal classifier; when the current first loss function meets the first convergence condition, the current feature extraction module is determined as the updated feature extraction module. Thus, the technical purpose of improving the generalization ability of the image classification model is achieved, and the technical effect of being able to improve the accuracy of the image classification model is achieved.

[0083] As an optional solution, the second parameter of the image classification module in the initial image classification model is updated through the sample query set until the second convergence condition is met to obtain an updated image classification module, including:

[0084] Perform the following steps until you get an updated image classification module:

[0085] S3-1, obtaining a sample feature vector obtained by performing feature extraction on the image data in the sample query set by the updated feature extraction module;

[0086] S3-2, inputting the sample feature vector into the current image classification module to obtain the current classification result output by the current image classification module, wherein the image classification module is obtained by the representation of the second parameter;

[0087] S3-3, using the current second loss function, measures the difference between the loss current classification result and the data label carried by the image data in the sample query set;

[0088] S3-4, when the current second loss function does not satisfy the second convergence condition, obtaining an updated second parameter, and using the image classification module represented by the updated second parameter as the current image classification module;

[0089] S3-5, when the current second loss function satisfies the second convergence condition, the current image classification module is determined as the updated image classification module.

[0090] In an optional embodiment, the sample feature vector may be, but is not limited to, a feature extracted from the sample query set by an updated feature extraction module.

[0091] In an optional embodiment, the second loss function may be, but is not limited to, used to measure the difference between the current classification result and the actual label of the image data in the sample query set.

[0092] In an optional embodiment, the second convergence condition may be, but is not limited to, referring to a process in which the image classification module is trained, and may be, but is not limited to, used to indicate whether the image classification module can perform accurate classification.

[0093] In general, first, by obtaining the image data in the sample query set, the sample feature vectors are extracted using the updated feature extraction module. Then, these sample feature vectors are input into the current image classification module to obtain preliminary classification results. Then, the difference between the classification result and the actual label is evaluated according to the second loss function. If the second loss function does not meet the second convergence condition, it means that the classification performance of the image classification module on the out-of-distribution data does not meet the requirements, and the second parameter needs to be updated until the image classification module achieves the expected classification performance on the out-of-distribution data. When the second loss function meets the second convergence condition, it is considered that the image classification module can effectively classify the out-of-distribution data and achieve the expected effect. That is, through iterative training, it is ensured that the image classification module can make accurate classifications when processing out-of-distribution data, thereby improving the accuracy of the image classification model when processing out-of-distribution data.

[0094] Through the embodiment of the present application, the updated feature extraction module is obtained to extract the image data in the sample query set, and the sample feature vector is obtained after the feature extraction; the sample feature vector is input into the current image classification module to obtain the current classification result output by the current image classification module, wherein the image classification module is obtained by the representation of the second parameter; the difference between the loss of the current classification result and the data label carried by the image data in the sample query set is measured by the current second loss function; when the current second loss function does not meet the second convergence condition, the updated second parameter is obtained, and the image classification module represented by the updated second parameter is used as the current image classification module; when the current second loss function meets the second convergence condition, the current image classification module is determined as the updated image classification module. Thus, the technical purpose of ensuring that the image classification module can make accurate classification when processing out-of-distribution data is achieved, thereby improving the technical effect of the accuracy of the image classification model when processing out-of-distribution data.

[0095] As an optional solution, before classifying the plurality of image sample data to obtain the plurality of first image data located in the first environment and the plurality of second image data located in the second environment, the method further includes:

[0096] S4-1, obtaining a plurality of confusion features corresponding to each of the plurality of image sample data, wherein the confusion features are features that affect the generalization ability of the image classification model;

[0097] S4-2, using multiple confusion features to train a deviation classifier, wherein the deviation classifier is used to identify and classify deviations caused by the confusion features;

[0098] Classifying a plurality of image sample data to obtain a plurality of first image data located in a first environment and a plurality of second image data located in a second environment, comprising:

[0099] S4-3, using a deviation classifier, classify multiple image sample data to obtain multiple first image data and multiple second image data.

[0100] In an optional embodiment, the confusing features may be, but are not limited to, features that vary in different environments but are irrelevant to or have little impact on image classification, and may include, but are not limited to, image background, lighting conditions, etc.

[0101] In an optional embodiment, the deviation classifier may be, but is not limited to, a model for identifying and classifying confusing features in image data.

[0102] It should be noted that the recognition of confusing features is based on the image classification model's understanding of image data. These confusing features may affect the generalization ability of the model due to changes in environmental conditions, such as background, lighting, etc. After the bias classifier is trained using the confusing features, the bias classifier can be used to identify the classification bias caused by confusing features in image data, helping the image classification model to understand and segment these confusing features that are not conducive to image classification, thereby improving the classification ability of the image classification model on out-of-distribution data.

[0103] Through the embodiment of the present application, multiple confusion features corresponding to each of the multiple image sample data are obtained, wherein the confusion features are features that affect the generalization ability of the image classification model; the multiple confusion features are used to train a deviation classifier, wherein the deviation classifier is used to identify and classify deviations caused by the confusion features; the deviation classifier is used to classify the multiple image sample data to obtain multiple first image data and multiple second image data. Thus, the technical purpose of helping the image classification model understand and segment these confusion features that are not conducive to image classification is achieved, thereby improving the technical effect of the classification ability of the image classification model on out-of-distribution data.

[0104] As an optional solution, a deviation classifier is used to classify a plurality of image sample data to obtain a plurality of first image data and a plurality of second image data, including:

[0105] S5-1 uses a deviation classifier to optimize the data distribution in the initial partition matrix to obtain an optimized partition matrix, wherein the partition matrix is ​​used to guide the partition of data into different environments, and the initial partition matrix is ​​a randomly generated matrix that conforms to the normal distribution;

[0106] S5-2, using the optimized division matrix, classify the multiple image sample data to obtain multiple first image data and multiple second image data.

[0107] In an optional embodiment, the partition matrix may be, but is not limited to, a data result in a matrix form, and may be, but is not limited to, used to guide the partitioning of data into different environments.

[0108] It should be noted that the initial partition matrix is ​​randomly generated and conforms to the normal distribution, but it lacks consideration of the confusing features in the image data and may not accurately guide the data partitioning. Through the training of the deviation classifier, the model can identify the confusing features in the image data, which are caused by environmental changes and may have a negative impact on the generalization ability of the model. The optimization of the deviation classifier adjusts the data distribution in the partition matrix to ensure that the image data in each partition can represent different environmental features. At the same time, it also maintains the balance of the partition, avoids the imbalance of the number of images in different partitions, and ensures that each environment has enough representative data for subsequent model training and testing. Using the optimized partition matrix for classification, multiple first image data and multiple second image data can be obtained. These two sets of data are used to construct the sample support set and the sample query set, respectively, which are the basis for model training and performance evaluation under the meta-learning framework. This optimization process is an important part of the out-of-distribution image classification method based on causal learning. It refines and balances the data environment partition to ensure that the image classification model can learn the causal features that are invariant across environments, thereby improving the generalization ability of the image classification model when processing out-of-distribution data.

[0109] Through the embodiment of the present application, the data distribution in the initial partition matrix is ​​optimized using a deviation classifier to obtain an optimized partition matrix, wherein the partition matrix is ​​used to guide the partitioning of data into different environments, and the initial partition matrix is ​​a randomly generated matrix that conforms to the normal distribution; the optimized partition matrix is ​​used to classify multiple image sample data to obtain multiple first image data and multiple second image data. Thus, the technical purpose of ensuring that the image classification model can learn causal features that are invariant across environments is achieved, thereby improving the technical effect of the generalization ability of the image classification model when processing out-of-distribution data.

[0110] S6-1, as an optional solution, uses a deviation classifier to optimize the data distribution in the initial partition matrix to obtain an optimized partition matrix, including:

[0111] S6-2, obtain the empirical risk minimization loss and the gradient norm penalty, wherein the empirical risk minimization loss is used to reflect the performance of the deviation classifier on each sample data set, and the gradient norm penalty is used to measure the optimality of the deviation classifier on each sample data set;

[0112] S6-3, adjust the data distribution in the initial partition matrix until the sum of the empirical risk minimization loss and the gradient norm penalty is maximized to obtain the optimized partition matrix.

[0113] In an optional embodiment, the empirical risk minimization loss can be, but is not limited to, an indicator for evaluating the performance of the deviation classifier on the training set, and can be, but is not limited to, used to reflect the gap between the prediction results of the deviation classifier on different sample data sets and the actual labels.

[0114] In an optional embodiment, the gradient norm penalty may be, but is not limited to, an additional loss term, and may be, but is not limited to, used to measure the difficulty of the deviation classifier reaching the optimal state on each sample data set.

[0115] It should be noted that the empirical risk minimization loss reflects the prediction performance of the deviation classifier on the sample data set under each environmental partition, while the gradient norm penalty is used to ensure the generalization ability and stability of the deviation classifier under different environmental distributions, and to avoid overfitting a specific environmental feature by measuring the difficulty of the deviation classifier to reach the optimal state. The image classification model adjusts the data distribution in the partition matrix to maximize the sum of the empirical risk minimization loss and the gradient norm penalty, ensuring that the deviation classifier has good generalization performance and stability under each environmental partition, while avoiding too few or too many samples in certain environments, thereby achieving a more fine-grained and balanced environmental partition, thereby improving the generalization ability of the image classification model when processing out-of-distribution data.

[0116] Through the embodiment of the present application, as an optional solution, the data distribution in the initial partition matrix is ​​optimized using a deviation classifier to obtain an optimized partition matrix, including: obtaining an empirical risk minimization loss and a gradient norm penalty, wherein the empirical risk minimization loss is used to reflect the performance of the deviation classifier on each sample data set, and the gradient norm penalty is used to measure the optimality of the deviation classifier on each sample data set; adjusting the data distribution in the initial partition matrix until the sum of the empirical risk minimization loss and the gradient norm penalty is maximized, and obtaining the optimized partition matrix. Thereby achieving the technical purpose of more fine-grained and more balanced environmental partitioning, and then improving the technical effect of the generalization ability of the image classification model when processing out-of-distribution data.

[0117] As an optional solution, in the process of optimizing the data distribution in the initial partition matrix by using the deviation classifier to obtain the optimized partition matrix, the method further includes:

[0118] S7-1, obtaining a first function and a second function, wherein the first function is used to focus on the quantity imbalance of different data partition intervals, and the second function is used to focus on maintaining a minimum adjustment during the data partitioning process;

[0119] S7-2, using the first function and the second function as a balance loss function in the process of optimizing the data distribution in the initial partition matrix;

[0120] S7-3, constrains the optimization process of the data distribution in the initial partition matrix by balancing the loss function.

[0121] In an optional embodiment, the balanced loss function may be, but is not limited to, a combination of the first function and the second function, and may be, but is not limited to, used to constrain data distribution in the process of optimizing the partition matrix.

[0122] It should be noted that in the process of optimizing the partition matrix, the first function focuses on the quantitative imbalance of different data partition intervals to ensure that each interval has sufficient representative data, which helps the model to fully learn and understand the causal relationship under different environmental features in subsequent training and improve the generalization ability of the model. The second function focuses on the minimum adjustment in the data partition process. By limiting the degree of modification to the initial partition matrix, the stability and effectiveness of the data partition are ensured, unnecessary changes in data distribution are avoided, and the randomness in the data partition optimization process is prevented from affecting the effectiveness of data partition. Combining the first and second functions, a balanced loss function is formed. It serves as a constraint in the optimization process to guide the model to pursue quantitative balance when adjusting the data distribution in the partition matrix, ensuring that each environmental feature has sufficient data for subsequent training, and maintaining the minimum adjustment of the partition process to avoid data distribution distortion caused by excessive modification of the initial partition matrix. By introducing the balanced function, a more refined and stable environmental partition process is provided for the image classification model, which helps to improve the classification performance and stability of the model on out-of-distribution data.

[0123] Through the embodiment of the present application, the first function and the second function are obtained, wherein the first function is used to focus on the quantitative imbalance of different data partition intervals, and the second function is used to focus on maintaining the minimum adjustment in the data partition process; the first function and the second function are used as the balance loss function in the process of optimizing the data distribution in the initial partition matrix; the optimization process of the data distribution in the initial partition matrix is ​​constrained by the balance loss function. Thus, the technical purpose of providing a more refined and stable environment partition process for the image classification model is achieved, and then the technical effect of helping to improve the classification performance and stability of the model on out-of-distribution data is achieved.

[0124] As an optional solution, the initial image classification model is trained through the sample support set and the sample query set to obtain a trained image classification model, including:

[0125] S8-1, iteratively train the initial image classification model through the sample support set and the sample query set to obtain the optimal round of image classification model;

[0126] S8-2, using the image data in the sample test set, to test the optimal round image classification model to obtain a test result, wherein the test result is used to measure whether the optimal round image classification model is a trained image classification model.

[0127] In an optional embodiment, the optimal round image classification model may be, but is not limited to, a model that achieves the best classification performance during the iterative training process.

[0128] In an optional embodiment, the sample test set may be, but is not limited to, a set of independent out-of-distribution data used to evaluate the generalization ability of an image classification model.

[0129] It should be noted that in the specific steps of iteratively training the image classification model using the meta-learning framework to obtain the optimal round of image classification model, a sample test set is also used to evaluate the model performance to determine whether the image classification model has been trained, thereby ensuring that the image classification model can quickly adapt to the new environment and improving the classification performance of the image classification model when processing out-of-distribution data, thereby improving the generalization ability of the image classification model.

[0130] Through the embodiment of the present application, the initial image classification model is iteratively trained through the sample support set and the sample query set to obtain the optimal round of image classification model; the image classification model of the optimal round is tested using the image data in the sample test set to obtain the test result, wherein the test result is used to measure whether the image classification model of the optimal round is used as a trained image classification model. Thus, the technical purpose of improving the classification performance of the image classification model when processing out-of-distribution data is achieved, and the technical effect of improving the generalization ability of the image classification model is achieved.

[0131] As an optional solution, the above-mentioned causal reasoning-based image classification method is applied in the scenario of deep learning image classification algorithm.

[0132] In an optional embodiment, the following steps are mainly divided into: 1. Extracting confusion features. 2. Dividing the data environment. 3. Generating meta-tasks. 4. Training the model to obtain causal features. 5. Outputting the image classification results. The model framework flow chart and model architecture diagram are shown in Figure 3 , Figure 4 As shown. Among them, Figure 3 The specific process is as follows:

[0133] First extract the confusion features, and then divide the data environment {t1, t2, ..., t m}, then generate meta-tasks, obtain causal features through the trained model, and finally output the image classification results.

[0134] To further illustrate, the optional Figure 4 The following is a model architecture diagram:

[0135] An out-of-distribution (OOD) data set is input into the model, and then the confusion features are obtained through the model. Then, according to the deviation classifier, a partition matrix is ​​obtained, where the partition matrix can be but not limited to partition 1, partition 2, and partition 3, and can also be but not limited to other partitions. Then, the support set is obtained according to partition 1, and the query set is obtained by performing data weighting according to partitions 1 and 2; then, the support set is input into the model to obtain causal features, and then the loss is obtained based on the causal features, and then the loss is gradient-passed back to perform a one-step gradient descent; the query set is input into the model to obtain causal features, and then the loss is obtained based on the causal features, and then the loss is gradient-passed back to the model, where the model input by the support set and the model input by the query set are the same model.

[0136] In an optional embodiment, the specific steps are as follows:

[0137] (1) Extracting confusing features: First, this embodiment proposes a balanced environment division method. Compared with the rough division by humans, this embodiment hopes to automatically divide the data of different environments by features, so as to obtain a sufficiently complex environment and better implement causal intervention. Specifically, features are extracted through deep learning models and attention modules. In this embodiment, the classic image classification model ResNet-18 is used as the baseline, and the attention uses a lightweight CBAM module. This embodiment adds two attention modules after the original feature extractor of the model. and , respectively used to obtain causal features and confusion features :

[0138] ,

[0139] ,

[0140] in represents the sigmoid function, represents the input image, represents the initial features obtained by the model's feature extractor, Represents dot product. The causal features and the confounding features are decoupled through model training. It is generally believed that the confounding features are the part that affects the generalization ability of the model, while the causal features are the part that is beneficial to the model in predicting the correct results.

[0141] (2) Dividing the data environment: This embodiment uses obfuscation features Train a bias classifier and minimize its cross-entropy loss (CE Loss):

[0142] ,

[0143] in represents the training environment, is the training data, is a bias classifier, is a confusing feature, It's a label.

[0144] Next, this example freezes the weights of the model and the classifier, and uses the model and the bias classifier To optimize the partition matrix, the partition matrix is ​​initially a randomly generated matrix that conforms to the normal distribution. This step uses the invariant risk minimization (IRM Loss) as a constraint, and optimizes the data distribution in the partition matrix by maximizing the loss, so that the environmental difference between different partitions guided by the partition matrix is ​​maximized.

[0145] ,

[0146] in, is the current data partition, Representation Division Depend on Decide, is the total number of training samples, is the number of partitioned subsets, Is a subset ERM loss under is the frozen trained bias classifier, is a scalar and dummy classifier, the gradient norm penalty is used to measure the dummy classifier in each subset The optimality under It is a regularizer that balances the cross entropy loss and the invariance of the predictor.

[0147] Since the partition matrix is ​​updated by the loss constraint, in order to avoid uncertainty and imbalance caused by the partition matrix update, such as a large difference in the number of images between different partitions, this embodiment further designs a balance loss to constrain it:

[0148] ,

[0149] in is the number of partitions, is the number of samples, is the number of samples in each partition, is the mean of the number of samples in the partition. represents the original partition matrix, Represents the updated partition matrix. The purpose of the first part of the balance loss is to reduce the imbalance of the number of different partitions, so that the number of images in the divided environment is as close as possible; the second part is to ensure that the balance is achieved with minimal adjustments, so as to avoid destroying the automatic training partition. Finally, this embodiment obtains the overall loss of the environment partition part, and minimizes the loss to obtain a balanced and fine-grained environment partition:

[0150] ,

[0151] (3) Generating meta-tasks: Assume that through the above steps, this embodiment divides the data set into three parts with different environments: ,Right now In the case of the three environments, the meta-tasks required for model training are further constructed from the current three environments. Usually, meta-learning divides tasks into support sets and query sets. The support sets and query sets have the same categories to help the model learn and converge quickly. Because causal intervention needs to be learned in different environments to obtain causal features that are invariant across environments. Based on the above principles, taking the dog category in the animal dataset as an example, this embodiment uses the environment Get An image of a dog, images as support set, images as part of the query set; and from the environment Get The image is the other part of the query set, where In order to enhance the generalization ability of the model, this embodiment uses the mixup image enhancement method to Each image in Get an image of the same category in the environment and interpolate it.

[0152] ,

[0153] ,

[0154] in is the probability value, ,Right now It follows the Beta distribution, and its parameters are Finally, a meta-task obtained in this embodiment is as follows:

[0155] ,

[0156] in represents the samples of the support set, The sample representing the query set. The query set contains two types of samples: one is the image that is consistent with the support set environment. , which is used to consolidate the classification ability of the model under known distribution; the other type is the image obtained after mixup interpolation that is different from the support set environment , which is used to improve the generalization ability of the model under distribution changes. and , and The middle loop constructs multiple meta-tasks for iterative learning of subsequent models.

[0157] (4) Training the model and obtaining causal features: In this step, model training and causal feature extraction are mainly completed. Meta-learning can enable the model to learn a wider range of features by simulating the training process of multiple tasks, thereby improving its adaptability to new tasks and new distributions. In out-of-distribution image classification tasks, the meta-learning model can quickly adapt to a small amount of data or a new distribution, thereby improving classification performance. Therefore, this part of this embodiment adopts a two-step gradient descent training strategy similar to meta-learning. This design aims to gradually expose the model to different distributions so that it can maintain high classification performance in different environments.

[0158] The model is first learned on the support set and then updated. The feature extractor is defined as ,parameter Define the classifier as Using feature extractor and causal classifiers To obtain causal features , and then the cross entropy loss of the first model update and the model after gradient descent are obtained:

[0159] ,

[0160] ,

[0161] in and It's a task Supports images and labels in the set, represents the function of cross entropy loss, is the step length. and is the updated parameter vector, updated using one or more gradient descents on the support set for the current loss computation.

[0162] Next, the updated model is used to train on the query set and obtain the training loss. According to the number of partitions, this embodiment treats m meta-tasks as a group:

[0163] ,

[0164] Next, this embodiment uses this loss to optimize the meta-model. Meta-optimization is the initial model parameters of the current task. Executed on:

[0165] ,

[0166] in is the meta-step size. By iterating on multiple tasks and continuously minimizing the sum of losses of all tasks, the model can accurately extract causal features, which helps improve the generalization ability of the model when facing new distributions.

[0167] The above is only an embodiment of the present embodiment patent, but the protection scope of the present embodiment patent is not limited thereto. Any technician familiar with the technical field can make equivalent substitutions or changes within the scope disclosed by the present embodiment patent according to the technical solution of the present embodiment patent and its invention patent concept, which are all within the protection scope of the present embodiment patent.

[0168] (5) Output image classification results: Load the best-round trained model and classifier, make predictions on the test machine set, and decide whether to continue training based on the prediction results.

[0169] Through the embodiments of the present application, the problem that traditional models are susceptible to environmental bias is solved through an out-of-distribution image classification method and system based on causal learning. Compared with traditional models, this embodiment can have stronger robustness in the face of complex real-world environments, and can also have certain learning capabilities for some small sample data in reality. It can be widely used in common image classification, judgment of monitoring images, pedestrian re-identification and other tasks. By iteratively training bias attention, bias classifiers and partitioning matrices, balanced data partitions with different environmental characteristics are automatically generated in a fine-grained manner; meta-tasks of different environments are constructed based on the partitions, and the learning of cross-environmental causal features is strengthened through meta-learning, thereby improving the performance and stability of the model when facing complex environments.

[0170] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0171] In this embodiment, a causal reasoning-based image classification device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0172] Figure 5 is a structural block diagram of an image classification device based on causal reasoning according to an embodiment of the present application, such as Figure 5 As shown, the device comprises:

[0173] An image acquisition unit 502, used to acquire image data to be classified;

[0174] An input unit 504 is used to input image data into a trained image classification model, wherein the trained image classification model is an image classification model trained using a sample support set and a sample query set, the image classification model is a neural network model that performs image classification through causal reasoning, the sample support set is a sample set constructed using a plurality of first image data, the sample query set is a sample set constructed using a plurality of first image data and a plurality of second image data, the plurality of first image data is a plurality of image data located in a first environment obtained by classifying a plurality of image sample data, the plurality of second image data is a plurality of image data located in a second environment obtained by classifying a plurality of image sample data, and the category of image data in the sample support set is the same as the category of image data in the sample query set;

[0175] The first acquisition unit 506 is used to acquire the image classification result output by the trained image classification model, wherein the image classification result is used to indicate the image data category to which the image data belongs.

[0176] For specific embodiments, reference may be made to the examples shown in the above-mentioned image classification method based on causal reasoning, which will not be described in detail in this example.

[0177] In this embodiment, a causal reasoning-based image classification model training device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0178] Figure 6 is a structural block diagram of an image classification model training device based on causal reasoning according to an embodiment of the present application, such as Figure 6 As shown, the device comprises:

[0179] The second acquisition unit 602 is used to acquire a plurality of image sample data;

[0180] A classification unit 604, configured to classify the plurality of image sample data to obtain a plurality of first image data located in the first environment and a plurality of second image data located in the second environment;

[0181] The first construction unit 606 is used to construct a sample support set using the plurality of first image data; the second construction unit is used to construct a sample query set using the plurality of first image data and the plurality of second image data, wherein the image data category in the sample support set is the same as the image data category in the sample query set;

[0182] The training unit 608 is used to train the initial image classification model through the sample support set and the sample query set to obtain a trained image classification model, wherein the image classification model is a neural network model that performs image classification through causal reasoning.

[0183] For specific embodiments, reference may be made to the examples shown in the above-mentioned image classification model training method based on causal reasoning, which will not be described in detail in this example.

[0184] As an optional solution, the training unit includes: a first updating module, used to update the first parameter of the feature extraction module in the initial image classification model through the sample support set until the first convergence condition is met, and an updated feature extraction module is obtained, wherein the feature extraction module is used to capture and represent causal features; a second updating module, used to update the second parameter of the image classification module in the initial image classification model through the sample query set until the second convergence condition is met, and an updated image classification module is obtained, wherein the image classification module is used to classify images using causal features.

[0185] For specific embodiments, reference may be made to the examples shown in the above-mentioned image classification model training method based on causal reasoning, which will not be described in detail in this example.

[0186] As an optional solution, the first updating module includes: a first acquisition submodule, used to obtain a current feature extractor represented by a first subparameter, and a current causal classifier represented by a second subparameter, wherein the first parameter includes the first subparameter and the second subparameter, and the feature extraction module includes the feature extractor and the causal classifier; a first extraction submodule, used to use the current feature extractor to perform feature extraction on the image data in the sample support set to obtain a current feature vector, wherein the feature vector includes causal features; a first classification submodule, used to use the current causal classifier to classify the current feature vector to obtain a current prediction result; a first measurement submodule The first block is used to measure the difference between the current prediction result and the data label carried by the image data in the sample support set through the current first loss function; the second acquisition submodule is used to obtain the updated first sub-parameter and the updated second sub-parameter when the current first loss function does not meet the first convergence condition, and use the feature extractor represented by the updated first sub-parameter as the current feature extractor, and use the causal classifier represented by the updated second sub-parameter as the current causal classifier; the second extraction submodule is used to determine the current feature extraction module as the updated feature extraction module when the current first loss function meets the first convergence condition.

[0187] For specific embodiments, reference may be made to the examples shown in the above-mentioned image classification model training method based on causal reasoning, which will not be described in detail in this example.

[0188] As an optional solution, the second updating module includes: a third acquisition submodule, which is used to obtain a sample feature vector obtained after the updated feature extraction module performs feature extraction on the image data in the sample query set; an input submodule, which is used to input the sample feature vector into the current image classification module to obtain the current classification result output by the current image classification module, wherein the image classification module is obtained through the representation of the second parameter; a second measurement submodule, which is used to measure the difference between the current classification result and the data label carried by the image data in the sample query set through the current second loss function; a fourth acquisition submodule, which is used to obtain the updated second parameter when the current second loss function does not meet the second convergence condition, and use the image classification module represented by the updated second parameter as the current image classification module; a determination submodule, which is used to determine the current image classification module as the updated image classification module when the current second loss function meets the second convergence condition.

[0189] For specific embodiments, reference may be made to the examples shown in the above-mentioned image classification model training method based on causal reasoning, which will not be described in detail in this example.

[0190] As an optional solution, the classification unit includes: an acquisition module, used to acquire multiple confusion features corresponding to each image sample data in a plurality of image sample data, wherein the confusion feature is a feature that affects the generalization ability of the image classification model; a training module, used to use the multiple confusion features to train a deviation classifier, wherein the deviation classifier is used to identify and classify deviations caused by the confusion features; and a deviation classification module, used to use the deviation classifier to classify the multiple image sample data to obtain multiple first image data and multiple second image data.

[0191] For specific embodiments, reference may be made to the examples shown in the above-mentioned image classification model training method based on causal reasoning, which will not be described in detail in this example.

[0192] As an optional solution, the deviation classification module includes: an optimization submodule, which is used to use a deviation classifier to optimize the data distribution in an initial partitioning matrix to obtain an optimized partitioning matrix, wherein the partitioning matrix is ​​used to guide the partitioning of data into different environments, and the initial partitioning matrix is ​​a randomly generated matrix that conforms to a normal distribution; a second classification submodule, which is used to use the optimized partitioning matrix to classify multiple image sample data to obtain multiple first image data, and multiple second image data.

[0193] For specific embodiments, reference may be made to the examples shown in the above-mentioned image classification model training method based on causal reasoning, which will not be described in detail in this example.

[0194] As an optional solution, the optimization submodule includes: a first acquisition subunit, used to obtain the empirical risk minimization loss and the gradient norm penalty, wherein the empirical risk minimization loss is used to reflect the performance of the deviation classifier on each sample data set, and the gradient norm penalty is used to measure the optimality of the deviation classifier on each sample data set; an adjustment subunit, used to adjust the data distribution in the initial partition matrix until the sum of the empirical risk minimization loss and the gradient norm penalty is maximized, thereby obtaining the optimized partition matrix.

[0195] For specific embodiments, reference may be made to the examples shown in the above-mentioned image classification model training method based on causal reasoning, which will not be described in detail in this example.

[0196] As an optional solution, the optimization submodule includes: a second acquisition subunit, used to acquire a first function and a second function, wherein the first function is used to focus on the quantitative imbalance of different data partition intervals, and the second function is used to focus on maintaining minimal adjustments during the data partitioning process; a setting subunit, used to use the first function and the second function as a balanced loss function in the process of optimizing the data distribution in the initial partitioning matrix; a constraint subunit, used to constrain the optimization process of the data distribution in the initial partitioning matrix through the balanced loss function.

[0197] For specific embodiments, reference may be made to the examples shown in the above-mentioned image classification model training method based on causal reasoning, which will not be described in detail in this example.

[0198] As an optional solution, the training unit includes: an iterative training module, which is used to iteratively train the initial image classification model through a sample support set and a sample query set to obtain an optimal round of image classification model; a testing module, which is used to test the optimal round of image classification model using image data in a sample test set to obtain a test result, wherein the test result is used to measure whether the optimal round of image classification model is used as a trained image classification model.

[0199] For specific embodiments, reference may be made to the examples shown in the above-mentioned image classification model training method based on causal reasoning, which will not be described in detail in this example.

[0200] It should be noted that the above-mentioned virtual devices (modules, units, sub-modules, sub-units, components, etc.) can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to this: the above-mentioned virtual devices are all located in the same processor; or, the above-mentioned virtual devices are located in different processors in any combination.

[0201] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0202] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0203] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0204] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0205] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0206] An embodiment of the present application further provides a computer program product, including a computer program, wherein when the computer program is configured to be executed by a processor, the computer program performs the steps in any of the above method embodiments.

[0207] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0208] Obviously, those skilled in the art should understand that the above-mentioned virtual devices or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0209] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A causal reasoning-based image classification method, characterized in that: include: Obtain image data to be classified; Input the image data into a trained image classification model, wherein the trained image classification model is an image classification model trained using a sample support set and a sample query set, the image classification model is a neural network model that performs image classification through causal reasoning, the sample support set is a sample set constructed using a plurality of first image data, the sample query set is a sample set constructed using a plurality of first image data and a plurality of second image data, the plurality of first image data are a plurality of image data located in a first environment obtained after classifying a plurality of image sample data, the plurality of second image data are a plurality of image data located in a second environment obtained after classifying the plurality of image sample data, and the image data category in the sample support set is the same as the image data category in the sample query set; Obtaining an image classification result output by the trained image classification model, wherein the image classification result is used to indicate the image data category to which the image data belongs; The trained image classification model is obtained by training through the following steps: updating the first parameter of the feature extraction module in the initial image classification model through the sample support set until the first convergence condition is met, and obtaining an updated feature extraction module, wherein the feature extraction module is used to capture and represent causal features; updating the second parameter of the image classification module in the initial image classification model through the sample query set until the second convergence condition is met, and obtaining an updated image classification module, wherein the image classification module is used to perform image classification using the causal features.

2. A causal reasoning-based image classification model training method, characterized in that: include: Acquire multiple image sample data; Classifying the plurality of image sample data to obtain a plurality of first image data located in a first environment and a plurality of second image data located in a second environment; Using the plurality of first image data, constructing a sample support set; Using the plurality of first image data and the plurality of second image data, a sample query set is constructed, wherein the image data category in the sample support set is the same as the image data category in the sample query set; The initial image classification model is trained by using the sample support set and the sample query set to obtain a trained image classification model, wherein the image classification model is a neural network model that performs image classification by causal reasoning; Among them, the initial image classification model is trained through the sample support set and the sample query set to obtain a trained image classification model, including: updating the first parameter of the feature extraction module in the initial image classification model through the sample support set until the first convergence condition is met, and obtaining an updated feature extraction module, wherein the feature extraction module is used to capture and represent causal features; updating the second parameter of the image classification module in the initial image classification model through the sample query set until the second convergence condition is met, and obtaining an updated image classification module, wherein the image classification module is used to perform image classification using the causal features.

3. The method according to claim 2, characterized in that The method of updating a first parameter of a feature extraction module in the initial image classification model by using the sample support set until a first convergence condition is satisfied to obtain an updated feature extraction module comprises: Perform the following steps until the updated feature extraction module is obtained: Obtaining a current feature extractor represented by a first sub-parameter and a current causal classifier represented by a second sub-parameter, wherein the first parameter includes the first sub-parameter and the second sub-parameter, and the feature extraction module includes the feature extractor and the causal classifier; Using the current feature extractor, extract features from the image data in the sample support set to obtain a current feature vector, wherein the feature vector includes a causal feature; Using the current causal classifier, classify the current feature vector to obtain a current prediction result; Measuring the difference between the current prediction result and the data label carried by the image data in the sample support set by using the current first loss function; In the case that the current first loss function does not satisfy the first convergence condition, obtaining an updated first sub-parameter and an updated second sub-parameter, and using the feature extractor represented by the updated first sub-parameter as the current feature extractor, and using the causal classifier represented by the updated second sub-parameter as the current causal classifier; When the current first loss function satisfies the first convergence condition, the current feature extraction module is determined as the updated feature extraction module.

4. The method according to claim 2, characterized in that: The method of updating the second parameter of the image classification module in the initial image classification model by using the sample query set until a second convergence condition is satisfied to obtain an updated image classification module comprises: Perform the following steps until the updated image classification module is obtained: Obtaining a sample feature vector obtained by extracting features from the image data in the sample query set by the updated feature extraction module; Inputting the sample feature vector into a current image classification module to obtain a current classification result output by the current image classification module, wherein the image classification module is obtained through the representation of the second parameter; Using the current second loss function, the difference between the current classification result and the data label carried by the image data in the sample query set is measured; When the current second loss function does not satisfy the second convergence condition, obtaining an updated second parameter, and using the image classification module represented by the updated second parameter as the current image classification module; When the current second loss function satisfies the second convergence condition, the current image classification module is determined as the updated image classification module.

5. The method according to claim 2, characterized in that: Before classifying the plurality of image sample data to obtain a plurality of first image data located in the first environment and a plurality of second image data located in the second environment, the method further includes: Acquire a plurality of confusion features corresponding to each of the plurality of image sample data, wherein the confusion features are features that affect the generalization ability of the image classification model; Using the multiple confusion features, training a deviation classifier, wherein the deviation classifier is used to identify and classify deviations caused by the confusion features; The method of classifying the plurality of image sample data to obtain a plurality of first image data located in the first environment and a plurality of second image data located in the second environment includes: utilizing the deviation classifier to classify the plurality of image sample data to obtain the plurality of first image data and the plurality of second image data.

6. The method according to claim 5, characterized in that The method of using the deviation classifier to classify the plurality of image sample data to obtain the plurality of first image data and the plurality of second image data includes: Using the deviation classifier, the data distribution in the initial partition matrix is ​​optimized to obtain an optimized partition matrix, wherein the partition matrix is ​​used to guide the partitioning of data into different environments, and the initial partition matrix is ​​a randomly generated matrix that conforms to the normal distribution; The optimized division matrix is ​​used to classify the plurality of image sample data to obtain the plurality of first image data and the plurality of second image data.

7. The method according to claim 6, characterized in that The method of optimizing the data distribution in the initial partition matrix by using the deviation classifier to obtain the optimized partition matrix includes: Obtaining an empirical risk minimization loss and a gradient norm penalty, wherein the empirical risk minimization loss is used to reflect the performance of the deviation classifier on each sample data set, and the gradient norm penalty is used to measure the optimality of the deviation classifier on each sample data set; The data distribution in the initial partition matrix is ​​adjusted until the sum of the empirical risk minimization loss and the gradient norm penalty is maximized, thereby obtaining the optimized partition matrix.

8. The method according to claim 6, characterized in that In the process of optimizing the data distribution in the initial partition matrix by using the deviation classifier to obtain the optimized partition matrix, the method further includes: Obtaining a first function and a second function, wherein the first function is used to focus on the quantity imbalance of different data partition intervals, and the second function is used to focus on maintaining a minimum adjustment during the data partitioning process; Using the first function and the second function as a balance loss function in a process of optimizing data distribution in the initial partition matrix; The optimization process of the data distribution in the initial partition matrix is ​​constrained by the balanced loss function.

9. The method according to any one of claims 2 to 8, characterized in that The initial image classification model is trained by using the sample support set and the sample query set to obtain a trained image classification model, including: Iteratively training the initial image classification model through the sample support set and the sample query set to obtain an optimal round of image classification model; The image classification model of the optimal round is tested using the image data in the sample test set to obtain a test result, wherein the test result is used to measure whether the image classification model of the optimal round is used as the trained image classification model.

10. An image classification device based on causal reasoning, characterized in that: include: An image acquisition unit, used for acquiring image data to be classified; An input unit, used for inputting the image data into a trained image classification model, wherein the trained image classification model is an image classification model trained using a sample support set and a sample query set, the image classification model is a neural network model that performs image classification through causal reasoning, the sample support set is a sample set constructed using a plurality of first image data, the sample query set is a sample set constructed using a plurality of first image data and a plurality of second image data, the plurality of first image data are a plurality of image data located in a first environment obtained after classification of a plurality of image sample data, the plurality of second image data are a plurality of image data located in a second environment obtained after classification of the plurality of image sample data, and the image data category in the sample support set is the same as the image data category in the sample query set; A first acquisition unit is used to acquire an image classification result output by the trained image classification model, wherein the image classification result is used to indicate the image data category to which the image data belongs; The trained image classification model is obtained by training through the following steps: updating the first parameter of the feature extraction module in the initial image classification model through the sample support set until the first convergence condition is met, and obtaining an updated feature extraction module, wherein the feature extraction module is used to capture and represent causal features; updating the second parameter of the image classification module in the initial image classification model through the sample query set until the second convergence condition is met, and obtaining an updated image classification module, wherein the image classification module is used to perform image classification using the causal features.

11. An image classification device based on causal reasoning, characterized in that: include: A second acquisition unit, used for acquiring a plurality of image sample data; A classification unit, used for classifying the plurality of image sample data to obtain a plurality of first image data located in a first environment and a plurality of second image data located in a second environment; A first construction unit, configured to construct a sample support set using the plurality of first image data; A second construction unit is configured to construct a sample query set using the plurality of first image data and the plurality of second image data, wherein the image data category in the sample support set is the same as the image data category in the sample query set; A training unit, used to train an initial image classification model using the sample support set and the sample query set to obtain a trained image classification model, wherein the image classification model is a neural network model that performs image classification through causal reasoning; Wherein, the training unit includes: a first updating module, used to update the first parameter of the feature extraction module in the initial image classification model through the sample support set until the first convergence condition is met, and an updated feature extraction module is obtained, wherein the feature extraction module is used to capture and represent causal features; a second updating module, used to update the second parameter of the image classification module in the initial image classification model through the sample query set until the second convergence condition is met, and an updated image classification module is obtained, wherein the image classification module is used to perform image classification using the causal features.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 or 2 to 9 when executed by a processor.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 or 2 to 9 are implemented.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 or 2 to 9 are implemented.

Images