Image Classification Prediction Method, Model Training Method, Device, Equipment and Medium

Abstract

The present invention provides an image classification prediction method, a model training method, a device, a device and a medium. The prediction method includes: obtaining image data to be classified; using a trained image classification prediction model to perform classification prediction processing on the image data to be classified, so as to obtain a predicted classification result of the image classification features. The image classification prediction model is obtained by training a counterfactual attention supervised classification model and removing the model obtained by the forged attention transformation in the trained counterfactual attention supervised classification model. The present invention extracts features from the image data to be classified, and combines the counterfactual long-tail classification and the counterfactual-based attention supervision mechanism by using the trained image classification prediction model, which not only improves the recall of the tail classes in the image data, but also has good key feature extraction ability in the face of complex long-tail classification tasks, improves the image classification effect and the robustness of the classification prediction model.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to an image classification prediction method, a model training method, a device, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the advent of the era of large AI models, endowing large models with human reasoning capabilities has attracted increasing attention, and causal reasoning is expected to become one of the effective ways to achieve this goal. Counterfactuals and interventions are two powerful analysis tools in causal reasoning. With the help of the do-expression in causal reasoning, the values or states of elements in the SCM model can be controlled, thereby achieving certain practical purposes. Long-tail classification means that in the data to be classified, a considerable proportion of classes have very few samples, only a few classes have more samples, located at the head, while most classes have fewer samples, located at the tail. The long-tail distribution will cause the model to overfit the features of the head classes and be difficult to learn effective tail class features, thus affecting the recall of the tail classes.

[0003] In related technologies, although the recall of tail classes can be effectively improved by means of causal relationships, the problems of class confusion and intra-class bias in long-tail classification cannot be effectively solved, thus affecting the robustness of long-tail classification.

[0004] Therefore, how to effectively solve the problems of intra-class bias and inter-class confusion in long-tail classification tasks to improve the robustness of long-tail classification is a problem to be solved at present. Summary of the Invention

[0005] The present invention provides an image classification prediction method, a model training method, a device, an electronic device, and a computer-readable storage medium to at least solve the problem in related technologies that the problems of intra-class bias and inter-class confusion in image classification tasks cannot be effectively solved, resulting in poor image classification effects. The technical solution of the present invention is as follows:

[0006] According to the first aspect of the embodiments of the present invention, an image classification prediction method is provided, including:

[0007] Obtain image data to be classified;

[0008] Use the trained image classification prediction model to perform classification prediction processing on the image data to be classified to obtain a predicted classification result of the image classification features, where the image classification prediction model is obtained by training a counterfactual attention supervised classification model and removing the model obtained by transforming based on forged attention from the trained counterfactual attention supervised classification model.

[0009] Optionally, classifying and predicting the image classification features by using the trained image classification prediction model to obtain a predicted classification result of the image classification features, including:

[0010] Extracting the image classification features of the image data to be classified;

[0011] Performing attention mapping calculation on the image classification features to obtain attention mapping features;

[0012] Performing weighted processing on the attention mapping features and the image classification features to extract key features;

[0013] Performing max pooling and linear transformation on the key features extracted after weighted processing to obtain attention linear transformation features;

[0014] Performing deconfounding processing on the attention linear transformation features to obtain the predicted classification result of the image classification features.

[0015] Optionally, the performing deconfounding processing on the attention linear transformation features to obtain the predicted classification result of the image classification features includes:

[0016] Using causal inference to remove the confounding factors in the attention linear transformation features to obtain the predicted classification result of the corrected image classification features.

[0017] Optionally, the method further includes: pre-training an image classification prediction model in the following manner to obtain a trained long-tail classification model:

[0018] Obtaining a training set, where the training set includes: various image data and corresponding true classification labels;

[0019] Inputting each image data into a counterfactual attention supervised classification model for training; during the training process, extracting the image classification features of each image data; performing counterfactual attention supervision processing on the image classification features to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features; performing deconfounding training on the attention linear transformation features and the pseudo-attention linear transformation features respectively to obtain the corresponding predicted results after deconfounding training; calculating loss values according to the differences between each predicted result and the true classification label; updating the parameters of the counterfactual attention supervised classification model through a backpropagation mechanism based on the loss values, and after multiple iterative trainings until the counterfactual attention supervised classification model converges, obtaining a trained counterfactual attention supervised classification model, and removing the model obtained based on the forged attention transformation in the trained counterfactual attention supervised classification model as the image classification prediction model.

[0020] Optionally, the counterfactual attention supervision processing of the image classification features to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features includes:

[0021] Performing attention mapping calculation on the image classification features to obtain attention mapping features;

[0022] Performing counterfactual random calculation on the image classification features to obtain pseudo-attention mapping features;

[0023] Performing weighted processing on the image classification features with the attention mapping features and the pseudo-attention mapping features respectively, and extracting key features in the attention mapping features and the pseudo-attention mapping features respectively;

[0024] Performing max pooling and linear transformation on the key features in the respectively extracted attention mapping features and pseudo-attention mapping features to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features.

[0025] Optionally, the calculating the loss value according to the difference between each prediction result and the true classification label includes:

[0026] Calculating the difference between the prediction result after deconfounding training of the attention linear transformation features and the prediction result after deconfounding training of the pseudo-attention linear transformation features as the first difference;

[0027] Calculating the difference between the first difference and the prediction result after deconfounding training of the attention linear transformation features as the second difference;

[0028] Calculating the cross-entropy loss function based on the second difference and the true classification label as the first cross-entropy loss value;

[0029] Calculating the difference between the prediction result after deconfounding training of the pseudo-attention linear transformation features and the true classification label y as the second cross-entropy loss value;

[0030] Setting a hyperparameter for the sum of the first calculated cross-entropy loss value and the first calculated cross-entropy loss value as the loss value; wherein, the value of the hyperparameter is set to any value between 0 and 1.

[0031] According to the second aspect of the embodiments of the present invention, there is provided a method for training an image classification prediction model, including:

[0032] Obtaining a training set, the training set including: various image data, and corresponding true classification labels;

[0033] Input each type of image data into the counterfactual attention supervised classification model for training; during the training process, extract the image classification features of each type of image data; perform counterfactual attention supervision processing on the image classification features to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features; perform deconfounding training on the image classification features respectively with the attention linear transformation features and the pseudo-attention linear transformation to obtain corresponding prediction results after deconfounding training; calculate the loss value according to the difference between each prediction result and the true classification label; update the parameters of the counterfactual attention supervised classification model based on the loss value through the backpropagation mechanism; after multiple iterative trainings until the counterfactual attention supervised classification model converges, obtain the trained counterfactual attention supervised classification model, and remove the model obtained after the forged attention transformation in the trained counterfactual attention supervised classification model as the image classification prediction model.

[0034] According to the third aspect of the embodiments of the present invention, there is provided an image classification device, including:

[0035] An acquisition module, configured to acquire image data to be classified;

[0036] A prediction module, configured to perform classification prediction processing on the image data to be classified by using the trained image classification prediction model to obtain a predicted classification result of the image classification features, where the image classification prediction model is obtained by training a counterfactual attention supervised classification model and removing the model obtained after the forged attention transformation in the trained counterfactual attention supervised classification model.

[0037] Optionally, the prediction module includes:

[0038] A first extraction module, configured to extract the image classification features of the image data to be classified;

[0039] A first attention mapping module, configured to perform attention mapping calculation on the image classification features to obtain attention mapping features;

[0040] A first weighting module, configured to perform weighting processing on the attention mapping features and the image classification features to extract key features;

[0041] A first transformation module, configured to perform max pooling and linear transformation on the weighted features to obtain attention linear transformation features;

[0042] A first deconfounding processing module, configured to perform deconfounding processing on the attention linear transformation features to obtain the predicted classification result of the image classification features.

[0043] Optionally, the first de - confounding processing module is specifically configured to use causal inference to remove the confounding factors in the attention linear transformation features, and obtain the predicted classification result of the corrected image classification features.

[0044] Optionally, the device further includes: a training module, configured to pre - train an image classification prediction model to obtain a trained long - tail classification model.

[0045] Optionally, the training module includes:

[0046] A second acquisition module, configured to acquire a training set, where the training set includes: various image data and corresponding true classification labels;

[0047] An input module, configured to input each type of image data into the counterfactual attention supervised classification model for training;

[0048] A second extraction module, configured to extract the image classification features of each type of image data during the training process;

[0049] A counterfactual attention supervision module, configured to perform counterfactual attention supervision processing on the image classification features to obtain corresponding attention linear transformation features and pseudo - attention linear transformation features;

[0050] A second de - confounding training module, configured to perform de - confounding training on the attention linear transformation features and the pseudo - attention linear transformation respectively to obtain corresponding predicted results after de - confounding training;

[0051] A loss calculation module, configured to calculate a loss value according to the difference between each predicted result and the true classification label;

[0052] An update module, configured to update the parameters of the counterfactual attention supervised classification model based on the loss value through a backpropagation mechanism;

[0053] An iterative training module, configured to perform iterative training multiple times until the counterfactual attention supervised classification model converges, obtain a trained counterfactual attention supervised classification model, and use the model obtained by removing the model based on the forged attention transformation in the trained counterfactual attention supervised classification model as the image classification prediction model.

[0054] Optionally, the counterfactual attention supervision module includes:

[0055] A second attention mapping module, configured to perform attention mapping calculation on the image classification features to obtain attention mapping features;

[0056] A counterfactual random calculation module, configured to perform counterfactual random calculation on the image classification features to obtain pseudo - attention mapping features;

[0057] The second weighted processing module is used to perform weighted processing on the image classification features and the attention mapping features and the pseudo-attention mapping features respectively, and extract key features in the attention mapping features and the pseudo-attention mapping features respectively;

[0058] The pooling transformation module is used to perform max pooling and linear transformation on the key features in the attention mapping features and the pseudo-attention mapping features extracted respectively, to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features.

[0059] Optionally, the loss calculation module includes:

[0060] The first difference calculation module is used to calculate the difference between the prediction result after deconfounding training of the attention linear transformation features and the prediction result after deconfounding training of the pseudo-attention linear transformation features as the first difference;

[0061] The second difference calculation module is used to calculate the difference between the first difference and the prediction result after deconfounding training of the attention linear transformation features as the second difference;

[0062] The first cross-entropy loss module is used to calculate the cross-entropy loss function based on the second difference and the true classification label as the first cross-entropy loss value;

[0063] The second cross-entropy loss calculation module is used to calculate the difference between the prediction result after deconfounding training of the pseudo-attention linear transformation features and the true classification label y as the second cross-entropy loss value;

[0064] The setting module sets a hyperparameter for the sum of the first calculated cross-entropy loss value and the first calculated cross-entropy loss value as the loss value, where the value of the hyperparameter is set to any value between 0 and 1.

[0065] According to the fourth aspect of the embodiments of the present invention, there is provided an image classification prediction model training device,

[0066] The acquisition module is used to acquire a training set, and the training set includes: various image data and corresponding true classification labels;

[0067] The input module is used to input each kind of image data into the counterfactual attention supervised classification model for training;

[0068] The extraction module extracts the image classification features of each kind of image data during the training process;

[0069] The counterfactual attention supervision module is used to perform counterfactual attention supervision processing on the image classification features to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features;

[0070] A de - obfuscation training module, configured to perform de - obfuscation training on the image classification features respectively with the attention linear transformation features and the pseudo - attention linear transformation, and obtain corresponding prediction results after de - obfuscation training;

[0071] A loss calculation module, configured to calculate a loss value according to the difference between each prediction result and the true classification label;

[0072] An update module, configured to update the parameters of the counterfactual attention supervised classification model based on the loss value through a backpropagation mechanism;

[0073] An iterative training module, configured to perform multiple iterations of training until the counterfactual attention supervised classification model converges, obtain a trained counterfactual attention supervised classification model, and use the model obtained by removing the model based on the forged attention transformation in the trained counterfactual attention supervised classification model as an image classification prediction model.

[0074] According to the fifth aspect of the embodiments of the present invention, there is also provided an electronic device, including:

[0075] A processor;

[0076] A memory for storing executable instructions of the processor;

[0077] Wherein, the processor is configured to execute the instructions to implement the image classification prediction method or the image classification prediction model training method as described above.

[0078] According to the sixth aspect of the embodiments of the present invention, there is also provided a computer - readable storage medium. When the instructions in the computer - readable storage medium are executed by a processor of an electronic device, the electronic device can execute the image classification prediction method or the image classification prediction model training method as described above.

[0079] According to the seventh aspect of the embodiments of the present invention, there is also provided a computer program product, including a computer program or instructions. When the computer program or instructions are executed by a processor of an electronic device, the electronic device executes the image classification prediction method or the image classification prediction model training method as described above.

[0080] The technical solutions provided by the embodiments of the present invention at least bring the following beneficial effects:

[0081] In an embodiment of the present invention, image data to be classified is obtained; the trained image classification prediction model is used to perform classification prediction processing on the image data to be classified, and a predicted classification result of the image classification features is obtained. Among them, the image classification prediction model is obtained by training a counterfactual attention supervised classification model and removing the model obtained by the forged attention transformation in the trained counterfactual attention supervised classification model. That is to say, in this embodiment, feature extraction is performed on the image data to be classified, and the trained image classification prediction model combines counterfactual long-tail classification and counterfactual attention supervision mechanism, which not only improves the recall of the tail classes in the image data, but also has good key feature extraction ability in the face of complex long-tail classification tasks, improves the image classification effect, and the robustness of the classification prediction model.

[0082] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present invention. Brief Description of the Drawings

[0083] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present invention, and are used together with the specification to explain the principles of the present invention and do not constitute an improper limitation of the present invention. In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0084] Figure 1 It is a flowchart of an image classification prediction method provided by an embodiment of the present invention.

[0085] Figure 2 It is an application example diagram of an image classification prediction method provided by an embodiment of the present invention.

[0086] Figure 3 It is a flowchart of an image classification prediction model training method provided by an embodiment of the present invention.

[0087] Figure 4 It is an application example diagram of an image classification prediction model training method provided by an embodiment of the present invention.

[0088] Figure 5 It is a block diagram of an image classification prediction device provided by an embodiment of the present invention.

[0089] Figure 6 It is a block diagram of an image classification prediction model training device provided by an embodiment of the present invention.

[0090] Figure 7 It is an application example diagram of an image classification prediction model training device provided by an embodiment of the present invention.

[0091] Figure 8 It is a block diagram of an electronic device provided by an embodiment of the present invention.

[0092] Figure 9 It is a block diagram of a device for image classification or training of an image classification prediction model provided by an embodiment of the present invention. Detailed implementation manners

[0093] In order to enable those of ordinary skill in the art to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0094] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order different from those illustrated or described herein. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present invention. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0095] In recent years, important progress has been made in the research of technologies such as computer vision, deep learning, machine learning, image processing, and image recognition based on artificial intelligence. Artificial Intelligence (AI) is a new science and technology that studies and develops theories, methods, technologies, and application systems for simulating and extending human intelligence. The discipline of artificial intelligence is a comprehensive discipline that involves many technical categories such as chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. As an important branch of artificial intelligence, computer vision specifically enables machines to recognize the world. Computer vision technology usually includes face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, Simultaneous Localization and Mapping (SLAM), computational photography, robot navigation and positioning, and other technologies. With the research and progress of artificial intelligence technology, this technology has been applied in many fields, such as security, urban management, traffic management, building management, park management, face access, face attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone imaging, cloud services, smart home, wearable devices, driverless, autonomous driving, intelligent healthcare, face payment, face unlocking, fingerprint unlocking, person-card verification, smart screen, smart TV, cameras, mobile Internet, webcasting, beauty, makeup, medical beauty, intelligent temperature measurement, and other fields.

[0096] Technical terms:

[0097] Causal Inference,

[0098] One of the core issues in statistics and data science, it is the process of inferring a causal relationship conclusion when a certain phenomenon has occurred.

[0099] Structural Causal Model (SCM), a method for expressing the causal relationship behind data in the form of a directed acyclic graph, used to represent the associations and interactions of individuals in the real world.

[0100] Counterfactual is a reflection on what has happened in the past, and it solves the problem of "if different actions were taken in the past, what would be the difference in the current result?" For example, Zhang San didn't pass the exam, and he would think if I didn't keep watching dramas frequently before the exam, would I be able to pass the final exam, etc.

[0101] Intervention, by implementing certain actions on a certain target to study a certain effect before and after the implementation of the actions.

[0102] Attention, a learnable module that helps the model capture key features in an image.

[0103] Based on the understanding of the above technical terms, please also refer to the following embodiments.

[0104] Figure 1 It is a flowchart of an image classification prediction method provided by an embodiment of the present invention. As Figure 1 shown, the method includes the following steps:

[0105] Step 101: Obtain image data to be classified;

[0106] Step 102: Use the trained image classification prediction model to perform classification prediction processing on the image data to be classified, and obtain a predicted classification result of the image classification features. Among them, the image classification prediction model is obtained by training a counterfactual attention supervised classification model and removing the model obtained after the forged attention transformation in the trained counterfactual attention supervised classification model.

[0107] The image classification prediction method described in the present invention can be applied to terminals, servers, etc., without limitation here. The terminal implementation devices can be electronic devices such as smart phones, laptop computers, tablet computers, desktop computers, personal digital assistants (PDAs), and wearable devices. The server can be an independent server, a server cluster, or a server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, intermediate services, domain name services, security services, content distribution networks, or a big data and artificial intelligence platform, etc., without limitation here.

[0108] Next, in conjunction with Figure 1 ,the specific implementation steps of an image classification prediction method provided by an embodiment of the present invention will be described in detail.

[0109] In step 101, obtain image data to be classified.

[0110] In this step, the image data to be classified can be obtained locally, remotely, in real time, etc., and this embodiment does not limit it. The image to be classified can be an ordinary image or a long-tail image, such as a wild animal image, etc. Since many wild animals look very similar, such as the aardvark and the anteater, it is usually difficult to distinguish them. They not only belong to the tail classes in the long-tail distribution, but also have little distinguishability, which poses greater challenges for the long-tail classification task. Therefore, for this long-tail image, if you want to identify what animal is in the long-tail image and which category it belongs to, you can use the image classification prediction model provided by the present invention to perform prediction classification on it.

[0111] In step 102, the trained image classification prediction model is used to perform classification prediction processing on the image data to be classified, and a prediction classification result of the image classification features is obtained. Among them, the image classification prediction model is obtained by training a counterfactual attention supervised classification model and removing the model obtained by the forged attention transformation in the trained counterfactual attention supervised classification model.

[0112] In this step, specifically, 1) Extract the image classification features of the image data to be classified.

[0113] In this step, the image classification features of the image data to be classified can be extracted through the backbone network in the image classification prediction model, where the backbone network can be a convolutional neural network (CNN, Convolutional Neural Networks), etc. The specific feature extraction process is already a well-known technology and will not be elaborated here.

[0114] 2) Perform attention mapping calculation on the image classification features to obtain attention mapping features.

[0115] In this step, the attention mapping calculation can be performed on the image classification features through the attention supervision module in the image classification prediction model to obtain attention mapping features; among them, the attention supervision module adopts an attention supervision mechanism, and the process of performing attention mapping calculation on the image classification features is already a well-known technology and will not be elaborated here.

[0116] 3) Perform weighted processing on the attention mapping features and the image classification features to extract key features.

[0117] In this step, the weighted processing is to perform element-wise dot multiplication on the attention mapping features and the image classification features, and then extract the key features in the feature mapping. The specific extraction process is detailed in the following training process and will not be elaborated here.

[0118] 4) Perform max pooling and linear transformation on the key features extracted after weighted processing to obtain attention linear transformation features.

[0119] In this step, first perform a max pooling operation on the weighted key features to convert them into one-dimensional feature vectors. Then, perform a linear transformation on the obtained one-dimensional feature vectors to obtain attention linear transformation features. Among them, the linear transformation can be to convolve the one-dimensional vector with a linear transformation matrix to obtain attention linear transformation features, etc. Of course, this embodiment is not limited to this linear transformation method, and other linear transformation methods can also be used, which are not limited in this embodiment.

[0120] 5) Perform deconfounding processing on the attention linear transformation features to obtain the predicted classification result of the image classification features.

[0121] Among them, the process of performing deconfounding processing on the attention linear transformation features to obtain the predicted classification result of the image classification features includes: using causal inference to remove the confounding factors in the attention linear transformation features to obtain the predicted classification result of the corrected image classification features. Use counterfactual and intervention learning in causal inference for deconfounding to eliminate the harmful effects of confounding factors and obtain the corrected prediction result. Among them, the specific process of learning to remove momentum confounding through intervention can be found in the corresponding process of model training described below, and will not be elaborated here.

[0122] It should be noted that in the embodiment of the present invention, the image classification prediction model is obtained by training the counterfactual attention supervised classification model and then removing the model obtained by the forged attention transformation in the trained counterfactual attention supervised classification model. That is to say, the classification prediction process of the image classification prediction model is basically similar to the training process of the counterfactual attention supervised classification model, except that the part based on the forged attention transformation in the counterfactual attention supervised classification model needs to be removed. The specific process is described below and will not be elaborated here.

[0123] In the embodiment of the present invention, image data to be classified is obtained; the trained image classification prediction model is used to perform classification prediction processing on the image data to be classified to obtain the predicted classification result of the image classification features, where the image classification prediction model is obtained by training the counterfactual attention supervised classification model and then removing the model obtained by the forged attention transformation in the trained counterfactual attention supervised classification model. That is to say, in this embodiment, feature extraction is performed on the image data to be classified, and the trained image classification prediction model combines counterfactual long-tail classification and counterfactual attention supervision mechanism, which not only improves the recall of the tail classes in the image data, but also has good key feature extraction ability for complex long-tail classification tasks, improves the image classification effect, and the robustness of the classification prediction model.

[0124] Please also refer toFigure 2 , which is an application example diagram of an image classification prediction method provided by an embodiment of the present invention. In this embodiment, the image data takes long-tailed image data as an example, but in actual applications, it is not limited to this.

[0125] As Figure 2 shown, first, input the long-tailed image into the backbone network (Backbone, or main network, etc., such as a network for feature extraction like ResNet, VGG, etc.) in the image classification prediction model to extract features, and obtain the image classification feature X (Feature map, that is, the feature map). Then, input the image classification feature into the attention mapping module (Attention Maps A) in the image classification prediction model to perform attention mapping calculation, and obtain the attention mapping feature; perform weighted processing on the attention mapping feature and the image classification feature to extract key features; input the key features extracted after weighted processing into the pooling linear transformation module (Pooling+FC) in the image classification prediction model to perform max pooling and linear transformation processing, and output the attention linear transformation feature; the specific max pooling and linear transformation processing process is described in detail below and will not be elaborated here. Finally, input the attention linear transformation feature into the causal inference head (Casual Head) in the image classification prediction model to perform deconfounding processing, remove the confounding factors in the attention linear transformation feature, and obtain the predicted classification result Y(A) of the corrected image classification feature.

[0126] That is to say, in the embodiment of the present invention, the long-tailed image is subjected to feature extraction through the backbone network, weighted with the attention mapping generated by the attention mapping module, the output after weighting is subjected to pooling and linear transformation operations, and finally input into the causal inference head for deconfounding processing to obtain the deconfounded output, that is, the predicted classification result.

[0127] It should be noted that in this embodiment, the application of the image classification prediction model is similar to the subsequent training process. The difference is that a process of intervention based on forged attention is added in the following training process, so that the model has the ability to extract key features for distinguishing different categories. Therefore, in the model inference stage (i.e., the application node), the forged attention mapping module does not need to continue to exist and can be directly removed from the model structure, thereby making the inference process more efficient than the training process.

[0128] Optionally, in another embodiment, based on the above embodiment, the method may further include: pre-training an image classification prediction model to obtain a trained long-tailed classification model. Specifically, it includes:

[0129] Obtain a training set, where the training set includes: various image data and corresponding true classification labels; input each type of image data into a counterfactual attention supervised classification model for training; during the training process, extract the image classification features of each type of image data; perform counterfactual attention supervised processing on the image classification features to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features; perform deconfounding training on the attention linear transformation features and the pseudo-attention linear transformation respectively to obtain corresponding prediction results after deconfounding training; calculate the loss value according to the difference between each prediction result and the true classification label; based on the loss value, update the parameters of the counterfactual attention supervised classification model through the backpropagation mechanism. After multiple iterations of training until the counterfactual attention supervised classification model converges, obtain a trained counterfactual attention supervised classification model, and remove the model obtained by the forged attention transformation in the trained counterfactual attention supervised classification model as an image classification prediction model.

[0130] In an embodiment of the present invention, each type of image data in the obtained training set is input into a counterfactual attention supervised classification model for training. During the training process, first extract the image classification features of the image data, and then perform a transformation calculation supervised by counterfactual attention on the extracted features, and send the two outputs supervised by counterfactual attention and the image classification features into the causal inference head module for deconfounding processing to output the prediction training result. Calculate the loss value based on the prediction training result, and based on the output of the forward inference of the loss value, update the parameters of the model through the backpropagation loss function value. In this embodiment, combining counterfactual long-tail classification with a counterfactual attention supervision mechanism for training can not only improve the recall of tail classes, but also have better key feature extraction ability in the face of complex long-tail classification tasks, improving the robustness of the model.

[0131] Optionally, in another embodiment, based on the above embodiment, the performing counterfactual attention supervised processing on the image classification features to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features includes: performing attention mapping calculation on the image classification features to obtain attention mapping features; performing counterfactual random calculation on the image classification features to obtain pseudo-attention mapping features; performing weighted processing on the image classification features with the attention mapping features and the pseudo-attention mapping features respectively, and extracting the key features in the attention mapping features and the pseudo-attention mapping features respectively; performing max pooling and linear transformation on the key features in the attention mapping features and the pseudo-attention mapping features extracted respectively to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features.

[0132] Optionally, in another embodiment, based on the above embodiment, calculating the loss value according to the difference between each prediction result and the true classification label respectively includes: calculating the difference between the prediction result after the anti-confusion training of the attention linear transformation feature and the prediction result after the anti-confusion training of the pseudo-attention linear transformation feature as the first difference; calculating the difference between the first difference and the prediction result after the anti-confusion training of the attention linear transformation feature as the second difference; calculating the cross-entropy loss function based on the second difference and the true classification label as the first cross-entropy loss value; calculating the difference between the prediction result after the anti-confusion training of the pseudo-attention linear transformation feature and the true classification label y as the second cross-entropy loss value; setting a hyperparameter for the sum of the first calculated cross-entropy loss value and the first calculated cross-entropy loss value as the loss value, where the value of the hyperparameter is set to any value between 0 and 1.

[0133] It should be noted that the model training process of this embodiment is described in detail below and will not be elaborated here.

[0134] Please also refer to Figure 3 , which is a flowchart of a method for training an image classification prediction model provided by an embodiment of the present invention. The method includes:

[0135] Step 301: Obtain a training set, where the training set includes various image data and corresponding true classification labels;

[0136] Step 302: Input each type of image data into the counterfactual attention supervised classification model for training; during the training process, extract the image classification features of each type of image data; perform counterfactual attention supervision processing on the image classification features to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features; perform anti-confusion training on the attention linear transformation features and the pseudo-attention linear transformation respectively to obtain corresponding prediction results after anti-confusion training; calculate the loss value according to the difference between each prediction result and the true classification label respectively; update the parameters of the counterfactual attention supervised classification model through the backpropagation mechanism based on the loss value; after multiple iterative trainings until the counterfactual attention supervised classification model converges, obtain the trained counterfactual attention supervised classification model, and remove the model obtained based on the forged attention transformation in the trained counterfactual attention supervised classification model as the image classification prediction model.

[0137] Optionally, in another embodiment, based on the above embodiment, the counterfactual attention supervision process of the image classification features to obtain the corresponding attention linear transformation features and pseudo-attention linear transformation features includes: performing attention mapping calculation on the image classification features to obtain attention mapping features; performing counterfactual random calculation on the image classification features to obtain pseudo-attention mapping features; weighting the image classification features with the attention mapping features and the pseudo-attention mapping features respectively, and extracting the key features in the attention mapping features and the pseudo-attention mapping features respectively; performing max pooling and linear transformation on the key features in the respectively extracted attention mapping features and pseudo-attention mapping features to obtain the corresponding attention linear transformation features and pseudo-attention linear transformation features.

[0138] Optionally, in another embodiment, based on the above embodiment, the calculation of the loss value according to the difference between each prediction result and the true classification label includes: calculating the difference between the prediction result after deconfounding training of the attention linear transformation features and the prediction result after deconfounding training of the pseudo-attention linear transformation features as the first difference; calculating the difference between the first difference and the prediction result after deconfounding training of the attention linear transformation features as the second difference; calculating the cross-entropy loss function based on the second difference and the true classification label as the first cross-entropy loss value; calculating the difference between the prediction result after deconfounding training of the pseudo-attention linear transformation features and the true classification label y as the second cross-entropy loss value; setting a hyperparameter for the sum of the first calculated cross-entropy loss value and the first calculated cross-entropy loss value as the loss value, where the value of the hyperparameter is set to any value between 0 and 1.

[0139] Please also refer to Figure 4 , which is an application example diagram of an image classification prediction model training method provided by an embodiment of the present invention. For the sake of illustration, the input image data in this embodiment is long-tailed image data, but in actual applications, it is not limited to this. Specifically, it includes:

[0140] As Figure 4 shown, 1) Input the long-tailed image into the backbone network (Backbone, or main network, etc.) in the image classification prediction model for feature extraction to obtain the image classification feature X (Feature map, that is, the feature map), and the dimension of the image classification feature X is H×W×C, where H and W respectively represent the height and width of the extracted channels, and C represents the number of feature channels.

[0141] 2) Input the image classification features into the attention mapping module (AttentionMaps A, which can be called the model) in the image classification prediction model to perform attention mapping calculation, and obtain the attention mapping feature A. The calculation formula of the attention mapping is shown in Figure Formula 1;

[0142]

[0143] represents a certain attention model, X represents the image feature map extracted by the backbone network (consisting of one or more two-dimensional square matrices), represents the attention feature map extracted by the attention model (consisting of one or more two-dimensional square matrices), A represents the set of calculated attention mappings, with a total of M, and A1, A2, and A M are all elements in the set. Among them, the square matrix is a two-dimensional matrix with equal width and height.

[0144] 3) Input the image classification features into the counterfactual attention mapping module (Counterfactual Attention Maps A) in the image classification prediction model to perform counterfactual attention mapping calculation, and obtain the pseudo-attention mapping feature A ; specifically, it can be generated by randomly sampling numerical values from a normal distribution or a uniform distribution, or by taking the inverse of the numerical values in the pseudo-attention mapping feature A. Its role is to use counterfactual theory to drive the attention module to achieve the learning of key features. The specific formal definition can refer to Formula (2) and Formula (3).

[0145]

[0146] Among them, and are forged attention mapping square matrices, C represents the classifier, represents the intervention operation, indicating that the image feature X is fixed and remains unchanged, and the output value of the classifier under the forged attention mapping. The other part of the parameters in this formula have been explained in other formulas. For details, please refer to other formulas.

[0147]

[0148] In this formula, γ represents the distribution followed by the generated forged attention mapping, and represents that the forged attention mapping follows the distribution represented by γ (such as a uniform distribution). E represents calculating the mean of the random variable, and Y(A = A, X = X) represents the output of the classifier when the attention mapping is A and the image feature is equal to X. The parameters of this formula are explained in Formula (2). The difference between the two, Y effectIt represents the direct effect of the true attention map learned by the model reflected in the prediction output.

[0149] 4) Perform weighted processing on the image classification features with the attention map features and the pseudo-attention map features respectively, and extract the key features corresponding to the attention map features and the pseudo-attention map features.

[0150] In this step, the image classification features, the pseudo-attention map features A and the attention map feature A are respectively element-wise multiplied with the image classification feature X, and then the corresponding key features in the feature map are extracted. The specific calculation method refers to formulas (4) and (5), and two sets of outputs are correspondingly generated.

[0151]

[0152] Among them, H and W respectively represent the height and width of X in formula (1), h and w respectively traverse from 1 to H and W, and X h，w represents the value at the position of the h-th row and the w-th column in X, i represents the index of the feature map, represents the value at the h-th row and the w-th column in the i-th feature map, represents global average pooling, and * represents element-wise multiplication of two tensors. The whole formula means using an attention map square matrix to weight the element values in X, and then obtaining a new square matrix h through global average pooling i .

[0153] h = normalize([h1, h2,..., h M ),(5)

[0154] Among them,

[0155] h1, h2 and h M are the outputs generated by formula (4), and normalize represents the normalization operation on these output square matrices, so that the values of all elements are limited between [-1, 1].

[0156] 5) Input the key features extracted after weighted processing into the pooling linear transformation module (Pooling + FC) in the image classification prediction model for max-pooling and linear transformation processing, and output the attention linear transformation features and the pseudo-attention linear transformation features.

[0157] In this step, the two sets of output results in step 4) are respectively pooled and linearly transformed through the max-pooling and linear transformation module, that is, the two sets of inputs are first converted into one-dimensional vectors through pooling, and then linearly transformed to output the attention linear transformation features and the pseudo-attention linear transformation features.

[0158] 6) Input the attention linear transformation feature and the pseudo-attention linear transformation feature into the causal inference head (Casual Head) in the image classification prediction model for deconfounding processing to obtain the corresponding prediction result after deconfounding training.

[0159] This step is to separately input the two groups of outputs in step 5) into the causal inference head module to perform deconfounding learning, and learn the model parameters after removing momentum confounding through intervention. The formal definition can refer to formulas (6) and (7) in [reference]. The predicted classification result of the output obtained from the attention mapping feature A is Y(A), and the predicted classification result of the output obtained from the pseudo-attention mapping feature A is Y(A).

[0160]

[0161] Among them, i represents the class number, argmax represents finding which class can obtain the maximum value in the formula, TDE is the abbreviation of the total direct effect, Y1 represents the causal effect of the i-th class, d represents the value of the mediating node, Y d =l represents the causal effect of the i-th class under the mediating effect, x represents the image feature, do(X = x) represents fixing the feature value of the image to a certain value, which belongs to the intervention means in causal inference, x0 represents an empty input, do(X = x) represents fixing the feature value of the image to an empty input, and the overall meaning is to calculate the total direct effect of class i after removing confounding factors.

[0162]

[0163] Among them, K represents the number of heads of the multi-head attention, τ is a scaling factor greater than 0. Since this formula draws on the theory of the energy model, γ is a regulation parameter related to the energy model, x k 、 and d k respectively represent the image feature generated by the k-th attention head, the special part in the image feature vector, and the projection part of the image feature in the head class direction. represents the weight matrix belonging to the i-th class in the k-th head. The whole formula represents the formalization of the model's deconfounding training.

[0164] 7) Calculate the loss value according to the difference between each prediction result and the true classification label respectively; update the parameters of the counterfactual attention supervised classification model through the backpropagation mechanism based on the loss value; after multiple iterative trainings until the counterfactual attention supervised classification model converges, obtain the trained counterfactual attention supervised classification model, and remove the model obtained after the forged attention transformation in the trained counterfactual attention supervised classification model as the image classification prediction model.

[0165] In this step, the loss value is calculated. This loss value is composed of two cross-entropy loss functions, L1 and L2. Loss = αL1+(1 - α)L2, where L1 = CE((Y(A)-Y(A)),Y); L2 = CE((Y(A),Y); where CE is the abbreviation of the cross-entropy loss function. Specifically, L1 and L2 are as follows:

[0166] Among them, the calculation formula of L1 is:

[0167] -ylog(Y(A)-Y( A )+(1 - y)log(1 - Y(A)+Y( A ))), (8)

[0168] This formula represents the difference (i.e., the first difference) between the predicted output (Y(A)-Y( A )) of the deconfounding training and the output Y(A) generated by the attention mapping feature A. The smaller this difference is, that is, the closer the two are, the closer they are, the higher the quality of the Attention that the model truly learns, and the stronger the ability to mine and distinguish key features of different categories. After that, the cross-entropy loss value is calculated between this first difference and the true label y to obtain the value of L1, which is called the first cross-entropy loss value. The calculation formula of L2 is:

[0169] -ylog(Y(A))+(1 - y)log(1 - Y(A)), (9)

[0170] Among them, this formula represents the gap (i.e., the second difference) between Y(A) and the true classification label y.

[0171] After that, calculate the sum of the first cross-entropy loss value and the second cross-entropy loss value, and set a hyperparameter α for the sum of the first cross-entropy loss value and the second cross-entropy loss value. α can be any value between 0 and 1. Its calculation formula is:

[0172] -α[ylog(Y(A)-Y( A )+(1 - y)log(1 - Y(A)+Y( A )))-ylog(Y(A))+(1 - y)log(1 - Y(A))] (10)

[0173] That is, a hyperparameter α is added between the two losses to regulate the learning intensity of the counterfactual attention. For some simple long-tail tasks, this parameter can be adjusted lower to focus on optimizing the causal head and the backbone network.

[0174] It should be noted that for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present disclosure is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential to the present invention.

[0175] Figure 5 It is a block diagram of an image classification prediction device provided by an embodiment of the present invention. The device includes: an acquisition module 501 and a prediction module 502, wherein,

[0176] The acquisition module 501 is used to acquire image data to be classified;

[0177] The prediction module 502 is used to perform classification prediction processing on the image data to be classified by using a trained image classification prediction model, and obtain a predicted classification result of the image classification feature, wherein the image classification prediction model is obtained by training a counterfactual attention supervised classification model and removing the model obtained by the forged attention transformation in the trained counterfactual attention supervised classification model.

[0178] Optionally, in another embodiment, on the basis of the above embodiment, the prediction module includes: a first extraction module, a first attention mapping module, a first weighting module, a first transformation module, a first transformation module and a first deconfusion processing module, wherein,

[0179] The first extraction module is used to extract the image classification feature of the image data to be classified;

[0180] The first attention mapping module is used to perform attention mapping calculation on the image classification feature to obtain an attention mapping feature;

[0181] The first weighting module is used to perform weighting processing on the attention mapping feature and the image classification feature to extract key features;

[0182] The first transformation module is used to perform max pooling and linear transformation on the weighted feature to obtain an attention linear transformation feature;

[0183] The first deconfusion processing module is used to perform deconfusion processing on the attention linear transformation feature to obtain a predicted classification result of the image classification feature.

[0184] Optionally, in another embodiment, based on the above embodiment, the first deconfounding processing module is specifically configured to use causal inference to remove the confounding factors in the attention linear transformation features, and obtain the predicted classification result of the corrected image classification features.

[0185] Optionally, in another embodiment, based on the above embodiment, the apparatus further includes: a training module, configured to pre-train an image classification prediction model to obtain a trained long-tail classification model.

[0186] Optionally, in another embodiment, based on the above embodiment, the training module includes:

[0187] A second acquisition module, configured to acquire a training set, where the training set includes: various image data and corresponding true classification labels;

[0188] An input module, configured to input each type of image data into the counterfactual attention supervised classification model for training;

[0189] A second extraction module, configured to extract the image classification features of each type of image data during the training process;

[0190] A counterfactual attention supervision module, configured to perform counterfactual attention supervision processing on the image classification features to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features;

[0191] A second deconfounding training module, configured to perform deconfounding training on the attention linear transformation features and the pseudo-attention linear transformation respectively to obtain corresponding predicted results after deconfounding training;

[0192] A loss calculation module, configured to calculate a loss value according to the difference between each predicted result and the true classification label;

[0193] An update module, configured to update the parameters of the counterfactual attention supervised classification model based on the loss value through a backpropagation mechanism;

[0194] An iterative training module, configured to perform multiple iterations of training until the counterfactual attention supervised classification model converges, obtain a trained counterfactual attention supervised classification model, and use the model obtained by removing the model obtained based on the forged attention transformation in the trained counterfactual attention supervised classification model as the image classification prediction model.

[0195] Optionally, in another embodiment, based on the above embodiment, the counterfactual attention supervision module includes:

[0196] The second attention mapping module is used to perform attention mapping calculation on the image classification features to obtain attention mapping features;

[0197] The counterfactual random calculation module is used to perform counterfactual random calculation on the image classification features to obtain pseudo-attention mapping features;

[0198] The second weighting processing module is used to perform weighting processing on the image classification features with the attention mapping features and the pseudo-attention mapping features respectively, and extract key features in the attention mapping features and the pseudo-attention mapping features respectively;

[0199] The pooling transformation module is used to perform max pooling and linear transformation on the key features in the attention mapping features and the pseudo-attention mapping features extracted respectively to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features.

[0200] Optionally, in another embodiment, based on the above embodiment, the loss calculation module includes:

[0201] The first difference calculation module is used to calculate the difference between the prediction result after deconfounding training of the attention linear transformation features and the prediction result after deconfounding training of the pseudo-attention linear transformation features as the first difference;

[0202] The second difference calculation module is used to calculate the difference between the first difference and the prediction result after deconfounding training of the attention linear transformation features as the second difference;

[0203] The first cross-entropy loss module is used to calculate the cross-entropy loss function based on the second difference and the true classification label as the first cross-entropy loss value;

[0204] The second cross-entropy loss calculation module is used to calculate the difference between the prediction result after deconfounding training of the pseudo-attention linear transformation features and the true classification label y as the second cross-entropy loss value;

[0205] The setting module sets a hyperparameter for the sum of the first calculated cross-entropy loss value and the first calculated cross-entropy loss value as the loss value, where the value of the hyperparameter is set to any value between 0 and 1.

[0206] Please also refer to Figure 6 , which is a block diagram of an image classification prediction model training device provided by an embodiment of the present invention. The device includes: an acquisition module 601, an input module 602, an extraction module 603, a counterfactual attention supervision module 604, a deconfounding training module 605, a loss calculation module 606, an update module 607, and an iterative training module 608, where,

[0207] An acquisition module 601, configured to acquire a training set, where the training set includes: various image data and corresponding true classification labels;

[0208] An input module 602, configured to input each type of image data into a counterfactual attention supervised classification model for training;

[0209] A second extraction module 603, which extracts image classification features of each type of image data during the training process;

[0210] A counterfactual attention supervision module 604, configured to perform counterfactual attention supervision processing on the image classification features to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features;

[0211] A deconfounding training module 605, configured to perform deconfounding training on the image classification features with the attention linear transformation features and the pseudo-attention linear transformation respectively to obtain corresponding prediction results after deconfounding training;

[0212] A loss calculation module 606, configured to calculate a loss value according to the difference between each prediction result and the true classification label;

[0213] An update module 607, configured to update the parameters of the counterfactual attention supervised classification model based on the loss value through a backpropagation mechanism;

[0214] An iterative training module 608, configured to update the parameters of the counterfactual attention supervised classification model based on the loss value through a backpropagation mechanism. After multiple iterative trainings until the counterfactual attention supervised classification model converges, a trained counterfactual attention supervised classification model is obtained, and the model obtained by removing the model obtained based on forged attention transformation in the trained counterfactual attention supervised classification model is used as an image classification prediction model.

[0215] Please also refer to Figure 7 , which is an application example diagram of an image classification prediction model training device provided by an embodiment of the present invention, including: a backbone network module 701, a counterfactual attention supervision module 702, a causal inference head module 703, and a loss value calculation module 704: where,

[0216] The backbone network module 701 is configured to extract image classification features of input image data and output the image classification features;

[0217] The counterfactual attention supervision module 702: It is used to generate a false attention that is opposite to the real attention learned from the image through methods such as random generation and uniform distribution sampling. Its role is to drive the Attention module of the model to focus the attention on the parts that distinguish key features of different categories; the role of attention is to focus on the key parts in the image (such as the long and pointed mouth of the anteater, etc.), and perform transformation processing on the image features corresponding to the key parts. This module simultaneously generates real attention transformation features and pseudo-attention transformation features, and outputs the attention transformation features and pseudo-attention transformation features. That is, the image classification features are subjected to counterfactual attention supervision processing to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features;

[0218] The causal reasoning head module 703: It is used to perform deconfounding training by using counterfactual and intervention techniques in causal reasoning, that is, the image classification features are respectively subjected to deconfounding training with the attention linear transformation features and the pseudo-attention linear transformation features to obtain corresponding prediction results after deconfounding training; then in the inference stage (i.e., the application stage), the harmful effects of confounding factors can be eliminated to obtain the corrected output prediction results;

[0219] The loss value calculation module 704: It is used to calculate the difference between the output of the causal reasoning head and the real label, and the obtained difference updates the relevant parameters of the backbone network module and the Attention module through the backpropagation mechanism, so as to achieve the goal of optimizing the network. That is, the loss value is calculated according to the difference between each prediction result and the real classification label; based on the loss value, the parameters of the counterfactual attention supervision classification model are updated through the backpropagation mechanism; after multiple iterative trainings until the counterfactual attention supervision classification model converges, the trained counterfactual attention supervision classification model is obtained, and the model obtained after the forged attention transformation in the trained counterfactual attention supervision classification model is removed as the image classification prediction model

[0220] In the embodiment of the present invention, combining the long-tail classification method with the counterfactual-based attention mechanism can not only supervise the model to learn to distinguish key features of different categories, but also improve the robustness of the model in complex classification scenarios.

[0221] Optionally, the present invention also provides an electronic device, including:

[0222] A processor;

[0223] A memory for storing executable instructions of the processor;

[0224] Wherein, the processor is configured to execute the instructions to implement the image classification prediction method or the image classification prediction model training method as described above.

[0225] Optionally, the present invention further provides a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the image classification prediction method or the image classification prediction model training method as described above.

[0226] Optionally, the present invention further provides a computer program product, including a computer program or instructions. When the computer program or instructions are executed by a processor of an electronic device, the image classification prediction method or the image classification prediction model training method as described above is implemented.

[0227] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0228] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0229] Figure 8 FIG. is a block diagram of an electronic device 800 provided by an embodiment of the present invention. For example, the electronic device 800 may be a mobile terminal or a server. In the embodiments of the present invention, the electronic device is taken as a mobile terminal as an example for illustration. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0230] Referring to Figure 8 , the electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0231] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-described methods. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0232] The memory 804 is configured to store various types of data to support the operation of the device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, and the like. The memory 804 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0233] The power component 806 provides power to the various components of the electronic device 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

[0234] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each of the front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0235] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) that is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

[0236] The I / O interface 812 provides an interface between the processing component 802 and a peripheral interface module, and the peripheral interface module may be a keyboard, a click wheel, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

[0237] The sensor component 814 includes one or more sensors for providing an assessment of various aspects of the status of the electronic device 800. For example, the sensor component 814 can detect the on / off state of the device 800, the relative positioning of components, such as the display and keypad of the electronic device 800. The sensor component 814 can also detect a change in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor component 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 814 may further include an acceleration sensor, a gyro sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0238] The communication component 816 is configured to facilitate communication between the electronic device 800 and other devices in a wired or wireless manner. The electronic device 800 can access a wireless network based on communication standards, such as WiFi, a carrier network (such as 2G, 3G, 4G, or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0239] In an embodiment, the electronic device 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the image classification prediction method or the image classification prediction model training method shown above.

[0240] In an embodiment, there is also provided a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of the electronic device, the electronic device 800 is enabled to execute the image classification prediction method or the image classification prediction model training method shown above. For example, the computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0241] In an embodiment, there is also provided a computer program product, including a computer program or instructions. When the computer program or instructions are executed by a processor 820 of the electronic device 800, the electronic device 800 is enabled to execute the image classification prediction method or the image classification prediction model training method described above.

[0242] Figure 9 It is a block diagram of an apparatus 900 for an image classification prediction method or an image classification prediction model training provided by an embodiment of the present invention. For example, the apparatus 900 may be provided as a server. Referring to Figure 9 , the apparatus 900 includes a processing component 922, which further includes one or more processors, and memory resources represented by a memory 932 for storing instructions executable by the processing component 922, such as application programs. The application programs stored in the memory 932 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 922 is configured to execute instructions to perform the method described above.

[0243] The apparatus 900 may further include a power component 926 configured to perform power management of the apparatus 900, a wired or wireless network interface 950 configured to connect the apparatus 900 to a network, and an input / output (I / O) interface 958. The apparatus 900 may operate based on an operating system stored in the memory 932, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, or the like.

[0244] Other embodiments of the present invention will be readily apparent to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include known common general knowledge or conventional technical means in the technical field not disclosed herein. The specification and examples are only to be considered as exemplary, and the true scope and spirit of the invention are pointed out by the following claims.

[0245] It should be understood that the present invention is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. An image classification and prediction method, characterized in that, Including: Obtain image data to be classified; Use the trained image classification prediction model to perform classification prediction processing on the image data to be classified, and obtain a predicted classification result of the image classification feature. Among them, the image classification prediction model is obtained by training the counterfactual attention supervised classification model and removing the model obtained by the forged attention transformation in the trained counterfactual attention supervised classification model; The using the trained image classification prediction model to perform classification prediction processing on the image classification feature to obtain the predicted classification result of the image classification feature includes: Extract the image classification feature of the image data to be classified; Perform attention mapping calculation on the image classification feature to obtain an attention mapping feature; Perform weighted processing on the attention mapping feature and the image classification feature, and extract key features; Perform max pooling and linear transformation on the key features extracted after weighted processing to obtain an attention linear transformation feature; Perform deconfounding processing on the attention linear transformation feature to obtain the predicted classification result of the image classification feature; The performing deconfounding processing on the attention linear transformation feature to obtain the predicted classification result of the image classification feature includes: Use causal inference to remove the confounding factor in the attention linear transformation feature, and obtain the predicted classification result of the corrected image classification feature.

2. The image classification prediction method according to claim 1, wherein The method further includes: pre-training the image classification prediction model in the following manner to obtain a trained long-tail classification model: Obtain a training set, where the training set includes: various image data and corresponding true classification labels; Input each image data into the counterfactual attention supervised classification model for training; during the training process, extract the image classification feature of each image data; perform counterfactual attention supervision processing on the image classification feature to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features; perform deconfounding training on the attention linear transformation feature and the pseudo-attention linear transformation feature respectively to obtain the corresponding predicted results after deconfounding training; calculate the loss value according to the difference between each predicted result and the true classification label; update the parameters of the counterfactual attention supervised classification model based on the loss value through the backpropagation mechanism, and after multiple iterative trainings until the counterfactual attention supervised classification model converges, obtain the trained counterfactual attention supervised classification model, and use the model obtained by removing the forged attention transformation in the trained counterfactual attention supervised classification model as the image classification prediction model.

3. The image classification prediction method according to claim 2, wherein The performing counterfactual attention supervision processing on the image classification feature to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features includes: Perform attention mapping calculation on the image classification feature to obtain an attention mapping feature; Perform counterfactual random calculation on the image classification feature to obtain a pseudo-attention mapping feature; Weight the image classification features with the attention mapping features and the pseudo-attention mapping features respectively, and extract the key features in the attention mapping features and the pseudo-attention mapping features respectively; Perform max pooling and linear transformation on the key features in the respectively extracted attention mapping features and pseudo-attention mapping features to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features.

4. The image classification prediction method according to claim 2, wherein The calculating of the loss value according to the difference between each prediction result and the true classification label includes: Calculate the difference between the prediction result after deconfounding training of the attention linear transformation features and the prediction result after deconfounding training of the pseudo-attention linear transformation features as the first difference; Calculate the cross-entropy loss function based on the first difference and the true classification label as the first cross-entropy loss value; Calculate the difference between the prediction result after deconfounding training of the attention linear transformation features and the true classification label y as the second cross-entropy loss value; Set a hyperparameter for the sum of the first cross-entropy loss value and the second cross-entropy loss value as the loss value, where the value of the hyperparameter is set to any value between 0 and 1.

5. A method for training an image classification prediction model, characterized in that, including: Obtain a training set, which includes: various image data and corresponding true classification labels; Input each kind of image data into the counterfactual attention supervised classification model for training; during the training process, extract the image classification features of each kind of image data; perform counterfactual attention supervision processing on the image classification features to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features; perform deconfounding training on the image classification features with the attention linear transformation features and the pseudo-attention linear transformation features respectively to obtain corresponding prediction results after deconfounding training; calculate the loss value according to the difference between each prediction result and the true classification label; update the parameters of the counterfactual attention supervised classification model through the backpropagation mechanism based on the loss value; after multiple iterative trainings until the counterfactual attention supervised classification model converges, obtain the trained counterfactual attention supervised classification model, and remove the model obtained after the forged attention transformation in the trained counterfactual attention supervised classification model as the image classification prediction model.

6. An image classification prediction device, characterized in that, including: An acquisition module for acquiring the image data to be classified; A prediction module for using the trained image classification prediction model to perform classification prediction processing on the image data to be classified to obtain a predicted classification result of the image classification features, where the image classification prediction model is based on the training of the counterfactual attention supervised classification model and removes the model obtained after the forged attention transformation in the trained counterfactual attention supervised classification model; The prediction module includes: a first extraction module, a first attention mapping module, a first weighting module, a first transformation module, a first transformation module and a first deconfounding processing module, where The first extraction module is used to extract the image classification features of the image data to be classified; The first attention mapping module is used to perform attention mapping calculation on the image classification features to obtain attention mapping features; The first weighting module is used to perform weighting processing on the attention mapping features and the image classification features to extract key features; The first transformation module is used to perform max pooling and linear transformation on the weighted features to obtain attention linear transformation features; The first deconfounding processing module is used to perform deconfounding processing on the attention linear transformation features to obtain the predicted classification result of the image classification features; The first deconfounding processing module is used to use causal reasoning to remove the confounding factors in the attention linear transformation features to obtain the corrected predicted classification result of the image classification features.

7. An image classification prediction model training device, characterized in that: The acquisition module is used to acquire a training set, and the training set includes: various image data and corresponding true classification labels; The input module is used to input each image data into the counterfactual attention supervised classification model for training; The extraction module extracts the image classification features of each image data during the training process; The counterfactual attention supervision module is used to perform counterfactual attention supervision processing on the image classification features to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features; The deconfounding training module is used to perform deconfounding training on the image classification features, the attention linear transformation features, and the pseudo-attention linear transformation features respectively to obtain the corresponding predicted results after deconfounding training; The loss calculation module is used to calculate the loss value according to the difference between each predicted result and the true classification label; The update module is used to update the parameters of the counterfactual attention supervised classification model through the backpropagation mechanism based on the loss value; The iterative training module is used to perform multiple iterative trainings until the counterfactual attention supervised classification model converges, obtain the trained counterfactual attention supervised classification model, and remove the model obtained based on the forged attention transformation in the trained counterfactual attention supervised classification model as the image classification prediction model.

8. An electronic device, characterized in that, Comprising: A processor; A memory for storing executable instructions of the processor; Wherein, the processor is configured to execute the instructions to implement the image classification prediction method according to any one of claims 1 to 4 or the image classification prediction model training method according to claim 7.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device can execute the image classification prediction method according to any one of claims 1 to 4 or the image classification prediction model training method according to claim 5.

Images