Image classification prediction method and device, model training method and device, equipment and medium

Abstract

The invention provides an image classification prediction method and device, a model training method and device, equipment and a medium. The prediction method comprises the following steps: acquiring to-be-classified image data; and performing classification prediction processing on the to-be-classified image data by using a trained image classification prediction model to obtain a prediction classification result of the image classification features, the image classification prediction model being obtained by training an anti-fact attention supervision classification model. And a model obtained on the basis of forged attention transformation in the trained anti-fact attention supervision classification model is removed. According to the method, feature extraction is carried out on the image data to be classified, the trained image classification prediction model is utilized, anti-fact long-tail classification and an anti-fact-based attention supervision mechanism are combined, recall of tail classes in the image data is improved, and in the face of a complex long-tail classification task, the accuracy of the long-tail classification is improved. The method has good key feature extraction capability, and improves the image classification effect and the robustness of a classification prediction model.

Description

Technical Field

[0001] The present invention relates to the field of image classification technology, and in particular to an image classification prediction method, a model training method, a device, an electronic device and a computer-readable storage medium. Background Art

[0002] With the advent of the era of artificial intelligence (AI) big models, giving big models human reasoning capabilities has attracted more and more attention, and causal reasoning is expected to become one of the effective ways to achieve this goal. Counterfactuals and interventions are two powerful analytical tools for causal reasoning. With the help of do-expressions in causal reasoning, the value or state of elements in the SCM model can be controlled to achieve certain practical purposes. Long-tail classification refers to the fact that a considerable number of categories in the data to be classified have very few samples, only a few categories have more samples, located at the head, while most categories have fewer samples, located at the tail. The long-tail distribution will cause the model to overfit the features of the head category, making it difficult to learn effective tail class features, thereby affecting the recall of the tail class.

[0003] In the related technology, although the causal relationship can be used to effectively improve the recall of the tail class, it cannot effectively solve the category confusion and intra-class bias problems in the long-tail classification, thus affecting the robustness of the long-tail classification.

[0004] Therefore, how to effectively solve the problems of intra-class bias and inter-class confusion in long-tail classification tasks in order to improve the robustness of long-tail classification is a problem that needs to be solved. Summary of the invention

[0005] The present invention provides an image classification prediction method, a model training method, an apparatus, an electronic device, and a computer-readable storage medium, to at least solve the problem that the related art cannot effectively solve the intra-class bias and inter-class confusion problems in image classification tasks, resulting in poor image classification results. The technical solution of the present invention is as follows:

[0006] According to a first aspect of an embodiment of the present invention, there is provided an image classification prediction method, comprising:

[0007] Obtain image data to be classified;

[0008] The image data to be classified is classified and predicted using a trained image classification prediction model to obtain a predicted classification result of the image classification feature, wherein the image classification prediction model is based on training a counterfactual attention supervised classification model and removing a model obtained after a pseudo-attention transformation is performed on the trained counterfactual attention supervised classification model.

[0009] Optionally, the using a trained image classification prediction model to perform classification prediction processing on the image classification feature to obtain a prediction classification result of the image classification feature includes:

[0010] Extracting image classification features of the image data to be classified;

[0011] Performing attention mapping calculation on the image classification features to obtain attention mapping features;

[0012] Performing weighted processing on the attention mapping feature and the image classification feature to extract key features;

[0013] Perform maximum pooling and linear transformation on the key features extracted after weighted processing to obtain the attention linear transformation features;

[0014] The attention linear transformation feature is deobfuscated to obtain a predicted classification result of the image classification feature.

[0015] Optionally, the deobfuscation processing is performed on the attention linear transformation feature to obtain a predicted classification result of the image classification feature, including:

[0016] Causal reasoning is used to remove the confusion factors in the attention linear transformation feature to obtain the predicted classification result of the corrected image classification feature.

[0017] Optionally, the method further includes: pre-training the image classification prediction model in the following manner to obtain a trained long-tail classification model:

[0018] Obtaining a training set, wherein the training set includes: various image data and corresponding true classification labels;

[0019] Each type of image data is input into a counterfactual attention supervised classification model for training; during the training process, image classification features of each type of image data are extracted; the image classification features are subjected to counterfactual attention supervision processing to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features; the attention linear transformation features and the pseudo-attention linear transformation are subjected to deconfusion training respectively to obtain corresponding prediction results after deconfusion training; the loss value is calculated according to the difference between each prediction result and the true classification label; the parameters of the counterfactual attention supervised classification model are updated through a back propagation mechanism based on the loss value, and after multiple iterative trainings until the counterfactual attention supervised classification model converges, a trained counterfactual attention supervised classification model is obtained, and the model obtained after removing the pseudo-attention transformation from the trained counterfactual attention supervised classification model is used as an image classification prediction model.

[0020] Optionally, the image classification features are subjected to counterfactual attention supervision processing to obtain corresponding attention linear transformation features and pseudo attention linear transformation features, including:

[0021] Performing attention mapping calculation on the image classification features to obtain attention mapping features;

[0022] Performing counterfactual random calculation on the image classification features to obtain pseudo attention mapping features;

[0023] The image classification feature is weighted with the attention mapping feature and the pseudo attention mapping feature, respectively, to extract key features from the attention mapping feature and the pseudo attention mapping feature;

[0024] The key features of the respectively extracted attention mapping features and pseudo attention mapping features are subjected to maximum pooling and linear transformation to obtain corresponding attention linear transformation features and pseudo attention linear transformation features.

[0025] Optionally, calculating the loss value according to the difference between each prediction result and the true classification label includes:

[0026] Calculate the difference between the prediction result after the attention linear transformation feature de-obfuscation training and the prediction result after the pseudo attention linear transformation feature de-obfuscation training as the first difference;

[0027] Calculate the difference between the first difference and the prediction result after the attention linear transformation feature de-obfuscation training as the second difference;

[0028] Calculate a cross entropy loss function based on the second difference and the true classification label as a first cross entropy loss value;

[0029] Calculate the difference between the prediction result after the pseudo attention linear transformation feature de-obfuscation training and the true classification label y as the second cross entropy loss value;

[0030] A hyperparameter is set for the sum of the first calculated cross entropy loss value and the first calculated cross entropy loss value as the loss value; wherein the value of the hyperparameter is set to any value between 0 and 1.

[0031] According to a second aspect of an embodiment of the present invention, there is provided an image classification prediction model training method, comprising:

[0032] Obtaining a training set, wherein the training set includes: various image data and corresponding true classification labels;

[0033] Each type of image data is input into a counterfactual attention supervised classification model for training; during the training process, image classification features of each type of image data are extracted; the image classification features are subjected to counterfactual attention supervision processing to obtain corresponding attention linear transformation features and pseudo attention linear transformation features; the image classification features are subjected to deconfusion training with the attention linear transformation features and the pseudo attention linear transformation respectively to obtain corresponding prediction results after deconfusion training; the loss value is calculated according to the difference between each prediction result and the true classification label; the parameters of the counterfactual attention supervised classification model are updated through a back propagation mechanism based on the loss value; after multiple iterative trainings until the counterfactual attention supervised classification model converges, a trained counterfactual attention supervised classification model is obtained, and the model obtained after removing the pseudo attention transformation from the trained counterfactual attention supervised classification model is used as an image classification prediction model.

[0034] According to a third aspect of an embodiment of the present invention, there is provided an image classification device, comprising:

[0035] An acquisition module, used for acquiring image data to be classified;

[0036] A prediction module is used to perform classification prediction processing on the image data to be classified using a trained image classification prediction model to obtain a predicted classification result of the image classification feature, wherein the image classification prediction model is based on training a counterfactual attention supervised classification model and removing a model obtained after a pseudo-attention transformation is performed on the trained counterfactual attention supervised classification model.

[0037] Optionally, the prediction module includes:

[0038] A first extraction module, used to extract image classification features of the image data to be classified;

[0039] A first attention mapping module, configured to perform attention mapping calculation on the image classification features to obtain attention mapping features;

[0040] A first weighting module, used for weighting the attention mapping feature and the image classification feature to extract key features;

[0041] The first transformation module is used to perform maximum pooling and linear transformation on the weighted features to obtain attention linear transformation features;

[0042] The first de-obfuscation processing module is used to perform de-obfuscation processing on the attention linear transformation feature to obtain a predicted classification result of the image classification feature.

[0043] Optionally, the first de-obfuscation processing module is specifically used to use causal reasoning to remove the confusion factor in the attention linear transformation feature to obtain the predicted classification result of the corrected image classification feature.

[0044] Optionally, the device further includes: a training module, used to pre-train the image classification prediction model to obtain a trained long-tail classification model.

[0045] Optionally, the training module includes:

[0046] A second acquisition module is used to acquire a training set, wherein the training set includes: various image data and corresponding real classification labels;

[0047] An input module, used to input each image data into the counterfactual attention supervised classification model for training;

[0048] A second extraction module is used to extract image classification features of each type of image data during training;

[0049] A counterfactual attention supervision module is used to perform counterfactual attention supervision processing on the image classification features to obtain corresponding attention linear transformation features and pseudo attention linear transformation features;

[0050] A second de-obfuscation training module is used to perform de-obfuscation training on the attention linear transformation feature and the pseudo attention linear transformation respectively to obtain a corresponding prediction result after the de-obfuscation training;

[0051] A loss calculation module, used to calculate the loss value according to the difference between each prediction result and the true classification label;

[0052] An updating module, configured to update the parameters of the counterfactual attention supervision classification model through a back-propagation mechanism based on the loss value;

[0053] An iterative training module is used to perform multiple iterative training until the counterfactual attention supervised classification model converges to obtain a trained counterfactual attention supervised classification model, and remove the model obtained after the fake attention transformation in the trained counterfactual attention supervised classification model as an image classification prediction model.

[0054] Optionally, the counterfactual attention supervision module includes:

[0055] A second attention mapping module is used to perform attention mapping calculation on the image classification features to obtain attention mapping features;

[0056] A counterfactual random calculation module, used for performing counterfactual random calculation on the image classification features to obtain pseudo attention mapping features;

[0057] A second weighted processing module is used to perform weighted processing on the image classification feature, the attention mapping feature and the pseudo attention mapping feature, respectively, to extract key features from the attention mapping feature and the pseudo attention mapping feature;

[0058] The pooling transformation module is used to perform maximum pooling and linear transformation on the key features of the attention mapping features and pseudo-attention mapping features extracted respectively, so as to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features.

[0059] Optionally, the loss calculation module includes:

[0060] A first difference calculation module, used to calculate the difference between the prediction result after the attention linear transformation feature de-obfuscation training and the prediction result after the pseudo attention linear transformation feature de-obfuscation training as a first difference;

[0061] A second difference calculation module is used to calculate the difference between the first difference and the prediction result after the attention linear transformation feature de-obfuscation training as the second difference;

[0062] A first cross entropy loss module, configured to calculate a cross entropy loss function based on the second difference and the true classification label as a first cross entropy loss value;

[0063] A second cross entropy loss calculation module is used to calculate the difference between the prediction result after the pseudo attention linear transformation feature de-obfuscation training and the true classification label y as a second cross entropy loss value;

[0064] A setting module is used to set a hyperparameter for the sum of the first calculated cross entropy loss value and the first calculated cross entropy loss value as a loss value, wherein the value of the hyperparameter is set to any value between 0 and 1.

[0065] According to a fourth aspect of an embodiment of the present invention, there is provided an image classification prediction model training device,

[0066] An acquisition module is used to acquire a training set, wherein the training set includes: various image data and corresponding true classification labels;

[0067] An input module, used to input each image data into the counterfactual attention supervised classification model for training;

[0068] An extraction module, which extracts image classification features of each type of image data during training;

[0069] A counterfactual attention supervision module is used to perform counterfactual attention supervision processing on the image classification features to obtain corresponding attention linear transformation features and pseudo attention linear transformation features;

[0070] A de-obfuscation training module, used to perform de-obfuscation training on the image classification features, the attention linear transformation features, and the pseudo attention linear transformation, respectively, to obtain corresponding prediction results after the de-obfuscation training;

[0071] A loss calculation module, used to calculate the loss value according to the difference between each prediction result and the true classification label;

[0072] An updating module, configured to update the parameters of the counterfactual attention supervision classification model through a back-propagation mechanism based on the loss value;

[0073] An iterative training module is used to perform multiple iterative training until the counterfactual attention supervised classification model converges to obtain a trained counterfactual attention supervised classification model, and remove the model obtained after the fake attention transformation in the trained counterfactual attention supervised classification model as an image classification prediction model.

[0074] According to a fifth aspect of an embodiment of the present invention, there is further provided an electronic device, including:

[0075] processor;

[0076] a memory for storing instructions executable by the processor;

[0077] Wherein, the processor is configured to execute the instructions to implement the image classification prediction method as described above or the image classification prediction model training method as described above.

[0078] According to the sixth aspect of an embodiment of the present invention, a computer-readable storage medium is also provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image classification prediction method as described above or the image classification prediction model training method as described above.

[0079] According to the seventh aspect of an embodiment of the present invention, a computer program product is also provided, including a computer program or instructions. When the computer program or instructions are executed by a processor of an electronic device, the electronic device executes the above-mentioned image classification prediction method or image classification prediction model training method.

[0080] The technical solution provided by the embodiments of the present invention brings at least the following beneficial effects:

[0081] In an embodiment of the present invention, image data to be classified is obtained; and a trained image classification prediction model is used to perform classification prediction processing on the image data to be classified to obtain a predicted classification result of the image classification feature, wherein the image classification prediction model is based on training a counterfactual attention supervision classification model and removing the model obtained after the fake attention transformation in the trained counterfactual attention supervision classification model. In other words, this embodiment performs feature extraction on the image data to be classified, and uses a trained image classification prediction model to combine the counterfactual long-tail classification with the counterfactual-based attention supervision mechanism, which not only improves the recall of the tail class in the image data, but also has a better key feature extraction capability in the face of complex long-tail classification tasks, thereby improving the image classification effect and the robustness of the classification prediction model.

[0082] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present invention, and are used together with the specification to explain the principles of the present invention, and do not constitute an improper limitation of the present invention. In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0084] Figure 1 It is a flowchart of an image classification prediction method provided by an embodiment of the present invention.

[0085] Figure 2 This is an application example diagram of an image classification prediction method provided by an embodiment of the present invention.

[0086] Figure 3 It is a flowchart of an image classification prediction model training method provided by an embodiment of the present invention.

[0087] Figure 4 This is an application example diagram of an image classification prediction model training method provided by an embodiment of the present invention.

[0088] Figure 5 It is a block diagram of an image classification prediction device provided by an embodiment of the present invention.

[0089] Figure 6 It is a block diagram of an image classification prediction model training device provided by an embodiment of the present invention.

[0090] Figure 7 This is a diagram of an application example of an image classification prediction model training device provided by an embodiment of the present invention.

[0091] Figure 8 It is a block diagram of an electronic device provided by an embodiment of the present invention.

[0092] Fig. 9 It is a block diagram of a device for image classification or image classification prediction model training provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0093] In order to enable ordinary persons in the art to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0094] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0095] In recent years, research on computer vision, deep learning, machine learning, image processing, image recognition and other technologies based on artificial intelligence has made important progress. Artificial Intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies and application systems for simulating and extending human intelligence. Artificial intelligence is a comprehensive discipline involving many types of technologies such as chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, neural networks, etc. Computer vision, as an important branch of artificial intelligence, specifically allows machines to recognize the world. Computer vision technology usually includes face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, target detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, robot navigation and positioning and other technologies. With the research and advancement of artificial intelligence technology, this technology has been applied in many fields, such as security, urban management, traffic management, building management, park management, facial access, facial attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone imaging, cloud services, smart homes, wearable devices, unmanned driving, automatic driving, smart medical care, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile Internet, live streaming, beauty, makeup, medical beauty, and smart temperature measurement.

[0096] Technical terms:

[0097] Causal Inference,

[0098] One of the core issues in statistics and data science is the process of drawing causal conclusions when a phenomenon has already occurred.

[0099] Structural causal model (SCM) is a method that expresses the causal relationship behind data in the form of a directed acyclic graph, which is used to describe the associations and interactions between individuals in the real world.

[0100] Counterfactual is a reflection on what happened in the past, and it solves the problem of "If I had acted differently in the past, how would the result be different now?" For example, Zhang San did not pass the exam, and he would think if he did not watch TV series frequently before the exam, would he pass the final exam?

[0101] Intervention is the process of implementing certain behaviors on a target to study certain effects before and after the behaviors are implemented.

[0102] Attention, a learnable module that helps the model capture key features in an image.

[0103] Based on the understanding of the above technical terms, please refer to the following embodiments.

[0104] Figure 1 is a flowchart of an image classification prediction method provided by an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0105] Step 101: Obtain image data to be classified;

[0106] Step 102: Use the trained image classification prediction model to perform classification prediction processing on the image data to be classified to obtain the predicted classification results of the image classification features, wherein the image classification prediction model is based on training the counterfactual attention supervised classification model and removing the model obtained after the trained counterfactual attention supervised classification model is based on the pseudo-attention transformation.

[0107] The image classification prediction method described in the present invention can be applied to terminals, servers, etc., without limitation herein. The terminal implementation device can be an electronic device such as a smart phone, a laptop computer, a tablet computer, a desktop computer, a personal digital assistant (PDA, Personal Digital Assistant) and a wearable device. The server can be an independent server or a server cluster, or a server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, intermediate services, domain name services, security services, content distribution networks, or a big data and artificial intelligence platform, etc., without limitation herein.

[0108] Combine the following Figure 1 , the specific implementation steps of an image classification prediction method provided by an embodiment of the present invention are described in detail.

[0109] In step 101, image data to be classified is obtained.

[0110] In this step, the image data to be classified is obtained, which can be obtained locally, remotely, or in real time, etc. This embodiment is not limited. The image to be classified can be an ordinary image or a long-tail image, such as a wild animal image, etc. Since many wild animals have very similar appearances, such as aardvarks and anteaters, they are usually difficult to distinguish. They not only belong to the tail class in the long-tail distribution, but also have low differentiation, which is more challenging for the long-tail classification task. Therefore, for the long-tail image, if you want to identify what animal in the long-tail image and which category it belongs to, you can use the image classification prediction model provided by the present invention to predict and classify it.

[0111] In step 102, the image data to be classified is classified and predicted using a trained image classification prediction model to obtain a predicted classification result of the image classification feature, wherein the image classification prediction model is based on training a counterfactual attention supervised classification model and removing a model obtained after a pseudo-attention transformation is performed on the trained counterfactual attention supervised classification model.

[0112] In this step, specifically, 1) image classification features of the image data to be classified are extracted.

[0113] In this step, the image classification features of the image data to be classified can be extracted through the backbone network in the image classification prediction model, wherein the backbone network can be a convolutional neural network (CNN, Convolutional Neural Networks), etc. The specific feature extraction process is already a well-known technology and will not be repeated here.

[0114] 2) Performing attention mapping calculation on the image classification features to obtain attention mapping features.

[0115] In this step, the attention supervision module in the image classification prediction model can perform attention mapping calculation on the image classification features to obtain attention mapping features; wherein the attention supervision module adopts an attention supervision mechanism, and its process of performing attention mapping calculation on the image classification features is a well-known technology and will not be repeated here.

[0116] 3) Perform weighted processing on the attention mapping features and the image classification features to extract key features.

[0117] In this step, the weighted processing is to perform element-by-element dot multiplication of the attention map feature and the image classification feature, thereby extracting the key features in the feature map. The specific extraction process is detailed in the following training process and will not be repeated here.

[0118] 4) Perform maximum pooling and linear transformation on the key features extracted after weighted processing to obtain the attention linear transformation features.

[0119] In this step, the maximum pooling operation is first performed on the key features after weighted processing to convert them into a one-dimensional feature vector. Then, the one-dimensional feature vector obtained after the conversion is linearly transformed to obtain the attention linear transformation feature, wherein the linear transformation can be convolving the one-dimensional vector with the linear transformation matrix to obtain the attention linear transformation feature, etc. Of course, this embodiment is not limited to this linear transformation method, and other linear transformation methods can also be used, which is not limited in this embodiment.

[0120] 5) De-obfuscating the attention linear transformation feature to obtain a predicted classification result of the image classification feature.

[0121] The de-obfuscation process of the attention linear transformation feature to obtain the predicted classification result of the image classification feature includes: using causal reasoning to remove the confusion factor in the attention linear transformation feature to obtain the corrected predicted classification result of the image classification feature. De-obfuscation is performed using counterfactuals and intervention learning in causal reasoning to eliminate the harmful effects of the confusion factor and obtain the corrected prediction result. The process of learning to remove momentum confusion through intervention is specifically described in the corresponding process of the model training below, which will not be repeated here.

[0122] It should be noted that in the embodiment of the present invention, the image classification prediction model is based on training the counterfactual attention supervision classification model and removing the model obtained after the counterfactual attention supervision classification model is trained based on the fake attention transformation. In other words, the classification prediction processing process of the image classification prediction model is basically similar to the training process of the counterfactual attention supervision classification model, except that the part based on the fake attention transformation in the counterfactual attention supervision classification model needs to be removed. The specific process is described below and will not be repeated here.

[0123] In an embodiment of the present invention, image data to be classified is obtained; and a trained image classification prediction model is used to perform classification prediction processing on the image data to be classified to obtain a predicted classification result of the image classification feature, wherein the image classification prediction model is based on training a counterfactual attention supervision classification model and removing the model obtained after the fake attention transformation in the trained counterfactual attention supervision classification model. In other words, this embodiment performs feature extraction on the image data to be classified, and uses a trained image classification prediction model to combine the counterfactual long-tail classification with the counterfactual-based attention supervision mechanism, which not only improves the recall of the tail class in the image data, but also has a better key feature extraction capability in the face of complex long-tail classification tasks, thereby improving the image classification effect and the robustness of the classification prediction model.

[0124] See also Figure 2 , which is an application example diagram of an image classification prediction method provided by an embodiment of the present invention. The image data in this embodiment takes long-tail image data as an example, but in actual applications, it is not limited to this.

[0125] like Figure 2 As shown, first, the long-tail image is input into the backbone network (Backbone, or backbone network, such as ResNet, VGG, etc.) in the image classification prediction model for feature extraction to obtain image classification features X (Feature map, i.e., feature map). After that, the image classification features are input into the attention mapping module (Attention Maps A) in the image classification prediction model for attention mapping calculation to obtain attention mapping features; the attention mapping features and the image classification features are weighted to extract key features; the key features extracted after weighted processing are input into the pooling linear transformation module (Pooling+FC) in the image classification prediction model for maximum pooling and linear transformation processing, and the attention linear transformation features are output; the specific maximum pooling and linear transformation processing process is described below and will not be repeated here. Finally, the attention linear transformation features are input into the causal reasoning head (Casual Head) in the image classification prediction model for de-obfuscation processing to remove the confusion factors in the attention linear transformation features and obtain the predicted classification result Y(A) of the corrected image classification features.

[0126] That is to say, in the embodiment of the present invention, the long-tail image is subjected to feature extraction through the backbone network and weighted with the attention map generated by the attention mapping module, the weighted output is pooled and linearly transformed, and finally input into the causal reasoning head for deobfuscation processing to obtain the deobfuscated output, that is, the predicted classification result.

[0127] It should be noted that, in this embodiment, the application of the image classification prediction model is similar to the subsequent training process. The difference is that an intervention process based on pseudo-attention is added to the following training process, so that the model has the ability to extract key features to distinguish different categories. Therefore, in the deployment model reasoning stage (ie, application node), the pseudo-attention mapping module is no longer necessary and can be directly removed from the model structure, thereby making the reasoning process more efficient than the training process.

[0128] Optionally, in another embodiment, based on the above embodiment, the method may further include: pre-training the image classification prediction model to obtain a trained long-tail classification model. Specifically including:

[0129] A training set is obtained, wherein the training set includes: various image data and corresponding real classification labels; each image data is input into a counterfactual attention supervised classification model for training; during the training process, image classification features of each image data are extracted; the image classification features are subjected to counterfactual attention supervision processing to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features; de-obfuscation training is performed on the attention linear transformation features and the pseudo-attention linear transformation respectively to obtain corresponding prediction results after de-obfuscation training; a loss value is calculated according to the difference between each prediction result and the real classification label; based on the loss value, the parameters of the counterfactual attention supervised classification model are updated through a back-propagation mechanism, and after multiple iterative trainings until the counterfactual attention supervised classification model converges, a trained counterfactual attention supervised classification model is obtained, and a model obtained after removing the pseudo-attention transformation from the trained counterfactual attention supervised classification model is used as an image classification prediction model.

[0130] In an embodiment of the present invention, each type of image data in the acquired training set is input into the counterfactual attention supervision classification model for training. During the training process, the image classification features of the image data are first extracted, and then the extracted features are transformed and calculated after counterfactual attention supervision, and the two-way output of counterfactual attention supervision and the image classification features are sent to the causal reasoning head module for de-obfuscation processing, and the predicted training results are output, the loss value is calculated based on the predicted training results, the output of forward reasoning based on the loss value, and the parameters of the model are updated by back-propagating the loss function value. In this embodiment, the counterfactual-based long-tail classification is combined with the counterfactual-based attention supervision mechanism for training, which can not only improve the recall of the tail class, but also have better key feature extraction capabilities in the face of complex long-tail classification tasks, thereby improving the robustness of the model.

[0131] Optionally, in another embodiment, based on the above-mentioned embodiment, the image classification features are subjected to counterfactual attention supervision processing to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features, including: performing attention mapping calculation on the image classification features to obtain attention mapping features; performing counterfactual random calculation on the image classification features to obtain pseudo-attention mapping features; weighting the image classification features with the attention mapping features and the pseudo-attention mapping features, respectively, to extract key features from the attention mapping features and the pseudo-attention mapping features; performing maximum pooling and linear transformation on the key features of the respectively extracted attention mapping features and pseudo-attention mapping features to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features.

[0132] Optionally, in another embodiment, based on the above embodiment, the loss value is calculated according to the difference between each prediction result and the true classification label, including: calculating the difference between the prediction result after the attention linear transformation feature deconfusion training and the prediction result after the pseudo attention linear transformation feature deconfusion training as a first difference; calculating the difference between the first difference and the prediction result after the attention linear transformation feature deconfusion training as a second difference; calculating the cross entropy loss function based on the second difference and the true classification label as a first cross entropy loss value; calculating the difference between the prediction result after the pseudo attention linear transformation feature deconfusion training and the true classification label y as a second cross entropy loss value; setting a hyperparameter for the sum of the first calculated cross entropy loss value and the first calculated cross entropy loss value as a loss value, wherein the value of the hyperparameter is set to any value between 0 and 1.

[0133] It should be noted that the model training process of this embodiment is described in detail below and will not be repeated here.

[0134] See also Figure 3 , is a flowchart of an image classification prediction model training method provided by an embodiment of the present invention, the method comprising:

[0135] Step 301: Obtain a training set, wherein the training set includes: various image data and corresponding real classification labels;

[0136] Step 302: Input each type of image data into a counterfactual attention supervised classification model for training; during the training process, extract the image classification features of each type of image data; perform counterfactual attention supervision processing on the image classification features to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features; perform deconfusion training on the attention linear transformation features and the pseudo-attention linear transformation respectively to obtain corresponding prediction results after deconfusion training; calculate the loss value according to the difference between each prediction result and the true classification label; update the parameters of the counterfactual attention supervised classification model through a back propagation mechanism based on the loss value; after multiple iterative trainings, until the counterfactual attention supervised classification model converges, obtain a trained counterfactual attention supervised classification model, and remove the model obtained after removing the pseudo-attention transformation from the trained counterfactual attention supervised classification model as an image classification prediction model.

[0137] Optionally, in another embodiment, based on the above-mentioned embodiment, the image classification features are subjected to counterfactual attention supervision processing to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features, including: performing attention mapping calculation on the image classification features to obtain attention mapping features; performing counterfactual random calculation on the image classification features to obtain pseudo-attention mapping features; weighting the image classification features with the attention mapping features and the pseudo-attention mapping features, respectively, to extract key features from the attention mapping features and the pseudo-attention mapping features; performing maximum pooling and linear transformation on the key features of the respectively extracted attention mapping features and pseudo-attention mapping features to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features.

[0138] Optionally, in another embodiment, based on the above embodiment, the loss value is calculated according to the difference between each prediction result and the true classification label, including: calculating the difference between the prediction result after the attention linear transformation feature deconfusion training and the prediction result after the pseudo attention linear transformation feature deconfusion training as a first difference; calculating the difference between the first difference and the prediction result after the attention linear transformation feature deconfusion training as a second difference; calculating the cross entropy loss function based on the second difference and the true classification label as a first cross entropy loss value; calculating the difference between the prediction result after the pseudo attention linear transformation feature deconfusion training and the true classification label y as a second cross entropy loss value; setting a hyperparameter for the sum of the first calculated cross entropy loss value and the first calculated cross entropy loss value as a loss value, wherein the value of the hyperparameter is set to any value between 0 and 1.

[0139] See also Figure 4 , is an application example diagram of an image classification prediction model training method provided by an embodiment of the present invention. For the sake of convenience, the input image data in this embodiment is long-tail image data, but in actual applications, it is not limited to this. Specifically including:

[0140] like Figure 4 As shown in the figure, 1) the long-tail image is input into the backbone network (Backbone, or main network, etc.) in the image classification prediction model for feature extraction to obtain the image classification feature X (Feature map, i.e., feature map), and the dimension of the image classification feature X is H×W×C, where H and W represent the height and width of the extracted channel respectively, and C represents the number of feature channels.

[0141] 2) Input the image classification features into the attention mapping module (AttentionMaps A, which can be called a model) in the image classification prediction model to perform attention mapping calculation to obtain the attention mapping feature A. The attention mapping calculation formula is shown in Formula 1.

[0142]

[0143] represents a certain attention model, X represents the image feature map extracted by the backbone network (consisting of one or more two-dimensional matrices), represents the attention feature map extracted by the attention model (composed of one or more two-dimensional matrices), and A represents the set of attention maps obtained. There are M sets of them, A1, A2, and A M are all elements in the set. A square matrix is ​​a two-dimensional matrix with equal width and height.

[0144] 3) Input the image classification features into the counterfactual attention mapping module (Counterfactual Attention Maps A) in the image classification prediction model to perform counterfactual attention mapping calculations and obtain pseudo attention mapping features A Specifically, it can be generated by random sampling of normal distribution or uniform distribution, or by negating the value in the pseudo attention mapping feature A. Its function is to use the counterfactual theory to drive the attention module to realize the learning of key features. The specific formal definition can refer to formula (2) and formula (3).

[0145]

[0146] in, and is the fake attention map matrix, c represents the classifier, Represents the intervention operation, which means that the fixed image feature X remains unchanged, and the classifier output value under the forged attention mapping. The other parameters in this formula have been explained in other formulas. Please refer to other formulas for details.

[0147]

[0148] In this formula, γ represents the distribution followed by the generated fake attention map, and indicates that the forged attention map obeys the distribution represented by γ (such as uniform distribution), represents the calculation of the mean of the random variable, Y(A=A,X=X) represents the classifier output when the attention map is A and the image feature is equal to X, The parameters of this formula are explained in formula (2). The difference between the two before and after is Y effectRepresents the direct effect of the true attention map learned by the model on the predicted output.

[0149] 4) Perform weighted processing on the image classification features, the attention mapping features and the pseudo-attention mapping features respectively, and extract key features corresponding to the attention mapping features and the pseudo-attention mapping features.

[0150] In this step, the image classification feature pseudo attention mapping feature A The attention map feature A is element-wise multiplied with the image classification feature X, and the corresponding key features in the feature map are extracted. The specific calculation method refers to formulas (4) and (5), and two sets of outputs are generated accordingly.

[0151]

[0152] Where H and W represent the height and width of X in formula (1), respectively. h and w start from 1 and traverse to H and W, respectively. h，w represents the value at the hth row and wth column in X, i represents the index of the feature map, represents the value of the hth row and wth column in the i-th feature map, represents global average pooling, and * represents element-by-element multiplication of two tensors. The entire formula represents the use of an attention mapping matrix to weight the element values ​​in X, and then obtain a new matrix h through global average pooling. i .

[0153] h=normalize([h1,h2,...,h M ]), (5)

[0154] in,

[0155] h1,h2 and h M is the output generated by formula (4), and normalize means to perform normalization operation on these output matrices so that the values ​​of all elements are limited to [-1, 1].

[0156] 5) The key features extracted after weighted processing are respectively input into the pooling linear transformation module (Pooling+FC) in the image classification prediction model for maximum pooling and linear transformation processing, and the attention linear transformation features and pseudo-attention linear transformation features are output.

[0157] In this step, the two sets of output results in step 4) are pooled and linearly transformed through the maximum pooling and linear transformation modules respectively, that is, the two sets of inputs are first converted into one-dimensional vectors through pooling, and then linearly transformed to output attention linear transformation features and pseudo-attention linear transformation features.

[0158] 6) Inputting the attention linear transformation features and pseudo attention linear transformation features into the causal reasoning head (Casual Head) in the image classification prediction model for de-obfuscation processing to obtain the corresponding prediction results after de-obfuscation training.

[0159] This step is to send the two sets of outputs in step 5) to the causal reasoning head module to perform deconfusion learning, and learn the model parameters after removing momentum confusion through intervention. The formal definition can refer to formulas (6) and (7). The predicted classification result of the output obtained by the attention mapping feature A is Y(A), and the predicted classification result of the output obtained by the pseudo-attention mapping feature A is Y(A).

[0160]

[0161] Where i represents the category number, argmax represents which category can obtain the maximum value in the formula, TDE is the abbreviation of total direct effect, and Y i represents the causal effect of the i-th class, d represents the value of the intermediate node, and Y d =i represents the causal effect of the i-th category under the mediation effect, x represents the image feature, do(X=x) means fixing the feature value of the image to a certain value, which is an intervention method in causal inference, x0 represents an empty input, do(X=x) means fixing the feature value of the image to an empty input. The overall meaning is to calculate the total direct effect of category i after removing confounding factors.

[0162]

[0163] Among them, K represents the number of heads of multi-head attention, τ is a scaling factor greater than 0, and since this formula draws on the theory of energy model, γ is a control parameter related to the energy model, and x k , and d k They represent the image features generated by the kth attention head, the special part of the image feature vector, and the projection part of the image features in the head class direction, respectively. represents the weight matrix belonging to the i-th category in the k-th head. The entire formula represents the formalization of model de-confusion training.

[0164] 7) Calculating the loss value according to the difference between each prediction result and the true classification label; updating the parameters of the counterfactual attention supervision classification model through the back propagation mechanism based on the loss value; after multiple iterations of training, until the counterfactual attention supervision classification model converges, a trained counterfactual attention supervision classification model is obtained, and the model obtained after removing the fake attention transformation from the trained counterfactual attention supervision classification model is used as the image classification prediction model.

[0165] In this step, the loss value is calculated, which is composed of two cross entropy loss functions L1 and L2. Loss = αL1 + (1-α)L2, where L1 = CE ((Y (A) - Y (A)), Y); L2 = CE ((Y (A), Y); where CE is the abbreviation of the cross entropy loss function, and L1 and L2 are specifically as follows:

[0166] Among them, the calculation formula of L1 is:

[0167] -ylog(Y(A)-Y( A )+1-,y)log(1-Y(A)+Y( A ))), (8)

[0168] This formula represents the predicted output of the de-obfuscation training (Y(A)-Y( A )) and the difference between the output Y(A) generated by the attention mapping feature A (i.e., the first difference). The smaller the difference, the closer the two should be. The closer the model is, the higher the quality of the Attention learned by the model is, and the stronger the ability to mine and distinguish key features of different categories is. After that, the cross entropy loss value is calculated for the first difference and the true label y to obtain the value of L1, which is called the first cross entropy loss value. The calculation formula for L2 is:

[0169] -ylog(Y(A))+(1-y)log(1-Y(A)), (9)

[0170] The formula represents the difference between Y(A) and the true classification label y (ie, the second difference).

[0171] After that, the sum of the first cross entropy loss value and the second cross entropy loss value is calculated, and the hyperparameter α is set for the sum of the first cross entropy loss value and the second cross entropy loss value, and α is any value between 0-1. The calculation formula is:

[0172] -α[ylog(Y(4)-Y( A )+(1-y)log(1-Y(A)+Y( A )))-ylog(Y(A))+(1-y)log(1-Y(A))|(10)

[0173] That is, a hyperparameter α is added between the two losses to adjust the learning intensity of counterfactual attention. For some simple long-tail tasks, this parameter can be lowered to focus on optimizing the causal head and backbone network.

[0174] It should be noted that, for the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that this implementation disclosure is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the present invention.

[0175] Figure 5 is a block diagram of an image classification prediction device provided by an embodiment of the present invention. The device comprises: an acquisition module 501 and a prediction module 502, wherein:

[0176] An acquisition module 501 is used to acquire image data to be classified;

[0177] The prediction module 502 is used to perform classification prediction processing on the image data to be classified using the trained image classification prediction model to obtain the predicted classification result of the image classification feature, wherein the image classification prediction model is based on training the counterfactual attention supervised classification model and removing the model obtained after the fake attention transformation in the trained counterfactual attention supervised classification model.

[0178] Optionally, in another embodiment, based on the above embodiment, the prediction module includes: a first extraction module, a first attention mapping module, a first weighting module, a first transformation module, a first transformation module and a first de-obfuscation processing module, wherein:

[0179] A first extraction module, used to extract image classification features of the image data to be classified;

[0180] A first attention mapping module, configured to perform attention mapping calculation on the image classification features to obtain attention mapping features;

[0181] A first weighting module, used for weighting the attention mapping feature and the image classification feature to extract key features;

[0182] The first transformation module is used to perform maximum pooling and linear transformation on the weighted features to obtain attention linear transformation features;

[0183] The first de-obfuscation processing module is used to perform de-obfuscation processing on the attention linear transformation feature to obtain a predicted classification result of the image classification feature.

[0184] Optionally, in another embodiment, based on the above embodiment, the first de-obfuscation processing module is specifically used to use causal reasoning to remove the confusion factor in the attention linear transformation feature to obtain the predicted classification result of the corrected image classification feature.

[0185] Optionally, in another embodiment, based on the above embodiment, the device further includes: a training module for pre-training the image classification prediction model to obtain a trained long-tail classification model.

[0186] Optionally, in another embodiment, based on the above embodiment, the training module includes:

[0187] A second acquisition module is used to acquire a training set, wherein the training set includes: various image data and corresponding real classification labels;

[0188] An input module, used to input each image data into the counterfactual attention supervised classification model for training;

[0189] A second extraction module is used to extract image classification features of each type of image data during training;

[0190] A counterfactual attention supervision module is used to perform counterfactual attention supervision processing on the image classification features to obtain corresponding attention linear transformation features and pseudo attention linear transformation features;

[0191] A second de-obfuscation training module is used to perform de-obfuscation training on the attention linear transformation feature and the pseudo attention linear transformation respectively to obtain a corresponding prediction result after the de-obfuscation training;

[0192] A loss calculation module, used to calculate the loss value according to the difference between each prediction result and the true classification label;

[0193] An updating module, configured to update the parameters of the counterfactual attention supervision classification model through a back-propagation mechanism based on the loss value;

[0194] An iterative training module is used to perform multiple iterative training until the counterfactual attention supervised classification model converges to obtain a trained counterfactual attention supervised classification model, and remove the model obtained after the fake attention transformation in the trained counterfactual attention supervised classification model as an image classification prediction model.

[0195] Optionally, in another embodiment, based on the above embodiment, the counterfactual attention supervision module includes:

[0196] A second attention mapping module is used to perform attention mapping calculation on the image classification features to obtain attention mapping features;

[0197] A counterfactual random calculation module, used for performing counterfactual random calculation on the image classification features to obtain pseudo attention mapping features;

[0198] A second weighted processing module is used to perform weighted processing on the image classification feature, the attention mapping feature and the pseudo attention mapping feature, respectively, to extract key features from the attention mapping feature and the pseudo attention mapping feature;

[0199] The pooling transformation module is used to perform maximum pooling and linear transformation on the key features of the attention mapping features and pseudo-attention mapping features extracted respectively, so as to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features.

[0200] Optionally, in another embodiment, based on the above embodiment, the loss calculation module includes:

[0201] A first difference calculation module, used to calculate the difference between the prediction result after the attention linear transformation feature de-obfuscation training and the prediction result after the pseudo attention linear transformation feature de-obfuscation training as a first difference;

[0202] A second difference calculation module is used to calculate the difference between the first difference and the prediction result after the attention linear transformation feature de-obfuscation training as the second difference;

[0203] A first cross entropy loss module, configured to calculate a cross entropy loss function based on the second difference and the true classification label as a first cross entropy loss value;

[0204] A second cross entropy loss calculation module is used to calculate the difference between the prediction result after the pseudo attention linear transformation feature de-obfuscation training and the true classification label y as a second cross entropy loss value;

[0205] A setting module is used to set a hyperparameter for the sum of the first calculated cross entropy loss value and the first calculated cross entropy loss value as a loss value, wherein the value of the hyperparameter is set to any value between 0 and 1.

[0206] See also Figure 6 , a block diagram of an image classification prediction model training device is also provided in an embodiment of the present invention, the device comprising: an acquisition module 601, an input module 602, an extraction module 603, a counterfactual attention supervision module 604, a de-confusion training module 605, a loss calculation module 606, an update module 607 and an iterative training module 608, wherein,

[0207] The acquisition module 601 is used to acquire a training set, wherein the training set includes: various image data and corresponding real classification labels;

[0208] An input module 602, used to input each image data into the counterfactual attention supervised classification model for training;

[0209] A second extraction module 603 extracts image classification features of each type of image data during training;

[0210] A counterfactual attention supervision module 604 is used to perform counterfactual attention supervision processing on the image classification features to obtain corresponding attention linear transformation features and pseudo attention linear transformation features;

[0211] A de-obfuscation training module 605 is used to perform de-obfuscation training on the image classification feature, the attention linear transformation feature and the pseudo attention linear transformation, respectively, to obtain a corresponding prediction result after the de-obfuscation training;

[0212] A loss calculation module 606 is used to calculate a loss value according to the difference between each prediction result and the true classification label;

[0213] An updating module 607, configured to update the parameters of the counterfactual attention supervision classification model based on the loss value through a back propagation mechanism;

[0214] The iterative training module 608 is used to update the parameters of the counterfactual attention supervision classification model through the back-propagation mechanism based on the loss value, and to obtain a trained counterfactual attention supervision classification model after multiple iterative training until the counterfactual attention supervision classification model converges. The model obtained after removing the fake attention transformation from the trained counterfactual attention supervision classification model is used as the image classification prediction model.

[0215] See also Figure 7 , which is an application example diagram of an image classification prediction model training device provided by an embodiment of the present invention, including: a backbone network module 701, a counterfactual attention supervision module 702, a causal reasoning head module 703 and a loss value calculation module 704: wherein,

[0216] The backbone network module 701 is used to extract image classification features of input image data and output image classification features;

[0217] The counterfactual attention supervision module 702 is used to generate a false attention that is opposite to the real attention learned from the image through random generation, uniform distribution sampling, etc., and its function is to drive the Attention module of the model to focus on the parts that distinguish the key features of different categories; the function of attention is to focus on the key parts of the image (such as the pointed and long mouth of the anteater, etc.), and transform the image features corresponding to the key parts. The module simultaneously generates real attention transformation features and pseudo-attention transformation features, and outputs attention transformation features and pseudo-attention transformation features. That is, the image classification features are subjected to counterfactual attention supervision processing to obtain the corresponding attention linear transformation features and pseudo-attention linear transformation features;

[0218] The causal reasoning head module 703 is used to perform de-obfuscation training using counterfactuals and intervention techniques in causal reasoning, that is, to perform de-obfuscation training on the image classification features, the attention linear transformation features, and the pseudo attention linear transformation, respectively, to obtain the corresponding prediction results after the de-obfuscation training; then, in the reasoning stage (i.e., the application stage), the harmful effects of the confusion factors can be eliminated to obtain the corrected output prediction results;

[0219] The loss value calculation module 704 is used to calculate the difference between the output of the causal reasoning head and the true label. The obtained difference is used to update the relevant parameters of the backbone network module and the Attention module through the back-propagation mechanism, thereby achieving the goal of optimizing the network. That is, the loss value is calculated according to the difference between each prediction result and the true classification label; the parameters of the counterfactual attention supervision classification model are updated through the back-propagation mechanism based on the loss value; after multiple iterations of training, until the counterfactual attention supervision classification model converges, a trained counterfactual attention supervision classification model is obtained, and the model obtained after removing the fake attention transformation from the trained counterfactual attention supervision classification model is used as the image classification prediction model.

[0220] In an embodiment of the present invention, the long-tail classification method is combined with the counterfactual-based attention mechanism, which can not only supervise the model to learn the key features to distinguish different categories, but also improve the robustness of the model in complex classification scenarios.

[0221] Optionally, the present invention further provides an electronic device, comprising:

[0222] processor;

[0223] a memory for storing instructions executable by the processor;

[0224] Wherein, the processor is configured to execute the instructions to implement the image classification prediction method as described above or the image classification prediction model training method as described above.

[0225] Optionally, the present invention also provides a computer-readable storage medium, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to execute the image classification prediction method as described above or the image classification prediction model training method as described above.

[0226] Optionally, the present invention also provides a computer program product, including a computer program or instructions, which, when executed by a processor of an electronic device, implements the image classification prediction method as described above or the image classification prediction model training method as described above.

[0227] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0228] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0229] Figure 8 800 is a block diagram of an electronic device 800 provided in an embodiment of the present invention. For example, the electronic device 800 may be a mobile terminal or a server, and the electronic device is a mobile terminal as an example for explanation in the embodiment of the present invention. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a message transceiver device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0230] Reference Figure 8 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0231] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0232] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0233] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0234] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0235] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0236] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0237] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0238] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0239] In an embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the image classification prediction method or image classification prediction model training method shown above.

[0240] In an embodiment, a computer-readable storage medium is also provided, and when the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device 800 can perform the image classification prediction method or image classification prediction model training method shown above. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0241] In an embodiment, a computer program product is also provided, including a computer program or instructions. When the computer program or instructions are executed by the processor 820 of the electronic device 800, the electronic device 800 executes the above-mentioned image classification prediction method or image classification prediction model training method.

[0242] Fig. 9 is a block diagram of a device 900 for image classification prediction method or image classification prediction model training provided by an embodiment of the present invention. For example, the device 900 can be provided as a server. Fig. 9 The apparatus 900 includes a processing component 922, which further includes one or more processors, and a memory resource represented by a memory 932 for storing instructions, such as an application, that can be executed by the processing component 922. The application stored in the memory 932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 922 is configured to execute instructions to perform the above method.

[0243] The device 900 may also include a power supply component 926 configured to perform power management of the device 900, a wired or wireless network interface 950 configured to connect the device 900 to a network, and an input / output (I / O) interface 958. The device 900 may operate based on an operating system stored in the memory 932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0244] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0245] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An image classification prediction method, characterized in that: include: Obtain image data to be classified; The image data to be classified is classified and predicted using a trained image classification prediction model to obtain a predicted classification result of the image classification feature, wherein the image classification prediction model is based on training a counterfactual attention supervised classification model and removing a model obtained after a pseudo-attention transformation is performed on the trained counterfactual attention supervised classification model.

2. The image classification prediction method according to claim 1, characterized in that: The method of using the trained image classification prediction model to perform classification prediction processing on the image classification features to obtain the prediction classification results of the image classification features includes: Extracting image classification features of the image data to be classified; Performing attention mapping calculation on the image classification features to obtain attention mapping features; Performing weighted processing on the attention mapping feature and the image classification feature to extract key features; Perform maximum pooling and linear transformation on the key features extracted after weighted processing to obtain the attention linear transformation features; The attention linear transformation feature is deobfuscated to obtain a predicted classification result of the image classification feature.

3. The image classification prediction method according to claim 2, characterized in that: The deobfuscation processing of the attention linear transformation feature to obtain the predicted classification result of the image classification feature includes: Causal reasoning is used to remove the confusion factors in the attention linear transformation feature to obtain the predicted classification result of the corrected image classification feature.

4. The image classification prediction method according to any one of claims 1 to 3, characterized in that: The method further includes: pre-training the image classification prediction model in the following manner to obtain a trained long-tail classification model: Obtaining a training set, wherein the training set includes: various image data and corresponding true classification labels; Each type of image data is input into a counterfactual attention supervised classification model for training; during the training process, image classification features of each type of image data are extracted; the image classification features are subjected to counterfactual attention supervision processing to obtain corresponding attention linear transformation features and pseudo-attention linear transformation features; the attention linear transformation features and the pseudo-attention linear transformation are subjected to deconfusion training respectively to obtain corresponding prediction results after deconfusion training; the loss value is calculated according to the difference between each prediction result and the true classification label; the parameters of the counterfactual attention supervised classification model are updated through a back propagation mechanism based on the loss value, and after multiple iterative trainings until the counterfactual attention supervised classification model converges, a trained counterfactual attention supervised classification model is obtained, and the model obtained after removing the pseudo-attention transformation from the trained counterfactual attention supervised classification model is used as an image classification prediction model.

5. The image classification prediction method according to claim 4, characterized in that: The processing of the image classification features with counterfactual attention supervision to obtain corresponding attention linear transformation features and pseudo attention linear transformation features includes: Performing attention mapping calculation on the image classification features to obtain attention mapping features; Performing counterfactual random calculation on the image classification features to obtain pseudo attention mapping features; The image classification feature is weighted with the attention mapping feature and the pseudo attention mapping feature, respectively, to extract key features from the attention mapping feature and the pseudo attention mapping feature; The key features of the respectively extracted attention mapping features and pseudo attention mapping features are subjected to maximum pooling and linear transformation to obtain corresponding attention linear transformation features and pseudo attention linear transformation features.

6. The image classification prediction method according to claim 4, characterized in that: The loss value is calculated according to the difference between each prediction result and the true classification label, including: Calculate the difference between the prediction result after the attention linear transformation feature de-obfuscation training and the prediction result after the pseudo attention linear transformation feature de-obfuscation training as the first difference; Calculate the difference between the first difference and the prediction result after the attention linear transformation feature de-obfuscation training as the second difference; Calculate a cross entropy loss function based on the second difference and the true classification label as a first cross entropy loss value; Calculate the difference between the prediction result after the pseudo attention linear transformation feature de-obfuscation training and the true classification label y as the second cross entropy loss value; A hyperparameter is set for the sum of the first calculated cross entropy loss value and the first calculated cross entropy loss value as the loss value, wherein the value of the hyperparameter is set to any value between 0 and 1.

7. A method for training an image classification prediction model, characterized in that: include: Obtaining a training set, wherein the training set includes: various image data and corresponding true classification labels; Each type of image data is input into a counterfactual attention supervised classification model for training; during the training process, image classification features of each type of image data are extracted; the image classification features are subjected to counterfactual attention supervision processing to obtain corresponding attention linear transformation features and pseudo attention linear transformation features; the image classification features are subjected to deconfusion training with the attention linear transformation features and the pseudo attention linear transformation respectively to obtain corresponding prediction results after deconfusion training; the loss value is calculated according to the difference between each prediction result and the true classification label; the parameters of the counterfactual attention supervised classification model are updated through a back propagation mechanism based on the loss value; after multiple iterative trainings until the counterfactual attention supervised classification model converges, a trained counterfactual attention supervised classification model is obtained, and the model obtained after removing the pseudo attention transformation from the trained counterfactual attention supervised classification model is used as an image classification prediction model.

8. An image classification prediction device, characterized in that: include: An acquisition module, used for acquiring image data to be classified; A prediction module is used to perform classification prediction processing on the image data to be classified using a trained image classification prediction model to obtain a predicted classification result of the image classification feature, wherein the image classification prediction model is based on training a counterfactual attention supervised classification model and removing a model obtained after a pseudo-attention transformation is performed on the trained counterfactual attention supervised classification model.

9. An image classification prediction model training device, characterized in that: An acquisition module is used to acquire a training set, wherein the training set includes: various image data and corresponding true classification labels; An input module, used to input each image data into the counterfactual attention supervised classification model for training; An extraction module, which extracts image classification features of each type of image data during training; A counterfactual attention supervision module is used to perform counterfactual attention supervision processing on the image classification features to obtain corresponding attention linear transformation features and pseudo attention linear transformation features; A de-obfuscation training module, used to perform de-obfuscation training on the image classification features, the attention linear transformation features, and the pseudo attention linear transformation, respectively, to obtain corresponding prediction results after the de-obfuscation training; A loss calculation module, used to calculate the loss value according to the difference between each prediction result and the true classification label; An updating module, configured to update the parameters of the counterfactual attention supervision classification model through a back-propagation mechanism based on the loss value; An iterative training module is used to perform multiple iterative training until the counterfactual attention supervised classification model converges to obtain a trained counterfactual attention supervised classification model, and remove the model obtained after the fake attention transformation in the trained counterfactual attention supervised classification model as an image classification prediction model.

10. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image classification prediction method as described in any one of claims 1 to 6 or the image classification prediction model training method as described in claim 7.

11. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image classification prediction method as described in any one of claims 1 to 6 or the image classification prediction model training method as described in claim 7.

Images