Model training method and device, image classification method, equipment and medium

Abstract

Embodiments of the present application provide a model training method and device, an image classification method, equipment and a medium, and relate to the technical field of artificial intelligence. The method inputs an image sample into an image classification model to obtain a feature matrix, obtains a class weight value corresponding to each feature channel according to the feature matrix based on a Shapley calculation principle, obtains a class activation map according to the class weight value of each feature channel, obtains a predicted image class corresponding to the image sample according to the class activation map, adjusts a model weight according to a loss value, until a convergence condition of a preset loss function is reached, and an image classification model is obtained. The class weight value corresponding to each feature channel is obtained based on the Shapley calculation principle, which is different from the related art which uses gradient direction propagation as the weight of the feature in the image, avoids neglecting or hiding the dependency between different elements in the feature, improves the classification accuracy of the image classification model obtained by training, and expands the application scenarios of image classification.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to image classification model training methods and apparatus, image classification methods, devices, and storage media. Background Technology

[0002] With advancements in machine learning, trained image classification models can now categorize images based on their content. The accuracy of image classification is typically related to the training level of the model. Currently, training image classification models usually involves inputting sample images into the model for training, resulting in a model capable of classifying images.

[0003] In related technologies, image classification models use gradient propagation as the weights for features in an image. This method of weight calculation ignores or hides the dependencies between different elements in the features, resulting in image classification models trained in this way being only suitable for images with low classification difficulty. Their classification accuracy is low when classifying more difficult images. Therefore, improving the accuracy of image classification has become an urgent technical problem to be solved. Summary of the Invention

[0004] The main objective of this invention is to propose an image classification model training method and apparatus, an image classification method, device, and storage medium, which improves the accuracy of image classification and expands the application scenarios of image classification based on the dependencies between different elements in the features.

[0005] To achieve the above objectives, a first aspect of the present invention proposes an image classification model training method, comprising:

[0006] Construct an image sample set, which includes multiple image samples. Each image sample includes an image and a category label. The number of categories in the category label is n, and each category label represents the image category corresponding to the image.

[0007] The image sample is input into an image classification model to obtain a feature matrix, which includes feature maps of multiple feature channels.

[0008] Based on the Shapley calculation principle, the category weight value corresponding to each feature channel is obtained according to the feature matrix;

[0009] A class activation map is obtained based on the category weight value of each feature channel;

[0010] The predicted image category corresponding to the image sample is obtained based on the class activation map;

[0011] Calculate the loss value between the predicted image category and the category label according to a preset loss function;

[0012] The model weights of the image classification model are adjusted according to the loss value until the convergence condition of the preset loss function is met, thus obtaining the image classification model.

[0013] In some embodiments, inputting the image samples into an image classification model to obtain a feature matrix includes:

[0014] The image in the image sample is preprocessed using one or more preprocessing layers to obtain preprocessed feature information;

[0015] The preprocessed feature information is sequentially extracted using one or more feature extraction layers to obtain the feature matrix.

[0016] In some embodiments, obtaining the category weight value corresponding to each feature channel based on the feature matrix according to the Shapley calculation principle includes:

[0017] Generate at least one subset of feature maps for the current feature channel based on the preset Shapley classification method;

[0018] Calculate the marginal contribution value of the feature map subset;

[0019] Calculate the weighting factor of the feature map subset;

[0020] The weight values ​​of the feature map subset are calculated based on the weighting factor and the marginal contribution value;

[0021] Calculate the category weight value of the current feature channel based on the weight values ​​of the feature map subset;

[0022] The category weight value for each feature channel is calculated one by one.

[0023] In some embodiments, generating the marginal contribution value of the feature map subset based on the feature map subset includes:

[0024] The first contribution value is calculated based on the subset of the feature map;

[0025] The second contribution value is calculated based on the feature map subset and the feature map of the current feature channel;

[0026] The marginal contribution value of the feature map subset is obtained based on the first contribution value and the second contribution value.

[0027] In some embodiments, calculating the weighting factor of the feature map subset includes:

[0028] Obtain the number of feature maps contained in the feature map subset;

[0029] The weighting factor is calculated based on the number of feature maps and the number of categories of the category labels.

[0030] In some embodiments, calculating the category weight value of the current feature channel based on the weight values ​​of the feature map subset includes:

[0031] Generate one or more feature map subsets for the current feature channel based on the preset Shapley classification method;

[0032] The feature map subset is sampled at least once based on a preset number of sampling times to obtain at least one feature map sampling subset;

[0033] Calculate the marginal contribution value of at least one of the feature map sampling subsets;

[0034] The average of the marginal contribution values ​​is calculated based on the preset number of samplings to obtain the category weight value of the current feature channel;

[0035] The category weight value for each feature channel is calculated one by one.

[0036] In some embodiments, before obtaining the class activation map based on the class weight value of each of the feature channels, the process includes:

[0037] The summation weight value is obtained by summing the category weight values ​​for each feature channel;

[0038] Calculate the average of the summed weight values ​​to obtain the average weight value;

[0039] The obtained average weight value is binarized to obtain the binarized category weight value.

[0040] To achieve the above objectives, a second aspect of this application provides an image classification method, comprising:

[0041] Acquire the target image;

[0042] The target image is input into an image classification model trained using the image classification model training method described in any of the first aspects to obtain the image category corresponding to the target image model.

[0043] To achieve the above objectives, a third aspect of this application provides an image classification model training apparatus, comprising:

[0044] A sample set construction unit is used to construct an image sample set, which includes multiple image samples. Each image sample includes an image and a category label. The number of categories in the category label is n, and the category label represents the image category corresponding to the image.

[0045] The feature matrix acquisition unit is used to input the image sample into the image classification model to obtain a feature matrix, wherein the feature matrix includes feature maps of multiple feature channels;

[0046] The category weight value calculation unit is used to obtain the category weight value corresponding to each feature channel based on the feature matrix according to the Shapley calculation principle.

[0047] A class activation map acquisition unit is used to obtain a class activation map based on the class weight value of each feature channel, wherein the class activation map is used to characterize the probability score of the class label;

[0048] An image category prediction unit is used to obtain the predicted image category corresponding to the image sample based on the class activation map;

[0049] The loss calculation unit is used to calculate the loss value between the predicted image category and the category label according to a preset loss function;

[0050] The model weight adjustment unit is used to adjust the model weights of the image classification model according to the loss value until the convergence condition of the preset loss function is reached, thereby obtaining the image classification model.

[0051] To achieve the above objectives, a fourth aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect or the method described in the second aspect.

[0052] To achieve the above objectives, a fifth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect or the method described in the second aspect.

[0053] The image classification model training method and apparatus, image classification method, device, and storage medium proposed in this invention construct an image sample set, input the image samples into the image classification model to obtain a feature matrix, obtain the class weight value corresponding to each feature channel based on the Shapley algorithm, obtain a class activation map based on the class weight value of each feature channel, obtain the predicted image class corresponding to the image sample based on the class activation map, calculate the loss value between the predicted image class and the class label according to a preset loss function, and adjust the model weights of the image classification model according to the loss value until the convergence condition of the preset loss function is reached, thus obtaining the image classification model. In this embodiment, the class weight value corresponding to each feature channel is obtained based on the Shapley algorithm, which differs from related technologies that use gradient direction propagation as the weight of features in the image. This avoids ignoring or hiding the dependencies between different elements in the features, improves the classification accuracy of the trained image classification model, and expands the application scenarios of image classification. Attached Figure Description

[0054] Figure 1 This is a flowchart of the image classification model training method provided in the embodiments of the present invention.

[0055] Figure 2 yes Figure 1 The flowchart for step S120.

[0056] Figure 3 yes Figure 1 The flowchart of step S130.

[0057] Figure 4 yes Figure 3 The flowchart for step S132 in the process.

[0058] Figure 5 yes Figure 3 The flowchart for step S133.

[0059] Figure 6 yes Figure 1 Another flowchart of step S130 in the process.

[0060] Figure 7 This is another flowchart of the image classification model training method provided in the embodiments of the present invention.

[0061] Figure 8 This is a flowchart of the image classification method provided in an embodiment of the present invention.

[0062] Figure 9 This is a structural block diagram of the image classification model training device provided in an embodiment of the present invention.

[0063] Figure 10This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0065] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0067] First, let's clarify some of the terms used in this invention:

[0068] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0069] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0070] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0071] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.

[0072] With advancements in machine learning, trained image classification models can now categorize images based on their content. The accuracy of image classification is typically related to the training level of the model. Currently, training image classification models usually involves inputting sample images into the model for training, resulting in a model capable of classifying images.

[0073] In related technologies, image classification models use gradient propagation as the weights for features in an image. This method of weight calculation ignores or hides the dependencies between different elements in the features, resulting in image classification models trained in this way being only suitable for images with low classification difficulty. Their classification accuracy is low when classifying more difficult images. Therefore, improving the accuracy of image classification has become an urgent technical problem to be solved.

[0074] Based on this, embodiments of the present invention provide an image classification model training method and apparatus, an image classification method, device, and storage medium. Based on the Shapley calculation principle, the category weight value corresponding to each feature channel is obtained. Unlike related technologies that use gradient direction propagation as the weight of features in the image, this avoids ignoring or hiding the dependencies between different elements in the features, improves the classification accuracy of the trained image classification model, and expands the application scenarios of image classification.

[0075] This invention provides an image classification model training method and apparatus, an image classification method, a device, and a storage medium, which are specifically described through the following embodiments. First, the image classification model training method in this invention is described.

[0076] This invention can acquire and process relevant data based on artificial intelligence (AI) technology. AI is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0077] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0078] The image classification model training method provided in this invention relates to the field of artificial intelligence technology, and particularly to the field of data mining technology. This method can be applied to a terminal, a server, or a computer program running on either a terminal or a server. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run, such as a client supporting image classification model training; it can also be a small program, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin. The terminal communicates with the server via a network. The image classification model training method can be executed by the terminal or the server, or by the terminal and the server working together.

[0079] In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc. Additionally, the terminal can also be a smart in-vehicle device. This smart in-vehicle device applies the image classification model training method of this embodiment to provide related services, enhancing the driving experience. The server can be a standalone server, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; it can also be a service node in a blockchain system, where the service nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). The server can install the server-side software of the image classification model training system, through which it can interact with the terminal. For example, the server can install corresponding software, which can be an application implementing the image classification model training method, but is not limited to the above forms. The terminal and the server can be connected via Bluetooth, USB (Universal Serial Bus), or network communication methods, and this embodiment does not impose any restrictions.

[0080] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0081] To facilitate understanding of the embodiments of this application, the concept of image classification will first be briefly introduced below with examples of specific application scenarios.

[0082] Image classification is an image processing method that distinguishes different categories of objects based on the different features they reflect in image information. It uses computers to quantitatively analyze images, classifying each pixel or region within an image into one of several categories, thus replacing human visual interpretation.

[0083] This application provides an image classification model training method that can be implemented based on artificial intelligence technology. The image classification model obtained by this method can be applied to scenarios involving image classification. For example, it can be used to perform biological classification based on a target image, and the classification result can be used to characterize the biological feature categories in the target image.

[0084] In one optional implementation, the image classification model provided in this embodiment can be used to classify medical images. The classification result is used to indicate the probability that the human organ represented by the medical image has developed a lesion. The following describes the application of the image classification model to classify tumors in order to determine the degree of tumor development:

[0085] In this embodiment, when training the image classification model, the server constructs an image sample set, which includes multiple tumor-related pathological image samples. Then, following the training method provided in this embodiment, the image samples are input into the image classification model to obtain a feature matrix. Based on the Shapley algorithm, the class weight value corresponding to each feature channel is obtained from the feature matrix. A class activation map is obtained based on the class weight value of each feature channel. The predicted image category corresponding to the image sample is obtained from the class activation map. The loss value between the predicted image category and the category label is calculated according to a preset loss function. The model weights of the image classification model are adjusted according to the loss value until the convergence condition of the preset loss function is met, thereby obtaining the image classification model. The terminal collects and uploads pathological images. The image classification model classifies the pathological images and predicts the image category. Based on the prediction output, it determines whether the pathological image is a normal pathological image or a tumor image, and if it is a tumor image, its stage of development.

[0086] In one optional implementation, the image classification model training method provided in this application embodiment can also be used to classify human body images for behavior. The classification result is used to indicate the probability that human behavior in a human body image belongs to a preset behavior pattern. The following describes the use of the image classification model to classify human behavior to determine which preset behavior pattern the human behavior belongs to:

[0087] In this embodiment, when training the image classification model, the server constructs an image sample set, which includes multiple human behavior image samples with different behavioral patterns. Then, following the training method provided in this embodiment, the image samples are input into the image classification model to obtain a feature matrix. Based on the Shapley algorithm, the class weight value corresponding to each feature channel is obtained from the feature matrix. A class activation map is obtained based on the class weight value of each feature channel. The predicted image category corresponding to the image sample is obtained from the class activation map. The loss value between the predicted image category and the category label is calculated according to a preset loss function. The model weights of the image classification model are adjusted according to the loss value until the convergence condition of the preset loss function is met, thereby obtaining the image classification model. The terminal collects and uploads human behavior images. The image classification model classifies the human behavior images and predicts the image category. Based on the prediction output, the human behavior pattern in the human behavior image is determined. Further operations such as business recommendation, content recommendation, or device control can be performed based on the identified behavior pattern.

[0088] It should be noted that the target images mentioned above are for biological classification only and are merely illustrative examples. The embodiments of this application are not limited to this. In some embodiments, the target images can also be other types of images, such as animal images, for animal species classification, etc.

[0089] This application does not limit the method of acquiring the target image. Exemplarily, the target image can be acquired by a computer device. Exemplarily, the target image can be sent to a computer device by an image acquisition device. Exemplarily, the target image can be extracted from an image library by a computer device. Exemplarily, the target image can be crawled from the network by a computer device, etc.

[0090] The image classification model training method in the embodiments of this application is described in detail below.

[0091] Figure 1 This is an optional flowchart of the image classification model training method provided in this embodiment of the invention. Figure 1 The method may include, but is not limited to, steps S110 to S170. It is also understood that this embodiment... Figure 1 The order of steps S110 to S170 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0092] Step S110: Construct an image sample set.

[0093] In one embodiment, the image sample set includes multiple image samples, each of which includes an image and a category label. In one embodiment, the number of category labels is n, and the category label represents the image category corresponding to the image, that is, how many categories the image is divided into according to the actual image classification requirements, with each category corresponding to one category label.

[0094] In one embodiment, if the image classification model is applied to pathological image recognition and classification, the image samples include tumor image samples and normal pathological image samples. The tumor image samples include images at different stages of development, such as grade I, grade II, and grade III lesions. These different development stages are determined based on the tumor's location, size, growth time, and extent of invasion. The corresponding number of category labels is four: normal, grade I lesion, grade II lesion, and grade III lesion.

[0095] In one embodiment, if the image classification model is applied to human behavior pattern recognition, the image samples include human behavior images with various different behavior patterns, such as reading mode, walking mode, jumping mode, or sleeping mode. The corresponding number of category labels is four: reading mode, walking mode, jumping mode, and sleeping mode.

[0096] As can be seen from the above, the embodiments of this application can set different image sample sets to train the image classification model according to actual application scenarios, so as to adapt to the usage requirements of different application scenarios. In this embodiment, after obtaining a large number of image samples for training, each image sample is input into the image classification model to perform the following steps for training.

[0097] Step S120: Input the image samples into the image classification model to obtain the feature matrix.

[0098] In one embodiment, the image classification model uses ResNet-101 as a skeleton to extract features from image samples, obtaining a feature matrix. This feature matrix includes feature maps for multiple feature channels. In one embodiment, one image category corresponds to one feature channel. For example, when extracting features from an RGB image, red (R), green (G), and blue (B) are used as three categories, and the three feature channels are designated as the red, green, and blue feature channels, respectively. The resulting feature matrix is ​​formed by combining the feature maps extracted from these three feature channels. Therefore, the feature matrix contains the feature map corresponding to each feature channel.

[0099] In one embodiment, reference is made to Figure 2 This is a flowchart illustrating a specific implementation of step S120 in one embodiment. In this embodiment, step S120, which involves inputting image samples into an image classification model to obtain a feature matrix, includes steps S121 to S122.

[0100] Step S121: Perform preprocessing operations on the image in the image sample using one or more preprocessing layers to obtain preprocessed feature information.

[0101] In one embodiment, the ResNet-101 backbone network in the image classification model consists of four preprocessing layers: two convolutional layers and two max-pooling layers. In one embodiment, the four preprocessing layers are connected in sequence. The convolutional kernels of the two convolutional layers are both 7*7, the pooling size of the two max-pooling layers is 3*3, and the computation stride of all four preprocessing layers is 2.

[0102] In one embodiment, the image classification model uses a preprocessing layer to preprocess the images in the image samples to obtain preprocessed feature information, and then performs the following steps to extract features.

[0103] Step S122: Use one or more feature extraction layers to sequentially extract features from the preprocessed feature information to obtain a feature matrix.

[0104] In one embodiment, the image classification model uses four feature extraction layers to sequentially extract features from the preprocessed feature information to obtain a feature matrix, wherein all feature extraction layers are convolutional layers.

[0105] As can be seen from the above, the embodiments of this application utilize the ResNet-101 skeleton network to extract the features of the image in the image samples to obtain the feature matrix. The ResNet-101 skeleton network contains a residual structure, which introduces an identity mapping between the input and output, which can alleviate the problem of network performance degradation as the number of layers increases and improve the accuracy of feature extraction.

[0106] Step S130: Based on the Shapley calculation principle, obtain the category weight value corresponding to each feature channel according to the feature matrix.

[0107] In one embodiment, the Shapley calculation principle is a method for distributing benefits in cooperative games. It distributes the benefits of alliance members based on Shapley values, which can reflect the degree of contribution of each member to the overall goal of the alliance. It avoids egalitarianism in distribution and is more reasonable and fair than distribution methods based solely on the value of resource input, resource allocation efficiency, or a combination of the two. It also reflects the process of mutual game among the members.

[0108] In related technologies, image classification models use gradient direction propagation as the weights of features in the image. However, image classification models trained in this way are only suitable for images with low classification difficulty, and their classification accuracy is not high when classifying images with high classification difficulty.

[0109] Therefore, in one embodiment, when calculating feature weights in an image classification model, unlike related technologies that use gradient direction propagation as the weights of features in the image, the category weight value corresponding to each feature channel is obtained based on the Shapley calculation principle. This avoids ignoring or hiding the dependencies between different elements in the features, improves the classification accuracy of the trained image classification model, and expands the application scenarios of image classification.

[0110] In one embodiment, reference is made to Figure 3 This is a flowchart illustrating a specific implementation of step S130 in one embodiment. In this embodiment, step S130, which obtains the category weight value corresponding to each feature channel based on the feature matrix according to the Shapley calculation principle, includes steps S131 to S136.

[0111] Step S131: Generate at least one subset of feature maps for the current feature channel according to the preset Shapley classification method.

[0112] In one embodiment, each feature channel corresponds to a feature map, and the number of feature channels corresponds to the number of image categories; that is, one feature channel is used to extract features related to one image category. Assuming the number of feature channels is n, this means there are n feature maps. According to the Shapley algorithm, one feature map represents one member of a consortium, and all feature maps together constitute the entire consortium. The consortium can also be described as the aforementioned feature matrix, and the feature map subset S represents a subset of some members within the consortium.

[0113] In one embodiment, a subset S of feature maps is selected from the consortium (a feature matrix composed of n feature maps) according to a preset Shapley classification method.

[0114] Assuming the current feature channel to be calculated is the first feature channel, and the corresponding feature map is N1, then the feature maps in the feature matrix other than N1 are combined. The number of feature maps in the resulting feature map subset S ranges from 1 to N-1. There are M ways to combine feature map subset S, represented as follows:

[0115]

[0116] As can be seen from the above, according to the Shapley calculation principle, a subset of feature maps is selected from the feature matrix. In principle, each subset of feature maps is related to other feature maps. Therefore, the dependencies between different elements in the features can be used for image classification, thereby improving the classification accuracy.

[0117] Step S132: Calculate the marginal contribution value of the feature map subset.

[0118] In one embodiment, the marginal contribution value is calculated based on the feature map of the current feature channel and a subset of the feature map. (Refer to...) Figure 4 This is a flowchart illustrating the specific implementation of step S132 in one embodiment. In this embodiment, step S132, which calculates the marginal contribution value of a subset of feature maps, may also include steps S1321 to S1323.

[0119] Step S1321: Calculate the first contribution value based on the feature map subset.

[0120] Step S1322: Calculate the second contribution value based on the feature map subset and the feature map of the current feature channel.

[0121] Step S1323: Obtain the marginal contribution value of the feature map subset based on the first contribution value and the second contribution value.

[0122] In one embodiment, the feature matrix is ​​represented as {N1, N2, N3, N4}, the current feature channel is the first feature channel, and the corresponding feature map is N1. The first feature channel corresponds to 7 feature map subsets S, namely {S=2}, {S=3}, {S=4}, {S=2, 3}, {S=3, 4}, {S=2, 4}, and {S=2, 3, 4}. The feature maps corresponding to the feature map subset S include: {N2}, {N3}, {N4}, {N2, N3}, {N3, N4}, {N2, N4}, and {N2, N3, N4}.

[0123] In one embodiment, in step S1321, the current feature channel is the first feature channel, and the corresponding feature map subset S is input into the image classification model to obtain the predicted probability, which is then used as the first contribution value. In one embodiment, as in the example above, the first contribution value is represented as: F1, F2, F3, F4, F5, F6, and F7.

[0124] In one embodiment, as in the example above, step S1322 combines the feature map corresponding to the feature map subset S and the feature map corresponding to the current feature channel, and inputs them into the image classification model to obtain the prediction probability, and uses the prediction probability as the second contribution value of the corresponding feature map subset S. In one embodiment, since the feature maps corresponding to the feature map subset S include: {N2}, {N3}, {N4}, {N2,N3}, {N3,N4}, {N2,N4}, {N2,N3,N4}, and the current feature channel is the first feature channel, the corresponding feature map is N1. The feature maps corresponding to the feature map subset S and the feature maps corresponding to the current feature channel are combined as: {N2,N1}, {N3,N1}, {N4,N1}, {N2,N3,N1}, {N3,N4,N1}, {N2,N4,N1}, {N2,N3,N4,N1}. The combined feature maps are input into the image classification model to obtain the predicted probability. The corresponding second contribution value F' is represented as: F1', F2', F3', F4', F5', F6', and F7'. It can be seen that the second contribution value is related to the selection of the feature map subset S.

[0125] In one embodiment, as in the example above, in step S1323, the marginal contribution value corresponding to the feature map subset S is calculated based on the obtained first contribution value (F1, F2, F3, F4, F5, F6, and F7) and second contribution value (F1', F2', F3', F4', F5', F6', and F7'). The marginal contribution value is the second contribution value minus the first contribution value; that is, for different feature map subsets S, the corresponding marginal contribution value Fs is represented as: F1'-F1, F2'-F2, F3'-F3, F4'-F4, F5'-F5, F6'-F6, and F7'-F7.

[0126] In one embodiment, the process of calculating the marginal contribution value of multiple feature map subsets S corresponding to the i-th feature channel is expressed as follows:

[0127] Fs = F′ - F = f(S∪{i}) - f(S)

[0128] Where S represents a subset of feature maps, {i} represents the feature map of the i-th feature channel, Fs represents the marginal contribution value of the feature map subset S, S∪{i} represents the combination information of the feature map corresponding to the feature map subset S and the feature map corresponding to the current feature channel, F represents the first contribution value calculated by the feature map subset, F' represents the second contribution value calculated by the feature map subset S, and f() represents the model function of the image classification model to obtain the prediction probability based on the input feature map information.

[0129] As described above, step S132 generates the marginal contribution value of each feature map subset based on the feature matrix. The marginal contribution value is related to the selection of the feature map subset S. Since the selection of the feature map subset S is related to the current feature channel, the marginal contribution value of the feature map subset S can reflect the degree of marginal contribution of the current feature channel to the result to a certain extent. After obtaining the marginal contribution value, this embodiment uses the following steps to calculate the weighting factor corresponding to the feature map subset S.

[0130] Step S133: Calculate the weighting factor of the feature map subset.

[0131] In one embodiment, reference is made to Figure 5 This is a flowchart illustrating a specific implementation of step S133 in one embodiment. In this embodiment, step S133, which calculates the weighting factor of a subset of feature maps, includes steps S1331 to S1332.

[0132] Step S1331: Obtain the number of feature maps contained in the feature map subset.

[0133] In one embodiment, the number of feature maps contained in the feature map subset S varies in different combinations. After obtaining the marginal contribution value corresponding to the current feature channel and the number of feature maps corresponding to the feature map subset S for different combinations (mainly reflected in the number of feature maps contained in the feature map subset), the weighting factor is calculated according to the following steps.

[0134] Step S1332: Calculate the weighting factor based on the number of feature maps and the number of categories of labels.

[0135] In one embodiment, the weighting factor w is calculated as follows:

[0136]

[0137] Where w represents the weighting factor of the feature map subset S, |S| represents the number of feature maps in the feature map subset, n represents the number of category labels, the number of image categories, and the number of feature channels, and "!" represents the factorial operation.

[0138] In one embodiment, the feature matrix is ​​represented as {N1, N2, N3, N4}, where the current feature channel is the first feature channel, and the corresponding feature map is N1. When the feature map subset S is {S = 2}, {S = 3}, or {S = 4}, the number of feature maps in feature map subset S is 1, and the corresponding weighting factors are all the same. When the feature map subset S is {S = 2, 3}, {S = 3, 4}, or {S = 2, 4}, the number of feature maps in feature map subset S is 2, and the corresponding weighting factors are all the same. When the feature map subset S is {S = 2, 3, 4}, the corresponding weighting factors are obtained.

[0139] By following the steps above, we can obtain the weighting factors for different combinations of feature map subset S.

[0140] Step S134: Calculate the weight values ​​of the feature map subset based on the weighting factor and marginal contribution value.

[0141] In one embodiment, multiplying the weighting factor and the marginal contribution value yields the weight value corresponding to the feature map subset S under the current feature channel. Represented as:

[0142]

[0143] in, Let w represent the weight value corresponding to the feature map subset S, w represent the weighting factor of the feature map subset S, and Fs represent the marginal contribution value of the feature map subset S.

[0144] Step S135: Calculate the category weight value of the current feature channel based on the weight values ​​of the feature map subset.

[0145] In one embodiment, since the feature map subset S includes multiple different combinations, the weighting factor of the feature map subset S under each combination is calculated in the manner described above, and then the weight value corresponding to the feature map subset S is calculated. The weight values ​​obtained by summing all combinations This will give you the class weight value of the current feature channel. Represented as:

[0146]

[0147] in, Let S represent the category weight value of the current feature channel, i represent the i-th feature channel, S represent the feature map subset, {i} represent the feature map of the i-th feature channel, S∪{i} represents the combination information of the feature map corresponding to the feature map subset S and the feature map corresponding to the current feature channel, and S∈N / {i} represents the selection of the feature map subset S from the feature matrix after removing the feature map of the i-th feature channel. Let S represent all combinations of the accumulated feature map subset S, N represent the feature map in the feature matrix, n represent the number of category labels, and also the number of image categories, f(S) represent the first contribution value calculated by the feature map subset S, and f(S∪{i}) represent the second contribution value calculated by the feature map subset S.

[0148] As can be seen from the above, since different feature map subsets S correspond to different partitioning methods, the correlation between feature maps in the obtained feature matrices is different. Therefore, the weight values ​​corresponding to the feature map subsets S under each combination method are accumulated. Obtain the total class weight value of the current feature channel. This approach leverages the dependencies between different elements in the features to improve the classification accuracy of the trained image classification model.

[0149] Step S136: Calculate the category weight value for each feature channel one by one.

[0150] In one embodiment, for each feature matrix, the category weight value of each feature channel is calculated one by one in the manner described above. In another embodiment, considering that the consensus method based on the Shapley computation principle involves a large amount of computation in dividing the feature map subset S one by one by enumeration, a sampling method is used to obtain a subset of feature maps S with some combinations from the total feature map subset S, thereby reducing the amount of computation and improving the speed of image classification and recognition.

[0151] In one embodiment, the sampling method is Monte Carlo sampling, which is an approximate inference sampling method that includes three types: direct sampling, rejection sampling, and importance sampling. Compared with random sampling, this sampling method can improve the reliability of the sampling results.

[0152] In one embodiment, reference is made to Figure 6 This is another specific implementation flowchart of step S130 shown in one embodiment. In this embodiment, step S130, which calculates the category weight value of the current feature channel based on the weight value of the feature map subset, may also include steps S137 to S1310.

[0153] Step S137: Generate one or more feature map subsets for the current feature channel according to the preset Shapley classification method.

[0154] In one embodiment, the process of generating a subset of feature maps in step S137 is the same as in step S131, and will not be described again here.

[0155] Step S138: Sample the feature map subset at least once based on a preset number of sampling times to obtain at least one feature map sampling subset.

[0156] In one embodiment, if there are many feature channels, many different feature map subsets can be generated. Performing calculations on each subset individually would result in a large computational load and low training efficiency for the image classification model. Therefore, to reduce computational load, this embodiment uses Monte Carlo sampling to sample from all feature map subsets. By continuously sampling, a gradual approximation effect is achieved, striking a balance between computational load and accuracy. In this embodiment, a preset number of sampling times is set based on computational capacity and accuracy requirements.

[0157] In one embodiment, multiple samplings can be performed to obtain multiple feature map sampling subsets. For example, the feature map subset S includes: S1, S2, S3, S4, S5, S6, S7, S8, and S9. If the preset number of samplings is 3, then sampling is performed three times, and the feature map subset S sampled each time is: S2, S4, and S7. That is, the obtained feature map sampling subset Si is S2, S4, and S7.

[0158] Step S139: Calculate the marginal contribution value of at least one feature map sampling subset.

[0159] In one embodiment, the calculation process of the marginal contribution value of the feature map sampling subset Si in step S139 is described as follows: first, the first contribution value of the feature map sampling subset Si is calculated; then, the second contribution value is calculated based on the feature map sampling subset Si and the feature map of the current feature channel; finally, the marginal contribution value of the feature map sampling subset Si is obtained based on the first contribution value and the second contribution value.

[0160] In one embodiment, a feature map sampling subset Si is input into an image classification model to obtain a predicted probability, and this predicted probability is used as a first contribution value. In one embodiment, as in the example above, the feature map sampling subset Si is S2, S4, and S7, and their first contribution values ​​are F2, F4, and F7, respectively.

[0161] In one embodiment, the feature map corresponding to the above-mentioned feature map sampling subset Si and the feature map corresponding to the current feature channel are combined and input into the image classification model to obtain the prediction probability. The prediction probability is used as the second contribution value of the corresponding feature map sampling subset Si. In one embodiment, the combined feature map is input into the image classification model to obtain the prediction probability, and the corresponding second contribution values ​​are represented as F2', F4', and F7'. It can be seen that the second contribution value is related to the selection of the feature map sampling subset Si.

[0162] In one embodiment, as in the example above, the marginal contribution value corresponding to the feature map sampling subset Si is calculated based on the obtained first contribution value (F2, F4, and F7) and second contribution value (F2', F4', and F7'). The marginal contribution value is the second contribution value minus the first contribution value; that is, for different feature map sampling subsets Si, the corresponding marginal contribution value Fsi is represented as: F2'-F2, F4'-F4, and F7'-F7.

[0163] In one embodiment, the process of calculating the marginal contribution value of multiple feature map sampling subsets Si corresponding to the i-th feature channel is expressed as follows:

[0164] Fsi=f(Si∪{i})-f(Si)

[0165] Where Fsi represents the marginal contribution value of the feature map sampling subset Si, Si represents the feature map sampling subset, {i} represents the feature map of the i-th feature channel, Fsi represents the marginal contribution value of the feature map sampling subset Si, Si∪{i} represents the combination information of the feature map corresponding to the feature map sampling subset Si and the feature map corresponding to the current feature channel, f(Si) represents the first contribution value calculated by the feature map sampling subset Si, f(Si∪{i}) represents the second contribution value calculated by the feature map sampling subset Si, and f() represents the model function of the image classification model to obtain the prediction probability based on the input feature map information.

[0166] As shown above, the marginal contribution value is related to the feature map sampling subset Si obtained from the sampling. Since the selection of the feature map sampling subset Si is related to the current feature channel, the marginal contribution value of the feature map sampling subset Si can, to some extent, reflect the degree of marginal contribution of the current feature channel to the result.

[0167] Step S1310: Calculate the average of the marginal contribution values ​​based on the preset number of samplings to obtain the category weight value of the current feature channel.

[0168] Step S1311: Calculate the category weight value for each feature channel one by one.

[0169] In one embodiment, after calculating the marginal contribution value Fsi corresponding to each feature map sampling subset Si, the marginal contribution values ​​Fsi obtained from all feature map sampling subsets Si are summed to obtain the class weight value of the current feature channel. Represented as:

[0170]

[0171] in, Let represent the category weight value of the current feature channel, i represent the i-th feature channel, Si represent the feature map sampling subset, {i} represent the feature map of the i-th feature channel, Si∪{i} represent the combination information of the feature map corresponding to the feature map sampling subset Si and the feature map corresponding to the current feature channel, f(Si∪{i}) represents the second contribution value calculated by the feature map sampling subset Si, f() represents the model function of the image classification model to obtain the prediction probability based on the input feature map information, T represents the preset number of samplings, and t represents the t-th sampling.

[0172] As can be seen from the above, since different feature map sampling subsets Si correspond to different partitioning methods, the correlation between feature maps in the obtained feature matrices is different. Therefore, the marginal contribution values ​​corresponding to the feature map sampling subsets Si under each combination method are summed and averaged to obtain the total category weight value of the current feature channel. This approach leverages the dependencies between different elements in the features to improve the classification accuracy of the trained image classification model.

[0173] In one embodiment, in order to reduce the non-essential information contained in the feature channel category weight values, step S1312 is performed after step S1311 or step S136 to process the category weight values ​​using a binarization operation.

[0174] Step S1312: The category weight values ​​are processed using a binarization operation.

[0175] In one embodiment, reference is made to Figure 7 This is a flowchart illustrating a specific implementation of step S1312 in one embodiment. In this embodiment, step S1312, which uses binarization to process the category weight values, includes steps S1313 to S1315.

[0176] Step S1313: Sum the category weight values ​​for each feature channel to obtain the summed weight values.

[0177] Step S1314: Calculate the average of the summed weight values ​​to obtain the average weight value.

[0178] Step S1315: Binarize the obtained average weight value to obtain the binarized category weight value.

[0179] The above steps describe the binarization process, represented as:

[0180]

[0181]

[0182]

[0183] Where i represents the current feature channel, and n represents the number of feature channels, which also represents the number of category labels. This represents the category weight value corresponding to the current feature channel. The summation weights are represented by , and the average weights are represented by . This represents the binary category weight value. ∏() represents the indicator function. If the value inside the function is true, the output is 1; otherwise, the output is 0.

[0184] In one embodiment, the binarized class weight values ​​are more robust than continuous values ​​between [0, 1], ensuring that they do not contain too much unimportant information and improving the accuracy of image classification.

[0185] After obtaining the category weight value corresponding to each feature channel, the class activation map is calculated using the following steps.

[0186] Step S140: Obtain the class activation map based on the class weight value of each feature channel.

[0187] In one embodiment, the class activation graph is also called channel attention. Its main idea is to obtain an attention heatmap by weighted summation of different weights for each feature channel. The class activation graph performs two operations at the same time: one is to inject the relevance information of the category label into the extracted feature matrix, and the other is to map the image features contained in the original feature matrix to a new feature space. In this embodiment, the class activation graph can be used to represent the probability score of the category label.

[0188] In one embodiment, the image classification model is a convolutional neural network, and the class activation map is represented as follows:

[0189]

[0190]

[0191] Among them, M c (x,y) represents the value of the feature at position (x,y) with respect to the category label c after feature channel weighting, where k represents the k-th feature channel. This represents the class weight value of the k-th feature channel. f represents the class weight value of the k-th feature channel after the binarization process. k (x,y) represents the value of the feature matrix at position (x,y) in feature channel k.

[0192] In one embodiment, the image classification model is a graph neural network model, and the class activation graph is represented as follows:

[0193]

[0194]

[0195] Where L represents the class activation map, X represents the input image, and A represents the adjacency matrix. This represents the value of the k-th column in the l-th layer of the feature matrix, where k represents the k-th feature channel. This represents the class weight value of the k-th feature channel. This represents the class weight value of the k-th feature channel after the binarization process.

[0196] It is understood that this embodiment does not restrict whether the feature channel category weight values ​​need to be binarized, nor does it restrict the model structure of the image classification model.

[0197] As described above, in image classification tasks, both convolutional networks and graph neural networks extract features from input samples as the basis for classification. The class activation map in this embodiment visualizes the weight information of feature channels, characterizing whether the underlying criteria for successful image classification by the image classification model are reasonable. For example, an image classification model that identifies whether an image contains a boat cannot determine whether the decision is based on the boat or the water. The class activation map provides a clear visual representation of the semantic information related to the feature channels learned by the image classification model. In one embodiment, multiple feature channels in the image classification model represent different learned feature information. The class activation map, based on Shapley's computational principle, characterizes the marginal contribution of each feature channel in the input image and can serve as the basis for subsequent classification decisions.

[0198] After obtaining the class activation map using the above steps, the class activation map is used for image classification prediction.

[0199] Step S150: Obtain the predicted image category corresponding to the image sample based on the class activation map.

[0200] In one embodiment, since the acquired class activation map is a weighted feature matrix, where each channel represents the feature map corresponding to the class label, the weighted features need to be mapped back to the original feature space. In one embodiment, the class activation map is input into a fully connected layer, and the output of the fully connected layer is used as the classification prediction, that is, the label probability corresponding to the image sample. Then, based on the label probability and a preset probability threshold, the image sample is determined to belong to which image category, thus obtaining the predicted image category. In one embodiment, the fully connected layer is a 1*1 convolutional layer.

[0201] Step S160: Calculate the loss value between the predicted image category and the category label according to the preset loss function.

[0202] In one embodiment, the preset loss function is the cross-entropy loss function, which calculates the cross-entropy between the probability corresponding to the predicted image category and the probability corresponding to the category label as the loss value.

[0203] Step S170: Adjust the model weights of the image classification model according to the loss value until the convergence condition of the preset loss function is met, and obtain the image classification model.

[0204] In one embodiment, the convergence condition of the preset loss function can be minimizing cross-entropy, or other convergence conditions, such as reaching a set maximum number of iterations. This embodiment does not specifically limit this. The weights of the image classification model are adjusted in each iteration until training is complete, resulting in the image classification model.

[0205] This embodiment of the disclosure constructs an image sample set, inputs the image samples into an image classification model to obtain a feature matrix, obtains the category weight value corresponding to each feature channel based on the Shapley calculation principle, obtains a class activation map based on the category weight value of each feature channel, obtains the predicted image category corresponding to the image sample based on the class activation map, calculates the loss value between the predicted image category and the category label based on a preset loss function, and adjusts the model weights of the image classification model based on the loss value until the convergence condition of the preset loss function is reached, thus obtaining the image classification model.

[0206] In one embodiment, the category weight value corresponding to each feature channel is obtained based on the Shapley calculation principle. Unlike related technologies that use gradient direction propagation as the weight of features in the image, this avoids ignoring or hiding the dependencies between different elements in the features, improves the classification accuracy of the trained image classification model, and expands the application scenarios of image classification.

[0207] This invention also provides an image classification method, referring to... Figure 8 The method includes, but is not limited to, steps S810 to S820:

[0208] Step S810: Obtain the target image.

[0209] In one embodiment, the method of acquiring the target image is not limited. Exemplarily, the target image may be acquired by a computer device. Exemplarily, the target image may be sent to a computer device by an image acquisition device. Exemplarily, the target image may be extracted from an image library by a computer device. Exemplarily, the target image may be crawled from the network by a computer device, etc.

[0210] Step S820: Input the target image into the image classification model to obtain the image category.

[0211] In one embodiment, the target image is input into an image classification model trained using the image classification model training method described above, to obtain the image category corresponding to the target image model, and the image is classified according to the image category. For example, biological classification is performed based on the target image, and the classification result is used to characterize the biological feature categories in the target image. Biological classification includes: medical image classification and human behavior recognition classification, where biological features refer to: organ lesion probability and human behavior pattern, respectively. It should be noted that the target image described above for biological classification is merely an exemplary example, and the embodiments of this application are not limited thereto. In some embodiments, the target image can also be other types of images, such as animal images, for animal species classification, etc.

[0212] The specific implementation method of classifying the target image by the image classification model in this embodiment is basically the same as the specific implementation method of the image classification model training method described above, and will not be repeated here.

[0213] This invention also provides an image classification model training device that can implement the above-described image classification model training method, referring to... Figure 9 The device includes:

[0214] The sample set construction unit 910 is used to construct an image sample set, which includes multiple image samples. Each image sample includes an image and a category label. The number of categories is n, and each category label represents the image category corresponding to the image.

[0215] The feature matrix acquisition unit 920 is used to input image samples into the image classification model to obtain a feature matrix, which includes feature maps of multiple feature channels.

[0216] The category weight value calculation unit 930 is used to obtain the category weight value corresponding to each feature channel based on the feature matrix according to the Shapley calculation principle.

[0217] The class activation map acquisition unit 940 is used to obtain a class activation map based on the class weight value of each feature channel. The class activation map is used to represent the probability score of the class label.

[0218] The image category prediction unit 950 is used to obtain the predicted image category corresponding to the image sample based on the class activation map.

[0219] The loss value calculation unit 960 is used to calculate the loss value between the predicted image category and the category label according to a preset loss function.

[0220] The model weight adjustment unit 970 is used to adjust the model weights of the image classification model according to the loss value until the convergence condition of the preset loss function is reached, thus obtaining the image classification model.

[0221] The specific implementation of the image classification model training device in this embodiment is basically the same as the specific implementation of the image classification model training method described above, and will not be repeated here.

[0222] This invention also provides an electronic device, comprising:

[0223] The device includes at least one memory; at least one processor; and at least one program. The program is stored in the memory, and the processor executes the at least one program to implement the image classification model training method or image classification method described above. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0224] Please see Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0225] The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0226] The memory 1002 can be implemented in the form of ROM (Read-Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 1002 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 to execute the image classification model training method or image classification method of the embodiments of this invention.

[0227] Input / output interface 1003 is used to implement information input and output;

[0228] The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and the bus 1005 is used to transmit information between the various components of the device (such as processor 1001, memory 1002, input / output interface 1003 and communication interface 1004).

[0229] The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0230] This application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described image classification model training method or image classification method.

[0231] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0232] The image classification model training method, image classification method, image classification model training device, electronic device, and storage medium proposed in this invention construct an image sample set, input the image samples into the image classification model to obtain a feature matrix, obtain the class weight value corresponding to each feature channel based on the Shapley algorithm, obtain a class activation map based on the class weight value of each feature channel, obtain the predicted image class corresponding to the image sample based on the class activation map, calculate the loss value between the predicted image class and the class label according to a preset loss function, and adjust the model weights of the image classification model according to the loss value until the convergence condition of the preset loss function is met, thus obtaining the image classification model. This embodiment obtains the class weight value corresponding to each feature channel based on the Shapley algorithm, unlike related technologies that use gradient direction propagation as the weight of features in the image. This avoids ignoring or hiding the dependencies between different elements in the features, improves the classification accuracy of the trained image classification model, and expands the application scenarios of image classification.

[0233] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0234] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0235] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0236] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0237] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0238] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0239] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0240] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0241] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0242] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0243] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for training an image classification model, characterized in that, include: Construct an image sample set, which includes multiple image samples. Each image sample includes an image and a category label. The number of categories in the category label is n, and each category label represents the image category corresponding to the image. The image sample is input into an image classification model to obtain a feature matrix, which includes feature maps of multiple feature channels. Based on the Shapley calculation principle, the category weight value corresponding to each feature channel is obtained according to the feature matrix; A class activation map is obtained based on the category weight value of each feature channel; The predicted image category corresponding to the image sample is obtained based on the class activation map; Calculate the loss value between the predicted image category and the category label according to a preset loss function; The model weights of the image classification model are adjusted according to the loss value until the convergence condition of the preset loss function is met, thereby obtaining the image classification model. The step of obtaining the category weight value corresponding to each feature channel based on the feature matrix according to the Shapley calculation principle includes: generating at least one feature map subset of the current feature channel according to a preset Shapley classification method; calculating the marginal contribution value of the feature map subset; calculating the weighting factor of the feature map subset; calculating the weight value of the feature map subset according to the weighting factor and the marginal contribution value; calculating the category weight value of the current feature channel according to the weight value of the feature map subset; and calculating the category weight value of each feature channel one by one.

2. The image classification model training method according to claim 1, characterized in that, The step of inputting the image samples into the image classification model to obtain the feature matrix includes: The image in the image sample is preprocessed using one or more preprocessing layers to obtain preprocessed feature information; The preprocessed feature information is sequentially extracted using one or more feature extraction layers to obtain the feature matrix.

3. The image classification model training method according to claim 1, characterized in that, The step of generating the marginal contribution value of the feature map subset based on the feature map subset includes: The first contribution value is calculated based on the subset of the feature map; The second contribution value is calculated based on the feature map subset and the feature map of the current feature channel; The marginal contribution value of the feature map subset is obtained based on the first contribution value and the second contribution value.

4. The image classification model training method according to claim 1, characterized in that, The calculation of the weighting factor for the feature map subset includes: Obtain the number of feature maps contained in the feature map subset; The weighting factor is calculated based on the number of feature maps and the number of categories of the category labels.

5. The image classification model training method according to claim 1, characterized in that, The step of calculating the category weight value of the current feature channel based on the weight values ​​of the feature map subset includes: Generate one or more feature map subsets for the current feature channel based on the preset Shapley classification method; The feature map subset is sampled at least once based on a preset number of sampling times to obtain at least one feature map sampling subset; Calculate the marginal contribution value of at least one of the feature map sampling subsets; The average of the marginal contribution values ​​is calculated based on the preset number of samplings to obtain the category weight value of the current feature channel; The category weight value for each feature channel is calculated one by one.

6. The image classification model training method according to any one of claims 1 to 5, characterized in that, Before obtaining the class activation map based on the category weight value of each feature channel, the process includes: The summation weight value is obtained by summing the category weight values ​​for each feature channel; Calculate the average of the summed weight values ​​to obtain the average weight value; The obtained average weight value is binarized to obtain the binarized category weight value.

7. An image classification method, characterized in that, include: Acquire the target image; The target image is input into the image classification model trained using the image classification model training method according to any one of claims 1 to 6 to obtain the image category corresponding to the target image model.

8. An image classification model training device, characterized in that, include: A sample set construction unit is used to construct an image sample set, which includes multiple image samples. Each image sample includes an image and a category label. The number of categories in the category label is n, and the category label represents the image category corresponding to the image. The feature matrix acquisition unit is used to input the image sample into the image classification model to obtain a feature matrix, wherein the feature matrix includes feature maps of multiple feature channels; The category weight value calculation unit is used to obtain the category weight value corresponding to each feature channel based on the feature matrix according to the Shapley calculation principle. A class activation map acquisition unit is used to obtain a class activation map based on the class weight value of each feature channel, wherein the class activation map is used to characterize the probability score of the class label; An image category prediction unit is used to obtain the predicted image category corresponding to the image sample based on the class activation map; The loss calculation unit is used to calculate the loss value between the predicted image category and the category label according to a preset loss function; The model weight adjustment unit is used to adjust the model weights of the image classification model according to the loss value until the convergence condition of the preset loss function is reached, thereby obtaining the image classification model. The step of obtaining the category weight value corresponding to each feature channel based on the feature matrix according to the Shapley calculation principle includes: generating at least one feature map subset of the current feature channel according to a preset Shapley classification method; calculating the marginal contribution value of the feature map subset; calculating the weighting factor of the feature map subset; calculating the weight value of the feature map subset according to the weighting factor and the marginal contribution value; calculating the category weight value of the current feature channel according to the weight value of the feature map subset; and calculating the category weight value of each feature channel one by one.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the image classification model training method according to any one of claims 1 to 6, or the image classification method according to claim 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the image classification model training method according to any one of claims 1 to 6, or the image classification method according to claim 7.

Images