Image classification method, training method of image classification model, and electronic device

Abstract

The present disclosure relates to an image classification method, a training method of an image classification model and an electronic device, and relates to the technical field of computers. The method of the present disclosure comprises: dividing an image into a plurality of image blocks, and inputting an image classification model; determining a correlation degree value between each image block based on a self-attention mechanism in the image classification model; determining a redundant image block corresponding to each image block from the plurality of image blocks according to the correlation degree value between each image block; updating the correlation degree value between each image block, and encoding the image according to the updated correlation degree value between each image block, wherein for each image block, the correlation degree value generated by the redundant image block corresponding to the image block to the image block is reduced to a preset value; and determining the type of the image according to the encoding of the image.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to an image classification method, a training method of an image classification model and an electronic device. BACKGROUND

[0002] With the continuous development of the field of computer vision, and inspired by the model structure design of natural language processing, many visual tasks use structures such as self-attention mechanisms to explore and optimize. For example, a model with a self-attention mechanism is used for feature extraction of an image in an image classification task.

[0003] The inventors have found that in the current image classification task, an image is first divided into mutually non-overlapping visual image blocks, and the embedding sequence of each image block is input into an image classification model with a self-attention mechanism; secondly, the self-attention mechanism is used to construct the attention of the region between the image blocks, which is used for image feature extraction. SUMMARY

[0004] The inventors have found that in the existing scheme, the correlation between each image block and all image blocks is calculated based on the self-attention mechanism, resulting in a redundant attention calculation cost, and the extracted image features contain redundant information, thereby reducing the accuracy of image classification.

[0005] One technical problem to be solved by the present disclosure is how to improve the accuracy of image classification.

[0006] According to some embodiments of the present disclosure, an image classification method is provided, including: dividing an image into a plurality of image blocks, and inputting the image into an image classification model; determining a correlation degree value between each image block based on a self-attention mechanism in the image classification model; determining a redundant image block corresponding to each image block from the plurality of image blocks according to the correlation degree value between each image block; updating the correlation degree value between each image block, and encoding the image according to the updated correlation degree value between each image block, wherein for each image block, the correlation degree value generated by the redundant image block corresponding to the image block to the image block is reduced to a preset value; and determining a type of the image according to the encoding of the image.

[0007] In some embodiments, the image classification model comprises a plurality of self-attention layers, the relevance degree values between the image blocks comprise relevance degree values between the image blocks in each self-attention layer, and determining the redundant image block corresponding to each image block from the plurality of image blocks according to the relevance degree values between the image blocks comprises: for each self-attention layer, grouping the relevance degree values between the image blocks corresponding to all self-attention heads in the self-attention layer into a first relevance tensor; determining a first uncertainty tensor according to the discrete degree of the first relevance tensor corresponding to each self-attention layer; and determining the redundant image block corresponding to each image block according to the first uncertainty tensor.

[0008] In some embodiments, each self-attention layer comprises a plurality of self-attention heads, the relevance degree values between the image blocks in each self-attention layer comprise relevance degree values between the image blocks corresponding to each self-attention head in the self-attention layer, and determining the redundant image block corresponding to each image block according to the first uncertainty tensor comprises: extracting the uncertainty values of the relevance degree between the image blocks corresponding to each self-attention head from the first uncertainty tensor respectively to obtain a second uncertainty tensor corresponding to each self-attention head; determining a first influence degree tensor according to the discrete degree of each second uncertainty tensor, wherein the first influence degree tensor comprises influence degree values between the image blocks during encoding; and determining the redundant image block corresponding to each image block according to the first influence degree tensor.

[0009] In some embodiments, determining the redundant image block corresponding to each image block according to the first influence degree tensor comprises at least one of the following: for each image block, selecting the redundant image block corresponding to the image block from the first influence degree tensor according to the influence degree value of the encoding of the image block on the image block; and for each image block, determining whether the image block is the redundant image block corresponding to each image block according to the discrete degree of the influence degree values of the encoding of the image block on each image block.

[0010] In some embodiments, determining the first uncertainty tensor according to the discrete degree of the first relevance tensor corresponding to each self-attention layer comprises: determining a first average relevance tensor according to the first relevance tensor corresponding to each self-attention layer; squaring and adding the difference between the first relevance tensor corresponding to each self-attention layer and the first average relevance tensor, and then dividing the added result by the number of layers of the self-attention layer to obtain the first uncertainty tensor.

[0011] In some embodiments, the determining the first influence degree tensor according to the discrete degrees of the respective second uncertainty tensors comprises: determining an average uncertainty tensor according to the respective second uncertainty tensors; squaring and summing the differences between the respective second uncertainty tensors and the average uncertainty tensor, and dividing the sum by the number of the self-attention heads to obtain the first influence degree tensor.

[0012] In some embodiments, each row element in the first influence degree tensor corresponds to an image block, and the selecting, for each image block, the redundant image block corresponding to the image block from the first influence degree tensor according to the influence degree values of the respective image blocks on the encoding of the image block comprises: for each row element in the first influence degree tensor, sorting the row element in descending order; selecting a first preset number of elements according to the sorting order, and determining the image blocks corresponding to the selected elements as the effective image blocks; and determining the image blocks other than the effective image blocks as the redundant image blocks corresponding to the image blocks corresponding to the row element.

[0013] In some embodiments, each row element in the first influence degree tensor corresponds to an image block, and the selecting, for each image block, the redundant image block corresponding to the image block from the first influence degree tensor according to the influence degree values of the respective image blocks on the encoding of the image block comprises: for each row element in the first influence degree tensor, sorting the row element in descending order; selecting a first preset number of elements according to the sorting order, and determining the image blocks corresponding to the selected elements; determining a minimum rectangular region containing the image blocks corresponding to the selected elements with the image block corresponding to the row element as the center; and determining the image blocks other than the image blocks in the minimum rectangular region as the redundant image blocks corresponding to the image blocks corresponding to the row element.

[0014] In some embodiments, each column element in the first influence degree tensor corresponds to an image block, and the determining, for each image block, whether the image block is the redundant image block corresponding to the respective image blocks according to the discrete degrees of the influence degree values of the image block on the encoding of the respective image blocks comprises: for each column element in the first influence degree tensor, determining the variance of the column element as the discrete degree of the influence degree values of the image block corresponding to the column element on the encoding of the respective image blocks; sorting the variances of the column elements in ascending order, selecting a second preset number of variances according to the sorting order, and determining the image blocks corresponding to the selected variances as the redundant image blocks corresponding to the respective image blocks.

[0015] In some embodiments, the image classification model comprises a plurality of self-attention layers, each self-attention layer comprises a plurality of self-attention heads, and updating the relevance degree values between the image blocks comprises at least one of: generating a binary matrix, multiplying the first relevance matrix composed of the relevance degree values between the image blocks in each self-attention head by the binary matrix to obtain an updated matrix, wherein for each image block, the binary matrix is set such that in the updated matrix, for each image block, the relevance degree value generated by the redundant image block corresponding to the image block to the image block is reduced to a preset value; and for each image block, in the case that the image block is a redundant image block that generates redundant information for the encoding of each image block, setting an element of the image block in an input tensor of the image classification model to a preset element to reduce the relevance degree value generated by the image block to other image blocks to a preset value.

[0016] In some embodiments, the image classification model comprises a plurality of self-attention layers, each self-attention layer comprises a plurality of self-attention heads, and determining the relevance degree values between the image blocks based on the self-attention mechanism in the image classification model comprises: in each self-attention head, determining a query matrix and a key matrix according to the feature matrix of the image input into the self-attention head; multiplying the query matrix by the key matrix and dividing the result by the square root of the number of columns of the key matrix to obtain an attention score matrix; and normalizing the attention score matrix to obtain a first relevance matrix composed of the relevance degree values between the image blocks.

[0017] In some embodiments, the image classification model comprises a plurality of encoding layers, each encoding layer comprises a self-attention layer, each self-attention layer comprises a plurality of self-attention heads, and encoding the image according to the updated relevance degree values between the image blocks comprises: in each encoding layer, inputting the feature matrix of the image input from the previous layer into each self-attention head of the self-attention layer; in each self-attention head, determining a value matrix according to the feature matrix of the image input into the self-attention head; multiplying the updated matrix corresponding to the self-attention head by the value matrix to obtain an attention encoding matrix corresponding to the self-attention head, wherein the updated matrix is composed of the updated relevance degree values between the image blocks; in the encoding layer, determining the feature matrix of the image output from the encoding layer according to the attention encoding matrix corresponding to each self-attention head; and taking the feature matrix of the image output from the last encoding layer as the encoding matrix of the image.

[0018] According to another embodiment of the present disclosure, a training method of an image classification model is provided, including: dividing a sample image into a plurality of sample image blocks, and inputting the image classification model for pre-training; determining a correlation degree value between each sample image block based on a self-attention mechanism when the pre-training is completed; determining a redundant sample image block corresponding to each sample image block from the plurality of sample image blocks according to the correlation degree value between each sample image block; re-training the pre-trained image classification model using the sample image, updating the correlation degree value between each sample image block in the training process, and encoding the sample image according to the updated correlation degree value between each sample image block, wherein for each sample image block, the correlation degree value generated by the redundant sample image block corresponding to the sample image block for the sample image block is reduced to a preset value, and the type of the sample image is output according to the encoding of the sample image, and the parameters of the image classification model are adjusted according to the output type of the sample image and the labeled type.

[0019] In some embodiments, the image classification model includes a plurality of self-attention layers, the correlation degree value between each sample image block includes a correlation degree value between each sample image block in each self-attention layer, and the redundant sample image block corresponding to each sample image block is determined from the plurality of sample image blocks according to the correlation degree value between each sample image block, including: for each self-attention layer, the correlation degree values between each sample image block corresponding to all self-attention heads in the self-attention layer are grouped into a second correlation tensor; determining a third uncertainty tensor according to the discrete degree of the second correlation tensor corresponding to each self-attention layer; and determining the redundant sample image block corresponding to each sample image block according to the third uncertainty tensor.

[0020] In some embodiments, each self-attention layer includes a plurality of self-attention heads, the correlation degree value between each sample image block in each self-attention layer includes a correlation degree value between each sample image block corresponding to each self-attention head in the self-attention layer, and the redundant sample image block corresponding to each sample image block is determined according to the third uncertainty tensor, including: extracting an uncertainty value of the correlation degree between each sample image block corresponding to each self-attention head from the third uncertainty tensor to obtain a fourth uncertainty tensor corresponding to each self-attention head; determining a second influence degree tensor according to the discrete degree of each fourth uncertainty tensor, wherein the second influence degree tensor includes an influence degree value between each sample image block when encoding; and determining the redundant sample image block corresponding to each sample image block according to the second influence degree tensor.

[0021] In some embodiments, determining, according to the second influence degree tensor, the redundant sample image block corresponding to each sample image block comprises at least one of the following: for each sample image block, selecting, from the second influence degree tensor, the redundant sample image block corresponding to the sample image block according to the influence degree value of the encoding of each sample image block on the sample image block; and for each sample image block, determining, from the second influence degree tensor, whether the sample image block is the redundant sample image block corresponding to each sample image block according to the dispersion degree of the influence degree value of the encoding of the sample image block on each sample image block.

[0022] In some embodiments, each row element in the second influence degree tensor corresponds to a sample image block, and selecting, for each sample image block, the redundant sample image block corresponding to the sample image block from the second influence degree tensor according to the influence degree value of the encoding of each sample image block on the sample image block comprises: for each row element in the second influence degree tensor, sorting the row element in descending order; selecting a third preset number of elements according to the sorting order; determining the sample image block corresponding to the selected element as an effective sample image block, or determining a minimum rectangular region containing the sample image block corresponding to the selected element with the sample image block corresponding to the row element as the center, and taking the sample image block in the minimum rectangular region as an effective sample image block; and taking the sample image block other than the effective sample image block as the redundant sample image block corresponding to the sample image block corresponding to the row element.

[0023] In some embodiments, each column element in the second influence degree tensor corresponds to a sample image block, and determining, for each sample image block, whether the sample image block is the redundant sample image block corresponding to each sample image block according to the dispersion degree of the influence degree value of the encoding of the sample image block on each sample image block from the second influence degree tensor comprises: for each column element in the second influence degree tensor, determining the variance of the column element as the dispersion degree of the influence degree value of the encoding of the sample image block corresponding to the column element on each sample image block; sorting the variances of the column elements in ascending order, selecting a fourth preset number of variances according to the sorting order, and determining the sample image block corresponding to the selected variance as the redundant sample image block corresponding to each sample image block.

[0024] In some embodiments, the image classification model comprises a plurality of self-attention layers, each self-attention layer comprising a plurality of self-attention heads, updating the correlation degree values between the respective sample image blocks comprises at least one of: generating a binary matrix, multiplying the second correlation matrix composed of the correlation degree values between the respective sample image blocks in each self-attention head by the binary matrix to obtain an updated matrix, wherein for each sample image block, the binary matrix is set such that in the updated matrix, for each sample image block, the correlation degree value generated by the redundant sample image block corresponding to the sample image block for the sample image block is reduced to a preset value; for each sample image block, in the case that the sample image block is a redundant sample image block that generates redundant information for the encoding of the respective sample image blocks, the element of the sample image block in the input tensor of the image classification model is set to a preset element, so that the correlation degree value generated by the sample image block for other sample image blocks is reduced to a preset value.

[0025] According to still some embodiments of the present disclosure, an image classification device is provided, comprising: an input module configured to divide an image into a plurality of image blocks and input an image classification model; a correlation degree value determination module configured to determine correlation degree values between the respective image blocks based on a self-attention mechanism in the image classification model; a redundant image block determination module configured to determine, according to the correlation degree values between the respective image blocks, redundant image blocks corresponding to the respective image blocks from the plurality of image blocks; an encoding module configured to update the correlation degree values between the respective image blocks and encode the image according to the updated correlation degree values between the respective image blocks, wherein for each image block, the correlation degree value generated by the redundant image block corresponding to the image block for the image block is reduced to a preset value; and a classification module configured to determine a type of the image according to the encoding of the image.

[0026] According to still another embodiment of the present disclosure, a training device of an image classification model is provided, comprising: an input module configured to divide a sample image into a plurality of sample image blocks, and input the image classification model for pre-training; a correlation degree value determination module configured to, when the pre-training is completed, determine a correlation degree value between each sample image block based on a self-attention mechanism; a redundant image block determination module configured to determine, according to the correlation degree value between each sample image block, a redundant sample image block corresponding to each sample image block from the plurality of sample image blocks; and a re-training module configured to re-train the pre-trained image classification model using the sample image, update the correlation degree value between each sample image block during the training process, and encode the sample image according to the updated correlation degree value between each sample image block, wherein, for each sample image block, the correlation degree value generated by the redundant sample image block corresponding to the sample image block for the sample image block is reduced to a preset value, the type of the sample image is output according to the encoding of the sample image, and the parameters of the image classification model are adjusted according to the output type of the sample image and the labeled type.

[0027] According to still another embodiment of the present disclosure, a training device of an image classification model is provided, comprising: an input module configured to divide a sample image into a plurality of sample image blocks, and input the image classification model for pre-training; a correlation degree value determination module configured to, when the pre-training is completed, determine a correlation degree value between each sample image block based on a self-attention mechanism; a redundant image block determination module configured to determine, according to the correlation degree value between each sample image block, a redundant sample image block corresponding to each sample image block from the plurality of sample image blocks; and a re-training module configured to re-train the pre-trained image classification model using the sample image, update the correlation degree value between each sample image block during the training process, and encode the sample image according to the updated correlation degree value between each sample image block, wherein, for each sample image block, the correlation degree value generated by the redundant sample image block corresponding to the sample image block for the sample image block is reduced to a preset value, the type of the sample image is output according to the encoding of the sample image, and the parameters of the image classification model are adjusted according to the output type of the sample image and the labeled type.

[0028] According to still another embodiment of the present disclosure, a training device of an image classification model is provided, comprising: an input module configured to divide a sample image into a plurality of sample image blocks, and input the image classification model for pre-training; a correlation degree value determination module configured to, when the pre-training is completed, determine a correlation degree value between each sample image block based on a self-attention mechanism; a redundant image block determination module configured to determine, according to the correlation degree value between each sample image block, a redundant sample image block corresponding to each sample image block from the plurality of sample image blocks; and a re-training module configured to re-train the pre-trained image classification model using the sample image, update the correlation degree value between each sample image block during the training process, and encode the sample image according to the updated correlation degree value between each sample image block, wherein, for each sample image block, the correlation degree value generated by the redundant sample image block corresponding to the sample image block for the sample image block is reduced to a preset value, the type of the sample image is output according to the encoding of the sample image, and the parameters of the image classification model are adjusted according to the output type of the sample image and the labeled type.

[0029] According to still another embodiment of the present disclosure, a training device of an image classification model is provided, comprising: an input module configured to divide a sample image into a plurality of sample image blocks, and input the image classification model for pre-training; a correlation degree value determination module configured to, when the pre-training is completed, determine a correlation degree value between each sample image block based on a self-attention mechanism; a redundant image block determination module configured to determine, according to the correlation degree value between each sample image block, a redundant sample image block corresponding to each sample image block from the plurality of sample image blocks; and a re-training module configured to re-train the pre-trained image classification model using the sample image, update the correlation degree value between each sample image block during the training process, and encode the sample image according to the updated correlation degree value between each sample image block, wherein, for each sample image block, the correlation degree value generated by the redundant sample image block corresponding to the sample image block for the sample image block is reduced to a preset value, the type of the sample image is output according to the encoding of the sample image, and the parameters of the image classification model are adjusted according to the output type of the sample image and the labeled type.

[0030] Other features and advantages of the present disclosure will be apparent from the following detailed description of the exemplary embodiments of the present disclosure, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those of ordinary skill in the art without creative effort based on these drawings.

[0032] Figure 1 A flowchart of an image classification method of some embodiments of the present disclosure is shown.

[0033] Figure 2 A structural diagram of an image classification model of some embodiments of the present disclosure is shown.

[0034] Figure 3 A flowchart of a training method of an image classification model of some embodiments of the present disclosure is shown.

[0035] Figure 4 A structural diagram of an image classification device of some embodiments of the present disclosure is shown.

[0036] Figure 5 A structural diagram of a training device of an image classification model of some embodiments of the present disclosure is shown.

[0037] Figure 6 A structural diagram of an electronic device of some embodiments of the present disclosure is shown.

[0038] Figure 7 A structural diagram of an electronic device of some other embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only some of the embodiments of the present disclosure, but not all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative, but not as any limitation on the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present disclosure.

[0040] The present disclosure proposes an image classification method, which will be described below in conjunction with Figures 1-2 .

[0041] Figure 1Flowcharts illustrating some embodiments of the image classification method of this disclosure. For example... Figure 1 As shown, the method of this embodiment includes steps S102 to S110.

[0042] In step S102, the image is divided into multiple image blocks and input into the image classification model.

[0043] An image can be divided into multiple image patches of the same size. For example... Figure 2 As shown, an image classification model may include: an input layer, one or more encoding layers, a classification layer, and an output layer. Each encoding layer may include a self-attention layer, and each self-attention layer may include one or more attention heads. Each encoding layer may also include a feedforward neural network or a multilayer perceptron (MLP), etc. The image classification model can employ existing models, such as the Vision Transformer (ViT), and is not limited to the examples given.

[0044] In the input layer, embedding features and positional codes for each image patch are generated. Based on the embedding features and positional codes of each image patch, an image representation matrix is ​​generated and input into the encoding layer.

[0045] In step S104, the correlation value between each image patch is determined based on the self-attention mechanism in the image classification model.

[0046] In some embodiments, the image classification model includes multiple self-attention layers, each self-attention layer including multiple self-attention heads. In each self-attention head, a query matrix and a key matrix are determined based on the feature matrix of the image input to the self-attention head. The query matrix and the key matrix are multiplied and then divided by the square root of the number of columns in the key matrix to obtain an attention score matrix. The attention score matrix is ​​normalized to obtain a first association matrix composed of the association degree values ​​between each image patch.

[0047] The first coding layer takes the representation matrix of the image output from the input layer as input. For each subsequent coding layer, the feature matrix (or coding matrix) of the image output from the previous coding layer is taken as input. For example, for the i-th coding layer, the feature matrix of the input image is represented as X. i The parameter matrix of the j-th self-attention head in the i-th coding layer is represented as follows: X i respectively with Multiplying yields the query matrix Q i,j Key matrix K i,j Value matrix V i,jThe first association matrix of the jth self-attention head in the ith encoding layer can be expressed by the following formula.

[0048]

[0049] In formula (1), d is the number of key matrix columns, which can also be set to other values and is not limited to the examples shown. Softmax() is a normalization function. Assuming that the image is divided into N image blocks, then A i,j contains N x N elements, each element representing the degree of association between image blocks, which can also be referred to as a self-attention weight value. For example, the element in the first row and the second column represents the degree of association of image block 2 to image block 1, that is, the degree of contribution of image block 2 to the encoding of image block 1, which can also be referred to as the self-attention weight of image block 2 to image block 1.

[0050] In step S106, according to the degree of association between each image block, the corresponding redundant image block of each image block is determined from the plurality of image blocks.

[0051] The interaction of image blocks is a relatively key part of the application of image classification models based on self-attention mechanisms in the visual field. The self-attention mechanism maps image information into Query (query matrix), Key (key matrix) and Value (value matrix), which is used to find the internal association of the input data, and further improves the self-attention layer by combining the mechanism of multi-head structure, that is, multiple self-attention mechanisms, to expand the ability of the model to focus on different positions. The reliability of the internal association of the input image blocks can be analyzed, that is, the interaction relationship between each image block and all image blocks.

[0052] After quantifying the analysis of the interaction relationship between image blocks, the corresponding redundant image block of each image block can be further determined. Specifically, after analyzing the image block interaction relationship based on uncertainty estimation, the analyzed data can be used as a quantification of the relationship between different image blocks. Thus, the interaction between image blocks has the ability to be explained, and a new relationship construction method is provided for exploring the interaction relationship between image blocks in the self-attention module calculation of the image classification network.

[0053] At present, the architecture of the image classification model based on the self-attention mechanism (for example, ViT) learns the global context information by using the full self-attention mechanism between all image blocks, ignoring the reliability (effectiveness) of the image block interaction. The scheme of the present disclosure analyzes the difference degree of self-attention weights at different levels and multi-head, and analyzes the reliability of image block interaction from the aspects of structure and feature, which helps to better understand the interaction and association relationship between image blocks, and further guides the attention mechanism design of the image classification model.

[0054] AsFigure 2 As shown, the disclosure designs an uncertainty perception model to measure the effectiveness of image block interaction. In the calculation of multi-layer multi-head self-attention, image blocks are calculated by information propagation with all image blocks, that is, an image block can interact densely with other image blocks that are highly related to it, thereby representing self-attention. However, such global redundant calculation leads to unstable attention scores and high uncertainty of associated image block interaction. Therefore, through the uncertainty perception model, redundant image blocks that lead to unstable attention scores and high uncertainty of associated image block interaction can be screened out.

[0055] In some embodiments, for each self-attention layer, a first association tensor is composed of the association degree values between the respective image blocks corresponding to all self-attention heads in the self-attention layer; a first uncertainty tensor is determined according to the dispersion degree of the first association tensor corresponding to each self-attention layer; and redundant image blocks corresponding to each image block are determined according to the first uncertainty tensor. The first uncertainty tensor can be regarded as a score describing the uncertainty of interaction between image blocks.

[0056] Further, in some embodiments, a first average association tensor is determined according to the first association tensor corresponding to each self-attention layer; the difference between the first association tensor corresponding to each self-attention layer and the first average association tensor is squared and added, and then the added result is divided by the number of layers of the self-attention layer, as the first uncertainty tensor.

[0057] For example, the first association matrix of the jth self-attention head in the ith encoding layer is represented as A i,j , and the first association tensor of the ith self-attention layer is represented as k is the number of self-attention heads. The first uncertainty tensor is represented by the following formula.

[0058]

[0059] In formula (2), l represents the number of layers of the self-attention layer, is the first average association tensor.

[0060] It should be noted that in the case of only one self-attention head in each self-attention layer, u is an N*N matrix, which describes the uncertainty of the association (interaction) between the image blocks. The n-th row of u represents the uncertainty of the contribution of each image block to the encoding of the n-th image block. The higher the uncertainty of the contribution of the m-th image block to the encoding of the n-th image block, the higher the reference value of the information injected by the m-th image block in the encoding of the n-th image block, and the lower the probability that the m-th image block is a redundant image block. Therefore, for example, the elements of each row of u are sorted in ascending order; the third preset number of elements are selected according to the sorting order, and the image blocks corresponding to the selected elements are determined as the redundant image blocks corresponding to the image blocks corresponding to the elements of the row; or, for each row element of u, the row elements are sorted in descending order; the fourth preset number of elements are selected according to the sorting order, and the image blocks corresponding to the selected elements are determined as the effective image blocks corresponding to the image blocks corresponding to the elements of the row, and the smallest rectangular region (interaction region) containing the effective image blocks is determined; the other image blocks in the smallest rectangular region are determined as the redundant image blocks corresponding to the image blocks corresponding to the elements of the row.

[0061] The n-th column of u represents the uncertainty of the contribution of the n-th image block to the encoding of each image block. The lower the dispersion of these uncertainties, the less differentiated the information injected by the n-th image in the encoding of the image blocks, and the higher the probability that the n-th image block is a redundant image block. Therefore, for example, for each column element in u, the variance of the column element is determined, the variances of the column elements are sorted in ascending order, the fifth preset number of variances are selected according to the sorting order, and the image blocks corresponding to the selected variances are determined as the redundant image blocks corresponding to each image block.

[0062] In the case where each self-attention layer includes multiple self-attention heads, the differences between the self-attention heads can be further analyzed. In some embodiments, the uncertainty values of the association degrees between each image block corresponding to each self-attention head are extracted from the first uncertainty tensor respectively, obtaining a second uncertainty tensor corresponding to each self-attention head; according to the dispersion of each second uncertainty tensor, a first influence degree tensor is determined, wherein the first influence degree tensor includes the influence degree values between each image block during encoding; according to the first influence degree tensor, the redundant image blocks corresponding to each image block are determined.

[0063] Further, in some embodiments, according to each second uncertainty tensor, an average uncertainty tensor is determined; the difference between each second uncertainty tensor and the average uncertainty tensor is squared and added, and then the sum is divided by the number of self-attention heads to obtain the first influence degree tensor.

[0064] In the case where each self-attention layer includes k self-attention heads, A second uncertainty tensor corresponding to each self-attention head is extracted from U, for example, the second uncertainty tensor corresponding to the jth self-attention head is denoted as u j , u j is an N x N matrix (tensor). The first influence degree tensor can be expressed by the following formula.

[0065]

[0066] In formula (3), denotes the average uncertainty tensor.

[0067] It should be noted that in the case where there is only one self-attention layer including multiple self-attention heads, formula (2) does not need to be calculated, and u j is transformed into a first association matrix composed of the association degree values between each image block corresponding to the jth self-attention head, and is an N x N matrix. The method for determining the redundant image block according to U can refer to the method for determining the redundant image block according to u in the case where each self-attention layer has only one self-attention head in the foregoing embodiments.

[0068] In some embodiments, determining the redundant image block that generates redundant information for the encoding of each image block according to the first influence degree tensor includes at least one of the following: for each image block, selecting the redundant image block corresponding to the image block from the first influence degree tensor according to the influence degree value of each image block on the encoding of the image block; and for each image block, determining whether the image block is the redundant image block corresponding to each image block according to the dispersion degree of the influence degree value of the image block on the encoding of each image block.

[0069] The nth row of U represents the influence degree value of each image block on the encoding of the nth image block. For the nth image block, the lower the influence degree value of the image block m on the encoding of the nth image block, the lower the reference value of the information injected by the image block m in the encoding of the nth image block, and the higher the probability that the image block m is a redundant image block. Therefore, in some embodiments, for each row element in the first influence degree tensor, the row element is sorted in descending order; the first preset number of elements are selected according to the sorting order, and the image blocks corresponding to the selected elements are determined as valid image blocks; and the image blocks other than the valid image blocks are determined as the redundant image blocks corresponding to the image blocks corresponding to the row element.

[0070] In some embodiments, to facilitate the processing of the image, an interaction region is divided for each image block, and for each row element in the first influence degree tensor, the row element is sorted in descending order; a first preset number of elements are selected according to the sorting order, and the image block corresponding to the selected element is determined; a minimum rectangular region (interaction region) containing the image block corresponding to the selected element is determined with the image block corresponding to the row element as the center; and other image blocks outside the minimum rectangular region are determined as the redundant image blocks corresponding to the image block corresponding to the row element. The image blocks in the interaction region are the image blocks that have an important influence on the image block corresponding to the row element.

[0071] For example, for an image block, the extreme values in the four directions of the coordinate axis with the image block as the center are calculated according to the two-dimensional coordinates of the image block and the effective image blocks, so as to form an interaction region.

[0072] The method of the above embodiments can find other image blocks with relatively reliable interaction relationship of the current image block by quantifying the interaction relationship between the image blocks based on the uncertainty estimation.

[0073] The nth list in U represents the influence degree value of the nth image block on the encoding of each image block. The lower the dispersion degree of these influence degree values, the smaller the differentiation of the information injected by the nth image block when encoding the image blocks, and the higher the probability that the nth image block is a redundant image block caused by such non-differentiated injection. Therefore, in some embodiments, for each column element in the first influence degree tensor, the variance of the column element is determined as the dispersion degree of the influence degree value of the image block corresponding to the column element on the encoding of each image block; the variances of the column elements are sorted in ascending order, a second preset number of variances are selected according to the sorting order, and the image block corresponding to the selected variance is determined as the redundant image block corresponding to each image block.

[0074] In the above embodiments, based on the quantification result, the non-discriminative interaction relationship in the image block interaction process is mined. Specifically, some high-value constant image blocks exist in U by the uncertainty perception model, these image blocks usually inject a large amount of information to any other image block without differentiation, such information is not representative for the feature representation of the image, and can be considered as a biased estimation (bias) for the feature extraction of the image, and these image blocks belong to non-discriminative image blocks, i.e. redundant image blocks.

[0075] The above two embodiments for determining the redundant image blocks, i.e. the division of the image block interaction region and the mining of the non-discriminative image blocks, can be applied separately or in combination.

[0076] In step S108, the correlation degree values between the image blocks are updated, and the image is encoded according to the updated correlation degree values between the image blocks. For each image block, the correlation degree value generated by the redundant image block corresponding to the image block to the image block is reduced to a preset value.

[0077] The preset value can be set to a very low value to reduce the impact on image encoding, for example, the preset value is set to 0. The correlation degree values in the first correlation tensor corresponding to each self-attention layer can be directly modified. However, since the image classification model can include multiple encoding layers, the results output by the previous encoding layer (self-attention layer) will affect the encoding of the subsequent encoding layer (self-attention layer), therefore, the image blocks can be re-input into the image classification model, or the representation matrix of the image output by the input layer is re-input into the first encoding layer, and the correlation degree values between the image blocks are updated at each encoding layer.

[0078] The foregoing embodiments describe two embodiments for determining redundant image blocks, namely the division of the interaction area of the image blocks and the mining of the image blocks without discriminativeness. Different methods can be used to update the correlation degree values between the image blocks for the two different embodiments.

[0079] For example, for the embodiment corresponding to the division of the interaction area of the image blocks, a binary matrix is generated, and the first correlation matrix composed of the correlation degree values between the image blocks in each self-attention head is multiplied by the binary matrix to obtain an updated matrix. For each image block, the binary matrix is set such that in the updated matrix, the correlation degree value generated by the redundant image block corresponding to the image block to the image block is reduced to a preset value.

[0080] The representation matrix of the image output by the input layer is re-input into the first encoding layer, and the first correlation matrix of each self-attention head is still determined by the method of formula (1), for example, the first correlation matrix of the jth self-attention head in the ith encoding layer is represented as A i,j , and the binary matrix is represented as W. For example, in the binary matrix, the elements corresponding to the multiplication positions of the correlation degree values between the image blocks that do not belong to the redundant image blocks are set to 1, and the elements of other positions are set to 0 or a certain fixed value, so that in the updated matrix, the correlation degree value generated by the redundant image block corresponding to the image block to the image block is reduced to a preset value. Q i,j ·K i,j T Multiplying by W has the same effect.

[0081] In the embodiment for mining non-discriminative image patches, since these redundant image patches are those that generate redundant information for all image patches, these redundant image patches can be processed from the input layer when each image patch is re-inputted into the image classification model, thereby updating the first correlation matrix corresponding to each self-attention head in the subsequent self-attention layers.

[0082] For example, for each image patch, if the image patch is a redundant image patch that generates redundant information for encoding other image patches, the elements of that image patch in the input tensor of the image classification model are set to preset elements, so that the correlation value of that image patch with other image patches is reduced to a preset value. For example, the elements of that image patch in the input tensor of the image classification model are set to 0.

[0083] In some embodiments, in each encoding layer, the feature matrix of the image input from the previous layer is input into each self-attention head of the self-attention layer; in each self-attention head, a value matrix is ​​determined based on the feature matrix of the image input to that self-attention head; the updated matrix corresponding to that self-attention head is multiplied by the value matrix to obtain the attention encoding matrix corresponding to that self-attention head, wherein the updated matrix is ​​composed of the updated correlation values ​​between each image patch; in that encoding layer, the feature matrix of the output image of that encoding layer is determined based on the attention encoding matrix corresponding to each self-attention head; the feature matrix of the image output by the last encoding layer is used as the encoding matrix of the image.

[0084] For example, in each coding layer, the attention coding matrices corresponding to each self-attention head are concatenated, multiplied with the parameter matrix corresponding to that coding layer, and then input into a feedforward neural network or MLP to obtain the feature matrix output by that coding layer, which is then input into the next coding layer.

[0085] In step S110, the type of the image is determined based on the image's encoding.

[0086] For example, the encoding matrix of an image is input into a classification layer to obtain the type of the output image.

[0087] like Figure 2 As shown, when two types of redundant image blocks are determined based on the two embodiments described above, the uncertainty perception model returns the result of the determined redundant image blocks to the input layer, re-inputs each image block into the image classification model, and returns the result of the division of the interactive region to each self-attention layer, such as returning the binary matrix to the self-attention layer, thereby realizing the classification of the image.

[0088] In the above embodiment, the image is divided into multiple image blocks, input into the image classification model, the correlation degree value between each image block is determined based on the self-attention mechanism in the image classification model, and the corresponding redundant image block of each image block is determined from the multiple image blocks according to the correlation degree value between each image block. These redundant image blocks are the image blocks that generate redundant information for the encoding of the image blocks. Further, the correlation degree value between the redundant image blocks and the corresponding image blocks is reduced to a preset value, and the image is encoded according to the updated correlation degree value between each image block, so as to determine the type of the image based on the encoding of the image. Reducing the correlation degree value between the redundant image blocks and the corresponding image blocks can reduce the addition of redundant information in the process of image block encoding, and pay more attention to the calculation of the self-attention mechanism between the effective image blocks, thereby improving the accuracy of image feature extraction, improving the accuracy of image classification, and improving the perception ability of the model.

[0089] In the above embodiment, the explainability of the image classification model based on the self-attention mechanism is explored. In order to improve the explainability of the model, the uncertainty introduced in the image block interaction process is analyzed, and the interaction relationship between the current image block and other image blocks is quantified. The quantification index directly reflects the influence degree of the self-attention operation between different image blocks on the final result of the model.

[0090] Based on the quantification analysis result, the design of image classification is inversely pointed out. Specifically, based on the above quantification analysis result, according to the importance of the self-attention operation between different image blocks, all image blocks with high importance corresponding to the current image block are divided, the interaction area of the current image block is divided, the interaction range of the image block is further constrained, and redundant calculation is reduced. And the image blocks without discriminability are mined, and the redundant information generated by the image blocks without discriminability is reduced as much as possible. These two technologies greatly reduce the influence of redundant information on image encoding in the self-attention operation process, thereby greatly improving the performance of the image classification model, and have great practical application potential.

[0091] The present disclosure also proposes a training method of an image classification model, which will be described below in combination with Figure 3 .

[0092] Figure 3 The flowchart of some embodiments of the training method of the image classification model of the present disclosure is shown in FIG. 3. As shown in FIG. 3, the method of this embodiment includes steps S302-S308. Figure 3

[0093] In step S302, the sample image is divided into multiple sample image blocks, and input into the image classification model for pre-training.

[0094] ​The training samples include a plurality of sample images, and the image classification model can be pre-trained using the training samples. The structure of the image classification model, and the processing of the input layer can refer to the related embodiments of step S102.

[0095] In step S304, when the pre-training is completed, the correlation degree values between the sample image blocks are determined based on the self-attention mechanism.

[0096] When the pre-training is completed, the parameters of each part of the image classification model are determined, and at this time, the correlation degree values between the sample image blocks are determined based on the self-attention mechanism. In some embodiments, the image classification model includes a plurality of self-attention layers, each self-attention layer includes a plurality of self-attention heads, in each self-attention head, the query matrix and the key matrix of the sample image are determined according to the feature matrix of the sample image input into the self-attention head; the query matrix of the sample image is multiplied by the key matrix and then divided by the square root of the number of key matrix columns to obtain the attention score matrix of the sample image; the attention score matrix of the sample image is normalized to obtain a second correlation matrix composed of the correlation degree values between the sample image blocks.

[0097] The determination method of the second correlation matrix can refer to the related embodiments of step S104 and formula (1), and will not be repeated here.

[0098] In step S306, according to the correlation degree values between the sample image blocks, the redundant sample image blocks corresponding to each sample image block are determined from the plurality of sample image blocks.

[0099] In some embodiments, the correlation degree values between the sample image blocks include the correlation degree values between the sample image blocks in each self-attention layer, and for each self-attention layer, the correlation degree values between the sample image blocks corresponding to all self-attention heads in the self-attention layer are composed into a second correlation tensor; according to the dispersion degree of the second correlation tensor corresponding to each self-attention layer, a third uncertainty tensor is determined; according to the third uncertainty tensor, the redundant sample image blocks corresponding to each sample image block are determined.

[0100] Further, in some embodiments, according to the second correlation tensor corresponding to each self-attention layer, a second average correlation tensor is determined; the difference between the second correlation tensor corresponding to each self-attention layer and the second average correlation tensor is squared and summed, and then the sum result is divided by the number of layers of the self-attention layer as the third uncertainty tensor. The third uncertainty tensor can be determined by referring to formula (2) and the related embodiments of the first uncertainty tensor described above.

[0101] In some embodiments, the correlation degree values between the respective sample image blocks in each self-attention layer include correlation degree values between the respective sample image blocks corresponding to each self-attention head in the self-attention layer, and uncertainty values of the correlation degrees between the respective sample image blocks corresponding to each self-attention head are extracted from the third uncertainty tensor respectively to obtain fourth uncertainty tensors corresponding to each self-attention head; and a second influence degree tensor is determined according to the dispersion degrees of the respective fourth uncertainty tensors, where the second influence degree tensor includes influence degree values between the respective sample image blocks during encoding.

[0102] Further, in some embodiments, an average uncertainty tensor is determined according to the respective fourth uncertainty tensors, and the second influence degree tensor is obtained by squaring and adding the differences between the respective fourth uncertainty tensors and the average uncertainty tensor, and then dividing the sum by the number of self-attention heads. The second influence degree tensor can be determined according to Formula (3) and the related embodiments of the first influence degree tensor.

[0103] In some embodiments, determining the redundant sample image blocks corresponding to the respective sample image blocks according to the second influence degree tensor includes at least one of the following: for each sample image block, selecting a redundant sample image block corresponding to the sample image block from the second influence degree tensor according to the influence degree value of the sample image block on the encoding of the sample image block; and for each sample image block, determining whether the sample image block is a redundant sample image block corresponding to the respective sample image blocks according to the dispersion degree of the influence degree values of the sample image block on the encoding of the respective sample image blocks.

[0104] In some embodiments, for each row element in the second influence degree tensor, the row element is sorted in descending order, third preset number of elements are selected according to the sorting order, a sample image block corresponding to the selected element is determined as an effective sample image block, or a minimum rectangular region containing the sample image block corresponding to the selected element is determined with the sample image block corresponding to the row element as the center, and the sample image blocks in the minimum rectangular region are determined as effective sample image blocks, and the other sample image blocks other than the effective sample image blocks are determined as redundant sample image blocks corresponding to the sample image block corresponding to the row element.

[0105] In some embodiments, for each column element in the second influence degree tensor, a variance of the column element is determined as the dispersion degree of the influence degree values of the sample image block corresponding to the column element on the encoding of the respective sample image blocks, the variances of the column elements are sorted in ascending order, fourth preset number of variances are selected according to the sorting order, and the sample image blocks corresponding to the selected variances are determined as the redundant sample image blocks corresponding to the respective sample image blocks.

[0106] Similar to the method of determining the redundant image block in the foregoing embodiments, the interactive region can also be divided and the non-discriminative sample image block can be determined for the sample image block, which can be applied separately or in combination. For details, reference can be made to the foregoing embodiments, which will not be described here again.

[0107] In step S308, the pre-trained image classification model is retrained using the sample image. In the training process, the correlation degree values between the sample image blocks are updated, and the sample image is encoded according to the updated correlation degree values between the sample image blocks. For each sample image block, the correlation degree value generated by the redundant sample image block corresponding to the sample image block to the sample image block is reduced to a preset value. According to the encoding of the sample image, the type of the sample image is output. According to the output type of the sample image and the labeled type, the parameters of the image classification model are adjusted.

[0108] As shown in FIG. 8, the uncertainty perception model returns the result of the determined redundant sample image block to the input layer and each self-attention layer, re-enters the sample image into the image classification model, and performs training. Figure 2

[0109] In some embodiments, a binary matrix is generated. In the retraining process, the second correlation matrix composed of the correlation degree values between the sample image blocks in each self-attention head is multiplied by the binary matrix to obtain an updated matrix. For each sample image block, the binary matrix is set such that in the updated matrix, the correlation degree value generated by the redundant sample image block corresponding to the sample image block to the sample image block is reduced to a preset value. For each sample image block, in the case that the sample image block is a redundant sample image block that generates redundant information for the encoding of each sample image block, the element of the sample image block in the input tensor of the image classification model is set to a preset element, so that the correlation degree value generated by the sample image block to other sample image blocks is reduced to a preset value. The preset element and the preset value are, for example, 0. The binary matrix can be a 0, 1 matrix.

[0110] ​In some embodiments, in each encoding layer, the feature matrix of the image input by the previous layer is input into each self-attention head of the self-attention layer; in each self-attention head, a value matrix is determined according to the feature matrix of the sample image input into the self-attention head; the updated matrix corresponding to the self-attention head is multiplied by the value matrix to obtain the attention encoding matrix corresponding to the self-attention head, wherein the updated matrix is composed of the correlation degree values between the updated image blocks; in the encoding layer, the feature matrix of the image output by the encoding layer is determined according to the attention encoding matrix corresponding to each self-attention head; and the feature matrix of the sample image output by the last encoding layer is taken as the encoding matrix of the sample image.

[0111] The encoding matrix of the sample image is input into the classification layer to obtain the type of the output sample image. According to the type of the output sample image and the labeled type, a loss function is calculated, and the parameters of the image classification model are adjusted according to the loss function. In the process of retraining, the addition of redundant information is reduced, so that the parameters of the model are further accurately adjusted, and the accuracy of the model is improved. The specific correlation degree value updating and encoding process can refer to the foregoing embodiments, which will not be described here.

[0112] In the above embodiments, the pre-training is performed first, and then the training is performed again after the redundant sample image blocks are determined. Alternatively, another training method can be used to replace the above method. For example, in each training cycle (epoch), the data in the current cycle is processed according to the information fed back by the uncertainty perception model in the previous cycle, and the redundant sample image blocks in the current cycle are determined and fed back to the next cycle. Specifically, in each training cycle, the correlation degree values between the sample image blocks in the current cycle are determined based on the self-attention mechanism and the redundant sample image blocks determined in the previous cycle, the redundant sample image blocks corresponding to the sample image blocks in the next cycle are determined according to the correlation degree values between the sample image blocks in the current cycle, the sample image is encoded according to the correlation degree values between the sample image blocks in the current cycle, the type of the sample image is output according to the encoding of the sample image, the parameters of the image classification model are adjusted according to the type of the output sample image and the labeled type, and the above process is repeated until the training is completed. This training method can accelerate the convergence speed of the model.

[0113] Through extensive experimentation, the inventors discovered that performance verification on the ImageNet dataset for image classification tasks effectively improved image classification accuracy while significantly reducing the computational cost per self-attention head. Further analysis of the interaction region segmentation results for each image patch revealed that both the interaction direction trend and the size of the interaction region are related to the target in the image. In other words, response field analysis of the interaction region can explore the effective interaction regions of image patches in image classification models.

[0114] This disclosure also proposes an image classification device, which is described below in conjunction with... Figure 4 Describe it.

[0115] Figure 4 These are structural diagrams of some embodiments of the image classification apparatus of this disclosure. For example... Figure 4 As shown, the device 40 in this embodiment includes: an input module 410, an association degree value determination module 420, a redundant image block determination module 430, an encoding module 440, and a classification module 450.

[0116] The input module 410 is used to divide the image into multiple image blocks and input the image classification model.

[0117] The correlation degree value determination module 420 is used to determine the correlation degree value between each image patch based on the self-attention mechanism in the image classification model.

[0118] In some embodiments, the correlation degree value determination module 420 is used to determine a query matrix and a key matrix in each self-attention head based on the feature matrix of the image input to the self-attention head; multiply the query matrix and the key matrix and divide by the square root of the number of columns in the key matrix to obtain an attention score matrix; normalize the attention score matrix to obtain a first correlation matrix composed of the correlation degree values ​​between each image block.

[0119] The redundant image block determination module 430 is used to determine the redundant image block corresponding to each image block from multiple image blocks based on the correlation degree value between each image block.

[0120] In some embodiments, the image classification model includes multiple self-attention layers, and the correlation degree value between each image patch includes the correlation degree value between each image patch in each self-attention layer. The redundant image patch determination module 430 is used to, for each self-attention layer, form a first correlation tensor by the correlation degree values ​​between each image patch corresponding to all self-attention heads in the self-attention layer; determine a first uncertainty tensor according to the discreteness of the first correlation tensor corresponding to each self-attention layer; and determine the redundant image patch corresponding to each image patch according to the first uncertainty tensor.

[0121] In some embodiments, each self-attention layer includes a plurality of self-attention heads, the correlation degree values between the respective image blocks in each self-attention layer include correlation degree values between the respective image blocks corresponding to each self-attention head in the self-attention layer, and the redundancy image block determination module 430 is configured to extract, from the first uncertainty tensor, an uncertainty value of the correlation degree between the respective image blocks corresponding to each self-attention head, respectively, to obtain a second uncertainty tensor corresponding to each self-attention head; determine a first influence degree tensor according to the discrete degrees of the respective second uncertainty tensors, wherein the first influence degree tensor includes influence degree values between the respective image blocks during encoding; and determine the redundancy image block corresponding to each image block according to the first influence degree tensor.

[0122] In some embodiments, the redundancy image block determination module 430 performs at least one of the following: for each image block, selecting a redundancy image block corresponding to the image block from the first influence degree tensor according to an influence degree value of the image block on the encoding of the image block; and for each image block, determining whether the image block is a redundancy image block corresponding to the respective image blocks according to the discrete degree of the influence degree values of the image block on the encoding of the respective image blocks.

[0123] In some embodiments, the redundancy image block determination module 430 is configured to determine a first average correlation tensor according to the first correlation tensors corresponding to the respective self-attention layers; square and add the differences between the first correlation tensors corresponding to the respective self-attention layers and the first average correlation tensor, and divide the added result by the number of layers of the self-attention layers to obtain the first uncertainty tensor.

[0124] In some embodiments, the redundancy image block determination module 430 is configured to determine an average uncertainty tensor according to the respective second uncertainty tensors; square and add the differences between the respective second uncertainty tensors and the average uncertainty tensor, and divide the added result by the number of self-attention heads to obtain the first influence degree tensor.

[0125] In some embodiments, each row element in the first influence degree tensor corresponds to an image block, and the redundancy image block determination module 430 is configured to, for each row element in the first influence degree tensor, sort the row element in descending order; select a first preset number of elements according to the sorting order, and determine the image blocks corresponding to the selected elements as effective image blocks; and determine the image blocks other than the effective image blocks as redundancy image blocks corresponding to the image blocks corresponding to the row element.

[0126] In some embodiments, the redundant image block determination module 430 is configured to, for each row element in the first influence degree tensor, sort the row element in descending order; select a first preset number of elements in the sorted order, and determine the image blocks corresponding to the selected elements; determine a minimum rectangular region containing the image blocks corresponding to the selected elements, with the image blocks corresponding to the row element as the center; and determine other image blocks in the minimum rectangular region as the redundant image blocks corresponding to the image blocks corresponding to the row element.

[0127] In some embodiments, the redundant image block determination module 430 is configured to, for each column element in the first influence degree tensor, determine the variance of the column element as the degree of dispersion of the influence degree values of the encoding of the image blocks corresponding to the column element; sort the variances of the column elements in ascending order, select a second preset number of variances in the sorted order, and determine the image blocks corresponding to the selected variances as the redundant image blocks corresponding to each image block.

[0128] The encoding module 440 is configured to update the association degree values between the image blocks, and encode the image according to the updated association degree values between the image blocks, wherein, for each image block, the association degree value generated by the redundant image block corresponding to the image block to the image block is reduced to a preset value.

[0129] In some embodiments, the encoding module 440 is configured to perform at least one of the following: generate a binary matrix, multiply the first association matrix composed of the association degree values between the image blocks in each self-attention head by the binary matrix to obtain an updated matrix, wherein, for each image block, the binary matrix is set such that, in the updated matrix, the association degree value generated by the redundant image block corresponding to the image block to the image block is reduced to a preset value; and for each image block, if the image block is a redundant image block that generates redundant information for the encoding of each image block, set the element of the image block in the input tensor of the image classification model to a preset element, so that the association degree value generated by the image block to other image blocks is reduced to a preset value.

[0130] In some embodiments, the encoding module 440 is configured to, in each encoding layer, input the feature matrix of the image input by the previous layer into each self-attention head of the self-attention layer; in each self-attention head, determine a value matrix according to the feature matrix of the image input into the self-attention head; multiply the updated matrix corresponding to the self-attention head by the value matrix to obtain an attention encoding matrix corresponding to the self-attention head, wherein the updated matrix is composed of the updated association degree values between the image blocks.

[0131] In this coding layer, the feature matrix of the output image is determined based on the attention coding matrix corresponding to each self-attention head; the feature matrix of the image output by the last coding layer is used as the coding matrix of the image.

[0132] The classification module 450 is used to determine the type of an image based on its encoding.

[0133] This disclosure also provides a training device for an image classification model, which is described below in conjunction with... Figure 5 Describe it.

[0134] Figure 5 These are structural diagrams of some embodiments of the training apparatus for the image classification model of this disclosure. For example... Figure 5 As shown, the device 50 in this embodiment includes: an input module 510, a correlation degree value determination module 520, a redundant image block determination module 530, and a retraining module 540.

[0135] The input module 510 is used to divide the sample image into multiple sample image blocks and input the image classification model for pre-training.

[0136] The correlation degree value determination module 520 is used to determine the correlation degree value between each sample image patch based on the self-attention mechanism when pre-training is completed.

[0137] The redundant image block determination module 530 is used to determine the redundant sample image block corresponding to each sample image block from multiple sample image blocks based on the correlation degree value between each sample image block.

[0138] In some embodiments, the image classification model includes multiple self-attention layers, and the correlation degree value between each sample image patch includes the correlation degree value between each sample image patch in each self-attention layer. The redundant image patch determination module 530 is used to, for each self-attention layer, form a second correlation tensor by the correlation degree values ​​between each sample image patch corresponding to all self-attention heads in the self-attention layer; determine a third uncertainty tensor according to the discreteness of the second correlation tensor corresponding to each self-attention layer; and determine the redundant sample image patch corresponding to each sample image patch according to the third uncertainty tensor.

[0139] In some embodiments, the redundant image block determination module 530 is used to extract the uncertainty value of the correlation degree between each sample image block corresponding to each self-attention head from the third uncertainty tensor to obtain a fourth uncertainty tensor corresponding to each self-attention head; determine a second influence degree tensor according to the discreteness of each fourth uncertainty tensor, wherein the second influence degree tensor includes the influence degree value between each sample image block during encoding; and determine the redundant sample image blocks corresponding to each sample image block according to the second influence degree tensor.

[0140] In some embodiments, the redundant image block determination module 530 is configured to perform at least one of the following: for each sample image block, selecting, from the second influence degree tensor, a redundant sample image block corresponding to each sample image block according to an influence degree value of each sample image block on the encoding of the sample image block; and for each sample image block, determining whether the sample image block is a redundant sample image block of each sample image block according to a dispersion degree of the influence degree value of the sample image block on the encoding of each sample image block.

[0141] In some embodiments, each row element in the second influence degree tensor corresponds to a sample image block, and the redundant image block determination module 530 is configured to, for each row element in the second influence degree tensor, sort the row element in descending order, select a third preset number of elements according to the sorting order, determine a sample image block corresponding to the selected element as an effective sample image block, or determine a minimum rectangular region containing the sample image block corresponding to the selected element with the sample image block corresponding to the row element as the center, and determine the sample image blocks in the minimum rectangular region as the effective sample image blocks, and determine the sample image blocks other than the effective sample image blocks as redundant sample image blocks of the sample image block corresponding to the row element.

[0142] In some embodiments, the redundant image block determination module 530 is configured to, for each column element in the second influence degree tensor, determine a variance of the column element as a dispersion degree of the influence degree value of the sample image block corresponding to the column element on the encoding of each sample image block, sort the variances of the column elements in ascending order, select a fourth preset number of variances according to the sorting order, and determine the sample image blocks corresponding to the selected variances as the redundant sample image blocks of each sample image block.

[0143] The retraining module 540 is configured to retrain the pre-trained image classification model using the sample images, update the association degree values between the sample image blocks during the training, and encode the sample images according to the updated association degree values between the sample image blocks, wherein for each sample image block, the association degree value of the redundant sample image block corresponding to the sample image block on the sample image block is reduced to a preset value, the type of the sample image is output according to the encoding of the sample image, and the parameters of the image classification model are adjusted according to the output type of the sample image and the labeled type.

[0144] In some embodiments, the retraining module 540 is configured to perform at least one of the following: generating a binary matrix, multiplying a second correlation matrix composed of correlation values ​​between sample image blocks in each self-attention head with the binary matrix to obtain an updated matrix, wherein, for each sample image block, the binary matrix is ​​set such that, for each sample image block, the correlation value generated by the redundant sample image block corresponding to the sample image block is reduced to a preset value in the updated matrix; for each sample image block, if the sample image block is a redundant sample image block that generates redundant information for encoding other sample image blocks, the elements of the sample image block in the input tensor of the image classification model are set to preset elements so that the correlation value generated by the sample image block to other sample image blocks is reduced to a preset value.

[0145] The electronic devices in the embodiments of this disclosure can each be implemented by various computing devices or computer systems, as described below. Figure 6 as well as Figure 7 Describe it.

[0146] Figure 6 This is a structural diagram of some embodiments of the electronic device disclosed herein. For example... Figure 6 As shown, the electronic device 60 of this embodiment includes a memory 610 and a processor 620 coupled to the memory 610. The processor 620 is configured to execute an image classification method or an image classification model training method in any of the embodiments of this disclosure based on instructions stored in the memory 610.

[0147] The memory 610 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory stores, for example, the operating system, application programs, boot loader, database, and other programs.

[0148] Figure 7 These are structural diagrams of some other embodiments of the electronic device disclosed herein. For example... Figure 7 As shown, the electronic device 70 of this embodiment includes a memory 710 and a processor 720, which are similar to the memory 610 and processor 620, respectively. It may also include an input / output interface 730, a network interface 740, a storage interface 750, etc. These interfaces 730, 740, 750, and the memory 710 and processor 720 can be connected, for example, via a bus 760. The input / output interface 730 provides a connection interface for input / output devices such as a display, mouse, keyboard, and touchscreen. The network interface 740 provides a connection interface for various networked devices, such as connecting to a database server or cloud storage server. The storage interface 750 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0149] Those skilled in the art will appreciate that embodiments of the disclosure can be supplied as a method, a system, or a computer program product. Thus, the disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the disclosure can take the form of a computer program product on one or more computer-usable non-transitory storage media (including but not limited to disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code thereon for use by or in connection with an instruction execution system. Program Code

[0150] The disclosure is described in reference to the flowchart and / or block diagrams of the method, apparatus (system) and computer program product according to embodiments of the disclosure. It should be understood that each flow and / or block in the flowchart and / or block diagrams, and a combination of flows and / or blocks in the flowchart and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor or other programmable data processing apparatus to produce a machine, so that the instructions, which are executed via the processor of the computer or other programmable data processing apparatus, generate a means for implementing the functions specified in the flowchart and / or block diagrams of the flowchart and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified in the flow or flows and / or block or blocks.

[0151] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufacture product including an instruction means, which implements the functions specified in the flowchart and / or block diagrams of the flowchart and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified in the flow or flows and / or block or blocks.

[0152] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus, so that a series of operational steps are performed on the computer or other programmable data processing apparatus to generate a computer implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a process for implementing the functions specified in the flowchart and / or block diagrams of the flowchart and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 Figure 1 means for carrying out the function specified in the flow or flows and / or block or blocks.

[0153] The above description is merely the preferred embodiments of the disclosure, and is not intended to limit the disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the disclosure shall be included in the protection scope of the disclosure.

Claims

1. An image classification method, comprising: The image is divided into multiple image patches and input into an image classification model, wherein the image classification model includes multiple self-attention layers; The correlation degree value between each image patch is determined based on the self-attention mechanism in the image classification model, wherein the correlation degree value between each image patch includes the correlation degree value between each image patch in each self-attention layer; Based on the correlation values ​​between each image patch, the redundant image patch corresponding to each image patch is determined from the plurality of image patches, including: for each self-attention layer, forming a first correlation tensor by the correlation values ​​between each image patch corresponding to all self-attention heads in the self-attention layer; determining a first average correlation tensor based on the first correlation tensor corresponding to each self-attention layer; squaring the difference between the first correlation tensor corresponding to each self-attention layer and the first average correlation tensor, summing the sums, and then dividing the sum by the number of self-attention layers to obtain a first uncertainty tensor; and determining the redundant image patch corresponding to each image patch based on the first uncertainty tensor. The correlation values ​​between each image block are updated, and the image is encoded according to the updated correlation values ​​between each image block. Specifically, for each image block, the correlation value of the redundant image block corresponding to that image block is reduced to a preset value. The type of the image is determined based on its encoding.

2. The image classification method according to claim 1, wherein, Each self-attention layer includes multiple self-attention heads. The correlation value between each image patch in each self-attention layer includes the correlation value between each image patch corresponding to each self-attention head in that self-attention layer. Determining the redundant image patch corresponding to each image patch based on the first uncertainty tensor includes: The uncertainty value of the correlation between each image patch corresponding to each self-attention head is extracted from the first uncertainty tensor to obtain the second uncertainty tensor corresponding to each self-attention head; Based on the degree of discreteness of each second uncertainty tensor, a first influence tensor is determined, wherein the first influence tensor includes the influence values ​​between each image block during encoding; Based on the first influence degree tensor, the redundant image blocks corresponding to each image block are determined.

3. The image classification method according to claim 2, wherein, The determination of redundant image blocks corresponding to each image block based on the first influence tensor includes at least one of the following: For each image block, a redundant image block corresponding to that image block is selected from the first influence tensor based on the influence value of each image block's encoding of that image block; For each image block, based on the dispersion of the influence value of the image block on the encoding of each image block in the first influence tensor, it is determined whether the image block is a redundant image block corresponding to each image block.

4. The image classification method according to claim 2, wherein, The determination of the first influence tensor based on the degree of discreteness of each second uncertainty tensor includes: Based on each of the second uncertainty tensors, determine the average uncertainty tensor; The differences between each second uncertainty tensor and the average uncertainty tensor are squared and summed. The sum is then divided by the number of self-attention heads to obtain the first degree of influence tensor.

5. The image classification method according to claim 3, wherein, Each row element in the first influence tensor corresponds to an image patch. The step of selecting a redundant image patch corresponding to each image patch from the first influence tensor based on the influence value of each image patch's encoding of that image patch includes: For each row of elements in the first influence tensor, sort the elements in that row from largest to smallest; Select a first preset number of elements according to the sorting order, and determine the image block corresponding to the selected element as a valid image block; Image blocks other than the valid image blocks are considered as redundant image blocks corresponding to the image blocks of the row element.

6. The image classification method according to claim 3, wherein, Each row element in the first influence tensor corresponds to an image patch. The step of selecting a redundant image patch corresponding to each image patch from the first influence tensor based on the influence value of each image patch's encoding of that image patch includes: For each row of elements in the first influence tensor, sort the elements in that row from largest to smallest; Select a first preset number of elements according to the sorting order, and determine the image block corresponding to the selected elements; Using the image patch corresponding to the element in that row as the center, determine the smallest rectangular region that contains the image patch corresponding to the selected element; Other image blocks outside the image block in the minimum rectangular region are considered as redundant image blocks corresponding to the image block corresponding to the row element.

7. The image classification method according to claim 3, wherein, Each column element in the first influence tensor corresponds to an image block. For each image block, determining whether the image block is a redundant image block corresponding to each image block is based on the dispersion of the influence value of the image block on the encoding of each image block in the first influence tensor, including: For each column element in the first influence tensor, the variance of that column element is determined as the degree of dispersion of the influence of the image block corresponding to that column element on the encoding of each image block. Sort the variances of each column of elements in ascending order, select a second preset number of variances according to the sorting order, and determine the image blocks corresponding to the selected variances as redundant image blocks for each image block.

8. The image classification method according to claim 3, wherein, The image classification model includes multiple self-attention layers, each self-attention layer includes multiple self-attention heads, and updating the correlation value between each image patch includes at least one of the following: A binary matrix is ​​generated. In each self-attention head, the first correlation matrix composed of the correlation values ​​between each image block is multiplied by the binary matrix to obtain an updated matrix. For each image block, the binary matrix is ​​set such that in the updated matrix, the correlation value of the redundant image block corresponding to the image block is reduced to a preset value. For each image block, if the image block is a redundant image block that generates redundant information for encoding other image blocks, the elements of the image block in the input tensor of the image classification model are set as preset elements so that the correlation value of the image block with other image blocks is reduced to a preset value.

9. The image classification method according to claim 1, wherein, Each self-attention layer includes multiple self-attention heads, and the determination of the correlation degree value between each image patch based on the self-attention mechanism in the image classification model includes: In each self-attention head, a query matrix and a key matrix are determined based on the feature matrix of the image input to that self-attention head; Multiply the query matrix by the key matrix and then divide by the square root of the number of columns in the key matrix to obtain the attention score matrix; The attention score matrix is ​​normalized to obtain a first correlation matrix composed of the correlation values ​​between each image patch.

10. The image classification method according to claim 1, wherein, The image classification model includes multiple encoding layers, each encoding layer includes a self-attention layer, and each self-attention layer includes multiple self-attention heads. Encoding the image based on the updated correlation values ​​between image patches includes: In each coding layer, the feature matrix of the image input from the previous layer is input into each self-attention head of the self-attention layer; In each self-attention head, a value matrix is ​​determined based on the feature matrix of the image input to that self-attention head; Multiply the updated matrix corresponding to the self-attention head by the value matrix to obtain the attention encoding matrix corresponding to the self-attention head, wherein the updated matrix is ​​composed of the updated correlation values ​​between each image patch; In this coding layer, the feature matrix of the image output by the coding layer is determined based on the attention coding matrix corresponding to each self-attention head; The feature matrix of the image output from the last coding layer is used as the coding matrix of the image.

11. A training method for an image classification model, comprising: The sample image is divided into multiple sample image blocks and input into the image classification model for pre-training. The image classification model includes multiple self-attention layers. When pre-training is complete, the correlation degree value between each sample image patch is determined based on the self-attention mechanism, wherein the correlation degree value between each sample image patch includes the correlation degree value between each sample image patch in each self-attention layer; Based on the correlation values ​​between various sample image blocks, redundant sample image blocks corresponding to each sample image block are determined from the plurality of sample image blocks, including: for each self-attention layer, forming a second correlation tensor by the correlation values ​​between various sample image blocks corresponding to all self-attention heads in the self-attention layer; determining a second average correlation tensor based on the second correlation tensor corresponding to each self-attention layer; summing the squared differences between the second correlation tensor corresponding to each self-attention layer and the second average correlation tensor, and then dividing the sum by the number of self-attention layers to obtain a third uncertainty tensor; and determining redundant sample image blocks corresponding to each sample image block based on the third uncertainty tensor. The pre-trained image classification model is retrained using the sample images. During training, the correlation values ​​between each sample image patch are updated, and the sample images are encoded according to the updated correlation values. Specifically, for each sample image patch, the correlation values ​​of the redundant sample image patches corresponding to that sample image patch are reduced to a preset value. Based on the encoding of the sample images, the type of the sample images is output. Based on the output type of the sample images and the type of annotation, the parameters of the image classification model are adjusted.

12. The training method according to claim 11, wherein, Each self-attention layer includes multiple self-attention heads. The correlation value between each sample image patch in each self-attention layer includes the correlation value between each sample image patch corresponding to each self-attention head in that self-attention layer. Determining the redundant sample image patch corresponding to each sample image patch based on the third uncertainty tensor includes: The uncertainty value of the correlation between each sample image block corresponding to each self-attention head is extracted from the third uncertainty tensor to obtain the fourth uncertainty tensor corresponding to each self-attention head; Based on the degree of discreteness of each fourth uncertainty tensor, a second influence tensor is determined, wherein the second influence tensor includes the influence values ​​between each sample image block during encoding; Based on the second influence degree tensor, the redundant sample image blocks corresponding to each sample image block are determined.

13. The training method according to claim 12, wherein, The determination of redundant sample image blocks corresponding to each sample image block based on the second influence degree tensor includes at least one of the following: For each sample image block, a redundant sample image block is selected from the second influence tensor based on the influence value of each sample image block on the encoding of that sample image block. For each sample image block, based on the dispersion of the influence value of the sample image block on the encoding of each sample image block from the second influence tensor, it is determined whether the sample image block is a redundant sample image block corresponding to each sample image block.

14. The training method according to claim 13, wherein, Each row element in the second influence tensor corresponds to a sample image block. The step of selecting redundant sample image blocks corresponding to each sample image block from the second influence tensor based on the influence value of each sample image block's encoding of that sample image block includes: For each row of elements in the second influence tensor, sort the elements in that row from largest to smallest; then select a third preset number of elements according to the sorting order. The sample image block corresponding to the selected element is determined as a valid sample image block, or, with the sample image block corresponding to the element in that row as the center, the smallest rectangular area containing the sample image block corresponding to the selected element is determined, and the sample image block in the smallest rectangular area is determined as a valid sample image block. Other sample image blocks besides the valid sample image blocks are regarded as redundant sample image blocks corresponding to the sample image blocks of the row element.

15. The training method according to claim 13, wherein, Each column element in the second influence tensor corresponds to a sample image block. For each sample image block, determining whether the sample image block is a redundant image sample block corresponding to each sample image block is based on the dispersion of the influence value of the sample image block on the encoding of each sample image block in the second influence tensor, including: For each column element in the second influence tensor, the variance of that column element is determined as the degree of dispersion of the influence of the sample image block corresponding to that column element on the encoding of each sample image block. Sort the variances of each column of elements in ascending order, select a fourth preset number of variances according to the sorting order, and determine the sample image blocks corresponding to the selected variances as redundant sample image blocks corresponding to each sample image block.

16. The training method according to claim 13, wherein, Each self-attention layer includes multiple self-attention heads, and updating the correlation value between each sample image patch includes at least one of the following: A binary matrix is ​​generated. In each self-attention head, the second correlation matrix composed of the correlation values ​​between each sample image block is multiplied by the binary matrix to obtain an updated matrix. For each sample image block, the binary matrix is ​​set such that in the updated matrix, the correlation value of the redundant sample image block corresponding to the sample image block is reduced to a preset value. For each sample image block, if the sample image block is a redundant sample image block that generates redundant information for encoding other sample image blocks, the elements of the sample image block in the input tensor of the image classification model are set as preset elements so that the correlation value of the sample image block with other sample image blocks is reduced to a preset value.

17. An image classification device, comprising: An input module is used to divide an image into multiple image patches and input them into an image classification model, wherein the image classification model includes multiple self-attention layers; The correlation degree value determination module is used to determine the correlation degree value between each image patch based on the self-attention mechanism in the image classification model, wherein the correlation degree value between each image patch includes the correlation degree value between each image patch in each self-attention layer; A redundant image block determination module is used to determine the redundant image blocks corresponding to each image block from the plurality of image blocks based on the correlation degree values ​​between each image block. This includes: for each self-attention layer, forming a first correlation tensor by constructing the correlation degree values ​​between each image block corresponding to all self-attention heads in that self-attention layer; determining a first average correlation tensor based on the first correlation tensor corresponding to each self-attention layer; squaring and summing the differences between the first correlation tensor corresponding to each self-attention layer and the first average correlation tensor; dividing the sum by the number of self-attention layers to obtain a first uncertainty tensor; and determining the redundant image blocks corresponding to each image block based on the first uncertainty tensor. The encoding module is used to update the correlation degree value between each image block and to encode the image according to the updated correlation degree value between each image block. Specifically, for each image block, the correlation degree value generated by the redundant image block corresponding to the image block is reduced to a preset value. A classification module is used to determine the type of the image based on its encoding.

18. A training device for an image classification model, comprising: The input module is used to divide the sample image into multiple sample image blocks and input the image classification model for pre-training, wherein the image classification model includes multiple self-attention layers; The correlation degree value determination module is used to determine the correlation degree value between each sample image patch based on the self-attention mechanism when the pre-training is completed. The correlation degree value between each sample image patch includes the correlation degree value between each sample image patch in each self-attention layer. A redundant image block determination module is used to determine redundant sample image blocks corresponding to each sample image block from the plurality of sample image blocks based on the correlation degree values ​​between each sample image block. This includes: for each self-attention layer, forming a second correlation tensor based on the correlation degree values ​​between each sample image block corresponding to all self-attention heads in that self-attention layer; determining a second average correlation tensor based on the second correlation tensor corresponding to each self-attention layer; squaring and summing the differences between the second correlation tensor corresponding to each self-attention layer and the second average correlation tensor; then dividing the sum by the number of self-attention layers to obtain a third uncertainty tensor; and determining the redundant sample image blocks corresponding to each sample image block based on the third uncertainty tensor. The retraining module is used to retrain the pre-trained image classification model using the sample images. During the training process, the correlation values ​​between each sample image patch are updated, and the sample images are encoded according to the updated correlation values ​​between each sample image patch. Specifically, for each sample image patch, the correlation values ​​of the redundant sample image patches corresponding to that sample image patch are reduced to a preset value. Based on the encoding of the sample images, the type of the sample images is output. Based on the output type of the sample images and the type of annotation, the parameters of the image classification model are adjusted.

19. An electronic device comprising: processor; as well as A memory coupled to the processor is used to store instructions that, when executed by the processor, cause the processor to perform the image classification method as described in any one of claims 1-10 or the training method for the image classification model as described in any one of claims 11-16.

20. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein, When executed by a processor, the program implements the image classification method according to any one of claims 1-10 or the training method for the image classification model according to any one of claims 11-16.

Images