Image classification method and image classification model training method and device

Abstract

The present disclosure provides an image classification method and a training method for an image classification model, which relates to the field of artificial intelligence, and specifically to the field of computer vision and deep learning technology. Among them, the image classification model includes a self-attention encoder, a global encoding network, a first local encoding network and a prediction network. The specific implementation scheme of the image classification method is: the image to be classified is divided into multiple image blocks to obtain an image block sequence; the image block sequence is self-attention encoded by a self-attention encoder to obtain a first feature map sequence; the first feature map sequence includes multiple feature maps for multiple image blocks respectively; the global encoding network is used to extract the global features of the first feature map sequence to obtain a global feature map; the first local encoding network is used to extract the first local features of the first feature map sequence to obtain a first local feature map; and the global feature map and the first local feature map are input into the prediction network to obtain classification information of the image to be classified.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, specifically to the field of computer vision and deep learning technology, and in particular to an image classification method based on an image classification model and a training method, device, electronic device and storage medium for the image classification model. Background Art

[0002] With the development of computer technology and network technology, deep learning technology has been widely used in many fields. For example, deep learning technology can be used to extract image features and predict the image category based on the image features. In actual scenarios, for example, the predicted image category can be the age category of the object in the image, thus providing potential value for application scenarios such as human-computer interaction. Summary of the invention

[0003] The present disclosure aims to provide an image classification method based on an image classification model for improving classification accuracy, and a training method, device, electronic device and storage medium for the image classification model.

[0004] According to one aspect of the present disclosure, there is provided an image classification method based on an image classification model, wherein the image classification model comprises a self-attention encoder, a global encoding network, a first local encoding network and a prediction network; the image classification method comprises: dividing an image to be classified into a plurality of image blocks to obtain an image block sequence; performing self-attention encoding on the image block sequence using a self-attention encoder to obtain a first feature map sequence; the first feature map sequence comprises a plurality of feature maps respectively for the plurality of image blocks; extracting global features of the first feature map sequence using a global encoding network to obtain a global feature map; extracting first local features of the first feature map sequence using a first local encoding network to obtain a first local feature map; and inputting the global feature map and the first local feature map into a prediction network to obtain classification information of the image to be classified.

[0005] According to one aspect of the present disclosure, a training method for an image classification model is provided, wherein the image classification model includes a self-attention encoder, a global encoding network, and a first local encoding network; the training method includes: dividing a sample image into multiple image blocks to obtain an image block sequence; using a self-attention encoder to perform self-attention encoding on the image block sequence to obtain a first feature map sequence; the first feature map sequence includes multiple feature maps for multiple image blocks respectively; using a global encoding network to extract global features of the first feature map sequence to obtain a global feature map; using a first local encoding network to extract first local features of the first feature map sequence to obtain a first local feature map; determining first classification information based on the global feature map, and determining second classification information based on the first local feature map; and training the image classification model based on a first difference between the first classification information and the second classification information.

[0006] According to one aspect of the present disclosure, there is provided an image classification device based on an image classification model, wherein the image classification model comprises a self-attention encoder, a global encoding network, a first local encoding network and a prediction network; the image classification device comprises: an image segmentation module, used to segment an image to be classified into multiple image blocks to obtain an image block sequence; a self-attention encoding module, used to perform self-attention encoding on the image block sequence using a self-attention encoder to obtain a first feature map sequence; the first feature map sequence comprises multiple feature maps respectively for multiple image blocks; a global encoding module, used to extract global features of the first feature map sequence using a global encoding network to obtain a global feature map; a local encoding module, used to extract first local features of the first feature map sequence using a first local encoding network to obtain a first local feature map; and a classification module, used to input the global feature map and the first local feature map into the prediction network to obtain classification information of the image to be classified.

[0007] According to one aspect of the present disclosure, a training device for an image classification model is provided, wherein the image classification model includes a self-attention encoder, a global encoding network, and a first local encoding network; the training device includes: an image segmentation module, used to segment a sample image into multiple image blocks to obtain an image block sequence; a self-attention encoding module, used to use the self-attention encoder to perform self-attention encoding on the image block sequence to obtain a first feature map sequence; the first feature map sequence includes multiple feature maps respectively for multiple image blocks; a global encoding module, used to use the global encoding network to extract global features of the first feature map sequence to obtain a global feature map; a local encoding module, used to use the first local encoding network to extract first local features of the first feature map sequence to obtain a first local feature map; a classification module, used to determine first classification information based on the global feature map, and determine second classification information based on the first local feature map; and a model training module, used to train the image classification model based on a first difference between the first classification information and the second classification information.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image classification method based on the image classification model and / or the training method of the image classification model provided by the present disclosure.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the image classification method based on the image classification model and / or the image classification model training method provided by the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the image classification method based on the image classification model and / or the image classification model training method provided by the present disclosure.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 It is a schematic diagram of an application scenario of an image classification method based on an image classification model and a training method and device for an image classification model according to an embodiment of the present disclosure;

[0014] Figure 2 is a flowchart of an image classification method based on an image classification model according to an embodiment of the present disclosure;

[0015] Figure 3 is a structural schematic diagram of an image classification model according to an embodiment of the present disclosure;

[0016] Figure 4 is a structural schematic diagram of an image classification model according to another embodiment of the present disclosure;

[0017] Figure 5 is a flowchart of a method for training an image classification model according to an embodiment of the present disclosure;

[0018] Figure 6 is a schematic diagram of the principle of a training method for an image classification model according to an embodiment of the present disclosure;

[0019] Figure 7 is a schematic diagram of the principle of a training method for an image classification model according to another embodiment of the present disclosure;

[0020] Figure 8 is a structural block diagram of an image classification device based on an image classification model according to an embodiment of the present disclosure;

[0021] Fig. 9 is a structural block diagram of a training device for an image classification model according to an embodiment of the present disclosure; and

[0022] Fig.10 It is a block diagram of an electronic device used to implement the image classification method based on the image classification model and / or the training method of the image classification model of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] The present disclosure provides an image classification method based on an image classification model, wherein the image classification model includes a self-attention encoder, a global encoding network, a first local encoding network and a prediction network. The method includes an image segmentation stage, a self-attention encoding stage, a global encoding stage, a local encoding stage and a classification stage. In the image segmentation stage, the image to be classified is segmented into a plurality of image blocks to obtain an image block sequence. In the self-attention encoding stage, the self-attention encoder is used to perform self-attention encoding on the image block sequence to obtain a first feature map sequence; the first feature map sequence includes a plurality of feature maps for the plurality of image blocks respectively. In the global encoding network, the global encoding network is used to extract the global features of the first feature map sequence to obtain a global feature map. In the local encoding stage, the first local encoding network is used to extract the first local features of the first feature map sequence to obtain a first local feature map. In the classification stage, the global feature map and the first local feature map are input into the prediction network to obtain classification information of the image to be classified.

[0025] The following will be combined Figure 1 The application scenarios of the method and device provided by the present disclosure are described.

[0026] Figure 1 It is a schematic diagram of application scenarios of an image classification method based on an image classification model and a training method and device for an image classification model according to an embodiment of the present disclosure.

[0027] like Figure 1 As shown, the application scenario 100 of this embodiment may include an electronic device 110, which may be various electronic devices with processing functions, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, servers, and the like.

[0028] The electronic device 110 may, for example, process the input image 120 to obtain classification information of the image 120. For example, in the application of facial age estimation, the image 120 is an image including a face, and the classification information may include the probability that the face in the image belongs to each of multiple age groups. According to the classification information, the age category 130 of the face in the image may be obtained. The multiple age groups may, for example, include 101 age groups from 0 to 100, and the interval size of each age group is 1.

[0029] According to the embodiments of the present disclosure, face age recognition is applied in scenarios such as public facility management, user portrait and product recommendation, and information security management. For example, in the public facility management scenario, the crowd gathering and age distribution in the public place can be determined to reasonably adjust the flow of people in various areas of the public place. For example, more attention can be paid to older or younger people, and more service prompts and help can be provided. For example, in the user portrait and product recommendation scenario, the age category can be determined to provide the target audience with appropriate push content and advertisements, so as to provide personalized services to users. For example, in the information security management scenario, the age category can be determined to give reminders to the target audience and automatically log off after exceeding the specified time. In this way, the occurrence of situations such as the target audience being addicted to the Internet can be reduced.

[0030] In one embodiment, the electronic device 110 may use an image classification model 140 to process the image 120, so as to obtain classification information of the image 120. The image classification model 140 may, for example, include a model composed of a convolutional neural network and a classifier, wherein the convolutional neural network is used to extract image features as a basis for determining the category, and the image features are processed by the classifier to obtain classification information. Alternatively, the image classification model 140 may be constructed based on a label distribution method, such as a deep label distribution (Deep Label Distributional Learning, DLDL)-v2 model. Alternatively, the image classification model may be constructed based on an adaptive variance distribution learning method, etc. Alternatively, the image classification model may also be constructed based on a vision transformer (ViT) network, which is not limited in the present disclosure.

[0031] In one embodiment, if Figure 1As shown, the application scenario may also include a server 150, and the electronic device 110 may be connected to the server 150 through a network. The network may be a wired or wireless communication link. For example, the electronic device 110 may send a model acquisition request to the server 150 through the network, and the server 150 may send the trained image classification model 140 to the electronic device 110 in response to the model acquisition request.

[0032] In one embodiment, the electronic device 110 may also send the image 120 to the server 150 , and the server 150 may use the trained image classification model 140 to process the image 120 to obtain classification information.

[0033] It should be noted that the image classification method based on the image classification model provided in the present disclosure can be executed by the electronic device 110 or by the server 150. Accordingly, the image classification device based on the image classification model provided in the present disclosure can be set in the electronic device 110 or in the server 150. The training method of the image classification model provided in the present disclosure can be executed by the server 150. Accordingly, the training device of the image classification model provided in the present disclosure can be set in the server 150.

[0034] It should be understood that Figure 1 The number and type of the electronic devices 110 and the server 150 in the embodiment are only for illustration. According to the implementation requirements, there may be any number and type of electronic devices 110 and the server 150.

[0035] The following will be combined Figure 2 to Figure 4 The image classification method based on the image classification model provided by the present disclosure is described in detail.

[0036] Figure 2 It is a flowchart of an image classification method based on an image classification model according to an embodiment of the present disclosure.

[0037] like Figure 2 As shown, the image classification method 200 based on the image classification model of this embodiment may include operations S210 to S250. The image classification model includes a self-attention encoder, a global encoding network, a first local encoding network and a prediction network. The self-attention encoder may be an encoder in a Transformer network.

[0038] In operation S210, the image to be classified is divided into a plurality of image blocks to obtain an image block sequence.

[0039] According to an embodiment of the present disclosure, the image to be classified can be divided into multiple image blocks, and then the multiple image blocks obtained after the division are flattened to obtain an image block sequence. For example, the size of the image to be classified is set to C×H×W, where C is the number of channels, H and W are the height and width of the image to be classified, and the size of each image block is 2×2, then a total of H / 2*W / 2 image blocks can be obtained. The H / 2*W / 2 image blocks are flattened to the same height along the height direction to obtain an image block sequence.

[0040] In operation S220, a self-attention encoder is used to perform self-attention encoding on the image block sequence to obtain a first feature map sequence.

[0041] According to an embodiment of the present disclosure, the first feature map sequence includes multiple feature maps for multiple image blocks respectively. The self-attention encoder may include a multi-layer encoder. In this embodiment, the embedded representation of the image block sequence may be input into the multi-layer encoder, and the last layer of encoder in the multi-layer encoder outputs the first feature map sequence. Each layer of encoder can be a block (Block) of the encoding part in the Transformer structure. Among them, the number of encoder layers included in the self-attention encoder can be set according to actual needs. For example, if the image classification model is built based on the ViT network, the number of encoder layers can be 6, 8, 9, etc. Any value greater than 1 and less than or equal to 12, and the present disclosure does not limit this.

[0042] Among them, the embedded representation of each image block in the image block sequence can be used as a token. It should be noted that when processing the classified image, this embodiment does not need to input a learnable category token into the self-attention encoder, but only needs to input multiple tokens obtained according to multiple image blocks. It is understandable that the embedded representation of each image block can also be added with the position information of each image block, and the position information can be obtained by a sine coding method or a cosine coding method, or by a learning method, and the present disclosure does not limit this.

[0043] It can be understood that a Block of the encoding part in the Transformer structure may include a self-attention layer, a first residual and normalization layer, a feed-forward network, and a second residual and normalization layer connected in sequence.

[0044] In operation S230, a global encoding network is used to extract global features of the first feature map sequence to obtain a global feature map.

[0045] According to an embodiment of the present disclosure, the global encoding network can be composed of other structures in the encoding part of the ViT network except the self-attention encoder. For example, if the encoding part of the ViT network includes a total of 12 blocks connected in sequence, and the self-attention encoder includes the first 9 blocks of the 12 blocks, then the global encoding network can include 3 blocks after the first 9 blocks. This is because the Transformer network of the self-attention mechanism has global modeling capabilities and can capture long-term relationships. In this way, the feature maps in the feature map sequence output by the 3 blocks after the first 9 blocks can be rearranged to obtain a global feature map. When rearranging the feature map, the position of the image block corresponding to the feature map in the image to be classified can be used as a basis.

[0046] In operation S240, a first local encoding network is used to extract a first local feature of a first feature map sequence to obtain a first local feature map.

[0047] According to an embodiment of the present disclosure, the first local coding network may, for example, only focus on part of the feature maps in the first feature map sequence, and encode the part of the feature map, so as to achieve the purpose of focusing only on the local information of the image to be classified, and extract the first local feature map. Alternatively, the first local coding network may use a network structure with a fixed and limited receptive field, such as a convolutional neural network, to process each feature map in the first feature map sequence, and rearrange the multiple feature maps obtained after processing to obtain the first local feature map. When rearranging the feature maps, the position of the image block corresponding to the processed feature map in the image to be classified may be used as a basis.

[0048] In operation S250, the global feature map and the first local feature map are input into a prediction network to obtain classification information of the image to be classified.

[0049] According to an embodiment of the present disclosure, the prediction network may include a fusion layer and a Softmax classifier. The fusion layer is used to fuse the global feature map and the first local feature map to obtain a fused feature map. After the fused feature map is processed by the Softmax classifier, the Softmax classifier may output classification information. The classification information may include probability distribution information (specifically, a probability vector) of the image to be classified for multiple predetermined categories, and the probability distribution information includes the predicted probability value of the image to be classified belonging to each predetermined category.

[0050] According to an embodiment of the present disclosure, the prediction network may further include a fully connected layer disposed between the fusion layer and the Softmax classifier, and the fused feature map may be projected into the category space after being processed by the fully connected layer. The features output by the fully connected layer are input into the Softmax classifier, and the classifier outputs classification information.

[0051] The fusion layer can achieve the fusion of the two feature maps by adding the first local feature map and the global feature map. The fusion of the two feature maps can also be achieved by calculating the weighted sum of the two feature maps. The weights used in the weighting are learned during the training process of the image classification model, and the present disclosure does not limit this.

[0052] Compared to the technical solution of setting a category token and using the output information corresponding to the category token as the classification information, the embodiment of the present disclosure predicts the classification information based on the extracted feature map in the image classification model based on the self-attention mechanism, so it can make full use of the information in the feature map, which is conducive to improving the classification accuracy. Furthermore, by setting a global encoding network to extract global features. And setting a local encoding branch to extract local features, and finally comprehensively considering the global features and local features to predict the classification information, it can further make full use of the rich local detail information in the feature map, which is conducive to further improving the classification accuracy.

[0053] It is understandable that in the age prediction application, the image to be classified may be an image with a human face, and the multiple predetermined categories include the aforementioned multiple age groups. In this way, since the classification information is often affected by aspects such as the facial skin texture, brightness, and wrinkle texture of the human face, if only the global information is focused on, the key details of the face will be ignored; and if only the local information is focused on, the overall structural information will be lost. The disclosed embodiment predicts age by comprehensively considering global features and local features, which can effectively improve the prediction accuracy.

[0054] Figure 3 It is a structural diagram of an image classification model according to an embodiment of the present disclosure.

[0055] According to an embodiment of the present disclosure, the aforementioned global encoding network may include, for example, a first encoding subnetwork and a first conversion subnetwork. Among them, the first encoding subnetwork is used to extract features based on the first feature map sequence, and the first conversion subnetwork is used to convert the extracted feature sequence into a global feature map. This is because when the input data of the encoding subnetwork is a feature map sequence, the output data is also a feature map sequence. In order to facilitate fusion with the first local feature map and predict classification information based on the fusion result, it is necessary to convert the feature map sequence into an overall feature map. In one embodiment, the first conversion subnetwork may also be used to adjust the size of the feature map in the feature map sequence, so that the size of the final global feature map is equal to that of the first local feature map, so as to facilitate the fusion of the two feature maps.

[0056] Exemplarily, the first coding subnetwork may include the 3 blocks following the first 9 blocks described above. Based on the first feature map sequence, the data obtained after processing by the first coding subnetwork may be a second feature map sequence. In this embodiment, the first feature map sequence may be input into the first coding subnetwork, and the second feature map sequence may be output by the first coding subnetwork. The second coding subnetwork may be used to convert the second feature map sequence into a feature map matrix, which may be used as a global feature map. It is understandable that the process of converting the second feature map sequence into a feature map matrix may be opposite to the process described above of expanding the multiple image blocks obtained by segmenting the image to be classified into an image block sequence. For example, the second feature map sequence includes multiple feature maps for multiple image blocks. In this embodiment, the feature map for the image block may be placed at the position of the image block in the image to be classified, so that the multiple feature maps may constitute an overall global feature map.

[0057] like Figure 3 As shown, in the embodiment 300, the self-attention encoder 310 is composed of 9 blocks of the encoding part in the Transformer network. After the image block sequence 301 is processed by the self-attention encoder 310, a first feature map sequence 302 can be obtained. In addition to the first encoding subnetwork 322 and the first conversion subnetwork 323, the global encoding network 320 can also include a first fusion subnetwork 321. The first fusion subnetwork 321 is used to globally fuse the feature maps in the first feature map sequence 302, so as to obtain a third feature map sequence 303. After the third feature map sequence 303 is processed by the first encoding subnetwork 322, a second feature map sequence can be obtained, and after the second feature map sequence is converted by the first conversion subnetwork 323, a global feature map 304 can be obtained. Through the setting of the first fusion subnetwork 321, information fusion between the feature maps of each image block can be achieved, and the correlation between the image blocks is considered to a certain extent, which is conducive to improving the expression ability of the obtained global feature map and the final classification accuracy.

[0058] For example, the first encoding subnetwork 322 may use the 3 blocks following the first 9 blocks described above.

[0059] Exemplarily, the first fusion subnetwork 321 can randomly divide the feature maps in the first feature map sequence 302 into several groups, each of which includes at least two feature maps. Subsequently, a weighted sum is calculated for at least two feature maps in each group of feature maps to obtain a weighted feature map, which can be used as a feature map in the third feature map sequence 303. In this way, each feature map in the second feature map sequence can be obtained by fusing the feature maps of at least two image blocks, which can reflect the global features of the image to be classified to a certain extent.

[0060] Exemplarily, the first fusion subnetwork 321 can obtain the third feature map sequence by using a reverse operation similar to the reorganized pixel map. In this way, the feature maps in the first feature map sequence can be fully integrated in the global range, which is beneficial to improving the expression ability of the obtained global features and improving the classification accuracy.

[0061] For example, the first fusion subnetwork 321 may include a first conversion layer, a first downsampling layer, and a second conversion layer. Among them, the first conversion layer may, for example, convert the first feature map sequence 302 into a first feature map matrix 321_1. Specifically, the first conversion layer is used to arrange each feature map in the first feature map sequence 302 according to the position of the image block to be classified in the image to which each feature map corresponds, thereby forming a first feature map matrix 321_1. For example, if the number of first image blocks is 6*6, the obtained first feature map sequence 302 includes 36 feature maps, and the 36 feature maps can constitute a 6×6 first feature map matrix 321_1. The first downsampling layer is used to sample the feature maps in the first feature map matrix 321_1 at intervals to obtain multiple fusion sub-matrices. The sampling interval value may be, for example, 2, 3, or any integer greater than 1. Taking the interval value as 2 and the size of the first feature map matrix 321_1 as 6×6 as an example, in one sampling, the first feature map, the third feature map and the fifth feature map of the cardinality row in the first feature map matrix 321_1 can be sampled, and the obtained feature maps can be used to form a fused sub-matrix. Similarly, the second feature map, the fourth feature map and the sixth feature map of the cardinality row in the first feature map matrix 321_1 can also be sampled to obtain a fused sub-matrix. Similarly, through multiple samplings, four non-repeating fused sub-matrices 321_2 can be obtained. The second conversion layer is used to convert multiple fused sub-matrices into multiple fused feature maps, thereby obtaining a third feature map sequence composed of multiple fused feature maps. Specifically, the second conversion layer can, for example, first arrange the four fused sub-matrices 321_2 in a direction perpendicular to the width and height. Thereby obtaining a tensor 321_3 of size 3×3×4. Subsequently, the four feature maps with the same position in the height direction and the width direction in the tensor 321_3 are spliced ​​to obtain a spliced ​​feature map, and a total of 3*3=9 spliced ​​feature maps 321_4 can be obtained. In this embodiment, the 9 spliced ​​feature maps can directly constitute the third feature map sequence 303. Alternatively, the 9 spliced ​​feature maps 321_4 can be sequentially input into a fully connected layer to adjust the channel dimension of each spliced ​​feature map, and the 9 feature maps output by the fully connected layer constitute the third feature map sequence 303. For example, the size of the channel dimension of the 9 feature maps output by the fully connected layer can be equal to the size of the channel dimension of the feature map in the first feature map sequence, and the present disclosure does not limit this.

[0062] It is understandable that the number of the above-mentioned first image blocks of 6*6 is only used as an example to facilitate the understanding of the present disclosure. For example, the number of the first image blocks can also be 14*14, etc., and the number of image blocks obtained by cutting in the width direction and the number of image blocks obtained by cutting in the height direction can also be different, and the present disclosure does not limit this. Furthermore, the above-mentioned sampling interval value is also only used as an example to facilitate the understanding of the present disclosure, and the present disclosure does not limit this. By setting the first downsampling layer, this embodiment can achieve the purpose of using fewer feature maps to replace the original feature maps for subsequent calculations, and thus can reduce the amount of calculation and video memory usage of the image classification model, while ensuring the image classification accuracy, it can improve the classification efficiency to a certain extent.

[0063] In one embodiment, similar to the global encoding network, such as Figure 3 As shown, the first local coding network 330 in the embodiment 300 may include a second fusion subnetwork 331, a second coding subnetwork 332, and a second conversion subnetwork 333. The second coding subnetwork 332 is similar to the first coding subnetwork 322, and both may use the 3 blocks arranged after the first 9 blocks described above.

[0064] Among them, the difference between the second fusion subnetwork 331 and the first fusion subnetwork 321 is that the second fusion subnetwork 331 only fuses part of the feature maps in the first feature map sequence 302 to achieve attention to the local area in the image to be classified, thereby extracting the first local feature map.

[0065] Specifically, in this embodiment, the second fusion subnetwork 331 can be used to fuse the feature maps of the first part in the first feature map sequence 302, so as to obtain the fourth feature map sequence 305. For example, the number of the first image blocks can also be set to 14*14, and the size of the first part can be any size such as 12*12, 10*10, etc. The feature map of the first part can include a plurality of continuous feature maps in the first feature map sequence 302, or a plurality of discontinuous feature maps, which is not limited in the present disclosure. When fusing the feature maps of the first part, a pooling operation can be used to achieve fusion. Among them, the pooling operation can include an average pooling operation, an adaptive average pooling operation, a maximum pooling operation, etc., which is not limited in the present disclosure. Through the pooling operation, while paying attention to the local features of the image to be classified, the purpose of using fewer feature maps to replace the original feature maps for subsequent calculations can be achieved, and thus the amount of calculation and the amount of video memory occupied by the image classification model can be reduced, while ensuring the accuracy of image classification, the classification efficiency can be improved to a certain extent. After obtaining the fourth feature map sequence 305, the fourth feature map sequence 305 can be input into the second encoding sub-network 332, and the second encoding sub-network 332 outputs the fifth feature map sequence. Finally, the fifth feature map sequence is converted into a feature map matrix by the second conversion sub-network 333 to obtain the first local feature map 306. The second conversion sub-network 333 has a similar structure and function to the first conversion sub-network 323, and will not be described in detail here.

[0066] In one embodiment, the feature map of the first part selected may be a feature map of an image block corresponding to the central area of ​​the image to be classified, so as to improve the richness of the content expressed by the local feature map obtained in the end. This is because the central area of ​​the image often contains richer information. Specifically, in this embodiment, the second fusion subnetwork may include a third conversion layer, a first pooling layer, and a fourth conversion layer. Among them, the third conversion layer is similar to the aforementioned first conversion layer, and is used to convert the first feature map sequence into a first feature map matrix, which will not be repeated here. The first pooling layer can be used to intercept the feature map of the first predetermined area in the first feature map matrix, and perform a pooling operation on the intercepted feature map to obtain a sixth feature map matrix. The first predetermined area can be centered on the center point of the first feature map matrix. The fourth conversion layer is used to convert the sixth feature map matrix into a fourth feature map sequence 305. The operation performed by the fourth conversion layer is opposite to the operation performed by the third conversion layer. For example, the size of the sixth feature map matrix is ​​set to 3×3, and after the matrix is ​​expanded, a total of 9 feature maps of 3*3 can be obtained, and the 9 feature maps can constitute the fourth feature map sequence 305. It should be noted that the number of feature maps in the fourth feature map sequence 305 should be equal to the number of feature maps in the third feature map sequence 303, so that the first local feature map and the global feature map have the same size.

[0067] According to an embodiment of the present disclosure, after obtaining the global feature map 304 and the first local feature map 306 , the global feature map 304 and the first local feature map 306 may be input into a prediction network 340 , and the prediction network 340 outputs classification information 307 .

[0068] Figure 4 It is a structural diagram of an image classification model according to another embodiment of the present disclosure.

[0069] In one embodiment, if Figure 4 As shown, the image classification model may include a second local encoding network 450 in addition to the self-attention encoder 410, the global encoding network 420, the first local encoding network 430 and the prediction network 440. The second local encoding network 450 has a similar function to the first local encoding network 430, and is used to extract local feature maps. The difference is that the local feature maps extracted by the two local encoding networks may have receptive fields of different sizes.

[0070] Specifically, the embodiment can use the second local encoding network 450 to extract the second local feature of the first feature map sequence 402 to obtain the second local feature map 409. Finally, the second local feature map 409, the global feature map 404 and the first local feature map 406 can all be input into the prediction network 440, and the classification information 407 of the image to be classified can be obtained after being processed by the prediction network 440. In this way, the classification information can be obtained by comprehensively considering the global features and the local features of two different receptive fields. Therefore, the accuracy of the classification information can be improved to a certain extent, that is, the classification accuracy of the image classification model is improved. For example, compared with the first local feature map, the second local feature map can focus on local information of smaller size, that is, the size of the second local feature is smaller than the size of the aforementioned first local feature, and the receptive field of the second local feature map is larger than the receptive field of the first local feature map.

[0071] In one embodiment, similar to the first local encoding network 430, the second local encoding network 450 may include a third fusion subnetwork 451, a third encoding subnetwork 452, and a third conversion subnetwork 453. The third encoding subnetwork 452 is similar to the second encoding subnetwork 432, and the third conversion subnetwork 453 is similar to the second conversion subnetwork 433. Specifically, the third fusion subnetwork 451 may be used to fuse the feature graphs of the second part of the first feature graph sequence to obtain the seventh feature graph sequence 408. The third fusion subnetwork 451 performs similar operations to the second fusion subnetwork 431, except that the number of feature graphs of the second part fused by the third fusion subnetwork 451 is less than the number of feature graphs of the first part fused by the second fusion subnetwork 431. The number of feature graphs in the seventh feature graph sequence 408, the number of feature graphs in the fourth feature graph sequence 405, and the number of feature graphs in the third feature graph sequence 403 may all be equal. The seventh feature graph sequence 408 is input into the third encoding subnetwork 452, and the third encoding subnetwork 452 may output the sixth feature graph sequence. Finally, the third conversion sub-network is used to convert the sixth feature map sequence into a feature map matrix, and the second local feature map 409 can be obtained.

[0072] In one embodiment, the third fusion subnetwork may include a fifth conversion layer, a second pooling layer, and a sixth conversion layer. Among them, the fifth conversion layer is the same as the aforementioned third conversion layer, and the sixth conversion layer is similar to the aforementioned fourth conversion layer. The difference between the second pooling layer and the aforementioned first pooling layer is that the size of the predetermined area where the intercepted feature map is located is different. For example, this embodiment can use the fifth conversion layer to convert the first feature map sequence 402 into a first feature map matrix. The second pooling layer is used to perform a pooling operation on the feature map of the second predetermined area in the intercepted first feature map matrix, and an eighth feature map matrix can be obtained. Among them, if the size of the first predetermined area is 12×12, the size of the second predetermined area in this embodiment can be, for example, 8×8, and the second predetermined area can also be centered on the center point of the first feature map matrix. The sixth conversion layer is used to convert the eighth feature map matrix into the seventh feature map sequence.

[0073] In order to facilitate the implementation of the image classification method based on the image classification model provided by the present disclosure, the present disclosure also provides a training method for the image classification model. Figure 5 to Figure 7 The training method is described in detail.

[0074] Figure 5 It is a flowchart of a method for training an image classification model according to an embodiment of the present disclosure.

[0075] like Figure 5As shown, the training method 500 of this embodiment may include operations S510 to S560. The image classification model may be the image classification model adopted by the image classification method 200 of the image classification model. For example, the image classification model at least includes a self-attention encoder, a global encoding network, and a first local encoding network.

[0076] In operation S510, a sample image is segmented into a plurality of image blocks to obtain an image block sequence.

[0077] In operation S520, a self-attention encoder is used to perform self-attention encoding on the image block sequence to obtain a first feature map sequence, wherein the first feature map sequence includes a plurality of feature maps respectively for the plurality of image blocks.

[0078] In operation S530, a global encoding network is used to extract global features of the first feature map sequence to obtain a global feature map.

[0079] In operation S540, a first local encoding network is used to extract a first local feature of the first feature map sequence to obtain a first local feature map.

[0080] Operations S510 to S540 are similar to operations S210 to S240 described above, respectively, and are not described again herein.

[0081] In operation S550, first classification information is determined based on the global feature map, and second classification information is determined based on the first local feature map.

[0082] According to an embodiment of the present disclosure, a prediction model composed of a fully connected layer and a Softmax classifier can be used to process the global feature map and the first local feature map respectively, thereby obtaining first classification information and second classification information respectively. The first classification information and the second classification information are similar to the classification information of the image to be classified described above, and will not be repeated here.

[0083] In operation S560, an image classification model is trained according to a first difference between the first classification information and the second classification information.

[0084] According to an embodiment of the present disclosure, the first difference can be represented by the Euclidean distance, Manhattan distance, cosine distance, etc. between the first classification information and the second classification information, or by the relative entropy (i.e., Kullback-Leibler divergence, KL divergence) between the first classification information and the second classification information, which is not limited by the present disclosure. This embodiment can use a back propagation algorithm to train an image classification model by minimizing the first difference.

[0085] This embodiment essentially uses an imitation learning method to train the image classification model, which can make the features output by the global network and the first local network in the image classification model tend to be consistent. The distance between the global feature and the first local feature is shortened. In this way, unsupervised training of the image classification model can be achieved, which is conducive to improving the robustness of the model.

[0086] Figure 6 It is a schematic diagram of the principle of the training method of the image classification model according to an embodiment of the present disclosure.

[0087] According to an embodiment of the present disclosure, similar to the embodiment 400 described above, in this embodiment 600, the image classification model may include a second local encoding network in addition to the self-attention encoder, the global encoding network, and the first local encoding network.

[0088] Accordingly, in the embodiment 600, while the global feature map 604 is obtained by the aforementioned operation S530 and the first local feature map 606 is obtained by the aforementioned operation S540, a second local encoding network can also be used to extract the second local features of the first feature map sequence, thereby obtaining a second local feature map 609. The principle of obtaining the second local feature map 609 is similar to the principle of obtaining the second local feature map 409 in the aforementioned embodiment 400, and will not be repeated here.

[0089] After the global feature map 604, the first local feature map 606, and the second local feature map 609, the embodiment can input the three feature maps into the prediction model 660 including the fully connected layer and the Softmax classifier, respectively, to obtain the first classification information 671, the second classification information 672, and the third classification information 673. After obtaining the three classification information, the image classification model can be trained according to the differences between the three classification information. For example, the KL divergence between the first classification information 671 and the second classification information 672 can be used as the first difference 681. The KL divergence between the first classification information 671 and the third classification information 673 is used as the second difference 682. The KL divergence between the second classification information 672 and the third classification information 673 is used as the third difference 683.

[0090] Exemplarily, the KL divergence between the first classification information 671 and the second classification information 672 may include, for example, the KL divergence of the first classification information 671 relative to the second classification information 672, and the KL divergence of the second classification information 672 relative to the first classification information 671. For example, assuming that the first classification information is p1 and the second classification information is p2, the first difference D1 681 may be calculated using the following formula (1):

[0091] D1=D KL (p1||p2)+DKL (p2||p1). Formula (1)

[0092] Assuming that the sample images include N images in total, for each of the N images, a first classification information, a second classification information and a third classification information can be obtained. Assuming that the number of the plurality of predetermined categories is M, the first classification information p1 includes the i-th sample x i The probability p1 of the mth predetermined category among the M predetermined categories m (x i ), the second classification information p2 includes the i-th sample x i The probability of the mth predetermined category among M predetermined categories , then D in formula (1) KL (p2||p1) can be calculated, for example, using the following formula (2):

[0093]

[0094] Similarly, if the third classification information is set to P3, the second difference 682 can be represented by D KL (p1||p3) and D KL The third difference 683 can be obtained by adding (p3||p1). KL (p2||p3) and D KL (p3||p2) is added together.

[0095] Through the method of this embodiment, the distance between the global encoding network, the first local encoding network and the second local encoding network can be shortened, so that the three networks can learn from each other in pairs, so that the accuracy of the three networks is improved synchronously.

[0096] Figure 7 It is a schematic diagram of the principle of a training method for an image classification model according to another embodiment of the present disclosure.

[0097] According to an embodiment of the present disclosure, the first classification information and the second classification information obtained above can also be compared with the true value category information of the sample image, and the first classification information and the second classification information can be made closer to the true value category information by training the image classification model. In this way, supervised training of the image classification model can be achieved, which is conducive to improving the accuracy of the image classification model. Accordingly, the sample image should include true value category information. The true value category information may include the true value category of the sample image, and the true value category belongs to the multiple predetermined categories described above.

[0098] In one embodiment, when the image classification model also includes a second local encoding network, the embodiment can also compare the aforementioned third classification information with the true value category information of the sample image, and make the third classification information closer to the true value category information by training the image classification model.

[0099] For example, the embodiment may first determine the fourth difference between any of the first classification information, the second classification information, and the third classification information and the true value category information. Subsequently, the image classification model is trained based on the three fourth differences obtained for the three classification information. The fourth difference may be calculated using cross entropy or the like. Alternatively, the embodiment may first determine the probability value for the true value category in any of the classification information, and determine the fourth difference based on the difference between the probability value and 1.

[0100] In one embodiment, if Figure 7 As shown, when determining the fourth difference between any classification information 701 and the true value category information 702, the embodiment 700 can first determine the predetermined distribution information 703 of the sample image for multiple predetermined categories according to the true value category indicated by the true value category information 702. The predetermined distribution information can satisfy the normal distribution, for example. For example, the predetermined distribution information is set to Pt, and the predetermined distribution information includes the probability value of the sample image for each of the M predetermined categories. For example, each predetermined category can have a corresponding numerical value, and the numerical value corresponding to the i-th category is set to k. i , the probability value p of the sample image for the i-th category in the predetermined distribution information i (k i ) can be calculated using the following formula (3):

[0101]

[0102] Wherein, μ is the numerical value corresponding to the true value category of the sample image. The value of σ is 1. It can be understood that when the multiple predetermined categories are multiple age groups, the numerical value corresponding to each predetermined category can be an age value, k i The value of includes integers in the range [0, 100].

[0103] In this way, a probability value may be obtained for each predetermined category, and a plurality of probability values ​​may constitute the predetermined distribution information 703 .

[0104] After obtaining the predetermined distribution information 703, the embodiment may determine the fourth difference according to the distribution difference 704 between the predetermined distribution information 703 and any classification information 701. For example, the fourth difference 705 may be the distribution difference 704 between the predetermined distribution information 703 and any classification information 701. The distribution difference 704 is similar to the first difference, the second difference, and the third difference described above. For example, the KL divergence between the predetermined distribution information 703 and any classification information 701 may be used to represent the distribution difference 704, which is not limited in the present disclosure.

[0105] This embodiment can improve the accuracy of the determined difference by expressing the difference between the predicted classification information and the true value category information according to the distribution difference, which is conducive to improving the accuracy of the image classification model.

[0106] In one embodiment, for any classification information 701, the weighted value of the sample image can also be determined by combining the numerical values ​​corresponding to multiple predetermined categories. The fourth difference between any classification information 701 and the true value category information 702 is represented by the difference between the numerical value corresponding to the true value category and the weighted value.

[0107] Specifically, in this embodiment, the probability value for a predetermined category in the probability distribution information of a sample image for a plurality of predetermined categories included in any classification information 701 can be used as the weight of the numerical value corresponding to the predetermined category, and the weighted sum of the numerical values ​​706 of the plurality of predetermined categories can be calculated to obtain the weighted value 707. Subsequently, the value difference 709 between the weighted value 707 and the numerical value 708 corresponding to the true value category is determined, and the fourth difference 705 is determined according to the value difference 709. The value difference 709 between the weighted value 707 and the numerical value 708 can be represented by the Euclidean distance or the Manhattan distance between the two numerical values, etc., which is not limited in the present disclosure.

[0108] It can be understood that the weighted value 707 is essentially a predicted value obtained by the depth expectation method. By determining the fourth difference 705 by the method of this embodiment and training the image classification model based on the fourth difference 705, the accuracy of the image classification model can be further improved and the inconsistency between the training stage and the prediction stage of the image classification model can be reduced.

[0109] In one embodiment, when determining the fourth difference 705 , both the distribution difference 704 and the value difference 709 may be considered. For example, a weighted sum of the values ​​of the two differences may be used as the value of the fourth difference 705 .

[0110] It is understandable that for each of the first classification information, the second classification information, and the third classification information, a fourth difference can be obtained. In this embodiment, the sum of the three fourth differences obtained can be used as the prediction loss of the image classification model, and the image classification model is trained according to the prediction loss.

[0111] In one embodiment, in addition to the aforementioned self-attention encoder, global encoding network and first local encoding network, the image classification model may also include a prediction network, the input of which includes the aforementioned global feature map and the first local feature map obtained according to the sample image. The prediction network can fuse the global feature map and the first local feature map to perform prediction to obtain fourth classification information. This embodiment can also train the image classification model according to the fifth difference between the fourth classification information and the true value category information. The fifth difference is similar to the way of obtaining the fourth difference described above, and will not be repeated here. In this way, the accuracy of the prediction network can be improved to a certain extent, which is conducive to improving the accuracy of the category predicted by the image classification model.

[0112] It is understandable that when the image classification model also includes a second local encoding network, the input of the prediction network may include the second local feature map in addition to the global feature map and the first local feature map. Then the fourth classification information is predicted after fusing the global feature map, the first local feature map and the second local feature map.

[0113] In one embodiment, when training the image classification model, the first difference, the second difference, the third difference, the three fourth differences and the fifth difference may be comprehensively considered, and the weighted sum of the values ​​of these differences is used as the loss value of the image classification model. The network parameters in the image classification model are adjusted to minimize the loss value, thereby achieving the training of the image classification model.

[0114] By applying the image classification model trained by the embodiment of the present disclosure to tasks such as face age recognition and face expression recognition, the completion effect of the tasks is significantly improved. In this way, the image classification model trained by the embodiment of the present disclosure can better support the development of various businesses in the scenarios of public facility management, product recommendation, and information security management described above.

[0115] Based on the image classification method based on the image classification model provided by the present disclosure, the present disclosure also provides an image classification device based on the image classification model. Figure 8 The device is described in detail.

[0116] Figure 8 It is a structural block diagram of an image classification device based on an image classification model according to an embodiment of the present disclosure.

[0117] like Figure 8 As shown, the image classification device 800 based on the image classification model of this embodiment may include an image segmentation module 810, a self-attention encoding module 820, a global encoding module 830, a first local encoding module 840 and a classification module 850. Among them, the image classification model includes a self-attention encoder, a global encoding network, a first local encoding network and a prediction network.

[0118] The image segmentation module 810 is used to segment the image to be classified into a plurality of image blocks to obtain an image block sequence. In one embodiment, the image segmentation module 810 can be used to perform the operation S210 described above, which will not be described in detail here.

[0119] The self-attention encoding module 820 is used to perform self-attention encoding on the image block sequence using a self-attention encoder to obtain a first feature map sequence. The first feature map sequence includes multiple feature maps for multiple image blocks respectively. In one embodiment, the self-attention encoding module 820 can be used to perform the operation S220 described above, which will not be repeated here.

[0120] The global encoding module 830 is used to extract the global features of the first feature map sequence using a global encoding network to obtain a global feature map. In one embodiment, the global encoding module 830 can be used to perform the operation S230 described above, which will not be repeated here.

[0121] The first local encoding module 840 is used to extract the first local features of the first feature map sequence using the first local encoding network to obtain the first local feature map. In one embodiment, the first local encoding module 840 can be used to perform the operation S240 described above, which will not be repeated here.

[0122] The classification module 850 is used to input the global feature map and the first local feature map into the prediction network to obtain classification information of the image to be classified. In one embodiment, the classification module 850 can be used to perform the operation S250 described above, which will not be repeated here.

[0123] According to an embodiment of the present disclosure, the global encoding network includes a first encoding subnetwork and a first conversion subnetwork. The above-mentioned global encoding module 830 may include a first encoding submodule and a first conversion submodule. The first encoding submodule is used to obtain a second feature map sequence based on the first feature map sequence using the first encoding subnetwork. The first conversion submodule is used to convert the second feature map sequence into a feature map matrix using the first conversion subnetwork to obtain a global feature map.

[0124] According to an embodiment of the present disclosure, the global encoding network also includes a first fusion subnetwork. The above-mentioned first encoding submodule may include a global fusion unit and a first encoding unit. The global fusion unit is used to use the first fusion subnetwork to globally fuse the feature maps in the first feature map sequence to obtain a third feature map sequence. The first encoding unit is used to input the third feature map sequence into the first encoding subnetwork to obtain a second feature map sequence.

[0125] According to an embodiment of the present disclosure, the first fusion subnetwork includes a first conversion layer, a first downsampling layer, and a second conversion layer. The above-mentioned global fusion unit includes a first conversion subunit, a sampling subunit, and a second conversion subunit. The first conversion subunit is used to convert the first feature map sequence into a first feature map matrix using the first conversion layer. The sampling subunit is used to sample the feature map intervals in the first feature map matrix using the first downsampling layer to obtain multiple fused sub-matrices. The second conversion subunit is used to convert the multiple fused sub-matrices into multiple fused feature maps using the second conversion layer to obtain a third feature map sequence composed of multiple fused feature maps.

[0126] According to an embodiment of the present disclosure, the first local coding network includes a second fusion subnetwork, a second coding subnetwork and a second conversion subnetwork. The above-mentioned first local coding module includes a first fusion submodule, a second coding submodule and a second conversion submodule. The first fusion submodule is used to fuse the feature graphs of the first part of the first feature graph sequence using the second fusion subnetwork to obtain a fourth feature graph sequence. The second coding submodule is used to input the fourth feature graph sequence into the second coding subnetwork to obtain a fifth feature graph sequence. The second conversion submodule is used to convert the fifth feature graph sequence into a feature graph matrix using the second conversion subnetwork to obtain a first local feature graph.

[0127] According to an embodiment of the present disclosure, the second fusion subnetwork includes a third conversion layer, a first pooling layer and a fourth conversion layer. The above-mentioned first fusion submodule includes a first conversion unit, a first pooling unit and a second conversion unit. The first conversion unit is used to convert the first feature map sequence into a first feature map matrix using the third conversion layer. The first pooling unit is used to perform a pooling operation on the feature map of the first predetermined area in the intercepted first feature map matrix using the first pooling layer to obtain a sixth feature map matrix. The second conversion unit is used to convert the sixth feature map matrix into a fourth feature map sequence using the fourth conversion layer.

[0128] According to an embodiment of the present disclosure, the image classification model also includes a second local encoding network. The above-mentioned device 800 may also include a second local encoding module, which is used to extract the second local feature of the first feature map sequence using the second local encoding network to obtain a second local feature map. The above-mentioned classification module 850 can also be used to input the global feature map, the first local feature map and the second local feature map into the prediction network to obtain classification information of the image to be classified. Wherein, the size of the first local feature is larger than the size of the second local feature.

[0129] According to an embodiment of the present disclosure, the second local coding network includes a third fusion subnetwork, a third encoding subnetwork and a third conversion subnetwork. The above-mentioned second local coding network may include a second fusion submodule, a third encoding submodule and a third conversion submodule. The second fusion submodule is used to fuse the feature graphs of the second part of the first feature graph sequence using the third fusion subnetwork to obtain a seventh feature graph sequence. The third encoding submodule is used to input the seventh feature graph sequence into the third encoding subnetwork to obtain a sixth feature graph sequence. The third conversion submodule is used to convert the sixth feature graph sequence into a feature graph matrix using the third conversion subnetwork to obtain a second local feature graph.

[0130] According to an embodiment of the present disclosure, the third fusion subnetwork includes a fifth conversion layer, a second pooling layer, and a sixth conversion layer. The above-mentioned second fusion submodule may include a third conversion unit, a second pooling unit, and a fourth conversion unit. The third conversion unit is used to convert the first feature map sequence into a first feature map matrix using the fifth conversion layer. The second pooling unit is used to perform a pooling operation on the feature map of the second predetermined area in the intercepted first feature map matrix using the second pooling layer to obtain an eighth feature map matrix. The fourth conversion unit is used to convert the eighth feature map matrix into a seventh feature map sequence using the sixth conversion layer.

[0131] Based on the training method of the image classification model provided by the present disclosure, the present disclosure also provides a training device for the image classification model. Fig. 9 The device is described in detail.

[0132] Fig. 9 It is a structural block diagram of a training device for an image classification model according to an embodiment of the present disclosure.

[0133] like Fig. 9 As shown, the training device 900 of the image classification model of this embodiment may include an image segmentation module 910, a self-attention encoding module 920, a global encoding module 930, a first local encoding module 940, a first classification module 950 and a model training module 960. Among them, the image classification model includes a self-attention encoder, a global encoding network, and a first local encoding network.

[0134] The image segmentation module 910 may be used to segment the sample image into a plurality of image blocks to obtain an image block sequence. In one embodiment, the image segmentation module 910 is used to perform the operation S510 described above, which will not be described in detail herein.

[0135] The self-attention encoding module 920 is used to perform self-attention encoding on the image block sequence using a self-attention encoder to obtain a first feature map sequence; the first feature map sequence includes multiple feature maps for multiple image blocks respectively. In one embodiment, the self-attention encoding module 920 can perform the operation S520 described above, which will not be repeated here.

[0136] The global encoding module 930 is used to extract the global features of the first feature map sequence using a global encoding network to obtain a global feature map. In one embodiment, the global encoding module 930 may perform the operation S530 described above, which will not be repeated here.

[0137] The first local encoding module 940 is used to extract the first local features of the first feature map sequence using the first local encoding network to obtain the first local feature map. In one embodiment, the first local encoding module 940 can perform the operation S540 described above, which will not be repeated here.

[0138] The first classification module 950 is used to determine the first classification information according to the global feature map, and determine the second classification information according to the first local feature map. In one embodiment, the first classification module 950 may perform the operation S550 described above, which will not be described in detail here.

[0139] The model training module 960 is used to train the image classification model according to the first difference between the first classification information and the second classification information. In one embodiment, the model training module 960 can perform the operation S560 described above, which will not be repeated here.

[0140] According to an embodiment of the present disclosure, the image classification model also includes a second local coding network. The above-mentioned device 900 may also include a second local coding module, which is used to extract the second local features of the first feature map sequence using the second local coding network to obtain a second local feature map. The above-mentioned first classification module 950 is also used to determine the third classification information based on the second local feature map. The above-mentioned model training module 960 can also be used to train the image classification model based on the second difference between the first classification information and the third classification information and the third difference between the second classification information and the third classification information.

[0141] According to an embodiment of the present disclosure, the sample image includes true value category information. The above-mentioned device 900 may also include a first difference determination module, which is used to determine the fourth difference between any classification information and the true value category information for any classification information among the first classification information, the second classification information and the third classification information. The above-mentioned model training module 960 may also be used to train the image classification model according to the fourth difference between the first classification information, the second classification information and the third classification information and the true value category information.

[0142] According to an embodiment of the present disclosure, any classification information includes probability distribution information of a sample image for a plurality of predetermined categories, and the true value category information includes the true value category to which the sample image belongs. The above-mentioned first difference determination module may include a distribution determination submodule and a first difference determination submodule. The distribution determination submodule is used to determine predetermined distribution information of the sample image for a plurality of predetermined categories according to the true value category. The difference determination submodule is used to determine a fourth difference between any classification information and the true value category information according to the distribution difference between the predetermined distribution information and the probability distribution information.

[0143] According to an embodiment of the present disclosure, any classification information includes probability distribution information of a sample image for a plurality of predetermined categories. Each of the plurality of predetermined categories has a corresponding numerical value, and the true value category information includes the true value category to which the sample image belongs. The above-mentioned first difference determination module may include a value determination submodule and a second difference determination submodule. The value determination submodule is used to determine the weighted sum of a plurality of numerical values ​​corresponding to the plurality of predetermined categories according to the probability distribution information as a weighted value. The second difference determination submodule is used to determine the fourth difference between any classification information and the true value category information according to the value difference between the numerical value corresponding to the true value category and the weighted value.

[0144] According to an embodiment of the present disclosure, the sample image includes true value category information, and the image classification model also includes a prediction network. The above-mentioned device 900 may also include a second classification module and a second difference determination module. The second classification module is used to input the global feature map and the first local feature map into the prediction network to obtain fourth classification information. The second difference determination module is used to determine the fifth difference between the fourth classification information and the true value category information. The above-mentioned model training module 960 is also used to train the image classification model according to the fifth difference.

[0145] It should be noted that the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals. In the technical solution of this disclosure, the user's authorization or consent is obtained before obtaining or collecting user personal information.

[0146] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0147] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an image classification method based on an image classification model and / or a training method for an image classification model of an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0148] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0149] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0150] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as an image classification method based on an image classification model and / or a training method for an image classification model. For example, in some embodiments, an image classification method based on an image classification model and / or a training method for an image classification model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into RAM 1003 and executed by computing unit 1001, one or more steps of the image classification method based on the image classification model and / or the training method of the image classification model described above may be performed. Alternatively, in other embodiments, computing unit 1001 may be configured to perform the image classification method based on the image classification model and / or the training method of the image classification model by any other appropriate means (e.g., by means of firmware).

[0151] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0152] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0153] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0154] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0155] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0156] A computer system may include a client and a server. The client and the server are generally far away from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. Among them, the server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server can also be a server of a distributed system, or a server combined with a blockchain.

[0157] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0158] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image classification method based on an image classification model, wherein: The image classification model includes a self-attention encoder, a global encoding network, a first local encoding network and a prediction network; the method includes: The image to be classified is divided into multiple image blocks to obtain an image block sequence; Using the self-attention encoder to perform self-attention encoding on the image block sequence to obtain a first feature map sequence; the first feature map sequence includes a plurality of feature maps respectively for the plurality of image blocks; Extracting global features of the first feature map sequence using the global encoding network to obtain a global feature map; extracting a first local feature of the first feature map sequence using the first local encoding network to obtain a first local feature map; and Inputting the global feature map and the first local feature map into the prediction network to obtain classification information of the image to be classified; The obtaining of the global feature map comprises: grouping the plurality of feature maps in the first feature map sequence, calculating a weighted sum of at least two feature maps in each group, and obtaining the global feature map according to the weighted feature maps of each group; Obtaining the first local feature map includes: fusing a first portion of feature maps in the first feature map sequence, and determining the first local feature map according to the fused feature maps, wherein the first portion of feature maps is a feature map of an image block in a central area of ​​the image to be classified.

2. The method according to claim 1, wherein: The global encoding network includes a first encoding subnetwork and a first conversion subnetwork; the extracting the global features of the first feature map sequence by using the global encoding network to obtain the global feature map includes: Based on the first feature graph sequence, using the first encoding subnetwork to obtain a second feature graph sequence; and The first conversion subnetwork is used to convert the second feature map sequence into a feature map matrix to obtain the global feature map.

3. The method according to claim 2, wherein: The global encoding network further includes a first fusion subnetwork; and based on the first feature graph sequence, obtaining a second feature graph sequence using the first encoding subnetwork includes: Using the first fusion sub-network to globally fuse the feature maps in the first feature map sequence to obtain a third feature map sequence; and The third feature map sequence is input into the first encoding sub-network to obtain the second feature map sequence.

4. The method according to claim 3, wherein: The first fusion subnetwork includes a first conversion layer, a first downsampling layer, and a second conversion layer; the first fusion subnetwork is used to globally fuse the feature maps in the first feature map sequence to obtain a third feature map sequence, including: Using the first conversion layer to convert the first feature map sequence into a first feature map matrix; Using the first downsampling layer to perform interval sampling on the feature maps in the first feature map matrix to obtain a plurality of fused sub-matrices; and The second conversion layer is used to convert the multiple fused sub-matrices into multiple fused feature maps to obtain a third feature map sequence composed of the multiple fused feature maps.

5. The method according to claim 1, wherein: The first local encoding network includes a second fusion subnetwork, a second encoding subnetwork, and a second transformation subnetwork; the first local feature map sequence is extracted using the first local encoding network to obtain the first local feature map, including: Using the second fusion sub-network to fuse the feature maps of the first part of the first feature map sequence, to obtain a fourth feature map sequence; Inputting the fourth feature map sequence into the second encoding subnetwork to obtain a fifth feature map sequence; and The second conversion sub-network is used to convert the fifth feature map sequence into a feature map matrix to obtain the first local feature map.

6. The method according to claim 5, wherein: The second fusion sub-network includes a third conversion layer, a first pooling layer and a fourth conversion layer; the fourth feature map sequence obtained by fusing the first part of the feature maps in the first feature map sequence using the second fusion sub-network includes: Using the third conversion layer to convert the first feature map sequence into a first feature map matrix; Using the first pooling layer to perform a pooling operation on the feature map of the first predetermined area in the intercepted first feature map matrix to obtain a sixth feature map matrix; and The sixth feature map matrix is ​​converted into the fourth feature map sequence using the fourth conversion layer.

7. The method according to claim 5, wherein: The image classification model also includes a second local encoding network; the method also includes: extracting second local features of the first feature map sequence using the second local encoding network to obtain a second local feature map; and Inputting the global feature map, the first local feature map and the second local feature map into the prediction network to obtain classification information of the image to be classified, Wherein, the size of the first local feature is greater than the size of the second local feature.

8. The method according to claim 7, wherein: The second local encoding network includes a third fusion subnetwork, a third encoding subnetwork and a third transformation subnetwork; the second local encoding network is used to extract the second local feature of the first feature map sequence to obtain the second local feature map, including: Using the third fusion sub-network to fuse the feature maps of the second part of the first feature map sequence, to obtain a seventh feature map sequence; Inputting the seventh feature map sequence into the third encoding subnetwork to obtain a sixth feature map sequence; and The third conversion sub-network is used to convert the sixth feature map sequence into a feature map matrix to obtain the second local feature map.

9. The method according to claim 8, wherein: The third fusion sub-network includes a fifth conversion layer, a second pooling layer and a sixth conversion layer; The step of fusing the feature graphs of the second part of the first feature graph sequence by the third fusion sub-network to obtain a seventh feature graph sequence includes: Using the fifth conversion layer to convert the first feature map sequence into a first feature map matrix; Using the second pooling layer to perform a pooling operation on the feature map of the second predetermined area in the intercepted first feature map matrix to obtain an eighth feature map matrix; and The eighth feature map matrix is ​​converted into the seventh feature map sequence using the sixth conversion layer.

10. A method for training an image classification model, wherein: The image classification model includes a self-attention encoder, a global encoding network, and a first local encoding network; the method includes: The sample image is divided into multiple image blocks to obtain an image block sequence; Using the self-attention encoder to perform self-attention encoding on the image block sequence to obtain a first feature map sequence; the first feature map sequence includes a plurality of feature maps respectively for the plurality of image blocks; Extracting global features of the first feature map sequence using the global encoding network to obtain a global feature map; Extracting a first local feature of the first feature map sequence using the first local encoding network to obtain a first local feature map; Determining first classification information according to the global feature map, and determining second classification information according to the first local feature map; and training the image classification model according to a first difference between the first classification information and the second classification information; The obtaining of the global feature map comprises: grouping the plurality of feature maps in the first feature map sequence, calculating a weighted sum of at least two feature maps in each group, and obtaining the global feature map according to the weighted feature maps of each group; Obtaining the first local feature map includes: fusing the first part of the feature maps in the first feature map sequence, and determining the first local feature map according to the fused feature maps, wherein the first part of the feature map is a feature map of an image block in a central area of ​​the sample image.

11. The method according to claim 10, wherein: The image classification model also includes a second local encoding network; the method also includes: Extracting second local features of the first feature map sequence using the second local encoding network to obtain a second local feature map; Determining third classification information according to the second local feature map; and The image classification model is trained according to a second difference between the first classification information and the third classification information and a third difference between the second classification information and the third classification information.

12. The method according to claim 11, wherein: The sample image includes true value category information; the method further includes: For any one of the first classification information, the second classification information, and the third classification information, determine a fourth difference between the any one of the classification information and the true value category information; and The image classification model is trained according to a fourth difference between each of the first classification information, the second classification information, and the third classification information and the true value category information.

13. The method according to claim 12, wherein: The any classification information includes probability distribution information of the sample image for a plurality of predetermined categories; The true value category information includes the true value category to which the sample image belongs; Determining a fourth difference between the any classification information and the true value category information includes: Determining, according to the true value category, predetermined distribution information of the sample image for the plurality of predetermined categories; as well as According to the distribution difference between the predetermined distribution information and the probability distribution information, a fourth difference between any one of the classification information and the true value category information is determined.

14. The method according to claim 12, wherein: The any classification information includes probability distribution information of the sample image for a plurality of predetermined categories; each of the plurality of predetermined categories has a corresponding numerical value; the true value category information includes the true value category to which the sample image belongs; Determining a fourth difference between the any classification information and the true value category information includes: Determine, according to the probability distribution information, a weighted sum of multiple numerical values ​​corresponding to the multiple predetermined categories as a weighted value; as well as A fourth difference between any one of the classification information and the true value category information is determined based on a difference between the numerical value corresponding to the true value category and the weighted value.

15. The method according to any one of claims 10 to 14, wherein The sample image includes true value category information; the image classification model also includes a prediction network; the method also includes: Inputting the global feature map and the first local feature map into the prediction network to obtain fourth classification information; determining a fifth difference between the fourth classification information and the true value classification information; and The image classification model is trained based on the fifth difference.

16. An image classification device based on an image classification model, wherein: The image classification model includes a self-attention encoder, a global encoding network, a first local encoding network and a prediction network; the device includes: An image segmentation module is used to segment the image to be classified into multiple image blocks to obtain an image block sequence; A self-attention encoding module, configured to perform self-attention encoding on the image block sequence using the self-attention encoder to obtain a first feature map sequence; the first feature map sequence includes a plurality of feature maps respectively for the plurality of image blocks; A global encoding module, used for extracting global features of the first feature map sequence using the global encoding network to obtain a global feature map; a local encoding module, configured to extract a first local feature of the first feature map sequence using the first local encoding network to obtain a first local feature map; and A classification module, used for inputting the global feature map and the first local feature map into the prediction network to obtain classification information of the image to be classified; The global encoding module is further used to group multiple feature maps in the first feature map sequence, calculate a weighted sum of at least two feature maps in each group, and obtain the global feature map according to each group of weighted feature maps; The local encoding module is also used to fuse the first part of the feature maps in the first feature map sequence, and determine the first local feature map according to the fused feature maps, wherein the first part of the feature map is the feature map of the image block in the central area of ​​the image to be classified.

17. A training device for an image classification model, wherein: The image classification model includes a self-attention encoder, a global encoding network, and a first local encoding network; the device includes: An image segmentation module is used to segment a sample image into multiple image blocks to obtain an image block sequence; A self-attention encoding module, configured to perform self-attention encoding on the image block sequence using the self-attention encoder to obtain a first feature map sequence; the first feature map sequence includes a plurality of feature maps respectively for the plurality of image blocks; A global encoding module, used for extracting global features of the first feature map sequence using the global encoding network to obtain a global feature map; A local encoding module, configured to extract a first local feature of the first feature map sequence using the first local encoding network to obtain a first local feature map; a classification module, configured to determine first classification information according to the global feature map, and determine second classification information according to the first local feature map; and a model training module, configured to train the image classification model according to a first difference between the first classification information and the second classification information; The global encoding module is further used to group multiple feature maps in the first feature map sequence, calculate a weighted sum of at least two feature maps in each group, and obtain the global feature map according to each group of weighted feature maps; The local encoding module is also used to fuse the first part of the feature maps in the first feature map sequence, and determine the first local feature map according to the fused feature maps, wherein the first part of the feature map is the feature map of the image block in the central area of ​​the sample image.

18. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of claims 1 to 15.

19. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 15.

20. A computer program product comprising a computer program / instructions, which when executed by a processor implement the steps of the method according to any one of claims 1 to 15.

Images