Image classification method and device, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to the field of artificial intelligence, and relates to an image classification method, which comprises the following steps: acquiring a target image, and generating a target feature vector of the target image; inputting the target feature vector into an encoder in an image classification model to obtain a hidden layer feature state output by each encoding layer in the encoder, and adding Gaussian blur calculation to each encoding layer; adding a weight to the hidden layer feature state of each coding layer through a layer attention mechanism, and generating a first feature matrix according to the hidden layer feature state with the weight of each coding layer; matrix weighting operation is carried out on the first feature matrix, pooling operation is carried out after matrix weighting, and a classification vector is obtained; and inputting the classification vector into a classifier in the image classification model to obtain an image category of the target image. The invention further provides an image classification device, computer equipment and a storage medium. According to the invention, the accuracy of image classification is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image classification method, apparatus, computer equipment and storage medium. Background Art

[0002] Existing image classification technologies mainly rely on deep learning models, especially convolutional neural networks (CNN) and transformer-based models, and have made significant progress in image classification tasks. For example, in the field of medical image classification, tongue coating images are automatically classified to determine the type of disease, or medical images are classified to determine whether there is a tumor or a mass.

[0003] Due to the scarcity and complexity of image data in certain fields (such as the medical field), existing image classification technologies still have some shortcomings in practical applications. First, convolutional neural networks such as VGG and ResNet may lose important features when processing complex images, resulting in low classification accuracy. The converter model performs better in image feature extraction and classification due to its global self-attention mechanism, but its stacked structure easily leads to the neglect of early-level features by deep-level features, thereby losing important information. In addition, image data in the medical field is often highly sparse and specific. When faced with images that do not appear in the training set, existing technologies are prone to overfitting, resulting in poor generalization ability and inability to effectively process new images. These problems all affect the accuracy of image classification. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image classification method, apparatus, computer device and storage medium to solve the problem of low image classification accuracy.

[0005] In order to solve the above technical problems, the present application embodiment provides an image classification method, which adopts the following technical solution:

[0006] Acquire a target image, and generate a target feature vector of the target image;

[0007] Inputting the target feature vector into an encoder in an image classification model to obtain a hidden layer feature state output by each encoding layer in the encoder, wherein each encoding layer is added with a Gaussian blur calculation;

[0008] Adding weights to the hidden layer feature states of each coding layer through a layer attention mechanism, and generating a first feature matrix according to the hidden layer feature states with weights of each coding layer;

[0009] Performing a matrix weighting operation on the first feature matrix, and performing a pooling operation after the matrix weighting to obtain a classification vector;

[0010] The classification vector is input into the classifier in the image classification model to obtain the image category of the target image.

[0011] In order to solve the above technical problems, the embodiment of the present application further provides an image classification device, which adopts the following technical solution:

[0012] An image acquisition module, used to acquire a target image and generate a target feature vector of the target image;

[0013] A vector input module, used for inputting the target feature vector into an encoder in an image classification model to obtain a hidden layer feature state output by each encoding layer in the encoder, wherein each encoding layer is added with Gaussian blur calculation;

[0014] A weight adding module, used for adding weights to the hidden layer feature states of each coding layer through a layer attention mechanism, and generating a first feature matrix according to the hidden layer feature states with weights of each coding layer;

[0015] A weighted pooling module, used for performing a matrix weighting operation on the first feature matrix, and performing a pooling operation after the matrix weighting to obtain a classification vector;

[0016] The image classification module is used to input the classification vector into the classifier in the image classification model to obtain the image category of the target image.

[0017] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0018] Acquire a target image, and generate a target feature vector of the target image;

[0019] Inputting the target feature vector into an encoder in an image classification model to obtain a hidden layer feature state output by each encoding layer in the encoder, wherein each encoding layer is added with a Gaussian blur calculation;

[0020] Adding weights to the hidden layer feature states of each coding layer through a layer attention mechanism, and generating a first feature matrix according to the hidden layer feature states with weights of each coding layer;

[0021] Performing a matrix weighting operation on the first feature matrix, and performing a pooling operation after the matrix weighting to obtain a classification vector;

[0022] The classification vector is input into the classifier in the image classification model to obtain the image category of the target image.

[0023] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0024] Acquire a target image, and generate a target feature vector of the target image;

[0025] Inputting the target feature vector into an encoder in an image classification model to obtain a hidden layer feature state output by each encoding layer in the encoder, wherein each encoding layer is added with a Gaussian blur calculation;

[0026] Adding weights to the hidden layer feature states of each coding layer through a layer attention mechanism, and generating a first feature matrix according to the hidden layer feature states with weights of each coding layer;

[0027] Performing a matrix weighting operation on the first feature matrix, and performing a pooling operation after the matrix weighting to obtain a classification vector;

[0028] The classification vector is input into the classifier in the image classification model to obtain the image category of the target image.

[0029] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: when performing image classification, the model's ability to capture the correlation between image blocks is enhanced through Gaussian blur and layer attention mechanism; by adding Gaussian blur calculation to the multi-layer structure of the encoder, the model improves the generalization ability of unseen image data and reduces the overfitting problem; the layer attention mechanism enables the model to dynamically assign different weights to the hidden feature states of each layer, avoiding the problem of information loss in too deep layers in traditional stacked models, and ensuring that both shallow and deep features can contribute to the final classification; pooling and matrix weighting operations further extract the most significant feature information, so that the classifier can accurately determine the image category. This hierarchical feature processing improves the accuracy of classification, and can improve the accuracy of classification when classifying complex and scarce images. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0032] Figure 2 is a flow chart of an embodiment of an image classification method according to the present application;

[0033] Figure 3 is a schematic structural diagram of an embodiment of an image classification device according to the present application;

[0034] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0036] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0037] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0038] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used as a medium for providing a communication link between the terminal device 101 and the server 103. The user may use the terminal device 101 to interact with the server 103 through the network 102, for example, to send a target image to the server 103, which classifies the image, and to obtain and view the image category returned by the server 103 through the terminal device 101.

[0039] It should be noted that the image classification method provided in the embodiment of the present application is generally executed by a server, and accordingly, the image classification device is generally set in the server.

[0040] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0041] Continue to refer Figure 2 , shows a flow chart of an embodiment of an image classification method according to the present application. The image classification method comprises the following steps:

[0042] Step S201, acquiring a target image and generating a target feature vector of the target image.

[0043] In this embodiment, the electronic device (eg, Figure 1 The server shown in the figure) can communicate with the terminal device through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0044] Specifically, the server obtains a target image to be classified. In practical applications, the target image can be an image in the medical field or other images that need to be classified. In one embodiment, the target image can be a medical image in the medical field, such as a patient's tongue coating picture. The doctor can upload the patient's medical image to the server via a laptop computer, and the server automatically classifies the image. The image category obtained can be used for medical diagnosis and treatment, thereby improving the efficiency of the consultation. The target image is encoded into a target feature vector, which represents the basic visual information of the image, including pixel information and position features to ensure that the spatial relationship of the image blocks is preserved (when generating the target feature vector, the target image will be cut into image blocks first).

[0045] Step S202, inputting the target feature vector into the encoder in the image classification model, obtaining the hidden layer feature state output by each encoding layer in the encoder, wherein each encoding layer is added with Gaussian blur calculation.

[0046] Specifically, the target feature vector is input into the image classification model, which can be built based on the transformer model (transformer network). The transformer model is a neural network model based on the self-attention mechanism for processing sequence data. It is essentially an Encoder-Decoder architecture, in which the encoding component consists of multiple layers of encoders (Encoder) and the decoding component consists of decoders (Decoder) with the same number of layers. The transformer has an encoder encoder, and the encoder can have multiple encoding layers. In one embodiment, the encoder has a total of 12 encoding layers.

[0047] Each encoding layer processes the input feature vector through the self-attention mechanism. In order to improve the generalization ability of the model, Gaussian blur calculation is added to the encoding layer to specifically blur the V (Value) vector in the attention mechanism to better capture the association between image blocks.

[0048] Step S203, adding weights to the hidden layer feature states of each coding layer through a layer attention mechanism, and generating a first feature matrix according to the hidden layer feature states with weights of each coding layer.

[0049] Specifically, the model uses a layer attention mechanism to dynamically add weights to the output of each encoding layer. By calculating the contribution of different layers to the classification, different weights are assigned to the hidden features of these layers, while ensuring that shallow features are not ignored. These weighted hidden feature states are combined into the first feature matrix, which represents the importance and interaction of features in each layer.

[0050] Step S204, performing a matrix weighting operation on the first feature matrix, and performing a pooling operation after the matrix weighting to obtain a classification vector.

[0051] Specifically, a weighted operation is performed on each row in the first feature matrix, that is, the inter-layer feature matrices are superimposed. The pooling operation is used to compress the feature dimension, retain the most significant features, and finally obtain a classification vector, which is the key feature expression used to determine the image category.

[0052] Step S205, inputting the classification vector into the classifier in the image classification model to obtain the image category of the target image.

[0053] Specifically, the classification vector is input into the classifier in the image classification model. The classifier calculates the probability of each category through the Softmax function and outputs the category to which the image belongs. The model uses the previously processed feature information to generate the final classification result of the target image. When the target image is a medical image, the image category can be the patient's disease category or the patient's disease feature, such as identifying tumors or masses in the target image, thereby realizing automatic judgment of the disease.

[0054] It can be understood that the present application can be applied to any field that requires image classification. For example, in the field of monitoring, the monitoring image can be input into the server as the target image, and the server automatically monitors whether there are suspicious target objects (for example, whether there are extra people in certain restricted areas), or whether the target object exhibits specific behaviors (for example, whether the vehicle is speeding or driving illegally), and generates monitoring results based on the image classification results.

[0055] In this embodiment, when performing image classification, the model's ability to capture the correlation between image blocks is enhanced through Gaussian blur and layer attention mechanisms; Gaussian blur calculations are added to the multi-layer structure of the encoder, and the model improves its generalization ability for unseen image data and reduces overfitting problems; the layer attention mechanism enables the model to dynamically assign different weights to the hidden feature states of each layer, avoiding the problem of information loss in too deep layers in traditional stacked models, and ensuring that both shallow and deep features can contribute to the final classification; pooling and matrix weighting operations further extract the most significant feature information, so that the classifier can accurately determine the image category. This hierarchical feature processing improves the accuracy of classification, and can improve the accuracy of classification when classifying complex and scarce images.

[0056] Furthermore, the above-mentioned step of generating a target feature vector of a target image may include: adjusting the target image to a preset size to obtain a first image; cutting the first image into a preset number of image blocks; performing convolution processing on each image block to obtain a first block feature vector of each image block; obtaining a position feature vector of each image block, and adding the first block feature vector of each image block and its corresponding position feature vector to obtain a second block feature vector; and determining the second block feature vector of each image block as the target feature vector of the target image.

[0057] Specifically, the target image is first adjusted to a fixed preset size, such as enlarging or reducing the target image to a 256*256 image size, so as to unify the format of the input data, ensure the consistency of subsequent processing, and avoid unstable feature extraction due to different image sizes.

[0058] The first image is cut into a preset number of image blocks of equal size, for example, into non-overlapping grid blocks, each of which is 64*64 in size, resulting in a total of 16 image blocks. This cutting method can help the model focus on local features, so that different areas of the image can be analyzed separately, improving the classification effect.

[0059] Apply convolution operation to each image block to extract local features. By sliding the convolution kernel (size can be 3*3), important information such as edges and textures can be captured to generate the first block feature vectors, which represent the feature description of each image block. Among them, each first block feature vector has a dimension of d, and a total of 16 first block feature vectors with a dimension of d are obtained.

[0060] Each image block has a corresponding position feature vector, which is used to encode the position information of the block in the overall image. The dimension of the position feature vector is also d. The first block feature vector is added to its corresponding position feature vector to generate the second block feature vector, ensuring that the model also considers the spatial structure when extracting features.

[0061] The second feature vectors of all image blocks are integrated to form the target feature vector of the target image, which represents the overall feature information of the image, including local features and location information. The target feature vector is used as the final input embedding and will be input into the image classification model.

[0062] In one embodiment, the position feature vector of the image block is obtained through model learning. The position feature vector comes from a 16*d learnable weight matrix (16 represents 16 image blocks). The initial value in this matrix is ​​random, and each row in the matrix represents the feature of the current position. The dimension of each row is d, which is consistent with the dimension of the block feature vector of each block. The values ​​in this matrix will be updated as the model is trained, and a position feature vector with a reasonable representation will be obtained after the training is completed.

[0063] In this embodiment, the target image is structured to improve the quality of image feature extraction; the image is adjusted to a preset size and cut into multiple image blocks, which can focus on local features and effectively capture subtle changes in different areas; convolution processing generates a high-dimensional feature vector for each image block, extracts important visual information, and the introduction of position feature vectors can not only obtain local features, but also maintain the spatial structure of the image and reduce feature loss; the second feature vector combines the convolution features and position information to ensure the comprehensiveness and accuracy of the target feature vector.

[0064] Furthermore, the step of inputting the target feature vector into the encoder in the image classification model to obtain the hidden layer feature state output by each coding layer in the encoder may include: inputting the target feature vector into the first coding layer of the encoder in the image classification model, and the first coding layer outputs the hidden layer feature state based on Gaussian blur calculation; inputting the hidden layer feature state output by the first coding layer into the second coding layer of the encoder for iteration until the last coding layer of the encoder, to obtain the hidden layer feature state output by each coding layer in the encoder.

[0065] Specifically, the target feature vector is first input into the first encoding layer of the encoder of the image classification model, which is the starting point of feature extraction. The target feature vector contains image information processed by convolution and position encoding and has strong semantic features.

[0066] The first encoding layer processes the input target feature vector and applies Gaussian blur calculation to enhance the expressiveness of the feature. Gaussian blur enables the input feature to be better associated with the surrounding features, reduces the impact of noise, and outputs the hidden layer feature state, which represents the information processed by the first encoding layer and retains important local features.

[0067] The hidden layer feature state output by the first encoding layer is used as input and passed to the second encoding layer of the encoder. The processing of each subsequent encoding layer is based on the output of the previous encoding layer. By stacking multiple layers, the model can gradually extract deeper features. This process is repeated until the last encoding layer.

[0068] Each encoding layer will output its own hidden layer feature state. These states contain feature information after abstraction at different levels, which is the key data for subsequent classification tasks. The hidden layer feature state of each encoding layer is 16*d.

[0069] In this embodiment, the target feature vector is processed layer by layer in the encoder, which improves the depth and accuracy of feature extraction; the Gaussian blur calculation is introduced, so that the encoding layer can effectively reduce the noise of the input features and enhance the correlation with adjacent features, ensuring that the feature state of each layer can better reflect the real information of the image; through layer-by-layer iteration, the gradual abstraction from local features to global features is captured, and the outputs of different encoding layers gradually accumulate and fuse multi-level information, avoiding the loss of too deep feature information in traditional models, enhancing the model's ability to understand complex images, and improving image classification accuracy and the generalization ability of the model.

[0070] Furthermore, the output of the multi-head attention mechanism in each encoding layer is expressed as:

[0071]

[0072] in, represents the output of the multi-head attention mechanism, h represents the hth head, Q h , K h and V h They represent the query representation, key representation, and value representation of the target feature vector in the hth head, respectively. σ is the standard deviation of the standard Gaussian distribution. k is the fixed window size in Gaussian blur. x is the input. When Gaussian blur calculation is introduced, the value representation in each head is used as the input of Gaussian blur. V h *g represents Gaussian blur calculation on the value representation.

[0073] In one embodiment, each encoding layer uses Gaussian blur to calculate the V value when performing multi-head attention calculation to increase the generalization ability of the model.

[0074] In this embodiment, the generalization ability of the model is enhanced by performing Gaussian blur processing on the V vector in the multi-head self-attention mechanism; Gaussian blur smoothes the eigenvalues, enabling the model to capture more global feature associations, reduce the impact of local noise, and reduce the risk of model overfitting; this enables the model to more accurately extract important features when faced with unseen image data, thereby improving classification accuracy and robustness in complex tasks.

[0075] Furthermore, the above step S203 may include: for the hidden layer feature state of each coding layer, performing maximum pooling on the block hidden state of each image block in the hidden layer feature state from the image block dimension to obtain a first pooling result; inputting the first pooling result of each image block into the softmax function to obtain the weight of each image block; multiplying the weight of each image block by the corresponding hidden layer feature state to obtain a hidden layer feature state with a weight; and generating a first feature matrix according to the hidden layer feature state with a weight of each coding layer.

[0076] Specifically, as mentioned above, the first image is cut into 16 image blocks, each image block is convolved to obtain a first feature vector, and then the position feature vector is superimposed to obtain a second feature vector. The second feature vector of each image block can constitute a target feature vector of the target image 16*d.

[0077] In the hidden layer feature state output for each coding layer, the hidden layer feature data (block hidden state) corresponding to each image block can be determined. The block hidden state of each image block in the hidden layer feature state is subjected to maximum pooling from the dimension of the image block, that is, for the block hidden state of each image block, the maximum value in each dimension is taken to select the most significant feature in each block, ensuring that important information is retained and reducing the impact of redundant information. After such pooling calculation, a 1*12*16 feature vector is obtained as a whole, and each image block in each coding layer will correspond to one of the values.

[0078] The first pooling result obtained in the previous step is input into the Softmax function to calculate the attention weight of each image block in each layer. The pooling result of each image block is normalized into a probability value through Softmax to ensure that the sum of all weights is 1, thereby reflecting the importance of each image block to the final classification.

[0079] The obtained weights will be multiplied by the corresponding hidden layer feature states to obtain the weighted hidden layer feature states. Each image block in each layer has a corresponding attention weight value. This ensures that the feature state of each image block is adjusted according to its importance, so that more important features have greater influence in subsequent processing.

[0080] Finally, the weighted hidden layer feature states are integrated to generate the first feature matrix. This feature matrix represents the weighted features of each encoding layer, providing a more accurate and useful feature representation for subsequent classification tasks.

[0081] In this embodiment, dynamic weighting and maximum pooling operations are implemented in the coding layer to improve the accuracy and effectiveness of feature extraction; by performing maximum pooling on the hidden layer feature states of each coding layer, the most significant features are extracted, redundant information is filtered out, and the key features in each image block are retained; the Softmax function is used to calculate the weight of each image block, so that the contribution of each image block to the final feature matrix can be adjusted according to its importance. The dynamic weighting mechanism not only enables the model to respond more flexibly to feature changes of different image blocks, but also improves the ability to process complex image data; the first feature matrix finally generated integrates feature information at different levels, enhances the expressiveness of the classifier, and helps to improve the accuracy and robustness of classification.

[0082] Furthermore, the above step S204 may include: adding the hidden layer feature states with weights of each coding layer in the first feature matrix to obtain a second feature matrix; and performing average pooling on each dimension of the second feature matrix to obtain a classification vector.

[0083] Specifically, the first feature matrix contains the weighted hidden layer feature states of each coding layer. The weighted hidden layer feature states of each coding layer are added layer by layer to obtain a 16*d second feature matrix to fuse the feature information extracted from different coding layers to form a comprehensive feature representation, ensuring that the features abstracted from different levels can work together to provide more comprehensive information.

[0084] After obtaining the second feature matrix, average pooling will be performed on each dimension. Average pooling is to average all features in each dimension (each d direction has 16 values, and the average is taken), further compress the feature matrix, extract the overall information of each dimension, and obtain a 1*d classification vector. This process can reduce the dimension of the data while retaining important features as the basic features of the subsequent classifier input.

[0085] In this embodiment, the feature fusion capability and accuracy of the image classification model are improved by layer-by-layer addition and average pooling operations; the weighted hidden layer feature states of each coding layer are added so that features from different layers can be effectively integrated, ensuring the organic combination of shallow features and deep features, avoiding information loss, and improving the expressiveness of features; by performing average pooling on the second feature matrix, the overall features of each dimension are extracted. This method of aggregating features can capture the most significant information in the image, ensuring that the final output classification vector is more representative, and improving the classification accuracy and generalization ability of the model.

[0086] Furthermore, the above step S205 may include: inputting the classification vector into the classifier in the image classification model to obtain a second classification vector; inputting the second classification vector into the softmax function to obtain the category probability of each candidate category; and selecting the candidate category corresponding to the maximum category probability as the image category of the target image.

[0087] Specifically, the classification vector is input into the classifier of the image classification model. The classifier can be a d*c matrix, where c represents the number of classification categories, and is responsible for making classification decisions based on the input feature vector. Based on the input classification vector, the classifier calculates a 1*c second classification vector, which contains a preliminary evaluation of each candidate category.

[0088] The second classification vector is input into the Softmax function. The Softmax function converts the second classification vector into a probability distribution, which represents the predicted probability of each candidate category. The probability of each category reflects the model's confidence in that category, and the sum of the probabilities of all categories is 1.

[0089] Selecting the category with the highest probability as the final classification result of the target image ensures that the output category is the category that the model considers to be the most likely, thereby giving the target image a clear category label.

[0090] In this embodiment, the accuracy and reliability of the image classification task are enhanced by combining the classifier and the Softmax function; the input classification vector is processed by the classifier to generate a second classification vector, so that the model can comprehensively consider the relationship between different features and provide a more detailed evaluation for each candidate category, ensuring that the classifier can extract meaningful information from complex features and improve the accuracy of decision-making; the second classification vector is converted into a probability distribution through the Softmax function, which clearly reflects the confidence of each candidate category, thereby effectively distinguishing among multiple candidate categories; the category corresponding to the maximum probability is selected to provide the most representative classification result for the target image, ensuring the efficiency and accuracy of the model in practical applications.

[0091] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned target image, the above-mentioned target image can also be stored in a node of a blockchain.

[0092] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0093] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0094] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0095] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0096] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0097] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of an image classification device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0098] like Figure 3 As shown, the image classification device 300 described in this embodiment includes: an image acquisition module 301, a vector input module 302, a weight adding module 303, a weighted pooling module 304 and an image classification module 305, wherein:

[0099] The image acquisition module 301 is used to acquire a target image and generate a target feature vector of the target image.

[0100] The vector input module 302 is used to input the target feature vector into the encoder in the image classification model to obtain the hidden layer feature state output by each encoding layer in the encoder, wherein each encoding layer is added with Gaussian blur calculation.

[0101] The weight adding module 303 is used to add weights to the hidden layer feature states of each coding layer through the layer attention mechanism, and generate a first feature matrix according to the hidden layer feature states with weights of each coding layer.

[0102] The weighted pooling module 304 is used to perform a matrix weighting operation on the first feature matrix, and perform a pooling operation after the matrix weighting to obtain a classification vector.

[0103] The image classification module 305 is used to input the classification vector into the classifier in the image classification model to obtain the image category of the target image.

[0104] In this embodiment, when performing image classification, the model's ability to capture the correlation between image blocks is enhanced through Gaussian blur and layer attention mechanisms; Gaussian blur calculations are added to the multi-layer structure of the encoder, and the model improves its generalization ability for unseen image data and reduces overfitting problems; the layer attention mechanism enables the model to dynamically assign different weights to the hidden feature states of each layer, avoiding the problem of information loss in too deep layers in traditional stacked models, and ensuring that both shallow and deep features can contribute to the final classification; pooling and matrix weighting operations further extract the most significant feature information, so that the classifier can accurately determine the image category. This hierarchical feature processing improves the accuracy of classification, and can improve the accuracy of classification when classifying complex and scarce images.

[0105] In some optional implementations of this embodiment, the image acquisition module 301 may also be used for:

[0106] The target image is adjusted to a preset size to obtain a first image.

[0107] The first image is cut into a preset number of image blocks.

[0108] Convolution processing is performed on each image block respectively to obtain the first feature vector of each image block.

[0109] The position feature vector of each image block is obtained, and the first feature vector of each image block and its corresponding position feature vector are added to obtain the second feature vector.

[0110] The second feature vector of each image block is determined as the target feature vector of the target image.

[0111] In some optional implementations of this embodiment, the vector input module 302 may also be used for:

[0112] The target feature vector is input into the first encoding layer of the encoder in the image classification model, and the first encoding layer outputs the hidden layer feature state based on Gaussian blur calculation.

[0113] The hidden layer feature state output by the first coding layer is input into the second coding layer of the encoder for iteration until the last coding layer of the encoder, and the hidden layer feature state output by each coding layer in the encoder is obtained.

[0114] In some optional implementations of this embodiment, the output result of the multi-head attention mechanism in each encoding layer is expressed as:

[0115]

[0116] in, represents the output of the multi-head attention mechanism, h represents the hth head, Q h , K h and V h They represent the query representation, key representation, and value representation of the target feature vector in the hth head, respectively. σ is the standard deviation of the standard Gaussian distribution. k is the fixed window size in Gaussian blur. x is the input. When Gaussian blur calculation is introduced, the value representation in each head is used as the input of Gaussian blur. V h *g represents Gaussian blur calculation on the value representation.

[0117] In some optional implementations of this embodiment, the weight adding module 303 may also be used to:

[0118] For the hidden layer feature state of each coding layer, the block hidden state of each image block in the hidden layer feature state is maximized pooled from the image block dimension to obtain a first pooling result.

[0119] The first pooling result of each image block is input into the softmax function to obtain the weight of each image block.

[0120] The weight of each image block is multiplied by the corresponding hidden layer feature state to obtain the hidden layer feature state with weight.

[0121] A first feature matrix is ​​generated according to the hidden layer feature states with weights in each coding layer.

[0122] In some optional implementations of this embodiment, the weighted pooling module 304 may also be used to:

[0123] The hidden layer feature states with weights in each encoding layer in the first feature matrix are added together to obtain a second feature matrix.

[0124] Average pooling is performed on each dimension of the second feature matrix to obtain a classification vector.

[0125] In some optional implementations of this embodiment, the image classification module 305 may also be used to:

[0126] The classification vector is input into the classifier in the image classification model to obtain a second classification vector.

[0127] The second classification vector is input into the softmax function to obtain the category probability of each candidate category.

[0128] The candidate category corresponding to the maximum category probability is selected as the image category of the target image.

[0129] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0130] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 with a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0131] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0132] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the image classification method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0133] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the image classification method.

[0134] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0135] The computer device provided in this embodiment can execute the above-mentioned image classification method. Here, the image classification method can be the image classification method of each of the above-mentioned embodiments.

[0136] In this embodiment, when performing image classification, the model's ability to capture the correlation between image blocks is enhanced through Gaussian blur and layer attention mechanisms; Gaussian blur calculations are added to the multi-layer structure of the encoder, and the model improves its generalization ability for unseen image data and reduces overfitting problems; the layer attention mechanism enables the model to dynamically assign different weights to the hidden feature states of each layer, avoiding the problem of information loss in too deep layers in traditional stacked models, and ensuring that both shallow and deep features can contribute to the final classification; pooling and matrix weighting operations further extract the most significant feature information, so that the classifier can accurately determine the image category. This hierarchical feature processing improves the accuracy of classification, and can improve the accuracy of classification when classifying complex and scarce images.

[0137] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the image classification method as described above.

[0138] In this embodiment, when performing image classification, the model's ability to capture the correlation between image blocks is enhanced through Gaussian blur and layer attention mechanisms; Gaussian blur calculations are added to the multi-layer structure of the encoder, and the model improves its generalization ability for unseen image data and reduces overfitting problems; the layer attention mechanism enables the model to dynamically assign different weights to the hidden feature states of each layer, avoiding the problem of information loss in too deep layers in traditional stacked models, and ensuring that both shallow and deep features can contribute to the final classification; pooling and matrix weighting operations further extract the most significant feature information, so that the classifier can accurately determine the image category. This hierarchical feature processing improves the accuracy of classification, and can improve the accuracy of classification when classifying complex and scarce images.

[0139] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0140] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. An image classification method, characterized in that: The steps include: Acquire a target image, and generate a target feature vector of the target image; Inputting the target feature vector into an encoder in an image classification model to obtain a hidden layer feature state output by each encoding layer in the encoder, wherein each encoding layer is added with a Gaussian blur calculation; Adding weights to the hidden layer feature states of each coding layer through a layer attention mechanism, and generating a first feature matrix according to the hidden layer feature states with weights of each coding layer; Performing a matrix weighting operation on the first feature matrix, and performing a pooling operation after the matrix weighting to obtain a classification vector; The classification vector is input into the classifier in the image classification model to obtain the image category of the target image.

2. The image classification method according to claim 1, characterized in that: The step of generating the target feature vector of the target image comprises: Adjusting the target image to a preset size to obtain a first image; Cutting the first image into a preset number of image blocks; Performing convolution processing on each image block respectively to obtain a first feature vector of each image block; Acquire a position feature vector of each image block, and add a first block feature vector of each image block and a corresponding position feature vector thereof to obtain a second block feature vector; The second block feature vector of each image block is determined as the target feature vector of the target image.

3. The image classification method according to claim 1, characterized in that: The step of inputting the target feature vector into an encoder in the image classification model to obtain the hidden layer feature state output by each encoding layer in the encoder comprises: Input the target feature vector into the first encoding layer of the encoder in the image classification model, and the first encoding layer outputs the hidden layer feature state based on Gaussian blur calculation; The hidden layer feature state output by the first coding layer is input into the second coding layer of the encoder for iteration until the last coding layer of the encoder, so as to obtain the hidden layer feature state output by each coding layer in the encoder.

4. The image classification method according to claim 3, characterized in that: The output of the multi-head attention mechanism in each encoding layer is expressed as: in, represents the output of the multi-head attention mechanism, h represents the hth head, Q h , K h and V h Respectively represent the query representation, key representation and value representation of the target feature vector in the hth head, σ is the standard deviation of the standard Gaussian distribution, k is the fixed window size in Gaussian blur, x is the input, and when the Gaussian blur calculation is introduced, the value representation in each head is used as the input of Gaussian blur, V h *g represents Gaussian blur calculation on the value representation.

5. The image classification method according to claim 2, characterized in that: The step of adding weights to the hidden layer feature states of each coding layer through a layer attention mechanism and generating a first feature matrix according to the hidden layer feature states with weights of each coding layer comprises: For the hidden layer feature state of each coding layer, performing maximum pooling on the block hidden state of each image block in the hidden layer feature state from the image block dimension to obtain a first pooling result; Input the first pooling result of each image block into the softmax function to obtain the weight of each image block; Multiplying the weight of each image block by the corresponding hidden layer feature state to obtain a hidden layer feature state with a weight; A first feature matrix is ​​generated according to the hidden layer feature states with weights in each encoding layer.

6. The image classification method according to claim 1, characterized in that: The step of performing a matrix weighting operation on the first feature matrix and performing a pooling operation after the matrix weighting to obtain a classification vector includes: Adding the weighted hidden layer feature states of each encoding layer in the first feature matrix to obtain a second feature matrix; Average pooling is performed on each dimension of the second feature matrix to obtain a classification vector.

7. The image classification method according to claim 1, characterized in that: The step of inputting the classification vector into the classifier in the image classification model to obtain the image category of the target image comprises: Inputting the classification vector into a classifier in the image classification model to obtain a second classification vector; Input the second classification vector into the softmax function to obtain the category probability of each candidate category; The candidate category corresponding to the maximum category probability is selected as the image category of the target image.

8. An image classification device, characterized in that: include: An image acquisition module, used to acquire a target image and generate a target feature vector of the target image; A vector input module, used for inputting the target feature vector into an encoder in an image classification model to obtain a hidden layer feature state output by each encoding layer in the encoder, wherein each encoding layer is added with Gaussian blur calculation; A weight adding module, used for adding weights to the hidden layer feature states of each coding layer through a layer attention mechanism, and generating a first feature matrix according to the hidden layer feature states with weights of each coding layer; A weighted pooling module, used for performing a matrix weighting operation on the first feature matrix, and performing a pooling operation after the matrix weighting to obtain a classification vector; The image classification module is used to input the classification vector into the classifier in the image classification model to obtain the image category of the target image.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the image classification method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the image classification method according to any one of claims 1 to 7 are implemented.

Images