Image classification method and device, equipment and medium
By decomposing the image into high-frequency and low-frequency components and feeding it into the CNN and transformer networks respectively, the problem of difficulty in capturing global representation in the prior art is solved, and more efficient image classification performance is achieved.
Patent Information
- Application Number
- CN202411668240.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-20
- Publication Date
- 2025-03-14
AI Technical Summary
The existing image classification methods have difficulty capturing global representations due to convolutional operations, resulting in certain limitations in recognition performance.
Natural images are divided into low spatial frequency components that describe the stable changing structure and high spatial frequency components that describe the fine details of fast changing, and are sent to different branch networks respectively. The CNN and transformer branch networks are used to make the information contained in the image more comprehensively utilize.
Through this method, the performance of image classification is improved, and images can be analyzed more accurately from local and overall angles, thereby improving the accuracy of recognition.
Smart Images
Figure CN119625386A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology and digital medicine, and in particular to an image classification method, device, equipment and medium. Background Art
[0002] Image classification is one of the most basic tasks in computer vision and is widely used in many different fields. For example, in medical application scenarios, sample images are medical images. Image classification, as the basis of image processing, can be used to assist in processing medical images and complete disease screening and diagnosis. In financial application scenarios, image classification can be used to assist in extracting bank card numbers from photos, etc.
[0003] With the development of artificial intelligence technology, from the simple MNIST task of handwritten digit recognition in grayscale images with 10 categories, to the larger 10-category CIFAR10 and 100-category CIFAR100 tasks, to the ImageNet task, image classification models have been gradually improved to today's level along with the growth of data sets. Now, in data sets such as ImageNet with more than 10 million images and more than 20,000 categories, the image classification level of computers has surpassed that of humans.
[0004] Existing image classification methods usually use convolutional neural networks (CNNs), which are feedforward neural networks inspired by the natural visual cognition mechanism of organisms. Now it has become one of the research hotspots in the field of pattern classification. Since the network avoids complex pre-processing of images and can directly input the original image, it has been more widely used. However, the convolution operation is good at extracting local features, but it is difficult to capture global representations, so the recognition performance has certain limitations. Summary of the invention
[0005] In view of this, the present application provides an image classification method, apparatus, medium and equipment, which divide natural images into low spatial frequency components that describe smoothly changing structures and high spatial frequency components that describe rapidly changing fine details, and send them into different branch networks respectively, so that the information contained in the image can be more comprehensively utilized, solving the problem of low recognition performance of existing image classification methods.
[0006] In a first aspect of the present application, a method for image classification is provided, the method comprising:
[0007] Acquire a target image, and decompose the target image into a first component and a second component, wherein a frequency of the first component is higher than a frequency of the second component;
[0008] Processing the first component using a first preset network to obtain a first classification result;
[0009] Processing the second component using a second preset network to obtain a second classification result;
[0010] The first classification result and the second classification result are weighted to obtain a target classification result.
[0011] In a second aspect of the present application, an image classification device is provided, the device comprising:
[0012] an initialization unit, configured to acquire a target image and decompose the target image into a first component and a second component, wherein a frequency of the first component is higher than a frequency of the second component;
[0013] A first classification unit, configured to process the first component using a first preset network to obtain a first classification result;
[0014] A second classification unit, configured to process the second component using a second preset network to obtain a second classification result;
[0015] A weighting unit is used to perform weighted processing on the first classification result and the second classification result to obtain a target classification result.
[0016] In a third aspect of the present application, a device is provided, including a storage medium, a processor, and instructions or codes stored on the storage medium and executable on the processor, wherein the processor implements the above-mentioned image classification method when executing the instructions or codes.
[0017] In a fourth aspect of the present application, a medium is provided, on which instructions or codes are stored, and when the instructions or codes are executed by a processor, the above-mentioned image classification method is implemented.
[0018] The scheme implemented by the above-mentioned image classification method, device, equipment and medium divides the original target image into a high spatial frequency component (i.e., the first component) that describes the fine details of rapid changes and a low spatial frequency component (i.e., the second component) that describes the structure of smooth changes, and sends them to the CNN branch network (i.e., the first preset network) and the transformer branch network (i.e., the second preset network) respectively, so that the information contained in the image can be more comprehensively utilized. In addition, the present application designs a bridging module to send the global features from the second preset network to the first preset network to enhance the global perception ability of the first preset network; and to send the local features from the first preset network to the second preset network to enhance the local perception ability of the second preset network. Furthermore, the present application also improves the Attention of the current layer, adding the Attention of the previous layer as a priori to enhance the ability of Attention, which not only maintains the performance of the traditional residual structure, but also improves the performance of the attention module.
[0019] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0021] Figure 1 A schematic diagram of an image classification method provided in an embodiment of the present application is shown;
[0022] Figure 2 A schematic diagram of a target image of another image classification method provided by an embodiment of the present application is shown;
[0023] Figure 3 A schematic diagram of a first component of another image classification method provided by an embodiment of the present application is shown;
[0024] Figure 4 A schematic diagram of a second component of another image classification method provided by an embodiment of the present application is shown;
[0025] Figure 5 A schematic diagram of the bottleneck layer structure of another image classification method provided in an embodiment of the present application is shown;
[0026] Figure 6 A schematic diagram showing feature connections between a first preset network and a second preset network of another image classification method provided by an embodiment of the present application is shown;
[0027] Figure 7 A structural block diagram of an image classification device provided in an embodiment of the present application is shown;
[0028] Figure 8 A structural block diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0030] The image classification method provided in the embodiment of the present application can be applied to electronic devices that have the ability to run instructions or programs. Among them, the electronic devices can be, but are not limited to, various servers, workstations, personal computers, laptops, etc. Running on different computing devices is only a difference in the execution subject of the scheme. People in this field can foresee that running on different computing devices can produce the same technical effect. The present invention is described in detail below through specific embodiments.
[0031] See also Figure 1 As shown, Figure 1 A flowchart of an image classification method provided by an embodiment of the present invention includes the following steps:
[0032] S20: Acquire a target image, and decompose the target image into a first component and a second component, wherein the frequency of the first component is higher than the frequency of the second component.
[0033] The method provided by the present invention is used in the field of image processing and can be classified according to the specific content contained in the image. The method can be widely used in many different fields. For example, in a specific usage scenario, the lesion area and normal tissue organs in the medical image can be classified, thereby achieving the purpose of assisting the diagnosis of whether the human body has a lesion according to the specific content in the medical image. The method can also be used to analyze the medical images taken by the patient based on the different manifestations of benign and malignant areas to determine whether the lesion area is benign or malignant.
[0034] In a specific application scenario, this embodiment decomposes the target image into a first component and a second component. Specifically, the first component is a high spatial frequency component that describes fast-changing fine details, which is used for fine detail encoding, and the second component is a low spatial frequency component that describes a smoothly numbered structure, which is used for global structure encoding. The first component and the second component are respectively input into the first preset network and the second preset network in the form of images. It can be understood that the high-frequency component, that is, the first component, corresponds to the part of the target image where the distance of change is, that is, the edge contour or noise and detail part of the target image; and the low-frequency component, that is, the second component, corresponds to the area in the target image where the grayscale value changes slowly, that is, the large flat area in the target image, which describes the main part of the target image. Therefore, dividing the target image into the first component and the second component and processing them separately can realize the classification of the target image from both the local and the overall perspectives, which is conducive to improving the performance of image classification.
[0035] The first preset network may be a convolutional neural network, and the second preset network may be a transformer network. It is understandable that convolutional neural networks are good at extracting local features, so using convolutional neural networks to process high-frequency first components can obtain better results. The transformer network uses a cascaded self-attention mechanism, is better at capturing long-distance feature dependencies, and is more suitable for processing low-frequency second components.
[0036] It can be understood that this embodiment is explained by taking the first preset network as a convolutional neural network and the second preset network as a transformer network as an example, which does not mean that the present application can only be applied to convolutional neural networks and transformer networks. Those skilled in the art can adopt other network structures that are good at extracting local features as the first preset network, and adopt other network structures that are good at extracting overall features as the second preset network, which is also within the scope of the present invention.
[0037] Figure 2 A target image is shown, Figure 3 as well as Figure 4 The first high-frequency component and the second low-frequency component obtained by processing the target image are shown respectively. The first component and the second component are images obtained by processing the target image in different ways. It can be seen that Figure 2 is the original, unprocessed image, which contains two penguins. Figure 3 The high frequency components extracted are mainly Figure 2 's outline, and therefore pay more attention to Figure 2 Part of the content, Figure 4 The low-frequency components of the Figure 4 The penguin's head, abdomen, and back can be seen, but its outline is blurry, so we focus more on the Figure 2 The entirety of the content.
[0038] In actual application scenarios, for example, in medical scenarios, medical images of patients with coronary heart disease are input, details such as contours and smoothly changing structures are extracted, and two different preset networks are used for processing respectively, so that the image can be analyzed from both details and the overall perspective. For example, from the analysis of details such as contours, there are certain differences in the shapes of healthy tissues and lesion tissues. From the analysis of the smoothly changing part, there are differences in the colors of healthy tissues and lesion tissues, resulting in different RGB values. Therefore, comprehensive analysis from both details and the overall perspective to determine whether the image is healthy tissue or lesion tissue can effectively improve the accuracy of image classification.
[0039] S40: Processing the first component using a first preset network to obtain a first classification result.
[0040] In this step, a first preset network is used to process a first high-frequency component, wherein the first preset network is a network that is good at extracting local features, and the first preset network analyzes the target image locally to obtain a first classification result for the target image.
[0041] The first preset network includes a plurality of first convolution blocks and a first classifier connected in sequence, wherein each first convolution block includes at least one bottleneck layer, and each bottleneck layer includes a convolution layer, a normalization layer and an activation function.
[0042] Specifically, the first preset network is similar to the resnet (residual network) structure. It can be understood that the residual network has many bypass branches that directly connect the input to the subsequent layers, so that the subsequent layers can directly learn the residual, thereby solving the degradation problem of ordinary direct-connected convolutional networks.
[0043] In terms of specific structure, the first preset network includes multiple convolution blocks, denoted as first convolution blocks, and also includes a classifier, denoted as first classifier. Multiple first convolution blocks are connected in sequence, and the first classifier is connected after all first convolution blocks, that is, at the end of the first preset network.
[0044] Each first convolution block includes at least one bottleneck layer, and each bottleneck layer includes a convolution layer, a normalization layer, and an activation function. Specifically, inside each bottleneck layer, there are sequentially connected 1x1 conv, 3x3conv, and 1x1 conv convolution layers, wherein the first 1x1 conv is a down-projection convolution, which is used to reduce the feature dimension, use a small convolution kernel for feature extraction, reduce the number of parameters, and thus reduce the amount of calculation. In addition, reducing the dimension is more conducive to feature extraction; 3x3conv is a spatial convolution, which is used to perform convolution operations of conventional convolution kernels; the second 1x1 conv is an up-projection convolution, which is used to restore the dimension reduced by the first 1x1 conv to ensure that the dimension of the bottleneck layer output is equal to the dimension of the input. The setting of the bottleneck layer increases the depth of the first preset network, reduces the parameters of the first preset network, and is more conducive to training the first preset network.
[0045] In practical application scenarios, for example, in medical scenarios, a medical image of a patient with coronary heart disease is input, and details such as contours are extracted to obtain a first component and input into a first preset network. For example, a contour image of the heart is extracted as the first component and input into the first preset network. In the first preset network, the pixel value of each pixel in the contour image of the heart is extracted, and then the initial features are generated according to the pixel values of different points to obtain an initial matrix. The elements in the initial matrix can represent the position and color of each point in the contour of the heart. It can be understood that since the size and shape of the diseased heart and the healthy heart may be different, the contour of the heart in the image of the diseased heart is different from that of the healthy heart, resulting in the pixel value corresponding to each pixel point also correspondingly differing from that of the healthy heart, thereby resulting in the difference in the initial matrix. In the specific calculation process, the initial matrix is sequentially passed through multiple convolution blocks, and a convolution operation is performed in each convolution block. For example, if the color of a part of the contour of the heart is the same and there is a certain regularity in the contour coordinates, a few eigenvalues can be used to represent this part of the contour, without each pixel in the contour corresponding to an eigenvalue. Through such an operation, the matrix calculation amount is reduced, the calculation speed is improved, and the recognition accuracy of the heart contour is not reduced.
[0046] In addition, in the bottleneck layer, each convolution is followed by a normalization layer and an activation function. Figure 5 As shown, Figure 5 A specific structural diagram of a bottleneck layer is shown, which includes three convolutions connected in sequence, namely 1x1 conv, 3x3conv and 1x1 conv convolution layers, and each convolution is followed by a BN normalization layer and a RELU activation function. It can be understood that in a scenario where the activation function is not used, the output of each layer is a linear function of the upper layer input, and after using the activation function, the activation function can introduce nonlinear factors, so the first preset network can be applied to nonlinear scenarios. The normalization layer (Batch Normalization, BN) is usually used in conjunction with the activation function. The normalization layer eliminates the mean forward transfer caused by the activation function to ensure that the average activation of each dimension is 0. In this embodiment, the activation function can be RELU or GELU, etc., and other activation functions, such as PRELU function or MaxOut function, etc., can also be used according to the actual application scenario. Replacement of these activation functions is within the protection scope of the present invention.
[0047] In the specific application process, the high-frequency first component is input into the first preset network in the format of an image, and the feature information of the image is extracted in the first convolution block of the first preset network to obtain a data matrix containing image features, that is, a feature matrix. Each subsequent convolution block receives the feature matrix output by the previous convolution block, and performs dimensionality increase or dimensionality reduction on it to obtain a new feature matrix and pass it to the next convolution block.
[0048] Furthermore, among the multiple first convolution blocks connected in sequence, the first convolution block that is sorted first includes a bottleneck layer, and each of the other first convolution blocks includes at least two bottleneck layers, further increasing the depth of the first preset network.
[0049] This embodiment sets the first preset network to a structure similar to a residual network, including a plurality of sequentially connected convolution blocks and a classifier, performs feature extraction through the convolution blocks, and the classifier classifies the image according to the extracted features. Taking advantage of the fact that the first preset network is good at extracting local features, feature extraction and classification are performed on the high-frequency components of the target image, making full use of the local information in the target image to ensure the accuracy of image classification.
[0050] S60: Process the second component using a second preset network to obtain a second classification result.
[0051] In this step, a second preset network is used to process the low-frequency second component, wherein the second preset network is a network that is good at extracting global features, and the second preset network analyzes the target image as a whole to obtain a second classification result for the target image.
[0052] The second preset network includes a plurality of second convolution blocks and a second classifier connected in sequence, wherein each second convolution block includes a multi-layer perceptron module, a layer normalization module and a multi-head self-attention module, and each multi-layer perceptron module includes a first fully connected layer for increasing the dimension and a second fully connected layer for reducing the dimension.
[0053] Specifically, the second preset network is based on the Vit (Vision in Transformer) structure. Compared with the transformer structure, the Vit structure has better performance in image processing. In the process of processing images using the Vit structure, the picture is first serialized. Specifically, the two-dimensional target image is divided into small blocks (patches) of fixed size, and a one-dimensional patch embedding is obtained through linear transformation to achieve the conversion of high-dimensional vectors to low-dimensional vectors and reduce the amount of calculation. Then, according to the position of each patch in the target image, it is used as the input of the second preset network in sequence. After the serialization is completed, patch-level operations are performed in the second preset network for training and prediction. Among them, the category label and the position vector are both learnable vectors. The category label is used to predict the classification, and the position vector is used to characterize the position information of each patch in the image.
[0054] In practical application scenarios, for example, in medical scenarios, a medical image of a patient with coronary heart disease is input, and a steadily changing part is extracted to obtain a second component and input into a second preset network. For example, a large flat area image of the heart is extracted as the second component and input into the second preset network. In the second preset network, the pixel value of each pixel in the heart image is extracted, and then the initial features are generated according to the pixel values of different points to obtain an initial matrix. The elements in the initial matrix can represent the position and color of each point in the heart. It can be understood that since the color of the lesion area and the healthy area may be different, the pixel values corresponding to the pixels in the diseased heart image are also different from those of the healthy heart, which leads to the difference in the initial matrix. In the specific calculation process, the initial matrix is passed through multiple convolution blocks in sequence, and a convolution operation is performed in each convolution block. For example, if a large part of the heart has the same color, a few eigenvalues can be used to represent this part, without corresponding each pixel in this part to an eigenvalue. Through such an operation, the matrix operation amount is reduced, the operation speed is improved, and the recognition accuracy of the heart color is not reduced.
[0055] In terms of specific structure, the second preset network includes multiple convolution blocks, denoted as second convolution blocks, and also includes a classifier, denoted as a second classifier. Multiple second convolution blocks are connected in sequence, and the second classifier is connected after all the second convolution blocks, that is, at the end of the second preset network.
[0056] Among them, each second convolution block includes a multilayer perceptron (MLP) module, a layer normalization module and a multi-head self-attention module. Specifically, inside each multilayer perceptron module, it includes an upward projected fully connected layer (FC), recorded as the first fully connected layer, and also includes a downward projected fully connected layer, recorded as the second fully connected layer. In actual application scenarios, the two-dimensional target image is split into multiple patches and then input into the second preset network for processing, resulting in a reduction in the dimension of the second preset network. Therefore, the dimension is increased through the first fully connected layer to facilitate feature connection between the second preset network and the first preset network; thereafter, the second fully connected layer is used to reduce the dimension so that the dimension is restored to the number before the first fully connected layer is increased.
[0057] In addition, each second convolution block also includes a normalization layer. The principle of the normalization layer is similar to the normalization layer in the first preset network, which will not be repeated here.
[0058] In the specific application process, the low-frequency second component is input into the second preset network in the format of an image, and passes through each convolution block of the second preset network in turn. The processing method is similar to the processing of the first component by the convolution block of the first preset network, and will not be repeated here.
[0059] In addition, each second convolution block also includes a multi-head self-attention module. It can be understood that the multi-head attention module simulates the human attention mechanism and focuses more on the key points to be paid attention to, that is, in the multi-head attention module, a higher weight is given to the key points to be paid attention to. In this embodiment, the multi-head self-attention module selects key information for processing and ignores other unimportant information to better achieve the purpose of image classification.
[0060] This embodiment sets the second preset network to a structure based on the Vit network, including a plurality of sequentially connected convolution blocks and a classifier, performs feature extraction through the convolution blocks, and the classifier classifies the image according to the extracted features. By taking advantage of the fact that the second preset network is good at extracting global features, feature extraction and classification are performed on the low-frequency components of the target image, making full use of the global information in the target image, and further improving the accuracy of image classification.
[0061] Among them, a bridge module is arranged between the first preset network and the second preset network, and the features output by the first convolution block are interactively connected with the features output by the second convolution block through the bridge module.
[0062] Specifically, a bridge module is set between the first preset network and the second preset network, and the first preset network and the second preset network realize feature connection through the bridge module. Specifically, the feature map output by the 3x3conv convolution in the bottleneck layer of the first preset network is converted to the second preset network through the bridge module, and the features after the dimension is increased by the first fully connected layer in the second preset network can also be converted to the first preset network through the bridge module. Through such an interactive connection, the coupling between the local features of the high-frequency component and the global features of the low-frequency component is realized, and the performance of image classification is further improved. Since this embodiment sends the global features from the second preset network to the first preset network, the global perception ability of the first preset network is enhanced; similarly, the local features from the first preset network are sent to the second preset network, which enhances the local perception ability of the second preset network. Therefore, the performance of the first preset network and the second preset network are improved, so that the final image classification effect is better.
[0063] In actual application scenarios, for example, in medical scenarios, details such as contours and smoothly changing structures are extracted and processed using two different preset networks. Since there are multiple convolution blocks in each preset network that perform convolution operations in sequence, in the first preset network, the first component changes in features after multiple convolution operations, and there may be distortion. For example, the first preset network causes a slight deformation of the heart contour after multiple convolution operations. In response to this situation, the first preset network exchanges features with the second preset network, and the first preset network can determine the different colors of the left and right ventricles and the ventricular septum in the image based on the features provided by the second preset network, and then readjust the contours of the left and right ventricles based on the boundary lines of different colors, so as to further improve the calculation accuracy of the preset network. For patients with ventricular septal defects, such operations can improve the accuracy of identifying the patient's ventricular septum and reduce the misdiagnosis rate.
[0064] In addition, since the feature maps in the first preset network will be converted to the second preset network for feature connection, the second preset network no longer designs PE (position embedding, i.e., position encoding).
[0065] Preferably, since the first convolution block of the first preset network and the first convolution block of the second preset network input both initial features, feature connection is not required for these two convolution blocks; and since the features of the convolution blocks following these two convolution blocks are changed due to calculations, feature interactive connection can be performed between the two preset networks for these convolution blocks.
[0066] In the specific application process, after the second convolution block of the second preset network receives the features transmitted by the first convolution block of the first preset network through the bridge module, it combines it with the features existing in the current second convolution block itself and passes it to the next second convolution block. Among them, the feature combination method can be selected according to actual business needs. For example, the feature matrix corresponding to the features transmitted by the first convolution block and the feature matrix corresponding to the features of the second convolution block itself can be added to obtain a new feature matrix and passed to the next second convolution block; the two feature matrices can also be averaged to obtain a new feature matrix and pass it to the next second convolution block.
[0067] Similarly, after the first convolution block of the first preset network receives the features transmitted by the second convolution block of the second preset network, it also combines it with the features existing in the current first convolution block and passes it to the next first convolution block. The feature combination method is similar to the aforementioned method and will not be repeated here.
[0068] Figure 6The schematic diagram of feature connection between the first preset network and the second preset network is shown. As shown in the figure, the image first enters the stem module, wherein the stem module includes a convolution layer for extracting initial features and feeding them into the first preset network and the second preset network, wherein the feature map generated by the stem module is also split into multiple blocks through the project module in the second preset network and input in sequence. The first preset network includes multiple first convolution blocks connected in sequence, and the first classifier is connected after all the first convolution blocks; the second preset network includes multiple second convolution blocks connected in sequence, and the second classifier is connected after all the second convolution blocks. There is no feature connection between the first convolution block and the second convolution block that is sorted first, while the first convolution block sorted later and the corresponding second convolution block are connected to realize feature interaction. After that, the first preset network inputs the features processed by the first convolution block into the first classifier, and the second preset network inputs the features processed by the second convolution block into the second classifier, so as to realize the processing of the first component and the second component by the two networks respectively.
[0069] In specific application scenarios, the number of feature connections can also be set according to actual needs. For example, the features output by the 3x3conv convolution in each bottleneck layer are connected to the second preset network. Since the coupling between local features and global features is more and the interval is smaller, the performance of the classification method is better. On the other hand, the feature connection increases the amount of computation of the entire algorithm. Based on this, when the accuracy requirement is not high, the number of feature connections can be appropriately reduced. For example, in the 3x3conv convolutions in multiple bottleneck layers, a feature connection is performed every other 3x3conv convolution, or a feature connection is performed every two 3x3conv convolutions. Since the coupling between local features and global features is reduced and the interval is increased, the performance of the classification method is slightly reduced, but on the other hand, the reduction of feature connections also reduces the amount of computation of the entire algorithm accordingly. Based on this, when the number of feature connections is actually set, the algorithm accuracy and the amount of computation can be comprehensively considered according to actual needs, so as to reduce the amount of computation and improve the computation speed while ensuring that the performance of image classification meets actual needs.
[0070] In this embodiment, a bridge module is set between the first preset network and the second preset network, and the features output by the first preset network and the second preset network can be connected through the bridge module to achieve coupling between global features and local features. In the process of the first preset network using the high-frequency first component for classification, the features corresponding to the low-frequency second component can be referred to, thereby improving the image classification accuracy of the first preset network; similarly, in the process of the second preset network using the low-frequency second component for classification, the features corresponding to the high-frequency first component can also be referred to, thereby improving the image classification accuracy of the second preset network. In this way, the information in the target image is fully utilized, and the classification performance of the entire method is improved, so that the output classification result is more accurate.
[0071] In addition, before using the second preset network to process the second component, the method further includes:
[0072] Each multi-head attention module is determined in turn as a module to be corrected, the previous layer of multi-head attention module corresponding to the module to be corrected is the prior module, and the multi-head attention matrix corresponding to the prior module is used to correct the multi-head attention matrix corresponding to the module to be corrected.
[0073] In this step, the multi-head attention module of the second preset network is improved. In the multi-head attention module of the second preset network, the attention matrix is generated by input data, so some priors can be added to improve the ability of the multi-head attention module. Specifically, since the focus of each layer is not much different from that of the adjacent layers, the attention of each layer is similar to that of the adjacent layers. Based on this, the multi-head attention matrix of the previous layer can be used as the prior of the current layer.
[0074] Specifically, each multi-head attention module is taken as a module to be corrected in turn, and its corresponding previous layer multi-head attention module is taken as a priori module. The multi-head attention matrix of the priori module is used as the prior corresponding to the module to be corrected, and the priori module is used to improve the module to be corrected.
[0075] In actual application scenarios, for example, in medical scenarios, for patients with ventricular septal defects, it is understandable that when performing image classification, the focus should be on the ventricular septum part of the heart image. Therefore, the weight corresponding to the ventricular septum part is higher, and the weights corresponding to other parts are relatively low. However, during multiple convolution operations, the focus may shift. At this time, the multi-head attention module of the previous layer can be used to correct the attention of the current layer. For example, if the focus of the current layer shifts from the ventricular septum to the left ventricle, and the focus of the multi-head attention module of the previous layer is on the ventricular septum, the focus of the previous layer can be used to correct the focus of the current layer, so that in the current layer, the weight of the ventricular septum is higher than the weight of other parts. It is understandable that in the traditional multi-head attention mechanism, the calculation formula of the attention matrix is as follows:
[0076] Attention(Q n , K n , V n )=softmax(A n )V n
[0077] Among them, Attention(Q n , K n , V n ) is the multi-head attention matrix of the nth layer. In the multi-head attention matrix, Q (Query), K (Key) and V (Value) are all vectors generated according to the input features, where V is the vector representing the input features, and Q and K are the feature vectors for calculating the Attention weights. n , K n , and V n They represent the Q, K and V values of the nth layer respectively.
[0078] Among them, softmax() is used to normalize the vector. In the matrix obtained after normalization, the larger the weight, the higher the similarity.
[0079] in, Indicates Q n and K n The similarity between .
[0080] That is, Attention(Q n , K n , V n ) is the current Q n And all K n Calculate the similarity and normalize it through softmax() to get a set of weights. n The product of .
[0081] In the specific operation process, this embodiment adds a priori on the basis of the traditional multi-head attention mechanism, and uses the multi-head attention matrix of the previous layer to improve the multi-head attention matrix of the current layer. Specifically, the improvement of the multi-head attention module can be achieved based on the following formula:
[0082] Attention(Q n , K n , V n )=softmax(A n )V n ,in
[0083] Different from the traditional calculation formula, in this formula, A n-1 A n This improvement is equivalent to putting the residual on the Attention matrix, while the overall PostLN structure in the traditional residual structure design is still maintained. Therefore, it not only maintains the performance of PostLN, but also integrates the residual.
[0084] S80: Weighting the first classification result and the second classification result to obtain a target classification result.
[0085] In this step, the first classifier and the second classifier are respectively set at the end of the first preset network and the second preset network, and the two preset networks are trained simultaneously, and the importance of the loss functions of the two preset networks is the same. The first preset network processes the high-frequency first component to obtain the first classification result, and the second preset network processes the low-frequency second component to obtain the second classification result. Both classification results are input into the final third classifier, and the third classifier performs weighted processing on the first classification result and the second classification result, assigns corresponding weights, and obtains the final target classification result.
[0086] In a specific application scenario, for example, a medical image of a patient is input, and the image contains an image of an internal organ. The contour features of the medical image are extracted using the first preset network to obtain a first classification result, and the features of the flat part of the medical image are extracted by the second preset network to obtain a second classification result. Among them, the first classification result and the second classification result are a plurality of different classification labels and the probabilities corresponding to each label. If for a certain part of the image, the first classification result is: lesion probability 30%, health probability 70%; the second classification result is: lesion probability 55%, health probability 45%. In the third classifier, a weight of 60% is assigned to the first classification result, and a weight of 40% is assigned to the second classification result, and the final classification result is: lesion probability 0.3×0.6+0.55×0.4=0.4, health probability 0.7×0.6+0.45×0.4=0.6, so it can be considered that the probability of this part being a healthy area is higher, and the probability of being a lesion area is lower.
[0087] This embodiment combines the results obtained by the first preset network and the second preset network to obtain a final target classification result. Since the two preset networks process the high-frequency first component and the low-frequency second component respectively, the final target result takes both the high-frequency and low-frequency components into consideration, thereby making more comprehensive use of the information contained in the target image.
[0088] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.
[0089] It can be seen that compared with traditional image classification methods, the present application divides the original target image into a high spatial frequency component (i.e., the first component) that describes fast-changing fine details and a low spatial frequency component (i.e., the second component) that describes smoothly changing structures, and sends them to the CNN branch network (i.e., the first preset network) and the transformer branch network (i.e., the second preset network) respectively, so that the information contained in the image can be more comprehensively utilized. In addition, the present application designs a bridging module to send the global features from the second preset network to the first preset network to enhance the global perception ability of the first preset network; and to send the local features from the first preset network to the second preset network to enhance the local perception ability of the second preset network. Furthermore, the present application also improves the Attention of the current layer, adding the Attention of the previous layer as a priori to enhance the ability of Attention, which not only maintains the performance of the traditional residual structure, but also improves the performance of the attention module.
[0090] In one embodiment, an image classification device is provided, which corresponds one-to-one to the image classification method in the above embodiment. Figure 7 As shown, the image classification device includes: an initialization unit, a first classification unit, a second classification unit and a weighting unit. Each functional unit is described in detail as follows:
[0091] an initialization unit, used for acquiring a target image and decomposing the target image into a first component and a second component, wherein a frequency of the first component is higher than a frequency of the second component;
[0092] A first classification unit, configured to process the first component using a first preset network to obtain a first classification result;
[0093] A second classification unit, used for processing the second component by using a second preset network to obtain a second classification result;
[0094] The weighting unit is used to perform weighted processing on the first classification result and the second classification result to obtain a target classification result.
[0095] In one embodiment, the first preset network includes a plurality of first convolution blocks and a first classifier connected in sequence, wherein each first convolution block includes at least one bottleneck layer, and each bottleneck layer includes a convolution layer, a normalization layer, and an activation function.
[0096] In one embodiment, the second preset network includes a plurality of second convolution blocks connected sequentially and a second classifier, wherein each second convolution block includes a multi-layer perceptron module, a layer normalization module and a multi-head self-attention module, and each multi-layer perceptron module includes a first fully connected layer for increasing the dimension and a second fully connected layer for reducing the dimension.
[0097] In one embodiment, a bridge module is provided between the first preset network and the second preset network, and features output by the first convolution block are interactively connected with features output by the second convolution block through the bridge module.
[0098] In one embodiment, the device further comprises an improvement unit, configured to:
[0099] Each multi-head attention module is determined in turn as a module to be corrected, the previous layer of multi-head attention module corresponding to the module to be corrected is the prior module, and the multi-head attention matrix corresponding to the prior module is used to correct the multi-head attention matrix corresponding to the module to be corrected.
[0100] In one embodiment, the first preset network is a convolutional neural network, and the second preset network is a transformer network.
[0101] In one embodiment, among the plurality of first convolution blocks connected sequentially, the first convolution block ordered first includes one bottleneck layer, and each of the other first convolution blocks includes at least two bottleneck layers.
[0102] In one embodiment, an electronic device is provided. The electronic device may be a mobile device such as a mobile phone, a tablet, a vehicle-mounted mobile terminal, or other device with program execution capability. The internal structure diagram of the electronic device may be as shown below: Figure 8 As shown. The electronic device includes a processor, a memory and a network module. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, instructions or codes. The internal memory provides an environment for the operation of the operating system and instructions or codes in the non-volatile storage medium. When the instructions or codes are executed by the processor, a function or step of the above-mentioned image classification method is implemented. The network module of the electronic device may include a network interface and / or a wireless network module, and the electronic device may communicate with other devices or service platforms through the network module. In addition, the electronic device may also include a display screen and an input device, etc.
[0103] In one embodiment, an electronic device is provided, including a memory, a processor, and instructions or codes stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the instructions or codes:
[0104] Acquire a target image, and decompose the target image into a first component and a second component, wherein a frequency of the first component is higher than a frequency of the second component;
[0105] Processing the first component using a first preset network to obtain a first classification result;
[0106] Processing the second component using a second preset network to obtain a second classification result;
[0107] The first classification result and the second classification result are weighted to obtain the target classification result.
[0108] In one embodiment, a storage medium is provided, on which instructions or codes are stored, and when the instructions or codes are executed by a processor, the following steps are implemented:
[0109] Acquire a target image, and decompose the target image into a first component and a second component, wherein a frequency of the first component is higher than a frequency of the second component;
[0110] Processing the first component using a first preset network to obtain a first classification result;
[0111] Processing the second component using a second preset network to obtain a second classification result;
[0112] The first classification result and the second classification result are weighted to obtain the target classification result.
[0113] It should be noted that the functions or steps that can be implemented by the above-mentioned storage medium or electronic device can refer to the relevant description in the aforementioned method embodiment. In order to avoid repetition, they will not be described one by one here.
[0114] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through instructions or codes, and the instructions or codes can be stored in a non-volatile readable storage medium. When the instructions or codes are executed, they can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0115] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the decomposition of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be decomposed into different functional units or modules to complete all or part of the functions described above.
[0116] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the units or processes in the accompanying drawings are not necessarily necessary for implementing the present application. Those skilled in the art will appreciate that the units in the system in the implementation scenario can be distributed in the system of the implementation scenario according to the implementation scenario description, or can be changed accordingly and located in one or more systems different from the present implementation scenario. The units of the above-mentioned implementation scenario can be combined into one unit, or can be further split into multiple sub-units.
[0117] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.
Claims
1. An image classification method, characterized in that: The method comprises: Acquire a target image, and decompose the target image into a first component and a second component, wherein a frequency of the first component is higher than a frequency of the second component; Processing the first component using a first preset network to obtain a first classification result; Processing the second component using a second preset network to obtain a second classification result; The first classification result and the second classification result are weighted to obtain a target classification result.
2. The method according to claim 1, characterized in that The first preset network includes a plurality of first convolution blocks and a first classifier connected in sequence, wherein each of the first convolution blocks includes at least one bottleneck layer, and each of the bottleneck layers includes a convolution layer, a normalization layer, and an activation function.
3. The method according to claim 2, characterized in that The second preset network includes a plurality of second convolution blocks connected in sequence and a second classifier, wherein each of the second convolution blocks includes a multi-layer perceptron module, a layer normalization module and a multi-head self-attention module, and each of the multi-layer perceptron modules includes a first fully connected layer for increasing the dimension and a second fully connected layer for reducing the dimension.
4. The method according to claim 3, characterized in that A bridge module is provided between the first preset network and the second preset network, and the features output by the first convolution block are interactively connected with the features output by the second convolution block through the bridge module.
5. The method according to claim 3, characterized in that: Before processing the second component by using a second preset network, the method further includes: Determine each of the multi-head attention modules as a module to be corrected in turn, the previous layer of multi-head attention module corresponding to the module to be corrected is a priori module, and use the multi-head attention matrix corresponding to the priori module to correct the multi-head attention matrix corresponding to the module to be corrected.
6. The method according to claim 1, characterized in that The first preset network is a convolutional neural network, and the second preset network is a transformer network.
7. The method according to claim 2, characterized in that Among the plurality of first convolution blocks connected sequentially, the first convolution block that is ordered first includes one bottleneck layer, and each of the other first convolution blocks includes at least two bottleneck layers.
8. An image classification device, characterized in that: The device comprises: an initialization unit, configured to acquire a target image and decompose the target image into a first component and a second component, wherein a frequency of the first component is higher than a frequency of the second component; A first classification unit, configured to process the first component using a first preset network to obtain a first classification result; A second classification unit, configured to process the second component using a second preset network to obtain a second classification result; A weighting unit is used to perform weighted processing on the first classification result and the second classification result to obtain a target classification result.
9. A storage medium having a program or instruction stored thereon, characterized in that: When the program or instruction is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
10. An electronic device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.