An image classification method and system based on a modular convolutional neural network
By using modular convolutional neural networks and attention-based gating units, the problems of large parameter count and slow convergence speed in deep neural networks are solved, resulting in a faster training process and higher classification accuracy.
Patent Information
- Application Number
- CN202210863541.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-21
- Publication Date
- 2026-01-06
- Estimated Expiration
- 2042-07-21
AI Technical Summary
Existing deep neural network architectures improve classification accuracy by increasing network depth, but this leads to excessively large parameter counts and slow convergence speeds during training.
A modular convolutional neural network is adopted. By flipping and increasing the brightness of the original image, multiple main modules and sub-modules are used for feature extraction. The feature weights are determined by gating units with attention mechanism, and unnecessary main modules are discarded to reduce the number of parameters.
This reduces the number of network parameters, thereby improving the network's convergence speed and classification accuracy.
Smart Images
Figure CN115205598B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer image classification, and in particular to an image classification method and system based on modular convolutional neural networks. Background Technology
[0002] In the field of computer image classification, well-trained deep neural networks can distinguish between different image categories and exhibit excellent performance. However, most deep neural network architectures are designed based on single modules, meaning that most deep neural network architectures achieve higher classification accuracy by increasing network depth. While increasing network depth can improve recognition accuracy, it leads to a huge number of parameters in the network. This results in a slow convergence speed during network training. Summary of the Invention
[0003] The purpose of this invention is to provide an image classification method and system based on modular convolutional neural networks. This invention can reduce the number of parameters in the network and accelerate the convergence speed of the network.
[0004] To achieve the above objectives, the present invention provides the following solution:
[0005] An image classification method based on modular convolutional neural networks, the method comprising the following steps:
[0006] Obtain the original image;
[0007] The original image is flipped and its brightness is increased to obtain a flipped image and a brightened image;
[0008] A modular convolutional neural network is used to extract features from the original image, the flipped image, and the image with increased brightness, resulting in several sets of features. The modular convolutional neural network includes several main modules, and each main module includes three sub-modules. The three sub-modules within the same main module are the same convolutional neural network, while the sub-modules in different main modules are different convolutional neural networks. The three sub-modules are used to extract features from the corresponding original image, the flipped image, and the image with increased brightness, respectively, to obtain original image features, flipped image features, and image with increased brightness features. Each main module corresponds to a set of features, and each set of features includes the features extracted by the three sub-modules.
[0009] The features extracted from the three sub-modules in each set of features are fused to obtain several sets of fused features.
[0010] The weights of each set of fused features are determined using a gating unit based on an attention mechanism.
[0011] Determine whether the weight of each set of fused features is greater than a preset threshold.
[0012] If not, the main module corresponding to a weight that is not greater than the preset threshold will be discarded;
[0013] If so, the main module corresponding to the weight that is greater than the preset threshold will be retained;
[0014] Perform a weighted fusion operation on each set of fusion features and corresponding weights in all retained main modules to obtain the weighted fusion result;
[0015] The probability value of the weighted fusion result is calculated using the SoftMax function, and the probability value is used to determine the classification result.
[0016] The present invention also provides an image classification system based on a modular convolutional neural network, the system comprising:
[0017] The raw image acquisition unit is used to acquire the raw image.
[0018] The preprocessing unit is used to flip and increase the brightness of the original image respectively, to obtain the flipped image and the image with increased brightness;
[0019] The feature extraction unit is used to extract features from the original image, the flipped image, and the image with increased brightness using a modular convolutional neural network to obtain several sets of features. The modular convolutional neural network includes several main modules, and each main module includes three sub-modules. The three sub-modules in the same main module are the same convolutional neural network, while the sub-modules in different main modules are different convolutional neural networks. The three sub-modules are used to extract features from the corresponding original image, the flipped image, and the image with increased brightness, respectively, to obtain original image features, flipped image features, and image with increased brightness features. Each main module corresponds to a set of features, and each set of features includes the features extracted by the three sub-modules.
[0020] The feature fusion unit is used to fuse the features extracted from the three sub-modules in each set of features to obtain several sets of fused features;
[0021] The weight determination unit is used to determine the weights of each set of fused features using an attention-based gating unit;
[0022] The judgment unit is used to determine whether the weight of each group of fused features is greater than a preset threshold.
[0023] The discard unit is used to discard the main module corresponding to a weight that is not greater than a preset threshold when the judgment result is negative.
[0024] The retention unit is used to retain the main module corresponding to the weight that is greater than the preset threshold when the judgment result is yes;
[0025] The weighted fusion unit is used to perform a weighted fusion operation on each set of fusion features and corresponding weights in all the retained main modules to obtain the weighted fusion result.
[0026] The probability value calculation unit is used to calculate the probability value of the weighted fusion result using the SoftMax function, and the probability value is used to determine the classification result.
[0027] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0028] This invention provides an image classification method and system based on a modular convolutional neural network. The method includes: acquiring an original image; flipping and increasing the brightness of the original image to obtain a flipped image and a brightened image; and using a modular convolutional neural network to extract features from the original image, the flipped image, and the brightened image to obtain several sets of features. The modular convolutional neural network includes several main modules, and each main module includes three sub-modules. The three sub-modules in the same main module are the same convolutional neural network, while the sub-modules in different main modules are different convolutional neural networks. The three sub-modules are used to extract features from the corresponding original image, the flipped image, and the brightened image to obtain the original image. The system includes three main modules: image features, flipped image features, and image features with increased brightness. Each main module corresponds to a set of features, and each set of features includes features extracted by the three sub-modules. The features extracted by the three sub-modules in each set of features are fused to obtain several sets of fused features. A gating unit based on an attention mechanism is used to determine the weight of each set of fused features. It is determined whether the weight of each set of fused features is greater than a preset threshold. If not, the main module corresponding to the weight of the set of fused features is discarded. If so, the main module corresponding to the weight of the set of fused features is retained. Each set of fused features and its corresponding weight in all retained main modules are subjected to a weighted fusion operation to obtain a weighted fusion result. The probability value of the weighted fusion result is calculated using the SoftMax function, and the probability value is used to determine the classification result. The modular convolutional neural network proposed in this invention includes multiple main modules, each of which contains multiple sub-modules. Each sub-module performs a different function. The number of convolutional layers in each sub-module is much smaller than the number of convolutional layers in a single-module convolutional neural network, which can reduce the number of parameters in the network. Furthermore, the number of main modules can be reduced through the attention mechanism gating unit, which in turn can reduce the number of parameters in the network. By reducing the number of parameters, the convergence speed of the network can be accelerated. Attached Figure Description
[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0030] Figure 1 This is a flowchart of an image classification method based on a modular convolutional neural network provided in Embodiment 1 of the present invention;
[0031] Figure 2 This is a block diagram of an image classification method based on a modular convolutional neural network.
[0032] Figure 3 A block diagram of the gating unit for the attention mechanism;
[0033] Figure 4 This is a structural block diagram of an image classification system based on a modular convolutional neural network, provided in Embodiment 2 of the present invention. Detailed Implementation
[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0035] The purpose of this invention is to provide an image classification method and system based on modular convolutional neural networks. This invention can reduce the number of parameters in the network and accelerate the convergence speed of the network.
[0036] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0037] Example 1:
[0038] Please see Figure 1 This invention provides an image classification method based on a modular convolutional neural network, the method comprising the following steps:
[0039] S1: Obtain the original image;
[0040] S2: Flip and increase the brightness of the original image respectively to obtain the flipped image and the brightened image;
[0041] It should be noted that in previous single-module models, most models preprocessed the data by uniformly applying methods such as rotation, flipping, and brightness adjustment to the input image, and then inputting the processed image into the model for training. This approach, which applies preprocessing methods such as flipping and increasing brightness to a single image, may result in the loss of some information in the processed image. Therefore, this invention separates the image preprocessing methods, that is, it flips and increases the brightness of the original image separately, obtaining a flipped image and a brightened image, which, together with the original image, yield three images.
[0042] S3: A modular convolutional neural network is used to extract features from the original image, the flipped image, and the image with increased brightness, resulting in several sets of features. The modular convolutional neural network includes several main modules, each main module comprising three sub-modules. The three sub-modules within the same main module are the same convolutional neural network, while the sub-modules in different main modules are different convolutional neural networks. The three sub-modules are used to extract features from the corresponding original image, the flipped image, and the image with increased brightness, respectively, resulting in original image features, flipped image features, and image features with increased brightness. Each main module corresponds to a set of features, and each set of features includes the features extracted by the three sub-modules. The structural block diagram of the modular convolutional neural network is shown below. Figure 2 As shown.
[0043] In a modular convolutional neural network, the operation of each main module is shown in Equation (1):
[0044] f i_1 f i_2 f i_3 =Module i (input, input) flip input bright (1)
[0045] Among them, Module i () represents the feature extraction operation performed on the input image by the i-th main module, where input is the original input image. flip The input is the flipped image. bright f is the input image with increased brightness. i_1 f is the feature extracted from the original image. i_2 To extract features from the flipped image, f i_3 Let i be the extracted features after brightness enhancement, where i∈{1,...,M} and M is a positive integer.
[0046] It should be noted that in this embodiment, the three sub-modules in each main module are the same convolutional neural network. As other possible implementations, the three sub-modules in each main module can have two identical convolutional neural networks or three different convolutional neural networks, as long as the structural composition of all main modules is not completely the same.
[0047] S4: Fuse the features extracted from the three sub-modules in each feature set to obtain several sets of fused features; the specific calculation formula is as follows:
[0048] F i =Fusion(f i_1 f i_2 f i_3 (2)
[0049] Where Fusion() is the concatenate operation, f i_1 f is the feature extracted from the original image. i_2 To extract features from the flipped image, f i_3 F is used to extract features from the image after brightness enhancement. i Let i be the fusion feature of the i-th group, i∈{1,...,M}, where M is a positive integer.
[0050] S5: Use an attention-based gating unit to determine the weights of each set of fused features;
[0051] This invention designs a gating unit with an attention mechanism, the structure of which is as follows: Figure 3 As shown, the specific calculation formula is as follows:
[0052] W i =Attention(F i (3)
[0053] Where Attention() is the operation by which the attention module determines the weights of the i-th group of fused features, F i For the i-th fusion feature, W i Let be the weight of the i-th fused feature group, i∈{1,...,M}, where M is a positive integer.
[0054] S6: Determine whether the weight of each group of fused features is greater than a preset threshold; the threshold is generally set to 0.4.
[0055] S7: If not, it is assumed that the main module corresponding to the weight that is not greater than the preset threshold contains fewer features that are helpful for classification, so the main module corresponding to the weight that is not greater than the preset threshold is discarded.
[0056] S8: If so, the main module corresponding to the weight that is greater than the preset threshold will be retained;
[0057] S9: Perform a weighted fusion operation on each set of fusion features and corresponding weights in all retained main modules to obtain the weighted fusion result; the specific calculation formula is as follows:
[0058] Output = W1*F1 + W2*F2 + ... + W M *F M (4)
[0059] Where Output is the weighted fusion result, W1 is the weight of the first group of fused features, W2 is the weight of the second group of fused features, and W... M Let F1 be the weight of the Mth fusion feature, F2 be the weight of the 1st fusion feature, and F3 be the weight of the 2nd fusion feature. M Let M be the Mth fusion feature, where M is a positive integer.
[0060] S10: The SoftMax function is used to calculate the probability value of the weighted fusion result. The probability value is used to determine the classification result. The specific calculation formula is as follows:
[0061] Label = SoftMax(Output)
[0062] Where Label is the probability value of the weighted fusion result, SoftMax() is the classification function, and Output is the weighted fusion result.
[0063] The SoftMax function is a classification function that produces a probability vector, where each probability value represents the probability of a class. The class with the highest probability is then selected as the classification result.
[0064] It should be noted that each probability and its corresponding category can be set as needed, and the present invention can better complete the image classification task by using parallel convolution modules.
[0065] Specifically, after step S2 and before step S3, the method further includes:
[0066] The original image, the flipped image, and the image with increased brightness were scaled using bilinear interpolation.
[0067] In summary, this invention achieves better image classification tasks through parallel convolutional modules. The modular convolutional neural network proposed in this invention comprises multiple main modules, each of which contains multiple sub-modules. Each sub-module performs a different function, and the number of convolutional layers in each sub-module is much smaller than that in a single-module convolutional neural network. This reduces the number of parameters in the network, achieves a better recognition effect, and accelerates the convergence speed of the network. Furthermore, the attention mechanism gate unit can reduce the number of main modules, thereby also reducing the number of parameters in the network.
[0068] Example 2:
[0069] Please see Figure 4 This invention provides an image classification system based on a modular convolutional neural network, the system comprising:
[0070] Original image acquisition unit 1 is used to acquire the original image;
[0071] Preprocessing unit 2 is used to flip and increase the brightness of the original image respectively to obtain the flipped image and the image with increased brightness;
[0072] Feature extraction unit 3 is used to extract features from the original image, the flipped image, and the image with increased brightness using a modular convolutional neural network to obtain several sets of features. The modular convolutional neural network includes several main modules, and each main module includes three sub-modules. The three sub-modules in the same main module are the same convolutional neural network, and the sub-modules in different main modules are different convolutional neural networks. The three sub-modules are used to extract features from the corresponding original image, the flipped image, and the image with increased brightness, respectively, to obtain original image features, flipped image features, and image with increased brightness features. Each main module corresponds to a set of features, and each set of features includes the features extracted by the three sub-modules.
[0073] Feature fusion unit 4 is used to fuse the features extracted from the three sub-modules in each set of features to obtain several sets of fused features;
[0074] Weight determination unit 5 is used to determine the weights of each set of fused features using a gating unit based on an attention mechanism;
[0075] Judgment unit 6 is used to determine whether the weight of each group of fused features is greater than a preset threshold.
[0076] Discard unit 7 is used to discard the main module corresponding to a weight that is not greater than a preset threshold when the judgment result is negative.
[0077] Retention unit 8 is used to retain the main module corresponding to the weight that is greater than the preset threshold when the judgment result is yes;
[0078] Weighted fusion unit 9 is used to perform a weighted fusion operation on each set of fusion features and corresponding weights in all the retained main modules to obtain a weighted fusion result.
[0079] The probability value calculation unit 10 is used to calculate the probability value of the weighted fusion result using the SoftMax function, and the probability value is used to determine the classification result.
[0080] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.
[0081] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A method for image classification based on a modular convolutional neural network, characterized in that, The method comprises the following steps: obtaining an original image; flipping and increasing brightness of the original image respectively to obtain a flipped image and an increased brightness image; extracting features of the original image, the flipped image and the increased brightness image by using a modular convolutional neural network to obtain a plurality of groups of features, wherein the modular convolutional neural network comprises a plurality of main modules, each main module comprises three sub-modules, the three sub-modules in the same main module are the same convolutional neural network, the sub-modules in different main modules are different convolutional neural networks, and the three sub-modules are used for extracting features of the corresponding original image, flipped image and increased brightness image respectively to obtain original image features, flipped image features and increased brightness image features; each main module corresponds to a group of features, and each group of features comprises features extracted by the three sub-modules; fusing the features extracted by the three sub-modules in each group of features to obtain a plurality of groups of fused features; determining the weight of each group of fused features by using a gating unit based on an attention mechanism; judging whether the weight of each group of fused features is greater than a preset threshold respectively; if not, discarding the main module corresponding to the weight not greater than the preset threshold; if yes, retaining the main module corresponding to the weight greater than the preset threshold; performing a weighted fusion operation on each group of fused features and the corresponding weight in all retained main modules to obtain a weighted fusion result; calculating a probability value of the weighted fusion result by using a SoftMax function, and the probability value is used to determine a classification result. 2.The method of claim 1, wherein, After the step of "flipping and increasing brightness of the original image respectively to obtain a flipped image and an increased brightness image", and before the step of "extracting features of the original image, the flipped image and the increased brightness image by using a modular convolutional neural network to obtain a plurality of groups of features", the method further comprises: performing scale adjustment on the original image, the flipped image and the increased brightness image respectively by using a bilinear interpolation method. 3.The method of claim 1, wherein, The calculation formula of the step of "extracting features of the original image, the flipped image and the increased brightness image by using a modular convolutional neural network to obtain a plurality of groups of features" is as follows: f i_1 , f i_2 , f i_3 = Module i (input, input flip , input bright ) wherein Module i () is an operation of feature extraction of the i-th main module to an input image, input is an original image input, input flip is an image input after flipping, input bright is an image input after increasing brightness, f i_1 is a feature of the original image extracted, f i_2 is a feature of the flipped image extracted, f i_3 is a feature extracted after increasing brightness, i∈{1,...,M}, M is a positive integer. 4.The method of claim 1, wherein, The calculation formula of the step of "fusing the features extracted by the three sub-modules in each group of features to obtain a plurality of groups of fused features" is as follows: F i = Fusion(f i_1 , f i_2 , f i_3 ) wherein Fusion() is a concatenate operation, f i_1 is a feature of the extracted original image, f i_2 is a feature of the extracted flipped image, f i_3 is a feature of the extracted image with increased brightness, F i is the i-th group of fused features, i ∈ {1,..., M}, and M is a positive integer. 5.The method of claim 1, wherein, The calculation formula of the step of "determining the weight of each group of fused features by using a gating unit based on an attention mechanism" is as follows: W i = Attention(F i ) wherein Attention() is an operation of determining a weight of the i-th group of fusion features by the attention module, F i is the i-th group of fusion features, W i is the weight of the i-th group of fusion features, i∈{1,...,M}, and M is a positive integer. 6.The method of claim 1, wherein, The calculation formula of the step of "performing a weighted fusion operation on each group of fused features and the corresponding weight in all retained main modules to obtain a weighted fusion result" is as follows: Output = W1*F1 + W2*F2 +... + W M *F M Wherein, Output is the weighted fusion result, W1 is the weight of the first group of fusion features, W2 is the weight of the second group of fusion features, W M is the weight of the Mth group of fusion features, F1 is the first group of fusion features, F2 is the second group of fusion features, F M is the Mth group of fusion features, and M is a positive integer.
7. The method of claim 1, wherein the method further comprises: The calculation formula of the step of "calculating a probability value of the weighted fusion result by using a SoftMax function" is as follows: Label = SoftMax (Output) wherein Label is the probability value of the weighted fusion result, SoftMax () is a classification function, and Output is the weighted fusion result.
8. A modular convolutional neural network based image classification system, characterized in that, The method comprises the following steps: an original image acquisition unit is configured to obtain an original image; A preprocessing unit is configured to flip and increase brightness of the original images respectively to obtain flipped images and increased brightness images; A feature extraction unit is configured to extract features from the original images, the flipped images and the increased brightness images by using a modular convolutional neural network to obtain a plurality of groups of features, wherein the modular convolutional neural network comprises a plurality of main modules, each of the main modules comprises three sub-modules, the three sub-modules in the same main module are the same convolutional neural network, the sub-modules in different main modules are different convolutional neural networks, the three sub-modules are configured to extract features from the corresponding original images, flipped images and increased brightness images respectively to obtain original image features, flipped image features and increased brightness image features, each of the main modules corresponds to a group of features, and each of the groups of features comprises the features extracted by the three sub-modules; A feature fusion unit is configured to fuse the features extracted by the three sub-modules in each of the groups of features to obtain a plurality of groups of fused features; A weight determination unit is configured to determine weights of the groups of fused features by using a gating unit based on an attention mechanism; A judgment unit is configured to judge whether the weights of each of the groups of fused features are greater than a preset threshold respectively; A discard unit is configured to discard the main module corresponding to the weight not greater than the preset threshold when the judgment result is no; A reservation unit is configured to reserve the main module corresponding to the weight greater than the preset threshold when the judgment result is yes; A weighted fusion unit is configured to perform a weighted fusion operation on each of the groups of fused features and the corresponding weights in all the reserved main modules to obtain a weighted fusion result; A probability value calculation unit is configured to calculate a probability value of the weighted fusion result by using a SoftMax function, and the probability value is used to determine a classification result.