Image classification method based on residual pose capsule routing

Abstract

The present application relates to the field of image processing, and particularly relates to an image classification method based on residual pose capsule routing, which comprises the following steps: constructing an image classification network based on residual pose routing capsule network; the image classification network based on residual pose routing capsule network comprises a basic feature extraction network and a residual pose routing capsule network; the basic feature extraction network is used for extracting basic deep features of an input image; the residual pose routing capsule network comprises a residual capsule routing module and a residual capsule classification module; the residual capsule routing module is used for extracting and establishing capsule features based on the basic deep features; the residual capsule classification module is used for classification based on the capsule features; training the constructed image classification network; and classifying a to-be-detected image based on the trained image classification network. The present application reduces network parameters and reduces routing complexity.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more particularly to an image classification method based on residual pose capsule routing. Background Technology

[0002] The goal of image classification is to distinguish images of different categories based on their semantic information, minimizing classification error. Specifically, the task is to assign a label to an image from a given set of categories.

[0003] Early image classification methods relied on extracting handcrafted features to provide a global description of the entire image. However, due to the limited representational power of these handcrafted features, the performance of these methods remained stagnant. The advent of deep learning, with its powerful feature representation capabilities, has significantly improved image classification performance. Deep convolutional neural networks (CNNs) achieve image recognition by detecting the probability of entity presence, i.e., invariance. However, simple perturbations can easily cause deep CNNs to fail to correctly recognize images. To address this, researchers have attempted to introduce equivalence into neural networks, such as capsule networks. (See: "S. Sabour, N. Frosst, and G.E.Hinton, 'Dynamic routing between capsules,' in: Proceedings of the Neural Information and Processing Systems, 1-11, 2017." and "G.E.Hinton, S. Sabour, and N. Frosst, 'Matrix capsules with EM routing,' in: Proceedings of the International Conference on Learning Representations, 3856–3866, 2018."). However, this unidirectional routing method tends to result in a large number of network parameters and high complexity, leading to slow inference speed. Summary of the Invention

[0004] The purpose of this invention is to provide an image classification method based on residual pose capsule routing, which reduces network parameters, lowers routing complexity, and thus accelerates the inference speed of the capsule routing-based image classification method.

[0005] To solve the above technical problems, the technical solution of the present invention is as follows:

[0006] Firstly, an image classification method based on residual pose capsule routing is provided, including:

[0007] Step S100: Construct an image classification network based on a residual pose routing capsule network; wherein,

[0008] Image classification networks based on residual pose routing capsule networks include a basic feature extraction network and a residual pose routing capsule network.

[0009] The basic feature extraction network is used to extract basic depth features from the input image;

[0010] The residual pose routing capsule network includes a residual capsule routing module and a residual capsule classification module. The residual capsule routing module is used to extract and construct capsule features based on basic deep features, and the residual capsule classification module is used to classify based on capsule features; Step S200: Train the constructed image classification network;

[0011] Step S300: Classify the image to be detected based on the trained image classification network.

[0012] Furthermore, in step S200, the error function Loss used when training the image classification network is... i for:

[0013]

[0014] Among them, a t Represents maximizing the activation value of the target class, a i Indicates the activation value of other classes, a t and a i The difference is less than the boundary m.

[0015] Further, in step S100, the basic feature extraction network includes a convolutional layer and an activation layer; the residual pose routing capsule network includes four residual capsule routing modules and one residual capsule classification module, wherein the residual capsule routing module includes a primary capsule layer and two residual pose routing layers, and the residual capsule classification module includes a primary capsule layer and a residual pose classification layer.

[0016] Further, step S300 includes:

[0017] Step S310: The basic feature extraction network performs convolutional activation operations on the image to be detected to extract basic depth features;

[0018] Step S320: After inputting the basic depth features into the residual capsule routing module of the residual pose routing capsule network, capsule features are extracted and constructed.

[0019] Step S330: Input the capsule features of the residual capsule routing module into the residual capsule classification module for capsule classification, and use the output capsule classification category number.

[0020] Furthermore, in step S310, the convolution activation operation is represented as follows:

[0021] F deep =f ReLU (f Conv (X))

[0022] Where X represents the detected image, f Conv (·) represents the convolution operation, f ReLU (·) represents the ReLU activation function, F deep This represents the extracted basic depth features.

[0023] Further, step S320 includes:

[0024] Step S321: Generate capsule features from the basic depth features using the primary capsule layer;

[0025] Two branches are created to calculate the pose matrix and activation values; the pose matrix is ​​calculated by applying a convolution operation to the input data; the activation values ​​are calculated by applying a convolution operation to the input data and performing a sigmoid activation function operation.

[0026] Concatenate the pose matrix and activation values ​​to form capsule features;

[0027] Step S322: Calculate the high-level capsule attitude matrix and activation values ​​using the residual attitude routing algorithm;

[0028] The method for calculating the attitude matrix of high-level capsules is as follows:

[0029] According to the protocol routing mechanism, the relationship between the attitude matrices of capsule i and capsule k in the same layer is considered to be:

[0030]

[0031] Among them, w ij and w kj These represent the correlation between capsule i and capsule k in the same layer and capsule j in the upper layer, respectively. Considering that these two weight matrices are obtained through learning, a learnable weight matrix w is used. ik replace Right now:

[0032] p k =p i w ik

[0033] Where, p i Let w represent the pose matrix of capsule i. ik This represents a weight matrix that can be learned through a deep learning network.

[0034] This yields the relevant expressions for the attitude matrices of capsule i and capsule k in the same layer;

[0035] Capsules i and k in layer L construct the pose matrix of the global capsule j in layer (L+1) that is familiar to them, and calculate it by vector addition:

[0036]

[0037] Generally, assuming there are N capsules in layer L, the expression for calculating the pose matrix of the capsule in layer (L+1) is:

[0038]

[0039] Among them, w i It is a learnable 4×4 dimensional weight matrix, where [·] represents the concatenation operation;

[0040] The method for calculating the activation value of the high-level capsule attitude is as follows:

[0041] After calculating the pose matrix of the (L+1) layer capsule, its activation values ​​can be obtained by implicitly encoding the pose matrix. A convolution operation is used to encode the pose matrix to obtain its implicit information, and then a sigmoid function is used to activate it to calculate the required activation values. The calculation expression is as follows:

[0042]

[0043] Among them, f sig (·) and f Conv (·) represents the Sigmoid function and the convolution operation, respectively.

[0044] Generally, assuming there are N capsules in layer L, the expression for calculating the activation value of the capsule in layer (L+1) is:

[0045]

[0046] Step S323: Utilize capsule integration to construct capsule features;

[0047] Each capsule is obtained by concatenating the pose matrix and activation values. Specifically, each capsule is obtained by concatenating the pose matrix and its corresponding eigenvalues ​​from each of the (L+1) layers. The expression is as follows:

[0048]

[0049] In this context, [·] represents the concatenation operation, and the superscript L+1 indicates the layer index, where l = 1, 2, ...;

[0050] Next, all capsules in layer (L+1) are integrated, and the overall features associated with all capsules are obtained using a single convolution operation and the ReLU activation function. The expression is as follows:

[0051]

[0052] Among them, f ReLU (·) represents the ReLU function, f Conv (·) represents a convolution operation, CapF... L+1 This represents the original characteristics obtained from capsule integration.

[0053] Furthermore, in step S330, the capsule integration part of the residual routing module is removed, capsule routing is performed again, and the number of capsule types output by the current layer is fixed as the final number of network categories.

[0054] Secondly, an image classification method system based on residual pose capsule routing is provided, including a memory and a processor; wherein, the memory stores a computer program, and when the program is executed by the processor, it can implement the above-mentioned image classification method based on residual pose capsule routing.

[0055] Thirdly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the above-described image classification method based on residual attitude capsule routing.

[0056] The present invention has the following beneficial effects:

[0057] I. This invention extracts basic deep features from the input image, then uses a pre-trained capsule network based on a residual pose routing algorithm to explore part-whole relationships in the scene, obtaining capsule features. Convolutional operations and activation functions are then used to learn the original capsule features. Finally, the learned features are input into pose routing classification to learn the image category. This invention employs a residual pose routing algorithm based on a sparse unidirectional connection mechanism, which significantly reduces the number of network parameters and computational complexity during the routing process, improving the ability to find representations of high-dimensional data distributions in complex scenes, thereby greatly accelerating the network's inference speed.

[0058] Second, this invention uses a capsule network based on a residual mechanism, which effectively avoids gradient vanishing and can be used for image classification in computer vision. Attached Figure Description

[0059] Figure 1 This is a flowchart of the method steps of the present invention;

[0060] Figure 2 This is a flowchart illustrating the implementation of the method of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] Please refer to Figure 1 and Figure 2 This invention relates to an image classification method based on residual attitude capsule routing, comprising: step S100: constructing an image classification network based on a residual attitude routing capsule network; wherein,

[0063] Image classification networks based on residual pose routing capsule networks include a basic feature extraction network and a residual pose routing capsule network.

[0064] The basic feature extraction network is used to extract basic depth features from the input image;

[0065] The residual pose routing capsule network includes a residual capsule routing module and a residual capsule classification module. The residual capsule routing module is used to extract and construct capsule features based on basic deep features, and the residual capsule classification module is used to classify based on capsule features.

[0066] Step S200: Train the constructed image classification network;

[0067] Step S300: Classify the image to be detected based on the trained image classification network.

[0068] The following sections respectively address... Figure 1 The specific steps in the process will be explained in detail.

[0069] In step S100, the basic feature extraction network includes a convolutional layer and an activation layer. The basic feature extraction network is used to extract basic multi-channel depth features from the input image.

[0070] The residual attitude routing capsule network consists of four residual capsule routing modules and one residual capsule classification module. Each residual capsule routing module includes a primary capsule layer and two residual attitude routing layers, and the residual capsule classification module includes a primary capsule layer and a residual attitude classification layer. The residual attitude routing capsule network is used to extract part-to-whole relationship information at multiple scales.

[0071] In step S200, when training the image classification network, based on the activation values ​​of the classification capsules, this invention uses the propagation loss function, i.e., the error function Loss. i The ResCaps network for training residual pose capsule routing is trained; specifically, the propagation loss is used to maximize the target a. t The difference between the activation value and the edited m activation value, if an error class ai Closer to a than the marginal m t Therefore, the propagation loss can be expressed as:

[0072]

[0073] Among them, Loss i Let represent the propagation loss for class i, and Loss represent the propagation loss for all classes; a i and a t Let represent the probabilities of class i and class t, and m represent the boundary value, which is initially 0.2 and increases linearly to 0.9 during training.

[0074] In step S300, step S300 includes:

[0075] Step S310: Input the image to be detected into a convolutional network and a ReLU activation layer of the basic feature extraction network. The basic feature extraction network performs convolutional activation operations on the image to be detected to extract basic depth features.

[0076] In this embodiment, the input image is first fed into a convolutional layer with a kernel size of 5×5, a stride of 1, and padding of 2. Then, ReLU activation is used to obtain depth features, resulting in a depth feature size of 352×352 with 32 feature channels. The specific process is as follows:

[0077] F deep =f ReLU (f Conv (X))

[0078] Where X represents the image to be detected, f Conv (·) represents the convolution operation, f ReLU (v) represents the ReLU activation function, F deep This represents the extracted basic depth features.

[0079] Step S320: After inputting the basic depth features into the residual capsule routing module of the residual pose routing capsule network, capsule feature extraction and assembly are performed; specifically including:

[0080] Step S321: Use the primary capsule layer to generate capsule features from the basic depth features, and construct the capsule pose matrix and activation values;

[0081] Specifically, two branches are created to calculate the pose matrix and activation values. The pose matrix is ​​calculated by applying a convolution operation to the input data. The activation values ​​are calculated by applying a convolution operation and the sigmoid activation function to the input data. The pose matrix and activation values ​​are then concatenated to form a capsule feature.

[0082] In this embodiment, the primary capsule attitude matrix is ​​constructed as follows:

[0083]

[0084] The method for constructing the initial capsule activation value is as follows:

[0085] a = f Sig {p}

[0086] in and This is the pose matrix and activation values ​​of the capsule network, where W, H, and C represent the width, height, and number of capsule types, respectively, and D = {D}. P =16,D A =1} is the dimension of the pose matrix and activation values, specifically, D P =16 indicates the dimension of the attitude matrix, D A =1 indicates the dimension of the activation value. Re represents the shape transformation operation, f Conv (·) represents the convolution operation, f Sig (·) represents the Sigmoid function. Specifically, when constructing activation values, the Sigmoid function is used to map the activation value a to (0,1).

[0087] Step S322: Calculate the high-level capsule attitude matrix and activation values ​​using the residual attitude routing algorithm;

[0088] The method for calculating the attitude matrix of high-level capsules is as follows:

[0089] According to the protocol routing mechanism, the relationship between the attitude matrices of capsule i and capsule k in the same layer is considered to be:

[0090]

[0091] Among them, w ij and w kj These represent the correlation between capsule i and capsule k in the same layer and capsule j in the upper layer, respectively. Considering that these two weight matrices are obtained through learning, a learnable weight matrix w is used. ik replace Right now:

[0092] p k =p i w ik

[0093] Where, p i Let w represent the pose matrix of capsule i. ik This represents a weight matrix that can be learned through a deep learning network.

[0094] This yields the relevant expressions for the attitude matrices of capsule i and capsule k in the same layer;

[0095] The pose matrices of capsules i and k in layer L are constructed from the pose matrix of the global capsule j in layer (L+1), which is familiar to them, and are calculated using a simple and efficient vector addition:

[0096]

[0097] Generally, assuming there are N capsules in layer L, the expression for calculating the pose matrix of the capsule in layer (L+1) is:

[0098]

[0099] Among them, w i It is a learnable 4×4 dimensional weight matrix, where [·] represents the concatenation operation;

[0100] The method for calculating the activation value of the high-level capsule attitude is as follows:

[0101] After calculating the pose matrix of the (L+1) layer capsule, its activation values ​​can be obtained by implicitly encoding the pose matrix. A convolution operation is used to encode the pose matrix to obtain its implicit information, and then a sigmoid function is used to activate it to calculate the required activation values. The calculation expression is as follows:

[0102]

[0103] Among them, f Sig (·) and f Conv (·) represents the Sigmoid function and the convolution operation, respectively.

[0104] Generally, assuming there are N capsules in layer L, the expression for calculating the activation value of the capsule in layer (L+1) is:

[0105]

[0106] Where p and a are the pose matrix and activation value of the capsule network, the subscript i indicates the capsule constructed by the primary capsule network layer, and f Sig f represents the Sigmoid function. Conv This represents convolution computation. Activation values ​​are constructed by encoding the implicit information of the pose matrix, and the sigmoid function is used to map the activation values ​​a to the range (0,1).

[0107] Step S323: Utilize capsule integration to construct capsule features;

[0108] Each capsule is obtained by concatenating the pose matrix and activation values. Specifically, each capsule is obtained by concatenating the pose matrix and its corresponding eigenvalues ​​from each of the (L+1) layers. The expression is as follows:

[0109]

[0110] In this context, [·] represents the concatenation operation, and the superscript L+1 indicates the layer index, where l = 1, 2, ...;

[0111] Similarly, it is impossible to calculate the pose matrix of each capsule individually. We integrate the learned capsules into layer L+1, that is, we integrate all the capsules in layer (L+1). Using a single convolution operation and the ReLU activation function, we obtain the overall features associated with all capsules, expressed as:

[0112]

[0113] Among them, f ReLU (·) represents the ReLU function, f conv (·) represents a convolution operation, CapF... L+1 This represents the original characteristics obtained from capsule integration.

[0114] Step S330: Input the capsule features of the residual capsule routing module into the residual capsule classification module for capsule classification, and use the output capsule classification category number.

[0115] The capsule integration part of the residual routing module is removed, and capsule routing is performed again. The number of capsule types output by the current layer is fixed as the final number of classifications in the network. Specifically, the pose matrix and activation values ​​in the input data are calculated. The convolutional layer is used to calculate the pose matrix, and the sigmoid function of the convolutional kernel is used to calculate the activation values. After concatenating the pose matrix and activation values ​​to obtain capsule features, the pose class layer performs capsule classification. Specifically, the structure of the pose class layer ResPC is similar to ResP, except that the output capsule type of the pose class layer in this invention is the classification category number, and it does not require integration by the final convolution and activation functions.

[0116] The technical effects of the present invention will be further explained below with reference to simulation experiments.

[0117] 1. Simulation conditions: We implemented the proposed network in PyTorch. During training, we used the Adam optimizer with an initial learning rate of 0.01. The model was trained on a NVIDIA 3090Ti GPU with a batch size of 128. The capsule type numbers for the five blocks were set to [32, 32, 16, 16, N]. c ]

[0118] 2. Simulation Content and Result Analysis

[0119] The present invention and related capsule-based CNN networks were used in image classification experiments on public image datasets MNIST, AfFNIST, SmallNORB, and CIFAR-10 / 100. Some results are shown in Table 1 for intuitive comparison.

[0120] Table 1. Comparison of CNN networks used in classifying public image datasets for this invention and related capsule works.

[0121]

[0122] As can be seen from Table 1, compared with the existing technology, the present invention has a better effect on image classification and is more accurate in complex scenes.

[0123] In another embodiment, the present invention also provides an image classification method system based on residual attitude capsule routing, including a memory and a processor; wherein the memory stores a computer program, which, when executed by the processor, can implement the above-described image classification method based on residual attitude capsule routing.

[0124] In another embodiment, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described image classification method based on residual pose capsule routing.

[0125] All parts not covered in this invention are the same as or implemented using existing technologies.

[0126] The above description, in conjunction with specific embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. An image classification method based on residual pose capsule routing, characterized in that: include: Step S100: Construct an image classification network based on a residual pose routing capsule network; wherein, Image classification networks based on residual pose routing capsule networks include a basic feature extraction network and a residual pose routing capsule network. The basic feature extraction network is used to extract basic depth features from the input image; The residual pose routing capsule network includes a residual capsule routing module and a residual capsule classification module. The residual capsule routing module is used to extract and construct capsule features based on basic deep features, and the residual capsule classification module is used to classify based on capsule features. Step S200: Train the constructed image classification network; Step S300: Classify the image to be detected based on the trained image classification network; In step S100, the basic feature extraction network includes a convolutional layer and an activation layer; the residual pose routing capsule network includes four residual capsule routing modules and one residual capsule classification module, wherein the residual capsule routing module includes a primary capsule layer and two residual pose routing layers, and the residual capsule classification module includes a primary capsule layer and a residual pose classification layer. Step S300 includes: Step S310: The basic feature extraction network performs convolutional activation operations on the image to be detected to extract basic depth features; Step S320: After inputting the basic depth features into the residual capsule routing module of the residual pose routing capsule network, capsule features are extracted and constructed. Step S330: Input the capsule features of the residual capsule routing module into the residual capsule classification module for capsule classification, and use the output capsule classification category number; Step S320 includes: Step S321: Generate capsule features from the basic depth features using the primary capsule layer; Two branches are created to calculate the pose matrix and activation values; the pose matrix is ​​calculated by applying a convolution operation to the input data; the activation values ​​are calculated by applying a convolution operation to the input data and performing a sigmoid activation function operation. Concatenate the pose matrix and activation values ​​to form capsule features; Step S322: Calculate the high-level capsule attitude matrix and activation values ​​using the residual attitude routing algorithm; The method for calculating the attitude matrix of high-level capsules is as follows: According to the protocol routing mechanism, the relationship between the attitude matrices of capsule i and capsule k in the same layer is considered to be: Among them, w ij and w kj These represent the correlation between capsule i and capsule k in the same layer and capsule j in the upper layer, respectively. Considering that these two weight matrices are obtained through learning, a learnable weight matrix w is used. ik replace Right now: p k =p i w ik Where, p i Let w represent the pose matrix of capsule i. ik This represents a weight matrix that can be learned through a deep learning network. This yields the relevant expressions for the attitude matrices of capsule i and capsule k in the same layer; Capsules i and k in layer L construct the pose matrix of the global capsule j in layer (L+1) that is familiar to them, and calculate it by vector addition: Assuming there are N capsules in layer L, the expression for calculating the pose matrix of the capsule in layer (L+1) is: Among them, w i It is a learnable 4×4 dimensional weight matrix, where [·] represents the concatenation operation; The method for calculating the activation value of the high-level capsule attitude is as follows: After calculating the pose matrix of the (L+1) layer capsule, its activation values ​​can be obtained by implicitly encoding the pose matrix. A convolution operation is used to encode the pose matrix to obtain its implicit information, and then a sigmoid function is used to activate it to calculate the required activation values. The calculation expression is as follows: Among them, f Sig (·) and f Conv (·) represent the Sigmoid function and the convolution operation, respectively; Assuming there are N capsules in layer L, the expression for calculating the activation value of the capsule in layer (L+1) is: Step S323: Utilize capsule integration to construct capsule features; Each capsule is obtained by concatenating the pose matrix and activation values. Specifically, each capsule is obtained by concatenating the pose matrix and its corresponding eigenvalues ​​of each (L+1) layer. The expression is as follows: In this context, [·] represents the concatenation operation, and the superscript L+1 indicates the layer index, where l = 1, 2, ...; Next, all capsules in layer (L+1) are integrated, and the overall features associated with all capsules are obtained using a single convolution operation and the ReLU activation function. The expression is as follows: Among them, f ReLU (·) represents the ReLU function, f Conv (·) represents a convolution operation, CapF... L+1 This represents the original characteristics obtained from capsule integration.

2. The image classification method based on residual attitude capsule routing according to claim 1, characterized in that: In step S200, the error function Loss is used when training the image classification network. i for: Among them, a t Represents maximizing the activation value of the target class, a i Indicates the activation value of other classes, a t and a i The difference is less than the boundary m.

3. The image classification method based on residual attitude capsule routing according to claim 1, characterized in that: In step S310, the convolution activation operation is represented as follows: F deep =f ReLU (f Conv (X)) Where X represents the detected image, f Conv (·) represents the convolution operation, f ReLU (·) represents the ReLU activation function, F deep This represents the extracted basic depth features.

4. The image classification method based on residual pose capsule routing according to claim 1, characterized in that: In step S330, the capsule integration part of the residual routing module is removed, capsule routing is performed again, and the number of capsule types output by the current layer is fixed as the final number of network categories.

5. An image classification system based on residual pose capsule routing, characterized in that: It includes a memory and a processor; wherein the memory stores a computer program, which, when executed by the processor, can implement the image classification method based on residual attitude capsule routing as described in any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image classification method based on residual attitude capsule routing as described in any one of claims 1 to 4.

Images