Image recognition method, device and medium based on sparse constrained reconstruction capsule network

By using sparse constraint reconstruction capsule network in image recognition, dynamic routing is achieved by optimizing reconstruction errors, which solves the shortcomings of convolutional neural networks in spatial relationship processing, improves recognition performance and efficiency, and is used in traditional neural networks. Less parameters to achieve better results.

CN114648657BActive Publication Date: 2024-11-15NANJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210151100.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-14
Publication Date
2024-11-15
Estimated Expiration
2042-02-14

AI Technical Summary

Technical Problem

The prior art has insufficient performance and efficiency in image recognition, especially when dealing with spatial relationships, the filters of convolutional neural networks are difficult to understand the relative position information on the fine-grained image level, and the clustering method and inverse in dynamic routing There are contradictions in propagation training, and the sparsity of the weight matrix affects network training.

Method used

The capsule network based on sparse constraint reconstruction is adopted to achieve dynamic routing by optimizing reconstruction errors, embedded in the WideResNet backbone network, and the pose and routing weights are alternately updated using an adaptive iterative method to ensure the theoretical convergence and efficiency of the routing algorithm.

Benefits of technology

Improve the performance and efficiency of image recognition, use fewer model parameters to achieve better recognition effect, and the routing algorithm can be embedded in mainstream depth models, reducing the amount of parameters and training time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114648657B_ABST
    Figure CN114648657B_ABST
Patent Text Reader

Abstract

From the perspective of feature compression and reconstruction, the present invention proposes an image recognition method, device and medium based on sparse constrained reconstruction capsule network, which applies a dynamic routing algorithm with sparsity constraints and embeds the algorithm into a traditional convolutional neural network to realize an image recognition method. The method uses a closed-form solution to alternately update the posture and routing weights, ensuring the theoretical convergence of dynamic routing. Thanks to the strict convergence guarantee, adaptive iteration can be used to further improve the efficiency of routing. The image recognition method proposed by the present invention is superior to the existing image recognition method based on capsule routing in terms of performance and efficiency, and achieves better network performance using fewer model parameters.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an image recognition method, device and medium based on sparse constrained reconstruction capsule network, belonging to the technical field of deep learning. Background Art

[0002] The essence of modern convolutional neural networks is the multiplication or addition of a large number of vectors and matrices, so the computational consumption of neural networks is very high; the network transmits key information of the image through convolution and pooling. Admittedly, the convolutional network has very good experimental results when classifying images that are very close to the dataset, but it has obvious defects in handling spatial relationships. The essence of the problem is that the filters in the network understand the image at a fine-grained level, and the relative position information between the image components is lost.

[0003] The recently proposed capsule network is very good at processing various types of visual signals and encoding information such as posture (position, size, direction), deformation, hue and texture. The capsule network uses vectors or matrices as input and output, and adopts a dynamic routing mechanism for parameter update. It can obtain spatial position information and use its directional posture vector to encode related attributes as auxiliary decision-making. At the same time, this structural information can also be used as additional regularization in the model learning process to further improve the performance of deep models.

[0004] An important function of capsule routing is to compress information through the routing weight matrix to obtain output capsules with high representativeness. Clustering methods have been proven to be a successful measure to solve such problems in capsule routing scenarios. However, the main problem with directly applying clustering methods in dynamic routing is the contradiction between clustering methods and back-propagation training. On the other hand, the higher the sparsity of the weight matrix, the more unfavorable it is for the training of the entire network. Using the softmax function on the routing weights can alleviate this contradiction. However, the calculation of the Gaussian kernel has high complexity and low numerical stability. The image recognition technology of the existing technology has great deficiencies in performance and efficiency. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art and provide an image recognition method, device and medium based on sparse constrained reconstruction capsule network, which can effectively improve the performance and efficiency of image recognition.

[0006] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0007] In a first aspect, the present invention provides an image recognition method based on a sparse constrained reconstruction capsule network, the method comprising the following steps:

[0008] Acquire an image of a target to be identified;

[0009] Inputting the image of the target to be identified into a pre-trained image recognition model;

[0010] Output recognition results;

[0011] The image recognition model is obtained based on neural network training. The neural network includes a capsule network layer. The capsule network layer is used to realize dynamic routing of features and output low-dimensional features to subsequent network layers for image recognition. The dynamic routing is realized by optimizing the reconstruction error. The formula for optimizing the reconstruction error is:

[0012]

[0013] st||r i ||2≤∈,||y j ||2≤1,

[0014] in, is the input capsule structure of the capsule network layer, y j is the output capsule structure of the capsule network layer; i∈{1...I} and j∈{1...J} are the input and output indices respectively; r ij is the routing weight, ||·||2 represents the length of the vector, and r is r ij The vector mode of Y and R is r ij and j The matrix form of , ∈ is a set normal number.

[0015] Furthermore, the neural network includes a backbone network composed of WideResNet, and the capsule network layer is embedded before the final global average operation of WideResNet.

[0016] Furthermore, the training method of the image recognition model includes:

[0017] Constructing a recognition training data set; the recognition training data set

[0018] According to the recognition training data set, train the corresponding neural network

[0019] Furthermore, the implementation of the dynamic routing includes the following steps:

[0020] The output feature map of the WideResNet backbone network is grouped into capsules to obtain the capsule structure;

[0021] Forward-expanding the capsule structure to obtain an expanded capsule structure;

[0022] The expanded capsule structure is used as the input of the capsule network layer, and the output capsule structure is obtained through dynamic routing reconstruction;

[0023] The output capsule structure is reorganized to regain the WideResNet backbone network feature map.

[0024] Furthermore, the method of capsule grouping the output feature map of the WideResNet backbone network includes:

[0025] The output feature map is grouped into capsules according to the channel dimension: From the channel dimension, each feature map is divided into multiple capsules.

[0026] Furthermore, the method for forward extending the capsule structure includes:

[0027] By dividing the capsule {x i |i=1,...,I} and transformation weight w ij Multiply to get the expanded capsule structure Among them, J is the number of compressed capsules finally generated, and I is the number of capsules obtained.

[0028] Furthermore, the method of obtaining the output capsule structure through dynamic routing reconstruction includes:

[0029] For the transformed capsule Perform reconstruction based on sparse constraints: The following algorithm is used to achieve Figure 2 The process of reconstructing candidate capsules into new output capsules:

[0030] Step A: First, initialize the input and output. The formula is:

[0031]

[0032] Among them, r ij is the routing weight, ||·||2 represents the length of the vector, and the input candidate capsule is The output capsule is y j ;

[0033] Step B: Pass and r ij y can be calculated j , the specific formula can be expressed as

[0034]

[0035] Step C: For the obtained y j Normalized, the specific formula can be expressed as

[0036]

[0037] Step D: Pass and the normalized y j Reverse ij, the specific formula can be expressed as

[0038]

[0039] Step E: Calculate the reconstruction error When the last reconstruction error old If the reconstruction error Error minus this one is less than the threshold ε, the iteration stops; otherwise, the iteration steps BD are continued until convergence.

[0040] Furthermore, the implementation of the dynamic routing also includes: reorganizing the output capsule structure and calculating activation, and the activation can be directly mapped to a specific label for classification, and the formula is:

[0041]

[0042] where a j is the activation value of the capsule, sigmoid(·) is the nonlinear activation function, k j and b j are learnable weights.

[0043] In a second aspect, the present invention provides an image recognition device, comprising:

[0044] Input module: used to obtain the image of the target to be identified;

[0045] Recognition module: used to input the image of the target to be recognized into a pre-trained image recognition model;

[0046] Output module: used to output recognition results;

[0047] The image recognition model is obtained based on neural network training. The neural network includes a capsule network layer. The capsule network layer is used to realize dynamic routing of features and output low-dimensional features to subsequent network layers for image recognition. The dynamic routing is realized by optimizing the reconstruction error. The formula for optimizing the reconstruction error is:

[0048]

[0049] st||r i ||2≤∈,||y j ||2≤1,

[0050] in, is the input capsule structure of the capsule network layer, y j is the output capsule structure of the capsule network layer; i∈{1...I} and j∈{1...J} are the input and output indices respectively; r ij is the routing weight, ||·||2 represents the length of the vector, and r is r ijThe vector mode of Y and R is r ij and j The matrix form of , ∈ is a set normal number.

[0051] In a third aspect, the present invention provides an image recognition device, comprising: a memory and a processor;

[0052] The memory is used to store programs;

[0053] The processor is used to execute the program to implement each step of the image recognition method described in the first aspect.

[0054] In a fourth aspect, the present invention provides a readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the image recognition method as described in the first aspect are implemented.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] 1. The present invention adopts a capsule network based on sparse constrained reconstruction for image recognition and realizes dynamic routing by optimizing the reconstruction error. This method embeds a capsule network layer in the traditional convolutional neural network; by alternately updating the posture and routing weights, the reconstruction error between the output and input capsules is minimized to realize the dynamic routing of the feature capsules.

[0057] 2. Based on the dynamic routing based on reconstruction, the present invention further proposes an adaptive iterative routing method. Different from the fixed number of iterations in the existing method, the adaptive iteration can independently decide when to complete the routing according to the convergence of the reconstruction error. The capsule network and routing method proposed in the present invention can achieve better image recognition performance with fewer model parameters based on the traditional neural network.

[0058] 3. The dynamic routing algorithm based on sparsity constraints proposed by the present invention adopts a closed-form solution to alternately update the posture and routing weights, which ensures the theoretical convergence of dynamic routing. The adaptive number of iterations is used to further improve the efficiency of routing.

[0059] 4. The routing algorithm of the present invention can also be embedded into mainstream deep models as a ready-to-use module, greatly improving the performance of the model and reducing the number of parameters and training time. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a schematic diagram of the neural network structure used in the present invention;

[0061] Figure 2 Schematic diagram of the capsule network algorithm architecture of the present invention. DETAILED DESCRIPTION

[0062] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0063] Embodiment 1:

[0064] This embodiment proposes an image recognition method based on sparse constrained reconstruction capsule network, which is implemented by the following technical solutions:

[0065] Obtain an image of the target to be recognized and construct a recognition training data set; train the corresponding neural network based on the constructed recognition image data set to obtain an image recognition model; input the image to be recognized into the trained neural network model; the model outputs the final image recognition result. The neural network described here adds a capsule network layer on the basis of a general convolutional neural network; the output feature groups before the capsule network layer form a capsule structure, and dynamic routing is performed in the capsule network to generate output capsules and activations, which are reorganized into traditional convolutional network feature maps, and forward calculations are continued; the entire network can be trained using a mature back-propagation algorithm. The neural network model used in the present invention is composed of a traditional WideResNet embedded in a capsule network layer, and the capsule network and its corresponding dynamic routing algorithm are implemented as follows: Figure 2 As shown, the specific implementation steps are as follows:

[0066] S1. Convolutional network feature extraction: The present invention is a module embedded in a convolutional neural network, whose input is the output X of the upper convolutional neural network, and its size is W×H×C, which respectively represent the width, height, and number of channels of the feature map.

[0067] S2. Group features into capsules according to the channel dimension: From the channel dimension, each feature of X with a length of C is divided into I capsules, that is, divided into W×H×I capsules, and the size of each capsule is like Figure 2 In the process of Feature to poses, taking a feature of a channel dimension as an example, a vector of length 32 is split into four poses of length 8.

[0068] S3, capsule forward expansion: The candidate capsule can be expressed as {x i |i=1,...,I}, where I is the number of capsules obtained in the S2 group. Then, by combining these capsules with the transformation weights w ij Multiply to get Where J is the number of compressed capsules generated in the end. This step is to provide a large number of affine versions for routing in the final capsule compression, and the number of capsules will increase from I to I×J. Figure 2In the process of converting poses to votes, the poses segmented in S2 are forward expanded to generate more votes, preparing for the subsequent reconstruction.

[0069] S4, reconstructing the input based on sparse constraints: The present invention implements the following algorithm: Figure 2 The process of reconstructing votes into new poses in the above example. Let the input votes be Output poses is y j .

[0070] S41. First, initialize the input and output. The formula is:

[0071]

[0072] where r ij is the routing weight, and ||·||2 represents the length of the vector.

[0073] S42, through and r ij y can be calculated j The specific formula can be expressed as

[0074]

[0075] S43, for the obtained y j Normalized, the specific formula can be expressed as

[0076]

[0077] S44, through and the normalized y j Reverse ij , the specific formula can be expressed as

[0078]

[0079] S45, iterate steps S42, S43, S44 to minimize the reconstruction error. The reconstruction error can be expressed as:

[0080]

[0081] st||r i ||2≤∈,||y j ||2≤1, (10)

[0082] in, is the input capsule structure of the capsule network layer, y j is the output capsule structure of the capsule network layer; i∈{1...I} and j∈{1...J} are the input and output indices respectively; rij is the routing weight, ||·||2 represents the length of the vector, and r is r ij The vector mode of Y and R is r ij and j The matrix form of , ∈ is a set normal number.

[0083] S46, the specific number of iterations is determined according to the threshold ε, the last reconstruction error Error old Subtract the current reconstruction error Error and iterate until it converges. old The algorithm converges when -Error<ε.

[0084] S5. Recombined routing capsule: The recombined compressed routing capsule can be directly mapped to a specific label for classification. The formula is

[0085]

[0086] where a j is the activation value of the capsule, sigmoid(·) is the nonlinear activation function, k j and b j The weights can be learned. In addition, the experimental results show that y j It can also be used as the input of the next layer.

[0087] First, in order to verify the capsule routing algorithm based on sparse constraint reconstruction, the present invention conducts experiments on MNIST, smallNORB, SVHN, and CIFAI10 datasets, and compares CNN baseline, original dynamic routing algorithm, routing method based on expected maximum algorithm, and routing algorithm based on mean shift. In the algorithm implementation, all poses are 16-dimensional vectors. During the affine transformation, the pose size becomes 4×4 and is multiplied with the W weight of the same size.

[0088] In order to compare different routing methods fairly. A relatively small model was designed for the experiment, with only two convolutional layers and a convolution kernel size of 9×9. The output channels of each layer are 64 and 128 respectively. The stride of the first convolution kernel is 1, while the stride of the other convolution kernel is 2. For the MNIST dataset, an unpadded convolution method is used with an output size of 6×6×128. For other datasets, padded convolution is used with an output size of 8×8×128. Margin loss and adam optimizer are used for training, and training is performed for 200 rounds. As shown in Table 1, the experimental results show that the present invention has improvements in both performance and speed.

[0089] Table 1 Performance comparison of different methods on three common standard image recognition datasets

[0090]

[0091]

[0092] At the same time, the routing algorithm of the present invention can also be embedded into mainstream deep models as a ready-to-use module, and combined with traditional convolutional networks to form a complete image recognition model. Taking the deep network composed of WideResNet backbone network and capsule as an example, Table 2 compares the specific performance of different configuration structures. It can be seen that when the channel width scaling factor is 10, WRN-16-4 performs best on the SVHN dataset. It is worth noting that the performance of this model is much better than the best result of WRN, WRN-16-8, and the number of parameters and training time are significantly reduced.

[0093] Table 2 Recognition performance of image recognition models composed of WideResNet and capsule networks with different configurations on SVHN

[0094] method Recognition error rate Number of parameters Training time WRN-16-2 1.68% 692K 0.068s WRN-16-2+ capsule (np=10) 1.68% 782K 0.086s WRN-16-2+ capsule (np=20) 1.61% 865K 0.112s WRN-16-2+ capsule (np=40) 1.64% 1,030K 0.169s WRN-16-2+ capsule (np=80) 1.65% 1,370K 0.278s WRN-16-4 1.64% 2.750K 0.097s WRN-16-4+ capsule (np=10) 1.53% 2,930K 0.146s WRN-16-4+ capsule (np=20) 1.58% 3,090K 0.201s WRN-16-4+ capsule (np=40) 1.69% 3,420K 0.313s WRN-16-8 1.60% 10,960K 0.195s WRN-16-8+ capsule (np=10) 1.58% 11,320K 0.303s WRN-16-8+ capsule (np=20) 1.56% 11,560K 0.412s WRN-16-8+CapProNet[4] 1.54% 2,700K -

[0095] In summary, the present invention proposes an image recognition method based on sparse constrained reconstruction capsule network. The neural network model adopted by the method is composed of a traditional convolutional network embedded in a capsule network layer. The capsule network uses a closed-form solution to alternately update the posture and routing weights to achieve dynamic routing, ensuring the theoretical convergence of the routing algorithm. And the routing efficiency is further improved by using an adaptive number of iterations. Experimental results on different data sets show that the image recognition method proposed by the present invention is superior to existing methods in both performance and efficiency, and has fewer model parameters and better network performance.

[0096] This method embeds a capsule network layer in the traditional convolutional neural network; by alternately updating the posture and routing weights, the reconstruction error between the output and input capsules is minimized to achieve dynamic routing of feature capsules. At the same time, based on the dynamic routing based on reconstruction, the present invention further proposes an adaptive iterative routing method. Different from the fixed number of iterations in the existing methods, the adaptive iteration can autonomously decide when to complete the routing according to the convergence of the reconstruction error. Experimental results show that the capsule network and routing method proposed in this paper can achieve better image recognition performance with fewer model parameters based on the traditional neural network.

[0097] Embodiment 2:

[0098] This embodiment provides an image recognition device, including:

[0099] Input module: used to obtain the image of the target to be identified;

[0100] Recognition module: used to input the image of the target to be recognized into a pre-trained image recognition model;

[0101] Output module: used to output recognition results;

[0102] The image recognition model is obtained based on neural network training. The neural network includes a capsule network layer. The capsule network layer is used to realize dynamic routing of features and output low-dimensional features to subsequent network layers for image recognition. The dynamic routing is realized by optimizing the reconstruction error. The formula for optimizing the reconstruction error is:

[0103]

[0104] st||r i ||2≤∈,||y j ||2≤1,

[0105] in, is the input capsule structure of the capsule network layer, y j is the output capsule structure of the capsule network layer; i∈{1...I} and j∈{1...J} are the input and output indices respectively; r ij is the routing weight, ||·||2 represents the length of the vector, and r is r ij The vector mode of Y and R is r ij and j The matrix form of , ∈ is a set normal number.

[0106] This embodiment can be used to implement the method described in the first embodiment.

[0107] Embodiment three:

[0108] This embodiment provides an image recognition device, including: a memory and a processor;

[0109] The memory is used to store programs;

[0110] The processor is used to execute the program to implement each step of the image recognition method described in the first embodiment.

[0111] Embodiment 4:

[0112] This embodiment provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, each step of the image recognition method described in the first embodiment is implemented.

[0113] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0114] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0115] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0117] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An image recognition method based on sparse constrained reconstruction capsule network, It is characterized in that The method comprises the following steps: Acquire an image of a target to be identified; Input the image to be recognized into the pre-trained image recognition model; Output recognition results; The image recognition model is obtained based on neural network training. The neural network includes a capsule network layer. The capsule network layer is used to realize dynamic routing of features and output low-dimensional features to subsequent network layers for image recognition. The dynamic routing is realized by optimizing the reconstruction error. The specific formula for optimizing the reconstruction error can be expressed as: s.t.||r i || 2 ≤∈,||y j || 2 ≤1, in, is the input capsule structure of the capsule network layer, y j is the output capsule structure of the capsule network layer; i∈{1...I} and j∈{1...J} are the input and output indices respectively; r ij is the routing weight, ||·|| 2 represents the length of the vector, r is r ij The vector mode of Y and R is r ij and j The matrix form of , ∈ is a set normal number.

2. The image recognition method according to claim 1, It is characterized in that The neural network includes a backbone network composed of WideResNet, and the capsule network layer is embedded before the final global average operation of WideResNet.

3. The image recognition method according to claim 2, It is characterized in that The implementation of the dynamic routing specifically includes the following steps: The output feature map of the WideResNet backbone network is grouped into capsules to obtain the capsule structure; Forward-expanding the capsule structure to obtain an expanded capsule structure; The expanded capsule structure is used as the input of the capsule network layer, and the output capsule structure is obtained through dynamic routing reconstruction; The output capsule structure is reorganized to regain the WideResNet backbone network feature map.

4. The image recognition method according to claim 3, It is characterized in that The implementation of the dynamic routing also includes reorganizing the routing capsules.

5. An image recognition device, It is characterized in that include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the image recognition method according to any one of claims 1 to 4.

6. A readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, each step of the image recognition method according to any one of claims 1 to 4 is implemented.