Image recognition method based on self-adaptive stacking capsule auto-encoder

Abstract

The invention provides an image recognition method based on a self-adaptive stacking capsule auto-encoder. The method comprises the following steps: acquiring an image to be analyzed; a self-adaptive stacking capsule auto-encoder (ASCAE) is constructed; an image to be analyzed is input into the ASCAE; and reconstructing the object by using the ASCAE, and outputting an identified image. The ASCAE comprises a component capsule auto-encoder (PCAE) and an object capsule auto-encoder (OCAE), and an adaptive module (AM) is arranged in a decoder of the PCAE. And a decoder of the PCAE maps the image features of the component capsule into an image space through an adaptive module (AM) to form a block of a local image, the posture of the block is changed through geometric transformation, and finally a reconstructed image is formed. Compared with a fixed template of SCAE, the adaptive module AM has higher capability of expressing complex images, and experimental results show that the ASCAE can extract effective image feature information, so that the classification accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to an image recognition method based on an adaptive stacked capsule autoencoder. Background Art

[0002] Capsule network (CapsNet) is an artificial neural network based on vector neurons (capsules). CapsNet can learn the spatial relationship between objects in an image. However, CapsNet has a flaw, that is, the spatial geometric information extracted by the capsule network is difficult to interpret. Currently, it can only be proved through experiments that the extracted capsules contain spatial information, but it is impossible to explain how the capsules represent the posture of the object. To solve this problem, Kosiorek et al. proposed a stacked capsule autoencoder (SCAE).

[0003] The SCAE model is divided into two parts: the component capsule autoencoder (PCAE) and the object capsule autoencoder (OCAE). The PCAE encoder is responsible for extracting capsules from the input image. The information in the capsule contains the special features, pose, and probability of existence of a certain part in the image. The template for drawing the image parts is defined in the PCAE decoder, which uses the pose and existence probability of the parts extracted by the encoder to reconstruct the input image, that is, to combine the parts into the original image. Unlike other capsule models, these poses are 6-dimensional vectors that describe translation, rotation, scaling, and shearing. The image reconstruction process is basically the same as the image construction process in computer graphics. In OCAE, special features are converted into object capsules. These object capsules represent objects present in the image. They also contain the pose of the object, but do not contain appearance information, that is, they are not used to reconstruct the image. In general, SCAE uses poses, templates, and existence probabilities to describe images, which is an intuitive and interpretable approach. And SCAE is the first unsupervised capsule model.

[0004] Although SCAE has achieved excellent results in image reconstruction and unsupervised classification tasks, its effectiveness is limited to simple images such as MNIST (a dataset of handwritten digit images). On complex datasets, SCAE performs much worse. For example, on the CIFAR10 dataset (a dataset of images of real objects), SCAE cannot reconstruct meaningful images and has low unsupervised classification accuracy. This is because MNIST only contains handwritten digits with a pure black background. These images have low noise and are easy to reconstruct and classify. However, CIFAR10 images contain complex appearance and background. This results in CIFAR10 images having complex structures and high noise. In SCAE, the template consists of trainable parameters, but the number is limited. After the model converges, the parameters of the template also tend to be stable and cannot be changed according to the characteristics of the image. Therefore, Kosiorek et al. call it a fixed template. However, the expressiveness of the fixed template is not enough to model complex images, resulting in the inability of SCAE to effectively reconstruct complex images such as CIFAR10 images. Summary of the invention

[0005] In view of the shortcomings of the above-mentioned prior art, the present invention proposes an image recognition method based on an adaptive stacked capsule autoencoder, which constructs a capsule autoencoder ASCAE based on an adaptive method. In the ASCAE, the PCAE encoder part first converts the image into a component capsule. The present invention improves the data structure inside the component capsule to achieve the decoupling of the information used to describe the local image from the special features.

[0006] The image recognition method based on the adaptive stacked capsule autoencoder proposed in the present invention comprises the following steps:

[0007] Step 1: Obtain the image to be analyzed;

[0008] Step 2: Build an adaptive stacked capsule autoencoder ASCAE;

[0009] Step 3: Input the image to be analyzed into ASCAE;

[0010] Step 4: Use ASCAE to reconstruct the image, use the classifier to realize image recognition, and output the recognized image.

[0011] Furthermore, the adaptive stacked capsule autoencoder ASCAE in step 2 includes a component capsule autoencoder PCAE and an object capsule autoencoder OCAE, wherein the capsule autoencoder PCAE includes an encoder and a decoder, and an adaptive module AM ​​is set in the decoder of PCAE, and the adaptive AM module is a fully connected neural network composed of two fully connected layers.

[0012] Furthermore, a ResCNN deep convolution module and an attention pooling layer connected to its output are set in the encoder of the capsule autoencoder PCAE. The ResCNN deep convolution module is used to perform convolution calculation on the input original image and obtain a feature map, and output the feature map to the attention pooling layer. The attention pooling layer is used to convert the input feature map into a component capsule.

[0013] Furthermore, the component capsule includes a special feature component, an image feature component, an existence probability component and a posture component:

[0014] The existence probability component is used to describe the possibility of the component existing in the image;

[0015] The posture component is used to describe the spatial geometric information of the component;

[0016] Image feature component, used to represent the appearance information of the component;

[0017] Special feature components are used to describe other image features besides the appearance of the parts and serve as input data for OCAE.

[0018] Furthermore, the adaptive module AM ​​is used to draw the appearance of the component capsule according to the image feature map of the component capsule to form blocks of the local image, and these blocks then change their postures through geometric transformation to finally form a reconstructed image.

[0019] Furthermore, the ResCNN convolution module includes 4 residual blocks connected in sequence, each residual block includes three convolutional layers and a residual connection from the first convolutional layer to the third convolutional layer.

[0020] Furthermore, the specific operation steps of step 4 include:

[0021] Step 4.1: Get the input image I to be analyzed, and record it as I(h,w,c), where h is the height, w is the width, and c is the channel;

[0022] Step 4.2: Input the image I into the encoder of PCAE and convert I into M component capsules, namely:

[0023]

[0024] Among them, PE represents the encoder of PCAE, x represents the posture, d represents the existence probability, z s Indicates special features, z i Represents image features, and the subscript 1:M represents from 1 to M;

[0025] Step 4.3: The adaptive module AM ​​projects all M image features into M image blocks through the following projection formula, each Each contains the appearance information of an encoded component capsule, and the projection formula is:

[0026]

[0027] Among them, T m The height, width and channel are h t ,w t ,c;m∈(1,M); Represents the image features contained in the mth component capsule;

[0028] Step 4.4: PCAE decoder for each image block T m Corresponding to learning an alpha channel

[0029] Step 4.5: Based on the pose x in the part capsule m , T m and Perform affine transformation to obtain the transformed image block and

[0030]

[0031] Among them, Transform represents affine transformation;

[0032] Step 4.6: Use the following formula to convert Existence probability d m Perform joint calculation to obtain the reconstructed image:

[0033]

[0034] in, represents the reconstructed image, ⊙ represents the product of the corresponding elements in the image;

[0035] Step 4.7: Using the obtained reconstructed image and the original image I using the mean square error L mse Calculate the loss value of the reconstructed image and the original image to train the PCAE;

[0036] Step 4.8: Special features of the component capsule Input into OCAE and convert x m d m and Connect together to get a token, and encode the M tokens formed by the M component capsules into K object capsules by the encoder of OCAE;

[0037] Step 4.9: Use OCAE's object decoder to decode the corresponding object capsule into 5 components:

[0038] OV k ,a k ,OP k,1:M ,a k，1:M ,λ k,1:M =MLP k (c k )

[0039] Among them, MLP k represents the object decoder implemented by a multi-layer perceptron, OV k represents the 3×3 object-observer relationship matrix, c k represents the kth object capsule, a k Indicates c k The probability of existence of OP k，1:M represents a 3×3 object-component relationship matrix, a k，1:M represents the predicted existence probability of the component capsule, λ k，1:M represents the standard deviation of the correlation;

[0040] Step 4.10: Get the predicted pose μ of the kth object capsule to the mth component capsule according to the following formula: k,m :

[0041] μ k,m =OV k OP k,m

[0042] Among them, OV k represents the affine transformation between the object capsule and the image observer, OP k，m Represents the affine transformation between the object capsule and the component capsule;

[0043] Step 4.11: Model the pose as a single Gaussian mixture by:

[0044] p(x m |k,m)=N(x m |μ k,m ,λ k,m )

[0045] Where p represents Gaussian mixture, λ k，m represents the standard deviation of the mth component capsule;

[0046] Step 4.12: Calculate the pose likelihood L by the following formula pose , and by maximizing L pose To train OCAE, the pose likelihood L pose The calculation formula is:

[0047]

[0048] Among them, a k represents the existence probability of the kth object capsule, and all the existence probabilities a k Constitute an existence vector with k elements; a k,m represents the prediction result of the kth object capsule for the existence probability of the mth component capsule, a i,j represents the prediction result of the existence probability of the i-th object capsule to the j-th component capsule;

[0049] Step 4.13: By minimizing L mse and maximize L pose Establish the loss function of ASCAE:

[0050] L=L mse -L pose

[0051] Among them, L represents the loss function of ASCAE;

[0052] Step 4.14: Use the existence probability a of all object capsules k The existence vector formed represents the feature code of the input image I. The ASCAE model is trained based on the constructed loss function L to optimize the feature code. The classifier then uses the optimized feature code of the image I to realize image classification and output the image recognition result.

[0053] Therefore, the present invention adopts the above-mentioned image recognition method based on adaptive stacked capsule autoencoder, which has the following beneficial effects:

[0054] First, the present invention proposes a novel capsule autoencoder with an adaptive module, which can project the information in the capsule into a template, i.e., an image block. Through this dynamically generated template, high-quality complex images can be efficiently reconstructed. Moreover, there is no need to increase the number of templates.

[0055] Second, the present invention proposes an image feature component in the data structure of the capsule. Through the image feature component, the capsule realizes the decoupling of information between image appearance and object, so that PCAE focuses more on reconstructing images, while OCAE focuses more on discovering objects.

[0056] Finally, the experimental results show that ASCAE has better image reconstruction performance than SCAE on CIFAR10 and FMNIST. In addition, the performance of ASCAE on both unsupervised and supervised image classification tasks has been improved.

[0057] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is the architecture of the ASCAE model proposed in the present invention; wherein, the solid arrows represent information flow, the gradient flows in the opposite direction, and the dotted arrows represent information flow where the gradient stops.

[0059] Figure 2 This is the architecture of ResCNN.

[0060] Figure 3 (a)-(c) are reconstructed images on CIFAR10; where Figure 3 (a) is the original image in the dataset. Figure 3 (b) is the ASCAE reconstructed image, Figure 3 (c) Reconstructed image for SCAE.

[0061] Figure 4 (a)-(c) are reconstructed images on FMNIST; Figure 4 (a) is the original image in the dataset. Figure 4 (b) is the ASCAE reconstructed image, Figure 4 (c) Reconstructed image for SCAE.

[0062] Figure 5 (a)-(c) are the reconstructed images on the transformed CIFAR10; Figure 5 (a) is the original image in the dataset. Figure 5 (b) is the ASCAE reconstructed image, Figure 5 (c) Reconstructed image for SCAE. DETAILED DESCRIPTION

[0063] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art make various changes or modifications to the present invention, and these equivalent forms also fall within the scope limited by the claims attached to the application.

[0064] The present invention proposes a capsule autoencoder based on an adaptive method, which improves the data structure inside the component capsule. The improved component capsule includes four components: image features, existence probability, posture and special feature components. Among them, the image feature component is a vector describing a local image area. In the PCAE decoder of ASCAE, an adaptive module AM ​​is introduced, which is implemented by a spatial mapping function. AM projects the image features into the image space to form a block of the reconstructed image (i.e., the local image, which is also the image presented by the component). The existence probability component describes the existence possibility of the block to which it corresponds. The posture component describes the affine transformation of the block. All these components except the special feature component are used to reconstruct the image. Based on computer graphics, multiple sub-images in the same image space are combined into a complete image. The special feature component is used to describe some abstract information of the local image and does not participate in image reconstruction. The special feature is input into the OCAE of ASCAE to discover the object.

[0065] 1. Architecture of the ASCAE model

[0066] like Figure 1 As shown, the ASCAE model proposed in the present invention includes two autoencoders: a component capsule autoencoder (PCAE) and an object capsule autoencoder (OCAE). Among them, PCAE is used to reconstruct images, and OCAE is used to discover objects present in images.

[0067] (1) PCAE

[0068] Combination Figure 1 In the encoder of PCAE, the image is first input into the deep convolutional network ResCNN for convolution calculation and obtains the feature map. Then the obtained feature map is input into the attention-based pooling layer, and the feature map is converted into component capsules through the attention-based pooling layer.

[0069] like Figure 2 As shown in the figure, ResCNN is a deep convolutional network composed of four residual modules connected in sequence. Conv in the figure represents a convolutional layer. For example, Conv(5,1,64) means that it has 64 convolutions, 5×5 kernels, a step size of 1, and ⊕ represents an addition operation. Figure 2 As can be seen in Figure 1, ResCNN contains four identical residual blocks, each residual block contains three convolutional layers and a residual connection from the first convolutional layer to the third convolutional layer. In addition, each layer in ResCNN includes a batch normalization process.

[0070] The component capsule contains four components: special features, image features, existence probability and posture components, where:

[0071] The existence probability component describes the possibility of the capsule's existence;

[0072] The pose component describes the image patch information represented by the spatial geometry capsule;

[0073] Image feature component, used to represent the appearance information of the tile;

[0074] Special feature components, which do not participate in the image reconstruction process, are used to describe other properties of the capsule and serve as input data for OCAE.

[0075] Unlike SCAE, the independent structure of image features and special features of component capsules makes PCAE focus more on image reconstruction, while OCAE focuses more on high-level object description. This capsule data structure can solve the information interference caused by using only special features in SCAE.

[0076] The image features in each component capsule are mapped to the image space by AM in the decoder of PCAE. In the decoder of PCAE, the projected vector is reshaped into a tile representing the local image. The AM module is a fully connected neural network consisting of two fully connected layers, namely a multilayer perceptron network. The first layer consists of 56 neurons, and the number of neurons in the second layer is determined by the size of the reconstructed image. If a 32-size color image is used, the number of neurons should be 32×32×3=3072. Therefore, it can be trained to adaptively generate images. The AM module draws the appearance of the component (i.e., the tiles that make up the complete image) based on the input image feature information. These tiles change their posture through geometric transformation and finally generate a reconstructed image. This geometric transformation requires not only the existence of probability and posture components, but also an alpha channel. Each tile of the local image corresponds to an alpha channel, which represents transparency and is used to compose the image. The alpha channel is independent of the tile and is defined by trainable parameters. Finally, the mean square error (MSE) between the reconstructed image and the input image can be used to train the PCAE.

[0077] (2)OCAE

[0078] OCAE is similar in structure to the OCAE in SCAE in the paper "Kosiorek, AR; Sabour, S.; Teh, YW; Hinton, GE Stacked Capsule Autoencoders. In Proceed-ings of the Proc. Adv. Neural Inf. Process. Syst., 2019, Vol. 32, pp. 15486–15496". OCAE is used for pose prediction. These predicted poses and the poses in the component capsules are used to train OCAE.

[0079] 2. Algorithm implementation of ASCAE model

[0080] The input image I is defined as an image of shape (h,w,c), where h is the height, w is the width, and c is the channel. The image I is converted into M component capsules through the ResCNN convolution module and the attention-based pooling layer. The process can be expressed as:

[0081]

[0082] Among them, PE represents the encoder of PCAE, x represents the posture, d represents the existence probability, z s Indicates special features, z i Represents image features, subscript 1:M represents from 1 to M, superscript s represents special, and i represents image;

[0083] For example, capsule m consists of four parts: posture x m , existence probability d m , Special Features and image features Contains the appearance information of capsule m. The AM module converts the image features in the mth component capsule into Projected into image block T m :

[0084]

[0085] Among them, T m The height, width and channel are h t ,w t ,c.

[0086] In addition, the calculation of each component capsule is implemented by the same AM module, so that all component capsules can be projected into the same image space, which is conducive to image synthesis.

[0087] To more efficiently combine tiles into a reconstructed image, the PCAE decoder also mLearning an alpha channel

[0088] Based on T m and The process of image reconstruction includes the following two steps:

[0089] First, use x through equation (3) m To T m Perform affine transformation to obtain the transformed block because The posture should be consistent with T m Correspondingly, so Perform the same affine transformation to convert it to

[0090]

[0091] Among them, Transform represents affine transformation;

[0092] Secondly, the blocks are transformed according to the following formula Alpha channel Existence probability d m Perform joint calculation to obtain the reconstructed image:

[0093]

[0094] in, represents the reconstructed image, d m represents the existence probability, ⊙ is the product of the corresponding elements in the image;

[0095] Finally, the reconstructed image and the original image I using the mean square error L mse The loss values ​​of the reconstructed image and the original image are calculated to train the PCAE and continuously improve the image synthesis capability of the adaptive stacked capsule autoencoder.

[0096] The special features in the component capsules do not participate in the image reconstruction work. These features are used to discover objects that may exist in the image. OCAE uses special features as the main input information, combined with the existence probability and posture of the component capsules, to output object capsules that describe objects that may exist in the image.

[0097] Since OCAE needs the component capsules output by PCAE to find the objects in image I. In OCAE, firstly, the x of the mth component capsule is m d m and The M tokens of the M component capsules are connected together to form a token, and the M tokens of the M component capsules are encoded into K object capsules by the OCAE encoder Set Transformer. Different from the OCAE in the prior art, in the OCAE of the present invention, T m There is no need to input it into the Set Transformer to identify the identity of the component capsule, because the identity information of the component capsule m is already included in middle.

[0098] In OCAE, the target decoder is implemented by a multi-layer perceptron (MLP), and each object capsule corresponds to an MLP. k , through which the decoder converts the kth object capsule c k Decoded into 5 components:

[0099] OV k ,a k ,OP k,1:M ,a k，1:M ,λ k,1:M =MLP k (c k ) (5)

[0100] Among them, MLP k represents the target decoder implemented by a multi-layer perceptron, OV k represents the 3×3 object-observer relationship matrix, a k Indicates c k The probability of existence of OP k，1:M represents a 3×3 object-component relationship matrix, a k，1:M represents the predicted existence probability of the component capsule, λ k，1:M represents the standard deviation of the correlation.

[0101] Next, the predicted posture μ of the current object capsule k to the component capsule m is calculated by formula (6): k,m :

[0102] μ k,m =OV k OP k,m (6)

[0103] Among them, OV k represents the affine transformation between the object capsule and the image observer, OP k，m Represents the affine transformation between the object capsule and the component capsule;

[0104] In OCAE, the pose x m is modeled as a single Gaussian mixture p, with each pose x m are interpreted as coming from the object capsule c k Independent mixture of predictions:

[0105] p(x m |k,m)=N(x m |μ k,m ,λ k,m ) (7)

[0106] Among them, μ k,m and λ k，m are the center and standard deviation of the mixture components.

[0107] By maximizing the pose likelihood L pose To train OCAE, the pose likelihood L pose =p(x 1:M ,d 1:M ):

[0108]

[0109] Among them, a k represents the existence probability of the kth object capsule, and all the existence probabilities a k Constitute an existence vector with k elements; a k,m represents the prediction result of the kth object capsule for the existence probability of the mth component capsule, a i,j It represents the prediction result of the existence probability of the i-th object capsule to the j-th component capsule.

[0110] ASCAE can be achieved by minimizing L mse and maximize L pose To construct the loss function L of ASCAE, the formula of the loss function is:

[0111] L=L mse -L pose (9).

[0112] The existence vector can describe which objects may exist in the input image. Therefore, the existence vector is used to represent the feature code of the input image, and the ASCAE model is trained through the loss function L to optimize the feature code. Finally, the classifier uses the optimized feature code of the input image to realize image classification and output the image recognition result.

[0113] Specifically, the present invention uses a LIN-MATCH classifier or a LIN-PRED classifier. LIN-MATCH is an unsupervised classifier that first clusters feature codes and then implements bipartite graph matching to obtain classification. The LIN-PRED classifier is a supervised classifier, which is composed of a single-layer fully connected neural network. Before use, it needs to be trained with labeled images to enable it to have classification capabilities. The fully connected neural network can directly convert the input feature code into the category of the image, thereby realizing image recognition.

[0114] In summary, the adaptive stacked capsule autoencoder proposed in the present invention first decomposes the image into multiple components, and then recombine the components. In this process, it analyzes the various constituent objects that may exist in the image, and then uses the existence probability of these objects to describe the characteristics of the image, and finally realizes image recognition.

[0115] Example

[0116] In order to verify the method proposed in the present invention, the following experiments were carried out.

[0117] 1. Model selection

[0118] In PCAE, T m The width h t and height w t Both are 14. On CIFAR10, the number of channels c is 3. On FMNIST, c is 1. Alpha channel Size and T m The number of component capsules M is 32. The number of object capsules K is 64. Special features and image features The number of elements of are all 16. Posture x m The number of elements in is 6. The AM module is implemented by a network with two fully connected layers, the first layer has 256 units and the second layer has h t × t ×c units. Other hyperparameter configurations of the ASCAE model are the same as those of the SCAE.

[0119] The ASCAE model was trained using the RMSProp optimizer, with an initial learning rate of 0.00003. The learning rate strategy was to reduce the learning rate by 4% every 10,000 model parameter updates. Data augmentation was not used for the images in all the above datasets. Since it is difficult for SCAE to reconstruct the original image in CIFAR10, Kosiorek et al. used the image after Sobel filtering as the reconstruction target to emphasize the importance of shape. However, the ASCAE proposed in the present invention can reconstruct the original image. Therefore, the ASCAE model does not require image preprocessing, such as Sobel filter, on both the FMNIST dataset and the CIFAR10 dataset. The number of images used in one training is 128, and the number of training iterations is 800. The main loss function is shown in formula (9), and other auxiliary loss functions are the same as SCAE, such as prior sparsity and posterior sparsity.

[0120] 2. Dataset settings

[0121] This embodiment mainly studies the reconstruction and recognition performance of ASCAE on complex images. Therefore, two commonly used and representative data sets are used to evaluate ASCAE, namely CIFAR10 and FMNIST data sets. Among them, CIFAR10 is a collection of real-world images, which has 10 categories and 6000 samples in each category. It includes a training set and a test set, the training set has 50,000 samples, and the test set has 10,000 samples, each of which is a 3-channel color image with a size of 32×32. The images in CIFAR10 contain objects with complex shapes and complex background noise. Therefore, it is difficult to classify samples in CIFAR10 using an unsupervised model.

[0122] FMNIST is a collection of grayscale images, where objects are clothes, shoes, etc. The objects in FMNIST are derived from real objects, so their appearance structure is more complex than that of MNIST. FMNIST is also a dataset commonly used to evaluate capsule models. It has 10 categories, each with 7,000 samples. Its training set has 60,000 samples and the test set has 10,000 samples. Each sample is a single-channel grayscale image of size 28×28. In the experiment of this embodiment, in order to be consistent with CIFAR10, the images of FMNIST are rescaled to 32×32.

[0123] 3. Image reconstruction

[0124] In order to study the performance of ASCAE, the reconstructed images of the ASCAE model are compared with those of the SCAE. Figure 3 and Figure 4 shown. Figure 3 (a) shows the original image in the CIFAR10 dataset; Figure 3 (b) shows the reconstructed image of ASCAE for the CIFAR10 dataset; Figure 3 (c) shows the reconstructed image of SCAE for the CIFAR10 dataset. Figure 4 (a) shows the original image in the FMNIST dataset; Figure 4 (b) shows the reconstructed image of ASCAE for the FMNIST dataset; Figure 4 (c) shows the reconstructed image of SCAE for the FMNIST dataset. Figure 3 (b) and Figure 3 As can be seen from the comparison in (c), in CIFAR10, ASCAE is significantly better than SCAE. The reconstructed image of SACE consists of only some color blocks, and in some images, even the outline of the object cannot be distinguished. These results intuitively show the defects of SCAE's fixed template.

[0125] In SCAE, fixed templates are trained to describe parts of the image. However, when the training is completed, the parameters in the template are fixed. In other words, the patches that the template can express are fixed. In addition, in SACE, the number of fixed templates is limited. The images in CIFAR10 are complex and cannot be effectively described by limited fixed templates. In addition, fixed templates are independently defined variables and are not closely related to the input image. These reasons lead to Figure 3 The phenomenon shown in (c) in the figure. The ASCAE model's tiles are generated through a projection process, which is a mapping of image features to image space. The trained AM module has sufficient ability to generate complex local images, overcoming the above-mentioned defects of fixed templates. In addition, compared Figure 3 (a) and Figure 3 In (b), it can be seen that the reconstructed image of ASCAE is closer to the original image.

[0126] Figure 4 The reconstruction results of ASCAE and SCAE on FMNIST. Figure 4 As can be seen in (c), although SCAE is able to reconstruct the image, the color blocks are still visible. However, the reconstructed image of ASCAE is visually closer to the original image. Although the images in FMNIST are relatively simple and can be represented by a fixed template, ASCAE still outperforms SCAE.

[0127] 4. Image Classification

[0128] PCAE provides special feature information for OCAE, which is the key to image classification. Therefore, this embodiment evaluates the classification performance of ASCAE.

[0129] There are two methods for evaluating the classification performance of ASCAE: LIN-MATCH and LIN-PRED. LIN-MATCH is an unsupervised method. LIN-PRED is a supervised image classification method. In LIN-PRED, a linear classifier is trained using labeled samples to achieve classification. Its input is the existence vector output by OCAE, just like LIN-MATCH. However, the classifier does not propagate gradients to ASCAE. This ensures that the parameters of ASCAE are still trained by unsupervised image reconstruction and are not affected by supervised classification.

[0130] Table 1 shows the results of ASCAE classification. The best results in Table 1 are shown in bold, and “-” indicates that the paper of this model did not report the results.

[0131] Table 1 Classification results of ASCAE

[0132]

[0133] As can be seen from Table 1, ASCAE is compared with SCAE and other unsupervised models on CIFAR10 and FMNIST. The results show that ASCAE has better performance than SCAE. In LIN-MATCH, ASCAE achieves performance improvements on both datasets. For example, on CIFAR10, the accuracy of ASCAE is 4.59% higher than that of SCAE. In LIN-PRED, the results of ASCAE are still better than SCAE. This performance improvement proves that ASCAE can extract better existence vectors for image description than SCAE. This is because the performance of PCAE in ASCAE proposed in the present invention has been significantly improved, so that more efficient image features can be learned. In addition, due to the independent structure of the component capsule, more reasonable special feature information can be provided for OCAE. The improved OCAE outputs a better existence vector.

[0134] 5. Robustness to affine transformations

[0135] Robustness to affine transformation is the main feature of capsule autoencoders. In SCAE, affine robustness is evaluated using transformed MNIST images. However, MNIST only contains easily recognizable digit images. For complex images, we use transformed CIFAR10 images to verify the affine robustness of ASCAE. The transformed CIFAR10 data is obtained by affine transformation of image samples from the original CIFAR10 test set. These affine transformation processes include: first, the image is randomly rotated by an angle between -20 and +20 degrees; second, the image is scaled by a multiple of 0.8 to 1.2 randomly selected; third, the image is randomly moved by 3 pixels in both the horizontal and vertical directions, so that the translation of the object will not cause the main part of the image to be moved out of the boundary. In this embodiment, ASCAE is trained on the CIFAR10 training set without any data augmentation. Then, the trained ASCAE is evaluated on the transformed CIFAR10 dataset. In this experiment, the transformed image samples used for testing are completely uninvolved in the training process.

[0136] Figure 5 (a) shows the original image in the CIFAR10 dataset after affine transformation. Figure 5 (b) shows the reconstructed image of ASCAE. Figure 5 (c) shows the reconstructed image of SCAE. Figure 5 (a)- Figure 5 (c) It can be seen that ASCAE is still able to reconstruct the objects in the transformed image. However, the reconstructed image of SCAE is still composed of color blocks and is less vivid than Figure 3 Medium (c) poor. Figure 5The results show that the reconstruction ability of ASCAE is more affine robust than that of SCAE.

[0137] 6. Ablation experiment

[0138] Table 2 Ablation experiment results of ASCAE on CIFAR10 dataset

[0139]

[0140] To further study ASCAE, ablation experiments were conducted on CIFAR10. The experimental results are shown in Table 2. Here, the image recognition accuracy obtained by two classifiers, LIN-MATCH and LIN-PRED, was used to conduct ablation experiments on ASCAE. The best results in Table 2 are shown in bold. The "No ResCNN" experiment refers to using a simple convolutional network instead of ResCNN in ASCAE. In LIN-MATCH, the classification accuracy dropped by 0.73%. This result shows that the feature map extracted by ResCNN is better. The "No alpha channel" experiment refers to removing the alpha channel from the image reconstruction process. The results show that the performance of ASCAE will deteriorate without the alpha channel. Therefore, the alpha channel is also very important for image reconstruction. The "alpha channel from mapping" experiment means that the alpha channel is generated by AM mapping image features. The experimental results show that this kind of mapped alpha channel is not reasonable. The alpha channel should be defined with trainable parameters independent of the patch. The "OCAE input with patches" refers to using the patches generated by AM as part of the OCAE input. However, the experiment shows that patches do not need to be input to OCAE. OCAE requires abstract information rather than visual information.

[0141] Experiments on CIFAR10 and FMNIST show that ASCAE can reconstruct images better than SCAE. The adaptive method of ASCAE is more effective than the fixed template. Improving PCAE with the adaptive method can also improve the unsupervised classification performance of ASCAE.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. An image recognition method based on an adaptive stacked capsule autoencoder, characterized in that: The following steps are involved: Step 1: Obtain the image to be analyzed; Step 2: Build an adaptive stacked capsule autoencoder ASCAE; Step 3: Input the image to be analyzed into ASCAE; Step 4: Use ASCAE to reconstruct the image, use the classifier to realize image recognition, and output the recognized image.

2. The image recognition method based on the adaptive stacked capsule autoencoder according to claim 1, characterized in that: The adaptive stacked capsule autoencoder ASCAE in step 2 includes a component capsule autoencoder PCAE and an object capsule autoencoder OCAE. The capsule autoencoder PCAE includes an encoder and a decoder. An adaptive module AM ​​is set in the decoder of PCAE. The adaptive AM module is a fully connected neural network composed of two fully connected layers.

3. The image recognition method based on the adaptive stacked capsule autoencoder according to claim 2, characterized in that: A ResCNN deep convolution module and an attention pooling layer connected to its output are set in the encoder of the capsule autoencoder PCAE. The ResCNN deep convolution module is used to perform convolution calculation on the input original image and obtain a feature map, and output the feature map to the attention pooling layer. The attention pooling layer is used to convert the input feature map into a component capsule.

4. The image recognition method based on the adaptive stacked capsule autoencoder according to claim 3, characterized in that: The component capsule includes a special feature component, an image feature component, an existence probability component, and a posture component: The existence probability component is used to describe the possibility of the component existing in the image; The posture component is used to describe the spatial geometric information of the component; Image feature component, used to represent the appearance information of the component; Special feature components are used to describe other image features besides the appearance of the parts and serve as input data for OCAE.

5. The image recognition method based on the adaptive stacked capsule autoencoder according to claim 4, characterized in that: The adaptive module AM ​​is used to draw the appearance of the component capsule according to the image feature map of the component capsule to form blocks of the local image, and these blocks then change their postures through geometric transformation to finally form a reconstructed image.

6. The image recognition method based on the adaptive stacked capsule autoencoder according to claim 4, characterized in that: The ResCNN convolution module includes 4 residual blocks connected in sequence, each residual block includes three convolutional layers and a residual connection from the first convolutional layer to the third convolutional layer.

7. The image recognition method based on the adaptive stacked capsule autoencoder according to claim 6, characterized in that: The specific steps of step 4 include: Step 4.1: Get the input image I to be analyzed, and record it as I(h,w,c), where h is the height, w is the width, and c is the channel; Step 4.2: Input the image I into the encoder of PCAE and convert I into M component capsules, namely: Among them, PE represents the encoder of PCAE, x represents the posture, d represents the existence probability, z s Indicates special features, z i Represents image features, and the subscript 1:M represents from 1 to M; Step 4.3: The adaptive module AM ​​projects all M image features into M image blocks through the following projection formula, each Each contains the appearance information of an encoded component capsule, and the projection formula is: Among them, T m The height, width and channel are h t ,w t ,c;m∈(1,M); Represents the image features contained in the mth component capsule; Step 4.4: PCAE decoder for each image block T m Corresponding to learning an alpha channel Step 4.5: Based on the pose x in the part capsule m , T m and Perform affine transformation to obtain the transformed image block and Among them, Transform represents affine transformation; Step 4.6: Use the following formula to convert Existence probability d m Perform joint calculation to obtain the reconstructed image: in, represents the reconstructed image, ⊙ represents the product of the corresponding elements in the image; Step 4.7: Using the obtained reconstructed image and the original image I using the mean square error L mse Calculate the loss value of the reconstructed image and the original image to train the PCAE; Step 4.8: Special features of the component capsule Input into OCAE and convert x m ,d m and Connect together to get a token, and encode the M tokens formed by the M component capsules into K object capsules by the encoder of OCAE; Step 4.9: Use OCAE's object decoder to decode the corresponding object capsule into 5 components: OV k ,a k ,OP k,1:M ,a k，1:M ,λ k,1:M MLP k (c k ) Among them, MLP k represents the object decoder implemented by a multi-layer perceptron, OV k represents the 3×3 object-observer relationship matrix, c k represents the kth object capsule, a k Indicates c k The probability of existence of OP k，1:M represents a 3×3 object-component relationship matrix, a k，1:M represents the predicted existence probability of the component capsule, λ k，1:M represents the standard deviation of the correlation; Step 4.10: Get the predicted pose μ of the kth object capsule to the mth component capsule according to the following formula: k,m : μ k,m =OV k OP k,m Among them, OV k represents the affine transformation between the object capsule and the image observer, OP k，m Represents the affine transformation between the object capsule and the component capsule; Step 4.11: Model the pose as a single Gaussian mixture by: p(x m |k,m)=N(x m |m k,m ,l k,m ) Where p represents Gaussian mixture, λ k，m represents the standard deviation of the mth component capsule; Step 4.12: Calculate the pose likelihood L by the following formula pose , and by maximizing L pose To train OCAE, the pose likelihood L pose The calculation formula is: Among them, a k represents the existence probability of the kth object capsule, and all the existence probabilities a k Constitute an existence vector with k elements; a k,m represents the prediction result of the kth object capsule for the existence probability of the mth component capsule, a i,j represents the prediction result of the existence probability of the i-th object capsule to the j-th component capsule; Step 4.13: By minimizing L mse and maximize L pose Establish the loss function of ASCAE: L=L mse -L pose Among them, L represents the loss function of ASCAE; Step 4.14: Use the existence probability a of all object capsules k The existence vector formed represents the feature code of the input image I. The ASCAE model is trained based on the constructed loss function L to optimize the feature code. The classifier then uses the optimized feature code of the image I to realize image classification and output the image recognition result.

Images