A design method of group equivariant capsule network based on spatial attention

Abstract

The application discloses a design method of a group equivariant capsule network based on spatial attention, comprising the following steps: S1, constructing a backbone network with equivariance, and extracting a group equivariant feature map from an input picture; S2, processing the group equivariant feature map by using a combination of global average pooling and maximum pooling to obtain a capsule descriptor with invariance; S3, using a group equivariant spatial attention mechanism to strengthen spatial information in the feature map to obtain a group equivariant capsule pose; S4, processing the capsule pose by using an average maximum pooling, so as to facilitate verification of the equivariance of the capsule pose by using a semantic segmentation task; and S5, interacting information in the capsule descriptor and the pose to improve the performance of the network in classification and segmentation tasks. According to the application, the equivariance can be mathematically proved, the model has a small amount of parameters, is suitable for the case that an input image will be subjected to affine transformation, and can be directly applied to classification and semantic segmentation tasks in practice.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a design method of a group equivariant capsule network based on spatial attention. BACKGROUND

[0002] In recent years, the great progress of artificial intelligence in the field of computer vision cannot be separated from the construction of a large number of labeled data sets, which enables deep neural networks to extract rich information from data to help improve network performance. However, the demand for large amounts of data has put the development of deep neural networks in a dilemma. Because it is difficult to obtain a large amount of labeled data in special fields such as medicine and aerospace. In order to improve the statistical efficiency of deep learning, many scholars hope to enable the network model to learn an invariant feature. However, learning such an invariant feature will lose the spatial relationship between features, and the spatial relationship is very important in computer vision. Therefore, other scholars propose to make the network have equivariance, that is, the output of the network model will change with the input in a linear and predictable manner, while preserving the spatial relationship between features.

[0003] Capsule networks learn the invariant spatial relationship between low-level capsules (a group of neurons) and high-level capsules, so that the pose of the high-level capsule can change with the change of the pose of the low-level capsule, thereby realizing the equivariance of the entire capsule network. However, this learned invariance and equivariance by the network cannot be strictly proven from a mathematical point of view. Compared with convolutional neural networks, traditional capsule networks require a large amount of computation, and only have good results in small-scale data sets, which affects the further development of capsule networks. Recently proposed capsule networks attempt to achieve equivariance in another way. First, in these newly proposed capsule networks, each capsule is divided into a pose and a descriptor. Then the pose and the description are constrained by a specially designed loss function to achieve equivariance and invariance, respectively. However, this method cannot guarantee that the obtained equivariance and invariance are strictly provable. Therefore, some capsule networks use group convolution that is strictly equivariant to group affine transformation to construct the network. Group convolution is completely equivariant to group transformation, and will also improve the robustness of the network to transformations outside the group. However, group convolution capsule networks still use routing algorithms like traditional capsule networks, which requires a large amount of computational cost. SUMMARY

[0004] In view of the deficiencies in the prior art, the purpose of the present application is to provide a design method of a group equivariant capsule network based on spatial attention, which can provide mathematically provable equivariance, has a smaller number of model parameters, is suitable for cases where the input image will undergo affine transformation, and can be directly applied to classification and semantic segmentation tasks in practice. In order to achieve the above purposes and other advantages according to the present application, a design method of a group equivariant capsule network based on spatial attention is provided, comprising:

[0005] S1, constructing a backbone network with equivariance, extracting group equivariant feature maps from input pictures;

[0006] S2, processing the group equivariant feature maps using a combination of global average pooling and max pooling to obtain a capsule descriptor with invariance;

[0007] S3, using a group equivariant spatial attention mechanism to strengthen spatial information in the feature maps to obtain a group equivariant capsule pose;

[0008] S4, processing the capsule pose using an average max pooling to facilitate the use of semantic segmentation tasks to verify the equivariance of the capsule pose;

[0009] S5, interacting information in the capsule descriptor and pose to improve the performance of the network in classification and segmentation tasks.

[0010] Preferably, in step S1, group convolution layers, group equivariant BN layers and group equivariant max pooling are used instead of traditional ResNet50 convolution layers, BN layers and down-sampling layers to make the network have provable equivariance.

[0011] Preferably, in step S2, the capsule descriptor with invariance is obtained by combining average pooling and max pooling, so that the network can recognize the input even if the input picture undergoes affine transformation.

[0012] Preferably, the information in the capsule descriptor and pose is interacted, the descriptor can provide global information for the pose, and the pose can provide spatial position information for the descriptor.

[0013] Preferably, it further includes two kinds of compound pooling operations, a total of three pooling operations, the first pooling is group average pooling GAP, the purpose is to down-sample the group features in the model backbone while maintaining equivariance; The second pooling is a compound pooling operation called GAMP, which is used to achieve invariance that plays a leading role in classification tasks; The third pooling is MALP proposed for semantic segmentation, which uses the maximum value of the average value of the feature map to determine the index of the pixel point.

[0014] Compared with the prior art, the present application has the beneficial effects that: firstly, in order to make the features extracted by the network have the invariance, the group equivariant convolution layer is used to construct the network. Secondly, in order to make the descriptor of the capsule in the network have the invariance, the global average pooling and the maximum pooling are combined to process the group equivariant features. Then, in order to strengthen the spatial information in the capsule pose while preserving the invariance, a group equivariant spatial attention mechanism based on the group equivariant features is proposed. The amount of calculation required by the attention mechanism is greatly reduced compared with the method of directly applying the non-local attention mechanism to the group equivariant features. In addition, in order to make the network be able to reduce the size of the feature map while preserving the invariance and be applied to semantic segmentation, two different group equivariant pooling methods are proposed, which greatly reduce the amount of calculation compared with the routing algorithm in the traditional capsule network. In the experimental part, the classification results under two indexes verify the invariance of the capsule descriptor of SAGE-CapsNet. In addition, the invariance of the capsule pose is verified through semantic segmentation and visualization. The method is suitable for the case that the input image will undergo affine transformation, and can be directly applied to the classification and semantic segmentation tasks in practice, and has the characteristics of small amount of calculation and strong robustness. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 A network model framework diagram of the design method of the group equivariant capsule network based on spatial attention according to the present application;

[0016] Figure 2 A visualization diagram of network invariance of the design method of the group equivariant capsule network based on spatial attention according to the present application;

[0017] Figure 3 A visualization diagram of p4 group feature map of the design method of the group equivariant capsule network based on spatial attention according to the present application;

[0018] Figure 4 A visualization diagram of connection operation of GSA of the design method of the group equivariant capsule network based on spatial attention according to the present application;

[0019] Figure 5 A visualization diagram of GAMP composite pooling of the design method of the group equivariant capsule network based on spatial attention according to the present application;

[0020] Figure 6 A visualization diagram of semantic segmentation result of the design method of the group equivariant capsule network based on spatial attention according to the present application. DETAILED DESCRIPTION

[0021] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the protection scope of the present application.

[0022] With reference to Figures 1-6 , a design method of a group equivariant capsule network based on spatial attention, comprising:

[0023] S1, constructing a backbone network with equivariance, extracting a group equivariant feature map from an input picture;

[0024] S2, processing the group equivariant feature map using a combination of global average pooling and maximum pooling to obtain a capsule descriptor with invariance;

[0025] S3, using a group equivariant spatial attention mechanism to strengthen spatial information in the feature map to obtain a group equivariant capsule pose;

[0026] S4, processing the capsule pose using an average maximum pooling to facilitate verification of the equivariance of the capsule pose using a semantic segmentation task;

[0027] S5, interacting information in the capsule descriptor and the pose to improve the performance of the network in classification and segmentation tasks.

[0028] Before specifically introducing each module of the network, the definition of equivariance is given. If a network or a layer F of the network is equivariant, it needs to satisfy that when its input x is subjected to an affine transformation g (written as T g x) and then passes through F, it has the same result as when the input first passes through F and then is subjected to the affine transformation g. Specifically, as shown in formula (1):

[0029] F(T g x)=T′ g F(x), (1)

[0030] Where T g and T′ g are two instances of the affine transformation g, generally written in the form of a matrix. T g and T′ g do not need to be exactly the same, but only need to be linear representations of the affine transformation group G, that is, T(g)T(h)=T(gh) holds for any g, hEG. The visualization result is shown in Figure 2 .

[0031] When T′ gIf F is an identity map, then equivariance degenerates into invariance, as shown in equation (2):

[0032] F(T g x)=F(x) (2)

[0033] Symmetry is a transformation that leaves the internal structure of an object unchanged. A symmetry group is a set of symmetry transformations that satisfy the group axioms. Take rotation in a 2D plane as an example. This group acts on the pixels in the plane Z 2 . It can be easily proved that the combination of two rotations is also a rotation. For example, rotating an object by 30° and then rotating it by 60° has the same effect as rotating it by 90° alone. In addition, the inverse of a rotation is also a rotation, but in the opposite direction.

[0034] In order to facilitate comparison with other models, the present application uses P4 and P4m groups as shown in equations (3) and (4).

[0035]

[0036] where r is an integer and 0≤r<4, (h, w)∈Z 2 . The P4 group contains all rotations by multiples of 90° and the combination of translation transformations.

[0037]

[0038] where m∈(0, 1), 0≤r<4, (h, w)∈Z 2 . The P4m group contains all mirror reflections and the combination of rotations by multiples of 90° and translation transformations.

[0039] Next, group convolution is introduced. Group convolution is a convolution defined on a group. Unlike traditional convolution, group convolution uses symmetry transformations defined in the group to transform the convolution kernel, and then uses the transformed convolution kernel to perform ordinary convolution:

[0040]

[0041] Here is the feature map in the l-th layer with K l channels, ψ k is the k-th channel of the convolution kernel, y and g are the spatial positions of the input and output respectively, and g is also a transformation in the group.

[0042] To help understand group convolution and group convolution feature maps, the visualization of the feature maps of P4 group convolution is shown in Figure 3 Figure 3 ​The left subfigure depicts four representative rotated feature patches, which are rotated by 0°, 90°, 180°, and 270°, respectively. Each pixel in the P4 feature map has a rotation coordinate (which pixel is in which feature patch) and two spatial coordinates (which position in the pixel patch is the pixel in). When the input of group convolution is transformed by rotation r, the P4 output feature map becomes Figure 3 the right, where each feature patch moves along its red arrow and the feature patch itself rotates 90° at the same time. Figure 3 is also an intuitive example of the equivariance of group convolution. The equivariance of group convolution can be represented by equation (6):

[0043] [L u [f]*ψ](g)=[L u [f*ψ]](g) (6)

[0044] For the analysis later, the following formula is introduced to define a set of feature maps under transformation g:

[0045] [L g f](x)=f(g -1 x) (7)

[0046] As can be seen from equation (7), in order to obtain the value of pixel x under transformation g, we need to look up the value at the g -1 x position in the original feature map, which is the only point that can be mapped to x by g. For example, we can look up the green small block 1 on the left to obtain the value of the green small block 2 on the right. Figure 3 Figure 3

[0047] Equivariance is more important than invariance in the context of capsule networks. Equivariance requires the network to preserve the spatial information in the input image, and the capsule network also needs to learn the spatial information between capsules. To this end, the present application proposes an equivariant spatial attention mechanism (GSA) to help our model pay special attention to important spatial information in group features. GSA is derived from the no local attention (NLA) mechanism. The main difference is that GSA operates on group features, while NLA operates on regular flat features. We first introduce NLA for feature map f(x) at pixel i:

[0048]

[0049] where the attention map A[f(x i ), f(x j )] = [f*θ] T (x i )[f*φ] T (x j ), and x i ​​, x j ∈Z 2 。l i is a normalization coefficient for point i, usually set as and are convolution kernels with size 1x1.

[0050] One important reason we choose to derive GSA from NLA is that NLA is equivariant to affine transformation. However, directly applying NLA to group feature maps will lead to a huge computational problem. Take P4 as an example, the shape of group feature in P4 is (C, R, H, W), which represents channel, rotation, height and width respectively. P4 group feature can be seen as a set of (R x H x W) feature vectors, each of which is C-dimensional. Then, through the NLA calculation defined in equation (8), we can get an attention map with shape (R x H x W, R x H x W). If we omit the computational cost of convolution when calculating the attention map A, the computational complexity here is O(C, R 2 , H 2 , W 2 ).

[0051] To reduce the computational overhead and ensure equivariance, the invention proposes group space attention GSA. GSA concatenates group features in the channel direction, making the shape of group features (C x R, H, W). In this way, P4 group features can be seen as (H x W) feature vectors, each of which has (C x R) dimensions. After concatenation, GSA adopts the same operation as NLA. Therefore, the attention map can be derived by calculating these (H x W) feature vectors. The computational complexity here is O(C, R, H 2 , W 2 ), which is obviously R times smaller than NLA. If the group contains more transformations than the P4 group, the gap between the computational overhead of GSA and NLA will become even larger. The efficiency of GSA should be attributed to the sharing of attention weights in R feature vectors in each group, which can be understood by observing the columns composed of some red small squares in Figure 4 .

[0052] In addition to the small computational cost, GSA is equivariant to affine transformation. As shown in Figure 4 , when the feature is transformed, the concatenated group features will change the order of channels. This change in the channel direction will not have any effect on the dot product result of every two points in the attention map, which means that the equivariance of group features will not be destroyed.

[0053] Next, we will give a theoretical proof to verify the above observation that GSA is equivariant to affine transformation. We first give the definition of GSA as follows:

[0054] x = f cat [f(g)], (10)

[0055] f(g)=f decat [x], (11)

[0056] f GSA (g i )=f decat [f NLA [x i (12)

[0057] Where f cat ,like Figure 4 The diagram illustrates the operation of connecting group features in the channel direction. decat This is the operation of transforming feature maps back into group features. g∈G and x∈Z 2 . Figure 4 The changes brought about by the join operation in the middle can be summarized as follows:

[0058] [L u f](g)=f decat [ρL u′ [f(g)]], (13)

[0059] Here, ρ is the operation that changes the order of the feature maps along the channel direction, making their order consistent with their original order before the transformation. The ρ operation is used because f cat and f decat The grouping transformation operation may change the order of the feature maps. Figure 4 It can be seen that when the group features are along the channel direction, the transformation L u It can be divided into transformation L u′ (its relationship with L) u Same, but acts on Z 2 ) and operation ρ.

[0060] Next, we will discuss the equivariance of GSA. In GSA, the operation f cat and f decat It has no effect on isovariability, therefore if f NLA If they are equivariant, then f GSA It is equivalent.

[0061] Theorem 1: Operation f under the GSA definition GSA It is equivalent.

[0062] Proof: As mentioned above, in order for the theorem to hold, we only need to prove that operation f works when the group features are connected. cat It is equivariant. To achieve this goal, we first proved the equivariance of the attention map A. Using the equivariance of the group convolution expressed by equation (6), we can derive:

[0063]

[0064] Obviously, Figure 4 ρ in the meanwhile changes the channel order of the two input vectors of the dot product, but does not change the dot product result. So ρ can be dropped. In contrast, L u′ is a function that changes the spatial location of the feature vectors, so it cannot be dropped. Instead, it should be put outside the square brackets. Based on the above discussion, we have:

[0065]

[0066] Then, based on the equivariance defined in equation (1), we can conclude that the attention map A is equivariant to affine transformations. After the attention map, we further prove the equivariance of f NLA . With the help of equation (15), we have:

[0067]

[0068] The ρ in equation (16) acts on all f(x j ) and the weight A[·,·] is a scalar with respect to specific i and j, so ρ can be put outside the summation sign. Also based on the equivariance defined in equation (1), we can conclude that f NLA has equivariance, and the proof is complete.

[0069] Considering the equivariance and invariance of CapsNets to group transformations, the present invention proposes two compound pooling operations for group features in addition to the commonly used average pooling, a total of three pooling operations. The first pooling is group average pooling GAP, which aims to down-sample the group features in the model backbone while maintaining equivariance. Fortunately, the traditional average pool itself can preserve the equivariance of the feature map. Therefore, GAP directly applies average pooling to the group features, but only moves the pooling filter on Z 2 For example, using a 2x2 average pool with a stride of 2 and no padding, the shape of the p4 group feature changes from (C, R, H, W) to

[0070] The second pooling is a composite pooling operation called GAMP. GAMP aims to realize the invariance that plays a dominant role in the classification task. The first step of GAMP is to apply GAP to the spatial dimensions H, W of group features. After that, the shape of p4 features changes from (C, R, H, W) to (C, R, 1, 1). It is worth noting that GAP produces invariance rather than equivariance when the kernel size is equal to the feature space dimension size (H, W in this case). Then, GAMP uses max pooling over the dimension R to further introduce the invariance of group features. After max pooling, the shape of group features is converted to (C, 1, 1). The visualization of GAMP is given in Figure 5 .

[0071] The third pooling is MALP proposed for semantic segmentation, which uses the maximum of the average values of feature maps to determine the index of a pixel. Since group feature maps have one more dimension than traditional CNN feature maps (for example, the shape of p4 group features is (C, R, H, W), while the shape of traditional CNN features is (C, H, W)), we cannot directly use group feature maps for segmentation. Therefore, we propose the MALP operation to downsample group feature maps along the R dimension (the additional dimension). Taking the input p4 feature X as an example, with the shape of (C, R, H, W), MALP first calculates the average value along the spatial dimension and the channel dimension X m , where the shape of X m is R. Then, MALP finds the index idx of the maximum value in X m . Finally, MALP uses the index to downsample the input X: X←X[ :, idx, :, : ].

[0072] If batch normalization (BN) uses a single scale and bias parameter for each group feature map (rather than one scale and bias for each spatial feature plane in G feature maps), the equivariance of BN can be maintained. As for ReLu, this point-wise nonlinear operation also preserves the equivariance of group feature maps. So far, all layers used by the model have equivariance or invariance. Next, the present invention proposes three sets of experiments to verify the excellent performance of SAGE-CapsNet. First, to verify the invariance of the capsule descriptor, SAGE-CapsNet is compared with other CapsNets and conventional group equivariant networks on many classification datasets. The second set of experiments aims to verify the equivariance of the capsule pose of SAGE-CapsNEt through semantic segmentation and visualization. The third is an ablation experiment on GSA and information exchange in SAGE-CapsNet.

[0073] For classification and segmentation, the backbone networks of SAGE-CapsNet are modified ResNet, in which the traditional convolutional layers and batch normalization are replaced by group convolutional layers and group batch normalization. However, they have different numbers of layers. The SAGE-CapsNet backbone for classification has a three-stage network. Stage i has 3N i layers (a bottleneck residual block contains three convolutional layers) and K i convolutional kernels. Here, we set N i = 3, 4, 6. SAGE-CapsNet includes an initial group convolutional layer, a transition layer, and a final classification layer, so there are (3 + 4 + 6) x 3 + 3 = 42 layers. For SAGE-CapsNet built with p4 equivariant layers, we choose K i = 16, 32, 64, the model has 2.52M trainable parameters. For SAGE-CapsNet built with p4m equivariant layers, we set K i = 8, 16, 32 to reduce the computational cost of the additional group transformation in p4m. The total number of trainable parameters of the model is 1.47M.

[0074] Unlike classification, the SAGE-CapsNet backbone for segmentation has four stages. We choose N i = 3, 4, 6, 3 and K i = 16, 32, 64, 128 to build the backbone p4. SAGE-CapsNet for segmentation includes an initial group convolutional layer, three transition layers (the first is used to reduce the number of channels, the second is used to inject feature maps, and the last is used to introduce pose information to the descriptor), and a final classification layer, so it has (3 + 4 + 6 + 3) x 3 + 5 = 53 layers and 7.78M trainable parameters. Since SAGE-CapsNet is built with group convolutional layers, it should have fewer parameters than models built with traditional convolutional layers.

[0075] A common way to evaluate the robustness of CapsNets to affine transformations is to compare the classification results with and without affine transformations. If the results are close, the CapsNet is robust. Here, two metrics are proposed to measure the difference in classification results under two cases, i.e., within the symmetry group and outside the symmetry group. The first metric is used to verify whether the model is robust to the transformations within the symmetry group, while the second metric is used to evaluate the robustness of the model to the transformations that can not be in the symmetry group. Specifically, the first metric is to transform the input image with the transformations within the symmetry group, and then calculate the average classification accuracy of all transformations as the final classification result. Take p4 as an example, the input image is only rotated by a multiple of 90°. For the second metric, we rotate the input image by a multiple of 10°, and then calculate the average of the classification accuracy of each rotation as the final result.

[0076] Next, SAGE-CapsNet is compared with various models related to invariance and equivariance, including dynamic routing capsule network (Dy-CapsNet), fast routing capsule network (Fr-CapsNet), equivariant convolutional network (G-CNN), attention group equivariant network (AGE-Net), group equivariant capsule network (GE-CapsNet), and deep group equivariant capsule network (SOVNET). We mainly conduct comparative experiments on CIFAR10, MNIST, and classification experiments on other datasets such as Tiny ImageNet and affNIST. The classification results are shown in Tables 1 and 2. The results of the models other than SAGE-CapsNet come from our reproduction and are averaged over 3 experiments.

[0077] Table 1 Robustness test experiments on MNIST (%)

[0078]

[0079] Table 2 Robustness test experiments on CIFAR10 (%)

[0080]

[0081] As shown in Table 1 and Table 2, SAGE-CapsNet outperforms other models in terms of classification accuracy, metric1 and metric2. For robustness, a model that achieves good classification results can perform poorly in metric1 and metric2, which means it has no good generalization ability for other affine transformations. For example, Fr-CapsNet achieves 99.67% accuracy on MNIST, but only 47.12% on metric2. For metric1, models built with p4 group equivariant layers (except SOVNET) have the same results in classification accuracy and metric1, which means these models are robust to transformations within the symmetric group. Since the capsule descriptor in SAGE-CapsNet is used for classification, the results in metric1 show that the capsule descriptor is invariant. In addition, SAGE-CapsNet achieves the best results in metric2, which means SAGE-CapsNet has better generalization ability for unfamiliar transformations. For other datasets, the performance of SAGE-CapsNets is similar. The classification accuracy on Tiny ImageNet and affNIST (SAGE-CapsNet is trained on MNIST and tested on affNIST) is 69.8% and 94.4%, respectively.

[0082] Next, we verify the equivariance by visualizing the segmentation results, and then intuitively observe whether the results are transformed in the same way as the input image. We conduct semantic segmentation experiments on the PASCAL Visual Object Classes (VOC) 2012 dataset. Considering that our main goal is to verify the equivariance of SAGE-CapsNet, we choose a relatively simple model, FCN (fully convolutional network with ResNet50 as backbone), for comparison. Similar to FCN, SAGE-CapsNet for segmentation combines feature maps at different stages of the backbone network, but in a concatenation manner. Next, SAGE-CapsNet bilinearly upsamples the concatenated feature maps to the same spatial size of the input image, and finally uses a 1x1 convolutional classifier to classify each pixel. For training details, we randomly crop images to a size of 480x480, flip images with a probability level of 0.5, and then normalize the images. Adaptive momentum estimation (Adam) is used as the optimizer with a learning rate of 0.001. The loss function uses cross-entropy loss. In the visualization experiment, we randomly rotate the input image by multiples of 90° and observe the changes in the segmentation results.

[0083] The segmentation results are given in Table 3. The performance of SAGE-CapsNet is comparable to FCN, but contains less parameters. This should be thanks to the p4 equivariant layer, which shares weights to a higher degree than traditional convolutional layers. The visualization of the segmentation results is shown in Figure 6 Table 3. It can be observed that the heat map of the features and the transformation of the segmentation results are the same as the input image, which directly proves the equivariance of SAGE-CapsNet. It also implicitly indicates the equivariance of all components of SAGE-CapsNet.

[0084] Table 3. It can be observed that the heat map of the features and the transformation of the segmentation results are the same as the input image, which directly proves the equivariance of SAGE-CapsNet. It also implicitly indicates the equivariance of all components of SAGE-CapsNet.

[0085]

[0086] After the segmentation experiment, the ablation experiment is carried out. As shown in Table 4, we carry out the ablation experiment on GSA and `P+D'. The benchmark model is a SAGE-CapsNet without group spatial attention and the fusion of descriptor information and capsule pose information. By adding GSA and `P+D' operation to the benchmark model respectively, we can observe that they both improve the performance of the model. This is because the working principle of GSA is similar to NLA, that is, GSA is a spatial attention operation for capturing long-range dependencies. `P+D' is an operation that combines global information in the descriptor with local information in the pose, which is of great help to complete the subsequent segmentation. P+D' also allocates weights to the capsule pose, which can also be regarded as channel attention.

[0087] The application discloses a design method of a group equivariant capsule network based on spatial attention. First, in order to make the features extracted by the network have equivariance, a group equivariant convolutional layer is used to construct the network. Second, in order to make the descriptors of the capsules in the network have invariance, a combination of global average pooling and maximum pooling is used to process the group equivariant features. Then, in order to strengthen the spatial information in the capsule pose while preserving the equivariance, a group equivariant spatial attention mechanism based on the group equivariant features is proposed. The amount of calculation required by the attention mechanism is greatly reduced compared with the method of directly applying the non-local attention mechanism to the group equivariant features. In addition, in order to make the network be able to reduce the size of the feature map while preserving the equivariance and be applied to semantic segmentation, two different group equivariant pooling methods are proposed, which greatly reduce the amount of calculation compared with the routing algorithm in the traditional capsule network. In the experimental part, the classification results under two indexes verify the invariance of the capsule descriptors of SAGE-CapsNet. In addition, the semantic segmentation and visualization verify the equivariance of the capsule pose. The method is suitable for the case that the input image will undergo affine transformation, can be directly applied to the classification and semantic segmentation tasks in practice, and has the characteristics of small amount of calculation and strong robustness.

[0088] Table 4 Ablation experiments on PASCAL VOC 2012 validation set

[0089]

[0090] The number of devices and the size of the processing are illustrative of the application and are not intended to limit the scope of the application to which the application is applicable. Modifications and variations are possible in light of the above teachings or can be acquired from practice of the application.

[0091] While embodiments of the application have been disclosed in connection with the illustrative embodiments shown and described above, it is not intended to limit the application to the details of the embodiments set forth hereinabove, rather it is intended to be illustrative of the broadness of the application. The scope of the application should be determined, not with reference to the above description, but should instead be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. A design method of a group equivariant capsule network based on spatial attention, characterized in that, The method comprises the following steps: S1, constructing a backbone network with equivariance, extracting group equivariant feature maps from input pictures; By using group convolution layers, group equivariant BN layers and group equivariant max pooling instead of the convolution layers, BN layers and down-sampling layers in the traditional ResNet50, the network has provable equivariance; S2, processing the group equivariant feature maps using a combination of global average pooling and max pooling to obtain a capsule descriptor with invariance; the capsule descriptor with invariance is obtained by combining average pooling and max pooling, so that the network can still recognize the input under the condition that the input picture is subject to affine transformation; the information in the capsule descriptor and the pose is interacted, the descriptor can provide global information for the pose, and the pose can provide spatial position information for the descriptor; S3, using a group equivariant spatial attention mechanism to strengthen the spatial information in the feature map to obtain a group equivariant capsule pose; the group equivariant spatial attention mechanism connects group features in the channel direction to share attention weights among each group transformation dimension, so as to reduce the calculation overhead while maintaining the equivariance of affine transformation; S4, processing the capsule pose using an average max pooling, which facilitates the verification of the equivariance of the capsule pose using the semantic segmentation task; the average max pooling calculates the average value along the spatial dimension and the channel dimension, and uses the maximum value index to down-sample the group feature map; S5, interacting the information in the capsule descriptor and the pose to improve the performance of the network in classification and segmentation tasks; For classification and segmentation, the backbone network of SAGE-CapsNet is a modified ResNet; SAGE-CapsNet includes an initial group convolution layer, a transition layer and a final classification layer; Unlike classification, the SAGE-CapsNet backbone for segmentation has four stages, and the SAGE-CapsNet for segmentation includes an initial group convolution layer, three transition layers and a final classification layer.

2. The design method of a group equivariant capsule network based on spatial attention according to claim 1, wherein, It also includes two composite pooling operations, a total of three pooling operations, the first pooling is group average pooling GAP, which aims to down-sample the group features in the model backbone while maintaining equivariance; The second pooling is a composite pooling operation called GAMP, which is used to achieve invariance that plays a leading role in the classification task; the third pooling is MALP proposed for semantic segmentation, which uses the maximum value of the average value of the feature map to determine the index of the pixel point.

Images