A convolutional neural network rotation pattern recognition method with pre-rotation only in the inference stage

By pre-rotating the image in the inference stage and calculating the classification confidence of multiple rotation channels, the problem of low efficiency in the rotation pattern recognition method in the prior art relying on rotation samples and rotation angle information processing during the training stage is solved, and efficient Rotation pattern recognition and angle estimation.

CN113989486BActive Publication Date: 2024-11-08NAVAL UNIV OF ENG PLA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111109202.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-22
Publication Date
2024-11-08
Estimated Expiration
2041-09-22

AI Technical Summary

Technical Problem

The existing rotation pattern recognition method needs to provide rotation samples during the network training stage, which leads to increased learning difficulty and poor interpretability of feature maps, and cannot explicitly give rotation angle information when classification of pattern categories, limiting the processing efficiency of rotation information.

Method used

In the inference phase, the image is pre-rotated to all possible directions, and the classification confidence of multiple rotation channels is calculated using parallel shared network structures and weights. The pattern categories and direction angles are predicted by the confidence maximum, and rotation pattern recognition and angle estimation are realized.

Benefits of technology

Without increasing learning tasks, the accuracy of rotation pattern recognition and direction angle estimation accuracy are improved, subjective control and efficient processing of rotation pattern recognition are realized, and are suitable for general images, medical images and other fields.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113989486B_ABST
    Figure CN113989486B_ABST
Patent Text Reader

Abstract

The present invention relates to a rotation pattern recognition method of a convolutional neural network that is pre-rotated only in the inference stage. In the training stage, the method enables the convolutional neural network to learn only one fixed-direction pattern. In the inference stage, the test pattern is first rotated to multiple directions and then respectively input into the convolutional neural network to calculate the classification confidence response. Then, the pattern classification and direction angle estimation are performed by selecting the largest confidence response of all rotation channels. This method realizes for the first time that the convolutional neural network can recognize any rotation pattern under the premise of learning only one direction pattern and estimate the direction of the rotation pattern without adding an angle regression task. It can be used for rotation pattern recognition and rotation object detection tasks of general optical images, medical images, remote sensing images, etc.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of image pattern recognition, and in particular relates to a convolutional neural network rotation pattern recognition method that is pre-rotated only in the inference stage. Background Art

[0002] Currently, rotation pattern recognition is a challenge for convolutional neural networks. The following two methods are usually used to solve this problem: (1) construct a dataset containing various rotation patterns of objects to train the neural network, which is essentially data augmentation; (2) rotate the convolutional neural network feature map or convolution kernel to learn rotation-invariant features. These methods are inefficient in processing angle information. An ideal neural network should be able to generalize to new perspectives without the need for additional sample learning.

[0003] Convolutional neural networks are one of the most effective and popular tools in the field of image object recognition. A typical convolutional neural network consists of convolutional layers and pooling layers. The design of the convolutional layer follows a basic rule of object recognition, that is, if a feature detector is effective at a certain position in the image, it should also be effective at other positions. Therefore, compared with the fully connected network, the "sliding convolution" and parameter sharing mechanism of the convolutional neural network ensures the equivariance of its feature extraction for translational objects and generalizes its object recognition ability to any position in the image plane.

[0004] In order to generalize the recognition ability of CNNs to any direction, the current solution is to establish a rotation-invariant learning task for CNNs, that is, the convolutional neural network needs to predict the same category label for the input image and its arbitrarily rotated copy. The essence of rotation-invariant learning is to use the powerful function fitting ability of CNNs to learn the mapping of multiple orientation patterns to a single category label, and this mapping can be learned by rotating the input image, feature map or convolution kernel by multiple angles during network training. Rotating images is the easiest rotation-invariant learning strategy to implement. The most widely used method is to use rotation data augmentation during network training. Data augmentation does not require changing the CNNs structure, but its effect depends heavily on the diversity of rotation patterns contained in the training set. In order to obtain more robust rotation-invariant pattern recognition effects, it is a common method to establish multiple rotation channels for CNNs during training and inference. D. Laptev et al. rotated the input images in the training set that were originally random in angle to 24 angles in the full-angle space, and then used a group of CNNs with the same structure and shared parameters to extract features, and applied "max pooling" to the obtained multi-channel feature vectors to extract the final invariant features. In addition to rotating images, there are also studies that establish rotation channels by rotating CNNs intermediate feature maps or convolution kernels.

[0005] Although rotation-invariant learning is widely accepted in rotational pattern recognition, it has two significant disadvantages: First, rotated samples need to be provided during the network training phase. Most studies on rotation-invariant recognition use the MNIST-rot dataset for network training, and the training set images of this dataset itself contain characters in any direction within a 360° range. Therefore, to some extent, it can be considered that rotation-invariant learning is forcing CNNs to "memorize" copies of the same pattern in different rotation directions, which will lead to a sharp increase in the difficulty of CNNs learning and a decrease in the interpretability of the extracted feature maps. Second, the rotation-invariant learning goal abandons the angle information carried by CNNs themselves, and uses a single category label as the learning goal for patterns of different rotation directions, which means that the prediction of the direction of the rotation pattern can only be achieved by adding an explicit angle regression task.

[0006] The difficulty of solving the above problems and defects is:

[0007] The current rotation pattern recognition methods all take rotation invariance as the learning goal, but the rotation invariance learning goal abandons the angle information carried by the two-dimensional features extracted by CNNs. For the rotation patterns of different direction angles, only the same category label is used as the learning goal, and the pattern rotation angle information cannot be explicitly given while the pattern category is classified. The processing of different rotation channel information is implemented in a serial manner. The feature extraction layer of the latter level needs to process all the rotation channel features of the previous level. The serial network structure needs to limit the number of rotation channels to ensure the real-time requirement of reasoning, and limiting the number of channels means reducing the processing efficiency of rotation information.

[0008] The significance of solving the above problems and defects is:

[0009] The existing convolutional neural network is equipped with the ability to recognize rotation patterns without changing the original structure, and this ability to recognize rotation patterns does not rely on learning rotation samples during the network training phase. According to the requirements of task category classification accuracy and angle classification accuracy, it can be achieved by artificially changing the number of parallel rotation channels, realizing subjective control of rotation pattern recognition accuracy and making the mechanism of rotation pattern recognition more explainable. The calculation of rotation channels is completely independent of each other, and can achieve higher rotation information processing efficiency with the help of parallel computing. Summary of the invention

[0010] In view of the problems existing in the prior art, the present invention provides a convolutional neural network plane recognition method, system and terminal based on image pre-rotation, and specifically relates to a convolutional neural network rotation pattern recognition method that only pre-rotates in the inference stage.

[0011] The present invention is implemented as follows: a convolutional neural network rotation pattern recognition method that is pre-rotated only in the inference stage includes two stages: a training stage and an inference stage. In the training stage, a CNN with a small number of parameters is trained using only a pattern in one direction. In the inference stage, multiple rotation channels are first established by pre-rotating the test sample to all possible directions; then, a group of CNN groups that share network structures and weights are used between the rotation channels to calculate the classification confidence in parallel; finally, the pattern category and direction are simultaneously predicted according to the maximum classification confidence of each channel.

[0012] Specifically, when processing an actual rotation pattern recognition task, the present invention can be implemented by the following steps:

[0013] Step 1: image sample collection, preprocessing and dataset creation;

[0014] Step 2: Design the convolutional neural network structure;

[0015] Step 3: CNN parameter learning without any rotation data augmentation;

[0016] Step 4, pre-rotate the random direction image to be tested to construct a pre-rotation channel;

[0017] Step 5, use the complete twin instance of the convolutional neural network trained in step 3 to infer the rotation pattern in parallel and calculate the multi-channel classification confidence matrix F;

[0018] Step 6: According to the maximum confidence criterion, the object category and direction angle are simultaneously predicted from the F matrix.

[0019] Furthermore, in step 1, the image sample collection, preprocessing and data set preparation include:

[0020] (1) For the image pattern to be recognized, set respective standard directions for different categories of patterns;

[0021] (2) Collect image samples for each type of pattern and try to ensure the balance of sample numbers between classes;

[0022] (3) The collected samples are normalized one by one and the samples are rotated to the standard direction;

[0023] The object's rotation angle is recorded as When the object direction is in the positive direction of the y axis When the object rotates counterclockwise, the direction angle is negative, and when it rotates clockwise, the direction angle is positive; for the collected image samples of the i-th mode, the direction angle may be arbitrary, and the arbitrary rotation mode is expressed as: The purpose of normalizing the samples is to convert any rotation mode Adjust to standard mode with a direction angle of 0 degrees

[0024] (4) Divide the preprocessed image samples into training, validation, and test sets in a ratio of 7:1:2;

[0025] (5) During training and testing, the input image is normalized into a square before input.

[0026] Further, in step 2, the designing of the convolutional neural network structure includes:

[0027] (1) Reasonably design the convolutional neural network structure according to the size of the task dataset to obtain higher generalization performance on the test set;

[0028] (2) The convolutional network structure for rotation pattern recognition can be designed manually or using automated machine learning methods such as neural architecture search. The core principle of the design is to ensure that the network size is adapted to the task dataset and to achieve the best generalization performance on the test dataset.

[0029] (3) The first layer of the network uses 32 convolution kernels of size 5, a convolution step of 1, and the “same padding” method to fill the image edge feature map output size unchanged;

[0030] The second layer of the network uses a maximum pooling operation with a step size of 2, which reduces the network size by half;

[0031] The third convolutional layer of the network uses 32 convolution kernels of size 5, a convolution step of 1, and the “same padding” method to fill the image edge feature map output size unchanged;

[0032] The fourth layer of the network uses a maximum pooling operation with a step size of 2, which reduces the network size by half;

[0033] The fifth, sixth, and seventh layers of the network use fully connected networks; the fifth layer receives the vector features of the fourth layer feature map flattened, and has 120 neuron outputs;

[0034] The sixth layer of the network is output by 84 neurons;

[0035] The seventh layer of the network consists of 10 neuron outputs, corresponding to the ten categories of the MNIST multi-classification task.

[0036] Further, in step 3, the convolutional neural network parameter learning includes:

[0037] (1) Softmax regression is used to process multiple neuron outputs of the convolutional neural network, cross entropy loss is used as the network learning criterion, and mini-batch gradient descent is used to learn network parameters;

[0038] (2) The standardized training sample set is input into the convolutional neural network model in small batches, the structured loss is calculated using the cross entropy loss function, and then the network is updated using small batch gradient descent; wherein the learning rate during the training is an adjustable learning rate.

[0039] Further, in step 4, the randomly pre-rotating the test set images to construct a rotation channel includes:

[0040] (1) For any input image with direction θ in the test set Among them, H, W, d represent the image height, width, and number of channels respectively; the image is rotated at equal intervals within the range of [-Φ, +Φ] degrees, the number of rotations is N, and the rotation angle interval ΔΦ = 2Φ / N;

[0041] (2) The modes and angles after rotation are:

[0042]

[0043]

[0044] in, Φ and ΔΦ represent the expected rotation pattern recognition angle estimation range and accuracy, respectively, which are set according to the task requirements, 0≤Φ≤π, and ΔΦ is adjusted by setting the number of rotation channels N;

[0045] (3) Image rotation is implemented using a bilinear interpolation algorithm. The rotation angle intervals are set to 1 degree and 0.1 degree, respectively, and the estimation intervals Φ are set to π and

[0046] Further, in step 5, the fully twinned instances of the trained convolutional neural network are used to infer the rotation patterns in parallel to calculate the multi-channel classification confidence matrix F, including:

[0047] (1) After pre-rotating the image, the trained convolutional neural network is used to extract features and predict categories on the copy of the object pattern in the nth rotation channel:

[0048]

[0049] Among them, f n is the classification response of the nth channel;

[0050] (2) Arrange the convolutional neural network classification confidence outputs of all rotation channels into a rotation multi-channel classification confidence matrix:

[0051]

[0052] in, C is the number of categories to be predicted.

[0053] Further, in step 6, the object category and direction angle are simultaneously predicted by the F matrix according to the maximum confidence criterion, including:

[0054] (1) Using classification confidence as the final criterion for pattern recognition, the maximum coordinates (j, k) of the multi-channel response matrix are used to simultaneously predict the object category and direction angle:

[0055]

[0056] (j,k)=argmax(F N×C );

[0057] (2) When using the classic convolutional neural network for supervised classification, the maximum classification confidence criterion is used to predict the category and angle. The maximum similarity criterion can also be used.

[0058] (3) Executing the above process on the Rotated MNIST and Rotated Fashion MNIST datasets respectively can obtain the category and angle estimation performance.

[0059] Another object of the present invention is to provide a convolutional neural network plane rotation pattern recognition system based on image pre-rotation, which uses the convolutional neural network rotation pattern recognition method that is pre-rotated only in the inference stage, and the convolutional neural network plane rotation pattern recognition system based on image pre-rotation includes:

[0060] Image preprocessing module, used for image sample collection, preprocessing and data set preparation;

[0061] Network structure design module, used to design convolutional neural network structure;

[0062] Parameter learning module, used to implement convolutional neural network parameter learning;

[0063] A rotation channel construction module, used for pre-rotating the random orientation image to be tested to construct a pre-rotation channel;

[0064] A multi-channel classification confidence matrix calculation module is used to infer the rotation pattern in parallel through the fully twinned instances of the trained convolutional neural network to calculate the multi-channel classification confidence matrix F;

[0065] The object category and direction angle prediction module is used to simultaneously predict the object category and direction angle from the F matrix according to the maximum confidence criterion.

[0066] Another object of the present invention is to provide an information data processing terminal, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the convolutional neural network rotation pattern recognition method that is only pre-rotated in the inference stage.

[0067] Another object of the present invention is to provide a computer-readable storage medium storing instructions, which, when executed on a computer, enables the computer to execute the convolutional neural network rotation pattern recognition method that is pre-rotated only in the inference stage.

[0068] Combining all the above technical solutions, the advantages and positive effects of the present invention are as follows:

[0069] The invention provides a convolutional neural network rotation pattern recognition method that only pre-rotates in the inference stage, which limits the convolutional neural network to learn only a set of standard rotation patterns during training. During the test period, multiple rotation channels are established by pre-rotating input images with random directions, and a set of parallel twin networks are used to calculate the classification confidence response of multiple rotation channels. In this way, the convolutional neural network can have the ability to simultaneously predict the pattern category and the rotation direction angle without increasing the learning tasks. The invention can be widely used in pattern recognition and object detection tasks of general images, medical images, remote sensing images, etc.

[0070] The convolutional neural network provided by the present invention only learns the rotation pattern in one direction during training, and only needs to store a set of network weights during storage; during testing, multiple rotation channels are established by pre-rotating the input image, and a set of parallel twin networks are used to calculate the classification confidence response of the multiple rotation channels; by reducing the angular rotation interval, the rotation pattern recognition accuracy and the direction angle estimation accuracy can be effectively increased.

[0071] The network training method of the present invention can simultaneously predict the category and direction angle of the rotation mode from the pre-rotated image in the inference stage without relying on rotation data enhancement; through this image pre-rotation method, the convolutional neural network can abandon the rotation data enhancement operation and reduce the number of parameters that need to be learned. At the same time, the superiority of the present invention in rotation pattern recognition and angle estimation has been verified on data sets such as Rotated MNIST, and can be used for pattern recognition and object detection tasks of general optical images, medical images, remote sensing images, etc.

[0072] The difference between the present invention and the current rotation pattern recognition method is mainly reflected in:

[0073] (1) In order to enable the convolutional neural network to have the ability to recognize rotation patterns, the present invention does not need to change the original structure and does not rely on learning rotation samples in the training phase.

[0074] (2) Most current rotation pattern recognition methods use a single-channel processing method due to computational complexity, while the image pre-rotation method in the inference stage uses a parallel multi-angle channel processing method for the rotation pattern, thereby allowing the angle information to be obtained by calculating the classification response during the object's rotation direction. Although D. Laptev and Dieleman et al. also used a multi-channel processing method, the method of the present invention does not rely on feature fusion decisions, but each rotation angle channel will give a prediction result, truly realizing parallel reasoning in pattern category prediction.

[0075] (3) In the inference stage, the image pre-rotation method can be artificially realized by changing the number of parallel rotation channels according to the task category classification accuracy and angle classification accuracy, thus realizing subjective control of the rotation pattern recognition accuracy and making the rotation pattern recognition mechanistically explainable.

[0076] The contribution of the present invention to the research on rotational pattern recognition is mainly reflected in that rotational pattern recognition is achieved without relying on the learning of rotational examples, and the category and angle can be predicted simultaneously, which expands the application mode for existing research, especially the research on the rotation and other properties of convolutional neural networks. The present invention can be effectively compatible with existing convolutional neural networks. It only needs to adjust the convolutional network structure and the number of channels to effectively improve the adaptability of the convolutional neural network to the rotational pattern without adding additional parameters and learning tasks. In the design of the network structure, a neural network architecture search algorithm can be selected for network structure design to achieve better rotational pattern recognition effect. In addition, the present invention can be effectively adapted to other computer vision tasks, such as target detection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0078] Figure 1 It is a flow chart of a convolutional neural network rotation pattern recognition method provided by an embodiment of the present invention that is pre-rotated only in the inference stage.

[0079] Figure 2 It is a schematic diagram of a convolutional neural network rotation pattern recognition method provided by an embodiment of the present invention that is pre-rotated only in the inference stage.

[0080] Figure 3 It is a structural block diagram of a convolutional neural network plane rotation pattern recognition system based on image pre-rotation provided by an embodiment of the present invention;

[0081] In the figure: 1. Image preprocessing module; 2. Network structure design module; 3. Parameter learning module; 4. Rotation channel construction module; 5. Multi-channel classification confidence matrix calculation module; 6. Object category and direction angle prediction module.

[0082] Figure 4 It is a schematic diagram of the direction angle definition of the plane image rotation mode provided by an embodiment of the present invention.

[0083] Figure 5 It is a flow chart of the rotational multi-channel construction and reasoning method provided by an embodiment of the present invention.

[0084] Figure 6 It is a schematic diagram of the effect of classifying and estimating the angle of the Rotated MNIST and Fashion MNIST data sets provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0085] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0086] In view of the problems existing in the prior art, the present invention provides a convolutional neural network rotation pattern recognition method that is pre-rotated only in the inference stage. The present invention is described in detail below in conjunction with the accompanying drawings.

[0087] like Figure 1 As shown, the convolutional neural network rotation pattern recognition method provided by the embodiment of the present invention, which is only pre-rotated in the inference stage, includes the following steps:

[0088] S101, image sample collection, preprocessing and dataset creation;

[0089] S102, design convolutional neural network structure;

[0090] S103, convolutional neural network parameter learning;

[0091] S104, performing random pre-rotation on the test set images to construct a rotation channel;

[0092] S105, using the complete twin instance of the convolutional neural network trained in S103 to infer the rotation pattern in parallel and calculate the multi-channel classification confidence matrix;

[0093] S106, predicting the object category and direction angle simultaneously from the multi-channel classification confidence matrix according to the maximum confidence criterion.

[0094] like Figure 3As shown, the convolutional neural network plane rotation pattern recognition system based on image pre-rotation provided by an embodiment of the present invention includes:

[0095] Image preprocessing module 1, used for image sample collection, preprocessing and data set preparation;

[0096] Network structure design module 2, used to design the convolutional neural network structure;

[0097] Parameter learning module 3, used to realize convolutional neural network parameter learning;

[0098] A rotation channel construction module 4 is used to randomly pre-rotate the test set images to construct a rotation channel;

[0099] A multi-channel classification confidence matrix calculation module 5 is used to infer the rotation pattern in parallel through the fully twinned instances of the trained convolutional neural network to calculate the multi-channel classification confidence matrix F;

[0100] The object category and direction angle prediction module 6 is used to simultaneously predict the object category and direction angle from the F matrix according to the maximum confidence criterion.

[0101] The present invention will be further described below in conjunction with the embodiments.

[0102] Different from the prior art and experimental methods, the present invention limits the convolutional neural network to learn only a set of standard rotation patterns during training. During the test, multiple rotation channels are established by pre-rotating the input image, and a set of parallel twin networks are used to calculate the classification confidence response of the multiple rotation channels.

[0103] The present invention can simultaneously predict the category and direction angle of the rotation pattern according to the maximum value of the multi-channel response calculated by this learning method; through this image pre-rotation method, the convolutional neural network can abandon the rotation data enhancement operation and reduce the number of parameters that need to be learned. At the same time, the superiority of the present invention in rotation pattern recognition and angle estimation has been verified on data sets such as Rotated MNIST, and can be used for pattern recognition and object detection tasks of general optical images, medical images, remote sensing images, etc.

[0104] In view of the inherent defects of convolutional neural network pattern recognition, the present invention explores a method for performing rotational pattern recognition on multiple rotation channels of an input image, sharing network weights between rotation channels to achieve parallel processing of rotation information, and can directly predict the plane rotation angle of the rotation pattern to be recognized relative to the standard pattern without adding learning tasks (such as regression tasks). Specifically, the present invention first constructs a training data set composed of standard direction patterns for network learning, and does not use rotation data enhancement during training. During reasoning, the image area to be classified is pre-rotated into M different directions and then sent to the same convolutional neural network for pattern classification, and at the same time, the relative direction angle of the object is directly predicted based on the maximum response of the rotation channel.

[0105] like Figure 2 As shown, the convolutional neural network rotation pattern recognition method provided by the embodiment of the present invention, which is only pre-rotated in the inference stage, includes the following steps:

[0106] Step S1: Image sample collection, preprocessing and dataset creation.

[0107] (a) For the image pattern to be recognized, set the respective canonical directions for different categories of patterns (the canonical directions are as follows: Figure 4 Arrow points to).

[0108] (b) Collect image samples for each type of pattern separately, and try to ensure the balance of the number of samples between classes.

[0109] (c) The collected samples are normalized one by one and the samples are rotated to the standard direction.

[0110] The object's rotation angle is recorded as When the object direction is in the positive direction of the y axis When the object rotates counterclockwise, the direction angle is negative, and when it rotates clockwise, the direction angle is positive. For the collected image samples of the i-th pattern, the direction angle may be arbitrary, and the arbitrary rotation pattern can be expressed as The purpose of normalizing the samples is to convert any rotation mode Adjust to standard mode with a direction angle of 0 degrees

[0111] (d) The preprocessed image samples are divided into training, validation and test sets according to a certain ratio. In this embodiment, the ratio is 7:1:2.

[0112] (e) To eliminate the impact of changes in object scale and aspect ratio during rotation, during training and testing, the present invention normalizes the input image into a square before inputting it.

[0113] Step S2: Design the convolutional neural network structure.

[0114] (a) The convolutional neural network structure is reasonably designed according to the task dataset size to obtain higher generalization performance on the test set.

[0115] (b) The convolutional network structure for rotation pattern recognition can be designed manually or using automated machine learning methods such as neural architecture search. The core principle of the design is to ensure that the network scale is adapted to the task dataset and achieve the best generalization performance on the test dataset.

[0116] (c) Specifically for this embodiment, the first layer of the network uses 32 convolution kernels of size 5, the convolution step size is 1, and the "same padding" method is used to fill the image edge feature map output size unchanged;

[0117] The second layer of the network uses a maximum pooling operation with a step size of 2, which reduces the network size by half;

[0118] The third convolutional layer of the network uses 32 convolution kernels of size 5, a convolution step of 1, and the “same padding” method to fill the image edge feature map output size unchanged;

[0119] The fourth layer of the network uses a maximum pooling operation with a step size of 2, which reduces the network size by half;

[0120] The fifth, sixth, and seventh layers of the network use fully connected networks. The fifth layer receives the vector features of the fourth layer feature map flattened, and has 120 neuron outputs;

[0121] The sixth layer of the network is output by 84 neurons;

[0122] The seventh layer of the network consists of 10 neuron outputs, corresponding to the ten categories of the MNIST multi-classification task.

[0123] Step S3: Convolutional neural network parameter learning.

[0124] Softmax regression is used to process multiple neuron outputs of the convolutional neural network, cross entropy loss is used as the network learning criterion, and mini-batch gradient descent is used to learn network parameters.

[0125] Specifically, it is necessary to input the standardized training sample set into the convolutional neural network model in small batches, and then use the cross entropy loss function to calculate the structured loss, and then use the small batch gradient descent to update the network. The learning rate during training uses an adjustable learning rate. Specifically in this embodiment, the network weights are initialized randomly; the initial learning rate is set to 1e-3, and the learning rate decay method uses a cosine annealing model; the gradient estimation correction uses the Adam algorithm, and the batch training size is set to 256. After 600 rounds of training, the parameters of the learned convolutional neural network can be obtained.

[0126] Step S4: Perform random pre-rotation on the test set images to construct a rotation channel.

[0127] (a) If Figure 5 As shown, for any input image with direction θ in the test set (H, W, d represent the image height, width, and number of channels, respectively). First, it is rotated at equal intervals within the range of [-Φ, +Φ] degrees. The number of rotations is N, and the rotation angle interval is ΔΦ = 2Φ / N.

[0128] (b) The modes and their angles after rotation are:

[0129]

[0130]

[0131] In the formula, Φ and ΔΦ represent the expected rotation pattern recognition angle estimation range and accuracy, respectively, which are set according to the task requirements, 0≤Φ≤π, and ΔΦ is adjusted by setting the number of rotation channels N.

[0132] (c) In this embodiment, the image rotation is implemented using a bilinear interpolation algorithm, the rotation angle intervals are set to 1 degree and 0.1 degree, and the estimation intervals Φ are set to π and

[0133] Step S5: Use the fully twinned instance of the convolutional neural network trained in step S3 to infer the rotation pattern in parallel and calculate the multi-channel classification confidence matrix F.

[0134] (a) After pre-rotating the image, the copy of the object pattern in the nth rotation channel is subjected to feature extraction and category prediction using the convolutional neural network trained in step S3:

[0135]

[0136] Among them, f n is the classification response of the nth channel.

[0137] (b) Arrange the convolutional neural network classification confidence outputs of all rotation channels into a rotation multi-channel classification confidence matrix: C is the number of categories to be predicted.

[0138] Step S6: According to the maximum confidence criterion, the object category and direction angle are simultaneously predicted from the F matrix.

[0139] (a) Using classification confidence as the final criterion for pattern recognition, the maximum coordinates (j, k) of the multi-channel response matrix are used to simultaneously predict the object category and direction angle:

[0140]

[0141] (b) When using the classic convolutional neural network for supervised classification, equation (5) uses the maximum classification confidence criterion to predict the category and angle. Other criteria can also be used to construct the F matrix, such as the maximum similarity criterion.

[0142] By executing the above process on the Rotated MNIST and Rotated Fashion MNIST datasets respectively, the category and angle estimation performance can be obtained as shown in Tables 1 and 2.

[0143] Table 1 Classification error of the test set on Rotated MNIST

[0144]

[0145] (Note: "rotX—>rotY" in the table means that the input is randomly rotated within the range of X degrees, and the output is randomly rotated within the range of Y degrees)

[0146] It can be seen from Table 1 that compared with the traditional rotation-invariant pattern recognition method, the pre-rotation method proposed in this paper can achieve better performance in various indicators; in addition, "rot0—>rot180" and "rot0—>rot360" show that even if the training samples do not contain rotation instances, the pre-rotation method can still effectively recognize patterns of unknown rotations.

[0147] Table 2 Classification error of the test set on Rotated Fashion MNIST

[0148]

[0149] As can be seen from Table 2, the present invention is also effective on a more complex model such as Rotated Fashion MNIST, and can achieve recognition performance that exceeds existing methods.

[0150] The difference between the present invention and the current rotation pattern recognition method is mainly reflected in the following aspects: (1) Current rotation pattern recognition mostly adopts a single-channel processing method due to the consideration of computational complexity, while the rotation maximum response method adopts a parallel multi-angle channel processing method for the rotation pattern, thereby allowing the angle information to be obtained by calculating the classification response during the object's rotation direction. (2) Different from the multi-channel processing method of D. Laptev and Dieleman et al., the rotation maximum response method does not rely on fusion decision, but each rotation angle channel will give a prediction result, which truly realizes parallel reasoning on pattern category prediction.

[0151] The contribution of the present invention to the study of rotational pattern recognition is mainly reflected in the realization of simultaneous prediction of categories and angles, which expands the application mode for existing research, especially the study of convolutional neural network rotation and other degeneration. The present invention can be effectively compatible with existing convolutional neural networks, and the adaptability of convolutional neural networks to rotational patterns can be effectively improved by simply adjusting the convolutional network structure and the number of channels, without adding additional parameters and learning tasks. In addition, the present invention can be effectively adapted to other computer vision tasks, such as target detection tasks.

[0152] Tables 1 and 2 show that the present invention can achieve higher rotation pattern classification accuracy than the existing methods. In addition, another advantage of the present invention over the existing pattern recognition is that it can also provide angle information while achieving pattern classification. According to formula (4) of step S6 (a), a schematic diagram of the effect of the present invention on angle estimation of the Rotated MNIST and Fashion MNIST data sets can be obtained. As can be seen from the figure, the present invention has a higher accuracy in rotation pattern angle estimation.

[0153] Figure 6 FIG. 1 is a schematic diagram of the effect of classifying and estimating the angle of the Rotated MNIST and Fashion MNIST datasets provided by an embodiment of the present invention. Figure 6 (a) Recognition effect diagram of Rotated MNIST dataset.

[0154] Figure 6 (b) Recognition effect diagram of Rotated Fashion MNIST dataset.

[0155] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When the use is implemented in whole or in part in the form of a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL) or wireless (e.g., infrared, wireless, microwave, etc.) mode) to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidState Disk (SSD)), etc.

[0156] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A convolutional neural network rotation pattern recognition method that is pre-rotated only in the inference stage, characterized in that: The convolutional neural network rotation pattern recognition method that is pre-rotated only in the inference stage includes a training stage and an inference stage; in the training stage, the convolutional neural network only learns images with a fixed direction; in the inference stage, multiple pre-rotation channels are established to identify and estimate the direction of any rotation pattern; Use the fully twinned instances of the trained convolutional neural network to infer the rotation patterns in parallel and compute the multi-channel classification confidence matrix F; According to the maximum confidence criterion, the object category and direction angle are predicted simultaneously by the F matrix; After pre-rotation of the image, the trained convolutional neural network is used for feature extraction and category prediction on the copy of the object pattern in the nth rotation channel: Among them, f n is the classification response of the nth channel; Arrange the convolutional neural network classification confidence outputs of all rotation channels into a rotation multi-channel classification confidence matrix: in, C is the number of categories to be predicted.

2. The convolutional neural network rotation pattern recognition method according to claim 1, wherein the rotation is performed only in the inference phase. The multiple pre-rotation channels are realized by rotating the input image or the intermediate feature map of the convolutional neural network or the convolution kernel of the convolutional neural network to multiple directions in a certain angle space.

3. The convolutional neural network rotation pattern recognition method according to claim 1, wherein the convolutional neural network rotation pattern recognition method is characterized in that: The category and direction of the rotation mode are estimated simultaneously according to the maximum value of the classification confidence response of each channel.

4. The method for recognizing rotation patterns of convolutional neural networks with pre-rotation only in the inference stage according to claim 1, characterized in that: When the described method of rotating the input image is used to form multiple pre-rotation channels, a group of convolutional neural networks with the same network structure, intermediate feature maps or convolution kernels and weights are used between the channels for inference, and each pre-rotation channel independently calculates the multi-classification confidence.

5. The convolutional neural network rotation pattern recognition method according to claim 1, wherein the rotation is performed only in the inference phase. When the intermediate feature maps of the pre-rotated convolutional neural network are used to form multiple pre-rotated channels, a convolutional neural network with a shared structure is used between the channels, but the intermediate feature maps of one or more layers of the convolutional neural network of each channel are pre-rotated to different directions and then the multi-classification confidence is calculated independently.

6. The method for recognizing rotation patterns of convolutional neural networks with pre-rotation only in the inference stage as claimed in claim 1, characterized in that: When multiple pre-rotated channels are formed by using the convolution kernel of the pre-rotated convolutional neural network, a convolutional neural network with a shared structure is used between the channels, but the convolution kernels of one or more layers of the convolutional neural network of each channel are pre-rotated to different directions and then the multi-classification confidence is calculated independently.

7. The method for recognizing rotation patterns of convolutional neural networks with pre-rotation only in the inference stage according to claim 1, characterized in that: The recognition or direction angle estimation result is obtained by integrating the multi-classification confidence calculated by multiple pre-rotation channels.

8. A convolutional neural network rotation pattern recognition system that is pre-rotated only in the inference stage, which implements the convolutional neural network rotation pattern recognition method that is pre-rotated only in the inference stage as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the convolutional neural network rotation pattern recognition method according to any one of claims 1 to 7, which is pre-rotated only in the inference stage.

10. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the convolutional neural network rotation pattern recognition method that is only pre-rotated in the inference stage as described in any one of claims 1 to 7.