A method and system for scale-invariant pattern perception of image scaling

Abstract

The application provides a scale-invariant image zoom scale mode perception method, comprising the following steps: training a data set of standard scale samples to obtain a trained convolutional neural network; zooming in and out each random scale sample in a test set to form a multi-layer image pyramid composed of multiple scale samples; using the trained convolutional neural network to construct a set of twin convolutional neural networks matched with the number of layers of the multi-layer image pyramid, inputting the multiple scale samples into the set of twin convolutional neural networks for parallel reasoning to obtain a multi-channel classification score matrix; and classifying and estimating the scale according to the classification score matrix. The method has good scale invariance, can classify and estimate the unknown scale at the same time, can be applied to scale-invariant mode recognition tasks of medical images, remote sensing images and the like, and has wider applicability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target recognition, and in particular to an image scaling scale mode perception method and system with scale equivariance. BACKGROUND

[0002] The inherent translation equivariance of convolutional neural networks (CNN) makes CNN achieve good results in target recognition and classification tasks. Real-world objects are three-dimensional, and the change in the distance between the object and the observer will directly lead to the change in the scale of the two-dimensional projection of the object on the retina, causing the enlargement and reduction of the object on the human visual image. Therefore, the scale change of the image is as common as the translation. The intuitive feeling of the human visual system to the object is "near large and far small", which reflects the relationship between the two-dimensional image presented on the human eye retina and the distance between the human eye and the target. Using this feature can represent the depth and volume of the object, so as to represent the three-dimensional object on the two-dimensional picture. The same object in the field of view presents different sizes, but in our cognition, they are the same class of objects under different scales, which is the equivariance of the human eye visual system. However, the traditional convolutional neural network does not have the scale invariance feature. The existing CNN uses pooling operation to adapt to the scale change, but its scale equivariance is limited.

[0003] In order to improve the scale equivariance of CNN, there are currently two methods of multi-scale training and single-scale training. Multi-scale training can be divided into two ways of transforming feature representation and transforming convolution kernel. Both of these two ways are to learn as many different scale samples as possible in the training stage to improve the generalization ability to the scale, but this will lead to an increase in the amount of parameters and computation. Single-scale training uses single-scale samples for training, and uses weight-shared twin CNNs for inference, which reduces the amount of parameters and computation, and achieves good classification effect on the MNIST Large Scale dataset. However, both of the above two methods lack the function of estimating unknown scale. SUMMARY

[0004] In view of at least one defect or improvement demand of the prior art mentioned in the background, the present application provides an image scaling scale mode perception method and system with scale equivariance, to solve the technical problem of how to estimate the scale while classifying under the premise of ensuring scale equivariance.

[0005] In order to solve the above technical problems, in a first aspect, the present application provides an image scaling scale mode perception method with scale equivariance, comprising:

[0006] training a data set of standard scale samples to obtain a trained convolutional neural network;

[0007] Enlarging and reducing each random scale sample in the test set to form a multi-layer image pyramid composed of multiple scale samples;

[0008] Using the trained convolutional neural network to construct a set of twin convolutional neural networks matched with the number of layers of the multi-layer image pyramid, sending the multiple scale samples into the set of twin convolutional neural networks for parallel inference to obtain a multi-channel classification score matrix;

[0009] Classifying and estimating scale according to the classification score matrix.

[0010] According to the scale mode perception method provided by the application, the training method for training by using the data set of standard scale samples specifically comprises:

[0011] A convolutional neural network with a fixed structure is used for model training, cross-entropy loss is used as the network learning criterion, and the Adam algorithm is used for network parameter optimization, and an equal interval adjustment learning rate strategy is used.

[0012] According to the scale mode perception method provided by the application, the enlargement and reduction of each random scale sample in the test set specifically comprises:

[0013] The random scale sample passing through the image enlargement channel is up-sampled by using a bilinear interpolation method;

[0014] The random scale sample passing through the image reduction channel is down-sampled by using a two-dimensional discrete wavelet transform combined with an interpolation method.

[0015] According to the scale mode perception method provided by the application, the multiple channels of the set of twin convolutional neural networks share weights.

[0016] According to the scale mode perception method provided by the application, the classification and scale estimation according to the classification score matrix specifically comprises:

[0017] First, the maximum value of the classification score matrix is obtained to obtain the corresponding horizontal and vertical coordinates, and then the prediction category and scale estimation value are obtained according to the horizontal and vertical coordinates.

[0018] According to the scale mode perception method provided by the application, the method further comprises:

[0019] The classification score matrix is subjected to a logsoftmax operation, and classification and scale estimation are performed based on the classification score matrix after the logsoftmax operation.

[0020] In a second aspect, the application further provides an image scaling scale mode perception system with scale invariance, which can realize the steps of the scale mode perception method described in any one of the above aspects.

[0021] The beneficial effects of the present application relative to the prior art are:

[0022] (1) The traditional learning task is scale-independent learning, that is, different scales of images under the same label are regarded as the same target, and the label is independent of the scale. In the process of network model testing, the present application produces different responses to the same label images of different scales, predicts the class and scale at the same time, and realizes scale-invariant pattern recognition without relying on learning multiple scale instances, better estimates the scale of the sample by using the scale equivariance of the network, and expands the application mode for existing research, especially the scale equivariance research of convolutional neural network.

[0023] (2) Since only one scale pattern needs to be learned in the training process, compared with the traditional data enhancement learning of multiple scale training samples, the method greatly reduces the training time. During testing, the parameters between different channels are shared, and a set of network weights can realize the recognition of multiple scale targets, which greatly reduces the parameter amount compared with the traditional data enhancement method. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0025] Figure 1 is a flow chart of an image scaling scale pattern perception method provided by an embodiment of the present application with scale equivariance;

[0026] Figure 2 is a flow chart of an image scaling scale pattern perception method provided by an embodiment of the present application with scale equivariance;

[0027] Figure 3 is an output response graph provided by an embodiment of the present application on a data set with a scale change range of [1 / 4, 4];

[0028] Figure 4 is an output response graph provided by an embodiment of the present application on a data set with a scale change range of [1 / 4, 4]; Figure 3 is a slice graph in the class dimension;

[0029] Figure 5 is a slice graph in the scale dimension; Figure 3 is a slice graph in the scale dimension;

[0030] Figure 6is a visualization result figure of scale estimation on MNIST Large Scale dataset provided by the embodiment of the present application, wherein (a) is an original test image, and (b) is an image after scale correction. DETAILED DESCRIPTION

[0031] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work belong to the protection scope of the present application.

[0032] Reference Figure 1 and Figure 2 An embodiment of the present application provides an image scaling scale mode perception method with scale invariance, specifically comprising the steps of S1-S4:

[0033] Step S1, training is performed by using a data set of standard scale samples to obtain a trained convolutional neural network.

[0034] Specifically, a convolutional neural network with a fixed structure is used for model training, cross-entropy loss is used as a network learning criterion, Adam algorithm is used for network parameter optimization, and an equal interval adjustment learning rate strategy is used.

[0035] More specifically, a data set containing 50,000 standard scale images is used for training. The structure of the network is 4 convolutional layers plus 2 fully connected layers, the number of feature channels of the convolutional layers is 16, 16, 32 and 32 respectively, there are batch normalization and ReLU activation operations after each convolution operation, and then maximum pooling is performed. The number of layers of the fully connected layers is 100 and 10 respectively, the first fully connected layer uses random inactivation with a probability of 15%. The Adam gradient descent optimization algorithm is used for training for 20 epochs, the batch size is 128, the initial learning rate is 3×10 -3 , the equal interval adjustment learning rate strategy is used, the learning rate is decayed every 2 rounds, the decay coefficient is 0.1, and the learning rate is decayed to 5×10 -5 .

[0036] Step S2, each random scale sample in the test set is enlarged and reduced to form a multi-layer image pyramid composed of multiple scale samples.

[0037] In the test phase, the embodiment uses a test set containing 10,000 random unknown scale samples. Each scale unknown input sample x is subjected to scaling processing to realize the enlargement and reduction of the input sample, and a multi-layer image pyramid composed of multiple scale samples is formed, as shown inFigure 2 In this embodiment, the scale of the test sample is unknown, and the scale change range is [1 / 2, 8]. The specific method is as follows: scale the input sample x according to the equal ratio scale set S = {s1, s2,..., sL} to generate an image pyramid composed of multiple scale samples L L is the number of channels (the number of channels is equal to the number of layers of the image pyramid), and in this embodiment L = 9. When s i > 1, that is, when the input sample passes through the image enlargement channel, the input sample is up-sampled, and the bilinear interpolation method is adopted as in FovMax proposed by Lindeberg; when s i < 1, that is, when the input sample passes through the image reduction channel, the input sample is down-sampled, and different from the method of introducing a Gaussian derivative limit bandwidth, here a biorthogonal discrete wavelet transform combined with interpolation is adopted to avoid spectral aliasing caused by down-sampling, that is, for the result after biorthogonal discrete wavelet transform, only the low-frequency component is retained. For biorthogonal reduction, a discrete wavelet transform is adopted; for non-biorthogonal reduction, a bilinear interpolation is performed on the result of the next level wavelet transform and then enlarged to obtain.

[0038] Step S3, using the trained convolutional neural network to construct a set of twin convolutional neural networks matched with the number of layers of the multi-layer image pyramid, and sending the multiple scale samples into the set of twin convolutional neural networks for parallel inference to obtain a multi-channel classification score matrix.

[0039] After the input sample x passes through the multi-layer image pyramid in the previous step, L images of different scales are generated and input into the twin CNNs for parallel inference. Preferably, the weights between the multiple channels of the twin CNNs are shared.

[0040] The twin CNNs have corresponding L channels. The classification score of the i-th channel is:

[0041]

[0042] The output f i of each inference channel is a flattened vector, and the prediction score of the i-th channel for the j-th class is f ij . The flattened vectors are stacked into an LxC two-dimensional array to obtain a two-dimensional classification score matrix:

[0043]

[0044] ​In order to prevent the overflow of the numerical value and improve the stability of the data, preferably, the two-dimensional classification score matrix F is subjected to a logsoftmax operation, and the specific formula is as follows:

[0045]

[0046] Step S4, classification and scale estimation are performed according to the classification score matrix.

[0047] According to the maximum value coordinates (i, j) of the two-dimensional classification score matrix F', the SA Net can obtain the category of the predicted input sample and the scale estimation value The formula is as follows:

[0048] (i, j) = argmax (F')

[0049]

[0050] The pseudo code of the classification and scale estimation algorithm is as follows:

[0051]

[0052]

[0053] In the embodiment, the classification results on the data set in the range of [1 / 2, 8] are shown in Table 1.

[0054]

[0055] Table 1

[0056] In order to evaluate the performance of the scale estimation, the lower limit of the data set can be expanded, that is, the data set of [1 / 4, 4] is used, and the three-dimensional output response graph shown in Figure 3 is obtained. In order to more intuitively evaluate the effect of the scale estimation, a slice graph can be drawn according to the three-dimensional output response graph, as shown in Figure 4 , 5 In the category dimension, the non-smooth curve leads to the appearance of multiple side lobes, and the side lobes will interfere with the classification task. The coordinates corresponding to the side lobes are the wrong categories, indicating that the correlation between adjacent categories in the classification task is weak. However, the amplitude of the side lobe is always smaller than that of the main lobe. The coordinates corresponding to the maximum response in the category dimension always correspond to the GT category, verifying that the SA Net can always achieve high classification performance in the classification task shown in Table 1. In the scale dimension, the curve is relatively smooth, and there is only one peak without side lobe interference. Even if the scale estimation is wrong, the estimated value deviates from the true scale value not far, which is acceptable within the error range. The relative error on different scales is represented by the relative value of RMSE, and the specific values are shown in Table 2. It can be seen that the relative RMSE can always be less than 30% on any scale.

[0057]

[0058] Table 2

[0059] The correction of the test sample scale using the scale estimate is equivalent to the inverse of step S2, and the corrected samples are shown in Figure 6

[0060] One embodiment of the present application also provides a scale-invariant image zoom scale mode perception system, which can implement the steps of the scale mode perception method described above.

[0061] Those skilled in the art can understand that the features described in various embodiments and / or claims of the present disclosure can be combined and / or integrated in various combinations, even if such combinations or integrations are not explicitly described in the present disclosure. In particular, the features described in various embodiments and / or claims of the present disclosure can be combined and / or integrated in various combinations without departing from the spirit and teachings of the present disclosure, and all such combinations and / or integrations fall within the scope of the present disclosure.

[0062] Although the present disclosure has been shown and described with reference to certain exemplary embodiments thereof, it should be understood by those skilled in the art that various changes in form and details can be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents. Therefore, the scope of the present disclosure should not be limited to the above-described embodiments, but should be determined by the appended claims alone, and should be defined by the equivalents of the appended claims.​

Claims

1. A method for scale-invariant pattern perception in image resizing with scale mode perception, characterized in that, The method comprises the following steps: training a convolutional neural network using a data set of standard scale samples to obtain a trained convolutional neural network; enlarging and reducing each random scale sample in a test set to form a multi-layer image pyramid composed of multiple scale samples; constructing a set of twin convolutional neural networks matched with the number of layers of the multi-layer image pyramid using the trained convolutional neural network, and inputting the multiple scale samples into the set of twin convolutional neural networks for parallel inference to obtain a multi-channel classification score matrix; classifying and estimating the scale according to the classification score matrix; wherein the classification and scale estimation according to the classification score matrix is specifically: first obtaining the corresponding horizontal and vertical coordinates according to the maximum value of the classification score matrix, and then obtaining the predicted category and scale estimation value according to the horizontal and vertical coordinates; The method further comprises: performing a logsoftmax operation on the classification score matrix, and classifying and estimating the scale based on the classification score matrix after the logsoftmax operation.

2. The scale mode perception method of claim 1, wherein, The training method using a data set of standard scale samples for training is specifically: using a convolutional neural network with a fixed structure for model training, using cross-entropy loss as the network learning criterion, using the Adam algorithm for network parameter optimization, and using an equal interval adjustment learning rate strategy.

3. The scale mode perception method of claim 1, wherein, The enlarging and reducing each random scale sample in a test set specifically comprises: upsampling the random scale sample through the image enlargement channel using a bilinear interpolation method; downsampling the random scale sample through the image reduction channel using a two-dimensional discrete wavelet transform combined with an interpolation method.

4. The scale mode perception method of claim 1, wherein, The multiple channels of the set of twin convolutional neural networks share weights.

5. A scale-invariant image scaling dimension mode perception system, comprising: The scale mode perception system can implement the steps of the scale mode perception method of any one of claims 1-4.

Images