Convolutional neural network and image processing method based on convolutional neural network

Abstract

The present disclosure relates to a convolutional neural network and an image processing method based on the convolutional neural network, wherein the convolutional neural network comprises: at least one convolutional layer, each convolutional layer comprises at least one convolutional kernel group, each convolutional kernel group comprises more than two convolutional kernels, each convolutional kernel of each convolutional kernel group extracts the same features of input image data with different rotation angles to obtain a feature map, and a convolutional kernel of the convolutional kernel group can obtain other convolutional kernels of the convolutional kernel group after rotation; and at least one pooling layer, each pooling layer uses a pooling window corresponding to each convolutional kernel of the convolutional kernel group to pool the feature map output by each convolutional kernel to output the pooled feature map, wherein the pooling window corresponding to the other convolutional kernels in the convolutional kernel group can be obtained by rotating the pooling window around the center of the pooling window corresponding to a convolutional kernel of the convolutional kernel group. The convolutional neural network of the present disclosure itself has rotation invariance.

Description

Technical Field

[0001] The present disclosure relates to the technical field of neural network, and in particular to a convolutional neural network and an image processing method based on the convolutional neural network. Background Art

[0002] In recent years, deep neural networks have been widely used in the field of pattern recognition. For example, deep neural networks have been applied to target detection, face recognition, semantic segmentation, etc. These applications are closely related to the ability of deep neural networks to learn features with strong expressive power (image representation methods). The manual feature extraction method can extract low-level features of images by introducing expert knowledge. Compared with the traditional manual feature extraction method, deep neural networks rely on data-driven and can learn various levels of features of images, including low-level, mid-level and high-level features. The working method of neural network to extract image features is similar to the human visual system, so it is better than the manual feature extraction method in terms of recognition effect.

[0003] The inherent translation invariance of convolutional neural networks makes them very robust to the location of the target in the image. However, convolutional neural networks are sensitive to rotation and scaling and are not rotation and scale invariant. Recently, some schemes have been proposed to introduce rotation invariance into convolutional neural networks. These schemes can be roughly divided into two categories:

[0004] 1) Using the data augmentation method, the samples in the training set are rotated to obtain an augmented data set, which is then used to train ordinary convolutional neural network structures or some specially designed convolutional neural network structures to obtain a convolutional neural network model with rotation invariance.

[0005] 2) There is no need to perform data augmentation on the samples in the training set, but the structure of the convolution kernel of the convolutional neural network is changed to obtain a convolutional neural network model with rotation invariance.

[0006] Data augmentation can achieve a certain degree of rotation invariance, but data augmentation requires data preprocessing. At the same time, augmentation increases the sample size of the training set exponentially, which increases the training cost. In addition, data augmentation requires a larger convolutional neural network model, which increases the risk of model overfitting. The existing method of changing the convolution kernel structure cannot actively perform a rotation transformation on the input. Summary of the invention

[0007] The present disclosure is made in consideration of the above-mentioned problems existing in the prior art. The present disclosure adopts a rotating convolution kernel and a pooling window to bring rotational invariance to the convolutional neural network, thereby overcoming one or all of the above-mentioned defects in the prior art.

[0008] According to one aspect of an embodiment of the present disclosure, a convolutional neural network is provided, comprising: at least one convolutional layer, each convolutional layer comprising at least one convolutional kernel group, each convolutional kernel group comprising more than two convolutional kernels, each convolutional kernel of each convolutional kernel group performing feature extraction on the same features of input image data with different rotation angles to obtain a feature map, wherein one convolution kernel of the convolutional kernel group can obtain other convolution kernels of the convolutional kernel group after rotation; and at least one pooling layer, each pooling layer using pooling windows corresponding to each convolution kernel of the convolutional kernel group to pool the feature maps output by each convolution kernel to output a pooled feature map, wherein by rotating the pooling window corresponding to a convolution kernel of the convolutional kernel group around the center of the pooling window, the pooling windows corresponding to other convolution kernels in the convolutional kernel group can be obtained.

[0009] In an exemplary embodiment according to the present disclosure, the convolutional neural network also includes a feature map selection layer, which compares the feature maps of each convolution kernel that has been pooled by all pooling layers and originates from the same convolution kernel group, and selects the feature map with the maximum value as the feature map extracted by the convolution kernel group for the image data.

[0010] In an exemplary embodiment according to the present disclosure, the convolution kernel group includes a first convolution kernel, the first convolution kernel is represented by F0, and the other convolution kernels in the convolution kernel group are represented by F i Indicates that other convolution kernels F i Obtained by the following formula:

[0011] F i (c, p, q) = F0(c, p′, q′), where

[0012] Among them, i represents the index of each convolution kernel in the convolution kernel group, c represents the number of channels of the convolution kernel, p and q represent the height and width indexes of the i-th convolution kernel respectively, and θ i It represents the angle of rotation of the i-th convolution kernel from the first convolution kernel, and p' and q' represent the indexes of the height and width of the first convolution kernel respectively.

[0013] In an exemplary embodiment according to the present disclosure, the weight of the first convolution kernel changes as the convolutional neural network is trained.

[0014] In an exemplary embodiment according to the present disclosure, the number of channels of each convolution kernel in the convolution layer that performs feature extraction on the feature map after pooling of the previous pooling layer is 1 / n of the number of channels of the feature map after pooling of its input, where n represents the number of different rotation angles and is a natural number greater than 2.

[0015] In an exemplary embodiment according to the present disclosure, the center of each pooling window corresponding to the convolution kernels other than the first convolution kernel in the convolution kernel group is expressed as (h θi ,w θi ), then the center of the pooling window (h θi ,w θi ) is obtained by the following formula:

[0016]

[0017] Among them, (h θo ,w θo ) represents the center of the pooling window corresponding to the first convolution kernel, θ i Denotes the self-center (h θo ,w θo )The angle of rotation.

[0018] According to another aspect of the embodiments of the present disclosure, there is provided an image processing method based on a convolutional neural network, the method comprising: extracting features of the same features of input image data with different rotation angles by each convolution kernel of each convolution kernel group in each convolution layer of the convolutional neural network, wherein one convolution kernel of the convolution kernel group can obtain other convolution kernels of the convolution kernel group after rotation; and pooling the feature maps output by each convolution kernel using pooling windows respectively corresponding to each convolution kernel of the convolution kernel group by each pooling layer of the convolutional neural network to output a pooled feature map, wherein by rotating the pooling window around the center of the pooling window corresponding to a convolution kernel of the convolution kernel group, the pooling windows corresponding to other convolution kernels in the convolution kernel group can be obtained.

[0019] In an exemplary embodiment according to the present disclosure, the image processing method based on convolutional neural network also includes: a feature map selection layer of the convolutional neural network compares the feature maps of each convolution kernel that has been pooled by all pooling layers and originates from the same convolution kernel group, and selects the feature map with the maximum value as the feature map extracted by the convolution kernel group for the image data.

[0020] In an exemplary embodiment according to the present disclosure, the convolution kernel group includes a first convolution kernel, the first convolution kernel is represented by F0, and the other convolution kernels in the convolution kernel group are represented by F i Indicates that other convolution kernels F i Obtained by the following formula:

[0021] F i (c, p, q) = F0(c, p′, q′), where

[0022] Among them, i represents the index of each convolution kernel in the convolution kernel group, c represents the number of channels of the convolution kernel, p and q represent the height and width indexes of the i-th convolution kernel respectively, and θ i It represents the angle of rotation of the i-th convolution kernel from the first convolution kernel, and p' and q' represent the indexes of the height and width of the first convolution kernel respectively.

[0023] In an exemplary embodiment according to the present disclosure, the weight of the first convolution kernel changes as the convolutional neural network is trained.

[0024] In an exemplary embodiment according to the present disclosure, the number of channels of each convolution kernel in the convolution layer that performs feature extraction on the feature map after pooling of the previous pooling layer is 1 / n of the number of channels of the feature map after pooling of its input, where n represents the number of different rotation angles and is a natural number greater than 2.

[0025] In an exemplary embodiment according to the present disclosure, the center of each pooling window corresponding to the convolution kernels other than the first convolution kernel in the convolution kernel group is expressed as (h θi ,w θi ), then the center of the pooling window (h θi ,w θi ) is obtained by the following formula:

[0026]

[0027] Among them, (h θo ,w θo ) represents the center of the pooling window corresponding to the first convolution kernel, θ i Denotes the self-center (h θo ,w θo )The angle of rotation.

[0028] According to another aspect of an embodiment of the present disclosure, there is provided an image processing device, characterized in that it includes a memory and a processor, the processor is connected to the memory, the memory stores program instructions executed by the processor, and the processor can execute the above-mentioned methods when executing the program instructions.

[0029] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein a computer program including program instructions is stored on the computer-readable storage medium, and the program instructions execute the above-mentioned methods when executed by a processor or a computer.

[0030] According to another aspect of an embodiment of the present disclosure, a processor is provided, wherein the processor is used to run a program, wherein the program executes the above methods when run by a computer or a processor.

[0031] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, wherein the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the above methods.

[0032] According to the convolutional neural network and the image processing method based on the convolutional neural network of each embodiment of the present disclosure, by directly improving the convolution kernel and the pooling layer, the convolutional neural network has rotation invariance from its own structure, so that the rotation invariance no longer depends on data augmentation. In addition, the convolutional neural network of the present disclosure reduces the number of channels of the convolution kernel and the parameters of the convolution kernel, thereby reducing the amount of calculation and memory usage while maintaining good performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:

[0034] Figure 1 is a schematic diagram of a multi-column neural network according to the prior art;

[0035] Figure 2 is a schematic diagram of a space transformer network (STN) according to the prior art;

[0036] Figure 3 is a schematic diagram of a convolutional neural network structure according to an embodiment of the present disclosure;

[0037] Figure 4 is a diagram of image data according to an exemplary embodiment of the present disclosure;

[0038] Figure 5A is a schematic diagram of a feature map output by a convolution kernel according to an exemplary embodiment of the present disclosure;

[0039] FIG. 5B to FIG. 5D is a schematic diagram of a pooling window according to an exemplary embodiment of the present disclosure;

[0040] Figure 6 is a flowchart of an image processing method based on a convolutional neural network according to another embodiment of the present disclosure; and

[0041] Figure 7 is a block diagram of an image processing apparatus according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The following exemplary embodiments are only used to illustrate the present disclosure, but are not used to limit the scope of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without making creative work should fall within the scope of protection of the present disclosure.

[0043] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or modules or units is not necessarily limited to those steps or modules or units explicitly listed, but may include other steps or modules or units that are not explicitly listed or inherent to these processes, methods, products or devices.

[0044] To facilitate understanding of the present disclosure, the following is a description of key terms:

[0045] Feature: A method of representing an image. Traditional methods use pixels in the three channels of RGB to represent images. In order to better utilize computers for recognition, it is necessary to filter out redundant information in RGB and extract more semantic features. Images are often represented as a vector, which is called a feature. Image features contain some prominent information in the image, such as contour edges, colors, etc.

[0046] Scale: The size of the image.

[0047] Feature invariance: How sensitive a feature is to geometric changes. If the image changes a little but the feature remains unchanged, then the feature is invariant.

[0048] Data augmentation: When the image is read into the neural network, some transformation is actively performed on the image, such as scaling, cropping, rotating, etc. Each transformation can provide corresponding feature invariance for the neural network.

[0049] Rotation invariance: When the input image is rotated, the output features remain unchanged. The formula is as follows: Among them, θ represents the rotation angle of the input image of any size, output represents the output feature, and input represents the input image.

[0050] Figure 1 is a schematic diagram of a multi-column neural network according to the prior art. Figure 1 The neural network shown is a multi-column neural network with rotation invariance obtained by using the data augmentation method. Figure 1 As shown in the figure, the bottom part is the input image, the square represented by the filter located on the top of the input image, the leftmost one is the standard filter, and the right side of the standard filter is a plurality of transformation filters. The aforementioned filters actively perform some transformation on the image read into the neural network, such as image scaling, cropping, rotation, etc. Each transformation can provide corresponding feature invariance for the neural network. The processed image is then input into the convolution layer and pooling layer of the convolutional neural network for feature extraction. The feature vector obtained by feature extraction of the image is cascaded and input into the top classification layer for classification.

[0051] Figure 1 The multi-column neural network shown adopts a multi-column structure. Each column network reads an image of a rotation angle, and finally the output feature vectors of each column network are cascaded as a multi-angle representation of the image. The neural network needs to perform data preprocessing in each of the N columns, and train the preprocessed images in the N columns, which increases the difficulty of training and the image classification accuracy is very low.

[0052] Figure 2 FIG. 1 is a schematic diagram of a space transformer network (STN) according to the prior art. Figure 2 The spatial transformer network shown is arranged before the neural network structure to rotate the image data to be input into the neural network structure.

[0053] like Figure 2 As shown, the left cube U represents the input of the space transformer network, and the right cube V represents the output of the space transformer network. The space transformer network 20 includes three submodules: a positioning network 22, a grid generator 24, and a sampler 26. The positioning network 22 and the grid generator 24 define the coordinate correspondence and sampling parameters of the input and output (e.g., image) of the space transformer network. The sampler 26 performs sampling of the input, such as bilinear sampling.

[0054] After the spatial transformer network 20 is set as an independent unit to the data layer of the neural network structure, it actively performs geometric transformations (scaling, rotation, translation, etc.) on the input data, so that the network learns the corresponding geometric invariance. The parameters of the geometric transformation are generally manually set initially and then updated during training. Each geometric transformation requires the introduction of a separate, corresponding spatial transformer network 20. For example, if you want to use the spatial transformer network 20 to model rotation, scaling, and translation, you need to add three spatial transformer networks 20 to the network structure.

[0055] When the spatial transformer network is used after the data (image) layer, its output corresponds to an image at another angle. The deep neural network can learn angle invariance by learning on these images that are different from the original angle. This is similar to the idea of ​​data augmentation. However, when the spatial transformer network is used in the middle or high level of the network, its input is the middle and high level features of the original angle, but the output is not the middle and high level features of another angle. In other words, if the image is interpolated and rotated, the output is the rotated image; but if the image features are interpolated and rotated, the output is not the rotated features. Therefore, at this time, the spatial transformer network cannot actively perform rotation transformation on the input.

[0056] like Figure 1 The first method of the example uses data augmentation, which can achieve a certain degree of rotation invariance, but the data augmentation method requires preprocessing of the data, which increases the sample size of the training set exponentially. Data augmentation requires a larger convolutional neural network model, for example, using a neural network model with N columns, which increases the risk of overfitting the model. Figure 2 Although the second method shown can use a single-column neural network model, it cannot be used for the middle or high layers of the network. In addition, although only a single-column neural network is used, the image classification effect is not good when the image is rotated at a large angle. In addition, other existing convolutional neural networks (CNNs) cannot be used without reference Figure 1 and Figure 2 The described method does not have rotation invariance when performing data augmentation. For example, in a classification task, a convolutional neural network model trained without data augmentation cannot recognize images taken after the camera is rotated. When data augmentation is performed, the data cost of the training set increases, which increases the amount of computation and memory usage.

[0057] The present disclosure introduces an active rotating filter (ARF) in an orientation response network (ORN), and adopts a rotating convolution kernel to bring rotation invariance to a convolutional neural network, which is a method that does not require data augmentation. The present disclosure introduces a rotating convolution kernel and a directional maximum pooling layer on the basis of an orientation response network, so that the convolutional neural network still has rotation invariance without data augmentation. The present disclosure enables the convolutional neural network to better recognize images taken after the camera is rotated as long as it can recognize upright (unrotated) images taken by a camera, for example.

[0058] According to one aspect of an embodiment of the present disclosure, a convolutional neural network is provided, comprising: at least one convolutional layer, each of the convolutional layers comprising at least one convolutional kernel group, each of the convolutional kernel group comprising more than two convolutional kernels, each of the convolutional kernel groups performing feature extraction on the same features of input image data with different rotation angles to obtain a feature map, wherein one convolution kernel of the convolutional kernel group can be rotated to obtain other convolution kernels of the convolutional kernel group; and at least one pooling layer, each pooling layer using pooling windows corresponding to each convolution kernel of the convolutional kernel group to pool the feature maps output by each convolution kernel to output a pooled feature map, wherein by rotating the pooling window corresponding to a convolution kernel of the convolutional kernel group around the center of the pooling window, the pooling windows corresponding to other convolution kernels in the convolutional kernel group can be obtained.

[0059] A convolutional neural network according to one aspect of an embodiment of the present disclosure is described in detail below with reference to the accompanying drawings. Figure 3 Schematically shows a convolutional neural network according to an embodiment of the present disclosure. Figure 3 As shown, the convolutional neural network 30 includes, from the top to the bottom, a data layer 302, a first convolutional layer 304, a first pooling layer 306, a second convolutional layer 308, a second pooling layer 310, a feature map selection layer 312, a classification layer 314, and third to Nth convolutional layers and third to Nth pooling layers not shown between the second pooling layer 310 and the feature map selection layer 312, where N represents a natural number greater than 3.

[0060] The data layer 302 represents an input image represented by three RGB channels. For example, the data layer 302 can be any image captured by a device such as a camera or a mobile phone. Figure 4 FIG. 1 is a schematic diagram of image data according to an exemplary embodiment of the present disclosure. Figure 4 The image data shown can be used as image data in a data layer. The image data includes an image of the captured number "3", the left side represents an unrotated (upright) image, that is, an image with a rotation angle of 0°, and the right side image represents an image after rotation of 180°. Figure 4 The image data shown is for illustration purposes only, and the rotation angle between the image data can be any angle.

[0061] The number of convolutional layers and pooling layers can be set according to the needs of data processing. Figure 3 FIG. 3 shows a first convolutional layer 304 and a second convolutional layer 308, but a convolutional neural network may have only one convolutional layer. Figure 3 A first pooling layer 306 and a second pooling layer 310 are shown in FIG. 1 , but a convolutional neural network may have only one pooling layer.

[0062] like Figure 3As shown, there are four convolution kernels in the first convolution layer 304 and six convolution kernels in the second convolution layer 308. Here, the number of convolution kernels is only an example, and the number of convolution kernels in each convolution layer can be an even number and can be different from each other. The number of convolution kernels in each convolution layer can be set according to actual needs.

[0063] like Figure 3 As shown, the first convolution layer 304 includes two convolution kernel groups, wherein two convolution kernels connected by parentheses on the left side form a first convolution kernel group, and two convolution kernels connected by parentheses on the right side form a second convolution kernel group. Here, the number of convolution kernel groups is only an example, and each convolution layer includes at least one convolution kernel group. According to the number of image features to be extracted, a considerable number of convolution kernel groups can be set. Figure 3 The number of convolution kernels in the convolution kernel group shown in FIG is 2, but this is only an example. Each convolution kernel of each convolution kernel group extracts the same features of the input image data with different rotation angles to obtain a feature map. Figure 3 The neural network shown is Figure 4 The image data is taken as an example to be processed. Figure 4 The image data shown only contains two rotation angles: 0° and 180°. Accordingly, each convolution kernel group includes two convolution kernels, which are used to process the image data of 0° and 180° respectively. Depending on the rotation angle of the image to be processed, the convolution kernel group may contain a considerable number of convolution kernels.

[0064] The two convolution kernel groups in the first convolution layer 304 respectively extract two different features of the input image. For example, the left convolution kernel group extracts the outline of the number "3" as a feature, and the right convolution kernel group extracts the color of the number "3" as a feature. In other examples, the two convolution kernel groups in the first convolution layer 304 can extract two identical features of the input image. Each convolution kernel in the convolution kernel group has the same elements, and can be obtained by rotating each other by corresponding angles. For example, for each convolution kernel group in the first convolution layer 304 and the second convolution layer 308, the convolution kernel on the right can be obtained by rotating the convolution kernel on the left by 180°. Accordingly, the left convolution kernel in each convolution kernel group outputs a feature map of an image with a rotation angle of 0°, and the right convolution kernel outputs a feature map of an image with a rotation angle of 180°, wherein the features extracted by the two convolution kernels in the same convolution kernel group are the same features.

[0065] A convolution kernel of a convolution kernel group can be rotated to obtain other convolution kernels of the convolution kernel group. According to the embodiment of the present disclosure, by introducing a rotation convolution kernel group in a convolution neural network, the convolution neural network can achieve rotation invariance of feature extraction. The rotation invariance of feature extraction is specifically described below through formulas 1 to 3.

[0066]

[0067] Formula 1 represents the convolution process of an image with a rotation angle of 0°, the size of the convolution kernel is 3x3, and the step size is 1.

[0068]

[0069] Formula 2 represents the convolution process for an image rotated 90° with the convolution kernel unchanged. It can be seen that the images of the same object will be obtained, but due to different shooting angles, different features will be extracted, resulting in the performance of the convolutional neural network being reduced.

[0070]

[0071] According to an embodiment of the present disclosure, the convolution kernel is also rotated 90°, as expressed in Formula 3, and its output is the same as the output of Formula 1, indicating that the same features are obtained. Therefore, although the image is rotated, the neural network of the present disclosure obtains the same features when processing each image before and after the rotation, indicating that the neural network of the present disclosure has achieved rotational invariance in image feature extraction.

[0072] The feature map output by the first convolution layer 304 has 4 channels, which are, from front to back, the feature map of the image with a rotation angle of 0° extracted by the left convolution kernel of the left convolution kernel group, the feature map of the image with a rotation angle of 180° extracted by the right convolution kernel of the left convolution kernel group, the feature map of the image with a rotation angle of 0° extracted by the left convolution kernel of the right convolution kernel group, and the feature map of the image with a rotation angle of 180° extracted by the right convolution kernel of the right convolution kernel group. The feature map with 4 channels is input to the first pooling layer 306. As mentioned above, the rotation angles of 0° and 180° are only for illustrative purposes, and the rotation angle can be any other angle. The number of convolution kernels set in the convolution kernel group may depend on the number of rotation angles.

[0073] Pooling layers reduce the complexity and spatial resolution of image data by reducing the size of features, making the output less sensitive to changes. Figure 3 As shown, each convolution layer is followed by a corresponding pooling layer. In other words, the convolution layer and the pooling layer appear in pairs. Each pooling layer uses the pooling window corresponding to each convolution kernel of the convolution kernel group to pool the feature map output by each convolution kernel to output the pooled feature map. Accordingly, Figure 3The pooling layer 306 shown will include two pooling windows, corresponding to the two convolution kernels in the convolution kernel group, specifically, the first pooling window corresponds to the convolution kernel on the left side of each convolution kernel group for feature extraction of images with a rotation angle of 0°, and is used to pool the feature map output by the convolution kernel on the left side of each convolution kernel group; the second pooling window corresponds to the convolution kernel on the right side of each convolution kernel group for feature extraction of images with a rotation angle of 180°, and is used to pool the feature map output by the convolution kernel on the right side of each convolution kernel group, that is, when the pooling function pools the feature map output by each convolution kernel, different pooling windows are applied according to the different rotation angles of the images to be processed for each convolution kernel. In the example disclosed in the present invention, pooling is achieved by taking the maximum value of a set of m×m elements output by the previous convolution layer, so the formula of the pooling function is as follows:

[0074] O=max(i0,i1,i2,...,i m×m-1 ) (Formula 4)

[0075] Where O represents the output, i0,i1,i2,...,i m×m-1 Represents the elements covered by the m×m pooling window, where m is an integer greater than 0.

[0076] A pooling window refers to a window used to cover a group of elements of a feature map when a pooling function processes the feature map. According to an embodiment of the present disclosure, by rotating the pooling window around the center of a pooling window corresponding to a convolution kernel of the convolution kernel group, a pooling window corresponding to other convolution kernels in the convolution kernel group can be obtained.

[0077] Refer to the following FIG. 5A to FIG. 5D and Figure 3 The pooling process of the pooling layer 306 is described in detail. Figure 5A It is shown by Figure 3 The feature map output by the first convolutional layer 304 is shown. Figure 5A As shown, the feature maps are extracted from front to back in the order of 0°, 180°, 0°, and 180° image data, hereinafter referred to as 0° feature map, 180° feature map, 0° feature map, and 180° feature map. Different pooling windows are used for feature maps with different rotation angles, specifically, pooling windows with different divisions are used for pooling processing.

[0078] FIG. 5B to FIG. 5D is a schematic diagram of pooling window division according to an exemplary embodiment of the present disclosure, wherein: Figure 5B The pooling window used when pooling the feature map at 0° is shown. Figure 5C The pooling window used when pooling the 45° feature map is shown. Figure 5Dshows the pooling window used when pooling the 180° feature map. indicates that the pooling window applied when pooling the 0° feature map is evenly distributed over the entire feature map from the upper left to the lower right, as shown in Figure 5B As shown in FIG. 1 , each rectangular window size is m×m. The pooling operation is to apply the pooling function to each m×m pixel. Different from the common pooling operation, in the exemplary embodiment of the present disclosure, for feature maps that are not 0°, the pooling window is rotated, as shown in FIG. Figure 5C and Figure 5D As shown. The pooling window with a rotation angle is obtained by rotating the pooling window used when the 0° feature map is pooled by a corresponding angle. Figure 5C and Figure 5D The position of the pooling window is shown by Figure 5B The pooling windows shown are obtained by rotating 45° and 180° respectively. Since the convolutional neural network of the embodiment of the present disclosure only uses a convolutional kernel group consisting of two convolutional kernels for processing Figure 4 The image data shown, therefore, can only be selected Figure 5B and Figure 5D The pooling window division method shown is used to pool the feature map.

[0079] like Figure 3 As shown, after pooling in the first pooling layer 306, the number of channels of the feature map remains unchanged, but the size of the feature map becomes smaller. The pooled feature map output by the first pooling layer 306 is input to the second convolutional layer 308, and after being processed by the second convolutional layer 308, it is input to the second pooling layer 310. Except that the number of convolution kernel groups may be different, the second convolutional layer 308 is the same as the first convolutional layer 304, so its detailed description is omitted here. Similarly, the description of the second pooling layer 310 is omitted.

[0080] In an exemplary embodiment according to the present disclosure, the convolutional neural network also includes a feature map selection layer, which compares the feature maps of each convolution kernel that has been pooled by all pooling layers and originates from the same convolution kernel group, and selects the feature map with the maximum value as the feature map extracted by the convolution kernel group for the image data.

[0081] like Figure 3As shown, after being processed by all convolution layers and pooling layers, a 1×1 feature map is output, and the number of channels of the feature map is 8. The feature map selection layer 312 compares the feature maps output by the two convolution kernels constituting the convolution kernel group, and selects the feature map with the maximum value as the feature map of the image data extracted by the convolution kernel group, and then outputs it to the classification layer 314 for classification. In other words, a 1×1×8 feature vector is input to the feature map selection layer 312, and the feature map selection layer 312 divides the 8 1×1 feature maps into four groups based on the convolution kernel group, selects the feature map with the maximum value from each group, and then outputs a 1×1×4 feature vector to the classification layer 314 for classification.

[0082] In an exemplary embodiment according to the present disclosure, the convolution kernel group includes a first convolution kernel, the first convolution kernel is represented by F0, and other convolution kernels in the convolution kernel group are represented by Fi, and the other convolution kernels Fi are obtained by the following formula:

[0083] F i (c, p, q) = F0(c, p′, q′), where

[0084] Among them, i represents the index of each convolution kernel in the convolution kernel group, c represents the number of channels of the convolution kernel, p and q represent the height and width indexes of the i-th convolution kernel respectively, and θ i It represents the angle of rotation of the i-th convolution kernel from the first convolution kernel, and p' and q' represent the indexes of the height and width of the first convolution kernel respectively.

[0085] exist Figure 3 Taking the convolution kernel group shown in the figure as an example, the convolution kernel on the left in each convolution kernel group is F0, and the convolution kernel on the right is F1. According to the above formula, F0 can be rotated 180° to obtain F1.

[0086]

[0087] In an exemplary embodiment according to the present disclosure, the weight of the first convolution kernel changes as the convolutional neural network is trained. Figure 3 In the example, the left convolution kernel in each convolution kernel group is F0, and the weight of convolution kernel F0 can change with the training of the neural network, and has learning ability. The other convolution kernels in the convolution kernel group do not change with the training of the convolution neural network, and therefore do not have learning ability. Since there is a rotation correspondence between the convolution kernels in the convolution kernel group, the other convolution kernels can change according to the change of the weight of the convolution kernel F0.

[0088] In an exemplary embodiment of the present disclosure, the number of channels of each convolution kernel in the convolution layer for extracting features from the feature map after pooling of the previous pooling layer is 1 / n of the number of channels of the feature map after pooling of its input, where n represents the number of different rotation angles and is a natural number greater than 2. Since the neural network is a layered structure, the output of one convolution layer is also the input of another convolution layer. The number of channels of the convolution kernel is reduced from the second convolution layer, and the number of channels becomes 1 / n, which means that the number of parameters is reduced and less memory is required during operation. Figure 3 As shown, the number of channels of the feature map after pooling in the first pooling layer is 4, and the number of rotation angles of 0° and 180° is 2, so the number of channels of each convolution kernel in the second convolution layer 308 is 4 / 2=2.

[0089] In an exemplary embodiment according to the present disclosure, the center of each pooling window corresponding to the convolution kernels other than the first convolution kernel in the convolution kernel group is expressed as (h θi ,w θi ), then the center of the pooling window (h θi ,w θi ) is obtained by the following formula:

[0090]

[0091] Among them, h θo ,w θo represents the center of the pooling window corresponding to the first convolution kernel, θ i Denotes the self-center (h θo ,w θo )The angle of rotation.

[0092] like FIG. 5B to FIG. 5D As shown, Figure 5B The pooling window used when pooling the feature map at 0° is shown. Figure 5C The pooling window used when pooling the 45° feature map is shown. Figure 5D shows the pooling window used when pooling the 180° feature map. Figure 5B The center point coordinates of the pooling window shown, Figure 5C and Figure 5D The center point coordinates of the pooling window shown can be obtained by replacing θ i =45° and 180° are substituted into Formula 6 and calculated.

[0093] According to another aspect of the embodiments of the present disclosure, there is provided an image processing method based on a convolutional neural network, comprising: extracting features of the same features of input image data with different rotation angles by each convolution kernel of each convolution kernel group in each convolution layer of the convolutional neural network, wherein one convolution kernel of the convolution kernel group can obtain other convolution kernels of the convolution kernel group after rotation; and pooling the feature maps output by each convolution kernel using pooling windows respectively corresponding to each convolution kernel of the convolution kernel group by each pooling layer of the convolutional neural network to output a pooled feature map, wherein by rotating the pooling window around the center of the pooling window corresponding to a convolution kernel of the convolution kernel group, the pooling windows corresponding to other convolution kernels in the convolution kernel group can be obtained.

[0094] The following describes an image processing method based on a convolutional neural network according to an embodiment of the present disclosure with reference to the accompanying drawings. Figure 6 FIG. 4 is a flowchart of an image processing method based on a convolutional neural network according to an embodiment of the present disclosure. Figure 6 As shown, the method includes step S601 and step S603.

[0095] In step S601, each convolution kernel of each convolution kernel group in each convolution layer of the convolution neural network extracts the same features of the input image data with different rotation angles to obtain a feature map, wherein one convolution kernel of the convolution kernel group can be rotated to obtain other convolution kernels of the convolution kernel group. In step S603, each pooling layer of the convolution neural network uses the pooling windows corresponding to each convolution kernel of the convolution kernel group to pool the feature maps output by each convolution kernel to output the pooled feature map, wherein the pooling windows corresponding to other convolution kernels in the convolution kernel group can be obtained by rotating the pooling window around the center of the pooling window corresponding to one convolution kernel of the convolution kernel group. As reference has been made to Figure 3 The image data processing process of the convolutional neural network of the exemplary embodiment of the present disclosure has been described in detail, and therefore, it will not be described in detail here.

[0096] In an exemplary embodiment according to the present disclosure, the image processing method based on convolutional neural network also includes: a feature map selection layer of the convolutional neural network compares the feature maps of each convolution kernel that has been pooled by all pooling layers and originates from the same convolution kernel group, and selects the feature map with the maximum value as the feature map extracted by the convolution kernel group for the image data.

[0097] In an exemplary embodiment according to the present disclosure, the convolution kernel group includes a first convolution kernel, the first convolution kernel is represented by F0, and the other convolution kernels in the convolution kernel group are represented by F i Indicates that other convolution kernels F i Obtained by the following formula:

[0098] Fi (c, p, q) = F0(c, p′, q′), where

[0099] Among them, i represents the index of each convolution kernel in the convolution kernel group, c represents the number of channels of the convolution kernel, p and q represent the height and width indexes of the i-th convolution kernel respectively, and θ i It represents the angle of rotation of the i-th convolution kernel from the first convolution kernel, and p' and q' represent the indexes of the height and width of the first convolution kernel respectively.

[0100] In an exemplary embodiment according to the present disclosure, the weight of the first convolution kernel changes as the convolutional neural network is trained.

[0101] In an exemplary embodiment according to the present disclosure, the number of channels of each convolution kernel in the convolution layer that performs feature extraction on the feature map after pooling of the previous pooling layer is 1 / n of the number of channels of the feature map after pooling of its input, where n represents the number of different rotation angles and is a natural number greater than 2.

[0102] In an exemplary embodiment according to the present disclosure, the center of each pooling window corresponding to the convolution kernels other than the first convolution kernel in the convolution kernel group is expressed as (h θi ,w θi ), then the center of the pooling window (h θi ,w θi ) is obtained by the following formula:

[0103]

[0104] Among them, (h θo ,w θo ) represents the center of the pooling window corresponding to the first convolution kernel, θ i Denotes the self-center (h θo ,w θo )The angle of rotation.

[0105] According to yet another aspect of an embodiment of the present disclosure, an image processing device is provided. Figure 7 FIG. 2 is a diagram showing an image processing apparatus according to an embodiment of the present disclosure. Figure 7 As shown, the image processing device 70 includes: a memory 701 and a processor 702, the processor is connected to the memory 701 through a bus 703, etc., the memory 701 stores program instructions executed by the processor 702, and the processor 702 can execute a method as described in any of the above embodiments when executing the program instructions.

[0106] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein a computer program including program instructions is stored on the computer-readable storage medium, and the program instructions execute the above-mentioned methods when executed by a processor.

[0107] According to another aspect of an embodiment of the present disclosure, a processor is provided, wherein the processor is used to run a program, wherein the above methods are executed when the program is run.

[0108] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, wherein the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the above methods.

[0109] In the above embodiments of the present disclosure, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0110] In addition, each functional unit or module in each embodiment of the present disclosure may be integrated into one processing unit or module, or each unit or module may exist physically separately, or two or more units or modules may be integrated into one unit or module. The above-mentioned integrated units or modules may be implemented in the form of hardware or in the form of software functional units or modules.

[0111] Those skilled in the art will appreciate that all or part of the steps for implementing the above method embodiments may be accomplished by hardware (e.g., circuits) associated with program instructions, and the aforementioned program may be stored in a computer-readable storage medium, which, when executed, executes the steps of the above method embodiments; and the aforementioned storage medium includes various media that can store program codes, such as ROM, RAM, disks, or optical disks.

[0112] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0113] The present disclosure may also be configured as follows:

[0114] 1) A convolutional neural network, characterized in that the convolutional neural network comprises:

[0115] at least one convolution layer, each convolution layer includes at least one convolution kernel group, each convolution kernel group includes more than two convolution kernels, each convolution kernel of each convolution kernel group extracts the same features of the input image data with different rotation angles to obtain a feature map, wherein one convolution kernel of the convolution kernel group can obtain other convolution kernels of the convolution kernel group after rotation; and

[0116] At least one pooling layer, each pooling layer uses the pooling windows corresponding to the convolution kernels of the convolution kernel group to pool the feature maps output by each convolution kernel to output the pooled feature maps, wherein the pooling windows corresponding to other convolution kernels in the convolution kernel group can be obtained by rotating the pooling window around the center of the pooling window corresponding to a convolution kernel of the convolution kernel group.

[0117] 2) A convolutional neural network according to clause 1), characterized in that the convolutional neural network also includes a feature map selection layer, which compares the feature maps of each convolution kernel that has been pooled by all pooling layers and originates from the same convolution kernel group, and selects the feature map with the maximum value as the feature map extracted by the convolution kernel group for the image data.

[0118] 3) A convolutional neural network according to clause 1), characterized in that the convolution kernel group includes a first convolution kernel, the first convolution kernel is represented by F0, and the other convolution kernels in the convolution kernel group are represented by F i Indicates that other convolution kernels F i Obtained by the following formula:

[0119] F i (c, p, q) = F0(c, p′, q′), where

[0120] Among them, i represents the index of each convolution kernel in the convolution kernel group, c represents the number of channels of the convolution kernel, p and q represent the height and width indexes of the i-th convolution kernel respectively, and θ i It represents the angle of rotation of the i-th convolution kernel from the first convolution kernel, and p' and q' represent the indexes of the height and width of the first convolution kernel respectively.

[0121] 4) A convolutional neural network according to clause 3), characterized in that the weights of the first convolution kernel change as the convolutional neural network is trained.

[0122] 5) A convolutional neural network according to clause 4), characterized in that the number of channels of each convolution kernel in the convolutional layer that extracts features from the feature map after pooling of the previous pooling layer is 1 / n of the number of channels of the feature map after pooling of its input, where n represents the number of different rotation angles and is a natural number greater than 2.

[0123] 6) A convolutional neural network according to clause 3), characterized in that the center of each pooling window corresponding to the convolution kernels other than the first convolution kernel in the convolution kernel group is represented by (h θi ,w θi ), then the center of the pooling window (h θi ,w θi ) is obtained by the following formula:

[0124]

[0125] Among them, (h θo ,w θo ) represents the center of the pooling window corresponding to the first convolution kernel, θ i Denotes the self-center (h θo ,w θo )The angle of rotation.

[0126] 7) An image processing method based on convolutional neural network, characterized in that the method comprises:

[0127] Each convolution kernel of each convolution kernel group in each convolution kernel layer of the convolution neural network extracts the same features of the input image data with different rotation angles to obtain a feature map, wherein one convolution kernel of the convolution kernel group can obtain other convolution kernels of the convolution kernel group after rotation; and

[0128] Each pooling layer of the convolutional neural network uses the pooling windows corresponding to each convolution kernel of the convolution kernel group to pool the feature map output by each convolution kernel to output the pooled feature map, wherein the pooling window corresponding to a convolution kernel of the convolution kernel group is rotated around the center of the pooling window corresponding to one convolution kernel of the convolution kernel group to obtain the pooling windows corresponding to other convolution kernels in the convolution kernel group.

[0129] 8) The image processing method based on convolutional neural network according to clause 7) is characterized in that it also includes: the feature map selection layer of the convolutional neural network compares the feature maps of each convolution kernel that has been pooled by all pooling layers and comes from the same convolution kernel group, and selects the feature map with the maximum value as the feature map extracted by the convolution kernel group for the image data. 9) The image processing method based on convolutional neural network according to clause 7) is characterized in that

[0130] The convolution kernel group includes the first convolution kernel, which is represented by F0, and the other convolution kernels in the convolution kernel group are represented by F i Indicates that other convolution kernels F i Obtained by the following formula:

[0131] F i (c, p, q) = F O(c, p′, q′), where

[0132] Among them, i represents the index of each convolution kernel in the convolution kernel group, c represents the number of channels of the convolution kernel, p and q represent the height and width indexes of the i-th convolution kernel respectively, and θ i It represents the angle of rotation of the i-th convolution kernel from the first convolution kernel, and p' and q' represent the indexes of the height and width of the first convolution kernel respectively.

[0133] 10) The convolutional neural network-based image processing method according to clause 9), characterized in that the weight of the first convolution kernel changes as the convolutional neural network is trained.

[0134] 11) An image processing method based on a convolutional neural network according to clause 10), characterized in that the number of channels of each convolution kernel in the convolution layer that performs feature extraction on the feature map after pooling of the previous pooling layer is 1 / n of the number of channels of its input feature map after pooling, where n represents the number of different rotation angles and is a natural number greater than 2.

[0135] 12) The image processing method based on convolutional neural network according to clause 9), characterized in that the center of each pooling window corresponding to the convolution kernels other than the first convolution kernel in the convolution kernel group is represented by (h θi ,w θi ), then the center of the pooling window (h θi ,w θi ) is obtained by the following formula:

[0136]

[0137] Among them, (h θo ,w θo ) represents the center of the pooling window corresponding to the first convolution kernel, θ i Denotes the self-center (h θo ,w θo )The angle of rotation.

[0138] 13) An image processing device, characterized in that it includes a memory and a processor, the processor is connected to the memory, the memory stores program instructions executed by the processor, and the processor is able to execute the method according to any one of clauses 7) to 12) when executing the program instructions.

[0139] 14) A computer-readable storage medium, characterized in that a computer program including program instructions is stored on the computer-readable storage medium, and when the program instructions are executed by a processor or a computer, a method according to any one of clauses 7) to 12) is performed.

[0140] The above are only preferred embodiments of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present disclosure. These improvements and modifications should also be regarded as the protection scope of the present disclosure.

Claims

1. A system based on convolutional neural network, characterized in that: The convolutional neural network-based system comprises: At least one convolution layer, each of the convolution layers includes at least one convolution kernel group, each of the convolution kernel group includes more than two convolution kernels, each of the convolution kernel groups performs feature extraction on the same features of the input image data with different rotation angles to obtain a feature map, wherein one convolution kernel of the convolution kernel group can be rotated to obtain other convolution kernels of the convolution kernel group; at least one pooling layer, each of the pooling layers using pooling windows corresponding to the convolution kernels of the convolution kernel group to pool the feature maps output by the convolution kernels to output the pooled feature maps, wherein the pooling windows corresponding to the other convolution kernels in the convolution kernel group can be obtained by rotating the pooling window around the center of the pooling window corresponding to the one convolution kernel of the convolution kernel group; and A feature map selection layer compares the feature maps of the convolution kernels that have been pooled by all the pooling layers and that come from the same convolution kernel group, and selects the feature map with the maximum value as the feature map extracted by the convolution kernel group for the image data.

2. The system based on convolutional neural network according to claim 1, characterized in that: The convolution kernel group includes a first convolution kernel, the first convolution kernel is represented by F0, and other convolution kernels in the convolution kernel group are represented by F i Indicates that the other convolution kernel F i Obtained by the following formula: F i (c, p, q) = F0(c, p′, q′), where Where i represents the index of each convolution kernel in the convolution kernel group, c represents the number of channels of the convolution kernel, p and q represent the height and width indexes of the i-th convolution kernel respectively, and θ i represents the angle of rotation of the i-th convolution kernel from the first convolution kernel, and p' and q' represent the indexes of the height and width of the first convolution kernel respectively.

3. The system based on convolutional neural network according to claim 2, characterized in that The weight of the first convolution kernel changes as the convolutional neural network is trained.

4. The system based on convolutional neural network according to claim 3, characterized in that The number of channels of each convolution kernel in the convolution layer that performs feature extraction on the feature map after pooling of the previous pooling layer is 1 / n of the number of channels of the feature map after pooling of its input, where n represents the number of different rotation angles and is a natural number greater than 2.

5. The system based on convolutional neural network according to claim 2, characterized in that The center of each pooling window corresponding to the convolution kernels other than the first convolution kernel in the convolution kernel group is represented as (h θi ,w θi ), then the center of the pooling window (h θi ,w θi ) is obtained by the following formula: Among them, (h θo ,w θo ) represents the center of the pooling window corresponding to the first convolution kernel, θ i Denotes the self-center (h θo ,w θo )The angle of rotation.

6. An image processing method based on convolutional neural network, characterized in that: The method comprises: Each convolution kernel of each convolution kernel group in each convolution kernel layer of the convolution neural network performs feature extraction on the same features of the input image data with different rotation angles to obtain a feature map, wherein one convolution kernel of the convolution kernel group can obtain other convolution kernels of the convolution kernel group after rotation; The pooling layers of the convolutional neural network use the pooling windows corresponding to the convolution kernels of the convolution kernel group to pool the feature maps output by the convolution kernels to output the pooled feature maps, wherein the pooling windows corresponding to the other convolution kernels in the convolution kernel group can be obtained by rotating the pooling window around the center of the pooling window corresponding to the one convolution kernel of the convolution kernel group; and The feature map selection layer of the convolutional neural network compares the feature maps of each convolution kernel that has been pooled by all pooling layers and originates from the same convolution kernel group, and selects the feature map with the maximum value as the feature map extracted by the convolution kernel group for the image data.

7. The image processing method based on convolutional neural network according to claim 6, characterized in that: The convolution kernel group includes a first convolution kernel, the first convolution kernel is represented by F0, and other convolution kernels in the convolution kernel group are represented by F i Indicates that the other convolution kernel F i Obtained by the following formula: F i (c, p, q) = F0(c, p′, q′), where Where i represents the index of each convolution kernel in the convolution kernel group, c represents the number of channels of the convolution kernel, p and q represent the height and width indexes of the i-th convolution kernel respectively, and θ i represents the angle of rotation of the i-th convolution kernel from the first convolution kernel, and p' and q' represent the indexes of the height and width of the first convolution kernel respectively.

8. The image processing method based on convolutional neural network according to claim 7, characterized in that: The weight of the first convolution kernel changes as the convolutional neural network is trained.

9. The image processing method based on convolutional neural network according to claim 8, characterized in that: The number of channels of each convolution kernel in the convolution layer that performs feature extraction on the feature map after pooling of the previous pooling layer is 1 / n of the number of channels of the feature map after pooling of its input, where n represents the number of different rotation angles and is a natural number greater than 2.

10. The image processing method based on convolutional neural network according to claim 7, characterized in that: The center of each pooling window corresponding to the convolution kernels other than the first convolution kernel in the convolution kernel group is represented as (h θi ,w θi ), then the center of the pooling window (h θi ,w θi ) is obtained by the following formula: Among them, (h θo ,w θo ) represents the center of the pooling window corresponding to the first convolution kernel, θ i Denotes the self-center (h θo ,w θo )The angle of rotation.

11. An image processing device, characterized in that: The method comprises a memory and a processor, wherein the processor is connected to the memory, the memory stores program instructions executed by the processor, and the processor can execute the method according to any one of claims 6 to 10 when executing the program instructions.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program including program instructions, which, when executed by a processor or a computer, perform the method according to any one of claims 6 to 10.

Images