Image processing method and image processing device based on convolutional neural network

Abstract

The present application discloses an image processing method and an image processing device based on a convolutional neural network in the field of artificial intelligence. The method may include: receiving an input image; preprocessing the input image to obtain preprocessed image information; performing a convolution operation on the image information using a convolutional neural network, and outputting a convolution operation result; wherein, when the nth convolutional layer is used to perform a convolution operation on the image information, after obtaining r auxiliary convolution kernels from the m main convolution kernels of the convolutional layer, the m main convolution kernels and r auxiliary convolution kernels of the convolutional layer are used to perform convolution operations on the image information respectively, and the obtained multiple feature maps are spliced ​​to obtain the convolution operation result of the convolutional layer. In an embodiment of the present application, the image processing device may store the main convolution kernels of each convolutional layer, and generate auxiliary convolution kernels using the main convolution kernels of each convolutional layer before performing a convolution operation on each convolutional layer, thereby reducing the memory occupied by the convolutional neural network.

Description

[0001] This application claims priority to U.S. non-provisional patent application No. 16359346, filed on March 20, 2019, entitled “Image processing method and image processing device based on convolutional neural network” and international patent application No. PCT / CN2019 / 104460, filed on September 4, 2019, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer vision, and in particular to an image processing method and an image processing device. Background Art

[0003] Computer vision is an integral part of various intelligent systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is a discipline about how to use cameras and computers to obtain the data and information we need about the objects being photographed. Figuratively speaking, it is to equip computers with eyes (cameras) and brains (algorithms) to replace human eyes in identifying, tracking, and measuring targets, so that computers can perceive the environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multidimensional data. In general, computer vision uses various imaging systems to replace visual organs to obtain input information, and then computers replace the brain to complete the processing and interpretation of these input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.

[0004] Thanks to the rapid development of graphics processing units (GPUs) and various neural network processors, convolutional neural networks (CNNs) have become an indispensable technology in the field of computer vision. Today, CNNs have a lot of application requirements on mobile devices (e.g., mobile phones, cameras, smart homes), such as image recognition, target detection, etc. At present, the number of parameters of the convolution kernel of each convolution layer in the commonly used CNNs can often reach tens of thousands or even hundreds of thousands, and the number of parameters (convolution kernel parameters) involved in the convolution operation in the entire CNN can add up to tens of millions. If each parameter in CNN is stored, it will take up hundreds of megabytes of memory space. However, the memory resources of mobile devices are very limited, so how to reduce the memory space occupied by CNN is an urgent problem to be solved. Summary of the invention

[0005] The embodiments of the present application provide a convolutional neural network-based image processing method and an image processing device, which can effectively reduce the memory occupied by the convolutional neural network.

[0006] In a first aspect, an embodiment of the present application provides an image processing method based on a convolutional neural network, the method comprising: receiving an input image; preprocessing the input image to obtain preprocessed image information; performing a convolution operation on the image information using a convolutional neural network, the convolutional neural network comprising N convolutional layers, the convolution operation comprising: using M convolution kernels in the nth convolutional layer to perform a convolution operation on the image information, wherein the M convolution kernels include m main convolution kernels and r auxiliary convolution kernels, the r auxiliary convolution kernels are obtained from the m main convolution kernels, one main convolution kernel corresponds to one or more auxiliary convolution kernels, and the number of non-zero elements contained in each of the r auxiliary convolution kernels is less than the number of non-zero elements contained in the corresponding main convolution kernel; wherein N and M are both integers greater than 0; n is an integer greater than 0 and less than or equal to N; m is an integer greater than 0 and less than M, and r is an integer greater than 0 and less than M; outputting the convolution operation result, the convolution operation result comprising M feature maps.

[0007] The executor of the image processing method is an image processing device, which may be a mobile phone, a tablet computer, a laptop computer, a game console, a server, a wearable device, etc. Optionally, an auxiliary convolution kernel is obtained from a part of the elements in a main convolution kernel, that is, the elements in the auxiliary convolution kernel are all derived from the main convolution kernel. The number of elements contained in the auxiliary convolution kernel obtained in this way is less than the number of elements contained in the main convolution kernel corresponding to the auxiliary convolution kernel. Optionally, an auxiliary convolution kernel is obtained by setting a part of the elements in a main convolution kernel to 0. The number of non-zero elements contained in the auxiliary convolution kernel obtained in this way is less than the number of non-zero elements contained in the main convolution kernel corresponding to the auxiliary convolution kernel.

[0008] The image processing device can store the main convolution kernel of each convolution layer in the convolution neural network. Before using each convolution layer to perform a convolution operation, multiple auxiliary convolution kernels corresponding to the convolution layer are generated according to the main convolution kernel of the convolution layer, and the main convolution kernel and the generated multiple auxiliary convolution kernels are used to perform a convolution operation on the image information input to the convolution layer. For example, when a convolution layer in the convolution neural network needs to use 6 convolution kernels to perform a convolution operation, the image processing device can only store 1 main convolution kernel of the convolution layer. Before the convolution layer performs a convolution operation, the image processing device generates 6 auxiliary convolution kernels according to the main convolution kernel, and uses the 6 auxiliary convolution kernels to perform a convolution operation on the input image information.

[0009] In an embodiment of the present application, the image processing device may first store the main convolution kernel of each convolution layer, and use the main convolution kernel of each convolution layer to generate the auxiliary convolution kernel required for each convolution layer before using each convolution layer to perform a convolution operation, thereby reducing the memory occupied by the convolutional neural network.

[0010] In an optional implementation, any auxiliary convolution kernel among the r auxiliary convolution kernels includes P channels, the main convolution kernel corresponding to any auxiliary convolution kernel includes P channels, and the two-dimensional matrix of any auxiliary convolution kernel on the p-th channel includes a sub-matrix of the two-dimensional matrix of the main convolution kernel corresponding to any auxiliary convolution kernel on the p-th channel; wherein P is an integer greater than 0, and p is an integer greater than 0 and not greater than P.

[0011] A matrix is ​​a set of complex or real numbers arranged in a rectangular array. For example, a table of (H×W) numbers arranged in H rows and W columns is called a two-dimensional matrix of H rows and W columns, or (H×W) matrix for short, and these (H×W) numbers are called elements of the matrix. A new matrix consisting of elements at the intersection of certain rows and certain columns selected from a matrix is ​​called a submatrix of the original matrix. It can be understood that the elements in the two-dimensional matrix of any auxiliary convolution kernel on each channel are all derived from the main convolution kernel corresponding to the auxiliary convolution kernel. In other words, the elements included in any auxiliary convolution kernel are all from the main convolution kernel corresponding to the auxiliary convolution kernel.

[0012] In this implementation, the image processing device obtains the two-dimensional matrix of the auxiliary convolution kernel on the p-th channel from the sub-matrix of the two-dimensional matrix of the main convolution kernel on the p-th channel. The auxiliary convolution kernel can be quickly obtained from the main convolution kernel, and the implementation is simple.

[0013] In an optional implementation, the elements included in the two-dimensional matrix of any auxiliary convolution kernel on the p-th channel are the elements of the two-dimensional matrix of the main convolution kernel on the p-th channel corresponding to the any auxiliary convolution kernel except the outermost K rows and the outermost K columns; K is an integer not less than 1.

[0014] In this implementation, the image processing device uses the matrix corresponding to the elements of the main convolution kernel except the outermost K rows and the outermost K columns as the auxiliary convolution kernel, and can quickly generate an auxiliary convolution kernel that is smaller than the main convolution kernel.

[0015] In an optional implementation, any auxiliary convolution kernel includes a two-dimensional matrix on P consecutive channels among Q channels included in the main convolution kernel corresponding to the auxiliary convolution kernel, where P is less than Q.

[0016] Optionally, any auxiliary convolution kernel includes a two-dimensional matrix on any P channels among the Q channels included in the main convolution kernel corresponding to the any auxiliary convolution kernel. The image processing device can obtain the any auxiliary convolution kernel from the two-dimensional matrix on any P channels among the Q channels of the main convolution kernel corresponding to the any auxiliary convolution kernel.

[0017] In this implementation, the image processing device can quickly generate an auxiliary convolution kernel that is smaller than the main convolution kernel.

[0018] In an optional implementation, any auxiliary convolution kernel is obtained by setting at least one element in the main convolution kernel corresponding to the auxiliary convolution kernel to a target value.

[0019] The target value can be 0 or other values, which is not limited in this application.

[0020] In this implementation, by setting at least one element in the main convolution kernel to a target value, an auxiliary convolution kernel different from the main convolution kernel can be quickly generated, which is simple to implement.

[0021] In an optional implementation, any auxiliary convolution kernel is obtained by setting the element at the target position in the two-dimensional matrix on each channel of the main convolution kernel corresponding to the auxiliary convolution kernel to the target value.

[0022] The elements at the target position in a two-dimensional matrix may be elements other than the outermost K rows and the outermost K columns in the two-dimensional matrix, where K is an integer not less than 1.

[0023] In this implementation, an auxiliary convolution kernel that is significantly different from the main convolution kernel can be quickly obtained, and the implementation is simple.

[0024] In an optional implementation, any auxiliary convolution kernel is obtained by setting all elements in a two-dimensional matrix on one or more channels of a main convolution kernel corresponding to the auxiliary convolution kernel to target values.

[0025] In this implementation, by setting the elements in the two-dimensional matrix on any channel of the main convolution kernel to the target value, an auxiliary convolution kernel including some elements of the main convolution kernel can be quickly generated.

[0026] In an optional implementation, any auxiliary convolution kernel is obtained by setting the two-dimensional matrix on at least one channel of the main convolution kernel corresponding to the auxiliary convolution kernel to a zero matrix. The zero matrix is ​​a matrix whose elements are all zero.

[0027] In this implementation, the two-dimensional matrix on at least one channel of the main convolution kernel is set to a zero matrix to obtain an auxiliary convolution kernel. Using the auxiliary convolution kernel to perform convolution operations can greatly reduce the number of floating-point calculations required to be performed.

[0028] In an optional implementation, any auxiliary convolution kernel is a matrix obtained by calculating the product of the elements at the corresponding positions in the main convolution kernel corresponding to the any auxiliary convolution kernel and the elements at the corresponding positions in the sampling matrix; the sampling matrix and the main convolution kernel corresponding to the any auxiliary convolution kernel are homogeneous matrices and the elements at each position in the sampling matrix correspond one-to-one to the elements at each position in the main convolution kernel, the elements at the target position of the sampling matrix are all 1 and the elements except the target position are all 0.

[0029] The elements at the target position of the sampling matrix may be elements other than the outermost K rows and the outermost K columns in each two-dimensional matrix of the sampling matrix in the channel dimension, where K is an integer not less than 1.

[0030] In this implementation, the auxiliary convolution kernel is obtained by calculating the product of the main convolution kernel and the sampling matrix, which is simple to implement.

[0031] In an optional implementation, any auxiliary convolution kernel is a matrix obtained by calculating the product of the elements at corresponding positions in the main convolution kernel corresponding to any auxiliary convolution kernel and the elements at corresponding positions in the sampling matrix; the sampling matrix includes (QP) zero matrices and P two-dimensional matrices whose elements are all 1 in the channel dimension, the sampling matrix and the main convolution kernel corresponding to any auxiliary convolution kernel are isotype matrices and the elements at each position in the sampling matrix correspond one-to-one to the elements at each position in the main convolution kernel; Q is an integer greater than 0, and P is an integer greater than 0 and not greater than Q.

[0032] In this implementation, the auxiliary convolution kernel is obtained by calculating the product of the main convolution kernel and the sampling matrix, which is simple to implement.

[0033] In an optional implementation, the image processing method further includes: obtaining a new auxiliary convolution kernel by using any auxiliary convolution kernel among the r auxiliary convolution kernels as a main convolution kernel.

[0034] In this implementation, the generated auxiliary convolution kernel is used as a new main convolution kernel to generate another auxiliary convolution kernel, which not only solves the problem of the limited number of auxiliary convolution kernels generated according to one main convolution kernel, but also makes fuller use of the main convolution kernel and further reduces the number of parameters in CNN.

[0035] In a second aspect, an embodiment of the present application provides a training method, the method comprising: performing image processing on a training sample using a convolutional neural network to obtain a prediction processing result; the convolutional neural network comprises N convolutional layers, and the image processing comprises: performing a convolution operation on the training sample using M convolutional kernels in the nth convolutional layer, wherein the M convolutional kernels include m main convolutional kernels and r auxiliary convolutional kernels, the r auxiliary convolutional kernels are obtained from the m main convolutional kernels, one main convolutional kernel corresponds to one or more auxiliary convolutional kernels, and each of the r auxiliary convolutional kernels has a corresponding auxiliary convolutional kernel. The number of non-zero elements contained in the product kernel is less than the number of non-zero elements contained in the corresponding main convolution kernel; wherein N and M are both integers greater than 0; n is an integer greater than 0 and less than or equal to N; m is an integer greater than 0 and less than M, and r is an integer greater than 0 and less than M; according to the predicted processing result and the standard result, the loss corresponding to the training sample is determined; the standard result is the processing result expected to be obtained by processing the training sample using the convolutional neural network; using the loss corresponding to the training sample, the parameters of the convolutional neural network are updated through an optimization algorithm.

[0036] The image processing may be image classification, object detection, image reconstruction, etc. Each training sample corresponds to a standard result. For example, the image processing device performs image classification on the training sample, and the standard result corresponding to the training sample is the labeled data (ground truth) of the training sample, that is, the real classification result.

[0037] In an embodiment of the present application, during the process of training a convolutional neural network, the image processing device uses the main convolution kernels of each convolutional layer of the convolutional neural network to generate the auxiliary convolution kernels required for each convolutional layer, thereby reducing the memory occupied by the convolutional neural network.

[0038] In an optional implementation, s auxiliary convolution kernels among the r auxiliary convolution kernels are obtained from any main convolution kernel among the m main convolution kernels; using the loss corresponding to the training sample, updating the parameters of the convolution neural network through an optimization algorithm includes: calculating the gradients of the s auxiliary convolution kernels obtained from any main convolution kernel among the m main convolution kernels according to the loss corresponding to the training sample; using the back propagation algorithm to take the sum of the gradients of the s auxiliary convolution kernels as the gradient of any main convolution kernel, and updating the parameters of any main convolution kernel.

[0039] In this implementation, the sum of the gradients of multiple auxiliary convolution kernels is used as the gradient of the main convolution kernel to update the main convolution kernel, so that the auxiliary convolution kernel generated according to the updated main convolution kernel can meet the needs of each convolution layer.

[0040] In an optional implementation, any auxiliary convolution kernel among the r auxiliary convolution kernels includes P channels, the main convolution kernel corresponding to any auxiliary convolution kernel includes P channels, and the two-dimensional matrix of any auxiliary convolution kernel on the p-th channel includes a sub-matrix of the two-dimensional matrix of the main convolution kernel corresponding to any auxiliary convolution kernel on the p-th channel; wherein P is an integer greater than 0, and p is an integer greater than 0 and not greater than P.

[0041] In an optional implementation, the elements included in the two-dimensional matrix of any auxiliary convolution kernel on the p-th channel are the elements of the two-dimensional matrix of the main convolution kernel on the p-th channel corresponding to the any auxiliary convolution kernel except the outermost K rows and the outermost K columns; K is an integer not less than 1.

[0042] In an optional implementation, any auxiliary convolution kernel includes a two-dimensional matrix on P consecutive channels among Q channels included in the main convolution kernel corresponding to the auxiliary convolution kernel, where P is less than Q.

[0043] In an optional implementation, any auxiliary convolution kernel is obtained by setting at least one element in the main convolution kernel corresponding to the auxiliary convolution kernel to a target value.

[0044] In an optional implementation, any auxiliary convolution kernel is obtained by setting the element at the target position in the two-dimensional matrix on each channel of the main convolution kernel corresponding to the auxiliary convolution kernel to the target value.

[0045] The elements at the target position in a two-dimensional matrix may be elements other than the outermost K rows and the outermost K columns in the two-dimensional matrix, where K is an integer not less than 1.

[0046] In an optional implementation, any auxiliary convolution kernel is obtained by setting all elements in a two-dimensional matrix on one or more channels of a main convolution kernel corresponding to the auxiliary convolution kernel to target values.

[0047] In an optional implementation, any auxiliary convolution kernel is obtained by setting the two-dimensional matrix on at least one channel of the main convolution kernel corresponding to the auxiliary convolution kernel to a zero matrix.

[0048] In an optional implementation, any auxiliary convolution kernel is a matrix obtained by calculating the product of the elements at the corresponding positions in the main convolution kernel corresponding to the any auxiliary convolution kernel and the elements at the corresponding positions in the sampling matrix; the sampling matrix and the main convolution kernel corresponding to the any auxiliary convolution kernel are homogeneous matrices and the elements at each position in the sampling matrix correspond one-to-one to the elements at each position in the main convolution kernel, the elements at the target position of the sampling matrix are all 1 and the elements at positions other than the target position are all 0.

[0049] In an optional implementation, any auxiliary convolution kernel is a matrix obtained by calculating the product of the elements at corresponding positions in the main convolution kernel corresponding to any auxiliary convolution kernel and the elements at corresponding positions in the sampling matrix; the sampling matrix includes (QP) zero matrices and P two-dimensional matrices whose elements are all 1 in the channel dimension, the sampling matrix and the main convolution kernel corresponding to any auxiliary convolution kernel are isotype matrices and the elements at each position in the sampling matrix correspond one-to-one to the elements at each position in the main convolution kernel; Q is an integer greater than 0, and P is an integer greater than 0 and not greater than Q.

[0050] In an optional implementation, the training method further includes: obtaining a new auxiliary convolution kernel by using any auxiliary convolution kernel among the r auxiliary convolution kernels as the main convolution kernel.

[0051] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, the memory being used to store a code; the processor reads the code stored in the memory to perform the following operations: receiving an input image; preprocessing the input image to obtain preprocessed image information; performing a convolution operation on the image information using a convolutional neural network, the convolutional neural network comprising N convolutional layers, the convolution operation comprising: performing a convolution operation on the image information using M convolutional kernels in the nth convolutional layer, wherein the M convolutional kernels include The invention comprises m main convolution kernels and r auxiliary convolution kernels, the r auxiliary convolution kernels are obtained by the m main convolution kernels, one main convolution kernel corresponds to one or more auxiliary convolution kernels, and the number of non-zero elements contained in each of the r auxiliary convolution kernels is less than the number of non-zero elements contained in the corresponding main convolution kernel; wherein N and M are both integers greater than 0; n is an integer greater than 0 and less than or equal to N; m is an integer greater than 0 and less than M, and r is an integer greater than 0 and less than M; the convolution operation result is output, and the convolution operation result includes M feature maps.

[0052] In an embodiment of the present application, the image processing device may first store the main convolution kernel of each convolution layer, and use the main convolution kernel of each convolution layer to generate the convolution kernel required for each convolution layer before using each convolution layer to perform a convolution operation, which can greatly reduce the memory occupied by the convolutional neural network.

[0053] In an optional implementation, any auxiliary convolution kernel among the r auxiliary convolution kernels includes P channels, the main convolution kernel corresponding to any auxiliary convolution kernel includes P channels, and the two-dimensional matrix of any auxiliary convolution kernel on the p-th channel includes a sub-matrix of the two-dimensional matrix of the main convolution kernel corresponding to any auxiliary convolution kernel on the p-th channel; wherein P is an integer greater than 0, and p is an integer greater than 0 and not greater than P.

[0054] In an optional implementation, the elements included in the two-dimensional matrix of any auxiliary convolution kernel on the p-th channel are the elements of the two-dimensional matrix of the main convolution kernel on the p-th channel corresponding to the any auxiliary convolution kernel except the outermost K rows and the outermost K columns; K is an integer not less than 1.

[0055] In an optional implementation, any auxiliary convolution kernel includes a two-dimensional matrix on P consecutive channels among Q channels included in the main convolution kernel corresponding to the auxiliary convolution kernel, where P is less than Q.

[0056] In an optional implementation, the processor is further configured to set at least one element in the main convolution kernel corresponding to any auxiliary convolution kernel to a target value to obtain the any auxiliary convolution kernel.

[0057] In an optional implementation, the processor is further used to set the element at the target position in the two-dimensional matrix on each channel of the main convolution kernel corresponding to any auxiliary convolution kernel to the target value to obtain any auxiliary convolution kernel.

[0058] In an optional implementation, the processor is further configured to set the elements in the two-dimensional matrix on one or more channels of the main convolution kernel corresponding to any auxiliary convolution kernel to target values ​​to obtain any auxiliary convolution kernel.

[0059] In an optional implementation, the processor is further used to set the two-dimensional matrix on at least one channel of the main convolution kernel corresponding to any auxiliary convolution kernel to a zero matrix to obtain any auxiliary convolution kernel.

[0060] In an optional implementation, the processor is also used to train a convolutional neural network; the training process includes: initializing m main convolution kernels in the nth convolution layer of the convolutional neural network; using the convolutional neural network to perform image processing on the training sample to obtain an image processing result; calculating the loss corresponding to the training sample based on the image processing result and the standard result, and the standard result is the processing result expected to be obtained by processing the training sample using the convolutional neural network; according to the loss corresponding to the training sample, respectively calculating the gradient of s auxiliary convolution kernels obtained by any main convolution kernel among the m main convolution kernels; using the back propagation algorithm to take the sum of the gradients of the s auxiliary convolution kernels as the gradient of any main convolution kernel, and updating the parameters in any main convolution kernel.

[0061] In an optional implementation, the processor is further used to calculate the product of the elements at corresponding positions in the main convolution kernel corresponding to any auxiliary convolution kernel and the elements at corresponding positions in the sampling matrix to obtain the any auxiliary convolution kernel; the sampling matrix includes (QP) zero matrices and P two-dimensional matrices whose elements are all 1 in the channel dimension, the sampling matrix and the main convolution kernel corresponding to any auxiliary convolution kernel are isotype matrices and the elements at each position in the sampling matrix correspond one-to-one to the elements at each position in the main convolution kernel; Q is an integer greater than 0, and P is an integer greater than 0 and not greater than Q.

[0062] In an optional implementation, the processor is further configured to obtain a new auxiliary convolution kernel by using any auxiliary convolution kernel among the r auxiliary convolution kernels as the main convolution kernel.

[0063] In an optional implementation, s auxiliary convolution kernels among the r auxiliary convolution kernels are obtained from any main convolution kernel among the m main convolution kernels; the processor is also used to calculate the gradients of the s auxiliary convolution kernels respectively to obtain s gradients; s is an integer not less than 1; and the back propagation algorithm is used to take the sum of the s gradients as the gradient of any main convolution kernel to update any main convolution kernel.

[0064] In a fourth aspect, an embodiment of the present application provides another electronic device comprising: a processor and a memory, the memory being used to store code; the processor reads the code stored in the memory to perform the following operations: performing image processing on a training sample using a convolutional neural network to obtain a prediction processing result; the convolutional neural network comprises N convolutional layers, and the image processing comprises: performing a convolution operation on the image information using M convolutional kernels in the nth convolutional layer, wherein the M convolutional kernels comprise m main convolutional kernels and r auxiliary convolutional kernels, the r auxiliary convolutional kernels are obtained from the m main convolutional kernels, and one main convolutional kernel corresponds to one or more A plurality of auxiliary convolution kernels, and the number of non-zero elements contained in each of the r auxiliary convolution kernels is less than the number of non-zero elements contained in the corresponding main convolution kernel; wherein N and M are both integers greater than 0; n is an integer greater than 0 and less than or equal to N; m is an integer greater than 0 and less than M, and r is an integer greater than 0 and less than M; according to the predicted processing result and the standard result, the loss corresponding to the training sample is determined; the standard result is the processing result expected to be obtained by processing the training sample using the convolution neural network; using the loss corresponding to the training sample, the parameters of the convolution neural network are updated through an optimization algorithm.

[0065] In an embodiment of the present application, during the process of training a convolutional neural network, the electronic device uses the main convolution kernels of each convolution layer of the convolutional neural network to generate the auxiliary convolution kernels required for each convolution layer, which can greatly reduce the memory occupied by the convolutional neural network.

[0066] In an optional implementation, s of the r auxiliary convolution kernels are obtained from any main convolution kernel of the m main convolution kernels;

[0067] The processor is specifically used to calculate the gradients of the s auxiliary convolution kernels respectively to obtain s gradients; s is an integer not less than 1; a back propagation algorithm is used to take the sum of the s gradients as the gradient of any main convolution kernel to update any main convolution kernel.

[0068] In this implementation, the main convolution kernel is updated using the sum of the gradients of the loss function relative to each auxiliary convolution kernel, so that the auxiliary convolution kernel generated according to the updated main convolution kernel can meet the needs of each convolution layer.

[0069] In an optional implementation, any auxiliary convolution kernel among the r auxiliary convolution kernels includes P channels, the main convolution kernel corresponding to any auxiliary convolution kernel includes P channels, and the two-dimensional matrix of any auxiliary convolution kernel on the p-th channel includes a sub-matrix of the two-dimensional matrix of the main convolution kernel corresponding to any auxiliary convolution kernel on the p-th channel; wherein P is an integer greater than 0, and p is an integer greater than 0 and not greater than P.

[0070] In an optional implementation, the elements included in the two-dimensional matrix of any auxiliary convolution kernel on the p-th channel are the elements of the two-dimensional matrix of the main convolution kernel on the p-th channel corresponding to the any auxiliary convolution kernel except the outermost K rows and the outermost K columns; K is an integer not less than 1.

[0071] In an optional implementation, any auxiliary convolution kernel includes a two-dimensional matrix on P consecutive channels among Q channels included in the main convolution kernel corresponding to the auxiliary convolution kernel, where P is less than Q.

[0072] In an optional implementation, the processor is further configured to set at least one element in the main convolution kernel corresponding to any auxiliary convolution kernel to a target value to obtain the any auxiliary convolution kernel.

[0073] In an optional implementation, the processor is further used to set the element at the target position in the two-dimensional matrix on each channel of the main convolution kernel corresponding to any auxiliary convolution kernel to the target value to obtain any auxiliary convolution kernel.

[0074] In an optional implementation, the processor is further configured to set the elements in the two-dimensional matrix on one or more channels of the main convolution kernel corresponding to any auxiliary convolution kernel to target values ​​to obtain any auxiliary convolution kernel.

[0075] In an optional implementation, the processor is further used to set the two-dimensional matrix on at least one channel of the main convolution kernel corresponding to any auxiliary convolution kernel to a zero matrix to obtain any auxiliary convolution kernel.

[0076] In an optional implementation, the processor is also used to calculate the product of the elements at the corresponding positions in the main convolution kernel corresponding to any auxiliary convolution kernel and the elements at the corresponding positions in the sampling matrix to obtain the any auxiliary convolution kernel; the sampling matrix and the main convolution kernel corresponding to any auxiliary convolution kernel are isotype matrices, and the elements at the target positions of the sampling matrix are all 1 and the elements at positions other than the target positions are all 0.

[0077] In an optional implementation, the processor is further used to calculate the product of the elements at corresponding positions in the main convolution kernel corresponding to any auxiliary convolution kernel and the elements at corresponding positions in the sampling matrix to obtain the any auxiliary convolution kernel; the sampling matrix includes (QP) zero matrices and P two-dimensional matrices whose elements are all 1 in the channel dimension, the sampling matrix and the main convolution kernel corresponding to any auxiliary convolution kernel are isotype matrices and the elements at each position in the sampling matrix correspond one-to-one to the elements at each position in the main convolution kernel; Q is an integer greater than 0, and P is an integer greater than 0 and not greater than Q.

[0078] In an optional implementation, the processor is further configured to obtain a new auxiliary convolution kernel by using any auxiliary convolution kernel among the r auxiliary convolution kernels as the main convolution kernel.

[0079] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes program instructions, and when the program instructions are executed by a processor, the processor executes the method of the first aspect or the second aspect.

[0080] In the sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions, which, when executed by a processor, enable the processor to execute the methods of the first aspect and the second aspect.

[0081] In the seventh aspect, an embodiment of the present application provides a chip, which includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface to execute the methods of the first aspect and the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 It is a structural diagram of the system architecture provided by the embodiment of the present application;

[0083] Figure 2 A schematic diagram of the structure of a convolutional neural network provided in an embodiment of the present application;

[0084] Figure 3 It is a schematic diagram of a chip hardware structure provided in an embodiment of the present application;

[0085] Figure 4A flow chart of a method for training a convolutional neural network provided in an embodiment of the present application;

[0086] Figure 5 A flow chart of an image processing method provided in an embodiment of the present application;

[0087] Fig. 6A A schematic diagram for comparing a main convolution kernel and an auxiliary convolution kernel provided in an embodiment of the present application;

[0088] Figure 6B A schematic diagram for comparing another main convolution kernel and an auxiliary convolution kernel provided in an embodiment of the present application;

[0089] Figure 7 A schematic diagram for comparing another main convolution kernel and an auxiliary convolution kernel provided in an embodiment of the present application;

[0090] Figure 8 A schematic diagram for comparing another main convolution kernel and an auxiliary convolution kernel provided in an embodiment of the present application;

[0091] Fig. 9 A schematic diagram of the structure of AlexNet provided in an embodiment of the present application;

[0092] Fig.10 A schematic diagram of the hardware structure of a convolutional neural network training device provided in an embodiment of the present application;

[0093] Fig.11 A schematic diagram of the hardware structure of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0094] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0095] The image processing method based on convolutional neural network provided in the embodiment of the present application can be applied to scenes such as image classification, target detection, image super-resolution reconstruction, and image restoration. Specifically, the image processing method based on convolutional neural network in the embodiment of the present application can be applied to image classification scenes and image super-resolution reconstruction scenes. The following briefly introduces the image classification scenes and image super-resolution reconstruction scenes respectively.

[0096] Image classification scenario 1: The image processing device uses a convolutional neural network to classify images collected by a data collection device (such as a camera) to obtain an image classification result, and can also perform corresponding subsequent operations based on the image classification result. The image processing device can be a terminal, such as a mobile terminal, a tablet computer, a laptop computer, an augmented reality device (AR), a virtual reality device (VR), a vehicle terminal, etc., and can also be a server. For example, the image processing device can be a fingerprint attendance machine, which collects the fingerprint image entered by the user through a fingerprint collection device, classifies the fingerprint image to determine the identity of the user, and generates a corresponding attendance record. For another example, the image processing device is a door access control system, which uses a camera to collect the user's facial image, recognizes the facial image to determine the user's identity, and after determining that the user is an authorized user, opens a channel for the user to pass through.

[0097] Image classification scenario 2: The image processing device uses a convolutional neural network to classify images from client devices to obtain image classification results, and performs corresponding subsequent operations based on the image classification results. The image processing device can be a device or server with data processing functions such as a cloud server, a network server, an application server, and a management server. The client device can be a mobile phone, a tablet computer, etc. For example, the image processing device is a cloud server. When a user uses a mobile phone to query social security through a software application, the user uses the mobile phone to collect his or her facial image and sends it to the image processing device; the image processing device recognizes the facial image to determine the identity of the user, and after determining the identity of the user, sends the user's social security information to the mobile phone.

[0098] Image super-resolution reconstruction scenario: The image processing device uses a convolutional neural network to perform super-resolution reconstruction on a lower-resolution input image to obtain an image with a higher resolution. The input image can be a remote sensing image, a medical image (for example, an image taken by a medical device such as an X-ray machine, an MRI scanner, etc.), a video image, etc. The image processing device can be a device or server with data processing functions such as a cloud server, a network server, an application server, and a management server. For example, the image processing device is a computer, and the image processing device performs image super-resolution reconstruction on a remote sensing image to obtain an image with a better resolution.

[0099] In the above scenario, before the image processing device uses each convolutional layer of the convolutional neural network to perform convolution operations on image information, the auxiliary convolution kernel is obtained from the main convolution kernel of each convolutional layer. Each convolutional layer uses the main convolution kernel and the auxiliary convolution kernel to perform convolution operations on the input image information, so that the memory space occupied by the convolutional neural network is greatly reduced.

[0100] The following describes the method provided by this application from the perspectives of model training and model application:

[0101] The training method of the convolutional neural network provided in the embodiment of the present application relates to the processing of computer vision, and can be specifically applied to image processing methods such as data training, machine learning, and deep learning, and the training data (such as the input image in the present application) is symbolized and formalized for intelligent information modeling, extraction, preprocessing, training, etc., and finally a trained convolutional neural network is obtained; and the image processing method based on the convolutional neural network provided in the embodiment of the present application can use the above-mentioned trained convolutional neural network to input the input data (such as the input image in the present application) into the trained convolutional neural network to obtain output data (such as the image processing result or processed image in the present application). It should be noted that the training method of the convolutional neural network provided in the embodiment of the present application and the image processing method based on the convolutional neural network are inventions based on the same concept, and can also be understood as two parts in a system, or two stages of an overall process: such as the model training stage and the model application stage.

[0102] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0103] (1) A convolutional neural network is a deep neural network with a convolutional structure. A convolutional neural network contains a feature extractor consisting of a convolutional layer and a subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve an input image or a convolutional feature plane (feature map). The convolutional feature plane can also be called a feature map. A convolutional layer refers to a neuron layer in a convolutional neural network that performs convolution processing on the input signal. In a convolutional layer of a convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neurons. The neural units in the same feature plane share weights, and the weight matrix corresponding to the shared weights is the convolution kernel. Shared weights can be understood as the way to extract image information is independent of position. The implicit principle is that the statistical information of a part of the image is the same as that of other parts, which means that the image information learned in one part can also be used in another part. Therefore, the same learned image information can be used for all positions on the image. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0104] The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the number of connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0105] (2) Loss Function

[0106] In the process of training a convolutional neural network, because we hope that the output of the convolutional neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the convolutional neural network). For example, if the predicted value of the network is high, adjust the weight vector to make its predicted value lower, and keep adjusting until the convolutional neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the convolutional neural network becomes a process of minimizing this loss as much as possible.

[0107] (3) Back propagation algorithm

[0108] Convolutional neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the convolutional neural network during the training process, so that the error loss between the predicted value output by the convolutional neural network and the desired target value becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the initial parameters in the convolutional neural network, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the convolutional neural network, such as the weight matrix, that is, the convolution kernel of the convolution layer.

[0109] The following introduces the system architecture provided by the embodiments of the present application.

[0110] See attached Figure 1, an embodiment of the present invention provides a system architecture 100. As shown in the system architecture 100, the data acquisition device 160 is used to collect training data. In the embodiment of the present application, the training data includes: one or more test images (training samples) and the actual results corresponding to the one or more test images, that is, the ideal results expected to be obtained by processing the one or more test images using a convolutional neural network; and the training data can be stored in a database 130, and the training device 120 can be trained based on the training data maintained in the database 130 to obtain a target model / rule 101 (101 is the model obtained through the training phase introduced above, which can be a convolutional neural network for implementing image processing operations such as image classification and image super-resolution reconstruction). One test image corresponds to one real result, that is, ground truth. The following will describe in more detail how the training device 120 obtains the target model / rule 101 based on the training data with the first embodiment. The target model / rule 101 can be used to implement the image processing method based on the convolutional neural network provided in the embodiment of the present application, that is, the input image, such as the low-resolution image to be super-resolved, the image to be image classified, etc., is input into the target model / rule 101 through the image information obtained after the relevant pre-processing, and the image processing result can be obtained. The target model / rule 101 in the embodiment of the present application can specifically be a convolutional neural network obtained by training. In the embodiment provided in the present application, the convolutional neural network is obtained by training the initialized convolutional neural network. It should be noted that in actual applications, the training data maintained in the database 130 may not all come from the collection of the data acquisition device 160, but may also be received from other devices. It should also be noted that the training device 120 may not necessarily train the target model / rule 101 based entirely on the training data maintained by the database 130, and it may also be possible to obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiment of the present application.

[0111] The target model / rule 101 trained by the training device 120 can be applied to different systems or devices, such as Figure 1 The execution device 110 shown in the figure may be a terminal, such as a mobile terminal, a tablet computer, a laptop computer, an AR device, a VR device, a vehicle terminal, etc., or a server, etc. Figure 1 In the embodiment, the execution device 110 is configured with an I / O interface 112 for data interaction with an external device. The user can input data to the I / O interface 112 through the client device 140. The input data in the embodiment of the present application may include: an input image, which can be an image collected by the execution device 110 through the data acquisition device 160, or an image in the database 130, or an image from the client device 140.

[0112] The preprocessing module 113 is used to perform preprocessing according to the input data (such as the input image) received by the I / O interface 112. In an embodiment of the present application, the preprocessing module 113 can be used to implement one or more operations of image filtering, image preprocessing enhancement, image preprocessing smoothing, image preprocessing restoration, etc., and can also be used to implement other preprocessing operations, which are not limited in this application.

[0113] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, code, etc. in the data storage system 150 to implement the corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0114] Finally, the I / O interface 112 returns the processing result, such as the image processing result obtained above, to the client device 140 for providing to the user.

[0115] It is worth noting that the training device 120 can obtain corresponding target models / rules 101 based on different training data for different goals or different tasks. The corresponding target models / rules 101 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0116] exist Figure 1 In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 130.

[0117] It is worth noting that Figure 1 This is only a schematic diagram of a system architecture provided by an embodiment of the present invention. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, Figure 1In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110. In the present application, the target model / rule 101 trained based on the training data may be a convolutional neural network for processing image processing tasks.

[0118] At present, a trained CNN usually contains a large number of redundant parameters. In order to reduce the storage resources occupied by CNN in the device, the following two methods are currently mainly used to reduce the number of parameters in CNN: Method 1, compress the convolution kernel in the trained CNN; Method 2, replace the convolution kernel with a larger size with a smaller size. The number of parameters in CNN refers to the number of parameters in CNN. Method 1 is to re-represent the parameters of the convolution kernel in the trained CNN by pruning, quantization, dictionary learning, etc. Although this can reduce the storage overhead of CNN, some special calculation methods need to be introduced in the process of convolution operation of the CNN compressed by method 1, such as sparse calculation, matrix decomposition, quantization, etc., which is difficult to deploy on a general computing platform (such as a mobile terminal). The disadvantage of using method 2 is that the convolution kernel in CNN is not fully utilized, and the convolution kernel contains a lot of redundant information. These two methods only consider how to remove and trim the parameters of the convolution kernel, and do not consider how to make more full use of the convolution kernel. This application mainly reduces the storage resources occupied by CNN from the perspective of maximizing the use of the convolution kernel in CNN. Table 1 below illustrates the problem that the parameters in CNN are not fully utilized and how to reasonably optimize the parameters in CNN to reduce the number of CNN parameters and improve or ensure the calculation accuracy of CNN.

[0119] Table 1

[0120]

[0121]

[0122] Table 1 shows the properties of some deep convolutional neural network (CNN) models that have achieved good results on the ImageNet dataset in recent years, including their parameter count, memory usage, floating-point calculation times, and accuracy. The calculation times in Table 1 are the number of floating-point calculations performed by each network in Table 1 to classify the images in the ImageNet dataset. Floating-point operations are real number operations. CNN is a deep neural network. The ImageNet dataset can be understood as a picture library for image / vision training, which is used to evaluate the performance of different algorithms and networks. AlexNet, VGGNet-16, GooleNet, ResNet-50, and ResNeXt-50 in Table 1 are different deep convolutional neural networks (CNNs), which are used to classify images in the ImageNet dataset. The Top1 error rate refers to the probability that the predicted classification with the highest probability is misclassified (top 1 prediction error rate); the Top5 error rate refers to the probability that there is no correct classification among the five predicted classifications with the highest probability (top 5 prediction error rate). In Table 1, the Top1 error rate and Top5 error rate of a network are negatively correlated with the classification accuracy of the network. As can be seen from Table 1, the classification accuracy of ResNet-50 and ResNeXt-50 with fewer parameters is higher than that of AlexNet and VGGNet-16 with more parameters. It can be understood that many parameters in CNN are redundant, and the more parameters in CNN, the higher the calculation accuracy of CNN. This motivates us to develop more efficient convolution calculation methods, hoping to complete more effective calculations with fewer parameters, while reducing the memory and computing overhead occupied by CNN, and improving the performance of image processing using CNN as much as possible. The present application provides an image processing solution with low storage overhead based on convolutional neural network.

[0123] like Figure 1 As shown, the target model / rule 101 is obtained by training according to the training device 120. The target model / rule 101 can be a convolutional neural network in the embodiment of the present application. Specifically, each convolutional layer of the convolutional neural network provided in the embodiment of the present application only includes a main convolutional kernel. Before using each convolutional layer to perform a convolution operation, an auxiliary convolutional kernel is obtained from the main convolutional kernel of each convolutional layer. Each convolutional layer uses the main convolutional kernel and the auxiliary convolutional kernel of the convolutional layer to perform a convolution operation on the input image information.

[0124] As mentioned in the previous basic concept introduction, convolutional neural network is a deep neural network with convolution structure, which is a deep learning architecture. Deep learning architecture refers to multiple levels of learning at different abstract levels through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, in which each neuron can respond to the image input into the feed-forward artificial neural network.

[0125] like Figure 2 As shown, the convolutional neural network (CNN) 200 may include an input layer 210 , a convolutional layer / pooling layer 220 (wherein the pooling layer is optional), and a neural network layer 230 .

[0126] Convolutional layer / pooling layer 220:

[0127] Convolutional Layer:

[0128] like Figure 2 The convolution layer / pooling layer 220 shown may include layers 221-226, for example: in one implementation, layer 221 is a convolution layer, layer 222 is a pooling layer, layer 223 is a convolution layer, layer 224 is a pooling layer, layer 225 is a convolution layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolution layers, layer 223 is a pooling layer, layer 224 and 225 are convolution layers, and layer 226 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0129] The following will take the convolution layer 221 as an example to introduce the internal working principle of a convolution layer.

[0130] The convolution layer 221 may include a plurality of convolution operators, which are also called convolution kernels. The convolution operator is equivalent to a filter for extracting specific information from an input image matrix in image processing. The convolution operator may be essentially a weight matrix, which is usually predefined. In the process of performing a convolution operation on an image, depending on the value of the stride, the weight matrix is ​​usually processed one pixel after another or two pixels after two pixels in the horizontal direction on the input image to complete the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing a convolution operation, the weight matrix will extend to the entire depth of the input image. The depth dimension is also the channel dimension, corresponding to the number of channels. Therefore, using a single weight matrix for convolution will produce a convolution output of a single depth dimension, but in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same size (row × column), that is, multiple isotype matrices, are applied. The output of each weight matrix is ​​stacked up to form the depth dimension of the convolution image, where the dimension can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unnecessary noise in the image, etc. The multiple weight matrices have the same size (rows × columns), and the size of the feature maps extracted by the multiple weight matrices of the same size is also the same, and then the extracted multiple feature maps of the same size are merged to form the output of the convolution operation. In various embodiments of the present application, before using each convolution layer to perform a convolution operation, an auxiliary convolution kernel is obtained from the main convolution kernel of each convolution layer, and each convolution layer uses the main convolution kernel and the auxiliary convolution kernel of the convolution layer to perform a convolution operation on the input image information.

[0131] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolution neural network 200 can make correct predictions. The weight matrix is ​​also the convolution kernel, and the weight values ​​in the weight matrix are also the parameters in the convolution kernel. The parameters in the convolution kernel can also be called the elements in the convolution kernel.

[0132] When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (for example, 221) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by the later convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0133] Pooling layer:

[0134] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. Figure 2 The layers 221-226 marked as 220 in the figure may be a convolution layer followed by a pooling layer, or multiple convolution layers followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and / or a maximum pooling operator for sampling the input image to obtain an image of smaller size. The average pooling operator may calculate the pixel values ​​in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator may take the pixel with the largest value within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0135] Neural Network Layer 230:

[0136] After being processed by the convolution layer / pooling layer 220, the convolution neural network 200 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 200 needs to use the neural network layer 230 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 230 may include multiple hidden layers (such as Figure 2 231, 232 to 23n) and the output layer 240 shown, the parameters contained in the multi-layer hidden layer can be pre-trained according to relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0137] After the multiple hidden layers in the neural network layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (such as Figure 2 The propagation from 210 to 240 is forward propagation), and the reverse propagation (such as Figure 2 The propagation from 240 to 210 is called back propagation) and then starts to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer (i.e., the above-mentioned image processing result) and the ideal result.

[0138] It should be noted that if Figure 2 The convolutional neural network 200 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.

[0139] The following introduces a chip hardware structure provided by an embodiment of the present application.

[0140] Figure 3 A chip hardware structure provided in an embodiment of the present invention includes a neural network processor 30. The chip can be arranged as follows: Figure 1 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 1 The training device 120 shown in FIG. 1 is used to complete the training work of the training device 120 and output the target model / rule 101. Figure 2 The algorithms of each layer in the convolutional neural network shown in the figure can be used in Figure 3 is implemented in the chip shown.

[0141] The neural network processor 30 can be a neural network processor (neural-network processing unit, NPU), a tensor processing unit (tensor processing unit, TPU), or a graphics processing unit (graphics processing unit, GPU) and any other processor suitable for large-scale XOR operation processing. Take NPU as an example: NPU can be mounted on a central processing unit (central processing unit, CPU), that is, a host CPU (HostCPU) as a coprocessor, and the host CPU assigns tasks to it, such as image processing tasks. The core part of NPU is the operation circuit 303, which is controlled by the controller 304 to extract the matrix data in the memory (301 and 302) and perform multiplication and addition operations.

[0142] In some implementations, the operation circuit 303 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 303 is a two-dimensional systolic array. The operation circuit 303 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 303 is a general-purpose matrix processor.

[0143] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit 303 takes the weight value of the matrix B from the weight memory 302 and caches it on each PE in the operation circuit 303. The operation circuit 303 takes the input data of the matrix A from the input memory 301, performs matrix operations based on the input data of the matrix A and the weight value of the matrix B, and stores the partial result or the final result of the matrix in the accumulator 308. The input data can be an input image, and the weight matrix is ​​the convolution kernel. The weight data can also be called a weight matrix.

[0144] The unified memory 306 is used to store input data and output data. The weight matrix is ​​transferred to the weight memory 302 through the direct memory access controller (DMAC) 305. The input data is also transferred to the unified memory 306 through the DMAC. The output data is the image processing result.

[0145] The bus interface unit (BIU) 310 is used for the interaction between the DMAC and the instruction fetch buffer 309. The bus interface unit 310 is also used for the instruction fetch buffer 309 to obtain instructions from the external memory. The bus interface unit 310 is also used for the storage unit access controller 305 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0146] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 306, or to transfer weight data to the weight memory 302, or to transfer input data to the input memory 301.

[0147] The vector calculation unit 307 includes a plurality of operation processing units, and further processes the output of the operation circuit 303 when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. The vector calculation unit 307 is mainly used for the calculation of the non-convolutional layer or the fully connected (FC) layer in the neural network, and can specifically process the calculation of pooling, normalization, etc. For example, the vector calculation unit 307 can apply a nonlinear function to the output of the operation circuit 303, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 307 generates a normalized value, a merged value, or both.

[0148] In some implementations, the vector calculation unit 307 stores the processed vectors in the unified memory 306. In some implementations, the vectors processed by the vector calculation unit 307 can be used as activation inputs of the operation circuit 303, for example, for use in subsequent layers in a neural network, such as Figure 2 As shown, if the current processing layer is hidden layer 1 (231), the vector processed by the vector calculation unit 307 can also be used in the calculation of hidden layer 2 (232).

[0149] The instruction fetch memory 309 connected to the controller 304 is used to store instructions used by the controller 304 .

[0150] The unified memory 306, the input memory 301, the weight memory 302 and the instruction fetch memory 309 are all on-chip memories. The external memory may be independent of the NPU hardware architecture.

[0151] in, Figure 2 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 303 or the vector calculation unit 307.

[0152] Embodiment 1:

[0153] Figure 4 A convolutional neural network training method 400 provided in Embodiment 1 of the present invention may include:

[0154] S401: The training device preprocesses the received input image to obtain preprocessed image information.

[0155] S402: The training device initializes the convolutional neural network.

[0156] Initializing the convolutional neural network includes initializing the main convolution kernels of each convolutional layer of the convolutional neural network and the parameters of other layers (such as pooling layers, neural network layers, and fully connected layers). The training device can use any initialization method, such as using Gaussian distribution random number sampling, uniform distribution random number sampling, etc. to initialize the convolutional neural network.

[0157] S403, the training device obtains a spatial auxiliary convolution kernel from the main convolution kernel.

[0158] The training device can obtain the spatial auxiliary convolution kernel from the main convolution kernel. The main convolution kernel can be the main convolution kernel of any convolution layer of the convolutional neural network. In practical applications, the training device can obtain the spatial auxiliary convolution kernel of each convolution layer from the main convolution kernel of each convolution layer. How to obtain the spatial auxiliary convolution kernel from the main convolution kernel will be described in detail later, and will not be described in detail here.

[0159] S404, the training device obtains a channel auxiliary convolution kernel from the main convolution kernel.

[0160] The training device can obtain the channel auxiliary convolution kernel from the main convolution kernel. The main convolution kernel can be the main convolution kernel of any convolution layer of the convolutional neural network. In practical applications, the training device can obtain the channel auxiliary convolution kernel of each convolution layer from the main convolution kernel of each convolution kernel. How to obtain the channel auxiliary convolution kernel from the main convolution kernel will be described in detail later, and will not be described in detail here.

[0161] The main convolution kernel in S404 may be the same as or different from the main convolution kernel in S403. The training device may execute only one of S403 and S404, in which case the main convolution kernel corresponding to the auxiliary convolution kernels in S403 and S404 may be the main convolution kernel in S402; the training device may also execute S403 and S404 independently, in which case the main convolution kernel corresponding to the auxiliary convolution kernels in S403 and S404 may both be the main convolution kernel in S402; the training device may also execute S403 first and then S404, and the spatial auxiliary convolution kernel obtained in S403 is the main convolution kernel corresponding to the channel auxiliary convolution kernel obtained in S404; or the training device may execute S404 first and then S403, and the channel auxiliary convolution kernel obtained in S404 is the main convolution kernel corresponding to the spatial auxiliary convolution kernel obtained in S403.

[0162] S405: The training device uses the auxiliary convolution kernel to perform a convolution operation on the image information to obtain multiple feature maps.

[0163] The training device may at least use the auxiliary convolution kernel to perform a convolution operation on the image information, that is, the training device may use the auxiliary convolution kernel and the main convolution kernel in step S402 to perform a convolution operation on the image information, or may only use the auxiliary convolution kernel to perform a convolution operation on the image information. Among them, the auxiliary convolution kernel may include one or more of the spatial auxiliary convolution kernel and the channel auxiliary convolution kernel. Optionally, the training device may only use the obtained spatial auxiliary convolution kernel to perform a convolution operation on the image information to obtain multiple feature maps. Optionally, the training device may only use the obtained channel auxiliary convolution kernel to perform a convolution operation on the image information to obtain multiple feature maps. Optionally, the training device may use each spatial auxiliary convolution kernel to perform a convolution operation on the image information to obtain a part of the feature maps in the multiple feature maps, and may use each channel auxiliary convolution kernel to perform a convolution operation on the image information to obtain another part of the feature maps in the multiple feature maps. In practical applications, the training device may use each convolution layer of the convolutional neural network in a certain order to perform a convolution operation on the image information inputted by each layer. For example, the output of the 3rd convolutional layer of the convolutional neural network used by the training device is the input of the 4th convolutional layer. The training device uses the convolution kernels of the 3rd convolutional layer (spatial auxiliary convolution kernel and channel auxiliary convolution kernel) to perform a convolution operation on the image information input by the 3rd layer, and concatenates the obtained feature maps and inputs them into the 4th convolutional layer for convolution operation.

[0164] S406: The training device concatenates the obtained multiple feature maps in the channel dimension as the output feature map of the convolution layer.

[0165] The channel dimension is also the depth dimension mentioned above. For example, a convolutional layer of a convolutional neural network includes 4 convolution kernels. The output feature map of the convolutional layer to be output includes 24 channels. The training device uses the 4 convolution kernels to perform convolution operations on the image information input to the convolutional layer to obtain 4 feature maps. Each feature map includes 6 channels. These 4 feature maps are sequentially spliced ​​together to obtain the output feature map of the convolutional layer. The 1st to 6th channels, the 7th to 12th channels, the 13th to 18th channels, and the 19th to 24th channels of the output feature map correspond to these 4 feature maps in sequence. In other words, the feature map obtained using each convolution kernel is used as part of the output feature map of the convolutional layer.

[0166] S407: The training device determines whether the convolutional neural network has converged.

[0167] If so, execute S409; otherwise, execute S408. The training device determines whether the convolutional neural network has converged by determining whether the number of times the main convolution kernel is updated reaches the iteration threshold; or it may determine whether the loss value of the convolutional neural network is lower than the loss threshold. The loss value of the convolutional neural network is the error between the image processing result output by the convolutional neural network and the ideal result, calculated by the training device using the loss function of the convolutional neural network. The loss function of the convolutional neural network is different for different training tasks of the training device. The iteration threshold may be the number of iterations preset by the training device, such as 10,000 times, 20,000 times, etc. The loss threshold may be preset by the training device. If the difference between the image processing result output by the convolutional neural network and the true result is less than the loss threshold, the training is terminated.

[0168] S408. The training device calculates the gradient of each auxiliary convolution kernel, takes the sum of the gradients of each auxiliary convolution kernel as the gradient of the main convolution kernel, and updates the parameters of the main convolution kernel.

[0169] Optionally, the training device calculates the gradient of each spatial auxiliary convolution kernel, takes the sum of the gradients of each spatial auxiliary convolution kernel as the gradient of the main convolution kernel, and updates the parameters of the main convolution kernel. Optionally, the training device calculates the gradient of each channel auxiliary convolution kernel, takes the sum of the gradients of each channel auxiliary convolution kernel as the gradient of the main convolution kernel, and updates the parameters of the main convolution kernel.

[0170] For a main convolution kernel, the sum of the gradients of the auxiliary convolution kernels obtained from the main convolution kernel can be used as the gradient of the main convolution kernel, where the auxiliary convolution kernel can include one or more of a spatial auxiliary convolution kernel and a channel auxiliary convolution kernel.

[0171] The following takes any convolutional layer as an example to introduce how to update the parameters of the main convolution kernel.

[0172] The convolution operation of the convolution kernel on the input image information can be realized through matrix multiplication: First, the input feature map x of a certain convolution layer (the image information input to this convolution layer) can be divided into z blocks, where z = H × W, and the size of each block is (d1 × d2 × P); the input feature map includes P channels, each channel corresponds to a (H × W) two-dimensional matrix, H and W are both integers greater than 0; the main convolution kernel f includes P channels, each channel corresponds to a (d1 × d2) two-dimensional matrix, d1, d2 and P are all integers greater than 0. Secondly, each block is vectorized to obtain the vectorized input feature map. Figure X ,X=[vec(x1),vec(x2),...,vec(x z )]∈R d1d2p×z Similarly, the output feature map y and its corresponding main convolution kernel f are also vectorized as Y = vec(y)∈R z×1and F = vec(f)∈R d1d2p×1 Then, the convolution operation can be written as follows:

[0173] Y=X T F+b;

[0174] Among them, X T represents the transpose of X, b is the bias term of the main convolution kernel f, and the gradient of the output feature map is expressed as L is the loss function of the entire convolutional neural network. The gradient of X and the gradient of F It can be calculated by back propagation algorithm:

[0175]

[0176]

[0177] For the auxiliary convolution kernel, the matrix form of the output feature map is:

[0178] Y=[X T F1+b1,X T F2+b2,...,X T F s +b s ];

[0179] Among them, F1 to F s are the s auxiliary convolution kernels obtained from the main convolution kernel f, b1 to b s F1 to F s The corresponding bias terms are, Y = [Y1, Y2, ..., Y s ]∈R z×s is the output feature map obtained by each auxiliary convolution kernel, Y1 to Y s are the feature maps obtained by each auxiliary convolution kernel. The gradient of the input feature map is:

[0180]

[0181] The gradient of each auxiliary convolution kernel is expressed as follows:

[0182]

[0183] Therefore, the gradient of the main convolution kernel is the sum of the gradients of the auxiliary convolution kernels:

[0184]

[0185] After calculating the gradient of the main convolution kernel, it is updated as follows:

[0186]

[0187] Where η is the learning rate. As an important hyperparameter in supervised learning and deep learning, the learning rate determines whether the loss function can converge to a local minimum and when it converges to the minimum. The learning rate is an important hyperparameter for deep learning. It controls the speed of adjusting the weights of the neural network based on the loss gradient. The smaller the learning rate, the slower the loss gradient decreases and the longer the convergence time. In practical applications, different learning rates can be set when the training device trains the convolutional neural network to perform different processing operations.

[0188] S409: End the training.

[0189] Optionally, the training device repeatedly executes S402 to S408 until a set number of iterations is reached.

[0190] The method 400 may be specifically performed as follows: Figure 1 The training device 120 shown in FIG. 4 is executed, and the input image in the method 400 can be as follows: Figure 1 The training data maintained in the database 130 shown. Optionally, S401 of the method 400 may be executed in the training device 120, or may be pre-executed by other functional modules before the training device 120, that is, the training data received or acquired from the database 130 is pre-processed, such as S401, to obtain the pre-processed image information as the input of the training device 120, and the training device 120 executes S402 to S409.

[0191] Optionally, the method 400 may be executed by a CPU, or may be processed jointly by a CPU and a GPU, or may use other processors suitable for neural network calculations without using a GPU, and this application does not impose any limitation thereto.

[0192] In an embodiment of the present application, an auxiliary convolution kernel is obtained from the main convolution kernel, and the gradient of the main convolution kernel is calculated by the gradient of the auxiliary convolution kernel to update the main convolution kernel; the main convolution kernel used to generate each auxiliary convolution kernel can be trained to obtain a convolutional neural network with a small number of parameters.

[0193] The following describes a scheme for image processing using a CNN containing a small number of convolution kernels (main convolution kernels). In this scheme, the CNN includes N convolution layers, and any of the N convolution layers uses m main convolution kernels and r auxiliary convolution kernels obtained from the m main convolution kernels to perform convolution operations on the image information input thereto. N, m, and r are all integers greater than 0. In this scheme, the image processing device may first store the main convolution kernels of each convolution layer of the CNN, and before any convolution layer performs a convolution operation on the input image information, multiple auxiliary convolution kernels are obtained from the main convolution kernel of any convolution layer, and any convolution layer then uses the main convolution kernel and the auxiliary convolution kernel to perform convolution operations on the input image information. It can be understood that the auxiliary convolution kernels are obtained from the main convolution kernel, rather than being pre-stored by the image processing device. The image processing device may obtain multiple auxiliary convolution kernels from one main convolution kernel, and the parameters in these auxiliary convolution kernels may all be obtained by sampling a portion of the elements in the main convolution kernel. In the present application, the image processing device only needs to store the main convolution kernel of each convolution layer of CNN. Before using each convolution layer to perform convolution operation, the auxiliary convolution kernel is obtained from the main convolution kernel of each convolution layer. This can greatly reduce the number of parameters of the convolution kernel in CNN, thereby reducing the memory space occupied by CNN.

[0194] Embodiment 2:

[0195] Figure 5 A neural network-based image processing method 500 is provided in the second embodiment of the present invention. The method may include:

[0196] S501: An image processing apparatus receives an input image.

[0197] The image processing device is Figure 1 The image processing device receiving the input image may be that the image processing device obtains the image (input image) collected by the data collection device (such as a camera); it may also be that the image is received from the client device through the I / O interface 112; or it may be that the image is stored in the database 130.

[0198] S502: The image processing device preprocesses the input image to obtain preprocessed image information.

[0199] The main purpose of the image processing device preprocessing the above input image is to eliminate irrelevant information in the above input image, restore useful real information, enhance the detectability of relevant information and simplify data to the maximum extent, so as to improve the reliability and accuracy of CNN's image processing operations such as feature extraction, image segmentation, image matching and image recognition on the above input image. The image processing device preprocesses the above input image by at least one of the following: image filtering, image preprocessing enhancement, image preprocessing smoothing and image preprocessing restoration. The image processing device can also perform other image preprocessing operations on the above input image, which are not limited in this application. Image filtering mainly includes adjusting the image size and denoising and smoothing the noise in the scaled image. Image preprocessing enhancement is to selectively strengthen and suppress the information in the image to improve the visual effect of the image, or to convert the image into a form more suitable for machine processing to facilitate data extraction or recognition. Image preprocessing smoothing is to eliminate random noise in the image. Image preprocessing restoration is to correct the image degradation caused by various reasons so that the reconstructed or estimated image is as close as possible to the ideal non-degraded image.

[0200] S503: The image processing device performs a convolution operation on the above image information using a convolutional neural network.

[0201] The above-mentioned convolutional neural network includes N convolutional layers, and the above-mentioned convolution operation includes: using the M convolution kernels in the nth convolutional layer to perform convolution operations on the above-mentioned image information. The above-mentioned M convolution kernels include m main convolution kernels and r auxiliary convolution kernels. The above-mentioned r auxiliary convolution kernels are obtained from the above-mentioned m main convolution kernels. One main convolution kernel corresponds to multiple auxiliary convolution kernels, and the number of non-zero elements contained in each of the above-mentioned r auxiliary convolution kernels is less than the number of non-zero elements contained in the corresponding main convolution kernel. Among them, N and M are both integers greater than 0; n is an integer greater than 0 and less than or equal to N; m is an integer greater than 0 and less than M, and r is an integer greater than 0 and less than M.

[0202] Any of the r auxiliary convolution kernels includes P channels, the main convolution kernel corresponding to any of the auxiliary convolution kernels includes Q channels, and the two-dimensional matrix of any of the auxiliary convolution kernels on the pth channel is a submatrix of the two-dimensional matrix of the main convolution kernel corresponding to any of the auxiliary convolution kernels on the qth channel; wherein P and Q are both integers greater than 0, p is an integer greater than 0 and not greater than P, and q is an integer greater than 0 and not greater than Q. The image processing device can obtain any of the auxiliary convolution kernels from the main convolution kernel corresponding to any of the auxiliary convolution kernels.

[0203] In an optional implementation, any of the above-mentioned auxiliary convolution kernels is obtained by the image processing device extracting some parameters (elements) in the main convolution kernel corresponding to any of the above-mentioned auxiliary convolution kernels. In the present application, the elements in the convolution kernel and the parameters in the convolution kernel are the same concept. In this implementation, each auxiliary convolution kernel is obtained by sampling some elements in the main convolution kernel. That is to say, the parameters (elements) of each auxiliary convolution kernel are included in the parameters of the main convolution kernel, and no new parameters are generated. For example, a main convolution kernel is a (11×11×3) three-dimensional matrix, that is, a three-dimensional tensor, and a (9×9×3) three-dimensional matrix included in the main convolution kernel is an auxiliary convolution kernel obtained by the main convolution kernel. For another example, a main convolution kernel is a (5×5×96) three-dimensional matrix, that is, the main convolution kernel includes 96 channels, each channel corresponds to a (5×5) two-dimensional matrix, and the main convolution kernel includes a (5×5×93) three-dimensional matrix, which is an auxiliary convolution kernel obtained by the main convolution kernel.

[0204] In an optional implementation, any of the above-mentioned auxiliary convolution kernels is a matrix obtained by the image processing device setting some elements in the main convolution kernel corresponding to any of the above-mentioned auxiliary convolution kernels to target values. The above-mentioned target value can be 0 or other values, which is not limited in this application. In this implementation, the auxiliary convolution kernel and the main convolution kernel are matrices of the same type, and different auxiliary convolution kernels can be obtained by setting the elements at different positions of a main convolution kernel to target values. For example, a main convolution kernel is a three-dimensional matrix (11×11×3), and the outermost 2 rows and the outermost 2 columns of each two-dimensional matrix of the main convolution kernel in the channel dimension are set to 0 to obtain an auxiliary convolution kernel. For another example, a main convolution kernel includes 96 channels, and each channel corresponds to a two-dimensional matrix, that is, the main convolution kernel includes 96 two-dimensional matrices, and any 3 two-dimensional matrices of the 96 two-dimensional matrices are set to zero matrices to obtain an auxiliary convolution kernel.

[0205] S504: The image processing apparatus outputs the convolution operation result.

[0206] The above-mentioned convolution operation result includes M feature maps. The image processing device uses the M convolution kernels in the above-mentioned n-th convolution layer to perform convolution operations on the above-mentioned image information respectively to obtain M feature maps; the obtained M feature maps are spliced ​​together to obtain the convolution operation result (the output feature map of the convolution layer). For example, a certain convolution layer of the convolutional neural network includes 4 convolution kernels, and the image processing device uses these 4 convolution kernels to perform convolution operations on the image information input to the convolution layer respectively to obtain 4 feature maps. If each feature map includes 6 channels, these 4 feature maps are spliced ​​together in sequence to obtain the output feature map of the convolution layer. The 1st channel to the 6th channel, the 7th channel to the 12th channel, the 13th channel to the 18th channel, and the 19th channel to the 24th channel of the output feature map correspond to these 4 feature maps in sequence. That is to say, the image processing device uses the feature map obtained by each convolution kernel of any convolution layer as a part of the output feature map of any convolution layer. S503 and S504 are operations performed by the image processing device during image processing such as image classification and target detection on the above image information. For example, when the image processing device uses a convolutional neural network to classify the image information, the image processing device needs to use the convolution layer, pooling layer, and neural network layer of the convolutional neural network to classify the image information to obtain an image classification result.

[0207] The method 500 may be specifically performed as follows: Figure 1 The execution device 110 shown in FIG. 1 is executed, and the input image in S501 in the method 500 may be as follows: Figure 1 The input data provided by the client device 140 shown in the figure, the preprocessing module 113 in the execution device 110 can be used to perform the preprocessing of the input image described in S502 in the method 500, and the calculation module 111 in the execution device 110 can be used to execute S503 and S504.

[0208] Optionally, the method 500 may be processed by a CPU, or by a CPU and a GPU together, or may use other processors suitable for neural network calculations without using a GPU, and this application does not impose any limitation thereto.

[0209] It can be understood that the first embodiment is the training phase of the convolutional neural network (such as Figure 1 The specific training is performed by the convolutional neural network provided by the first embodiment; and the second embodiment can be understood as the application stage of the convolutional neural network (such as Figure 1 The execution device 110 shown in the figure executes the phase), which can be specifically embodied by using the convolutional neural network trained by the first embodiment and obtaining the image processing result according to the image input by the user.

[0210] Before the image processing device uses the auxiliary convolution kernels in any convolution layer of the convolution neural network to perform a convolution operation on the input image information, it is necessary to obtain these auxiliary convolution kernels from the main convolution kernel of any convolution layer. The following introduces the implementation method of obtaining the auxiliary convolution kernel from any main convolution kernel in any convolution layer of the convolution neural network.

[0211] Method 1: The image processing device obtains a spatial auxiliary convolution kernel from a main convolution kernel.

[0212] The two-dimensional matrix of the spatial auxiliary convolution kernel on the pth channel includes a submatrix of the two-dimensional matrix of the main convolution kernel corresponding to the spatial auxiliary convolution kernel on the qth channel. The spatial auxiliary convolution kernel and the main convolution kernel corresponding to the spatial auxiliary convolution kernel both include P channels, and p, q, and P are all integers greater than 0. The two-dimensional matrix of the spatial auxiliary convolution kernel on any channel includes a submatrix of the two-dimensional matrix of the main convolution kernel on one channel, and the two-dimensional matrices of the spatial auxiliary convolution kernel on any two channels are different. The following introduces two methods for obtaining the spatial auxiliary convolution kernel from the main convolution kernel.

[0213] Method 1: The image processing device obtains the sub-matrices corresponding to the target positions in each two-dimensional matrix of the main convolution kernel on the channel dimension as the two-dimensional matrices of the spatial auxiliary convolution kernel on the channel dimension.

[0214] Each main convolution kernel includes one or more channels, each channel has a two-dimensional matrix, and the two-dimensional matrix on each channel is the two-dimensional matrix of the main convolution kernel on the channel dimension. The submatrix corresponding to the target position in each two-dimensional matrix is ​​any submatrix in each two-dimensional matrix. Optionally, the elements corresponding to the target position in each two-dimensional matrix are the elements in each two-dimensional matrix except the outermost K rows and the outermost K columns, and K is an integer not less than 1.

[0215] In an optional implementation, any main convolution kernel includes P channels, and the spatial auxiliary convolution kernel obtained by the any main convolution kernel includes P channels. The image processing device can use the submatrices corresponding to the target positions in the two-dimensional matrix of the any main convolution kernel on the 1st channel to the Pth channel as the two-dimensional matrices of the spatial auxiliary convolution kernel on the 1st channel to the Pth channel. For example, a main convolution kernel includes P channels, that is, the first channel to the Pth channel, and the two-dimensional matrix on each channel is a (7×7) matrix. The image processing device obtains a (5×5) submatrix in each of the two-dimensional matrices on the first channel to the Pth channel included in the main convolution kernel as the two-dimensional matrices on the first channel to the Pth channel included in the spatial auxiliary convolution kernel.

[0216] In an optional implementation, any main convolution kernel includes P channels, and the spatial auxiliary convolution kernel obtained by the any main convolution kernel includes P channels. The image processing device can use the submatrix corresponding to the target position in the two-dimensional matrix of the any main convolution kernel on the qth channel as the two-dimensional matrix of the spatial auxiliary convolution kernel obtained by the any main convolution kernel on the pth channel, where P, p, and q are all integers greater than 0, and p is not equal to q. For example, if a main convolution kernel includes three channels, the image processing device uses the submatrix of the two-dimensional matrix of the main convolution kernel on the first channel as the two-dimensional matrix of the spatial auxiliary convolution kernel on the second channel, the submatrix of the two-dimensional matrix of the main convolution kernel on the second channel as the two-dimensional matrix of the spatial auxiliary convolution kernel on the third channel, and the submatrix of the two-dimensional matrix of the main convolution kernel on the third channel as the two-dimensional matrix of the spatial auxiliary convolution kernel on the first channel.

[0217] In an optional implementation, the image processing device can obtain elements at different positions of each of the P two-dimensional matrices to obtain a spatial auxiliary convolution kernel. The P two-dimensional matrices are two-dimensional matrices on the P channels included in any main convolution kernel. The elements included in the two-dimensional matrix of the spatial auxiliary convolution kernel on the p1th channel are the elements at the first position in the two-dimensional matrix of any main convolution kernel on the q1th channel; the elements included in the two-dimensional matrix of the auxiliary convolution kernel on the p2th channel are the elements at the second position in the two-dimensional matrix of any main convolution kernel on the q2th channel, and the first position and the second position are different. The spatial auxiliary convolution kernel includes P channels, and P is greater than 0. p1, p2, q1, and q2 are all integers greater than 0 and not greater than P. The image processing device may use the submatrix corresponding to the element at the first position in the two-dimensional matrix of any main convolution kernel on the q1th channel as the two-dimensional matrix of the spatial auxiliary convolution kernel on the p1th channel; and use the submatrix corresponding to the element at the second position in the two-dimensional matrix of any main convolution kernel on the q2th channel as the two-dimensional matrix of the spatial auxiliary convolution kernel on the p2th channel. For example, for a (3×3×256) main convolution kernel, there is a (3×3) two-dimensional matrix on each channel, and the two-dimensional matrix on the first channel of the spatial auxiliary convolution kernel obtained by the main convolution kernel is the matrix at the upper left corner (2×2) of the two-dimensional matrix on the first channel of the main convolution kernel, and the two-dimensional matrix on the second channel of the spatial auxiliary convolution kernel is the matrix at the lower right corner (2×2) of the two-dimensional matrix on the second channel of the main convolution kernel. It can be understood that the image processing device can extract elements at different positions in the two-dimensional matrix of any main convolution kernel on each channel to obtain the auxiliary convolution kernel.

[0218] The image processing device can obtain multiple spatial auxiliary convolution kernels from any main convolution. The image processing device can obtain the elements of the third position of each two-dimensional matrix in P two-dimensional matrices to obtain the first auxiliary convolution kernel; obtain the elements of the fourth position of each two-dimensional matrix in the P two-dimensional matrices to obtain the second auxiliary convolution kernel; if the third position is different from the fourth position, the first auxiliary convolution kernel and the second auxiliary convolution kernel are different. The P two-dimensional matrices are the two-dimensional matrices of any main convolution kernel on P channels. This application does not limit the position of the elements in each two-dimensional matrix obtained by the image processing device, that is, it does not limit the target position. Fig. 6A A schematic diagram of a comparison between a main convolution kernel and an auxiliary convolution kernel provided in an embodiment of the present application is shown in FIG. Fig. 6A As shown, 601 is the main convolution kernel, and 602, 603, and 604 are all spatial auxiliary convolution kernels obtained from the main convolution kernel 601. Fig. 6A It can be seen that the spatial auxiliary convolution kernel 602, the spatial auxiliary convolution kernel 603, and the spatial auxiliary convolution kernel 604 are all included in the main convolution kernel 601. The two-dimensional matrix on each channel of the spatial auxiliary convolution kernel obtained by any main convolution kernel is obtained from the two-dimensional matrix on each channel of any main convolution kernel in the same way. The following takes the acquisition of the two-dimensional matrix on one channel of any main convolution kernel as an example to introduce how to obtain one or more spatial auxiliary convolution kernels from any main convolution kernel.

[0219] Figure 6B A schematic diagram for comparing another main convolution kernel and an auxiliary convolution kernel provided in an embodiment of the present application, Figure 6B The four matrices included are Fig. 6A The two-dimensional matrix of the four convolution kernels included in the p-th channel. Figure 6B As shown, 605 is the two-dimensional matrix of the main convolution kernel 601 on the pth channel, 606 is the two-dimensional matrix of the spatial auxiliary convolution kernel 602 on the pth channel, 607 is the two-dimensional matrix of the spatial auxiliary convolution kernel 603 on the pth channel, and 608 is the two-dimensional matrix of the spatial auxiliary convolution kernel 604 on the pth channel. Figure 6B It can be seen that 606 , 607 , and 608 are all sub-matrices included in 605 . Figure 6B In the figure, the matrices corresponding to the three solid black boxes from left to right are respectively the two-dimensional matrix of the spatial auxiliary convolution kernel 602 on the p-th channel, the two-dimensional matrix of the spatial auxiliary convolution kernel 603 on the p-th channel, and the two-dimensional matrix of the spatial auxiliary convolution kernel 604 on the p-th channel.

[0220] A method for obtaining multiple spatial auxiliary convolution kernels from any main convolution kernel is introduced below. The image processing device can obtain the elements except the outermost 2 rows and the outermost 2 columns in each two-dimensional matrix of any main convolution kernel in the channel dimension to obtain a spatial auxiliary convolution kernel, and extract the elements except the outermost 4 rows and the outermost 4 columns in each two-dimensional matrix to obtain another spatial auxiliary convolution kernel. It can be understood that the image processing device can obtain a series of spatial auxiliary convolution kernels from any main convolution kernel in a similar manner until the number of obtained spatial auxiliary convolution kernels meets the requirements. In the case where any main convolution kernel or the two-dimensional matrix of any main convolution kernel in the channel is a (d1×d2) matrix, the image processing device can obtain at most (s-1) spatial auxiliary convolution kernels in this manner; wherein, d = min (d1, d2), d is the smaller number between d1 and d2, The round-up symbol.

[0221] The image processing device adopts method 1 to obtain a spatial auxiliary convolution kernel that is smaller than the main convolution kernel from the main convolution kernel, which can not only quickly obtain the required spatial auxiliary convolution kernel but also improve the calculation efficiency.

[0222] Method 2: The image processing device sets all elements at positions other than the target position in each two-dimensional matrix of the main convolution kernel in the channel dimension to target values.

[0223] The above target value can be 0 or other values. In general, the target value is 0, which is not limited in this application. Optionally, the elements outside the target position in each two-dimensional matrix are the elements of the outermost K rows and the outermost K columns in each two-dimensional matrix, and K is an integer not less than 1. The image processing device can set at least one element in the two-dimensional matrix on any channel of any main convolution kernel to the target value to obtain a spatial auxiliary convolution kernel.

[0224] In an optional implementation, the image processing device sets the elements outside the target position in each of the P two-dimensional matrices to the target value to obtain a spatial auxiliary convolution kernel. The P two-dimensional matrices are P two-dimensional matrices on the P channels included in any main convolution kernel. The spatial auxiliary convolution kernel includes P channels, and P is an integer greater than 0. The image processing device may set the element at the fifth position of each of the P two-dimensional matrices to the target value to obtain a spatial auxiliary convolution kernel; set the element at the sixth position of each of the P two-dimensional matrices to the target value to obtain another spatial auxiliary convolution kernel; if the fifth position is different from the sixth position, the two spatial auxiliary convolution kernels are different. Optionally, the image processing device sets the elements of each of the P two-dimensional matrices except the outermost K rows and the outermost K columns to the target value; it may also be to set the elements at other positions in each two-dimensional matrix to the target value, which is not limited in this application. K is an integer not less than 1.

[0225] The following is a method for obtaining multiple spatial auxiliary convolution kernels from any main convolution kernel. The image processing device can set the outermost 2 rows and outermost 2 columns of each two-dimensional matrix of any main convolution kernel in the channel dimension as the target value to obtain the first auxiliary convolution kernel; set the outermost 4 rows and outermost 4 columns of each two-dimensional matrix as the target value to obtain the second auxiliary convolution kernel; set the outermost 2(s-1) rows and outermost 2(s-1) columns of each two-dimensional matrix as the target value to obtain the sth auxiliary convolution kernel. In the case that any main convolution kernel or the two-dimensional matrix of any main convolution kernel in the channel dimension is a (d1×d2) matrix, the image processing device can obtain at most (s-1) auxiliary convolution kernels by using this method; wherein, d = min (d1, d2), d is the smaller number between d1 and d2, The symbol for rounding up. The image processing device can use a similar method to set the elements at the same position in each two-dimensional matrix of any main convolution kernel on the channel dimension to the target value to obtain a spatial auxiliary convolution kernel. The following takes setting a two-dimensional matrix on a channel of any main convolution kernel as an example to introduce how to obtain one or more auxiliary convolution kernels from any main convolution kernel. Figure 7 A schematic diagram of a comparison between a main convolution kernel and an auxiliary convolution kernel provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, 701 is a two-dimensional matrix of a main convolution kernel on the tth channel, 702, 703, and 704 are two-dimensional matrices of spatial auxiliary convolution kernels obtained by the main convolution kernel on the tth channel, and t is an integer greater than 0. Figure 7 It can be seen that the image processing device sets the outermost row or rows and the outermost column or columns in each two-dimensional matrix of the main convolution kernel in the channel dimension to 0, and can obtain a spatial auxiliary convolution kernel of the same type as the main convolution kernel.

[0226] The following provides a method for setting the elements outside the target position in each two-dimensional matrix of the main convolution kernel in the channel dimension to the target value to obtain the spatial auxiliary convolution kernel.

[0227] The image processing device calculates the product of the elements at the corresponding positions in any main convolution kernel and the elements at the corresponding positions in the sampling matrix to obtain the spatial auxiliary convolution kernel; the sampling matrix and the any main convolution kernel are isotype matrices and the elements at each position in the sampling matrix correspond one-to-one to the elements at each position in the main convolution kernel, the elements at the target position of the sampling matrix are all 1 and the elements at positions other than the target position are all 0. The elements at the target position of the sampling matrix can be the elements other than the outermost K rows and the outermost K columns in each two-dimensional matrix of the sampling matrix in the channel dimension, and K is an integer not less than 1.

[0228] In an optional implementation, the sampling matrix and any main convolution kernel are homogeneous matrices, and the elements included in the outermost K rows and the outermost K columns of each two-dimensional matrix of the sampling matrix in the channel dimension are all 0, and the elements in the sampling matrix other than these elements are all 1; the image processing device calculates the product of the elements at the corresponding positions in any main convolution kernel and the elements at the corresponding positions in the sampling matrix to obtain the spatial auxiliary convolution kernel. It can be understood that the spatial auxiliary convolution kernel obtained by calculating the product of any main convolution kernel and the sampling matrix is ​​the same as the spatial auxiliary convolution kernel obtained by setting the elements included in the outermost K rows and the outermost K columns of each two-dimensional matrix of any main convolution kernel in the channel dimension to 0. The image processing device can respectively calculate the product of any main convolution kernel and multiple different sampling matrices to obtain multiple spatial auxiliary convolution kernels. In this implementation, the mathematical expression of the sampling matrix is ​​as follows:

[0229]

[0230] Among them, the sampling matrix M i is a three-dimensional matrix (d1×d2×Q), i is an integer not less than 1, and d is the smaller number between d1 and d2. i (a, b, :) represents the sampling matrix M i The ath row and bth column in each 2D matrix in the channel dimension. The sampling matrix M i is the i-th sampling matrix corresponding to any main convolution kernel, and is the same type matrix as any main convolution kernel, Q is the sampling matrix M i The image processing device can use the following formula to calculate the product of the element at the corresponding position in any main convolution kernel and the element at the corresponding position in the sampling matrix:

[0231]

[0232] Among them, f i is the ith spatial auxiliary convolution kernel obtained by any main convolution kernel, f is any main convolution kernel, Indicates the multiplication of the elements at corresponding positions of two matrices. When any of the main convolution kernels or the two-dimensional matrix of any of the main convolution kernels on the channel is a (d1×d2) matrix, any of the main convolution kernels corresponds to at most (s-1) sampling matrices. Among them, d = min (d1, d2), d is the smaller number between d1 and d2, The image processing device calculates the product of the elements at the corresponding positions in any one of the main convolution kernels and the elements at the corresponding positions in the (s-1) sampling matrices, respectively, to obtain (s-1) spatial auxiliary convolution kernels.

[0233] The image processing device adopts method 2 to obtain a spatial auxiliary convolution kernel of the same type as the main convolution kernel from the main convolution kernel, which is simple to implement.

[0234] Method 2: The image processing device obtains the channel auxiliary convolution kernel from the main convolution kernel.

[0235] The two-dimensional matrix of the channel auxiliary convolution kernel on the pth channel is a submatrix of the two-dimensional matrix of the main convolution kernel corresponding to the channel auxiliary convolution kernel on the qth channel, and p and q are both integers greater than 0. The channel auxiliary convolution kernel includes P channels, and the main convolution kernel corresponding to the channel auxiliary convolution kernel includes Q channels, P and Q are both integers greater than 0, and P is not equal to Q. The following introduces two methods for obtaining the channel auxiliary convolution kernel from the main convolution kernel.

[0236] Method 1: The image processing device obtains a two-dimensional matrix on any P channels of the Q channels included in the main convolution kernel to obtain a channel auxiliary convolution kernel, where Q is an integer greater than 0; the channel auxiliary convolution kernel includes P channels, where P is an integer greater than 0 and less than Q.

[0237] Optionally, the P channels may be P consecutive channels among the Q channels. Assuming that a main convolution kernel includes channels from the first channel to the Qth channel in sequence, then the uth channel to the (P+u-1)th channel are P consecutive channels. Wherein, (P+u-1) is not greater than Q, and u is an integer greater than 0.

[0238] Optionally, P is equal to 1, and the image processing device uses the two-dimensional matrix on any channel of any main convolution kernel as the two-dimensional matrix on one channel of the channel auxiliary convolution kernel.

[0239] Optionally, P is greater than 1, and the image processing device uses the two-dimensional matrix on any channel of any main convolution kernel as the two-dimensional matrix of each of the T channels of the channel auxiliary convolution kernel, where T is an integer greater than 1 and less than or equal to P. For example, the image processing device may use the two-dimensional matrix on the first channel of the main convolution kernel as the two-dimensional matrix on each of the first to 48th channels of the channel auxiliary convolution kernel, and use the two-dimensional matrix on the second channel of the main convolution kernel as the two-dimensional matrix on each of the 49th to 96th channels of the channel auxiliary convolution kernel. It can be understood that the two-dimensional matrices on each channel of a channel auxiliary convolution kernel all come from the main convolution kernel corresponding to the channel auxiliary convolution kernel.

[0240] The following is a method for obtaining multiple channel auxiliary convolution kernels from any main convolution kernel. The image processing device can sequentially use the P two-dimensional matrices on the 1st channel to the Pth channel of any main convolution kernel as the two-dimensional matrices on the 1st channel to the Pth channel of the first channel auxiliary convolution kernel; sequentially use the P two-dimensional matrices on the (1+g)th channel to the (P+g)th channel of any main convolution kernel as the two-dimensional matrices on the 1st channel to the Pth channel of the second channel auxiliary convolution kernel; sequentially use the P two-dimensional matrices on the (1+g*(f-1))th channel to the (P+g*(f-1))th channel of any main convolution kernel as the two-dimensional matrices on the 1st channel to the Pth channel of the fth channel auxiliary convolution kernel. Wherein, g is an integer greater than 0, which can be understood as the step size of the channel auxiliary convolution kernel, that is, the sliding step size when the image processing device samples on the channel of the main convolution kernel; f is an integer greater than 1 and less than or equal to ((QP) / g+1).

[0241] The two-dimensional matrix of a channel auxiliary convolution kernel obtained by any main convolution kernel on the pth channel is the matrix of any main convolution kernel on the qth channel, and p and q are both integers greater than 0. For example, a main convolution kernel includes 96 channels, each channel has a two-dimensional matrix, and the auxiliary convolution kernel obtained by the main convolution kernel includes 93 channels. The image processing device can use the 93 two-dimensional matrices on the 1st to 93rd channels of the 96 channels in sequence as the two-dimensional matrices on the 1st to 93rd channels of the auxiliary convolution kernel of the first channel; use the 93 two-dimensional matrices on the 2nd to 94th channels of the 96 channels in sequence as the two-dimensional matrices on the 1st to 93rd channels of the auxiliary convolution kernel of the second channel; use the 93 two-dimensional matrices on the 3rd to 95th channels of the 96 channels in sequence as the two-dimensional matrices on the 1st to 93rd channels of the auxiliary convolution kernel of the third channel; and use the 93 two-dimensional matrices on the 4th to 96th channels of the 96 channels in sequence as the two-dimensional matrices on the 1st to 93rd channels of the auxiliary convolution kernel of the fourth channel. Figure 8A schematic diagram of a comparison between a main convolution kernel and an auxiliary convolution kernel provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, 801 is the main convolution kernel, 802 is the first channel auxiliary convolution kernel, 803 is the second channel auxiliary convolution kernel, 804 is the third channel auxiliary convolution kernel, and 805 is the fourth channel auxiliary convolution kernel. These channel auxiliary convolution kernels are obtained by the image processing device obtaining a two-dimensional matrix on multiple continuous channels of the main convolution kernel 801. The number of channels included in the channel auxiliary convolution kernel obtained by any main convolution kernel is less than the number of channels included in any main convolution kernel. In other words, the size of the channel auxiliary convolution kernel is smaller than the size of the main convolution kernel, and the speed of convolution operation using the channel auxiliary convolution kernel is faster.

[0242] Method 2: The image processing device sets the elements in the two-dimensional matrix on any one or more channels of the Q channels included in the main convolution kernel to the target value to obtain the channel auxiliary convolution kernel. Q is an integer greater than 0. The target value can be 0 or other values.

[0243] In an optional implementation, the image processing device sets the (QP) two-dimensional matrices on any (QP) channels of the Q channels included in any main convolution kernel to zero matrices to obtain a channel auxiliary convolution kernel. The (QP) channels may be (QP) continuous channels of the Q channels. Q is an integer greater than 0, and P is an integer greater than 0 and less than Q. The image processing device can obtain multiple channel auxiliary convolution kernels from a main convolution kernel in a similar manner. A method for obtaining multiple channel auxiliary convolution kernels from any main convolution kernel is described below. The image processing device can set the two-dimensional matrices on each channel except the 1st channel to the Pth channel of any main convolution kernel to zero matrices to obtain the first channel auxiliary convolution kernel; set the two-dimensional matrices on each channel except the (1+g)th channel to the (P+g)th channel of any main convolution kernel to zero matrices to obtain the second channel auxiliary convolution kernel; set the two-dimensional matrices on each channel except the (1+g*(u-1))th channel to the (P+g*(u-1))th channel of any main convolution kernel to zero matrices to obtain the uth channel auxiliary convolution kernel. Wherein, g is an integer greater than 0, which can be understood as the step size of the channel auxiliary convolution kernel; u is an integer greater than 1 and less than or equal to ((PH) / g+1).

[0244] The following provides a method for setting one or more two-dimensional matrices of any main convolution kernel in the channel dimension to a zero matrix to obtain a channel auxiliary convolution kernel.

[0245] In an optional implementation, the sampling matrix and any main convolution kernel are homogeneous matrices, the (QP) two-dimensional matrices on the (QP) channels among the Q channels of the sampling matrix are all zero matrices, and the elements in the two-dimensional matrices on the channels other than the (QP) channels in the sampling matrix are all 1. Q is an integer greater than 0, and P is an integer greater than 0 and less than Q. The image processing device calculates the product of the elements at the corresponding positions in any main convolution kernel and the elements at the corresponding positions in the sampling matrix to obtain the channel auxiliary convolution kernel. The image processing device can respectively calculate the product of any main convolution kernel and multiple different sampling matrices to obtain multiple channel auxiliary convolution kernels. In this implementation, the mathematical expression of the sampling matrix is ​​as follows:

[0246]

[0247] Among them, the sampling matrix M j is a three-dimensional matrix (d1×d2×Q), j is an integer not less than 1, and g is an integer greater than 0. The sampling matrix M j is the jth sampling matrix corresponding to any main convolution kernel, and is the same type matrix as any main convolution kernel, Q is the sampling matrix M j (:,:,l) includes the number of channels. j (:,:,l) represents the sampling matrix M j All elements in the two-dimensional matrix on the lth channel. The image processing device uses the following formula to calculate the product of the elements at the corresponding position in any main convolution kernel and the elements at the corresponding position in the sampling matrix:

[0248]

[0249] Among them, f j is the jth auxiliary convolution kernel obtained from any main convolution kernel, Indicates the multiplication of the elements at corresponding positions of the two matrices. In this implementation, any main convolution kernel corresponds to at most sampling matrices. Where (QP) is the number of zero matrices included in any sampling matrix in the channel dimension; g is an integer greater than 0, which can be understood as the step size of the channel auxiliary convolution kernel; In this implementation, the auxiliary convolution kernel obtained by the image processing device from any main convolution kernel is a channel auxiliary convolution kernel.

[0250] The image processing device may pre-store multiple sampling matrices corresponding to any one of the main convolution kernels, and use these sampling matrices and any one of the main convolution kernels to obtain multiple channel auxiliary convolution kernels. The image processing device obtains the channel auxiliary convolution kernel by calculating the product of the elements at the corresponding positions in the main convolution kernel and the elements at the corresponding positions in the sampling matrix, and can quickly obtain multiple channel auxiliary convolution kernels different from the main convolution kernel, which is simple to implement.

[0251] Method three: The image processing device uses the auxiliary convolution kernel obtained from the main convolution kernel as the new main convolution kernel, and obtains a new auxiliary convolution kernel from the new main convolution kernel.

[0252] In an optional implementation, the image processing device may first obtain multiple spatial auxiliary convolution kernels from one or more main convolution kernels; and then use these spatial auxiliary convolution kernels as new main convolution kernels to obtain channel auxiliary convolution kernels. For example, the image processing device obtains 4 spatial auxiliary convolution kernels from one main convolution kernel; and uses each spatial auxiliary convolution kernel as a new main convolution kernel to obtain 3 channel auxiliary convolution kernels, so that 12 auxiliary convolution kernels can be obtained.

[0253] In an optional implementation, the image processing device may first obtain multiple channel auxiliary convolution kernels from one or more main convolution kernels; and then use these channel auxiliary convolution kernels as new main convolution kernels to obtain spatial auxiliary convolution kernels. For example, the image processing device obtains 4 channel auxiliary convolution kernels from one main convolution kernel; and uses each channel auxiliary convolution kernel as a new main convolution kernel to obtain 3 spatial auxiliary convolution kernels, so that 12 auxiliary convolution kernels can be obtained.

[0254] In method three, the image processing device uses the generated auxiliary convolution kernel as a new main convolution kernel to generate another auxiliary convolution kernel, which not only solves the problem of the limited number of auxiliary convolution kernels generated based on a main convolution kernel, but also makes fuller use of the main convolution kernel, further reducing the number of parameters in CNN.

[0255] The parameter quantity of the auxiliary convolution kernel obtained by the image processing device using any of the above methods is less than or equal to the corresponding main convolution kernel, so the computational complexity of the convolution operation using each auxiliary convolution kernel is no higher than the computational complexity of the main convolution kernel. In other words, the auxiliary convolution kernel obtained using the above method can reduce the computational complexity of the convolution operation and improve the computational efficiency.

[0256] In the present application, the image processing device may also use other methods to obtain an auxiliary convolution kernel from any main convolution kernel, and the method of obtaining an auxiliary convolution kernel from a main convolution kernel is not limited to the above three methods. It should be understood that the scheme of obtaining an auxiliary convolution kernel from a main convolution kernel in other ways also belongs to the protection scope of the present application.

[0257] The foregoing embodiments describe various implementation methods for obtaining auxiliary convolution kernels from a main convolution kernel. The following describes a process for storing and using auxiliary convolution kernels obtained from a main convolution kernel.

[0258] In an optional implementation, after the image processing device obtains the auxiliary convolution kernel from the main convolution kernel, it directly uses the obtained auxiliary convolution kernel to perform convolution operation on the image information and stores the obtained feature map. In this implementation, the image processing device directly performs convolution operation using the auxiliary convolution kernel obtained by sampling or adjusting the main convolution kernel, and does not need to store these auxiliary convolution kernels, which can effectively reduce storage overhead.

[0259] In an optional implementation, after the image processing device obtains the auxiliary convolution kernel from the main convolution kernel of any convolution layer of the convolutional neural network, it stores the obtained auxiliary convolution kernel. After the convolution operation is completed by using these auxiliary convolution kernels in any convolution layer, the stored auxiliary convolution kernel is cleared; and the convolution operation of the next convolution layer is continued. For example, when the image processing device performs the convolution operation of the first convolution layer of the convolution neural network, the auxiliary convolution kernel is obtained from the main convolution kernel of the first convolution layer and stored, and the main convolution kernel and the auxiliary convolution kernel of the first convolution layer are used to perform convolution operation on the input image information, and the obtained feature map is stored; before performing the convolution operation of the second convolution layer of the convolution neural network, the stored auxiliary convolution kernel of the first convolution layer is cleared; when performing the convolution operation of the second convolution layer of the convolution neural network, the auxiliary convolution kernel is obtained from the main convolution kernel of the second convolution layer and stored, and the main convolution kernel and the auxiliary convolution kernel of the second convolution layer are used to perform convolution operation on the input image information, and the obtained feature map is stored; the previous operation is repeated until the convolution operation of each convolution layer is completed. In this implementation, the image processing device does not need to store the auxiliary convolution kernels of each convolution layer at the same time, but stores the auxiliary convolution kernels of each convolution layer in sequence during the process of performing the convolution operation of each convolution layer; the storage space occupied by the convolution kernel can be effectively reduced.

[0260] In this application, the auxiliary convolution kernel obtained from the main convolution kernel can be used in any convolutional neural network. The following takes AlexNet as an example to introduce how to use the auxiliary convolution kernel in the process of using CNN to implement image classification. Fig. 9 A schematic diagram of the structure of AlexNet provided in the embodiment of the present application is shown in FIG. Fig. 9 As shown in the figure, AlexNet contains 5 convolutional layers and 3 fully connected layers. 901 to 905 are the image information input from the first convolutional layer to the fifth convolutional layer, 902 to 906 are the output feature maps from the first convolutional layer to the fifth convolutional layer, and 907 to 909 are 3 fully connected layers. Fig. 9It can be seen that the sizes of the sliding windows (two-dimensional matrices on each channel of the convolution kernel) from the first convolution layer to the fifth convolution layer are 11×11, 5×5, 3×3, 3×3, and 3×3, respectively. The number of input channels (the number of channels of input image information) from the first convolution layer to the fifth convolution layer are 3, 96, 256, 384, and 384, respectively, and the number of output channels (the number of channels of the output feature map) are 96, 256, 384, 384, and 256, respectively. It can be understood that the convolution kernel used in the first convolution layer is a three-dimensional matrix of (11×11×3); the convolution kernel used in the second convolution layer is a three-dimensional matrix of (5×5×96); and the size of the convolution kernel of each layer can be determined by analogy. Fig. 9 The AlexNet in the embodiment may also include a pooling layer. Since the main invention of this application lies in the convolution layer, the pooling layer will not be described in detail here.

[0261] Use of spatial auxiliary convolution kernels: Take the first convolution layer as an example to introduce the use of spatial auxiliary convolution kernels. The image processing device samples the main convolution kernel (11×11×3) in space, and can obtain 6 spatial auxiliary convolution kernels of different sizes {11×11×3, 9×9×3, 7×7×3, 5×5×3, 3×3×3, 1×1×3}, and their parameters are all from the main convolution kernel. Since the number of output channels of the first convolution layer of AlexNet is 96, in order to keep the number of channels consistent, the number of output channels of the main convolution kernel (11×11×3) is 16, that is, the number of output channels of the feature map obtained by the main convolution kernel is 16, so the number of output channels of the 6 spatial auxiliary convolution kernels obtained by the main convolution kernel is also 16. The 6 output feature maps obtained by these 6 spatial auxiliary convolution kernels are spliced ​​together, and the number of output channels of the feature map finally obtained is 16×6=96. In this case, the number of parameters of the spatial auxiliary convolution kernel of this convolution layer is (11×11×3×16=5808), while the number of parameters of this convolution layer in the original AlexNet is (11×11×3×96=34848), which is reduced by 34848 / 5808=6 times. The calculation speed is accelerated by (11×11×6) / (11×11+9×9+7×7+5×5+3×3+1×1)=2.54 times. For other convolution layers, the operation is similar.

[0262] Use of channel auxiliary convolution kernel: The use of channel auxiliary convolution kernel is introduced by taking the second convolution layer with an input channel number of 96 as an example. If the sampling step is set to 1 and the channel auxiliary convolution kernel has a channel number of 93, the image processing device samples on the channel of the main convolution kernel to obtain 4 channel auxiliary convolution kernels with an input channel number of 93. Since the output channel number of the second convolution layer in AlexNet should be 256, in order to keep the number of channels consistent, the output channel number of the main convolution kernel is 64, so the output channel number of the 4 channel auxiliary convolution kernels is also 64. The 4 output feature maps obtained by these 4 channel auxiliary convolution kernels are spliced ​​together, and the output channel number of the feature map finally obtained is 64×4=256. In this case, the number of parameters of the channel auxiliary convolution kernel of this convolution layer is 5×5×96×64=153600, while the number of parameters of this convolution layer in the original AlexNet is 5×5×96×256=614400, which is reduced by 614400 / 153600=4 times. For other convolution layers, the operation is similar.

[0263] The method in the embodiment of the present application is applicable to all CNN structures. For networks other than AlexNet, such as ResNet-50 and ResNeXt-50, the auxiliary convolution kernel can replace the original convolution kernel in a manner similar to the above description, so as to reduce the number of network parameters and increase the calculation speed.

[0264] The above-mentioned embodiment introduces how to use auxiliary convolution kernels in the process of implementing image classification using CNN. The following introduces the image classification effect achieved by the image processing method provided in the embodiment of the present application.

[0265] The algorithm proposed in this application was tested on the ImageNet dataset for image classification. Table 2 shows the experimental results of several convolutional neural networks (AlexNet, ResNet-50 and ResNeXt-50) using the auxiliary convolution kernels proposed in this application. In the experiment, the structure of these neural network models (number of layers, convolution kernel size of each layer, parameters, etc.) is not changed, and the number of convolution kernels of each layer is only reduced according to the calculation method of the auxiliary convolution kernel proposed in this application.

[0266] Table 2

[0267]

[0268] Table 2 shows the parameter quantity, memory usage, floating point calculation times and accuracy of various convolutional neural networks when performing image classification experiments on the ImageNet dataset. In Table 2, Versatile-AlexNet represents AlexNet using spatial auxiliary convolution kernels, Versatile v2-AlexNet represents AlexNet using spatial auxiliary convolution kernels and channel auxiliary convolution kernels; Versatile-ResNet-50 represents ResNet-50 using spatial auxiliary convolution kernels, Versatile v2-ResNet-50 represents ResNet-50 using spatial auxiliary convolution kernels and channel auxiliary convolution kernels; Versatile v2-ResNeXt-50 represents ResNeXt-50 using spatial auxiliary convolution kernels and channel auxiliary convolution kernels. It can be seen from Table 2 that after replacing the traditional convolution kernels in the existing convolutional neural network with the auxiliary convolution kernels proposed in this application, the number of parameters of the convolutional neural network is greatly reduced, the memory usage and the amount of calculation are also reduced, and the impact on the image classification effect is very small. Furthermore, the auxiliary convolution kernel proposed in this application is used to replace the traditional convolution kernel in some lightweight deep convolutional neural networks (such as ShuffleNet), and trained on the ImageNet dataset. This can verify whether the auxiliary convolution kernel proposed in this application is effective when applied to lightweight deep convolutional neural networks.

[0269] Table 3

[0270] network Parameter quantity Memory usage Calculation times Top1 Error Rate ShuffleNet <![CDATA[0.7×10 7 ]]> 20.6MB <![CDATA[0.5×10 9 ]]> 26.3％ Versatile v2-ShuffleNet <![CDATA[0.4×10 7 ]]> 14.0MB <![CDATA[0.5×10 9 ]]> 27.6％

[0271] The size of most convolution kernels in ShuffleNet is 1x1. As can be seen from Table 3, the application of the auxiliary convolution kernel proposed in this application can reduce its memory overhead by more than 30% and has little effect on the accuracy of image classification.

[0272] In addition to being applied to image classification tasks, the auxiliary convolution kernel proposed in this application can also be applied to other image processing tasks. Taking the image super-resolution task as an example, a large number of convolution operations are required to perform image super-resolution tasks using the VDSR (Very Deep CNN for Image Super-resolution) network. The traditional convolution kernel in the VDSR network can be replaced with the auxiliary convolution kernel in this application, and the replacement method is similar to AlexNet. The VDSR network is an image super-resolution CNN model published in CVPR 2016. CVPR is the abbreviation of IEEE conference on computer vision and pattern recognition, i.e., the IEEE International Conference on Computer Vision and Pattern Recognition. The specific experimental results on the super-resolution standard dataset are shown in Table 4.

[0273] Table 4

[0274] network Memory usage Calculation times PSNR(×2) PSNR(×4) VDSR 2.82MB <![CDATA[48.39×10 9 ]]> 37.53dB 31.35dB Versatile-VDSR 1.41MB <![CDATA[26.90×10 9 ]]> 37.64dB 31.41dB Versatile v2-VDSR 0.69MB <![CDATA[26.46×10 9 ]]> 37.58dB 31.37dB

[0275] In Table 4, Versatile-VDSR represents a VDSR network that uses a spatial auxiliary convolution kernel, and Versatile v2-VDSR represents a VDSR network that uses a spatial auxiliary convolution kernel and a channel auxiliary convolution kernel. PSNR (Peak signal-to-noise ratio) refers to the peak signal-to-noise ratio. PSNR (×4) represents the peak signal-to-noise ratio of an image obtained by image super-resolution processing after it is magnified 4 times, and PSNR (×2) represents the peak signal-to-noise ratio of an image obtained by image super-resolution processing after it is magnified 2 times. It can be seen from Table 4 that the auxiliary convolution kernel proposed in this application can maintain or even improve the super-resolution effect while achieving the purpose of model compression and acceleration.

[0276] The above embodiments introduce the image processing method and the training method. The following describes the structure of the training device and the image processing apparatus for implementing these methods.

[0277] Fig.10 It is a schematic diagram of the hardware structure of a convolutional neural network training device provided in an embodiment of the present application. Fig.10 The training device 120 shown (the training device 120 may be a computer device) includes a memory 1001, a processor 1002, a communication interface 1003 and a bus 1004. The memory 1001, the processor 1002 and the communication interface 1003 are connected to each other through the bus 1004.

[0278] The memory 1001 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1001 may store a program. When the program stored in the memory 1001 is executed by the processor 1002, the processor 1002 and the communication interface 1003 are used to execute the various steps of the training method of the convolutional neural network of the embodiment of the present application.

[0279] Processor 1002 can adopt a general-purpose central processing unit, a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the convolutional neural network training device of the embodiment of the present application, or to execute the convolutional neural network training method of the method embodiment of the present application.

[0280] The processor 1002 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the training method of the convolutional neural network of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 1002. The above-mentioned processor 1002 may also be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1001, and the processor 1002 reads the information in the memory 1001, and combines its hardware to complete the functions required to be performed by the units included in the convolutional neural network training device of the embodiment of the present application, or executes the convolutional neural network training method of the method embodiment of the present application.

[0281] The communication interface 1003 uses a transceiver such as, but not limited to, a transceiver to implement communication between the training device 1000 and other devices or a communication network. For example, the communication interface 1003 can be used to obtain training data (such as the input image described in the first embodiment of the present application).

[0282] The bus 1004 may include a pathway for transmitting information between various components of the training device 120 (eg, the memory 1001 , the processor 1002 , and the communication interface 1003 ).

[0283] Fig.11 It is a schematic diagram of the hardware structure of the image processing device provided in an embodiment of the present application. Fig.11 The image processing device 1100 shown (the device 1100 may be a computer device, for example Figure 1 The execution device 110 shown includes a memory 1101, a processor 1102, a communication interface 1103 and a bus 1104. The memory 1101, the processor 1102 and the communication interface 1103 are connected to each other through the bus 1104.

[0284] The memory 1101 may be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1101 may store a program. When the program stored in the memory 1101 is executed by the processor 1102, the processor 1102 and the communication interface 1103 are used to execute the various steps of the image processing method based on the convolutional neural network of the embodiment of the present application.

[0285] Processor 1102 can adopt a general-purpose central processing unit, a microprocessor, an application-specific integrated circuit, a graphics processor or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the image processing device of the embodiment of the present application, or to execute the image processing method of the method embodiment of the present application.

[0286] The processor 1102 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the image processing method based on the convolution neural network of the present application may be completed by an integrated logic circuit of the hardware in the processor 1102 or an instruction in the form of software. The above-mentioned processor 1102 may also be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a ready-made programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, or the like. The storage medium is located in the memory 1101, and the processor 1102 reads the information in the memory 1101, and combines its hardware to complete the functions required to be performed by the units included in the image processing device of the embodiment of the present application, or executes the image processing method based on the convolutional neural network of the method embodiment of the present application.

[0287] The communication interface 1103 uses a transceiver such as but not limited to a transceiver to implement communication between the device 1100 and other devices or communication networks. For example, training data (such as the input image described in the second embodiment of the present application) can be obtained through the communication interface 1103.

[0288] The bus 1104 may include a path for transmitting information between various components of the image processing apparatus 1100 (eg, the memory 1101 , the processor 1102 , and the communication interface 1103 ).

[0289] It should be noted that although Fig.10 and Fig.11 The training device 120 and the execution device 110 (image processing apparatus 1100) shown in the figure only show the memory, processor, and communication interface, but in the specific implementation process, those skilled in the art should understand that the training device 120 and the execution device 110 also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the training device 120 and the execution device 110 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the training device 120 and the execution device 110 may also only include the devices necessary for implementing the embodiments of the present application, and do not necessarily include Fig.10 or Fig.11 All devices shown in .

[0290] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0291] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0292] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0293] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0294] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0295] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0296] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method based on convolutional neural network, characterized in that: include: receiving an input image; Preprocessing the input image to obtain preprocessed image information; A convolution operation is performed on the image information using a convolution neural network, the convolution neural network includes N convolution layers, and the convolution operation includes: using M convolution kernels in the nth convolution layer to perform a convolution operation on the image information, wherein the M convolution kernels include m main convolution kernels and r auxiliary convolution kernels, the r auxiliary convolution kernels are obtained from the m main convolution kernels, and one main convolution kernel corresponds to one or more auxiliary convolution kernels; wherein N and M are both integers greater than 0; n is an integer greater than 0 and less than or equal to N; m is an integer greater than 0 and less than M, and r is an integer greater than 0 and less than M; The convolution operation result is output, where the convolution operation result includes M feature maps.

2. The method according to claim 1, characterized in that: Any auxiliary convolution kernel among the r auxiliary convolution kernels includes P channels, the main convolution kernel corresponding to any auxiliary convolution kernel includes P channels, and the two-dimensional matrix of any auxiliary convolution kernel on the p-th channel includes a sub-matrix of the two-dimensional matrix of the main convolution kernel corresponding to any auxiliary convolution kernel on the p-th channel; wherein P is an integer greater than 0, and p is an integer greater than 0 and not greater than P.

3. The method according to claim 2, characterized in that The elements included in the two-dimensional matrix of any auxiliary convolution kernel on the p-th channel are the elements of the two-dimensional matrix of the main convolution kernel corresponding to any auxiliary convolution kernel on the p-th channel except the elements in the outermost K rows and the outermost K columns; K is an integer not less than 1.

4. The method according to claim 1, characterized in that: Any of the r auxiliary convolution kernels includes P channels, and any of the auxiliary convolution kernels includes a two-dimensional matrix on P continuous channels among the Q channels included in the main convolution kernel corresponding to the any of the auxiliary convolution kernels, where P and Q are both integers greater than 0, and P is less than Q.

5. The method according to any one of claims 1 to 4, characterized in that: S of the r auxiliary convolution kernels are obtained from any one of the m main convolution kernels, s is an integer greater than or equal to 1, and before using the convolution neural network to perform a convolution operation on the image information, the method further includes: training the convolution neural network; the training of the convolution neural network includes: Using the convolutional neural network to perform a convolution operation on the training sample to obtain a convolution operation result; According to the convolution operation result, the gradients of the s auxiliary convolution kernels are calculated respectively; According to the gradient of any one of the main convolution kernels, the elements in the any one of the main convolution kernels are updated, and the gradient of any one of the main convolution kernels is the sum of the gradients of the s auxiliary convolution kernels.

6. An image processing device, characterized in that: include: At least one processor and a non-transitory computer-readable storage medium coupled to the at least one processor, the non-transitory computer-readable storage medium being used to store programming instructions executed by the at least one processor; the programming instructions instructing the at least one processor to perform the following operations: Receiving an input image; preprocessing the input image to obtain preprocessed image information; A convolution operation is performed on the image information using a convolutional neural network, wherein the convolutional neural network includes N convolutional layers, and the convolution operation includes: Use M convolution kernels in the nth convolution layer to perform a convolution operation on the image information, wherein the M convolution kernels include m main convolution kernels and r auxiliary convolution kernels, the r auxiliary convolution kernels are obtained by the m main convolution kernels, and one main convolution kernel corresponds to one or more auxiliary convolution kernels; wherein N and M are both integers greater than 0; n is an integer greater than 0 and less than or equal to N; m is an integer greater than 0 and less than M, and r is an integer greater than 0 and less than M; output the convolution operation result, and the convolution operation result includes M feature maps.

7. The image processing device according to claim 6, characterized in that: Any auxiliary convolution kernel includes P channels, the main convolution kernel corresponding to any auxiliary convolution kernel includes P channels, and the two-dimensional matrix of any auxiliary convolution kernel on the p-th channel includes a sub-matrix of the two-dimensional matrix of the main convolution kernel corresponding to any auxiliary convolution kernel on the p-th channel; wherein P is an integer greater than 0, and p is an integer greater than 0 and not greater than P.

8. The image processing device according to claim 7, characterized in that: The elements included in the two-dimensional matrix of any auxiliary convolution kernel on the p-th channel are the elements of the two-dimensional matrix of the main convolution kernel corresponding to any auxiliary convolution kernel on the p-th channel except the outermost K rows and the outermost K columns; K is an integer not less than 1.

9. The image processing device according to claim 6, characterized in that: Any auxiliary convolution kernel includes P channels, and any auxiliary convolution kernel includes a two-dimensional matrix on P continuous channels among the Q channels included in the main convolution kernel corresponding to the any auxiliary convolution kernel, where P and Q are both integers greater than 0, and P is less than Q.

10. The image processing device according to any one of claims 6 to 9, characterized in that: S of the r auxiliary convolution kernels are obtained from any one of the m main convolution kernels, s is an integer greater than or equal to 1, and the programming instruction instructs the at least one processor to train the convolutional neural network by the following operations: Using the convolutional neural network to perform a convolution operation on the training sample to obtain a convolution operation result; According to the convolution operation result, the gradients of the s auxiliary convolution kernels are calculated respectively; According to the gradient of any one of the main convolution kernels, the elements in the any one of the main convolution kernels are updated, and the gradient of any one of the main convolution kernels is the sum of the gradients of the s auxiliary convolution kernels.

11. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores programming instructions executed by at least one processor, the programming instructions instructing the at least one processor to execute the method according to any one of claims 1-5.

12. A chip, comprising at least one processor and a data interface, wherein the at least one processor reads instructions stored in a memory through the data interface to execute the method according to any one of claims 1 to 5.

Images