Image processing method and device based on multi-channel convolution, and storage medium

Abstract

The invention provides an image processing method and device based on multi-channel convolution, and a storage medium. The method comprises the following steps: acquiring the number of input channels of an original image; setting first convolution kernels equal to the number of the input channels for the convolutional network model, and independently applying one first convolution kernel to each input channel to perform first convolution so as to output a plurality of intermediate feature maps; creating a full connection layer for the convolutional network model, and converting a feature matrix corresponding to the plurality of intermediate feature maps into a low-rank matrix through the full connection layer; performing second convolution on the low-rank matrix by using a second convolution kernel with the size of 1 * 1 through a convolutional network model so as to output a plurality of output feature maps; and obtaining a target image according to the plurality of output feature maps. On the basis, the calculation amount and the parameter amount when the convolutional network model processes the image can be reduced, and the image processing effect is ensured.

Description

Technical Field

[0001] The present application relates to the fields of image processing technology, neural network technology, and deep learning technology, and specifically to an image processing method, device, and storage medium based on multi-channel convolution. Background Art

[0002] Compared to traditional ISP (Image Signal Processing) pipeline (ISP) denoising algorithms, artificial intelligence (AI) technologies, represented by convolutional neural networks (CNNs), can significantly improve image denoising quality. Consequently, they have seen significant growth in the image processing field in recent years and are widely used in computer vision, with successful applications in image recognition, image compression, and object detection. However, existing CNN models (also known as "convolutional network models," "convolutional models," "network models," "neural models," or simply "models") contain a large number of redundant parameters, resulting in excessive computational complexity and excessive consumption of computing resources, both in terms of computational effort and parameter requirements. Consequently, these algorithms face numerous challenges in practical implementation, particularly on edge devices. For example, in scenarios like intelligent transportation, the high computational resource consumption makes real-time image display a significant challenge. Numerous solutions have been proposed, such as pruning and quantizing CNN models. However, these solutions are often performed manually after model training, essentially sacrificing performance to alleviate computational resource constraints, resulting in poor image processing results. Summary of the Invention

[0003] In view of this, the present application provides an image processing method, device, and storage medium based on multi-channel convolution, which can improve the problem of large computational complexity and large number of parameters when traditional convolutional network models process images.

[0004] This application provides an image processing method based on multi-channel convolution, including: In response to inputting an original image into a convolutional network model, obtaining the number of input channels of the original image; Setting a first convolution kernel equal to the number of input channels for the convolutional network model, and independently applying a first convolution kernel to each input channel to perform a first convolution to output a plurality of intermediate feature maps; Creating a fully connected layer for the convolutional network model, and converting the feature matrices corresponding to the plurality of intermediate feature maps into a low-rank matrix through the fully connected layer; Performing a second convolution on the low-rank matrix using a second convolution kernel of size 1*1 through the convolutional network model to output a plurality of output feature maps; A target image is obtained according to the plurality of output feature maps.

[0005] Optionally, converting the feature matrices corresponding to the plurality of intermediate feature maps into a low-rank matrix through the fully connected layer includes: The rank is obtained according to the several intermediate feature maps and the following relationship:

[0006]

[0007] Wherein, m represents the number of intermediate feature maps output by the first convolution, V1 is the one-dimensional feature vector corresponding to the m intermediate feature maps, n represents the number of output feature maps output by the second convolution, V2 is the one-dimensional feature vector corresponding to the n output feature maps, is the matrix multiplication function, is a one-dimensional feature vector matrix corresponding to m intermediate feature maps and n output feature maps, and is a non-full rank matrix; r represents the rank of the low rank matrix, is a one-dimensional feature vector matrix based on m intermediate feature maps and the corresponding rank, is a one-dimensional feature vector matrix corresponding to the rank and n output feature maps; A low-rank matrix is ​​obtained according to the rank.

[0008] Optional, is a linear function.

[0009] Optionally, the method further includes: Creating an activation layer for the convolutional network model; Performing a third convolution on the low-rank matrix using a nonlinear activation function through the activation layer to output a plurality of activation feature maps; The performing a second convolution on the low-rank matrix includes: A second convolution is performed on the plurality of activation feature maps.

[0010] Optionally, a second convolution is performed on the matrix using the following relationship:

[0011]

[0012]

[0013] in, represents the one-dimensional eigenvector corresponding to the low-rank matrix r, Represents the one-dimensional feature vectors corresponding to the several activation feature maps, represents the nonlinear activation function used by the activation layer, Represents a matrix transpose function.

[0014] Optionally, the number of nodes in the activation layer is equal to the number of nodes in the fully connected layer.

[0015] Optionally, perform a second convolution on the low-rank matrix to output a plurality of output feature maps, including: Divide the low-rank matrix into a plurality of subgroups, where the number of subgroups is the result of the integer division operation between the number of output feature maps and the parallelism of the plurality of subgroups; Perform a second convolution based on each subgroup to output a plurality of output feature maps.

[0016] Optionally, create a fully connected layer for the convolutional network model according to the parallelism of the plurality of subgroups and the following relationship: r < m, r < n, r is x * k, x is an integer and x is divisible by both m and n, where k is the parallelism of the plurality of subgroups.

[0017] Optionally, the fully connected layer uses a non-linear activation function to perform a second convolution on the two low-rank matrices.

[0018] Optionally, the non-linear activation function includes at least one of a rectified linear unit (ReLU) function, a logistic sigmoid function, and a hyperbolic tangent (Tanh) function.

[0019] An image processing device provided by the present application includes a processor and a memory. An image processing program is stored on the memory. When the image processing program is executed by the processor, the steps of the image processing method based on multi-channel convolution described in any one of the above are implemented.

[0020] A storage medium provided by the present application stores a computer program. When the computer program is executed by a processor, the steps of the image processing method based on multi-channel convolution described in any one of the above are implemented.

[0021] As described above, the present application performs convolution operations on each input channel independently through the first convolution, and uses a convolution kernel (i.e., the first convolution kernel) for each input channel to generate output, so that spatial features can be extracted on each channel. Since cross-channel combination is not performed, the cross-channel combination calculation can be reduced, thereby significantly reducing the amount of calculation and parameter amount when the convolution network model processes images; by creating a fully connected layer, a large feature matrix corresponding to several intermediate feature maps is converted into a smaller matrix (i.e., a low-rank matrix), which can significantly reduce the amount of parameters that need to be processed, and shares parameters to reduce redundancy, so that the model only needs to learn a small number of parameters during fine-tuning, thereby reducing the amount of calculation; through the second convolution, a 1*1 convolution kernel (i.e., the second convolution kernel) is used to combine the output across channels to generate the final output feature map, and the convolution of each channel has a higher nonlinear expression ability, thereby increasing the nonlinear expression ability of the network. In summary, this application can significantly reduce the amount of computation and the number of parameters while ensuring the image processing effect. In other words, the model of this application can be made deeper with the same number of parameters, or have a faster running speed and training speed with the same model structure, which is conducive to improving the training efficiency and generalization ability of the network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 1 is a flowchart of an image processing method based on multi-channel convolution according to an embodiment of the present application; Figure 2 This is a schematic diagram of the first convolution performed on the original image in this application; Figure 3 This is a schematic diagram of the convolutional network model of the present application performing convolution based on three convolutional layers; Figure 4 This is a schematic diagram of the low-rank decomposition of an m*n matrix by the fully connected layer of this application; Figure 5 1 is a flowchart of an image processing method based on multi-channel convolution according to another embodiment of the present application; Figure 6 This is a schematic diagram of the convolutional network model of the present application performing convolution based on four convolutional layers; Figure 7 This is a flowchart of an image processing method based on multi-channel convolution according to another embodiment of the present application. DETAILED DESCRIPTION

[0023] To address the aforementioned issues in the prior art, this application provides an image processing method, device, and storage medium based on multi-channel convolution. These protected claims are based on the same concept and address the same or similar principles. The implementation methods of each protected claim can be referenced across them, and any repetitions will not be repeated here.

[0024] In the scheme of the present application, a convolution operation is performed independently on each input channel through the first convolution, and each input channel uses a convolution kernel to generate an output, so that spatial features can be extracted on each channel. Since cross-channel combination is not performed, the cross-channel combination calculation is reduced, and the amount of calculation and parameters when the convolution network model processes the image is significantly reduced; by creating a fully connected layer, a large feature matrix corresponding to several intermediate feature maps is converted into a smaller matrix, thereby significantly reducing the amount of parameters that need to be processed, and sharing parameters to reduce redundancy, so that the model only needs to learn a small number of parameters during fine-tuning, reducing the amount of calculation; through the second convolution, a 1*1 convolution kernel is used to combine the output across channels to generate the final output feature map, and the convolution of each channel has a higher nonlinear expression ability, thereby increasing the nonlinear expression ability of the network.

[0025] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly described below in conjunction with specific embodiments and corresponding drawings. Obviously, the embodiments described below are only some of the embodiments of this application, not all of them. Unless there is a conflict, the following embodiments and their technical features can be combined with each other and also belong to the technical solutions of this application.

[0026] Figure 1 This is a flowchart of an image processing method based on multi-channel convolution provided by an embodiment of the present application. This image processing method based on multi-channel convolution may also be referred to as an "image processing method" or "method." The execution entity of each step may be an adapted image processing device, such as a mobile phone, or a storage medium, processor, controller, etc. with relevant image processing capabilities.

[0027] like Figure 1 As shown, the method includes at least the following steps S1 to S5: S1: In response to inputting the original image into the convolutional network model, the number of input channels of the original image is obtained.

[0028] The input channels (in_channels) of the original image (that is, the image originally input) refer to the components of the original image (which can be regarded as "input data") in the depth dimension.

[0029] Based on this, in one example, the number of input channels can be obtained according to the type of the original image. If the original image is a color image, that is, an RGB type, the number of input channels can be determined to be 3, corresponding to red (R), green (G), and blue (B), respectively. Each channel stores the pixel intensity of the corresponding color; if the original image is a gray image, the number of input channels can be determined to be 1, representing the brightness value of the pixel.

[0030] S2: Set the first convolution kernel equal to the number of input channels for the convolutional network model, and independently apply a first convolution kernel to each input channel to perform the first convolution to output several intermediate feature maps.

[0031] In a convolutional network model, a convolution kernel is a learnable weight matrix used to extract local features from the input data. Acting as a local feature detector, a convolution kernel can identify low-level features such as edges and textures in the original image, or combine these low-level features into high-level semantic features. Using different convolution kernels, a variety of features can be extracted, thereby constructing the hierarchical feature representation of the convolutional network model.

[0032] Combine Figure 2 As shown, taking the original image as an RGB type color image as an example, where each small box represents the pixel intensity of a pixel of the corresponding color, the number of input channels is 3, and the original image is decomposed into an image corresponding to R, an image corresponding to G, and an image corresponding to B. Here, three first convolution kernels, that is, three filters (Filter), can be used. The size of each first convolution kernel can be 3*3. Each first convolution kernel moves on the corresponding decomposed image to perform the first convolution. The convolution principle and process adopted by the first convolution can refer to the existing technology in this field and will not be repeated here. In this way, features of the original image in different aspects, such as low-level features such as edges, textures, corners, etc. of the original image are extracted respectively. Each first convolution kernel can extract a feature and obtain a new feature map. Finally, multiple feature maps are generated through the first convolution. These feature maps are called "intermediate feature maps".

[0033] A convolution operation is performed independently using a convolution kernel for each input channel to output the intermediate feature map, so that spatial features can be extracted on each channel. Compared with traditional standard convolution, step S2 does not perform cross-channel combination, which can reduce cross-channel combination calculations, thereby significantly reducing the amount of calculation and parameters when the convolutional network model processes images.

[0034] S3: Create a fully connected layer for the convolutional network model, and convert the feature matrices corresponding to several intermediate feature maps into a low-rank matrix through the fully connected layer.

[0035] S4: Using a second convolution kernel of size 1*1 through the convolutional network model, a second convolution is performed on the low-rank matrix to output several output feature maps.

[0036] The low-rank matrix can be regarded as a fully connected layer that converts the feature matrices corresponding to several intermediate feature maps into several low-rank feature maps, and the one-dimensional feature vectors corresponding to the several low-rank feature maps form the low-rank matrix. Figure 3As shown, the fully connected layer is a convolutional layer of the convolutional network model, which can also be called the "intermediate layer". Each circle of the fully connected layer represents the parameters corresponding to each second convolution kernel. The left column is the first convolutional layer, and each circle in the left column represents the feature parameters corresponding to several intermediate feature maps output by the step S2. The right column is the second convolutional layer, and each circle in the right column represents the feature parameters corresponding to several low-rank feature maps output by the fully connected layer.

[0037] Further integration Figure 4 As shown, the fully connected layer can convert the feature matrices corresponding to the several intermediate feature maps into a low-rank matrix through the following relationship: Relationship 1 Equation 2 Wherein, m represents the number of intermediate feature maps output by the first convolution, V1 is the one-dimensional feature vector corresponding to the m intermediate feature maps, n represents the number of output feature maps output by the second convolution, V2 is the one-dimensional feature vector corresponding to the n output feature maps, is the matrix multiplication function, is a one-dimensional feature vector matrix corresponding to m intermediate feature maps and n output feature maps, and is a non-full rank matrix; r represents the rank of the low rank matrix, is a one-dimensional feature vector matrix based on m intermediate feature maps and the corresponding rank, It is a one-dimensional feature vector matrix corresponding to the rank and n output feature maps; that is, the rank is obtained according to several intermediate feature maps and a relationship, and then a low-rank matrix is ​​obtained according to the rank.

[0038] Figure 4 Each rectangular box in represents a feature map. Figure 4 As shown in the figure above, the above equation 1 can be considered as a mapping from a one-dimensional 1*m eigenvector V1 to another one-dimensional n*1 eigenvector V2. If we only consider linear transformation, that is, set this mapping relationship as a linear transformation relationship, then the operation from one-dimensional eigenvector V1 to one-dimensional eigenvector V2 can be regarded as a linear transformation represented by an m*n matrix. Correspondingly, is a linear function.

[0039] Combine Figure 4 As shown in the figure below, the above relationship 2 can be regarded as a low-rank decomposition of the m*n matrix. For non-full-rank matrices , then theoretically the larger matrix can be decomposed into an approximate expression of the product of two matrices (i.e., an m*r matrix and an r*n matrix). The specific principles and processes of decomposing the two matrices can be referred to the existing technology of low-rank decomposition, which will not be described here. Then, the low-rank matrix can be obtained according to the rank r. The low-rank matrix can be considered as a one-dimensional 1*r eigenvector. The role of the fully connected layer is to convert a one-dimensional 1*m eigenvector V1 into a one-dimensional 1*r eigenvector.

[0040] like Figure 3 As shown, the input channel of the fully connected layer is m, the output channel is r, and the corresponding output r low-rank feature maps can be regarded as a one-dimensional r*1 feature vector .

[0041] S5: Obtain a target image based on several output feature maps.

[0042] An image obtained by superimposing the plurality of output feature maps is used as the target image.

[0043] As described above, the present application performs convolution operations on each input channel independently through the first convolution, and uses a convolution kernel (i.e., the first convolution kernel) for each input channel to generate output, which is equivalent to performing a depth-wise convolution without crossing channels. The second convolution uses a 1*1 convolution kernel (i.e., the second convolution kernel) to combine the outputs across channels to generate the final output feature map, which is equivalent to performing a point-by-point convolution across channels. That is, the first convolution and the second convolution of the present application are equivalent to depth-wise separable convolution.

[0044] The depth-wise separable convolution can significantly reduce the number of multiplication operations. For example, for a traditional standard convolution with an input feature map size of H* W*D and an output feature map size of H'*W'*N, the computational complexity is D*K 2 *H'*W'*N (where K is the size of the convolution kernel); the computational complexity of depthwise separable convolution is only D*K 2 *H'*W'+D*H'*W'*N. From this, we can see that when the number of output channels N is large, the computational advantage of depthwise separable convolution is particularly obvious‌.

[0045] For another example, for an original image with an input feature map size of 12*12*3, 256 convolution kernels of 5*5*3 are used to output an output feature map of size 8*8*256. The computational complexity of the traditional standard convolution method is: 256*(5*5*3)*(8*8)=1228800, where the computational complexity of a single-channel 5*5 convolution kernel is 5*5, and the computational complexity of the three channels is 5*5*3. The convolution kernel moves 8 times horizontally and vertically in the original image respectively; while the computational complexity of using depthwise separable convolution is: 3*(5*5*1)*(8*8) +256*(1*1*3)*(8*8)=53952, where depth convolution is performed first, and 3 5*5*1 first convolution kernels are used to output the original image with a feature map size of 12*12*3 into an intermediate feature map with a size of 8*8*3. The amount of calculation is 3*(5*5*1)*(8*8)=4800, and then 256 1*1*3 second convolution kernels are used to output the output feature map with a size of 8*8*256 for the intermediate feature map with a size of 8*8*3. The amount of calculation is 256*(1*1*3)*(8*8)=49152. It can be seen that the use of depth-separable convolution in this application can greatly reduce the amount of calculation.

[0046] The second convolution kernel is used to combine the outputs across channels through the second convolution, so that the convolution of each channel has a higher nonlinear expression ability, which can increase the nonlinear expression ability of the network model, thereby helping to ensure the accuracy of the image processing effect.

[0047] In addition, by creating a fully connected layer to convert a large feature matrix corresponding to several intermediate feature maps into a smaller low-rank matrix, the number of parameters that need to be processed can be significantly reduced. Parameters are shared to reduce redundancy, so that the model only needs to learn a small number of parameters during fine-tuning, thereby reducing the amount of computation. The specific reasons are as follows: 1. Reduce the number of parameters: For example, the weight matrix W∈R corresponding to the original image m×n If all parameters of the weight matrix W are updated directly, m×n parameters need to be updated; however, through step S3, the weight matrix W can be approximately converted into a low-rank matrix, which is equivalent to two matrices A∈R m×r and B∈R r×n The product of r≪min(m,n), min() is the minimum function, here, the number of parameters becomes m×r+r×n=r(m+n). Since r is much smaller than m and n, the number of parameters is greatly reduced.

[0048] 2. Parameter Sharing: A∈R m×r and B∈R r×nThe product of can be viewed as capturing the main information of weight updates through a low-dimensional space (rank r). This decomposition assumes that the main changes in weight updates are concentrated in the low-dimensional subspace, rather than the entire high-dimensional space. Therefore, only the parameters in the low-dimensional space need to be learned, without updating the entire high-dimensional matrix. This parameter sharing idea allows the network model to only learn a small number of parameters during fine-tuning, thereby reducing the amount of computation.

[0049] 3. Simplify the calculation process: Through the low-rank transformation in step S3, the weight matrix W∈R m×n The complex matrix operation is simplified to two simple matrices A∈R m×r and B∈R r×n Since the rank of the decomposed matrix is ​​small, the computational complexity and storage requirements are also correspondingly low.

[0050] This shows that in image processing processes such as image compression and feature extraction, convolution based on low-rank decomposition can be used to remove redundant information from images, reducing the number of parameters and the amount of computation. By converting the image matrix into a low-rank matrix, the image data can be effectively compressed while retaining the main information. This can improve the nonlinear expression ability of the convolution kernel and ensure the accuracy of the image processing effect.

[0051] In one example, the fully connected layer can use a nonlinear activation function to obtain the low-rank matrix, thereby adding nonlinear characteristics to the network model, so that no matter how many layers the network model has, the output of any layer is a nonlinear combination of the input variables of the layer, and the input variables obtained by the next layer are also a nonlinear combination, which is equivalent to the network model being a multi-layer nonlinear model, which can better learn and process the complex data characteristics of the image.

[0052] In actual scenarios, the nonlinear activation function includes but is not limited to at least one of a linear rectification ReLU function, a logistic Sigmoid function, and a hyperbolic tangent Tanh function.

[0053] Figure 5 FIG. 1 is a flow chart of an image processing method based on multi-channel convolution according to another embodiment of the present invention. Figure 5 As shown, the method at least includes the following steps S1 to S5: S1: In response to inputting the original image into the convolutional network model, obtaining the number of input channels of the original image; S2: Setting a first convolution kernel equal to the number of input channels for the convolutional network model, and independently applying a first convolution kernel to each input channel to perform a first convolution to output a number of intermediate feature maps; S3: Create a fully connected layer for the convolutional network model, and convert the feature matrices corresponding to several intermediate feature maps into a low-rank matrix through the fully connected layer; S31: Create an activation layer for the convolutional network model; S32: Perform a third convolution on the low-rank matrix using a nonlinear activation function through the activation layer to output several activation feature maps; S4: performing a second convolution on the plurality of activation feature maps using a second convolution kernel of size 1*1 through a convolutional network model to output a plurality of output feature maps; S5: Obtain a target image based on several output feature maps.

[0054] For Figure 5 and Figure 1 The steps with the same number and the same technical features can be executed in the same manner, and this application will not describe them one by one. Although step codes such as S1 and S2 are used herein, their purpose is to express the corresponding content more clearly and concisely, and do not constitute a substantial limitation on the order. When implementing the steps, those skilled in the art may first execute some steps with later numbers, and then execute some steps with earlier numbers. For example, the aforementioned step S31 may be executed first, and then steps S3 and S32 to S5 may be executed, but these should all be within the scope of protection of this application.

[0055] This example is equivalent to adding a convolutional layer, namely the activation layer, after the aforementioned fully connected layer. The activation layer uses a nonlinear activation function to convolve the low-rank matrix, which can add nonlinear characteristics to the network model, so that the output of any subsequent layer is a nonlinear combination of the input variables of this layer. The input variables obtained by the next layer are also a nonlinear combination, which is equivalent to the network model being a multi-layer nonlinear model, which can better learn and process the complex data characteristics of images.

[0056] Based on the activation layer, further combined Figure 6 As shown, step S4 of this embodiment performs a second convolution on the low-rank matrix through the following relationship: Equation 3 Equation 4 Equation 5 in, represents the one-dimensional eigenvector corresponding to the low-rank matrix, Represents the one-dimensional feature vectors corresponding to the several activation feature maps, represents the nonlinear activation function used by the activation layer, Represents the matrix transpose function.

[0057] Based on the above equation 3, after passing through the fully connected layer, the corresponding output r low-rank feature maps can be regarded as a one-dimensional r*1 feature vector Then, in order to ensure nonlinearity, these r low-rank feature maps are input into the activation layer. Based on the above relationship 4, the third convolution is performed to obtain r new feature maps, that is, the one-dimensional feature vectors corresponding to the several activated feature maps are obtained. , and then after point-by-point convolution through the above relational expression 5, these activated feature maps can be linearly transformed into n new feature maps, which are the output feature maps of step S5.

[0058] Combine Figure 6 As shown, each circle of the activation layer represents the parameters corresponding to each convolution kernel. In one example, the number of nodes used in the activation layer is equal to the number of nodes used in the fully connected layer. Here, this example can independently apply a convolution kernel to each output channel of the fully connected layer to perform a convolution operation to output a feature map, that is, no cross-channel combination is performed, thereby further reducing the amount of computation and the amount of parameters when the convolutional network model processes images.

[0059] Figure 7 FIG. 1 is a flow chart of an image processing method based on multi-channel convolution according to another embodiment of the present application. Figure 7 As shown, the method at least includes the following steps S1 to S5: S1: In response to inputting the original image into the convolutional network model, obtaining the number of input channels of the original image; S2: Setting a first convolution kernel equal to the number of input channels for the convolutional network model, and independently applying a first convolution kernel to each input channel to perform a first convolution to output a number of intermediate feature maps; S3: Create a fully connected layer for the convolutional network model, and convert the feature matrices corresponding to several intermediate feature maps into a low-rank matrix through the fully connected layer; S41: Divide the plurality of low-rank matrices into a plurality of subgroups, where the number of the subgroups is an integer division result between the number of output feature maps and the degree of parallelism of the plurality of subgroups; S42: Perform a second convolution on each subgroup using a second convolution kernel of size 1*1 through the convolutional network model to output a plurality of output feature maps; S5: Obtain a target image based on several output feature maps.

[0060] For Figure 7 and Figure 1 The steps with the same labels and the same technical features can be executed in the same manner, and this application will not describe them one by one.

[0061] This example is equivalent to introducing grouped convolution into depthwise separable convolution. Grouped convolution divides the relevant feature maps into multiple subgroups in the channel dimension and performs convolution operations independently within each subgroup. Specifically, grouped convolution reduces the computational complexity to 1 / g of the original by dividing the corresponding input channels and output channels into g non-overlapping subgroups, so that each subgroup only processes 1 / g of the channel data, and improves the running efficiency of the model by reducing the number of parameters and the amount of computation in the convolution operation.

[0062] Different from the problem that there is no information transmission between channels in traditional grouped convolution, this example uses a fully connected layer combined with grouped convolution to increase the parallelism between each subgroup. For example, according to the parallelism of the several subgroups and the following relational expression, a fully connected layer is created for the convolutional network model: r < m, r < n, r is x * k, x is an integer and x is divisible by both m and n, where k is the parallelism of the several subgroups and can be preset according to, for example, the precision requirements of the device applicable to the model.

[0063] In a specific implementation, first, a depth convolution kernel, that is, the first convolution kernel, is established according to the number of input channels, and then a fully connected layer is created according to the parallelism k. The way to take the rank (i.e., rank) of this fully connected layer follows the above relational expression, so as to solve the problem that there is no inter-channel communication in the subsequent use of grouped convolution. Then, a grouped pointwise convolution is created according to the parallelism k, where g = n / / k for the grouped convolution, and g represents the number of subgroups, that is, the number of subgroups is equal to the result of the integer division operation between the number of output channels and the parallelism.

[0064] For example, in a model scenario where it is necessary to design a model with 128 input channels and 256 output channels, the parallelism preset according to the device applicable to the model is 32. The specific implementation method is as follows: First, a depth convolution kernel is established according to the number of input channels (i.e., 128), and then a fully connected layer is created according to the parallelism k (i.e., 32). The way to take the rank of this fully connected layer is less than 128 and rank is x times of 32, that is, rank can take the value of 32 or 64. In this example, rank can take the value of 64. Then, a grouped pointwise convolution is created according to the parallelism k, where g = (256 / / 32) = 8, that is, the pointwise convolution adopts a grouping method of 8, and finally several output feature maps are output. Based on this, it can be regarded that: 4 channels rc0~rc3 of the fully connected layer will be mapped to 32 output channels oc0~oc31, 4 channels rc4~rc8 of the fully connected layer will be mapped to 32 output channels oc32~oc63, and so on, until 4 channels rc60~r6c3 of the fully connected layer will be mapped to oc224~oc255 of 32 output channels, totaling 8 subgroups. The introduction of the fully connected layer can force inter-channel communication while reducing the number of parameters, so as to eliminate the isolation of information between input channels.

[0065] An embodiment of the present application also provides a storage medium, on which an image processing program is stored. The image processing program is essentially a computer program, and when the image processing program is executed by a processor, the steps of the image processing method based on multi-channel convolution as in any example are implemented.

[0066] The storage medium includes but is not limited to any one of a read-only memory (ROM), a random access memory (RAM), a magnetic disk, and an optical disk.

[0067] Since the program stored in the storage medium can execute the steps in the image processing method of any embodiment provided in this application, the beneficial effects that can be achieved by the image processing method based on multi-channel convolution in any of the aforementioned embodiments can be achieved. Please see the aforementioned embodiments for details and will not be repeated here.

[0068] An embodiment of the present application also provides an image processing device or chip, including a memory and a processor, wherein an image processing program is stored on the memory, and when the image processing program is executed by the processor, the steps of the image processing method based on multi-channel convolution of any of the aforementioned embodiments are implemented; and / or, the image processing device or chip is provided with a storage medium as shown in the above example, and the processor loads the storage medium to execute the steps of the image processing method based on multi-channel convolution of any of the aforementioned embodiments, thereby achieving the beneficial effects that can be achieved by the image processing method based on multi-channel convolution of the corresponding embodiment.

[0069] The above are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. For ordinary technicians in this field, any equivalent structural changes made using the contents of this specification and drawings are also included in the patent protection scope of the present application.

[0070] Although the terms "first," "second," and the like are used herein to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. In addition, the singular forms "a," "an," and "the" are intended to include the plural forms as well. The terms "or" and "and / or" are to be interpreted as inclusive, meaning any one or any combination. Exceptions to this definition occur only when a combination of elements, functions, steps, or operations is inherently mutually exclusive in some manner.

Claims

1. An image processing method based on multi-channel convolution, characterized in that: include: In response to inputting an original image into a convolutional network model, obtaining the number of input channels of the original image; Setting a first convolution kernel equal to the number of input channels for the convolutional network model, and independently applying a first convolution kernel to each input channel to perform a first convolution to output a plurality of intermediate feature maps; Creating a fully connected layer for the convolutional network model, and converting the feature matrices corresponding to the plurality of intermediate feature maps into a low-rank matrix through the fully connected layer; Performing a second convolution on the low-rank matrix using a second convolution kernel of size 1*1 through the convolutional network model to output a plurality of output feature maps; A target image is obtained according to the plurality of output feature maps.

2. The method according to claim 1, characterized in that The converting the feature matrices corresponding to the plurality of intermediate feature maps into a low-rank matrix through the fully connected layer includes: The rank is obtained according to the several intermediate feature maps and the following relationship: Wherein, m represents the number of intermediate feature maps output by the first convolution, V1 is the one-dimensional feature vector corresponding to the m intermediate feature maps, n represents the number of output feature maps output by the second convolution, V2 is the one-dimensional feature vector corresponding to the n output feature maps, is the matrix multiplication function, is a one-dimensional feature vector matrix corresponding to m intermediate feature maps and n output feature maps, and is a non-full rank matrix; r represents the rank of the low rank matrix, is a one-dimensional feature vector matrix based on m intermediate feature maps and the corresponding rank, is a one-dimensional feature vector matrix corresponding to the rank and n output feature maps; A low-rank matrix is ​​obtained according to the rank.

3. The method according to claim 2, characterized in that is a linear function.

4. The method according to claim 2, characterized in that The method further comprises: Creating an activation layer for the convolutional network model; Performing a third convolution on the low-rank matrix using a nonlinear activation function through the activation layer to output a plurality of activation feature maps; The performing a second convolution on the low-rank matrix includes: A second convolution is performed on the plurality of activation feature maps.

5. The method according to claim 4, characterized in that The second convolution is performed on the low-rank matrix using the following relationship: in, represents the one-dimensional eigenvector corresponding to the low-rank matrix, Represents the one-dimensional feature vectors corresponding to the several activation feature maps, represents the nonlinear activation function used by the activation layer, Represents the matrix transpose function.

6. The method according to claim 4, characterized in that The number of nodes in the activation layer is equal to the number of nodes in the fully connected layer.

7. The method according to any one of claims 2 to 6, characterized in that Perform a second convolution on the low-rank matrix to output several output feature maps, including: Divide the low-rank matrix into a plurality of subgroups, where the number of the subgroups is an integer division result between the number of the output feature maps and the degree of parallelism of the plurality of subgroups; A second convolution is performed based on each subgroup to output several output feature maps.

8. The method according to claim 7, characterized in that A fully connected layer is created for the convolutional network model according to the parallelism of the plurality of subgroups and the following relationship: r < m, r < n, r is x*k, x is an integer and is divisible by both m and n, where k is the parallelism of the plurality of subgroups.

9. An image processing device, characterized in that The method comprises a processor and a memory, wherein an image processing program is stored in the memory, and when the image processing program is executed by the processor, the steps of the image processing method based on multi-channel convolution as claimed in any one of claims 1 to 8 are implemented.

10. A storage medium, characterized in that: A computer program is stored, which, when executed by a processor, implements the steps of the image processing method based on multi-channel convolution as described in any one of claims 1 to 8.

Images