Image processing method and device based on multi-channel convolution, and storage medium

Abstract

The application provides an image processing method and device based on multi-channel convolution, and a storage medium. The method comprises the following steps: acquiring an input channel number of an original image; setting a first convolution kernel equal to the input channel number for a convolution network model, and independently applying a first convolution kernel to each input channel to perform first convolution to output a plurality of intermediate feature maps; creating a full connection layer for the convolution network model, and converting a feature matrix corresponding to the plurality of intermediate feature maps into a low-rank matrix through the full connection layer; performing second convolution on the low-rank matrix by using a second convolution kernel with a size of 1*1 through the convolution network model to output a plurality of output feature maps; and obtaining a target image according to the plurality of output feature maps. Therefore, the application can reduce the calculation amount and parameter amount of the convolution network model when processing the image, and ensure the image processing effect.

Description

Technical Field

[0001] This application relates to the fields of image processing technology and neural network and deep learning technology, specifically to an image processing method and device based on multi-channel convolution, and a storage medium. Background Technology

[0002] Compared to traditional ISP (Image Signal Processing) denoising algorithms, artificial intelligence (AI) technologies, represented by convolutional neural network (CNN) algorithms, can significantly improve image denoising quality. Therefore, they have seen tremendous growth in the field of image processing in recent years and are widely used in computer vision, achieving successful applications in image recognition, image compression, and object detection. However, existing CNN models (also known as "convolutional network models," "convolutional models," "network models," "neural models," or simply "models") contain a large number of redundant parameters, resulting in excessive computational complexity and excessive consumption of computing resources. These computational resources are divided into computational load and parameter load, thus posing numerous difficulties when implementing algorithms in real-world scenarios, especially in edge devices. For example, in scenarios such as intelligent transportation, the high computational resource consumption makes real-time image display a significant challenge. Currently, the industry has proposed many solutions, such as pruning and quantizing CNN models. However, these solutions are usually performed manually after model training, essentially sacrificing some performance to alleviate computational resource constraints, resulting in poor image processing effects. Summary of the Invention

[0003] In view of this, this application provides an image processing method, device, and storage medium based on multi-channel convolution, which can improve the problem of large computational load and parameter quantity when traditional convolutional network models process images.

[0004] This application provides an image processing method based on multi-channel convolution, including: In response to inputting the original image into the convolutional network model, the number of input channels of the original image is obtained; The convolutional network model is set with a first convolutional kernel equal to the number of input channels, and a first convolutional kernel is applied independently to each input channel to perform the first convolution, so as to output several intermediate feature maps. A fully connected layer is created for the convolutional network model, and the feature matrix corresponding to the several intermediate feature maps is converted into a low-rank matrix through the fully connected layer. The convolutional network model uses a second convolutional kernel of size 1*1 to perform a second convolution on the low-rank matrix to output several output feature maps. A target image is obtained based on the aforementioned output feature maps.

[0005] Optionally, the step of converting the feature matrices corresponding to the plurality of intermediate feature maps into a low-rank matrix through the fully connected layer includes: The rank is obtained based on the aforementioned intermediate feature maps and the following relational formula;

[0006]

[0007] Where m represents the number of intermediate feature maps output from the first convolution, V1 is the one-dimensional feature vector corresponding to the m intermediate feature maps, n represents the number of output feature maps output from the second convolution, and V2 is the one-dimensional feature vector corresponding to the n output feature maps. This is the matrix multiplication function. Let be a one-dimensional feature vector matrix based on m intermediate feature maps and n output feature maps, and Let r be a non-full-rank matrix; r represents the rank of the low-rank matrix. It is a one-dimensional feature vector matrix based on m intermediate feature maps and the rank. It is a one-dimensional feature vector matrix based on the rank and n output feature maps; The low-rank matrix is ​​obtained based on the rank.

[0008] Optional, It is a linear function.

[0009] Optionally, the method further includes: Create an activation layer for the convolutional network model; The activation layer employs a nonlinear activation function to perform a third convolution on the low-rank matrix, thereby outputting several activation feature maps. The second convolution of the low-rank matrix includes: A second convolution is performed on the aforementioned activation feature maps.

[0010] Optionally, the matrix can be subjected to a second convolution using the following relation:

[0011]

[0012]

[0013] in, This represents the one-dimensional eigenvector corresponding to the low-rank matrix described in r. This represents a one-dimensional feature vector corresponding to the plurality of activated feature maps. This indicates the nonlinear activation function used in the activation layer. Represents a matrix transpose function.

[0014] Optionally, the number of nodes in the activation layer is equal to the number of nodes in the fully connected layer.

[0015] Optionally, perform a second convolution on the low-rank matrix to output a plurality of output feature maps, including: Divide the low-rank matrix into a plurality of subgroups, where the number of subgroups is the result of the integer division operation between the number of output feature maps and the parallelism of the plurality of subgroups; Perform a second convolution based on each subgroup to output a plurality of output feature maps.

[0016] Optionally, create a fully connected layer for the convolutional network model according to the parallelism of the plurality of subgroups and the following relational expression: r < m, r < n, r is x * k, x is an integer and x is divisible by both m and n, where k is the parallelism of the plurality of subgroups.

[0017] Optionally, the fully connected layer uses a non-linear activation function to perform a second convolution on the two low-rank matrices.

[0018] Optionally, the non-linear activation function includes at least one of a rectified linear unit (ReLU) function, a logistic sigmoid function, and a hyperbolic tangent (Tanh) function.

[0019] An image processing device provided by the present application includes a processor and a memory. An image processing program is stored on the memory. When the image processing program is executed by the processor, the steps of the image processing method based on multi-channel convolution described in any one of the above are implemented.

[0020] A storage medium provided by the present application stores a computer program. When the computer program is executed by a processor, the steps of the image processing method based on multi-channel convolution described in any one of the above are implemented.

[0021] As described above, this application performs independent convolution operations on each input channel through the first convolution, with each input channel using a convolution kernel (i.e., the first convolution kernel) to generate the output. This allows for the extraction of spatial features on each channel. Since cross-channel combination is not performed, cross-channel combination computation is reduced, thereby significantly reducing the computational cost and number of parameters when the convolutional network model processes images. By creating a fully connected layer, a large feature matrix corresponding to several intermediate feature maps is transformed into a smaller matrix (i.e., a low-rank matrix), which significantly reduces the number of parameters that need to be processed. Furthermore, sharing parameters reduces redundancy, allowing the model to learn only a small number of parameters during fine-tuning, thus reducing computational cost. The second convolution uses a 1*1 convolution kernel (i.e., the second convolution kernel) to combine the output across channels, thereby generating the final output feature map. This ensures that the convolution of each channel has high non-linear expressive power, increasing the non-linear expressive power of the network. In summary, this application can significantly reduce computational load and the number of parameters while maintaining image processing performance. In other words, the model proposed in this application can be deeper with the same number of parameters, or have faster running and training speeds with the same model structure, thereby improving the training efficiency and generalization ability of the network model. Attached Figure Description

[0022] Figure 1 This is a schematic flowchart of an image processing method based on multi-channel convolution according to an embodiment of this application; Figure 2 This is a schematic diagram of the first convolution of the original image in this application; Figure 3 This is a schematic diagram of the convolutional network model of this application, which is based on three convolutional layers for convolution. Figure 4 This is a schematic diagram of the low-rank decomposition of an m*n matrix by the fully connected layer of this application; Figure 5 This is a flowchart illustrating another embodiment of the image processing method based on multi-channel convolution in this application; Figure 6 This is a schematic diagram of the convolutional network model of this application, which performs convolution based on four convolutional layers; Figure 7 This is a flowchart illustrating another embodiment of the image processing method based on multi-channel convolution in this application. Detailed Implementation

[0023] To address the aforementioned problems in the prior art, this application provides an image processing method, apparatus, and storage medium based on multi-channel convolution. These protection subjects are based on the same concept, and their problem-solving principles are essentially the same or similar. The implementation methods of each protection subject can be referred to mutually, and repeated details will not be elaborated.

[0024] In this application's scheme, each input channel is independently convolved using the first convolution, with each input channel using a convolution kernel to generate the output. This allows for the extraction of spatial features on each channel. Since cross-channel combination is not performed, cross-channel combination computation is reduced, significantly reducing the computational cost and number of parameters when the convolutional network model processes images. By creating a fully connected layer, a large feature matrix corresponding to several intermediate feature maps is transformed into a smaller matrix, significantly reducing the number of parameters that need to be processed. Furthermore, parameter sharing reduces redundancy, allowing the model to learn only a small number of parameters during fine-tuning, further reducing computational cost. The second convolution uses a 1*1 convolution kernel to combine the output across channels, thereby generating the final output feature map. This ensures that the convolution of each channel has high non-linear expressive power, increasing the non-linear expressive power of the network.

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly described below in conjunction with specific embodiments and corresponding drawings. Obviously, the embodiments described below are only a part of the embodiments of this application, and not all of them. Unless otherwise specified, the following embodiments and their technical features can be combined with each other, and also belong to the technical solutions of this application.

[0026] Figure 1 This is a flowchart illustrating an image processing method based on multi-channel convolution provided in an embodiment of this application. This image processing method based on multi-channel convolution can also be called an "image processing method" or simply "method." The execution entity for each step can be a suitable image processing device, such as a mobile phone, or a storage medium, processor, controller, etc., with relevant image processing functions.

[0027] like Figure 1 As shown, the method includes at least the following steps S1 to S5: S1: In response to inputting the original image into the convolutional network model, obtain the number of input channels of the original image.

[0028] The input channels of the original image (i.e., the initial input image) refer to the components of the original image (which can be regarded as "input data") in the depth dimension.

[0029] Therefore, in one example, the number of input channels can be determined based on the type of the original image. If the original image is a color image, i.e., RGB type, then the number of input channels can be determined to be 3, corresponding to red (R), green (G), and blue (B) respectively, with each channel storing the pixel intensity of the corresponding color; if the original image is a grayscale image, then the number of input channels can be determined to be 1, representing the brightness value of the pixel.

[0030] S2: Set a first convolution kernel equal to the number of input channels for the convolutional network model, and apply a first convolution kernel independently to each input channel to perform the first convolution, so as to output several intermediate feature maps.

[0031] In a convolutional network model, a convolutional kernel is a learnable weight matrix used to extract local features from the input data. As a local feature detector, the convolutional kernel can identify low-level features such as edges and textures in the original image, or combine these low-level features into high-level semantic features. By using different convolutional kernels, multiple features can be extracted, thereby constructing a hierarchical feature representation for the convolutional network model.

[0032] Combination Figure 2 As shown, taking an RGB color image as an example, where each small square represents the pixel intensity of a pixel corresponding to the color, the number of input channels is 3. The original image is decomposed into an image corresponding to R, an image corresponding to G, and an image corresponding to B. Here, 3 first convolution kernels, i.e., 3 filters, can be used. The size of each first convolution kernel can be 3*3. Each first convolution kernel moves on the corresponding decomposed image to perform the first convolution. The convolution principle and process used by the first convolution can be found in the existing technology in this field, and will not be repeated here. In this way, features of the original image in different aspects are extracted, such as low-level features such as edges, textures, and corners of the original image. Each first convolution kernel can extract one feature and obtain a new feature map. Finally, multiple feature maps are generated through the first convolution. These feature maps are called "intermediate feature maps".

[0033] The intermediate feature map is output by using a separate convolution kernel for each input channel. This allows spatial features to be extracted in each channel. Compared to traditional standard convolution, step S2 does not perform cross-channel combination, which reduces cross-channel combination calculations and thus significantly reduces the computational load and parameter count of the convolutional network model when processing images.

[0034] S3: Create a fully connected layer for the convolutional network model, and convert the feature matrices corresponding to several intermediate feature maps into a low-rank matrix through the fully connected layer.

[0035] S4: The low-rank matrix is ​​convolved by a second convolution kernel of size 1*1 through a convolutional network model to output several output feature maps.

[0036] A low-rank matrix can be viewed as a fully connected layer transforming the feature matrices corresponding to several intermediate feature maps into several low-rank feature maps, and the one-dimensional eigenvectors corresponding to these low-rank feature maps form the low-rank matrix. Combined with... Figure 3As shown, the fully connected layer is a convolutional layer in the convolutional network model, also known as an "intermediate layer". Each circle in the fully connected layer represents the parameter corresponding to each second convolutional kernel. The left column is the first convolutional layer, and each circle in the left column represents the feature parameters corresponding to several intermediate feature maps output in step S2. The right column is the second convolutional layer, and each circle in the right column represents the feature parameters corresponding to several low-rank feature maps output by the fully connected layer.

[0037] Further integration Figure 4 As shown, the fully connected layer can convert the feature matrices corresponding to the intermediate feature maps into a low-rank matrix using the following relationship: Relation 1 Relation 2 Where m represents the number of intermediate feature maps output from the first convolution, V1 is the one-dimensional feature vector corresponding to the m intermediate feature maps, n represents the number of output feature maps output from the second convolution, and V2 is the one-dimensional feature vector corresponding to the n output feature maps. This is the matrix multiplication function. Let be a one-dimensional feature vector matrix based on m intermediate feature maps and n output feature maps, and Let r be a non-full-rank matrix; r represents the rank of the low-rank matrix. It is a one-dimensional feature vector matrix based on m intermediate feature maps and the rank. The rank is a one-dimensional feature vector matrix corresponding to the n output feature maps; that is, the rank is obtained based on several intermediate feature maps and the relation, and then the low-rank matrix is ​​obtained based on the rank.

[0038] Figure 4 Each rectangle in the diagram represents a feature map. Combined with... Figure 4 As shown in the diagram above, relation 1 can be considered as a mapping from a one-dimensional 1*m eigenvector V1 to another one-dimensional n*1 eigenvector V2. If only linear transformations are considered, i.e., if this mapping relationship is set as a linear transformation relationship, then the operation from one-dimensional eigenvector V1 to one-dimensional eigenvector V2 can be regarded as a linear transformation represented by an m*n matrix. Correspondingly, It is a linear function.

[0039] Combination Figure 4 As shown in the figure below, the above relation 2 can be regarded as performing a low-rank decomposition on the m*n matrix, for non-full-rank matrices. Theoretically, this large matrix can be decomposed into an approximate expression of the product of two matrices (i.e., an m*r matrix and an r*n matrix). The specific principles and processes of decomposing to obtain the two matrices can be found in the existing techniques of low-rank decomposition, which will not be elaborated here. Then, the low-rank matrix can be obtained according to the rank r. This low-rank matrix can be regarded as a one-dimensional 1*r eigenvector. The role of the fully connected layer is to transform a one-dimensional 1*m eigenvector V1 into a one-dimensional 1*r eigenvector.

[0040] like Figure 3 As shown, the fully connected layer has m input channels and r output channels. The r low-rank feature maps output can be viewed as a one-dimensional r*1 feature vector. .

[0041] S5: Obtain a target image based on several output feature maps.

[0042] The image obtained by superimposing the several output feature maps is the target image.

[0043] As described above, this application performs convolution operations independently on each input channel through the first convolution, and uses a convolution kernel (i.e., the first convolution kernel) to generate the output for each input channel, which is equivalent to performing a depthwise convolution without crossing channels. The second convolution uses a 1*1 convolution kernel (i.e., the second convolution kernel) to combine the output across channels, thereby generating the final output feature map, which is equivalent to performing a pointwise convolution across channels. That is, the first and second convolutions of this application are equivalent to depthwise separable convolutions.

[0044] The depthwise separable convolution can significantly reduce the number of multiplication operations. For example, for a traditional standard convolution with an input feature map size of H*W*D and an output feature map size of H'*W'*N, its computational complexity is D*K. 2 *H'*W'*N (where K is the kernel size); while the computational complexity of depthwise separable convolution is only D*K. 2 *H'*W'+D*H'*W'*N. Therefore, when the number of output channels N is large, the computational advantage of depthwise separable convolution is particularly significant.

[0045] For example, for an original image with an input feature map size of 12*12*3, using 256 convolutional kernels of size 5*5*3 to output an output feature map of size 8*8*256, the computational cost using the traditional standard convolution method is: 256*(5*5*3)*(8*8) = 1,228,800. Here, the computational cost of a single-channel 5*5 convolutional kernel is 5*5, so the computational cost of 3 channels is 5*5*3. The convolutional kernel moves 8 times horizontally and 8 times vertically in the original image. However, the computational cost using depthwise separable convolution is: 3*(5*5*1)*(8*8). +256*(1*1*3)*(8*8)=53952, where, firstly, depthwise convolution is performed, using three 5*5*1 first convolution kernels to output an intermediate feature map of size 8*8*3 from the original feature map of size 12*12*3, with a computational cost of 3*(5*5*1)*(8*8)=4800. Then, 256 1*1*3 second convolution kernels are used on the intermediate feature map of size 8*8*3 to output an output feature map of size 8*8*256, with a computational cost of 256*(1*1*3)*(8*8)=49152. Therefore, it can be seen that the use of depthwise separable convolution in this application can significantly reduce the computational cost.

[0046] By using a second convolution kernel to combine the output across channels through a second convolution, each channel's convolution has a high non-linear expressive power, which can increase the non-linear expressive power of the network model, thereby helping to ensure the accuracy of image processing results.

[0047] Furthermore, by creating a fully connected layer to transform a large feature matrix corresponding to several intermediate feature maps into a smaller low-rank matrix, the number of parameters that need to be processed can be significantly reduced. Sharing parameters also reduces redundancy, allowing the model to learn only a small number of parameters during fine-tuning, thereby reducing computational cost. The main reasons for this are as follows: 1. Reduce the number of parameters: For example, for the weight matrix W∈R corresponding to the original image m×n If all parameters of the weight matrix W are updated directly, m×n parameters need to be updated; however, step S3 can approximate the weight matrix W into a low-rank matrix, equivalent to two matrices A∈R. m×r and B∈R r×n The product of r and min(), where r ≪ min(m, n), min() is the minimum function. Here, the number of parameters becomes m×r + r×n = r(m+n). Since r is much smaller than m and n, the number of parameters is greatly reduced.

[0048] II. Parameter Sharing: A∈R m×r and B∈R r×nThe product can be viewed as capturing the main information of weight updates through a low-dimensional space (rank r). This decomposition assumes that the main changes in weight updates are concentrated in the low-dimensional subspace, rather than the entire high-dimensional space. Therefore, only the parameters in the low-dimensional space need to be learned, without updating the entire high-dimensional matrix. This idea of ​​parameter sharing allows the network model to learn only a small number of parameters during fine-tuning, thereby reducing computational cost.

[0049] III. Simplified Calculation Process: Through the low-rank transformation in step S3, the weight matrix W∈R can be transformed. m×n Complex matrix operations are simplified to two simple matrices A∈R m×r and B∈R r×n The combination of operations results in a smaller rank matrix after decomposition, leading to lower computational complexity and storage requirements.

[0050] Therefore, in image processing such as image compression and feature extraction, low-rank decomposition-based convolution can be used to remove redundant information from images, reducing the number of parameters and computational load. By converting the image matrix into a low-rank matrix, image data can be effectively compressed while retaining the main information, improving the nonlinear expressive power of the convolution kernel and ensuring the accuracy of image processing results.

[0051] In one example, the fully connected layer can use a nonlinear activation function to obtain the low-rank matrix. This adds nonlinearity to the network model, so that no matter how many layers the network model has, the output of any subsequent layer is a nonlinear combination of the input variables of that layer, and the input variables of the next layer are also a nonlinear combination. This is equivalent to the network model being a multi-layer nonlinear model, which can better learn and process the features of complex data in images.

[0052] In practical scenarios, the nonlinear activation function includes, but is not limited to, at least one of the linear rectified ReLU function, the logical Sigmoid function, and the hyperbolic tangent Tanh function.

[0053] Figure 5 This is a flowchart illustrating another embodiment of an image processing method based on multi-channel convolution. Figure 5 As shown, the method includes at least the following steps S1 to S5: S1: In response to inputting the original image into the convolutional network model, obtain the number of input channels of the original image; S2: Set a first convolution kernel equal to the number of input channels for the convolutional network model, and apply a first convolution kernel independently to each input channel to perform the first convolution, so as to output several intermediate feature maps; S3: Create a fully connected layer for the convolutional network model, and convert the feature matrices corresponding to several intermediate feature maps into a low-rank matrix through the fully connected layer; S31: Create an activation layer for the convolutional network model; S32: The low-rank matrix is ​​convolved by a non-linear activation function through the activation layer to output several activation feature maps; S4: Using a second convolution kernel of size 1*1, the convolutional network model performs a second convolution on the several activation feature maps to output several output feature maps; S5: Obtain a target image based on several output feature maps.

[0054] For Figure 5 and Figure 1 Steps with the same reference numerals and the same technical features can be implemented using the same methods, and will not be described in detail here. Although step designations such as S1 and S2 are used herein, their purpose is to more clearly and concisely describe the corresponding content, and they do not constitute a substantial limitation on the order. In specific implementation, those skilled in the art may execute some steps with later numbers first, and then execute some steps with earlier numbers. For example, the aforementioned step S31 may be executed first, followed by steps S3 and S32 to S5, but all of these should be within the protection scope of this application.

[0055] This example is equivalent to adding a convolutional layer, or activation layer, after the aforementioned fully connected layer. This activation layer uses a non-linear activation function to convolve the low-rank matrix, which can add non-linear characteristics to the network model. This makes the output of any subsequent layer a non-linear combination of the input variables of that layer, and the input variables of the next layer are also non-linear combinations. This means that the network model is a multi-layer non-linear model, which can better learn and process the features of complex data in images.

[0056] Based on the activation layer, further combined Figure 6 As shown, step S4 of this embodiment performs a second convolution on the low-rank matrix using the following relation: Relation 3 Relation 4 Relation 5 in, This represents the one-dimensional eigenvector corresponding to the low-rank matrix. This represents a one-dimensional feature vector corresponding to the plurality of activated feature maps. This indicates the nonlinear activation function used in the activation layer. This represents the matrix transpose function.

[0057] Based on the above relation 3, after passing through the fully connected layer, the corresponding output r low-rank feature maps can be regarded as a one-dimensional r*1 feature vector. Next, to ensure nonlinearity, these r low-rank feature maps are input into the activation layer. Based on the above relation 4, a third convolution is performed to obtain r new feature maps, which are the one-dimensional feature vectors corresponding to the aforementioned activation feature maps. Then, after pointwise convolution through the above relation 5, these activation feature maps can be linearly transformed into n new feature maps, which are the output feature maps of step S5.

[0058] Combination Figure 6 As shown, each circle in the activation layer represents the parameter corresponding to each convolution kernel. In one example, the number of nodes used in the activation layer is equal to the number of nodes used in the fully connected layer. Therefore, in this example, a convolution kernel can be independently applied to each output channel of the fully connected layer to perform convolution operations to output the feature map, that is, cross-channel combination is not performed, which can further reduce the amount of computation and parameters when the convolutional network model processes images.

[0059] Figure 7 This is a flowchart illustrating another embodiment of the image processing method based on multi-channel convolution according to this application. Figure 7 As shown, the method includes at least the following steps S1 to S5: S1: In response to inputting the original image into the convolutional network model, obtain the number of input channels of the original image; S2: Set a first convolution kernel equal to the number of input channels for the convolutional network model, and apply a first convolution kernel independently to each input channel to perform the first convolution, so as to output several intermediate feature maps; S3: Create a fully connected layer for the convolutional network model, and convert the feature matrices corresponding to several intermediate feature maps into a low-rank matrix through the fully connected layer; S41: Divide a number of low-rank matrices into a number of subgroups, the number of which is the result of an integer division operation between the number of output feature maps and the parallelism of the subgroups; S42: The convolutional network model uses a second convolutional kernel of size 1*1 to perform a second convolution on each subgroup to output several output feature maps; S5: Obtain a target image based on several output feature maps.

[0060] For Figure 7 and Figure 1 Steps with the same reference numerals and the same technical features can be implemented using the same methods, and will not be described in detail here.

[0061] This example is equivalent to introducing grouped convolution into depthwise separable convolution. Grouped convolution divides the relevant feature maps into multiple subgroups in the channel dimension and performs convolution operations independently within each subgroup. Specifically, grouped convolution reduces the computational complexity to 1 / g of the original by dividing the corresponding input channels and output channels into g non-overlapping subgroups, with each subgroup only processing 1 / g of the channel data, thus improving the running efficiency of the model by reducing the number of parameters and the amount of computation in the convolution operation.

[0062] Different from the problem that there is no information transmission between channels in traditional grouped convolution, this example uses a fully connected layer in combination with grouped convolution to increase the parallelism between each subgroup. For example, according to the parallelism of the several subgroups and the following relational expression, a fully connected layer is created for the convolutional network model: r < m, r < n, r is x * k, where x is an integer and x is divisible by both m and n, and k is the parallelism of the several subgroups, which can be preset according to, for example, the precision requirements of the device applicable to the model.

[0063] In a specific implementation, first, a depthwise convolutional kernel, that is, the first convolutional kernel, is established according to the number of input channels. Then, a fully connected layer is created according to the parallelism k, and the way to take the rank (i.e., rank) of this fully connected layer follows the above relational expression, which can solve the problem of no inter-channel communication in subsequent grouped convolution. Then, grouped pointwise convolution is created according to the parallelism k, where g = n / / k for the grouped convolution, and g represents the number of subgroups, that is, the number of subgroups is equal to the result of the integer division operation between the number of output channels and the parallelism.

[0064] For example, in a model scenario where it is required to design a model with 128 input channels and 256 output channels, the parallelism preset according to the device applicable to the model is 32. The specific implementation method is as follows: First, a depthwise convolutional kernel is established according to the number of input channels (i.e., 128). Then, a fully connected layer is created according to the parallelism k (i.e., 32), and the way to take the rank of this fully connected layer is less than 128 and is x times of 32, that is, the rank can take values of 32 or 64. In this example, the rank can take the value of 64. Then, grouped pointwise convolution is created according to the parallelism k, where g = (256 / / 32) = 8, that is, the pointwise convolution adopts a grouping method of 8, and finally, several output feature maps are output. Based on this, it can be regarded that 4 channels rc0~rc3 of the fully connected layer will be mapped to 32 output channels oc0~oc31, 4 channels rc4~rc8 of the fully connected layer will be mapped to 32 output channels oc32~oc63, and so on, until 4 channels rc60~r6c3 of the fully connected layer will be mapped to oc224~oc255 of 32 output channels, totaling 8 subgroups. The introduction of the fully connected layer can force inter-channel communication while reducing the number of parameters, thereby eliminating the isolation of information between input channels.

[0065] This application embodiment also provides a storage medium storing an image processing program, which is essentially a computer program, and when executed by a processor, implements the steps of an image processing method based on multi-channel convolution as in any example.

[0066] The storage medium includes, but is not limited to, any one of read-only memory (ROM), random access memory (RAM), magnetic disk, and optical disk.

[0067] Since the program stored in the storage medium can execute the steps in the image processing method of any embodiment provided in this application, the beneficial effects that the multi-channel convolution-based image processing method of any of the foregoing embodiments can achieve can be realized. For details, please refer to the foregoing embodiments, which will not be repeated here.

[0068] This application also provides an image processing device or chip, including a memory and a processor. The memory stores an image processing program, which, when executed by the processor, implements the steps of the multi-channel convolution-based image processing method of any of the foregoing embodiments. Alternatively, the image processing device or chip may be provided with a storage medium as shown in the above example, and the processor loads the storage medium to execute the steps of the multi-channel convolution-based image processing method of any of the foregoing embodiments, thereby achieving the beneficial effects achievable by the multi-channel convolution-based image processing method of the corresponding embodiments.

[0069] The above are only some embodiments of this application and do not limit the patent scope of this application. For those skilled in the art, any equivalent structural transformations made using the content of this specification and drawings are similarly included within the patent protection scope of this application.

[0070] Although this document uses terms such as "first," "second," etc., to describe various types of information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. Furthermore, the singular forms "a," "an," and "the" are intended to also include the plural forms. The terms "or" and "and / or" are interpreted as inclusive, or meaning either one or any combination thereof. Exceptions to this definition only arise when combinations of elements, functions, steps, or operations are inherently mutually exclusive in some way.

Claims

1. An image processing method based on multi-channel convolution, characterized in that, include: In response to inputting the original image into the convolutional network model, the number of input channels of the original image is obtained; The convolutional network model is set with a first convolutional kernel equal to the number of input channels, and a first convolutional kernel is applied independently to each input channel to perform the first convolution, so as to output several intermediate feature maps. A fully connected layer is created for the convolutional network model, and the feature matrix corresponding to the several intermediate feature maps is converted into a low-rank matrix through the fully connected layer. An activation layer is created for the convolutional network model, the number of nodes in the activation layer being equal to the number of nodes in the fully connected layer; The activation layer employs a nonlinear activation function to perform a third convolution on the low-rank matrix, thereby outputting several activation feature maps. The convolutional network model uses a second convolution kernel of size 1*1 to perform a second convolution on the plurality of activation feature maps to output a plurality of output feature maps. This includes: dividing the low-rank matrix into several subgroups, the number of which is the integer division result between the number of output feature maps and the parallelism of the subgroups; and performing a second convolution on each subgroup to output a plurality of output feature maps. The parallelism between the fully connected layer and the plurality of subgroups satisfies the relationship r < m, r < n, where r is x*k, x is an integer divisible by both m and n, where r represents the rank of the low-rank matrix, m represents the number of intermediate feature maps output by the first convolution, n represents the number of output feature maps output by the second convolution, and k is the parallelism of the plurality of subgroups. The second convolution on the plurality of activation feature maps is performed according to the following relationship: V1* Math(MAT m,r ) = V mid V’ mid = ACT(Transpose(V mid )) V' mid Matmul(MAT r,n ) = V2 V1 is the one-dimensional feature vector corresponding to m intermediate feature maps, V2 is the one-dimensional feature vector corresponding to n output feature maps, and Matmul() is a matrix multiplication function that is linear. m,r MAT is a one-dimensional eigenvector matrix based on m intermediate feature maps and the rank. r,n V is the one-dimensional feature vector matrix corresponding to the rank and n output feature maps; mid V' represents the one-dimensional eigenvector corresponding to the low-rank matrix. mid represents the one-dimensional feature vector corresponding to the plurality of activated feature maps, ACT() represents the non-linear activation function used in the activation layer, and Transpose() represents the matrix transpose function; A target image is obtained based on the aforementioned output feature maps.

2. The method according to claim 1, characterized in that, The step of converting the feature matrices corresponding to the intermediate feature maps into a low-rank matrix through the fully connected layer includes: The rank is obtained based on the aforementioned intermediate feature maps and the following relational formula; V1* Matmul(MAT m,n ) = V2 FOOD m,r = Food mul(FOOD r,n ) ≈ FOOD m,n Among them, MAT m,n Let MAT be a one-dimensional feature vector matrix based on m intermediate feature maps and n output feature maps, and MAT m,n It is a non-full rank matrix; The low-rank matrix is ​​obtained based on the rank.

3. An image processing device, characterized in that, It includes a processor and a memory, wherein the memory stores an image processing program, and when the image processing program is executed by the processor, it implements the steps of the image processing method based on multi-channel convolution as described in claim 1 or 2.

4. A storage medium, characterized in that, The system contains a computer program that, when executed by a processor, implements the steps of the image processing method based on multi-channel convolution as described in claim 1 or 2.

Images