UNet model-based image processing method and device, and storage medium

Abstract

The invention provides an image processing method and device based on a UNet model, and a storage medium. In the encoding stage, in response to inputting an original image into the UNet model, input features of the original image are obtained; performing jump connection on the input features; performing first channel rearrangement on the input features to realize down-sampling and output down-sampling features; inputting the down-sampling feature into an encoder for convolution to obtain an encoding feature; obtaining coding output features according to the coding features; in the decoding stage, residual features are obtained based on jump connection; inputting the coding output features into a decoder for convolution; performing second channel rearrangement on the convolution result to realize up-sampling and output decoding features; and obtaining a target image according to the residual feature and the decoding feature. On the basis, the calculation amount and the parameter amount when the UNet model processes the image can be reduced, and the image processing effect is ensured.

Description

Technical Field

[0001] The present application relates to the fields of image processing technology and deep learning technology, and specifically to an image processing method, device, and storage medium based on a UNet (U-shaped network) model. Background Art

[0002] Compared to traditional ISP (Image Signal Processing) pipeline (ISP) denoising algorithms, artificial intelligence (AI) technologies, such as convolutional neural networks (CNNs), can significantly improve image denoising quality. Consequently, they have seen significant growth in the image processing field in recent years and are widely used in computer vision, with successful applications in areas such as image recognition, image compression, and object detection. However, existing CNN models (also known as "neural network models," "convolutional models," "network models," "neural models," or simply "models") contain a large number of redundant parameters, resulting in excessive computational complexity and excessive consumption of computing resources, both in terms of computational effort and parameter requirements. Consequently, these algorithms face numerous challenges in practical implementation, particularly on edge devices. For example, in scenarios like intelligent transportation, the high computational resource consumption makes real-time image display a significant challenge. Numerous solutions have been proposed, such as pruning and quantizing CNN models. However, these solutions are often performed manually after model training, essentially sacrificing performance to alleviate computational resource constraints, resulting in poor image processing results. Summary of the Invention

[0003] In view of this, the present application provides an image processing method, device, and storage medium based on the UNet model, which can improve the problem of large computational complexity and large number of parameters when processing images using the traditional UNet model.

[0004] This application provides an image processing method based on the UNet model, including: Coding phase: In response to inputting an original image into a UNet model, obtaining input features of the original image; Performing a skip connection on the input features; Performing a first channel rearrangement on the input features to achieve downsampling and outputting downsampled features; Inputting the downsampled features into the encoder for convolution to obtain encoded features; Obtaining a coding output feature according to the coding feature; Decoding stage: Obtaining residual features based on the skip connection; Inputting the encoded output features into a decoder for convolution; The convolution result is reordered in the second channel to achieve upsampling and output the decoded features; A target image is obtained according to the residual features and the decoded features.

[0005] Optionally, the encoder includes multiple convolutional layers, and the encoding features are obtained after the multiple convolutional layers; the convolution performed by each convolutional layer includes: Pre-set weight coefficients k and 1-k; Input the downsampled features into the encoder for a first convolution; Perform the third convolution on the result of the first convolution; Input the downsampled features into the encoder for a second convolution; The product of the result of the third convolution and the weight coefficient k and the product of the result of the second convolution and the weight coefficient 1-k are input into the next convolution layer.

[0006] Optionally, the first convolution and the third convolution both use a convolution kernel of size 3*3, and the second convolution uses a convolution kernel of size 3*3.

[0007] Optionally, in the encoding stage, the method further includes: Performing wavelet transform on the input features to obtain high-frequency features and low-frequency features; Performing skip connections on the high-frequency features; In the decoding stage, the method further includes: Obtaining the high-frequency features based on the skip connection; Performing convolution on the high-frequency features; The convolution result is rearranged in the third channel to output the wavelet residual feature; The obtaining of a target image according to the residual feature and the decoding feature includes: A target image is obtained according to the wavelet residual feature, the residual feature and the decoding feature map.

[0008] Optionally, in the encoding stage, the method further includes: Performing convolution on the low-frequency features; The obtaining of the encoding output feature according to the encoding feature includes: According to the convolution result of the coding feature and the low-frequency feature, a coding output feature is obtained.

[0009] Optionally, the first channel rearrangement includes: Convert the input feature tensor shape [B1, C1, H1, W1] to a downsampled feature of the tensor shape [B1, r1*r1*C1, H1 / r1, W1 / r1], where B1 is the number of groups, C1 is the number of channels, H1*W1 is the channel size, r1 is a positive integer divisible by H1 and W1, and r1 ≥ 2; The second channel rearrangement or the third channel rearrangement includes: The convolution result tensor shape [B2, r1*r1C2, H2 / r1, W2 / r1] is transformed into features of tensor shape [B2, C2, H2, W2]; where B2 is the number of groups, C2 is the number of channels, H2*W2 is the channel size, and r1 is a positive integer divisible by H2 and W2.

[0010] Optionally, the convolution includes: In response to inputting the corresponding feature into the UNet model, obtaining the corresponding number of input channels; Setting a convolution kernel equal to the number of input channels for the UNet model, and independently applying a convolution kernel to each input channel to perform convolution to output intermediate features; Creating a fully connected layer for the UNet model, and converting the feature matrix corresponding to the intermediate features into a low-rank matrix through the fully connected layer; The low-rank matrix is ​​convolved using a convolution kernel of size 1*1 through the UNet model.

[0011] Optionally, perform convolution on the low-rank matrix, including: Divide the low-rank matrix into a plurality of subgroups, where the number of the subgroups is the integer division result between the number of output channels and the degree of parallelism of the plurality of subgroups; Convolution is performed based on each subgroup.

[0012] The present application provides an image processing device, comprising a processor and a memory, wherein an image processing program is stored in the memory. When the image processing program is executed by the processor, the steps of the image processing method based on the UNet model as described in any one of the above items are implemented.

[0013] The present application provides a storage medium storing a computer program. When the computer program is executed by a processor, the steps of the image processing method based on the UNet model as described in any one of the above items are implemented.

[0014] As mentioned above, the present application realizes downsampling in the encoding stage and upsampling in the decoding stage through channel rearrangement. Channel rearrangement divides the input features into multiple groups in the encoding stage, and each group performs convolution operations independently. Since cross-channel combination calculations are not performed, the amount of calculation and parameters when the UNet model processes images can be reduced. In addition, channel rearrangement disrupts the order of each group of features, and the data information of different channels is fused and reorganized in the decoding stage, which can increase the learning ability and feature representation ability of the UNet model, better capture complex features, and thus help ensure the image processing effect. In summary, the present application can significantly reduce the amount of calculation and the number of parameters while ensuring the image processing effect. In other words, the UNet model of the present application can be made deeper with the same number of parameters, or have a faster running speed and training speed with the same model structure, which is beneficial to improve the training efficiency and generalization ability of the network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 1 is a flowchart of an image processing method based on a UNet model according to an embodiment of the present application; Figure 2 This is a flow chart of image processing performed during the encoding phase of this application; Figure 3 This is a flow chart of image processing performed during the decoding phase of the present application; Figure 4 It is a schematic diagram of the first channel rearrangement of the present application; Figure 5 This is a schematic diagram of the convolution of fine-grained receptive field in this application; Figure 6 It is a schematic diagram of the second channel rearrangement performed in this application; Figure 7 1 is a flowchart of an image processing method based on a UNet model according to another embodiment of the present application; Figure 8 It is a schematic diagram of wavelet transform performed in this application; Figure 9 1 is a flow chart of a convolution method according to an embodiment of the present application; Figure 10 This is a schematic diagram of the application performing convolution on the input channels of the corresponding features; Figure 11 This is a schematic diagram of convolution based on three convolutional layers in this application; Figure 12 This is a schematic diagram of the low-rank decomposition of an m*n matrix by the fully connected layer of this application; Figure 13 This is a schematic diagram of convolution based on four convolutional layers in this application. DETAILED DESCRIPTION

[0016] To address the aforementioned issues in the prior art, this application provides an image processing method, device, and storage medium based on the UNet model. These several protected claims are based on the same concept and address the same or similar principles. The implementation methods of each protected claim can be referenced across them, and any repetitions will not be repeated here.

[0017] In the solution of the present application, downsampling in the encoding stage and upsampling in the decoding stage are achieved through channel rearrangement. Channel rearrangement divides the input features into multiple groups in the encoding stage, and each group performs convolution operations independently, thereby reducing the amount of calculation and parameters when the UNet model processes images. In addition, channel rearrangement disrupts the order of each group of features, and fuses and reorganizes data information from different channels in the decoding stage, thereby increasing the learning ability and feature representation ability of the UNet model and better capturing complex features.

[0018] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly described below in conjunction with specific embodiments and corresponding drawings. Obviously, the embodiments described below are only some of the embodiments of this application, not all of them. Unless there is a conflict, the following embodiments and their technical features can be combined with each other and also belong to the technical solutions of this application.

[0019] Figure 1 The figure is a flowchart of an image processing method based on a UNet model according to an embodiment of the present application. The image processing method based on a UNet model may also be referred to as an "image processing method" or "method." Each step may be performed by an appropriate image processing device, such as a mobile phone, or a storage medium, processor, or controller with relevant image processing capabilities.

[0020] Combine Figures 1 to 3 As shown, the method includes an encoding phase and a decoding phase which are performed successively.

[0021] The encoding stage includes the following steps S11 to S15: S11: In response to inputting the original image into the UNet model, obtaining input features of the original image.

[0022] The input features of the original image can be obtained based on the input channels (in_channels), which refer to the components of the original image in the depth dimension. For example, the number of input channels can be obtained based on the type of the original image. If the original image is a color image, that is, an RGB type, the number of input channels can be determined to be 3, corresponding to red (R), green (G), and blue (B), respectively. Each channel stores the pixel intensity of the corresponding color; based on this, the features corresponding to the original image in each channel, such as edges, textures, corners, etc., are obtained. These features are the input features, and the input features can be regarded as existing in the form of feature maps in the UNet model, that is, the original image is split into feature maps under these three channels. The "certain features" in this application can also be called "certain feature maps"; if the original image is a gray image, the number of input channels can be determined to be 1, representing the brightness value of the pixel, thereby obtaining a pair of input feature maps under this channel.

[0023] S12: Perform skip connections on the input features.

[0024] The so-called skip connection refers to the direct transfer of input features from the shallow network layer of the UNet model (i.e., the network layer corresponding to the encoding stage) to the deep network layer of the decoding stage (i.e., the network layer corresponding to the decoding stage), such as Figure 3 As shown, in the encoding stage, the deep layer of the network can use the input feature as a residual feature to facilitate fusion with deep output features such as decoding features to obtain the final output feature.

[0025] S13: Rearrange the first channel of the input features to achieve downsampling and output the downsampled features.

[0026] The core principle of channel rearrangement is to fuse and reorganize data information (i.e., features) from different channels. The first channel rearrangement involves converting the input feature tensor shape [B1, C1, H1, W1] into downsampled features of the shape [B1, r1*r1*C1, H1 / r1, W1 / r1], where B1 is the number of groups, C1 is the number of channels, H1*W1 is the channel size, and r1 is a positive integer divisible by H1 and W1, and r1 ≥ 2.

[0027] Combine Figure 4As shown, the input features correspond to two feature maps chY1 and chY2, and the corresponding tensor shapes are both [1, 2, 4, 4]. Taking r1 = 2 times downsampling as an example, the number of channels obtained after the first channel is rearranged is r1*r1*C1 = 8, and the size of each channel is 2*2. That is, the downsampled features are represented by a set of 8 downsampled feature maps, labeled ch20 to ch27, and the tensor shape of each downsampled feature map is [1, 8, 2, 2]. Specifically, the data information of feature map chY1 (that is, each small box and its represented eigenvalue) is reorganized to obtain 4 downsampled feature maps ch20 to ch23, and the data information of feature map chY2 is reorganized to obtain the remaining 4 downsampled feature maps ch24 to ch27. Among them, the specific method of fusion and reorganization can be adapted according to actual needs.

[0028] S14: Input the downsampled features into the encoder for convolution to obtain encoded features.

[0029] The encoder may include at least Conv (Convolution), Norm (Normalization), and Act (Activation). The convolution layer is used to perform convolution operations on the downsampled features. The normalization layer is used to adjust the output of the convolution layer so that it has zero mean and unit variance to accelerate the training process and improve the generalization ability of the model. The activation function is applied to the convolution operation of the convolution layer, so that the entire UNet model can learn the nonlinear relationship between feature channels, so as to better fit complex functions. In actual scenarios, the activation function includes but is not limited to at least one of the linear rectification ReLU function, the logistic Sigmoid function, and the hyperbolic tangent Tanh function. ‌

[0030] The downsampled features are convolved in the encoder through the cooperation of these three. It should be noted that the encoder includes multiple convolutional layers and is not limited to Figure 2 The three convolutional layers shown in the figure each include Conv, Norm, and Act, and the corresponding convolution is performed by the three in cooperation. The specific principles and processes of convolution can be referred to in the art. The convolution result after passing through multiple convolutional layers in sequence is the encoding feature.

[0031] The convolution principle and process adopted can be referred to the existing technology in this field and will not be described in detail here. The features of the image in different aspects, such as edges, textures, corners and other low-level features, are extracted respectively. A convolution kernel in each convolution layer can extract a feature and obtain a new feature map. Finally, multiple feature maps are generated through the multi-layer convolution of the encoder. These feature maps represent multiple encoding features.

[0032] The noise contained in the input features of the original image mainly includes Poisson noise, Gaussian noise and other noises. Among them, the parameters of Poisson noise are mainly related to the sensor type used to obtain the original image and the pixel values ​​of the real clean image, and will appear as medium-low frequency noise or high-frequency noise according to the size of the ideal image (that is, the ideal noise-free image used as the training target). The parameters of Gaussian noise are mainly related to the sensitivity and sensor type, and will always appear as high-frequency noise according to the size of the ideal image. Therefore, the noise of the input features of the original image includes low-frequency noise and high-frequency noise. The convolution method used by each convolution layer in the traditional UNet model is fixed. For example, a "5*5 superimposed on 5*5+3*3" convolution kernel is generally used for convolution operations, resulting in a larger receptive field. When faced with high-frequency noise, this will cause the details of the high-frequency noise to be easily lost, and complex features cannot be well captured, which is not conducive to image processing effects.

[0033] Based on this, the present application can set a fine-grained receptive field for the convolution layer of the encoder. Specifically, for the traditional convolution operation of the convolution kernel of "5*5 superimposed on 5*5+3*3", the present application can be regarded as performing three convolutions on each convolution layer, namely, the first convolution and the third convolution based on two convolution kernels 3*3, and the second convolution based on the convolution kernel 3*3, that is, the first convolution and the second convolution performed by two 3*3 convolution kernels are superimposed to produce a convolution equivalent to that based on a 5*5 convolution kernel, then combined with Figure 5 As shown in the figure, the convolution performed by each convolutional layer can include the following steps Step 1 to Step 5: Step 1. Pre-set weight coefficients k and 1-k; Step 2. Input the downsampled features into the encoder for the first convolution; Step 3. Perform the third convolution on the result of the first convolution; Step 4. Input the downsampled features into the encoder for the second convolution; Step 5. Input the product of the result of the third convolution and the weight coefficient k and the product of the result of the second convolution and the weight coefficient 1-k into the next convolution layer.

[0034] This application uses weight coefficients k and 1-k to control the ratio of 5*5 convolution kernel and 3*3 convolution kernel flowing into each layer of the encoder, so as to achieve a finer-grained receptive field, avoid the loss of details of high-frequency noise, better capture complex features, and facilitate image processing effects.

[0035] S15: Obtain encoding output features according to the encoding features.

[0036] The decoding stage includes the following steps S21 to S24: S21: Obtain residual features based on skip connections.

[0037] The residual is the difference between the observed value and the estimated value calculated based on the UNet model. The calculation principle and process of the residual can be referred to the existing technology in this field.

[0038] S22: Input the encoded output features into the decoder for convolution.

[0039] The output features of the encoding stage, that is, the encoded output features, are used as input features of the decoder. The decoder can also include at least Conv, Norm and Act. The encoded output features are convolved through the three in the decoder. The decoder also includes multiple convolutional layers and is not limited to Figure 3 The three convolutional layers shown in the figure can be called "encoded intermediate features" after the convolution results of multiple convolutional layers in sequence.

[0040] Among them, the decoder can also use the aforementioned steps Step 1 to Step 5 for convolution. The specific principles and processes are not repeated here. A finer-grained receptive field can also be achieved here. It should be understood that any convolution operation of this application can be performed in a manner similar to the aforementioned Step 1 to Step 5.

[0041] S23: Perform a second channel rearrangement on the convolution result to achieve upsampling and output the decoding features.

[0042] S24: Obtain a target image based on the residual features and the decoded features.

[0043] The second channel rearrangement includes: changing the tensor shape of the convolution result [B2, r1*r1C2, H2 / r1, W2 / r1] to the feature of the tensor shape [B2, C2, H2, W2], where B2 is the number of groups, C2 is the number of channels, H2*W2 is the channel size, r1 is a positive integer divisible by H2 and W2, and r1 ≥ 2.

[0044] Combine Figure 6As shown, the number of channels corresponding to the feature map output by the decoder is 8, that is, it is expressed as a group of 8 feature maps ch10~ch17, and the tensor shape of each feature is [1,8,2,2]. Taking r1=2 times upsampling as an example, the number of channels obtained after the second channel rearrangement is C2=2, and the size of each channel is 4*4, that is, the upsampling feature is expressed as a group of 2 upsampling feature maps, labeled chX1 and chX2, and the tensor shape of each upsampling feature map is [1,2,4,4]. Specifically, the data information of the feature maps ch10~ch13 (that is, each small box and the eigenvalue it represents) is fused and reorganized to obtain one of the upsampling feature maps chX1, and the data information of the feature maps ch14~ch17 are fused and reorganized to finally obtain another upsampling feature map chX2. Among them, the specific method of fusion and reorganization can be determined according to the adaptability of actual needs. For example, the rearrangement of the second channel can be regarded as the inverse reorganization of the eigenvalues ​​of the first channel rearrangement, and the eigenvalue of another feature map is inserted in sequence between the two adjacent eigenvalues ​​of the first feature map ch10, and so on, until the eigenvalues ​​of other feature maps are inserted between the two adjacent eigenvalues ​​of each feature map.

[0045] In step S24, an image obtained by superimposing the residual features and the decoded features is used as the target image.

[0046] As mentioned above, the present application realizes downsampling in the encoding stage and upsampling in the decoding stage by channel rearrangement. Channel rearrangement divides the input features into multiple groups (such as the aforementioned Figure 4 As shown in the figure, the eight input feature maps ch10 to ch17 are convolved independently for each set of corresponding features. Since no cross-channel combined calculations are performed, the amount of computation and parameters used by the UNet model to process images can be reduced. In addition, channel rearrangement disrupts the order of each set of features. Fusing and reorganizing data information from different channels during the decoding phase can increase the learning and feature representation capabilities of the UNet model, better capturing complex features and thus ensuring image processing effects. In summary, this application can significantly reduce the amount of computation and the number of parameters while ensuring image processing effects. In other words, the UNet model of this application can be made deeper with the same number of parameters, or have a faster running speed and training speed with the same model structure, thereby improving the training efficiency and generalization ability of the network model.

[0047] Figure 7 This is a flowchart of an image processing method based on a UNet model according to another embodiment of the present application. Figure 2 、 Figure 3 and Figure 7 As shown, the method includes the following steps: The encoding stage includes the following steps S11 to S15: S11: In response to inputting the original image into the UNet model, obtaining input features of the original image; S12: Perform skip connection on input features; S13: Rearrange the first channel of the input feature to achieve downsampling and output the downsampled feature; S14: Input the downsampled features into the encoder for convolution to obtain encoded features; S150: Perform wavelet transform on the input features to obtain high-frequency features and low-frequency features; S151: perform skip connection on high-frequency features; S15: Obtain encoding output features according to the encoding features.

[0048] The decoding stage includes the following steps S21 to S24: S21: Obtain residual features based on skip connections; S22: Input the encoded output features into the decoder for convolution; S23: Rearrange the second channel of the convolution result to achieve upsampling and output decoding features; S240: Obtain high-frequency features based on skip connections; S241: Convolution of high-frequency features; S242: performing a third channel rearrangement on the convolution result to output a wavelet residual feature; Accordingly, the aforementioned step S24 can be expressed as: obtaining the target image according to the wavelet residual feature, the residual feature and the decoding feature map.

[0049] Compared with the above Figure 1 In the example shown, this example further includes S150 and S151 in the encoding stage, and further includes S240, S241 and S242 in the decoding stage.

[0050] For Figure 7 and Figure 1 The steps with the same number and the same technical features can be executed in the same manner, and this application will not describe them one by one. Although step codes such as S11 and S12 are used herein, their purpose is to express the corresponding content more clearly and concisely, and do not constitute a substantial limitation on the order. When implementing the steps, those skilled in the art may first execute some steps with later numbers, and then execute some steps with earlier numbers, for example, they may first execute the aforementioned step S240 and then execute step S21, but these should all be within the scope of protection of this application.

[0051] In step S150, wavelet transform can be understood as extracting features corresponding to low-frequency components and features corresponding to high-frequency components from input features, that is, obtaining Figure 8 The low-frequency feature map and high-frequency feature map shown in the figure can be expressed by tensor shapes as follows: the tensor shape [B0, 4*C0, H0, W0] corresponding to the input feature is changed to the tensor shape [B0, C0, (H0 / 2), (W0 / 2)] corresponding to the low-frequency feature, and the tensor shape [B0, 3*C0, (H0 / 2), (W0 / 2)] corresponding to the high-frequency feature. Figure 8 In the example shown, the tensor shape of the input feature map is [1, 1, 4, 4], the tensor shape of the low-frequency feature map obtained after the wavelet transform is [1, 1, 2, 2], and the tensor shape of the high-frequency feature map is [1, 3, 2, 2].

[0052] Step S241 can be performed as follows: Figure 5 The fine-grained receptive field performs convolution on high-frequency features.

[0053] In step S242, the third channel rearrangement includes changing the tensor shape of the decoder convolution result from [B2, r1*r1C2, H2 / r1, W2 / r1] to [B2, C2, H2, W2]. The principle and process of this third channel rearrangement can be found in the second channel rearrangement described above and will not be repeated here.

[0054] In the aforementioned Figure 1 Based on the previous example, this example adds high-frequency noise to the skip connection and adds the residual features (i.e., wavelet residual features) obtained based on the high-frequency noise to the generation of the target image. This is equivalent to adding more dimensional feature factors to the generation of the target image, which can achieve better image processing effects.

[0055] Please also refer to Figure 2 、 Figure 3 and Figure 7 As shown, in the encoding stage, the following steps may also be included: S152: Convolution of low-frequency features; Accordingly, the aforementioned step S15 can be expressed as: obtaining a coding output feature according to the convolution result of the coding feature and the low-frequency feature.

[0056] This example incorporates features derived from low-frequency noise into the generation of the target image, which is equivalent to adding more dimensional feature factors to the generation of the target image and can also achieve better image processing results.

[0057] In any of the above examples, any convolution operation can be performed by Figure 9 Execute as shown.

[0058] like Figure 9 As shown, the convolution may include the following steps S1 to S4: S1: In response to inputting the corresponding feature into the UNet model, obtaining the corresponding number of input channels.

[0059] The input channel (in_channels) of the corresponding feature refers to the component of the feature (which can be regarded as "input data") in the depth dimension.

[0060] S2: Set the convolution kernel of the UNet model to the number of input channels, and apply a convolution kernel to each input channel independently to perform convolution to output intermediate features.

[0061] In the UNet model, a convolution kernel is a learnable weight matrix used to extract local features from the input data. Acting as a local feature detector, a convolution kernel can identify low-level features such as edges and textures, or combine these low-level features into high-level semantic features. Using different convolution kernels, a variety of features can be extracted, thereby constructing the hierarchical feature representation of the UNet model.

[0062] Combine Figure 10 As shown, taking the corresponding feature as RGB type as an example, where each small box represents the pixel intensity of a pixel of the corresponding color, the number of input channels is 3, and the feature map corresponding to the feature is decomposed into an image corresponding to R, an image corresponding to G, and an image corresponding to B. Here, three convolution kernels, also known as depth convolution kernels, that is, three filters (Filter), can be used. The size of each convolution kernel can be 3*3. Each convolution kernel moves on the corresponding decomposed image for convolution. The convolution principle and process adopted by the convolution can refer to the existing technology in this field and will not be repeated here. Features of the feature map in different aspects, such as low-level features such as edges, textures, and corners, are extracted respectively. Each convolution kernel can extract a feature and obtain a new feature map. Finally, multiple feature maps are generated through convolution. These feature maps are called "intermediate feature maps".

[0063] A convolution operation is performed independently on each input channel using a convolution kernel to output the intermediate feature map, so that spatial features can be extracted on each channel. Compared with traditional standard convolution, step S2 does not perform cross-channel combination, which can reduce cross-channel combination calculations, thereby significantly reducing the amount of calculation and parameters when the UNet model processes images.

[0064] S3: Create a fully connected layer for the UNet model, and convert the feature matrix corresponding to the intermediate features into a low-rank matrix through the fully connected layer.

[0065] S4: Convolution of the low-rank matrix using a 1*1 convolution kernel through the UNet model.

[0066] The low-rank matrix can be regarded as a fully connected layer that converts the feature matrices corresponding to several intermediate feature maps into several low-rank feature maps, and the one-dimensional feature vectors corresponding to the several low-rank feature maps form the low-rank matrix. Figure 11 As shown, the fully connected layer is a convolutional layer of the UNet model, which can also be called the "intermediate layer". Each circle of the fully connected layer represents the parameters corresponding to each convolution kernel. The left column is the depth convolution layer, and each circle in the left column represents the feature parameters corresponding to the several intermediate feature maps output by the step S2. The right column is the point-by-point convolution layer, and each circle in the right column represents the feature parameters corresponding to the several low-rank feature maps output by the fully connected layer.

[0067] Further integration Figure 12 As shown, the fully connected layer can convert the feature matrices corresponding to the several intermediate feature maps into a low-rank matrix through the following relationship: Relationship 1 Equation 2 Wherein, m represents the number of intermediate feature maps for the convolution output, V1 is the one-dimensional feature vector corresponding to m intermediate feature maps, n represents the number of output feature maps for the convolution output, V2 is the one-dimensional feature vector corresponding to n output feature maps, is the matrix multiplication function, is a one-dimensional feature vector matrix corresponding to m intermediate feature maps and n output feature maps, and is a non-full rank matrix; r represents the rank of the low rank matrix, is a one-dimensional feature vector matrix based on m intermediate feature maps and the corresponding rank, It is a one-dimensional feature vector matrix corresponding to the rank and n output feature maps; that is, the rank is obtained according to several intermediate feature maps and a relationship, and then a low-rank matrix is ​​obtained according to the rank.

[0068] Figure 12 Each rectangular box in represents a feature map. Figure 12 As shown in the figure above, the above equation 1 can be considered as a mapping from a one-dimensional 1*m eigenvector V1 to another one-dimensional n*1 eigenvector V2. If we only consider linear transformation, that is, set this mapping relationship as a linear transformation relationship, then the operation from one-dimensional eigenvector V1 to one-dimensional eigenvector V2 can be regarded as a linear transformation represented by an m*n matrix. Correspondingly, is a linear function.

[0069] Combine Figure 12 As shown in the figure below, the above relationship 2 can be regarded as a low-rank decomposition of the m*n matrix. For non-full-rank matrices , then theoretically the larger matrix can be decomposed into an approximate expression of the product of two matrices (i.e., an m*r matrix and an r*n matrix). The specific principles and processes of decomposing the two matrices can be referred to the existing technology of low-rank decomposition, which will not be described here. Then, the low-rank matrix can be obtained according to the rank r. The low-rank matrix can be considered as a one-dimensional 1*r eigenvector. The role of the fully connected layer is to convert a one-dimensional 1*m eigenvector V1 into a one-dimensional 1*r eigenvector.

[0070] like Figure 11 and Figure 12 As shown, the input channel of the fully connected layer is m, the output channel is r, and the corresponding output r low-rank feature maps can be regarded as a one-dimensional r*1 feature vector .

[0071] As described above, the present application performs convolution operations on each input channel independently, and uses a convolution kernel (i.e., the first convolution kernel) for each input channel to generate output, which is equivalent to performing a depth-wise convolution without crossing channels. The output is combined across channels through convolution using a 1*1 convolution kernel (i.e., the second convolution kernel) to generate the final output feature map, which is equivalent to performing a point-by-point convolution across channels. That is, the convolution and convolution of the present application are equivalent to depth-wise separable convolution.

[0072] The depth-wise separable convolution can significantly reduce the number of multiplication operations. For example, for a traditional standard convolution with an input feature map size of H* W*D and an output feature map size of H'*W'*N, the computational complexity is D*K 2 *H'*W'*N (where K is the size of the convolution kernel); the computational complexity of depthwise separable convolution is only D*K 2 *H'*W'+D*H'*W'*N. From this, we can see that when the number of output channels N is large, the computational advantage of depthwise separable convolution is particularly obvious‌.

[0073] For another example, for a feature map with an input feature map size of 12*12*3, 256 convolution kernels of 5*5*3 are used to output an output feature map with an output size of 8*8*256. The computational cost of the traditional standard convolution method is: 256*(5*5*3)*(8*8)=1228800, where the computational cost of a single-channel 5*5 convolution kernel is 5*5, and the computational cost of the three channels is 5*5*3. The convolution kernel moves 8 times horizontally and vertically in the feature map respectively; while the computational cost of using depthwise separable convolution is: 3*(5*5*1)*(8*8 )+256*(1*1*3)*(8*8)=53952, where depth convolution is performed first, and three 5*5*1 convolution kernels are used to output an intermediate feature map of size 8*8*3 from the original map of size 12*12*3. The amount of calculation is 3*(5*5*1)*(8*8)=4800. Then, 256 1*1*3 convolution kernels are used to output an output feature map of size 8*8*256 from the intermediate feature map of size 8*8*3. The amount of calculation is 256*(1*1*3)*(8*8)=49152. It can be seen that the use of depthwise separable convolution in this application can greatly reduce the amount of calculation.

[0074] By using the corresponding convolution kernel to combine the outputs across channels, the convolution of each channel has a higher nonlinear expression ability, which can increase the nonlinear expression ability of the network model, thereby ensuring the accuracy of the image processing effect.

[0075] In addition, by creating a fully connected layer to convert a large feature matrix corresponding to several intermediate feature maps into a smaller low-rank matrix, the number of parameters that need to be processed can be significantly reduced. Parameters are shared to reduce redundancy, so that the model only needs to learn a small number of parameters during fine-tuning, thereby reducing the amount of computation. The specific reasons are as follows: 1. Reduce the number of parameters: For example, the weight matrix W∈R corresponding to the input features m×n If all parameters of the weight matrix W are updated directly, m×n parameters need to be updated; however, through step S3, the weight matrix W can be approximately converted into a low-rank matrix, which is equivalent to two matrices A∈R m×r and B∈R r×n The product of r≪min(m,n), min() is the minimum function, here, the number of parameters becomes m×r+r×n=r(m+n). Since r is much smaller than m and n, the number of parameters is greatly reduced.

[0076] 2. Parameter Sharing: A∈R m×r and B∈R r×nThe product of can be viewed as capturing the main information of weight updates through a low-dimensional space (rank r). This decomposition assumes that the main changes in weight updates are concentrated in the low-dimensional subspace, rather than the entire high-dimensional space. Therefore, only the parameters in the low-dimensional space need to be learned, without updating the entire high-dimensional matrix. This parameter sharing idea allows the network model to only learn a small number of parameters during fine-tuning, thereby reducing the amount of computation.

[0077] 3. Simplify the calculation process: Through the low-rank transformation in step S3, the weight matrix W∈R m×n The complex matrix operation is simplified to two simple matrices A∈R m×r and B∈R r×n Since the rank of the decomposed matrix is ​​small, the computational complexity and storage requirements are also correspondingly low.

[0078] This shows that in image processing processes such as image compression and feature extraction, convolution based on low-rank decomposition can be used to remove redundant information in feature maps, reducing the number of parameters and computational complexity. By converting the feature map matrix into a low-rank matrix, image data can be effectively compressed while retaining the main information, improving the nonlinear expression ability of the convolution kernel and ensuring the accuracy of the image processing effect.

[0079] In one example, the fully connected layer can use a nonlinear activation function to obtain the low-rank matrix, thereby adding nonlinear characteristics to the network model, so that no matter how many layers the network model has, the output of any layer is a nonlinear combination of the input variables of the layer, and the input variables obtained by the next layer are also a nonlinear combination, which is equivalent to the network model being a multi-layer nonlinear model, which can better learn and process the complex data characteristics of the image.

[0080] The present application also provides a convolution method according to another embodiment, comprising the following steps S1 to S5: S1: In response to inputting the corresponding feature into the UNet model, obtaining the corresponding number of input channels; S2: Set the convolution kernel of the UNet model to the number of input channels, and apply a convolution kernel to each input channel independently to perform convolution to output intermediate features; S3: Create a fully connected layer for the UNet model, and convert the feature matrix corresponding to the intermediate features into a low-rank matrix through the fully connected layer; S31: Create an activation layer for the UNet model; S32: Convolve the low-rank matrix using a nonlinear activation function through the activation layer to output several activation feature maps; S4: Convolution of the activation feature map using a 1*1 convolution kernel through the UNet model.

[0081] This example is equivalent to adding a convolutional layer, namely the activation layer, after the aforementioned fully connected layer. The activation layer uses a nonlinear activation function to convolve several activation feature maps, which can add nonlinear characteristics to the network model, so that the output of any subsequent layer is a nonlinear combination of the input variables of this layer. The input variables obtained by the next layer are also a nonlinear combination, which is equivalent to the network model being a multi-layer nonlinear model, which can better learn and process the complex data characteristics of images.

[0082] Based on the activation layer, further combined Figure 13 As shown, step S4 of this embodiment is equivalent to convolving the low-rank matrix through the following relationship: Equation 3 Equation 4 Equation 5 in, represents the one-dimensional eigenvector corresponding to the low-rank matrix, Represents the one-dimensional feature vectors corresponding to the several activation feature maps, represents the nonlinear activation function used by the activation layer, Represents the matrix transpose function.

[0083] Based on the above equation 3, after passing through the fully connected layer, the corresponding output r low-rank feature maps can be regarded as a one-dimensional r*1 feature vector Then, in order to ensure nonlinearity, these r low-rank feature maps are input into the activation layer, and r new feature maps are obtained after convolution based on the above relationship 4, that is, the one-dimensional feature vectors corresponding to the several activated feature maps are obtained. , and then after point-by-point convolution through the above equation 5, these activated feature maps can be linearly transformed into n new feature maps.

[0084] Combine Figure 13 As shown, each circle of the activation layer represents the parameters corresponding to each convolution kernel. In one example, the number of nodes used in the activation layer is equal to the number of nodes used in the fully connected layer. Here, this example can independently apply a convolution kernel to each output channel of the fully connected layer to perform a convolution operation to output a feature map, that is, no cross-channel combination is performed, thereby further reducing the amount of computation and the amount of parameters when the UNet model processes images.

[0085] The present application also provides a convolution method according to another embodiment, comprising the following steps S1 to S42: S1: In response to inputting the corresponding feature into the UNet model, obtaining the corresponding number of input channels; S2: Set the convolutional kernel of the UNet model to be equal to the number of input channels, and independently apply a convolutional kernel to each input channel for convolution to output intermediate features; S3: Create a fully connected layer for the UNet model, and convert the feature matrix corresponding to the intermediate features into a low-rank matrix through the fully connected layer; S41: Divide several low-rank matrices into several subgroups, where the number of subgroups is the result of the integer division operation between the number of output channels and the parallelism of the several subgroups; S42: Use a convolutional kernel of size 1*1 through the UNet model to perform convolution based on each subgroup.

[0086] This example is equivalent to introducing grouped convolution to depthwise separable convolution. Grouped convolution divides the relevant feature maps into multiple subgroups in the channel dimension and performs convolution operations independently within each subgroup. Specifically, grouped convolution divides the corresponding input channels and output channels into g non-overlapping subgroups, so that each subgroup only processes 1 / g of the channel data, thereby reducing the computational complexity to 1 / g of the original. By reducing the number of parameters and computational amount of the convolution operation, the running efficiency of the model is improved.

[0087] Different from the problem that there is no information transmission between channels in traditional grouped convolution, this example uses a fully connected layer combined with grouped convolution to increase the parallelism between each subgroup. For example, according to the parallelism of the several subgroups and the following relational expression, create a fully connected layer for the UNet model: r < m, r < n, r is x*k, x is an integer and x is divisible by both m and n, where k is the parallelism of the several subgroups, which can be preset according to, for example, the precision requirements of the device applicable to the model.

[0088] In the specific implementation, first establish a depth convolutional kernel according to the number of input channels, then create a fully connected layer according to the parallelism k. The way to take the rank (i.e., rank) of this fully connected layer follows the above relational expression, which can solve the problem of no communication between channels in subsequent use of grouped convolution. Then create a grouped pointwise convolution according to the parallelism k, where g = n / / k for the grouped convolution, and g represents the number of subgroups, that is, the number of subgroups is the result of the integer division operation between the number of output channels and the parallelism.

[0089] For example, in a model scenario where it is necessary to design a model with 128 input channels and 256 output channels, the parallelism pre-set according to the applicable device of the model is 32. The specific implementation method is: first, establish a deep convolution kernel according to the number of input channels (i.e., 128), and then create a fully connected layer according to the parallelism k (i.e., 32). The rank of the fully connected layer is less than 128 and the rank is x times 32, that is, the rank can be 32 or 64. In this example, the rank can be 64. Then, create a grouped point-by-point convolution according to the parallelism k, where g of the grouped convolution = (256 / / 32) = 8, that is, the point-by-point convolution adopts a grouping method of 8, and finally outputs several feature maps. Based on this, we can think of the following as follows: the four channels rc0-rc3 of the fully connected layer are mapped to the 32 output channels oc0-oc31; the four channels rc4-rc8 of the fully connected layer are mapped to the 32 output channels oc32-oc63; and so on, until the four channels rc60-r6c3 of the fully connected layer are mapped to the 32 output channels oc224-oc255, for a total of 8 subgroups. The introduction of fully connected layers can reduce the number of parameters while forcing communication between channels, thereby eliminating the isolation of information between input channels.

[0090] An embodiment of the present application also provides a storage medium, on which an image processing program is stored. The image processing program is essentially a computer program, and when the image processing program is executed by a processor, the steps of the image processing method based on the UNet model as in any example are implemented.

[0091] The storage medium includes but is not limited to any one of a read-only memory (ROM), a random access memory (RAM), a magnetic disk, and an optical disk.

[0092] Since the program stored in the storage medium can execute the steps in the image processing method of any embodiment provided in this application, the beneficial effects that can be achieved by the image processing method based on the UNet model of any of the aforementioned embodiments can be achieved. Please see the aforementioned embodiments for details and will not be repeated here.

[0093] An embodiment of the present application also provides an image processing device or chip, including a memory and a processor, wherein an image processing program is stored on the memory, and when the image processing program is executed by the processor, the steps of the image processing method based on the UNet model of any of the aforementioned embodiments are implemented; and / or, the image processing device or chip is provided with a storage medium as shown in the above example, and the processor loads the storage medium to execute the steps of the image processing method based on the UNet model of any of the aforementioned embodiments, thereby achieving the beneficial effects that can be achieved by the image processing method based on the UNet model of the corresponding embodiment.

[0094] The above are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. For ordinary technicians in this field, any equivalent structural changes made using the contents of this specification and drawings are also included in the patent protection scope of the present application.

[0095] Although the terms "first," "second," and the like are used herein to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. In addition, the singular forms "a," "an," and "the" are intended to include the plural forms as well. The terms "or" and "and / or" are to be interpreted as inclusive, meaning any one or any combination. Exceptions to this definition occur only when a combination of elements, functions, steps, or operations is inherently mutually exclusive in some manner.

Claims

1. An image processing method based on the UNet model, characterized in that: include: Coding phase: In response to inputting an original image into a UNet model, obtaining input features of the original image; Performing a skip connection on the input features; Performing a first channel rearrangement on the input features to achieve downsampling and outputting downsampled features; Inputting the downsampled features into the encoder for convolution to obtain encoded features; Obtaining a coding output feature according to the coding feature; Decoding stage: Obtaining residual features based on the skip connection; Inputting the encoded output features into a decoder for convolution; The convolution result is reordered in the second channel to achieve upsampling and output the decoded features; A target image is obtained according to the residual features and the decoded features.

2. The method according to claim 1, characterized in that The encoder includes multiple convolutional layers, and the encoding features are obtained after the multiple convolutional layers; the convolution performed by each convolutional layer includes: Pre-set weight coefficients k and 1-k; Input the downsampled features into the encoder for a first convolution; Perform the third convolution on the result of the first convolution; Input the downsampled features into the encoder for a second convolution; The product of the result of the third convolution and the weight coefficient k and the product of the result of the second convolution and the weight coefficient 1-k are input into the next convolution layer.

3. The method according to claim 2, characterized in that The first convolution and the third convolution both use a convolution kernel of size 3*3, and the second convolution uses a convolution kernel of size 3*3.

4. The method according to claim 1, wherein During the encoding phase, the method further comprises: Performing wavelet transform on the input features to obtain high-frequency features and low-frequency features; Performing skip connections on the high-frequency features; In the decoding stage, the method further includes: Obtaining the high-frequency features based on the skip connection; Performing convolution on the high-frequency features; The convolution result is rearranged in the third channel to output the wavelet residual feature; The obtaining of a target image according to the residual feature and the decoding feature includes: A target image is obtained according to the wavelet residual feature, the residual feature and the decoding feature map.

5. The method according to claim 4, characterized in that During the encoding phase, the method further comprises: Performing convolution on the low-frequency features; The obtaining of the encoding output feature according to the encoding feature includes: According to the convolution result of the coding feature and the low-frequency feature, a coding output feature is obtained.

6. The method according to claim 4, characterized in that The first channel rearrangement includes: Convert the input feature tensor shape [B1, C1, H1, W1] to a downsampled feature of the tensor shape [B1, r1*r1*C1, H1 / r1, W1 / r1], where B1 is the number of groups, C1 is the number of channels, H1*W1 is the channel size, r1 is a positive integer divisible by H1 and W1, and r1 ≥ 2; The second channel rearrangement or the third channel rearrangement includes: The convolution result tensor shape [B2, r1*r1C2, H2 / r1, W2 / r1] is transformed into features of tensor shape [B2, C2, H2, W2]; where B2 is the number of groups, C2 is the number of channels, H2*W2 is the channel size, and r1 is a positive integer divisible by H2 and W2.

7. The method according to any one of claims 1 to 6, characterized in that The convolution includes: In response to inputting the corresponding feature into the UNet model, obtaining the corresponding number of input channels; Setting a convolution kernel equal to the number of input channels for the UNet model, and independently applying a convolution kernel to each input channel to perform convolution to output intermediate features; Creating a fully connected layer for the UNet model, and converting the feature matrix corresponding to the intermediate features into a low-rank matrix through the fully connected layer; The low-rank matrix is ​​convolved using a convolution kernel of size 1*1 through the UNet model.

8. The method according to claim 7, characterized in that Convolution of low-rank matrices, including: Divide the low-rank matrix into a plurality of subgroups, where the number of the subgroups is the integer division result between the number of output channels and the degree of parallelism of the plurality of subgroups; Convolution is performed based on each subgroup.

9. An image processing device, characterized in that The method comprises a processor and a memory, wherein an image processing program is stored in the memory, and when the image processing program is executed by the processor, the steps of the image processing method based on the UNet model as described in any one of claims 1 to 8 are implemented.

10. A storage medium, characterized in that: A computer program is stored, which, when executed by a processor, implements the steps of the image processing method based on the UNet model described in any one of claims 1 to 8.

Images