A method for different visual processing based on a lightweight deep neural network

Abstract

The application discloses a method for different visual processing based on a light-weight deep neural network, which comprises the following steps: according to different visual tasks to be processed, data sets corresponding to the visual tasks are respectively acquired, and the data sets are divided into a training set and a test set according to a proportion; a light-weight deep neural network is constructed, the light-weight deep neural network comprises a three-branch feature extraction framework based on a multi-scale hollow convolution and a residual-dense-attention interactive mixed feature reconstruction framework; test set data is input into the trained light-weight deep neural network, the residual-dense-attention interactive mixed feature reconstruction framework is used to perform more deep-level feature reconstruction on the initially reconstructed features, and finally, reconstructed feature maps are obtained; and finally, the residual connection and transposed convolution are used to obtain the final output images under different visual tasks.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence / next-generation information technology, and relates to a method for performing different visual processing based on a lightweight deep neural network. Background Technology

[0002] Deep neural networks (DNNs) are artificial neural networks capable of meeting the needs of various computer vision tasks. They are one of the main implementation forms in next-generation information technology. Specifically, they employ complex connection operations through multiple convolutional layers, pooling layers, and activation functions to fully acquire and represent image features, and then reconstruct these features to complete the corresponding computer vision tasks. However, to improve network performance, current deep neural networks often have extremely deep layers. This significantly increases the probability of gradient vanishing or exploding during training. Even though introducing dense connections or residual connections can alleviate gradient vanishing or exploding to some extent, the complex network structure inevitably leads to the over-extraction of redundant features, significantly increasing the network's operating efficiency.

[0003] Existing visual processing methods, including image super-resolution, image enhancement, image denoising, and image restoration, employ different deep neural network architectures. Regarding complexity and computational cost: some deep neural network architectures can be extremely complex, requiring significant computational resources and time for training and inference. This can make them difficult to use in resource-constrained environments. Regarding overfitting: complex network architectures may perform well on training data but poorly on unseen data, leading to overfitting. This results in decreased generalization performance in real-world scenarios. Regarding handling multi-scale problems: some networks may have limited capabilities for processing multi-scale inputs, potentially causing performance degradation when dealing with images of different sizes. Regarding handling specific image types: some networks may perform well on specific image types but poorly on different image types. This can lead to problems when used in a wide range of applications. Summary of the Invention

[0004] To address the aforementioned problems, the technical solution adopted by this invention is: a method for performing different visual processing based on a lightweight deep neural network, comprising the following steps:

[0005] Based on the different visual tasks to be processed, obtain the datasets corresponding to the visual tasks respectively, and divide the datasets into training sets and test sets according to the proportions.

[0006] A lightweight deep neural network is constructed, which includes a three-branch feature extraction framework based on multi-scale dilated convolution and a feature reconstruction framework based on residual-dense-attention interaction hybrid model.

[0007] The three-branch feature extraction framework based on multi-scale dilated convolution and the feature reconstruction framework based on residual-dense-attention interaction hybrid are cascaded in sequence;

[0008] A lightweight deep neural network is trained using training set data from different visual tasks to obtain the trained lightweight deep neural network.

[0009] The test set data is input into the post-trained lightweight deep neural network. The three-branch feature extraction framework of multi-scale dilated convolution is used to extract all features of the input test data under the multi-scale receptive field. Through channel merging operation, all feature representations are initially fused to obtain feature map.

[0010] Based on the residual-dense-attention interactive hybrid feature reconstruction framework, we can perform deeper feature reconstruction on the features that have been initially reconstructed, and finally obtain the reconstructed feature map. Then, we can obtain the final output image for different visual tasks through residual connection and transposed convolution.

[0011] Furthermore: the first feature A used to extract the feature representation of the input data in the multi-scale receptive field (1) Second feature A (2) and the third feature A (3) And through channel merging operations, all features, i.e., the first feature A, are merged. (1) Second feature A (2) and the third feature A (3) This indicates that preliminary fusion is performed to obtain feature map X. E The procedure expression is as follows:

[0012]

[0013]

[0014]

[0015] X E =[A (1) A (2) A (3) (4)

[0016] in, The weights of the current convolution are r, the dilation rate is r, the kernel size is k×k, δ represents the ReLU activation function, * represents the convolution operation, and [] represents the channel connection operation.

[0017] Furthermore, the aforementioned residual-dense-attention interactive hybrid feature reconstruction framework is used to perform deeper feature reconstruction on the initially reconstructed features, ultimately obtaining the reconstructed feature map X.R The process of obtaining the final output Y through residual connection and transpose convolution is expressed as follows:

[0018]

[0019] I SR =T 1×1 (v(W 1×1 *X R ))+(T ↑S *I LR (7)

[0020] Among them, X R For the output of the two-branch reconstruction structure; X U T is the output of the upsampling stage of the super-resolution task. ↑s H represents the transposed convolution with a scaling factor of s; enhancement, restoration, and denoising do not require upsampling operations. D (·,θ D ) represents the expression by θ D Parameterized mapping function for dense branches; H R (·,θ R ) represents θ R The mapping function for parameterized residual branches; This indicates that the parameters for embedded systems are mostly attention reconstruction modules; I SR Represents the predicted SR image; where T 1×1 This represents a transposed convolution with a kernel size of 1×1.

[0021] Furthermore: the residual-dense-attention interactive hybrid feature reconstruction framework includes a multi-level perceptual attention mechanism that realizes synchronous enhancement of image features in three dimensions: pixels, space, and channels. The multi-level perceptual attention mechanism includes a pixel attention module, a collaborative attention module, a channel attention module, and a global context module.

[0022] The output of the pixel attention module is connected to the input of the collaborative attention module and the input of the channel attention module, respectively.

[0023] The output terminals of the collaborative attention module and the channel attention module are both connected to the global context module.

[0024] Furthermore, the residual-dense-attention interactive hybrid feature reconstruction framework is used to perform deeper feature reconstruction on the initially reconstructed features, and the process of finally obtaining the reconstructed feature map is as follows:

[0025] First, the number of channels is adjusted by a 1*1 convolution; then, the pixel information of the preprocessed image is passed to the pixel attention module.

[0026] Next, the image information processed by pixel attention is divided into two branches, which are fed into cooperative attention and channel attention respectively; then, the information from cooperative attention and channel attention is combined through a channel concatenation operation.

[0027] Next, after global average pooling, the process is divided into two branches. After average pooling, the number of channels is adjusted by 1*1 convolution, and then the sigmoid activation function is used to multiply the information of cooperative attention and channel attention respectively.

[0028] To prevent information loss, the information after the initial 1*1 convolution is combined with the information before the global context module through channel concatenation; finally, the reconstructed image information is obtained after passing through the global context module.

[0029] The pixel attention module consists of a 1*1 convolution and a sigmoid function, and multiplies the information before the convolution with the information after the sigmoid function.

[0030] The collaborative attention module consists of two channels concatenated after global average pooling, followed by a 3x3 dilated convolution with a dilation rate of 1, then fed into a sigmoid function and a BN normalization layer. It is then divided into two channels: a first channel and a second channel. The first channel consists of a 3x3 dilated convolution with a dilation rate of 2 and a sigmoid function, while the second channel consists of a 3x3 dilated convolution with a dilation rate of 3 and a sigmoid function. Finally, the information before average pooling is multiplied by the information after sigmoid to obtain the output of the writing attention.

[0031] The channel attention module consists of a global average pooling layer, a 3*3 convolution and a sigmoid function, and multiplies the information before global average pooling with the information after sigmoid to obtain the output of Tongandu attention;

[0032] The global context module consists of a 3x3 dilated convolution with a dilation rate of 1, a softmax activation function, and then multiplies the information before and after the softmax convolution. This is followed by a 3x3 dilated convolution with a dilation rate of 2, LN normalization, and ReLU activation function, then a 3x3 dilated convolution with a dilation rate of 3. Finally, residual connections are used to add the information before the 3x3 convolution to the information after the 3x3 convolution.

[0033] Furthermore: The three-branch feature extraction framework based on multi-scale dilated convolution has the following structure: it includes a first branch, a second branch, and a third branch; the input ends of the first branch, the second branch, and the third branch are connected, and the output ends of the first branch, the second branch, and the third branch are connected.

[0034] The first branch includes a first 3*3 convolution with a dilation rate of 1 and a second 3*3 convolution with a dilation rate of 1, and the first 3*3 convolution with a dilation rate of 1 and the second 3*3 convolution with a dilation rate of 1 are connected.

[0035] The second branch includes a first 3*3 convolution with a dilation rate of 2 and a second 3*3 convolution with a dilation rate of 2, and the first 3*3 convolution with a dilation rate of 2 and the second 3*3 convolution with a dilation rate of 2 are connected.

[0036] The third branch includes a first 3*3 convolution with a dilation rate of 3 and a second 3*3 convolution with a dilation rate of 3, and the first 3*3 convolution with a dilation rate of 3 and the second 3*3 convolution with a dilation rate of 3 are connected.

[0037] Each branch has residual links from before the first convolution to after the second convolution, and then the three branches are brought together by summation.

[0038] Furthermore: the residual-dense-attention interactive hybrid feature reconstruction framework structure includes four residual-dense-multidimensional attention reconstruction combinations; namely, the first residual-dense-multidimensional attention reconstruction combination, the second first residual-dense-multidimensional attention reconstruction combination, the third residual-dense-multidimensional attention reconstruction combination, and the fourth residual-dense-multidimensional attention reconstruction combination;

[0039] The first residual-dense-multidimensional attention reconstruction combination, the second residual-dense-multidimensional attention reconstruction combination, the third residual-dense-multidimensional attention reconstruction combination, and the fourth residual-dense-multidimensional attention reconstruction combination have the same structure;

[0040] The first residual-dense-multidimensional attention reconstruction combination includes a residual module, a dense module, and a multidimensional attention module, wherein the output terminals of the residual module and the dense module are connected to the input terminal of the multidimensional attention module;

[0041] The first residual-dense-multidimensional attention reconstruction combination, the second residual-dense-multidimensional attention reconstruction combination, the third residual-dense-multidimensional attention reconstruction combination, and the fourth residual-dense-multidimensional attention reconstruction combination are cascaded in sequence;

[0042] The output of the residual module of the first residual-dense-multidimensional attention reconstruction combination is also connected to the input of the dense module of the second residual-dense-multidimensional attention reconstruction combination.

[0043] The output of the residual module of the second residual-dense-multidimensional attention reconstruction combination is also connected to the input of the dense module of the third residual-dense-multidimensional attention reconstruction combination.

[0044] The output of the residual module of the third residual-dense-multidimensional attention reconstruction combination is also connected to the input of the dense module of the fourth residual-dense-multidimensional attention reconstruction combination, and the output of the dense module of the third residual-dense-multidimensional attention reconstruction combination is also connected to the input of the residual module of the fourth residual-dense-multidimensional attention reconstruction combination.

[0045] The outputs of the residual module and the dense module of the fourth residual-dense-multidimensional attention reconstruction combination adjust the number of channels through a 1*1 convolution.

[0046] Furthermore: the residual module consists of two 3*3 convolutional modules and residual connections; the dense module consists of two 3*3 convolutional modules that aggregate information through channel splicing operations.

[0047] This invention provides a method for performing different visual processing based on a lightweight deep neural network. Specifically, it constructs a three-branch feature extraction framework based on multi-scale dilated convolution, a 1*1 convolution-transposed convolution-1*1 convolution feature adjustment framework, and a residual-dense-attention interactive hybrid feature reconstruction framework. These frameworks are then concatenated to form a general-purpose deep neural network. This neural network is lightweight and can be applied to various types of computer vision tasks, including but not limited to image super-resolution, image enhancement, image denoising, and image restoration. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is an overall block diagram of the lightweight deep neural network constructed in this invention;

[0050] Figure 2 This is a diagram of the three-branch feature extraction structure based on multi-scale dilated convolution constructed in this invention;

[0051] Figure 3This is a structural diagram (b) of the dense module (a) and residual module constructed in this invention;

[0052] Figure 4 This is a diagram of the residual-dense-attention interactive hybrid feature reconstruction structure constructed in this invention;

[0053] Figure 5 This is a structural diagram of the multi-level perceptual attention mechanism constructed in this invention;

[0054] Figure 6 These are the effect diagrams of the deep neural network structure constructed by the present invention in different computer vision tasks in the experimental examples, where (a) is the effect diagram for image super-resolution task; (b) is the effect diagram for image enhancement task; (c) is the effect diagram for image restoration task; and (d) is the effect diagram for image denoising task. Detailed Implementation

[0055] It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0058] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0059] In the description of this invention, it should be understood that the orientation or positional relationship indicated by directional terms such as "front, back, up, down, left, right", "horizontal, vertical, horizontal" and "top, bottom" is generally based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing this invention and simplifying the description. Unless otherwise stated, these directional terms do not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the scope of protection of this invention. The directional terms "inner" and "outer" refer to the inner and outer contours relative to the outline of each component itself.

[0060] For ease of description, spatial relative terms such as "above," "over," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation besides the orientation of the device as described in the figures. For example, if the device in the figures is inverted, a device described as "above" or "above" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0061] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, the above terms have no special meaning and therefore should not be construed as limiting the scope of protection of this invention.

[0062] A method for performing various visual processing tasks based on lightweight deep neural networks includes the following steps:

[0063] S1: Based on the different visual tasks to be processed, obtain the datasets corresponding to the visual tasks respectively, and divide the datasets into training sets and test sets according to the proportions;

[0064] S2: Construct a lightweight deep neural network, which includes a three-branch feature extraction framework based on multi-scale dilated convolution and a feature reconstruction framework based on residual-dense-attention interaction hybrid model.

[0065] The three-branch feature extraction framework based on multi-scale dilated convolution and the feature reconstruction framework based on residual-dense-attention interaction hybrid are cascaded in sequence; Figure 1 This is an overall block diagram of the lightweight deep neural network constructed in this invention;

[0066] S3: Train a lightweight deep neural network based on training set data for different visual tasks to obtain a trained lightweight deep neural network.

[0067] S4: Input the test set data into the post-trained lightweight deep neural network. The three-branch feature extraction framework of multi-scale dilated convolution is used to extract all features of the input test data under the multi-scale receptive field. Through channel merging operation, all feature representations are initially fused to obtain feature map.

[0068] S5: Based on the residual-dense-attention interactive hybrid feature reconstruction framework, it is used to perform deeper feature reconstruction on the features that have been initially reconstructed, and finally obtain the reconstructed feature map. Then, the final output image under different visual tasks is obtained through residual connection and transposed convolution.

[0069] Steps S1 / S2 / S3 / S4 / S5 are executed sequentially;

[0070] The visual processing tasks include image super-resolution, image enhancement, image denoising, and image restoration.

[0071] The image super-resolution requires preprocessing, which includes downsampling and upsampling operations.

[0072] The image enhancement, image denoising, and image restoration visual processes do not require preprocessing.

[0073] The first feature A is used to extract the feature representation of the visual processing task input data in the multi-scale receptive field. (1) Second feature A (2) and the third feature A (3) And through channel merging operations, all features, i.e., the first feature A, are merged.(1) Second feature A (2) and the third feature A (3) This indicates that preliminary fusion is performed to obtain feature map X. E The procedure expression is as follows:

[0074]

[0075]

[0076]

[0077] X E =[A (1) A (2) A (3) (4)

[0078] in, The weights of the current convolution are r, the dilation rate is r, the kernel size is k×k, δ represents the ReLU activation function, * represents the convolution operation, and [] represents the channel connection operation.

[0079] The proposed framework is based on a residual-dense-attention interactive hybrid feature reconstruction approach. This framework is used to perform deeper feature reconstruction on the features that have been initially reconstructed, ultimately obtaining the reconstructed feature map X. R The process of obtaining the final output Y through residual connection and transpose convolution is expressed as follows:

[0080]

[0081] I SR =T 1×1 (δ(W 1×1 *x R ))+(T ↑S *I LR (7)

[0082] Among them, X R For the output of the two-branch reconstruction structure; X U T is the output of the upsampling stage of the super-resolution task. ↑s H represents the transposed convolution with a scaling factor of s. Tasks such as enhancement, denoising, and restoration do not require upsampling operations. D (·,θ D ) represents the expression by θ D Parameterized mapping function for dense branches; H R (·,θ R ) represents θ R The mapping function for parameterized residual branches; This indicates that the parameters for embedded systems are mostly attention reconstruction modules; I SR Represents the predicted SR image; where T1×1 This represents a transposed convolution with a kernel size of 1×1.

[0083] Figure 2 This is a diagram of the three-branch feature extraction structure based on multi-scale dilated convolution constructed in this invention;

[0084] The three-branch feature extraction framework based on multi-scale dilated convolution has the following structure: it includes a first branch, a second branch, and a third branch; the inputs of the first branch, the second branch, and the third branch are connected, and the outputs of the first branch, the second branch, and the third branch are connected.

[0085] The first branch includes a first 3*3 convolution with a dilation rate of 1 and a second 3*3 convolution with a dilation rate of 1, and the first 3*3 convolution with a dilation rate of 1 and the second 3*3 convolution with a dilation rate of 1 are connected.

[0086] The second branch includes a first 3*3 convolution with a dilation rate of 2 and a second 3*3 convolution with a dilation rate of 2, and the first 3*3 convolution with a dilation rate of 2 and the second 3*3 convolution with a dilation rate of 2 are connected.

[0087] The third branch includes a first 3*3 convolution with a dilation rate of 3 and a second 3*3 convolution with a dilation rate of 3, and the first 3*3 convolution with a dilation rate of 3 and the second 3*3 convolution with a dilation rate of 3 are connected.

[0088] Each branch has residual links from before the first convolution to after the second convolution, and then the three branches are brought together by summation.

[0089] The information after three-branch multi-scale feature extraction is passed to the residual module and the dense module through two branches respectively. Then, the information after passing through the two is summarized and passed to the multi-dimensional attention module. After passing through the multi-dimensional attention module, it is passed to the next dense-residual module.

[0090] Figure 3 This is a diagram of the residual-dense-attention interactive hybrid feature reconstruction structure constructed in this invention;

[0091] Figure 4 (a) is the dense module constructed in this invention, and (b) is the structural diagram of the residual module;

[0092] The residual-dense-attention interactive hybrid feature reconstruction framework includes a multi-level perceptual attention mechanism that enables synchronous enhancement of image features in three dimensions: pixels, space, and channels. The multi-level perceptual attention mechanism includes a pixel attention module, a collaborative attention module, a channel attention module, and a global context module.

[0093] The output of the pixel attention module is connected to the input of the collaborative attention module and the input of the channel attention module, respectively.

[0094] The output terminals of the collaborative attention module and the channel attention module are both connected to the global context module.

[0095] The residual-dense-attention interactive hybrid feature reconstruction framework is structured as follows: it includes four residual-dense-multidimensional attention reconstruction combinations: namely, the first residual-dense-multidimensional attention reconstruction combination, the second first residual-dense-multidimensional attention reconstruction combination, the third residual-dense-multidimensional attention reconstruction combination, and the fourth residual-dense-multidimensional attention reconstruction combination;

[0096] The first residual-dense-multidimensional attention reconstruction combination, the second residual-dense-multidimensional attention reconstruction combination, the third residual-dense-multidimensional attention reconstruction combination, and the fourth residual-dense-multidimensional attention reconstruction combination have the same structure;

[0097] The first residual-dense-multidimensional attention reconstruction combination includes a residual module, a dense module, and a multidimensional attention module, wherein the output terminals of the residual module and the dense module are connected to the input terminal of the multidimensional attention module;

[0098] The first residual-dense-multidimensional attention reconstruction combination, the second residual-dense-multidimensional attention reconstruction combination, the third residual-dense-multidimensional attention reconstruction combination, and the fourth residual-dense-multidimensional attention reconstruction combination are cascaded in sequence;

[0099] The output of the residual module of the first residual-dense-multidimensional attention reconstruction combination is also connected to the input of the dense module of the second residual-dense-multidimensional attention reconstruction combination.

[0100] The output of the residual module of the second residual-dense-multidimensional attention reconstruction combination is also connected to the input of the dense module of the third residual-dense-multidimensional attention reconstruction combination.

[0101] The output of the residual module of the third residual-dense-multidimensional attention reconstruction combination is also connected to the input of the dense module of the fourth residual-dense-multidimensional attention reconstruction combination, and the output of the dense module of the third residual-dense-multidimensional attention reconstruction combination is also connected to the input of the residual module of the fourth residual-dense-multidimensional attention reconstruction combination.

[0102] The outputs of the residual module and the dense module of the fourth residual-dense-multidimensional attention reconstruction combination adjust the number of channels through a 1*1 convolution.

[0103] The residual module consists of two 3x3 convolutional modules and residual connections; the dense module consists of two 3x3 convolutional modules that aggregate information through channel splicing.

[0104] Figure 5 This is a structural diagram of the multi-level perceptual attention mechanism constructed in this invention;

[0105] The residual-dense-attention interactive hybrid feature reconstruction framework includes a multi-level perceptual attention mechanism that enables synchronous enhancement of image features in three dimensions: pixels, space, and channels. The multi-level perceptual attention mechanism includes a pixel attention module, a collaborative attention module, a channel attention module, and a global context module.

[0106] The output of the pixel attention module is connected to the input of the collaborative attention module and the input of the channel attention module, respectively.

[0107] The output terminals of the collaborative attention module and the channel attention module are both connected to the global context module.

[0108] The residual-dense-attention interactive hybrid feature reconstruction framework is used to perform deeper feature reconstruction on the features that have been initially reconstructed, and the process of finally obtaining the reconstructed feature map is as follows:

[0109] First, the number of channels is adjusted using a 1x1 convolution; then, the pixel information is passed to the pixel attention module.

[0110] Next, the image information processed by the pixel attention module is divided into two branches, which are fed into collaborative attention and channel attention respectively; then, the information of collaborative attention and channel attention is combined through a channel concatenation operation.

[0111] Next, after global average pooling, the image is divided into two branches. Then, after adjusting the number of channels through 1*1 convolution, the image is multiplied by the information from collaborative attention and channel attention respectively through the sigmoid activation function. To prevent information loss, the information from the initial 1*1 convolution is combined with the information from the previous global context module through channel concatenation. Finally, the image is reconstructed and output through the global context module.

[0112] The pixel attention module consists of a 1*1 convolution and a sigmoid function, and multiplies the information before the convolution with the information after the sigmoid function.

[0113] The collaborative attention module consists of two channels concatenated after global average pooling, followed by a 3x3 dilated convolution with a dilation rate of 1, then fed into a sigmoid function and a BN normalization layer. It is then divided into two channels: a first channel and a second channel. The first channel consists of a 3x3 dilated convolution with a dilation rate of 2 and a sigmoid function, while the second channel consists of a 3x3 dilated convolution with a dilation rate of 3 and a sigmoid function. Finally, the information before average pooling is multiplied by the information after sigmoid to obtain the output of the writing attention.

[0114] The channel attention module consists of a global average pooling layer, a 3*3 convolution and a sigmoid function, and multiplies the information before global average pooling with the information after sigmoid to obtain the output of Tongandu attention;

[0115] The global context module consists of a 3x3 dilated convolution with a dilation rate of 1, a softmax activation function, and then multiplies the information before and after the softmax convolution. This is followed by a 3x3 dilated convolution with a dilation rate of 2, LN normalization, and ReLU activation function, then a 3x3 dilated convolution with a dilation rate of 3. Finally, residual connections are used to add the information before the 3x3 convolution to the information after the 3x3 convolution.

[0116] Figure 6 The images shown are the results of the deep neural network structure constructed in this invention in different computer vision tasks in the experimental examples. (a) is the result for image super-resolution task; (b) is the result for image enhancement task; (c) is the result for image restoration task; and (d) is the result for image denoising task.

[0117] Table 1

[0118] Image super-resolution 27.7333 0.8481 Image enhancement 23.6044 0.7981 Image restoration 23.5496 0.7989 Image Denoising 30.0934 0.8475

[0119] Table 2 shows the computational complexity using super-resolution as an example.

[0120] VDSR 0.6659 6.1365 SRResNet 1.4017 13.3772 EDSR 1.3699 12.6500 IMDN 0.6944 6.3542 RFDN 0.4172 3.6514 BSRGAN 16.6611 155.5100 DualCNN 0.6175 5.4653 VapSR 0.4436 4.0459 Our method 0.3066 2.2682

[0121] Table 1 shows that for four visual processing tasks—image super-resolution, image enhancement, image denoising, and image denoising—the proposed method has advantages in both parameter quantity and model computational power. Table 2 shows that the proposed method has advantages in both parameter quantity and model computational power.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for performing different visual processing based on a lightweight deep neural network, characterized in that: Includes the following steps: Based on the different visual tasks to be processed, obtain the datasets corresponding to the visual tasks respectively, and divide the datasets into training sets and test sets according to the proportions. A lightweight deep neural network is constructed, which includes a three-branch feature extraction framework based on multi-scale dilated convolution and a feature reconstruction framework based on residual-dense-attention interaction hybrid model. The three-branch feature extraction framework based on multi-scale dilated convolution and the feature reconstruction framework based on residual-dense-attention interaction hybrid are cascaded in sequence; A lightweight deep neural network is trained using training set data from different visual tasks to obtain the trained lightweight deep neural network. The test set data is input into the trained lightweight deep neural network. The three-branch feature extraction framework of multi-scale dilated convolution is used to extract all features of the input test data under the multi-scale receptive field. Through channel merging operation, all feature representations are initially fused to obtain feature map. Based on the residual-dense-attention interactive hybrid feature reconstruction framework, we can perform deeper feature reconstruction on the initially reconstructed features to obtain the reconstructed feature map. Then, we connect the residual to the transposed convolution to obtain the final output image for different visual tasks. The residual-dense-attention interactive hybrid feature reconstruction framework includes a multi-level perceptual attention mechanism that enables synchronous enhancement of image features in three dimensions: pixels, space, and channels. The multi-level perceptual attention mechanism includes a pixel attention module, a collaborative attention module, a channel attention module, and a global context module. The output of the pixel attention module is connected to the input of the collaborative attention module and the input of the channel attention module, respectively. The output terminals of the collaborative attention module and the channel attention module are both connected to the global context module. The residual-dense-attention interactive hybrid feature reconstruction framework is used to perform deeper feature reconstruction on the features that have been initially reconstructed, and the process of finally obtaining the reconstructed feature map is as follows: First, after 1 A convolution of 1 is used to adjust the number of channels; then the pixel information of the preprocessed image is passed to the pixel attention module; The image information, after being processed by pixel attention, is divided into two branches, which are then fed into collaborative attention and channel attention respectively. The information from collaborative attention and channel attention is then combined through a channel concatenation operation. Next, after global average pooling, the process is divided into two branches. The average pooling steps are then passed through 1... After adjusting the number of channels through a convolution of 1, the signal is multiplied by information from collaborative attention and channel attention respectively using a sigmoid activation function; To prevent information loss, the initial 1 The information after convolution is combined with the information before the global context module through channel concatenation; finally, the reconstructed image information is obtained through the global context module. The pixel attention module consists of 1 It consists of a convolution of size 1 and a sigmoid function, and multiplies the information before the convolution with the information after the sigmoid function; The collaborative attention module is composed of two globally average pooled channels concatenated, and then processed by 3... A dilated convolution with a dilation rate of 1 is passed to a sigmoid function and a batch normalization layer, and then divided into two channels: a first channel and a second channel; the first channel has a dilation rate of 3. A dilated convolution with a dilation rate of 2 and a sigmoid function, where the second channel has a dilation rate of 3. A dilated convolution with a dilation rate of 3 and a sigmoid function are used. Finally, the information before average pooling is multiplied by the information after sigmoid to obtain the output of collaborative attention. The channel attention module consists of a global average pooling layer, a 3... It consists of a convolution of size 3 and a sigmoid function, and the information before global average pooling is multiplied with the information after sigmoid to obtain the output of channel attention; The global context module consists of 3 A dilated convolution with a dilation rate of 1 is applied, followed by a softmax activation function. The information before and after the softmax activation is then multiplied together and fed into a 3x3 matrix. A dilated convolution with a dilation rate of 2 is applied, followed by LN normalization and ReLU activation, and then passed in 3... A dilated convolution with a dilation rate of 3 is applied, and finally, the 3-dimensional convolution is performed using residual connections. The information before the 3-convolution is added to the information after the 3-convolution with a dilatation factor of 3. The three-branch feature extraction framework based on multi-scale dilated convolution has the following structure: it includes a first branch, a second branch, and a third branch; the inputs of the first branch, the second branch, and the third branch are connected, and the outputs of the first branch, the second branch, and the third branch are connected. The first branch includes the first 3 3-hole convolution with a dilation rate of 1, the second 3 A convolution with a dilation rate of 1 of 3, the first 3 3 convolutions with a dilation rate of 1 and the second 3 Three convolutional connections with a dilation rate of 1; The second branch includes the first 3 3 convolutions with a dilation rate of 2, the second 3 A convolution with a dilation rate of 2, the first 3 A convolution with a dilation rate of 3 and a second dilation rate of 2 Convolutional connections with a dilation rate of 2; The third branch includes the first 3 Convolution with a dilation rate of 3, the second 3 A convolution with a dilation rate of 3, the first 3 A convolution with a dilation rate of 3 and a second 3 A convolutional link with a dilation rate of 3; Each branch has residual links from before the first convolution to after the second convolution, and then the three branches are brought together by summation.

2. The method for performing different visual processing based on a lightweight deep neural network according to claim 1, characterized in that: The first feature A used to extract the feature representation of the input test data in the multi-scale receptive field. (1) Second feature A (2) and the third feature A (3) And through channel merging operations, all features, i.e., the first feature A, are merged. (1) Second feature A (2) and the third feature A (3) This indicates that preliminary fusion is performed to obtain feature map X. E The procedure expression is as follows: （1） （2） （3） （4） in, The current convolution weights have an dilation rate of 0. , × The size of the nucleus Represents the ReLU activation function. [ ] represents convolution operation; [ ] represents channel concatenation operation.

3. The method for performing different visual processing based on a lightweight deep neural network according to claim 1, characterized in that: The proposed framework is based on a residual-dense-attention interactive hybrid feature reconstruction method, used to perform deeper feature reconstruction on the initially reconstructed features, ultimately obtaining the reconstructed feature map X. R The process of obtaining the final output Y by connecting the residual to the transposed convolution is expressed as follows: （6） （7） in, The output of the two-branch reconstruction structure; T is the output of the upsampling stage of the super-resolution task. ↑s The scaling factor is Transposed convolutions do not require upsampling operations for enhancement, restoration, and denoising; H (·, ) indicates by Parameterized mapping function for dense branches; H (·, ) indicates with The mapping function for parameterized residual branches; This indicates that the parameters for embedded systems are mostly attention reconstruction modules; I SR Represents the predicted SR image; where T 1×1 This represents a transposed convolution with a kernel size of 1×1.

4. The method for performing different visual processing based on a lightweight deep neural network according to claim 1, characterized in that: The residual-dense-attention interactive hybrid feature reconstruction framework includes four residual-dense-multidimensional attention reconstruction combinations: namely, the first residual-dense-multidimensional attention reconstruction combination, the second residual-dense-multidimensional attention reconstruction combination, the third residual-dense-multidimensional attention reconstruction combination, and the fourth residual-dense-multidimensional attention reconstruction combination. The first residual-dense-multidimensional attention reconstruction combination, the second residual-dense-multidimensional attention reconstruction combination, the third residual-dense-multidimensional attention reconstruction combination, and the fourth residual-dense-multidimensional attention reconstruction combination have the same structure; The first residual-dense-multidimensional attention reconstruction combination includes a residual module, a dense module, and a multidimensional attention module, wherein the output terminals of the residual module and the dense module are connected to the input terminal of the multidimensional attention module; The first residual-dense-multidimensional attention reconstruction combination, the second residual-dense-multidimensional attention reconstruction combination, the third residual-dense-multidimensional attention reconstruction combination, and the fourth residual-dense-multidimensional attention reconstruction combination are cascaded in sequence; The output of the residual module of the first residual-dense-multidimensional attention reconstruction combination is also connected to the input of the dense module of the second residual-dense-multidimensional attention reconstruction combination. The output of the residual module of the second residual-dense-multidimensional attention reconstruction combination is also connected to the input of the dense module of the third residual-dense-multidimensional attention reconstruction combination. The output of the residual module of the third residual-dense-multidimensional attention reconstruction combination is also connected to the input of the dense module of the fourth residual-dense-multidimensional attention reconstruction combination, and the output of the dense module of the third residual-dense-multidimensional attention reconstruction combination is also connected to the input of the residual module of the fourth residual-dense-multidimensional attention reconstruction combination. The outputs of the residual module and the dense module of the fourth residual-dense-multidimensional attention reconstruction combination are connected via 1 The number of channels in a convolution of size 1 is adjusted.

5. The method for performing different visual processing based on a lightweight deep neural network according to claim 4, characterized in that: The residual module includes two 3 The dense module consists of a 3x3 convolutional module and residual connections; the dense module includes two 3x3 convolutional modules. The convolutional module of module 3 combines information through channel splicing.

Images