Image restoration method and device for multi-task learning in rain and snow environment

Abstract

The invention provides an image restoration method and device for multi-task learning in a rain and snow environment, and the method comprises the steps: carrying out the double down-sampling and quadruple down-sampling of an RGB image collected by a vehicle, and obtaining a first down-sampling image and a second down-sampling image; when the vehicle is in a rainy day environment, processing the RGB image, the first down-sampling image and the second down-sampling image by using a pre-trained first image restoration backbone network to obtain a feature map; processing the feature map by using a pre-trained first regression layer to obtain a restored image; or, when the vehicle is in the snowy environment, processing the RGB image, the first down-sampling image and the second down-sampling image by using a pre-trained second image restoration backbone network to obtain a feature map; and processing the feature map by using a pre-trained second regression layer to obtain a restored image. According to the invention, the imaging quality of the infrared camera in rainy and snowy weather is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to an image restoration method and device for multi-task learning in a rainy and snowy environment. Background Art

[0002] Autonomous driving is a highly complex and challenging task that requires the system to accurately, comprehensively and reliably perceive and understand the surrounding environment. As a key sensor on an autonomous vehicle, the visual camera is responsible for capturing information about the surrounding scene and performing key functions such as detecting pedestrians and vehicles, recognizing traffic signs, and sensing the intentions of other road users. However, in an open outdoor environment, severe weather conditions such as rain and snow often occur, posing a severe test to the performance of autonomous vehicles. When raindrops or snowflakes adhere to the camera lens, they block part of the field of view, resulting in a significant decrease in image quality, which directly affects the accuracy of the image processing algorithm, reduces the vehicle's ability to perceive the environment, and increases safety risks during driving.

[0003] Currently, there is no solution for how to effectively remove interference such as raindrops and snowflakes from images and improve the imaging quality of self-driving cars in severe rainy and snowy weather. Summary of the invention

[0004] In view of this, the present application provides an image restoration method and device for multi-task learning in a rainy and snowy environment to solve the above-mentioned technical problems.

[0005] In a first aspect, an embodiment of the present application provides an image restoration method for multi-task learning in a rainy and snowy environment, comprising:

[0006] Get the RGB image captured by the vehicle's infrared camera;

[0007] Using a classification network to perform classification detection on the RGB image to obtain a classification result; determining whether the vehicle is in a rainy environment or a snowy environment according to the classification result;

[0008] Downsampling the RGB image by a factor of two to obtain a first downsampled image; downsampling the RGB image by a factor of four to obtain a second downsampled image;

[0009] When the vehicle is in a rainy environment, the RGB image, the first downsampled image, and the second downsampled image are processed using a pre-trained first image restoration backbone network to obtain a feature map; the feature map is processed using a pre-trained first regression layer to obtain a restored image; or,

[0010] When the vehicle is in a snowy environment, the RGB image, the first down-sampled image, and the second down-sampled image are processed using a pre-trained second image restoration backbone network to obtain a feature map; the feature map is processed using a pre-trained second regression layer to obtain a restored image.

[0011] Furthermore, the RGB image, the first down-sampled image, and the second down-sampled image are processed using a pre-trained first image restoration backbone network to obtain a feature map, including:

[0012] The first convolutional layer is used to extract features from the RGB image to obtain a first feature map of 128 dimensions; the second convolutional layer is used to extract features from the first downsampled image to obtain a second feature map of 128 dimensions; the third convolutional layer is used to extract features from the second downsampled image to obtain a third feature map of 128 dimensions;

[0013] The first feature map is processed by a fourth convolutional layer to obtain a fourth feature map; the fourth feature map is processed by a fifth convolutional layer with a step size of 2 to obtain a fifth feature map; the fifth feature map and the second feature map are concatenated in the channel dimension to obtain a sixth feature map with a dimension of 256;

[0014] The sixth feature map is processed by reducing the dimension by using the sixth convolutional layer to obtain a seventh feature map with a dimension of 128;

[0015] The seventh feature map is processed by a seventh convolutional layer with a step size of 2 to obtain an eighth feature map, and the eighth feature map and the third feature map are concatenated in the channel dimension to obtain a ninth feature map;

[0016] The eighth convolutional layer is used to reduce the dimension of the ninth feature map to obtain a tenth feature map with a dimension of 128;

[0017] The tenth feature map is processed by a first deconvolution layer with a step size of 2 to obtain an eleventh feature map, and the eleventh feature map and the third feature map are concatenated in the channel dimension to obtain a twelfth feature map;

[0018] The twelfth feature map is processed by reducing the dimension by using the ninth convolutional layer to obtain the thirteenth feature map;

[0019] The thirteenth feature map is processed by improving the resolution using a second deconvolution layer with a step size of 2 to obtain a fourteenth feature map; the fourteenth feature map and the fourth feature map are concatenated in the channel dimension to obtain a fifteenth feature map;

[0020] The tenth convolutional layer is used to reduce the dimension of the fifteenth feature map to obtain a feature map with a dimension of 128.

[0021] Further, the RGB image, the first down-sampled image, and the second down-sampled image are processed using a pre-trained second image restoration backbone network to obtain a feature map, and the feature map includes:

[0022] The eleventh convolutional layer is used to extract features from the RGB image to obtain a sixteenth feature map of 128 dimensions; the twelfth convolutional layer is used to extract features from the first downsampled image to obtain a seventeenth feature map of 128 dimensions; the thirteenth convolutional layer is used to extract features from the second downsampled image to obtain an eighteenth feature map of 128 dimensions;

[0023] The sixteenth feature map is processed by the fourteenth convolution layer to obtain the nineteenth feature map; the nineteenth feature map is processed by the fifteenth convolution layer with a step size of 2 to obtain the twentieth feature map; the twentieth feature map and the seventeenth feature map are concatenated in the channel dimension to obtain the twenty-first feature map with a dimension of 256;

[0024] The 21st feature map is processed by reducing the dimension by using the 16th convolutional layer to obtain the 22nd feature map with a dimension of 128;

[0025] The twenty-second feature map is processed using a seventeenth convolutional layer with a step size of 2 to obtain a twenty-third feature map, and the twenty-third feature map and the eighteenth feature map are concatenated in the channel dimension to obtain a twenty-fourth feature map;

[0026] The 24th feature map is processed by reducing the dimension by using the 18th convolutional layer to obtain the 25th feature map with a dimension of 128;

[0027] The twenty-fifth feature map is processed by a third deconvolution layer with a step size of 2 to obtain a twenty-sixth feature map, and the twenty-sixth feature map and the eighteenth feature map are concatenated in the channel dimension to obtain a twenty-seventh feature map;

[0028] The 27th feature map is processed by reducing the dimension by using the 19th convolutional layer to obtain the 28th feature map;

[0029] A fourth deconvolution layer with a step size of 2 is used to increase the resolution of the twenty-eighth feature map to obtain a twenty-ninth feature map; the twenty-ninth feature map and the nineteenth feature map are concatenated in the channel dimension to obtain a thirtieth feature map;

[0030] The twentieth convolutional layer is used to reduce the dimension of the thirtieth feature map to obtain a feature map with a dimension of 128.

[0031] Further, the first regression layer includes two connected twenty-first convolutional layers and a twenty-second convolutional layer;

[0032] The feature map is processed using the pre-trained first regression layer to obtain the restored image, including:

[0033] The 21st convolutional layer is used to reduce the dimension of the feature map to obtain an intermediate feature map with a dimension of 64;

[0034] The 22nd convolutional layer is used to reduce the dimension of the intermediate feature map with a dimension of 64 to obtain the restored image.

[0035] Further, the second regression layer includes two connected twenty-third convolutional layers and a twenty-fourth convolutional layer;

[0036] The feature map is processed using the pre-trained second regression layer to obtain the restored image, including:

[0037] The 23rd convolutional layer is used to reduce the dimension of the feature map to obtain an intermediate feature map with a dimension of 64;

[0038] The 24th convolutional layer is used to reduce the dimension of the intermediate feature map with a dimension of 64 to obtain the restored image.

[0039] Furthermore, the method further includes: a step of jointly training the first image restoration backbone network, the first regression layer, the second image restoration backbone network and the second regression layer, specifically including:

[0040] Constructing a training network, including a first branch and a second branch in parallel, wherein the first branch includes a first image restoration backbone network and a first regression layer, and the second branch includes a second image restoration backbone network and a second regression layer;

[0041] Establishing a rainy day data set and a snowy day data set; the rainy day data set includes N first RGB image samples collected on rainy days; the snowy day data set includes N second RGB image samples collected on snowy days;

[0042] Downsampling the first RGB image sample by two times to obtain a first downsampled image sample; downsampling the first RGB image sample by four times to obtain a second downsampled image sample;

[0043] Downsampling the second RGB image sample by a factor of two to obtain a third downsampled image sample; downsampling the second RGB image sample by a factor of four to obtain a fourth downsampled image sample;

[0044] Using the first image restoration backbone network to process the first RGB image sample, the first downsampled image sample, and the second downsampled image sample to obtain a first feature map sample; using the first regression layer to process the first feature map sample to obtain a first restored image sample;

[0045] Determining a first loss value based on the first restored image sample and the first RGB image sample;

[0046] Using the second image restoration backbone network to process the second RGB image sample, the third down-sampled image sample and the fourth down-sampled image sample to obtain a second feature map sample; using the second regression layer to process the second feature map sample to obtain a second restored image sample;

[0047] Determining a second loss value based on the second restored image sample and the second RGB image sample;

[0048] Using the first loss value and the second loss value, a cross-task fine-grained parameter adaptive sharing strategy is adopted to update the weight parameters of the first image restoration backbone network and the second image restoration backbone network, and the weight parameters of the first regression layer and the second regression layer are updated at the same time.

[0049] Furthermore, the first loss value and the second loss value are used to update the weight parameters of the first image restoration backbone network and the second image restoration backbone network by adopting a cross-task fine-grained parameter adaptive sharing strategy, and the weight parameters of the first regression layer and the second regression layer are updated at the same time, including:

[0050] The weight parameters of each convolution layer and deconvolution layer of the first image restoration backbone network are divided into shared parameters and specific parameters; the weight parameters of each convolution layer and deconvolution layer of the second image restoration backbone network are divided into shared parameters and specific parameters;

[0051] Using the first loss value, updating the shared parameters and specific parameters of each convolution layer in the first image restoration backbone network, the shared parameters and specific parameters of each deconvolution layer, and the weight parameters of the two convolution layers of the first regression layer;

[0052] Using the second loss value, updating the shared parameters and specific parameters of each convolution layer and the shared parameters and specific parameters of each deconvolution layer in the second image restoration backbone network, and the weight parameters of two convolution layers of the second regression layer, wherein the shared parameters of the convolution layer and the deconvolution layer at the same position in the first image restoration backbone network and the second image restoration backbone network are the same;

[0053] Determine weight parameters of each convolution layer and weight parameters of each deconvolution layer according to shared parameters and specific parameters of each convolution layer and shared parameters and specific parameters of each deconvolution layer in the first image restoration backbone network;

[0054] According to the shared parameters and specific parameters of each convolution layer and the shared parameters and specific parameters of each deconvolution layer in the second image restoration backbone network, the weight parameters of each convolution layer and the weight parameters of each deconvolution layer are determined respectively.

[0055] Further, according to the shared parameters and specific parameters of each convolution layer and the shared parameters and specific parameters of each deconvolution layer in the first image restoration backbone network, the weight parameters of each convolution layer and the weight parameters of each deconvolution layer are determined respectively; including:

[0056] When the shared parameter of a convolutional layer or deconvolutional layer is w sh , the specific parameter is w sp , the weight coefficient is α, then the weight parameter w ac for:

[0057] w ac =αw sh +(1-α)w sp .

[0058] In a second aspect, the embodiment of the present application provides an image restoration device for multi-task learning in a rainy and snowy environment, comprising:

[0059] An acquisition unit, used to acquire an RGB image captured by an infrared camera of the vehicle;

[0060] A classification unit is used to classify and detect the RGB image using a classification network to obtain a classification result; and determine whether the vehicle is in a rainy environment or a snowy environment according to the classification result;

[0061] A downsampling unit, configured to perform two-fold downsampling on the RGB image to obtain a first downsampling image; and perform four-fold downsampling on the RGB image to obtain a second downsampling image;

[0062] A processing unit is used to process the RGB image, the first down-sampled image, and the second down-sampled image using a pre-trained first image restoration backbone network to obtain a feature map when the vehicle is in a rainy environment; and to process the feature map using a pre-trained first regression layer to obtain a restored image; or

[0063] When the vehicle is in a snowy environment, the RGB image, the first down-sampled image, and the second down-sampled image are processed using a pre-trained second image restoration backbone network to obtain a feature map; the feature map is processed using a pre-trained second regression layer to obtain a restored image.

[0064] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of the embodiment of the present application when executing the computer program.

[0065] This application effectively improves the imaging quality of infrared cameras in rainy and snowy weather, and improves the safety of self-driving cars. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0067] Figure 1 A flowchart of an image restoration method for multi-task learning in a rainy and snowy environment provided in an embodiment of the present application;

[0068] Figure 2 A schematic diagram of the structure of a first image restoration backbone network provided in an embodiment of the present application;

[0069] Figure 3 A schematic diagram of a cross-task fine-grained parameter adaptive sharing strategy provided in an embodiment of the present application;

[0070] Figure 4 A functional structure diagram of an image restoration device for multi-task learning in a rainy and snowy environment provided in an embodiment of the present application;

[0071] Figure 5 A functional structure diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0073] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0074] The technical solution provided in the embodiments of the present application is described below.

[0075] like Figure 1 As shown, the present application implements a method for image restoration using multi-task learning in a rainy and snowy environment, comprising:

[0076] Step 101: Obtain an RGB image captured by the infrared camera of the vehicle;

[0077] Among them, the RGB image is a three-channel image with a size of: W×H, W is the width of the image, and H is the height of the image;

[0078] Step 102: using a classification network to perform classification detection on the RGB image to obtain a classification result; determining whether the vehicle is in a rainy environment or a snowy environment according to the classification result;

[0079] According to the current research results on atmospheric imaging models, rainy days and snowy days have certain similarities in essence for image restoration tasks. In this embodiment, two networks with the same structure but different parameters are used to process the RGB images collected in the rainy environment and the RGB images collected in the snowy environment respectively.

[0080] Step 103: down-sampling the RGB image by a factor of two to obtain a first down-sampled image; down-sampling the RGB image by a factor of four to obtain a second down-sampled image;

[0081] Among them, the size of the first downsampled image is The size of the second downsampled image is

[0082] Step 104: When the vehicle is in a rainy environment, the RGB image, the first downsampled image, and the second downsampled image are processed using a pre-trained first image restoration backbone network to obtain a feature map; the feature map is processed using a pre-trained first regression layer to obtain a restored image; or,

[0083] When the vehicle is in a snowy environment, the RGB image, the first down-sampled image, and the second down-sampled image are processed using a pre-trained second image restoration backbone network to obtain a feature map; the feature map is processed using a pre-trained second regression layer to obtain a restored image.

[0084] The first image restoration backbone network and the second image restoration backbone network have the same structure but different parameters, and the parameters of the two networks are jointly trained. The first regression layer and the second regression layer have the same structure but different parameters.

[0085] This embodiment effectively uses multi-task learning technology to prompt the neural network to encode common attributes between rainy and snowy days, thereby effectively improving the imaging quality of the camera in rainy and snowy weather and ensuring the safe operation of the autonomous driving car.

[0086] In some embodiments, Figure 2 As shown, the RGB image, the first down-sampled image and the second down-sampled image are processed using a pre-trained first image restoration backbone network to obtain a feature map, including:

[0087] The first convolution layer is used to extract features from the RGB image to obtain a first feature map F1 of 128 dimensions; the second convolution layer is used to extract features from the first downsampled image to obtain a second feature map F2 of 128 dimensions; the third convolution layer is used to extract features from the second downsampled image to obtain a third feature map F3 of 128 dimensions;

[0088] The first feature map F1 is processed by the fourth convolutional layer to obtain the fourth feature map F 12 ; Use a fifth convolutional layer with a step size of 2 to convolution the fourth feature map F 12 Processing is performed to obtain a fifth feature map; the fifth feature map and the second feature map F2 are concatenated in the channel dimension to obtain a sixth feature map with a dimension of 256;

[0089] The sixth convolutional layer is used to reduce the dimension of the sixth feature map to obtain the seventh feature map F with a dimension of 128. 22 ;

[0090] The seventh feature map F is convolutionally transformed using a seventh convolutional layer with a stride of 2. 22 Processing is performed to obtain an eighth feature map, and the eighth feature map and the third feature map F3 are spliced ​​in the channel dimension to obtain a ninth feature map;

[0091] The eighth convolutional layer is used to reduce the dimension of the ninth feature map to obtain the tenth feature map F with a dimension of 128. 32 ;

[0092] The tenth feature map F is processed using a first deconvolution layer with a step size of 2. 32 Processing is performed to obtain the eleventh feature map F 33 , the eleventh feature map F 33 Concatenate the third feature map F3 with the third feature map F3 in the channel dimension to obtain the twelfth feature map;

[0093] The twelfth feature map is processed by reducing the dimension by using the ninth convolutional layer to obtain the thirteenth feature map;

[0094] The second deconvolution layer with a step size of 2 is used to improve the resolution of the thirteenth feature map to obtain the fourteenth feature map F 23 ; The fourteenth feature map F 23 Concatenate the fourth feature map with the fourth feature map in the channel dimension to obtain the fifteenth feature map;

[0095] The tenth convolutional layer is used to reduce the dimension of the fifteenth feature map to obtain a feature map with a dimension of 128.

[0096] In some embodiments, the RGB image, the first downsampled image, and the second downsampled image are processed using a pre-trained second image restoration backbone network to obtain a feature map, and obtaining the feature map includes:

[0097] The eleventh convolutional layer is used to extract features from the RGB image to obtain a sixteenth feature map of 128 dimensions; the twelfth convolutional layer is used to extract features from the first downsampled image to obtain a seventeenth feature map of 128 dimensions; the thirteenth convolutional layer is used to extract features from the second downsampled image to obtain an eighteenth feature map of 128 dimensions;

[0098] The sixteenth feature map is processed by the fourteenth convolution layer to obtain the nineteenth feature map; the nineteenth feature map is processed by the fifteenth convolution layer with a step size of 2 to obtain the twentieth feature map; the twentieth feature map and the seventeenth feature map are concatenated in the channel dimension to obtain the twenty-first feature map with a dimension of 256;

[0099] The 21st feature map is processed by reducing the dimension by using the 16th convolutional layer to obtain the 22nd feature map with a dimension of 128;

[0100] The twenty-second feature map is processed using a seventeenth convolutional layer with a step size of 2 to obtain a twenty-third feature map, and the twenty-third feature map and the eighteenth feature map are concatenated in the channel dimension to obtain a twenty-fourth feature map;

[0101] The 24th feature map is processed by reducing the dimension by using the 18th convolutional layer to obtain the 25th feature map with a dimension of 128;

[0102] The twenty-fifth feature map is processed by a third deconvolution layer with a step size of 2 to obtain a twenty-sixth feature map, and the twenty-sixth feature map and the eighteenth feature map are concatenated in the channel dimension to obtain a twenty-seventh feature map;

[0103] The 27th feature map is processed by reducing the dimension by using the 19th convolutional layer to obtain the 28th feature map;

[0104] A fourth deconvolution layer with a step size of 2 is used to increase the resolution of the twenty-eighth feature map to obtain a twenty-ninth feature map; the twenty-ninth feature map and the nineteenth feature map are concatenated in the channel dimension to obtain a thirtieth feature map;

[0105] The twentieth convolutional layer is used to reduce the dimension of the thirtieth feature map to obtain a feature map with a dimension of 128.

[0106] In some embodiments, the first regression layer includes two connected twenty-first convolutional layers and twenty-second convolutional layers;

[0107] The feature map is processed using the pre-trained first regression layer to obtain the restored image, including:

[0108] The 21st convolutional layer is used to reduce the dimension of the feature map to obtain an intermediate feature map with a dimension of 64;

[0109] The 22nd convolutional layer is used to reduce the dimension of the intermediate feature map with a dimension of 64 to obtain the restored image.

[0110] In some embodiments, the second regression layer includes two connected twenty-third convolutional layers and twenty-fourth convolutional layers;

[0111] The feature map is processed using the pre-trained second regression layer to obtain the restored image, including:

[0112] The 23rd convolutional layer is used to reduce the dimension of the feature map to obtain an intermediate feature map with a dimension of 64;

[0113] The 24th convolutional layer is used to reduce the dimension of the intermediate feature map with a dimension of 64 to obtain the restored image.

[0114] In some embodiments, the method further includes: a step of jointly training the first image restoration backbone network, the first regression layer, the second image restoration backbone network, and the second regression layer.

[0115] In this embodiment, the rainy day dataset Outdoorrain and the snowy day dataset Snow100K are used for multi-task joint training. It is crucial to give the network the ability to autonomously judge which parameters tend to be shared and which tend not to be shared, so that the network can flexibly encode common features between different datasets during training. This embodiment proposes a cross-task fine-grained parameter adaptive sharing strategy for the same convolutional layer and the same deconvolutional layer of the two image restoration backbone networks.

[0116] like Figure 3 As shown, taking a convolutional layer in the image restoration backbone network as an example, the initial weight is regarded as a cross-task shared parameter, which is used to encode common attributes between different tasks. out represents the output dimension, C in represents the input dimension, and s represents the size of the convolution kernel. Subsequently, an additional task-specific parameter is defined Used to encode properties specific to each task. Considering the parameter w sh and w sp Each element in may encode scene-general or scene-specific attributes, defining a learnable weighting coefficient And use it to integrate w sh and wsp , generate activation parameters w ac Forward propagation for participating networks:

[0117] w ac =αw sh +(1-α)w sp

[0118] During the network back propagation process, w sh 、w sp and α can be optimized directly. Note that only the activation parameter w ac Participates in the inference phase after training. Therefore, only the activation parameter w needs to be saved ac , without saving w sh and w sp , the number of network parameters will not increase significantly. In addition, during the training process, the distributed data parallel technology (DDP) is used to share the parameters w across tasks. sh Assigned to the global communication group to ensure that during the training process, w on different GPUs sh The initial value and gradient of are kept consistent.

[0119] Specifically, the training steps include:

[0120] Constructing a training network, including a first branch and a second branch in parallel, wherein the first branch includes a first image restoration backbone network and a first regression layer, and the second branch includes a second image restoration backbone network and a second regression layer;

[0121] Establishing a rainy day data set and a snowy day data set; the rainy day data set includes N first RGB image samples collected on rainy days; the snowy day data set includes N second RGB image samples collected on snowy days;

[0122] Downsampling the first RGB image sample by two times to obtain a first downsampled image sample; downsampling the first RGB image sample by four times to obtain a second downsampled image sample;

[0123] Downsampling the second RGB image sample by a factor of two to obtain a third downsampled image sample; downsampling the second RGB image sample by a factor of four to obtain a fourth downsampled image sample;

[0124] Using the first image restoration backbone network to process the first RGB image sample, the first downsampled image sample, and the second downsampled image sample to obtain a first feature map sample; using the first regression layer to process the first feature map sample to obtain a first restored image sample;

[0125] Based on the first restored image sample and the first RGB image sample, a first loss value is determined: wherein the restored image F is calculated resand the true value image F gt The absolute error of each pixel between is taken as the loss value L:

[0126]

[0127] Using the second image restoration backbone network to process the second RGB image sample, the third down-sampled image sample and the fourth down-sampled image sample to obtain a second feature map sample; using the second regression layer to process the second feature map sample to obtain a second restored image sample;

[0128] Determining a second loss value based on the second restored image sample and the second RGB image sample;

[0129] Using the first loss value and the second loss value, a cross-task fine-grained parameter adaptive sharing strategy is adopted to update the weight parameters of the first image restoration backbone network and the second image restoration backbone network, and the weight parameters of the first regression layer and the second regression layer are updated at the same time.

[0130] Furthermore, the first loss value and the second loss value are used to update the weight parameters of the first image restoration backbone network and the second image restoration backbone network by adopting a cross-task fine-grained parameter adaptive sharing strategy, and the weight parameters of the first regression layer and the second regression layer are updated at the same time, including:

[0131] The weight parameters of each convolution layer and deconvolution layer of the first image restoration backbone network are divided into shared parameters and specific parameters; the weight parameters of each convolution layer and deconvolution layer of the second image restoration backbone network are divided into shared parameters and specific parameters;

[0132] Using the first loss value, updating the shared parameters and specific parameters of each convolution layer in the first image restoration backbone network, the shared parameters and specific parameters of each deconvolution layer, and the weight parameters of the two convolution layers of the first regression layer;

[0133] Using the second loss value, updating the shared parameters and specific parameters of each convolution layer and the shared parameters and specific parameters of each deconvolution layer in the second image restoration backbone network, and the weight parameters of two convolution layers of the second regression layer, wherein the shared parameters of the convolution layer and the deconvolution layer at the same position in the first image restoration backbone network and the second image restoration backbone network are the same;

[0134] Determine weight parameters of each convolution layer and weight parameters of each deconvolution layer according to shared parameters and specific parameters of each convolution layer and shared parameters and specific parameters of each deconvolution layer in the first image restoration backbone network;

[0135] According to the shared parameters and specific parameters of each convolution layer and the shared parameters and specific parameters of each deconvolution layer in the second image restoration backbone network, the weight parameters of each convolution layer and the weight parameters of each deconvolution layer are determined respectively.

[0136] In some embodiments, according to the shared parameters and specific parameters of each convolution layer and the shared parameters and specific parameters of each deconvolution layer in the first image restoration backbone network, the weight parameters of each convolution layer and the weight parameters of each deconvolution layer are determined respectively; including:

[0137] When the shared parameter of a convolutional layer or deconvolutional layer is w sh , the specific parameter is w sp , the weight coefficient is α, then the weight parameter w ac for:

[0138] w ac =αw sh +(1-α)w sp .

[0139] Based on the above embodiments, the present application embodiment provides an image restoration device for multi-task learning in a rainy and snowy environment. Figure 4 As shown, the image restoration device 200 for multi-task learning in a complex rainy and snowy environment provided by the embodiment of the present application at least includes:

[0140] An acquisition unit 201 is used to acquire an RGB image captured by an infrared camera of the vehicle;

[0141] A classification unit 202 is used to classify and detect the RGB image using a classification network to obtain a classification result; and determine whether the vehicle is in a rainy environment or a snowy environment according to the classification result;

[0142] The down-sampling unit 203 is used to perform two-fold down-sampling on the RGB image to obtain a first down-sampled image; and perform four-fold down-sampling on the RGB image to obtain a second down-sampled image;

[0143] The processing unit 204 is used to process the RGB image, the first down-sampled image and the second down-sampled image using a pre-trained first image restoration backbone network to obtain a feature map when the vehicle is in a rainy environment; and process the feature map using a pre-trained first regression layer to obtain a restored image; or

[0144] When the vehicle is in a snowy environment, the RGB image, the first down-sampled image, and the second down-sampled image are processed using a pre-trained second image restoration backbone network to obtain a feature map; the feature map is processed using a pre-trained second regression layer to obtain a restored image.

[0145] It should be noted that the principle of solving the technical problem of the image restoration device 200 for multi-task learning in a rainy and snowy environment provided in the embodiment of the present application is similar to the method provided in the embodiment of the present application. Therefore, the implementation of the image restoration device 200 for multi-task learning in a rainy and snowy environment provided in the embodiment of the present application can refer to the implementation of the method provided in the embodiment of the present application, and the repeated parts will not be repeated.

[0146] Based on the above embodiments, the present application also provides an electronic device, referring to Figure 5 As shown, the electronic device 300 provided in the embodiment of the present application includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, the image restoration method of multi-task learning in a rainy and snowy environment provided in the embodiment of the present application is implemented.

[0147] The electronic device 300 provided in the embodiment of the present application may further include a bus 303 connecting different components (including the processor 301 and the memory 302). The bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and the like.

[0148] The memory 302 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 3021 and / or a cache memory 3022 , and may further include a read-only memory (ROM) 3023 .

[0149] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, the program modules 3024 including but not limited to: an operating subsystem, one or more applications, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0150] The electronic device 300 may also communicate with one or more external devices 304 (e.g., keyboards, remote controls, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 300 (e.g., mobile phones, computers, etc.), and / or communicate with any device that enables the electronic device 300 to communicate with one or more other electronic devices 300 (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 305. Furthermore, the electronic device 300 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 306. Figure 5 As shown, the network adapter 306 communicates with other modules of the electronic device 300 via the bus 303. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) subsystems, tape drives, and data backup storage subsystems.

[0151] It should be noted that Figure 5 The electronic device 300 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0152] The embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions, and when the computer instructions are executed by the processor, the image restoration method of multi-task learning in a rainy and snowy environment provided by the embodiment of the present application is implemented. Specifically, the executable program can be built-in or installed in the electronic device 300, so that the electronic device 300 can implement the image restoration method of multi-task learning in a rainy and snowy environment provided by the embodiment of the present application by executing the built-in or installed executable program.

[0153] The image restoration method of multi-task learning in a rainy and snowy environment provided in an embodiment of the present application can also be implemented as a program product, which includes a program code. When the program product can be run on an electronic device 300, the program code is used to enable the electronic device 300 to execute the image restoration method of multi-task learning in a rainy and snowy environment provided in an embodiment of the present application.

[0154] The program product provided in the embodiments of the present application may adopt any combination of one or more readable media, wherein the readable medium may be a readable signal medium or a readable storage medium, and the readable storage medium may be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, RAM, ROM, Erasable Programmable Read Only Memory (EPROM), optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0155] The program product provided in the embodiment of the present application may adopt a CD-ROM and include program code, and may also be run on a computing device. However, the program product provided in the embodiment of the present application is not limited thereto. In the embodiment of the present application, the readable storage medium may be any tangible medium containing or storing a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.

[0156] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0157] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application is described in detail with reference to the embodiments, a person skilled in the art should understand that any modification or equivalent replacement of the technical solution of the present application does not depart from the spirit and scope of the technical solution of the present application and should be included in the scope of the claims of the present application.

Claims

1. A multi-task learning image restoration method in a rainy and snowy environment, characterized in that: include: Get the RGB image captured by the vehicle's infrared camera; Using a classification network to perform classification detection on the RGB image to obtain a classification result; Determine whether the vehicle is in a rainy environment or a snowy environment according to the classification result; Down-sample the RGB image by a factor of two to obtain a first down-sampled image; Downsample the RGB image by four times to obtain a second downsampled image; When the vehicle is in a rainy environment, the RGB image, the first downsampled image, and the second downsampled image are processed using a pre-trained first image restoration backbone network to obtain a feature map; the feature map is processed using a pre-trained first regression layer to obtain a restored image; or, When the vehicle is in a snowy environment, the RGB image, the first down-sampled image, and the second down-sampled image are processed using a pre-trained second image restoration backbone network to obtain a feature map; the feature map is processed using a pre-trained second regression layer to obtain a restored image.

2. The method according to claim 1, characterized in that The RGB image, the first downsampled image, and the second downsampled image are processed using a pre-trained first image restoration backbone network to obtain a feature map, including: The first convolution layer is used to extract features from the RGB image to obtain a first feature map of 128 dimensions; the second convolution layer is used to extract features from the first downsampled image to obtain a second feature map of 128 dimensions; the third convolution layer is used to extract features from the second downsampled image to obtain a third feature map of 128 dimensions; The first feature map is processed by a fourth convolutional layer to obtain a fourth feature map; the fourth feature map is processed by a fifth convolutional layer with a step size of 2 to obtain a fifth feature map; the fifth feature map and the second feature map are concatenated in the channel dimension to obtain a sixth feature map with a dimension of 256; The sixth feature map is processed by reducing the dimension by using the sixth convolutional layer to obtain a seventh feature map with a dimension of 128; The seventh feature map is processed by a seventh convolutional layer with a step size of 2 to obtain an eighth feature map, and the eighth feature map and the third feature map are concatenated in the channel dimension to obtain a ninth feature map; The eighth convolutional layer is used to reduce the dimension of the ninth feature map to obtain a tenth feature map with a dimension of 128; The tenth feature map is processed by a first deconvolution layer with a step size of 2 to obtain an eleventh feature map, and the eleventh feature map and the third feature map are concatenated in the channel dimension to obtain a twelfth feature map; The twelfth feature map is processed by reducing the dimension by using the ninth convolutional layer to obtain the thirteenth feature map; The thirteenth feature map is processed by improving the resolution using a second deconvolution layer with a step size of 2 to obtain a fourteenth feature map; the fourteenth feature map and the fourth feature map are concatenated in the channel dimension to obtain a fifteenth feature map; The tenth convolutional layer is used to reduce the dimension of the fifteenth feature map to obtain a feature map with a dimension of 128.

3. The method according to claim 2, characterized in that The RGB image, the first down-sampled image, and the second down-sampled image are processed using a pre-trained second image restoration backbone network to obtain a feature map, and the feature map includes: The eleventh convolutional layer is used to extract features from the RGB image to obtain a sixteenth feature map of 128 dimensions; the twelfth convolutional layer is used to extract features from the first downsampled image to obtain a seventeenth feature map of 128 dimensions; the thirteenth convolutional layer is used to extract features from the second downsampled image to obtain an eighteenth feature map of 128 dimensions; The sixteenth feature map is processed by the fourteenth convolution layer to obtain the nineteenth feature map; the nineteenth feature map is processed by the fifteenth convolution layer with a step size of 2 to obtain the twentieth feature map; the twentieth feature map and the seventeenth feature map are concatenated in the channel dimension to obtain the twenty-first feature map with a dimension of 256; The 21st feature map is processed by reducing the dimension by using the 16th convolutional layer to obtain the 22nd feature map with a dimension of 128; The twenty-second feature map is processed using a seventeenth convolutional layer with a step size of 2 to obtain a twenty-third feature map, and the twenty-third feature map and the eighteenth feature map are concatenated in the channel dimension to obtain a twenty-fourth feature map; The 24th feature map is processed by reducing the dimension by using the 18th convolutional layer to obtain the 25th feature map with a dimension of 128; The twenty-fifth feature map is processed by a third deconvolution layer with a step size of 2 to obtain a twenty-sixth feature map, and the twenty-sixth feature map and the eighteenth feature map are concatenated in the channel dimension to obtain a twenty-seventh feature map; The 27th feature map is processed by reducing the dimension by using the 19th convolutional layer to obtain the 28th feature map; The 28th feature map is processed to improve the resolution by using a fourth deconvolution layer with a step size of 2 to obtain a 29th feature map; the 29th feature map and the 19th feature map are concatenated in the channel dimension to obtain a 30th feature map; The twentieth convolutional layer is used to reduce the dimension of the thirtieth feature map to obtain a feature map with a dimension of 128.

4. The method according to claim 3, characterized in that The first regression layer includes two connected twenty-first convolutional layers and a twenty-second convolutional layer; The feature map is processed using the pre-trained first regression layer to obtain the restored image, including: The 21st convolutional layer is used to reduce the dimension of the feature map to obtain an intermediate feature map with a dimension of 64; The 22nd convolutional layer is used to reduce the dimension of the intermediate feature map with a dimension of 64 to obtain the restored image.

5. The method according to claim 4, characterized in that The second regression layer includes two connected twenty-third convolutional layers and a twenty-fourth convolutional layer; The feature map is processed using the pre-trained second regression layer to obtain the restored image, including: The 23rd convolutional layer is used to reduce the dimension of the feature map to obtain an intermediate feature map with a dimension of 64; The 24th convolutional layer is used to reduce the dimension of the intermediate feature map with a dimension of 64 to obtain the restored image.

6. The method according to claim 5, characterized in that The method further includes: a step of jointly training the first image restoration backbone network, the first regression layer, the second image restoration backbone network and the second regression layer, specifically including: Constructing a training network, including a first branch and a second branch in parallel, wherein the first branch includes a first image restoration backbone network and a first regression layer, and the second branch includes a second image restoration backbone network and a second regression layer; Establishing a rainy day data set and a snowy day data set; the rainy day data set includes N first RGB image samples collected on rainy days; the snowy day data set includes N second RGB image samples collected on snowy days; Downsampling the first RGB image sample by two times to obtain a first downsampled image sample; downsampling the first RGB image sample by four times to obtain a second downsampled image sample; Downsampling the second RGB image sample by a factor of two to obtain a third downsampled image sample; downsampling the second RGB image sample by a factor of four to obtain a fourth downsampled image sample; Using the first image restoration backbone network to process the first RGB image sample, the first downsampled image sample, and the second downsampled image sample to obtain a first feature map sample; using the first regression layer to process the first feature map sample to obtain a first restored image sample; Determining a first loss value based on the first restored image sample and the first RGB image sample; Using the second image restoration backbone network to process the second RGB image sample, the third down-sampled image sample and the fourth down-sampled image sample to obtain a second feature map sample; using the second regression layer to process the second feature map sample to obtain a second restored image sample; Determining a second loss value based on the second restored image sample and the second RGB image sample; Using the first loss value and the second loss value, a cross-task fine-grained parameter adaptive sharing strategy is adopted to update the weight parameters of the first image restoration backbone network and the second image restoration backbone network, and the weight parameters of the first regression layer and the second regression layer are updated at the same time.

7. The method according to claim 6, characterized in that Using the first loss value and the second loss value, a cross-task fine-grained parameter adaptive sharing strategy is adopted to update the weight parameters of the first image restoration backbone network and the second image restoration backbone network, and the weight parameters of the first regression layer and the second regression layer are updated at the same time, including: The weight parameters of each convolution layer and deconvolution layer of the first image restoration backbone network are divided into shared parameters and specific parameters; the weight parameters of each convolution layer and deconvolution layer of the second image restoration backbone network are divided into shared parameters and specific parameters; Using the first loss value, updating the shared parameters and specific parameters of each convolution layer in the first image restoration backbone network, the shared parameters and specific parameters of each deconvolution layer, and the weight parameters of the two convolution layers of the first regression layer; Using the second loss value, updating the shared parameters and specific parameters of each convolution layer and the shared parameters and specific parameters of each deconvolution layer in the second image restoration backbone network, and the weight parameters of two convolution layers of the second regression layer, wherein the shared parameters of the convolution layer and the deconvolution layer at the same position in the first image restoration backbone network and the second image restoration backbone network are the same; Determine weight parameters of each convolution layer and weight parameters of each deconvolution layer according to shared parameters and specific parameters of each convolution layer and shared parameters and specific parameters of each deconvolution layer in the first image restoration backbone network; According to the shared parameters and specific parameters of each convolution layer and the shared parameters and specific parameters of each deconvolution layer in the second image restoration backbone network, the weight parameters of each convolution layer and the weight parameters of each deconvolution layer are determined respectively.

8. The method according to claim 7, characterized in that According to the shared parameters and specific parameters of each convolution layer and the shared parameters and specific parameters of each deconvolution layer in the first image restoration backbone network, respectively determine the weight parameters of each convolution layer and the weight parameters of each deconvolution layer; including: When the shared parameter of a convolutional layer or deconvolutional layer is w sh , the specific parameter is w sp , the weight coefficient is α, then the weight parameter w ac for: w ac =αw sh +(1-a)w sp 。 9. A multi-task learning image restoration device in a rainy and snowy environment, characterized in that: include: An acquisition unit, used to acquire an RGB image captured by an infrared camera of the vehicle; A classification unit, used to classify and detect the RGB image using a classification network to obtain a classification result; Determine whether the vehicle is in a rainy environment or a snowy environment according to the classification result; A downsampling unit, used for downsampling the RGB image by a factor of two to obtain a first downsampled image; Downsample the RGB image by four times to obtain a second downsampled image; A processing unit is used to process the RGB image, the first down-sampled image, and the second down-sampled image using a pre-trained first image restoration backbone network to obtain a feature map when the vehicle is in a rainy environment; and to process the feature map using a pre-trained first regression layer to obtain a restored image; or, When the vehicle is in a snowy environment, the RGB image, the first down-sampled image, and the second down-sampled image are processed using a pre-trained second image restoration backbone network to obtain a feature map; the feature map is processed using a pre-trained second regression layer to obtain a restored image.

10. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

Images