Image restoration method and device for multi-task learning in rain and snow environment

Abstract

The application provides a method and device for image restoration in a rain and snow environment through multi-task learning, wherein the method comprises: performing double down-sampling and four times down-sampling on an RGB image collected by a self vehicle to obtain a first down-sampled image and a second down-sampled image; when the self vehicle is in a rainy environment, processing the RGB image, the first down-sampled image and the second down-sampled image by using a first image restoration backbone network trained in advance to obtain a feature map; processing the feature map by using a first regression layer trained in advance to obtain a restored image; or when the self vehicle is in a snowy environment, processing the RGB image, the first down-sampled image and the second down-sampled image by using a second image restoration backbone network trained in advance to obtain a feature map; processing the feature map by using a second regression layer trained in advance to obtain a restored image. The application effectively improves the imaging quality of an infrared camera in rainy and snowy weather.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a multi-task learning image restoration method and device in a rainy and snowy environment. BACKGROUND

[0002] Automatic driving is a highly complex and challenging task that requires systems to accurately, comprehensively, and reliably perceive and understand the surrounding environment. Visual cameras, as key sensors on autonomous vehicles, are responsible for capturing information about the surrounding scene, performing key functions such as pedestrian and vehicle detection, traffic sign recognition, and perception of the intentions of other road users. However, in outdoor open environments, adverse weather conditions such as rain and snow often occur, posing a serious challenge to the performance of autonomous vehicles. When raindrops or snowflakes adhere to the camera lens, they block part of the field of view, causing a significant decrease in image quality, directly affecting the accuracy of image processing algorithms, reducing the vehicle's ability to perceive the environment, and increasing the safety risks during driving.

[0003] Currently, there is no solution to effectively remove raindrops, snowflakes, and other disturbances from images and improve the imaging quality of autonomous vehicles in adverse weather conditions. SUMMARY

[0004] Therefore, the present application provides a multi-task learning image restoration method and device in a rainy and snowy environment to solve the above technical problems.

[0005] In a first aspect, the present application provides a multi-task learning image restoration method in a rainy and snowy environment, comprising:

[0006] obtaining an RGB image collected by an infrared camera of the ego vehicle;

[0007] using a classification network to perform classification detection on the RGB image to obtain a classification result; determining whether the ego vehicle is in a rainy environment or a snowy environment according to the classification result;

[0008] performing two times down-sampling on the RGB image to obtain a first down-sampled image; performing four times down-sampling on the RGB image to obtain a second down-sampled image;

[0009] when the ego vehicle is in a rainy environment, using a pre-trained first image restoration backbone network to process the RGB image, the first down-sampled image, and the second down-sampled image to obtain a feature map; using a pre-trained first regression layer to process the feature map to obtain a restored image; or,

[0010] When the ego vehicle is in a snowy environment, the RGB image, the first down-sampled image and the second down-sampled image are processed by using a second image restoration backbone network which is pre-trained to obtain a feature map, and the feature map is processed by using a second regression layer which is pre-trained to obtain a restored image.

[0011] Further, the RGB image, the first down-sampled image and the second down-sampled image are processed by using a first image restoration backbone network which is pre-trained to obtain a feature map, including:

[0012] The RGB image is processed by using a first convolutional layer to obtain a first feature map with a dimension of 128; the first down-sampled image is processed by using a second convolutional layer to obtain a second feature map with a dimension of 128; and the second down-sampled image is processed by using a third convolutional layer to obtain a third feature map with a dimension of 128;

[0013] The first feature map is processed by using a fourth convolutional layer to obtain a fourth feature map; the fourth feature map is processed by using a fifth convolutional layer with a step of 2 to obtain a fifth feature map; and the fifth feature map and the second feature map are spliced in a channel dimension to obtain a sixth feature map with a dimension of 256;

[0014] The sixth feature map is processed by using a sixth convolutional layer to reduce the dimension to obtain a seventh feature map with a dimension of 128;

[0015] The seventh feature map is processed by using a seventh convolutional layer with a step of 2 to obtain an eighth feature map, and the eighth feature map and the third feature map are spliced in a channel dimension to obtain a ninth feature map;

[0016] The ninth feature map is processed by using an eighth convolutional layer to reduce the dimension to obtain a tenth feature map with a dimension of 128;

[0017] The tenth feature map is processed by using a first deconvolutional layer with a step of 2 to obtain an eleventh feature map, and the eleventh feature map and the third feature map are spliced in a channel dimension to obtain a twelfth feature map;

[0018] The twelfth feature map is processed by using a ninth convolutional layer to reduce the dimension to obtain a thirteenth feature map;

[0019] The thirteenth feature map is processed by using a second deconvolutional layer with a step of 2 to increase the resolution to obtain a fourteenth feature map; and the fourteenth feature map and the fourth feature map are spliced in a channel dimension to obtain a fifteenth feature map;

[0020] The fifteenth feature map is processed by using a tenth convolutional layer to reduce the dimension to obtain a feature map with a dimension of 128.

[0021] Further, the RGB image, the first down-sampled image and the second down-sampled image are processed by using the second image recovery backbone network which is pre-trained to obtain a feature map, and the feature map comprises:

[0022] The RGB image is processed by using an eleventh convolutional layer to obtain a sixteenth feature map with a dimension of 128; the first down-sampled image is processed by using a twelfth convolutional layer to obtain a seventeenth feature map with a dimension of 128; and the second down-sampled image is processed by using a thirteenth convolutional layer to obtain an eighteenth feature map with a dimension of 128.

[0023] The sixteenth feature map is processed by using a fourteenth convolutional layer to obtain a nineteenth feature map; the nineteenth feature map is processed by using a fifteenth convolutional layer with a step of 2 to obtain a twentieth feature map; and the twentieth feature map and the seventeenth feature map are spliced in a channel dimension to obtain a twenty-first feature map with a dimension of 256.

[0024] The twenty-first feature map is processed by using a sixteenth convolutional layer to reduce the dimension to obtain a twenty-second feature map with a dimension of 128.

[0025] The twenty-second feature map is processed by using a seventeenth convolutional layer with a step of 2 to obtain a twenty-third feature map, and the twenty-third feature map and the eighteenth feature map are spliced in a channel dimension to obtain a twenty-fourth feature map.

[0026] The twenty-fourth feature map is processed by using an eighteenth convolutional layer to reduce the dimension to obtain a twenty-fifth feature map with a dimension of 128.

[0027] The twenty-fifth feature map is processed by using a third deconvolutional layer with a step of 2 to obtain a twenty-sixth feature map, and the twenty-sixth feature map and the eighteenth feature map are spliced in a channel dimension to obtain a twenty-seventh feature map.

[0028] The twenty-seventh feature map is processed by using a nineteenth convolutional layer to reduce the dimension to obtain a twenty-eighth feature map.

[0029] The twenty-eighth feature map is processed by using a fourth deconvolutional layer with a step of 2 to increase the resolution to obtain a twenty-ninth feature map, and the twenty-ninth feature map and the nineteenth feature map are spliced in a channel dimension to obtain a thirtieth feature map.

[0030] The thirtieth feature map is processed by using a twentieth convolutional layer to reduce the dimension to obtain a feature map with a dimension of 128.

[0031] Further, the first regression layer comprises two connected twenty-first and twenty-second convolutional layers.

[0032] The first regression layer is used to process the feature map to obtain a restored image, including:

[0033] The twenty-first convolutional layer is used to process the feature map to obtain an intermediate feature map with a dimension of 64;

[0034] The twenty-second convolutional layer is used to process the intermediate feature map with a dimension of 64 to obtain a restored image.

[0035] Further, the second regression layer includes a twenty-third convolutional layer and a twenty-fourth convolutional layer connected in series;

[0036] The second regression layer is used to process the feature map to obtain a restored image, including:

[0037] The twenty-third convolutional layer is used to process the feature map to obtain an intermediate feature map with a dimension of 64;

[0038] The twenty-fourth convolutional layer is used to process the intermediate feature map with a dimension of 64 to obtain a restored image.

[0039] Further, the method further includes the step of jointly training the first image restoration backbone network, the first regression layer, the second image restoration backbone network and the second regression layer, specifically including:

[0040] A training network is constructed, including a first branch and a second branch in parallel, the first branch including the first image restoration backbone network and the first regression layer, and the second branch including the second image restoration backbone network and the second regression layer;

[0041] A rainy day dataset and a snowy day dataset are established; the rainy day dataset includes N first RGB image samples collected in a rainy day; and the snowy day dataset includes N second RGB image samples collected in a snowy day;

[0042] The first RGB image sample is subjected to two times down-sampling to obtain a first down-sampled image sample; and the first RGB image sample is subjected to four times down-sampling to obtain a second down-sampled image sample;

[0043] The second RGB image sample is subjected to two times down-sampling to obtain a third down-sampled image sample; and the second RGB image sample is subjected to four times down-sampling to obtain a fourth down-sampled image sample;

[0044] The first image restoration backbone network is used to process the first RGB image sample, the first down-sampled image sample and the second down-sampled image sample to obtain a first feature map sample; and the first regression layer is used to process the first feature map sample to obtain a first restored image sample;

[0045] determine a first loss value based on the first restored image sample and the first RGB image sample;

[0046] process the second RGB image sample, the third down-sampled image sample and the fourth down-sampled image sample by using the second image restoration backbone network to obtain a second feature map sample; process the second feature map sample by using the second regression layer to obtain a second restored image sample;

[0047] determine a second loss value based on the second restored image sample and the second RGB image sample;

[0048] update the weight parameters of the first image restoration backbone network and the second image restoration backbone network and the weight parameters of the first regression layer and the second regression layer by using the cross-task fine-grained parameter adaptive sharing strategy based on the first loss value and the second loss value.

[0049] Further, update the weight parameters of the first image restoration backbone network and the second image restoration backbone network and the weight parameters of the first regression layer and the second regression layer by using the cross-task fine-grained parameter adaptive sharing strategy based on the first loss value and the second loss value, including:

[0050] divide the weight parameters of each convolutional layer and deconvolutional layer of the first image restoration backbone network into shared parameters and specific parameters; divide the weight parameters of each convolutional layer and deconvolutional layer of the second image restoration backbone network into shared parameters and specific parameters;

[0051] update the shared parameters and specific parameters of each convolutional layer and the shared parameters and specific parameters of each deconvolutional layer in the first image restoration backbone network and the weight parameters of the two convolutional layers of the first regression layer based on the first loss value;

[0052] update the shared parameters and specific parameters of each convolutional layer and the shared parameters and specific parameters of each deconvolutional layer in the second image restoration backbone network and the weight parameters of the two convolutional layers of the second regression layer based on the second loss value, wherein the shared parameters of the convolutional layers and the deconvolutional layers at the same position in the first image restoration backbone network and the second image restoration backbone network are the same;

[0053] determine the weight parameters of each convolutional layer and the weight parameters of each deconvolutional layer based on the shared parameters and specific parameters of each convolutional layer and the shared parameters and specific parameters of each deconvolutional layer in the first image restoration backbone network, respectively;

[0054] determine the weight parameters of each convolutional layer and the weight parameters of each deconvolutional layer based on the shared parameters and specific parameters of each convolutional layer and the shared parameters and specific parameters of each deconvolutional layer in the second image restoration backbone network, respectively.

[0055] Further, according to the shared parameters and the specific parameters of each convolution layer in the first image recovery backbone network and the shared parameters and the specific parameters of each deconvolution layer, the weight parameters of each convolution layer and the weight parameters of each deconvolution layer are determined respectively; comprising:

[0056] When the shared parameters of a convolution layer or a deconvolution layer are w sh , the specific parameters are w sp , and the weighting coefficient is α, the weight parameters w ac are:

[0057] w ac =αw sh +(1-α)w sp .

[0058] In a second aspect, the embodiments of the present application provide an image recovery device for multi-task learning in a rain and snow environment, comprising:

[0059] An acquisition unit is configured to acquire an RGB image collected by an infrared camera of a vehicle;

[0060] A classification unit is configured to perform classification detection on the RGB image by using a classification network to obtain a classification result, and determine whether the vehicle is in a rainy environment or a snowy environment according to the classification result;

[0061] A down-sampling unit is configured to perform two times down-sampling on the RGB image to obtain a first down-sampled image, and perform four times down-sampling on the RGB image to obtain a second down-sampled image;

[0062] A processing unit is configured to, when the vehicle is in the rainy environment, perform processing on the RGB image, the first down-sampled image and the second down-sampled image by using a first image recovery backbone network trained in advance to obtain a feature map, and perform processing on the feature map by using a first regression layer trained in advance to obtain a recovered image; or

[0063] When the vehicle is in the snowy environment, perform processing on the RGB image, the first down-sampled image and the second down-sampled image by using a second image recovery backbone network trained in advance to obtain a feature map, and perform processing on the feature map by using a second regression layer trained in advance to obtain a recovered image.

[0064] In a third aspect, the embodiments of the present application provide an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the method of the embodiments of the present application when executing the computer program.

[0065] The present application effectively improves the imaging quality of the infrared camera in rainy and snowy weather, and improves the safety of the autonomous vehicle. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0067] Figure 1 A flowchart illustrating the image restoration method for multi-task learning in rain and snow environments provided in this application embodiment;

[0068] Figure 2 A schematic diagram of the structure of the first image restoration backbone network provided in the embodiments of this application;

[0069] Figure 3 This is a schematic diagram of a cross-task fine-grained parameter adaptive sharing strategy provided in an embodiment of this application;

[0070] Figure 4 A functional structural diagram of the image restoration device for multi-task learning in rain and snow environments provided in the embodiments of this application;

[0071] Figure 5 A functional structure diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0073] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0074] The technical solutions provided in the embodiments of this application will be described below.

[0075] like Figure 1 As shown, this application provides an image restoration method for multi-task learning in rain and snow environments, including:

[0076] Step 101: Acquire the RGB image captured by the vehicle's infrared camera;

[0077] The RGB image is a three-channel image with a size of W×H, where W is the width of the image and H is the height of the image.

[0078] Step 102: Use a classification network to classify and detect the RGB image to obtain the classification result; determine whether the vehicle is in a rainy or snowy environment based on the classification result.

[0079] Current research on atmospheric imaging models indicates that rainy and snowy weather present certain similarities for image restoration tasks. In this embodiment, two networks with identical structures but different parameters are used to process RGB images acquired in rainy and snowy conditions, respectively.

[0080] Step 103: Downsample the RGB image by a factor of 2 to obtain the first downsampled image; downsample the RGB image by a factor of 4 to obtain the second downsampled image;

[0081] The size of the first downsampled image is The size of the second downsampled image is

[0082] Step 104: When the vehicle is in a rainy environment, the RGB image, the first downsampled image, and the second downsampled image are processed using a pre-trained first image recovery backbone network to obtain a feature map; the feature map is then processed using a pre-trained first regression layer to obtain the recovered image; or,

[0083] When the vehicle is in a snowy environment, the RGB image, the first downsampled image and the second downsampled image are processed by the pre-trained second image recovery backbone network to obtain a feature map; the feature map is then processed by the pre-trained second regression layer to obtain the recovered image.

[0084] The first and second image restoration backbone networks have the same structure but different parameters, and their parameters are jointly trained. The first and second regression layers also have the same structure but different parameters.

[0085] This embodiment effectively uses multi-task learning technology to enable neural networks to encode common attributes between rainy and snowy weather, thereby effectively improving the imaging quality of cameras in rainy and snowy weather and ensuring the safe operation of autonomous vehicles.

[0086] In some embodiments, such as Figure 2 As shown, the RGB image, the first downsampled image, and the second downsampled image are processed using a pre-trained first image recovery backbone network to obtain feature maps, including:

[0087] The first convolutional layer is used for feature extraction of the RGB image to obtain a first feature map F1 with a dimension of 128; the second convolutional layer is used for feature extraction of the first down-sampled image to obtain a second feature map F2 with a dimension of 128; and the third convolutional layer is used for feature extraction of the second down-sampled image to obtain a third feature map F3 with a dimension of 128;

[0088] The fourth convolutional layer is used for processing the first feature map F1 to obtain a fourth feature map F 12 ; the fifth convolutional layer with a step of 2 is used for processing the fourth feature map F 12 to obtain a fifth feature map; and the fifth feature map and the second feature map F2 are spliced in the channel dimension to obtain a sixth feature map with a dimension of 256;

[0089] The sixth convolutional layer is used for dimension reduction processing of the sixth feature map to obtain a seventh feature map F 22 with a dimension of 128;

[0090] The seventh convolutional layer with a step of 2 is used for processing the seventh feature map F 22 to obtain an eighth feature map; and the eighth feature map and the third feature map F3 are spliced in the channel dimension to obtain a ninth feature map;

[0091] The eighth convolutional layer is used for dimension reduction processing of the ninth feature map to obtain a tenth feature map F 32 with a dimension of 128;

[0092] The first deconvolutional layer with a step of 2 is used for processing the tenth feature map F 32 to obtain an eleventh feature map F 33 ; the eleventh feature map F 33 and the third feature map F3 are spliced in the channel dimension to obtain a twelfth feature map;

[0093] The ninth convolutional layer is used for dimension reduction processing of the twelfth feature map to obtain a thirteenth feature map;

[0094] The second deconvolutional layer with a step of 2 is used for resolution enhancement processing of the thirteenth feature map to obtain a fourteenth feature map F 23 ; the fourteenth feature map F 23 and the fourth feature map are spliced in the channel dimension to obtain a fifteenth feature map;

[0095] The tenth convolutional layer is used for dimension reduction processing of the fifteenth feature map to obtain a feature map with a dimension of 128.

[0096] In some embodiments, the RGB image, the first down-sampled image and the second down-sampled image are processed by using a second image restoration backbone network which is pre-trained to obtain a feature map, the feature map comprising:

[0097] The RGB image is processed by using an eleventh convolutional layer to obtain a sixteenth feature map with a dimension of 128; the first down-sampled image is processed by using a twelfth convolutional layer to obtain a seventeenth feature map with a dimension of 128; and the second down-sampled image is processed by using a thirteenth convolutional layer to obtain an eighteenth feature map with a dimension of 128.

[0098] The sixteenth feature map is processed by using a fourteenth convolutional layer to obtain a nineteenth feature map; the nineteenth feature map is processed by using a fifteenth convolutional layer with a step of 2 to obtain a twentieth feature map; and the twentieth feature map and the seventeenth feature map are spliced in a channel dimension to obtain a twenty-first feature map with a dimension of 256.

[0099] The twenty-first feature map is processed by using a sixteenth convolutional layer to obtain a twenty-second feature map with a dimension of 128.

[0100] The twenty-second feature map is processed by using a seventeenth convolutional layer with a step of 2 to obtain a twenty-third feature map, and the twenty-third feature map and the eighteenth feature map are spliced in a channel dimension to obtain a twenty-fourth feature map.

[0101] The twenty-fourth feature map is processed by using an eighteenth convolutional layer to obtain a twenty-fifth feature map with a dimension of 128.

[0102] The twenty-fifth feature map is processed by using a third deconvolutional layer with a step of 2 to obtain a twenty-sixth feature map, and the twenty-sixth feature map and the eighteenth feature map are spliced in a channel dimension to obtain a twenty-seventh feature map.

[0103] The twenty-seventh feature map is processed by using a nineteenth convolutional layer to obtain a twenty-eighth feature map.

[0104] The twenty-eighth feature map is processed by using a fourth deconvolutional layer with a step of 2 to obtain a twenty-ninth feature map, and the twenty-ninth feature map and the nineteenth feature map are spliced in a channel dimension to obtain a thirtieth feature map.

[0105] The thirtieth feature map is processed by using a twentieth convolutional layer to obtain a feature map with a dimension of 128.

[0106] In some embodiments, the first regression layer comprises two connected twenty-first and twenty-second convolutional layers.

[0107] The feature maps are processed using the pre-trained first regression layer to obtain the reconstructed image, including:

[0108] The 21st convolutional layer is used to reduce the dimensionality of the feature map, resulting in an intermediate feature map with a dimension of 64.

[0109] The 22nd convolutional layer is used to reduce the dimensionality of the intermediate feature map with a dimension of 64, resulting in the restored image.

[0110] In some embodiments, the second regression layer includes two connected twenty-third and twenty-fourth convolutional layers;

[0111] The feature maps are processed using a pre-trained second regression layer to obtain the recovered image, including:

[0112] The 23rd convolutional layer is used to reduce the dimensionality of the feature map, resulting in an intermediate feature map with a dimension of 64.

[0113] The 24th convolutional layer is used to reduce the dimensionality of the intermediate feature map with a dimension of 64, resulting in the restored image.

[0114] In some embodiments, the method further includes the step of jointly training a first image restoration backbone network, a first regression layer, a second image restoration backbone network, and a second regression layer.

[0115] In this embodiment, the Outdoorrain and Snow100K datasets are used for multi-task joint training. Enabling the network to autonomously determine which parameters tend to be shared and which do not, thus enabling the network to flexibly encode common features across different datasets during training, is crucial. This embodiment proposes a cross-task fine-grained parameter adaptive sharing strategy for sharing between identical convolutional and deconvolutional layers in two image restoration backbone networks.

[0116] like Figure 3 As shown, taking a convolutional layer in an image restoration backbone network as an example, the initial weights are... Considered as a cross-task shared parameter, used to encode common attributes across different tasks. Where C... out Indicates the output dimension, C in Let represent the input dimension, and s represent the size of the convolution kernel. Then, define an additional task-specific parameter. Used to encode attributes specific to each task. Considering the parameter w sh and w sp Each element in the dataset can encode either a scene-general property or a scene-specific property, defining a learnable weighting coefficient. and use it to integrate w sh and wsp , generate activation parameter w ac for forward propagation of participating networks:

[0117] w ac = αw sh + (1-α)w sp

[0118] During the backward propagation of the network, w sh , w sp and α can be directly optimized. It should be noted that only the activation parameter w ac participates in the inference stage after the end of training. Therefore, only the activation parameter w ac needs to be saved, and w sh and w sp do not need to be saved, and the amount of network parameters will not increase significantly. In addition, during the training process, the distributed data parallel technology (DDP) is used to distribute the shared parameters w sh across tasks to the global communication group, ensuring that the initial value and gradient of w sh on different GPUs remain consistent during the training process.

[0119] Specifically, the training steps include:

[0120] constructing a training network, including a parallel first branch and a second branch, the first branch including a first image restoration backbone network and a first regression layer, and the second branch including a second image restoration backbone network and a second regression layer;

[0121] establishing a rainy day dataset and a snowy day dataset; the rainy day dataset includes N first RGB image samples collected on a rainy day; the snowy day dataset includes N second RGB image samples collected on a snowy day;

[0122] performing two times down-sampling on the first RGB image sample to obtain a first down-sampled image sample; performing four times down-sampling on the first RGB image sample to obtain a second down-sampled image sample;

[0123] performing two times down-sampling on the second RGB image sample to obtain a third down-sampled image sample; performing four times down-sampling on the second RGB image sample to obtain a fourth down-sampled image sample;

[0124] processing the first RGB image sample, the first down-sampled image sample and the second down-sampled image sample by using the first image restoration backbone network to obtain a first feature map sample; processing the first feature map sample by using the first regression layer to obtain a first restored image sample;

[0125] determining a first loss value based on the first restored image sample and the first RGB image sample: wherein, the restored image F resand the ground truth image F gt the absolute error of each pixel as the loss value L:

[0126]

[0127] The second RGB image sample, the third down-sampled image sample and the fourth down-sampled image sample are processed by using the second image recovery backbone network to obtain a second feature map sample; the second feature map sample is processed by using the second regression layer to obtain a second recovery image sample;

[0128] Based on the second recovery image sample and the second RGB image sample, a second loss value is determined;

[0129] The first loss value and the second loss value are used to update the weight parameters of the first image recovery backbone network and the second image recovery backbone network by using the cross-task fine-grained parameter adaptive sharing strategy, and the weight parameters of the first regression layer and the second regression layer are updated.

[0130] Further, the first loss value and the second loss value are used to update the weight parameters of the first image recovery backbone network and the second image recovery backbone network by using the cross-task fine-grained parameter adaptive sharing strategy, and the weight parameters of the first regression layer and the second regression layer are updated, including:

[0131] The weight parameters of each convolutional layer and deconvolutional layer of the first image recovery backbone network are divided into shared parameters and specific parameters; the weight parameters of each convolutional layer and deconvolutional layer of the second image recovery backbone network are divided into shared parameters and specific parameters;

[0132] The shared parameters and specific parameters of each convolutional layer and the shared parameters and specific parameters of each deconvolutional layer in the first image recovery backbone network and the weight parameters of the two convolutional layers of the first regression layer are updated by using the first loss value;

[0133] The shared parameters and specific parameters of each convolutional layer and the shared parameters and specific parameters of each deconvolutional layer in the second image recovery backbone network and the weight parameters of the two convolutional layers of the second regression layer are updated by using the second loss value, wherein the shared parameters of the convolutional layers and the deconvolutional layers at the same position in the first image recovery backbone network and the second image recovery backbone network are the same;

[0134] The weight parameters of each convolutional layer and the weight parameters of each deconvolutional layer are determined according to the shared parameters and specific parameters of each convolutional layer and the shared parameters and specific parameters of each deconvolutional layer in the first image recovery backbone network;

[0135] According to the shared parameters and the specific parameters of each convolution layer in the second image recovery backbone network and the shared parameters and the specific parameters of each deconvolution layer, weight parameters of each convolution layer and weight parameters of each deconvolution layer are determined respectively.

[0136] In some embodiments, according to the shared parameters and the specific parameters of each convolution layer in the first image recovery backbone network and the shared parameters and the specific parameters of each deconvolution layer, weight parameters of each convolution layer and weight parameters of each deconvolution layer are determined respectively; comprising:

[0137] When the shared parameters of a convolution layer or a deconvolution layer are w sh , the specific parameters are w sp , and the weighting coefficient is a, the weight parameters w ac are:

[0138] w ac = a w sh + (1-a) w sp .

[0139] Based on the above embodiments, the embodiments of the present application provide an image recovery device for multi-task learning in a rain and snow environment, as shown in Figure 4 The image recovery device 200 for multi-task learning in a complex rain and snow environment provided by the embodiments of the present application at least comprises:

[0140] An acquisition unit 201 is configured to acquire an RGB image collected by an infrared camera of a vehicle;

[0141] A classification unit 202 is configured to perform classification detection on the RGB image by using a classification network to obtain a classification result; and determine whether the vehicle is in a rainy environment or a snowy environment according to the classification result;

[0142] A downsampling unit 203 is configured to perform two times downsampling on the RGB image to obtain a first downsampled image; and perform four times downsampling on the RGB image to obtain a second downsampled image;

[0143] A processing unit 204 is configured to, when the vehicle is in a rainy environment, perform processing on the RGB image, the first downsampled image and the second downsampled image by using a first image recovery backbone network which is trained in advance to obtain a feature map; and perform processing on the feature map by using a first regression layer which is trained in advance to obtain a recovered image; or,

[0144] When the vehicle is in a snowy environment, perform processing on the RGB image, the first downsampled image and the second downsampled image by using a second image recovery backbone network which is trained in advance to obtain a feature map; and perform processing on the feature map by using a second regression layer which is trained in advance to obtain a recovered image.

[0145] It should be noted that the rain and snow environment multi-task learning image restoration device 200 provided in the embodiments of the present application solves the technical problems in the same way as the method provided in the embodiments of the present application, and therefore, the implementation of the rain and snow environment multi-task learning image restoration device 200 provided in the embodiments of the present application can refer to the implementation of the method provided in the embodiments of the present application, and the repeated parts will not be described herein.

[0146] Based on the above embodiments, the embodiments of the present application also provide an electronic device, as shown in Figure 5 The electronic device 300 provided in the embodiments of the present application at least includes a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301, and the processor 301 implements the rain and snow environment multi-task learning image restoration method provided in the embodiments of the present application when executing the computer program.

[0147] The electronic device 300 provided in the embodiments of the present application can further include a bus 303 connecting different components (including the processor 301 and the memory 302). Among them, the bus 303 represents one or more of several bus structures, including a memory bus, a peripheral bus, a local bus, etc.

[0148] The memory 302 can include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 3021 and / or a cache memory 3022, and can further include a read-only memory (ROM) 3023.

[0149] The memory 302 can further include a program tool 3025 having a set of (at least one) program modules 3024, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of these examples or some combination thereof can include the implementation of a network environment.

[0150] The electronic device 300 can also communicate with one or more external devices 304 (such as a keyboard, a remote control, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 300 (such as a mobile phone, a computer, etc.), and / or communicate with any device that enables the electronic device 300 to communicate with one or more other electronic devices 300 (such as a router, a modem, etc.). Such communication can be carried out through an input / output (I / O) interface 305. Moreover, the electronic device 300 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 306. AsFigure 5 As shown, network adapter 306 communicates with other modules of electronic device 300 over bus 303. It will be appreciated that although not shown, other hardware and / or software modules could be used in conjunction with electronic device 300. These include, but are not limited to: microcode, device drivers, redundant processors, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) subsystems, tape drives, and data archival storage subsystems, etc. Figure 5 As shown, network adapter 306 communicates with other modules of electronic device 300 over bus 303. It will be appreciated that although not shown, other hardware and / or software modules could be used in conjunction with electronic device 300. These include, but are not limited to: microcode, device drivers, redundant processors, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) subsystems, tape drives, and data archival storage subsystems, etc.

[0151] It should be noted that, Figure 5 Electronic device 300 as shown is merely an example and should not limit the function and usage range of the embodiments of the present application.

[0152] The embodiments of the present application further provide a computer readable storage medium, which stores computer instructions, and the computer instructions are executed by a processor to implement the image restoration method in a rain and snow environment based on multi-task learning provided by the embodiments of the present application. Specifically, the executable program can be built-in or installed in the electronic device 300, so that the electronic device 300 can implement the image restoration method in a rain and snow environment based on multi-task learning provided by the embodiments of the present application by executing the built-in or installed executable program.

[0153] The image restoration method in a rain and snow environment based on multi-task learning provided by the embodiments of the present application can also be implemented as a program product, and the program product includes program codes, and the program codes are used to make the electronic device 300 execute the image restoration method in a rain and snow environment based on multi-task learning provided by the embodiments of the present application when the program product can run on the electronic device 300.

[0154] The program product provided by the embodiments of the present application can adopt any combination of one or more readable media, and the readable media can be readable signal media or readable storage media, and the readable storage media can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above, and more specifically, more specific examples (non-exhaustive list) of readable storage media include: electrical connections with one or more conductive wires, portable disks, hard disks, RAM, ROM, Erasable Programmable Read Only Memory (EPROM), optical fibers, portable Compact Disc Read-Only Memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0155] The program product provided by the embodiments of the present application can adopt a CD-ROM and include program codes, and can also run on a computing device. However, the program product provided by the embodiments of the present application is not limited to this, and in the embodiments of the present application, the readable storage medium can be any tangible medium containing or storing programs, which can be used by or in combination with an instruction execution system, device or apparatus.

[0156] It should be noted that, although several units or sub-units of the apparatus are mentioned in the foregoing detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into units for embodiment.

[0157] In addition, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all of the shown operations must be performed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, a plurality of steps can be combined into one step, and / or one step can be divided into a plurality of steps.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application is described in detail with reference to the embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and they should be covered in the scope of claims of the present application.

Claims

1. An image restoration method for multi-task learning in a rain and snow environment, characterized in that, The method comprises the following steps: acquiring an RGB image collected by an infrared camera of a vehicle; performing classification detection on the RGB image by using a classification network to obtain a classification result; determining whether the vehicle is in a rainy environment or a snowy environment according to the classification result; performing double down-sampling on the RGB image to obtain a first down-sampled image; performing four times down-sampling on the RGB image to obtain a second down-sampled image; when the vehicle is in the rainy environment, performing processing on the RGB image, the first down-sampled image and the second down-sampled image by using a first image restoration backbone network which is trained in advance to obtain a feature map, and performing processing on the feature map by using a first regression layer which is trained in advance to obtain a restored image; or, when the vehicle is in the snowy environment, performing processing on the RGB image, the first down-sampled image and the second down-sampled image by using a second image restoration backbone network which is trained in advance to obtain a feature map, and performing processing on the feature map by using a second regression layer which is trained in advance to obtain a restored image; the first image restoration backbone network and the second image restoration backbone network have the same structure but different parameters; performing processing on the RGB image, the first down-sampled image and the second down-sampled image by using the first image restoration backbone network which is trained in advance to obtain a feature map, which comprises the following steps: performing feature extraction on the RGB image by using a first convolutional layer to obtain a first feature map with a dimension of 128, performing feature extraction on the first down-sampled image by using a second convolutional layer to obtain a second feature map with a dimension of 128, and performing feature extraction on the second down-sampled image by using a third convolutional layer to obtain a third feature map with a dimension of 128; performing processing on the first feature map by using a fourth convolutional layer to obtain a fourth feature map, performing processing on the fourth feature map by using a fifth convolutional layer with a step of 2 to obtain a fifth feature map, and splicing the fifth feature map and the second feature map in a channel dimension to obtain a sixth feature map with a dimension of 256; performing dimension reduction processing on the sixth feature map by using a sixth convolutional layer to obtain a seventh feature map with a dimension of 128; performing processing on the seventh feature map by using a seventh convolutional layer with a step of 2 to obtain an eighth feature map, splicing the eighth feature map and the third feature map in a channel dimension to obtain a ninth feature map; performing dimension reduction processing on the ninth feature map by using an eighth convolutional layer to obtain a tenth feature map with a dimension of 128; performing processing on the tenth feature map by using a first deconvolutional layer with a step of 2 to obtain an eleventh feature map, splicing the eleventh feature map and the third feature map in a channel dimension to obtain a twelfth feature map; performing dimension reduction processing on the twelfth feature map by using a ninth convolutional layer to obtain a thirteenth feature map; performing resolution enhancement processing on the thirteenth feature map by using a second deconvolutional layer with a step of 2 to obtain a fourteenth feature map, and splicing the fourteenth feature map and the fourth feature map in a channel dimension to obtain a fifteenth feature map; performing dimension reduction processing on the fifteenth feature map by using a tenth convolutional layer to obtain a feature map with a dimension of 128. The method further comprises the step of jointly training the first image restoration backbone network, the first regression layer, the second image restoration backbone network and the second regression layer, specifically comprising: A training network is constructed, including a first branch and a second branch in parallel, the first branch including a first image restoration backbone network and a first regression layer, and the second branch including a second image restoration backbone network and a second regression layer; A rainy day dataset and a snowy day dataset are established; the rainy day dataset includes N first RGB image samples collected in rainy days; the snowy day dataset includes N second RGB image samples collected in snowy days; The first RGB image samples are subjected to two times down-sampling to obtain first down-sampled image samples, and four times down-sampling to obtain second down-sampled image samples; The second RGB image samples are subjected to two times down-sampling to obtain third down-sampled image samples, and four times down-sampling to obtain fourth down-sampled image samples; The first image restoration backbone network is used to process the first RGB image samples, the first down-sampled image samples and the second down-sampled image samples to obtain first feature map samples; the first regression layer is used to process the first feature map samples to obtain first restored image samples; Based on the first restored image samples and the first RGB image samples, a first loss value is determined; The second image restoration backbone network is used to process the second RGB image samples, the third down-sampled image samples and the fourth down-sampled image samples to obtain second feature map samples; the second regression layer is used to process the second feature map samples to obtain second restored image samples; Based on the second restored image samples and the second RGB image samples, a second loss value is determined; The first image restoration backbone network and the second image restoration backbone network are updated in a cross-task fine-grained parameter adaptive sharing strategy using the first loss value and the second loss value, and the weight parameters of the first regression layer and the second regression layer are simultaneously updated; The first image restoration backbone network and the second image restoration backbone network are updated in a cross-task fine-grained parameter adaptive sharing strategy using the first loss value and the second loss value, and the weight parameters of the first regression layer and the second regression layer are simultaneously updated, including: The weight parameters of each convolutional layer and deconvolutional layer of the first image restoration backbone network are divided into shared parameters and specific parameters; the weight parameters of each convolutional layer and deconvolutional layer of the second image restoration backbone network are divided into shared parameters and specific parameters; The shared parameters and specific parameters of each convolutional layer and deconvolutional layer in the first image restoration backbone network and the weight parameters of two convolutional layers of the first regression layer are updated using the first loss value; The shared parameters and specific parameters of each convolutional layer and deconvolutional layer in the second image restoration backbone network and the weight parameters of two convolutional layers of the second regression layer are updated using the second loss value, wherein the shared parameters of the convolutional layers and deconvolutional layers at the same position in the first image restoration backbone network and the second image restoration backbone network are the same; According to the shared parameters and the specific parameters of each convolutional layer in the first image recovery backbone network and the shared parameters and the specific parameters of each deconvolutional layer, weight parameters of each convolutional layer and weight parameters of each deconvolutional layer are determined respectively; According to the shared parameters and the specific parameters of each convolutional layer in the second image recovery backbone network and the shared parameters and the specific parameters of each deconvolutional layer, weight parameters of each convolutional layer and weight parameters of each deconvolutional layer are determined respectively; According to the shared parameters and the specific parameters of each convolutional layer in the first image recovery backbone network and the shared parameters and the specific parameters of each deconvolutional layer, weight parameters of each convolutional layer and weight parameters of each deconvolutional layer are determined respectively; comprising: When the shared parameters of a convolutional layer or deconvolutional layer are , the specific parameters are , and the weighting coefficients are , the weight parameters are: 。 2. The method of claim 1, wherein, The pre-trained second image recovery backbone network is used to process the RGB image, the first down-sampled image and the second down-sampled image to obtain a feature map, comprising: The eleventh convolutional layer is used to extract features of the RGB image to obtain a sixteenth feature map with a dimension of 128; the twelfth convolutional layer is used to extract features of the first down-sampled image to obtain a seventeenth feature map with a dimension of 128; and the thirteenth convolutional layer is used to extract features of the second down-sampled image to obtain an eighteenth feature map with a dimension of 128; The fourteenth convolutional layer is used to process the sixteenth feature map to obtain a nineteenth feature map; a fifteenth convolutional layer with a step of 2 is used to process the nineteenth feature map to obtain a twentieth feature map; and the twentieth feature map and the seventeenth feature map are spliced in the channel dimension to obtain a twenty-first feature map with a dimension of 256; The sixteenth convolutional layer is used to process the twenty-first feature map to reduce the dimension to obtain a twenty-second feature map with a dimension of 128; The seventeenth convolutional layer with a step of 2 is used to process the twenty-second feature map to obtain a twenty-third feature map; and the twenty-third feature map and the eighteenth feature map are spliced in the channel dimension to obtain a twenty-fourth feature map; The eighteenth convolutional layer is used to process the twenty-fourth feature map to reduce the dimension to obtain a twenty-fifth feature map with a dimension of 128; The third deconvolutional layer with a step of 2 is used to process the twenty-fifth feature map to obtain a twenty-sixth feature map; and the twenty-sixth feature map and the eighteenth feature map are spliced in the channel dimension to obtain a twenty-seventh feature map; The nineteenth convolutional layer is used to process the twenty-seventh feature map to reduce the dimension to obtain a twenty-eighth feature map; The fourth deconvolutional layer with a step of 2 is used to process the twenty-eighth feature map to increase the resolution to obtain a twenty-ninth feature map; and the twenty-ninth feature map and the nineteenth feature map are spliced in the channel dimension to obtain a thirtieth feature map; The twentieth convolutional layer is used to process the thirtieth feature map to reduce the dimension to obtain a feature map with a dimension of 128.

3. The method of claim 2, wherein, The first regression layer comprises two connected twenty-first and twenty-second convolutional layers; The pre-trained first regression layer is used to process the feature map to obtain a recovered image, comprising: The twenty-first convolutional layer is used to process the feature map to reduce the dimension to obtain an intermediate feature map with a dimension of 64; The twenty-first convolutional layer is used to process the feature map to reduce the dimension to obtain an intermediate feature map with a dimension of 64; The intermediate feature map with the dimension of 64 is processed by the twenty-second convolutional layer to reduce the dimension, and a restored image is obtained.

4. The method of claim 3, wherein, The second regression layer includes two connected twenty-third and twenty-fourth convolutional layers. The feature map is processed by the second regression layer which is pre-trained to obtain a restored image, including: The feature map is processed by the twenty-third convolutional layer to reduce the dimension, and an intermediate feature map with the dimension of 64 is obtained. The intermediate feature map with the dimension of 64 is processed by the twenty-fourth convolutional layer to reduce the dimension, and a restored image is obtained.

5. An image restoration device for multi-task learning in a rain and snow environment, characterized by, including: An acquisition unit is configured to acquire an RGB image collected by an infrared camera of a vehicle; A classification unit is configured to perform classification detection on the RGB image by using a classification network to obtain a classification result; It is determined that the vehicle is in a rainy environment or the vehicle is in a snowy environment according to the classification result; A down-sampling unit is configured to perform two times down-sampling on the RGB image to obtain a first down-sampled image; The RGB image is four times down-sampled to obtain a second down-sampled image; A processing unit is configured to, when the vehicle is in a rainy environment, process the RGB image, the first down-sampled image and the second down-sampled image by using a first image restoration backbone network which is pre-trained, to obtain a feature map; and process the feature map by using a first regression layer which is pre-trained to obtain a restored image. Or, When the vehicle is in a snowy environment, the RGB image, the first down-sampled image and the second down-sampled image are processed by using a second image restoration backbone network which is pre-trained to obtain a feature map; and the feature map is processed by using a second regression layer which is pre-trained to obtain a restored image. The first image restoration backbone network and the second image restoration backbone network have the same structure but different parameters. The first image restoration backbone network which is pre-trained is used to process the RGB image, the first down-sampled image and the second down-sampled image to obtain a feature map, including: The RGB image is processed by using a first convolutional layer to extract features, and a first feature map with the dimension of 128 is obtained; the first down-sampled image is processed by using a second convolutional layer to extract features, and a second feature map with the dimension of 128 is obtained; and the second down-sampled image is processed by using a third convolutional layer to extract features, and a third feature map with the dimension of 128 is obtained; The first feature map is processed by using a fourth convolutional layer to obtain a fourth feature map; the fourth feature map is processed by using a fifth convolutional layer with a step of 2 to obtain a fifth feature map; and the fifth feature map and the second feature map are spliced in the channel dimension to obtain a sixth feature map with the dimension of 256; The sixth feature map is processed by using a sixth convolutional layer to reduce the dimension, and a seventh feature map with the dimension of 128 is obtained; The seventh feature map is processed by using a seventh convolutional layer with a step of 2 to obtain an eighth feature map; the eighth feature map and the third feature map are spliced in the channel dimension to obtain a ninth feature map; The ninth feature map is processed by using an eighth convolutional layer to reduce the dimension, and a tenth feature map with the dimension of 128 is obtained. The tenth feature map is processed by using a first deconvolutional layer with a step size of 2 to obtain an eleventh feature map, and the eleventh feature map and the third feature map are spliced in the channel dimension to obtain a twelfth feature map; The twelfth feature map is processed by using a ninth convolutional layer to reduce the dimension to obtain a thirteenth feature map; The thirteenth feature map is processed by using a second deconvolutional layer with a step size of 2 to increase the resolution to obtain a fourteenth feature map; and the fourteenth feature map and the fourth feature map are spliced in the channel dimension to obtain a fifteenth feature map; The fifteenth feature map is processed by using a tenth convolutional layer to reduce the dimension to obtain a feature map with a dimension of 128; The device further comprises a training unit configured to jointly train the first image restoration backbone network, the first regression layer, the second image restoration backbone network and the second regression layer, and specifically configured to: A training network is constructed, including a first branch and a second branch in parallel, the first branch including the first image restoration backbone network and the first regression layer, and the second branch including the second image restoration backbone network and the second regression layer; A rainy day data set and a snowy day data set are established; the rainy day data set includes N first RGB image samples collected in a rainy day; and the snowy day data set includes N second RGB image samples collected in a snowy day; The first RGB image sample is down-sampled by two times to obtain a first down-sampled image sample; and the first RGB image sample is down-sampled by four times to obtain a second down-sampled image sample; The second RGB image sample is down-sampled by two times to obtain a third down-sampled image sample; and the second RGB image sample is down-sampled by four times to obtain a fourth down-sampled image sample; The first RGB image sample, the first down-sampled image sample and the second down-sampled image sample are processed by using the first image restoration backbone network to obtain a first feature map sample; and the first feature map sample is processed by using the first regression layer to obtain a first restored image sample; Based on the first restored image sample and the first RGB image sample, a first loss value is determined; The second RGB image sample, the third down-sampled image sample and the fourth down-sampled image sample are processed by using the second image restoration backbone network to obtain a second feature map sample; and the second feature map sample is processed by using the second regression layer to obtain a second restored image sample; Based on the second restored image sample and the second RGB image sample, a second loss value is determined; The first loss value and the second loss value are used to update the weight parameters of the first image restoration backbone network and the second image restoration backbone network by using a cross-task fine-grained parameter adaptive sharing strategy, and the weight parameters of the first regression layer and the second regression layer are updated at the same time; The first loss value and the second loss value are used to update the weight parameters of the first image restoration backbone network and the second image restoration backbone network by using a cross-task fine-grained parameter adaptive sharing strategy, and the weight parameters of the first regression layer and the second regression layer are updated at the same time, including: The weight parameters of each convolutional layer and deconvolutional layer of the first image restoration backbone network are divided into shared parameters and specific parameters; the weight parameters of each convolutional layer and deconvolutional layer of the second image restoration backbone network are divided into shared parameters and specific parameters; The shared parameters and specific parameters of each convolutional layer and the shared parameters and specific parameters of each deconvolutional layer in the first image restoration backbone network are updated by using the first loss value, and the weight parameters of the two convolutional layers of the first regression layer are updated; The shared parameters and specific parameters of each convolutional layer and the shared parameters and specific parameters of each deconvolutional layer in the second image restoration backbone network are updated by using the second loss value, and the weight parameters of the two convolutional layers of the second regression layer are updated, wherein the shared parameters of the convolutional layers and the deconvolutional layers at the same position in the first image restoration backbone network and the second image restoration backbone network are the same; The weight parameters of each convolutional layer and the weight parameters of each deconvolutional layer are determined according to the shared parameters and specific parameters of each convolutional layer and the shared parameters and specific parameters of each deconvolutional layer in the first image restoration backbone network; The weight parameters of each convolutional layer and the weight parameters of each deconvolutional layer are determined according to the shared parameters and specific parameters of each convolutional layer and the shared parameters and specific parameters of each deconvolutional layer in the second image restoration backbone network; The weight parameters of each convolutional layer and the weight parameters of each deconvolutional layer are determined according to the shared parameters and specific parameters of each convolutional layer and the shared parameters and specific parameters of each deconvolutional layer in the first image restoration backbone network; comprising: When the shared parameters of a convolutional layer or deconvolutional layer are , the specific parameters are , the weighting coefficients are , the weight parameters are: 。 6. An electronic device, comprising: comprising: A memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1-4 when executing the computer program.

Images