Image super-resolution reconstruction and dehazing method and system based on loss classification and dual-branch network

Abstract

The present invention relates to a method and system for image super-resolution reconstruction and defogging based on loss classification and a dual-branch network. The method comprises the following steps: S1, inputting an image to be super-resolution reconstructed or defogged and its corresponding real reference image, performing preprocessing, and cropping the image to be super-resolution reconstructed or defogged and the real reference image to the same size and position; S2, constructing a dual-branch network model for training, and using an encoder-decoder network to train the image using a loss classifier and a constraint function; S3, inputting the image to be super-resolution reconstructed or defogged into the trained dual-branch network model, and obtaining the final result image through feature fusion. The present invention adopts a dual-branch network structure and proposes a loss classifier method, so that the first branch of the dual branch focuses on smooth areas, while the second branch focuses on edge areas. This solves the shortcoming of the end-to-end network's poor performance in edge areas and improves the visual effect of the final result image.

Description

Technical Field

[0001] The present invention relates to the field of digital image processing and computer vision technology, and in particular to an image super-resolution reconstruction and defogging method and system based on loss classification and a dual-branch network. Background Art

[0002] Digital images are an information medium that people are exposed to all the time. Especially in the context of the mobile Internet era, digital images have become an indispensable part of high-quality content.

[0003] However, due to factors such as network transmission speed, imaging environment, recording equipment, transmission media, and processing methods, digital images will always be distorted to some extent. Therefore, the key issue is how to maximize image resolution and thereby improve image quality within limited hardware and transmission impairments. Super-resolution reconstruction, which transforms a low-resolution image into a corresponding high-resolution image through a specific algorithm, can overcome the resolution limitations of imaging hardware and compensate for pixel loss during image transmission and compression.

[0004] On the other hand, people often encounter foggy weather when taking photos outdoors. Digital images taken in such weather often cannot distinguish the scenery and objects behind the fog due to the obstruction of the fog. In addition, the fog may also cause image degradation problems such as low contrast, color shift, and poor visual effects. Therefore, how to remove the fog effect from foggy images and obtain clearer fog-free images is a major challenge in the field of computer vision.

[0005] Most of the existing mainstream image super-resolution reconstruction and image dehazing methods are end-to-end super-resolution reconstruction or dehazing networks for single images. Based on the success of deep learning in the field of image vision, these methods can often achieve good visual effects.

[0006] These end-to-end methods typically take an image as input and feed it into a trained network to produce a resulting image. While these resulting images often offer some visual quality in smooth areas, they often struggle with edge-rich regions. For example, in dehazing tasks, edges retain a foggy effect, while in image super-resolution, edges tend to become even more blurred. Previous networks typically achieved greater detail by deepening the network or adding dense connections. These methods consume more computing resources while also offering limited visual improvement. Summary of the Invention

[0007] In order to solve the technical problems existing in the prior art, the present invention provides an image super-resolution reconstruction and defogging method and system based on loss classification and a dual-branch network. The method can complete both the super-resolution reconstruction task and the image defogging task. The method has two branches. During the training process, the loss classifier is used to divide the image preliminarily super-resolution reconstructed or defogged in the first branch into a smooth part and an edge part, and a mask is generated. The smooth area of ​​the image obtained by the branch and the edge area of ​​the image obtained by the second branch can be extracted through the mask. The two are combined to form a better super-resolution reconstructed or defogged image. During the training process, the constraints on the combined image can naturally make the first branch in the dual-branch network have good super-resolution reconstruction or defogging capabilities for the smooth area, while ensuring that the second branch can also have a good visual effect on the edge area. Finally, the method further obtains the best super-resolution reconstructed or defogged image through feature fusion.

[0008] The method of the present invention is implemented by the following technical solution: an image super-resolution reconstruction and defogging method based on loss classification and a dual-branch network, comprising the following steps:

[0009] S1. Prepare the image to be super-resolution reconstructed or defogged and its corresponding real reference image, input the image and perform preprocessing, crop the image to be super-resolution reconstructed or defogged and the real reference image to the same size and position, and perform data augmentation.

[0010] S2. Build and train a dual-branch network model. The dual-branch network model uses an encoder-decoder network and is trained using a loss classifier and a constraint function.

[0011] S3. Input the image to be super-resolution reconstructed or dehazed into the trained two-branch network model, and obtain the final result image through feature fusion.

[0012] The system of the present invention adopts the following technical solutions to achieve: an image super-resolution reconstruction and defogging system based on loss classification and a dual-branch network, comprising:

[0013] The preprocessing module is used to input the prepared image to be super-resolution reconstructed or dehazed and its corresponding real reference image, and perform preprocessing, cropping the image to be super-resolution reconstructed or dehazed and the real reference image to the same size and position, and performing data enhancement operations;

[0014] The model training module is used to build and train a two-branch network model. The two-branch network model uses an encoder-decoder network and is trained through a loss classifier and constraint function.

[0015] The model application module is used to input the image to be super-resolution reconstructed or dehazed into the trained two-branch network model and obtain the final result image through feature fusion.

[0016] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0017] 1. This paper adopts a dual-branch network structure and innovatively proposes a loss classifier method, so that the first branch of the dual branch focuses on smooth areas, while the second branch focuses on edge areas. This solves the shortcoming of previous end-to-end networks that have poor performance in edge areas and improves the visual effect of the final result image.

[0018] 2. The present invention is applicable to a variety of tasks. Compared with previous networks that are only effective in image dehazing or image super-resolution reconstruction, the present invention can be used for both image dehazing and image super-resolution reconstruction to achieve better results. Secondly, in the application of image super-resolution reconstruction, any multiple of super-resolution reconstruction can be achieved. Compared with previous super-resolution reconstruction networks that are often limited to 2x or 4x super-resolution reconstruction due to network structure, the system provided by the present invention can perform 1x or even any multiple of super-resolution reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of the method of the present invention;

[0020] Figure 2 It is a flow chart of the specific steps of the method of the present invention;

[0021] Figure 3 It is a schematic diagram of the specific structure of one branch in the dual-branch network of the present invention;

[0022] Figure 4 It is a schematic diagram of the specific structure of each layer of the dual-branch network of the present invention;

[0023] Figure 5 Schematic diagram of the loss classifier of the present invention;

[0024] Figure 6 is a network diagram of the present invention during the training process;

[0025] Figure 7 It is a network diagram of the present invention during application;

[0026] Figure 8 It is a grayscale schematic diagram of the smoothed mask image Mask generated by the loss classifier during the training process of the image defogging system of the present invention;

[0027] Figure 9 (a) is a schematic diagram of fog;

[0028] Figure 9 (b) is the real reference image GT;

[0029] Figure 9 (c) is a schematic diagram of the dehazing output of the DCP method;

[0030] Figure 9 (d) is a schematic diagram of the dehazing output of the AOD method;

[0031] Figure 9 (e) is a schematic diagram of the dehazing output of the GDN method;

[0032] Figure 9 (f) is a schematic diagram of the defogging output of the method of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0034] Example

[0035] like Figure 1 、 Figure 2 As shown, the image super-resolution reconstruction and defogging method based on loss classification and a dual-branch network in this embodiment includes the following steps:

[0036] S1. Prepare the image to be super-resolution reconstructed or defogged and its corresponding real reference image, input the image and perform preprocessing, crop the image to be super-resolution reconstructed or defogged and the real reference image to the same size and position, and perform data augmentation.

[0037] S2. Build and train a dual-branch network model. The dual-branch network model uses an encoder-decoder network and is trained using a loss classifier and a constraint function.

[0038] S3. Input the image to be super-resolution reconstructed or dehazed into the trained two-branch network model, and obtain the final result image through feature fusion.

[0039] Specifically, in this embodiment, the image to be super-resolution reconstructed or dehazed is of the same size as its corresponding real reference image, and is normalized before being input into the dual-branch network model. Specifically, the pixel value range from 0 to 255 is mapped to the range from -1 to 1; secondly, during the training and application stages, certain image preprocessing is performed on the image according to different tasks.

[0040] Specifically, in this embodiment, one branch of the dual-branch network model can be called the first branch, and the other branch can be called the second branch. The specific structure of any one of the branches is as follows: Figure 3As shown, any branch in the dual-branch network model is composed of several feature extraction layers, residual layers, downsampling layers, upsampling layers, residual attention layers, and feature recovery layers. The specific structure of each layer is as follows Figure 4 shown.

[0041] Specifically, the construction process of the dual-branch network model is as follows:

[0042] S211, the input image to be super-resolution reconstructed or defogged is passed through a feature extraction layer and a residual layer to obtain low-level features, and then the low-level features are down-sampled and encoded three times, wherein the first two down-sampling codes are a down-sampling layer followed by a residual layer, and the third down-sampling code is a down-sampling layer followed by a residual attention layer;

[0043] S212. In downsampling coding, the image size is continuously reduced while the number of feature channels is continuously increased, which is called downsampling coding. The downsampling coding is followed by three upsampling decodings, where each upsampling decoding is an upsampling layer followed by a residual attention layer. In the upsampling coding process, the image size increases while the number of feature channels decreases. In addition, the result of each upsampling decoding is spliced ​​with the corresponding downsampling encoding output of the same size in the encoder in the channel dimension as the input of the next upsampling decoding.

[0044] S213. Concatenate the result of the third upsampling decoding with the original low-level feature in the channel dimension and input it into the feature recovery layer. The output dimension of the feature recovery layer is an image of 3, which can be used as the final output of this branch.

[0045] Specifically, the specific structure of each layer in the dual-branch network model is as follows:

[0046] The feature extraction layer consists of a convolutional layer followed by a ReLU activation layer;

[0047] The residual layer is a convolutional layer followed by a ReLU activation layer, and the ReLU activation layer is followed by a convolutional layer. The output of the convolutional layer is added to the input of the residual layer to form the residual;

[0048] The downsampling layer consists of a convolutional layer followed by a ReLU activation layer. The stride of the convolutional layer is set to 2, the padding is set to 1, and the convolution kernel size is 3, which halves the size of the feature map.

[0049] The upsampling layer is composed of a sub-pixel upsampling layer followed by a convolutional layer, and the output of the convolutional layer is then activated by a ReLU layer. The sub-pixel upsampling layer achieves the enlargement of the feature map size.

[0050] The residual attention layer is a convolutional layer followed by a ReLU layer. The output of the ReLU layer is added to the input of the convolutional layer to form a residual, which is then input into a convolutional layer. The output of the convolutional layer is input into a channel attention layer. The output of the channel attention layer is input into a pixel attention layer. The output of the pixel attention layer is added to the input of the entire residual attention layer to form a large residual as the output of the entire residual attention. Specifically, the channel attention layer is first an average pooling layer, which pools each feature map to a size of 1×1, followed by a convolutional layer to reduce the number of channels of the feature map. The convolutional layer is followed by a ReLU activation layer. Re The LU activation layer is followed by a convolutional layer to restore the number of channels of the feature map to its original number. The convolutional layer is followed by a Sigmoid activation layer to obtain the weight of each feature map. Finally, the weight is multiplied by the input feature map of the channel attention layer to obtain the feature map output with channel attention. The pixel attention layer is first a convolutional layer to reduce the number of channels of the feature map, followed by a ReLU activation layer. After the activation layer, another convolutional layer is followed to further reduce the number of feature maps to 1. Then, a Sigmoid activation layer is used to obtain the weight of each pixel. Finally, the weight is multiplied by the input of the pixel attention layer to obtain the feature map output with pixel attention.

[0051] The feature recovery layer is a convolutional layer followed by a Tanh activation layer.

[0052] Specifically, if Figure 5 As shown in Figure 2, the loss classifier in step S2 divides the smooth area and edge area of ​​the output image through L1 loss. The specific division process is as follows:

[0053] S221. The super-resolution reconstructed or dehazed image a obtained by the first branch (abbreviated as image a) is subjected to L1 loss with the real reference image GT during training. The specific formula of LosL1 is as follows:

[0054]

[0055] Where, I a is image a; I GT is the corresponding true reference image GT; C, W, H are the number of channels, width and height of the true fog-free image respectively; x, y, z are the number of channels, width and height of the image pixels respectively;

[0056] S222. Use the absolute value of the difference between image a and the true reference image GT to obtain Loss (x, y, z). The specific calculation formula is as follows:

[0057] Loss(x,y,z)=I a (x,y,z)-I GT (x,y,z);

[0058] S223. After obtaining Loss(x,y,z), compare its value at position (x,y,z) with LossL1, and set 1 or 0 at the corresponding position to obtain Mask(x,y,z). Specifically, taking the smoothing mask as an example, if the value of Loss(x,y,z) is smaller than LossL1, then 1 is taken, otherwise 0 is taken; the specific formula of the smoothing mask is as follows:

[0059]

[0060] Then the edge mask is expressed as 1-Mask(x,y,z);

[0061] S224. Apply the smoothing mask to image a and the edge mask to the preliminary super-resolution reconstructed or defogging image b (abbreviated as image b). Add the two together to obtain a better super-resolution reconstructed or defogging image c (abbreviated as c). The specific calculation process is as follows:

[0062] C(x,y,z)=A(x,y,z)×Mask(x,y,z)+B(x,y,z)×(1-Mask(x,y,z)),

[0063] Among them, A(x,y,z), B(x,y,z), C(x,y,z) represent images a, b and c respectively;

[0064] S225, the input of the feature recovery layer in each branch of the dual-branch network model is spliced ​​with the output of the feature recovery layer, i.e., the preliminary restored image, in the channel dimension, and the splicing result of one branch is spliced ​​again with the splicing result of the other branch in the channel dimension, and finally a feature fusion layer is passed to obtain the best super-resolution reconstruction or defogging image d, referred to as image d; wherein the feature fusion layer refers to a residual attention layer followed by a convolution layer, and finally a Tanh activation layer. The specific splicing process is as follows Figure 6 As shown;

[0065] S226. After obtaining image c and image d, the two are subjected to L1 loss and perceptual loss compared with the true reference image GT, and the dual-branch network model can be trained through the back propagation mechanism and gradient descent mechanism of the neural network, so that it can focus on both smooth areas and edge areas in the process of obtaining the final result image.

[0066] Taking image c as an example, the formula for its L1 loss with the true reference image GT is as follows:

[0067]

[0068] Where, I c is the network output result graph, IGT is the corresponding true reference image GT, and Net is a branch in the image dual-branch network model, which is the first branch in this case.

[0069] Taking image c as an example, the perceptual loss between it and the true reference image GT will be calculated using the pre-trained VGG16 network. When calculating the perceptual loss, the network output image c and the true reference image GT are input into the VGG16 network, and the mean square error is calculated by taking the two feature maps before the 5th pooling layer and after the 4th convolution layer in the corresponding VGG network. The specific calculation formula is as follows:

[0070]

[0071] In the formula, the processing process of VGG16() is to input the input image into the pre-trained VGG16 network and take the two feature maps corresponding to the network before the 5th pooling layer and after the 4th convolution layer.

[0072] Therefore, the total loss function during training can be expressed as follows:

[0073] L total =L mc +λL pc +L md +λL pd

[0074] Where, L total Represents the total loss function during training; L mc With L pc They represent L1 loss and perceptual loss for image c respectively; L md With L pd They represent L1 loss and perceptual loss for image d respectively; λ is the weight parameter of perceptual loss. When λ is set to 0.04, the best effect is achieved.

[0075] The present invention provides a method for image super-resolution reconstruction and defogging based on loss classification and dual-branch network, and includes the following steps in the training phase: Figure 6 shown.

[0076] In this embodiment, step S3 is the application of the model. In the application stage, Figure 7 As shown in Figure 1, due to the lack of a true reference image GT, there is no need to use the loss classifier to fuse images a and b to obtain c. Instead, the image d obtained by feature fusion is directly output as the final result image.

[0077] Based on the same inventive concept, the present invention also proposes an image super-resolution reconstruction and defogging system based on loss classification and a dual-branch network, comprising:

[0078] The preprocessing module is used to input the prepared image to be super-resolution reconstructed or dehazed and its corresponding real reference image, and perform preprocessing, cropping the image to be super-resolution reconstructed or dehazed and the real reference image to the same size and position, and performing data enhancement operations;

[0079] The model training module is used to build and train a two-branch network model. The two-branch network model uses an encoder-decoder network and is trained through a loss classifier and constraint function.

[0080] The model application module is used to input the image to be super-resolution reconstructed or dehazed into the trained two-branch network model and obtain the final result image through feature fusion.

[0081] Specifically, in this embodiment, the preprocessing module is divided into a preprocessing module in the model training process and a preprocessing module in the model application; specifically, in the model training process, the image preprocessing requires randomly cropping the image into a fixed size of 512×512 to obtain the image to be super-resolution reconstructed or to be defogged during training, and randomly flipping it horizontally and vertically. This ensures data preprocessing on the one hand and also enhances the data; and in the model application process, the processing process of the preprocessing module for the super-resolution reconstruction task is to determine the super-resolution reconstruction multiple of the image to be super-resolution reconstructed, that is, to clarify whether the size of the super-resolution reconstructed image to be output is to maintain the original size or to enlarge the pixel size by 2 times or 4 times. If it is the latter, the bicubic interpolation method is used to enlarge the image size of the super-resolution reconstructed image by the corresponding multiple. If it is the former, it is not necessary; secondly, in the model application stage, the image to be super-resolution reconstructed of the input network needs to be size-aligned. Specifically, a number of pixels are cropped from the super-resolution reconstructed image so that the length and width of the image are multiples of 8 respectively. The size of the cropped image can be expressed by the following formula:

[0082] Width new =Width old -(Width old mod 8), Highth new =Higth old -(Higth old mod 8)

[0083] Where, Width old , Higth old Respectively represent the width and height of the original image to be super-resolution reconstructed, Width new , Higth newThey represent the width and height of the cropped image to be super-resolution reconstructed that is actually input to the dual-branch network, and mod represents the modulo operation.

[0084] Furthermore, in the model application stage, the image preprocessing process for the dehazing task uses bicubic interpolation to resize it to 800×600.

[0085] Specifically, in this embodiment, the model training module inputs the pre-processed image to be super-resolution reconstructed or defogging image into the two-branch network to obtain preliminary super-resolution reconstructed image or defogging image a and b. The loss classifier is used to obtain the smoothing mask Mask and the edge mask 1-Mask based on the preliminary super-resolution reconstructed image or defogging image a and the real reference image GT. The grayscale image of the smoothing mask Mask is as follows: Figure 8 As shown in , we can see that the Mask image basically reflects the general outline of the super-resolution reconstructed image or the defogging image, where the white part belongs to the smooth area and the black part belongs to the edge area. Figure 5 As shown, the obtained smooth mask and edge mask can be fused to obtain a better super-resolution reconstructed image or defogging image c. In addition, Figure 6 The method shown in the figure performs feature fusion to obtain the best super-resolution reconstructed image or defogging image d. During the training process, the L1 loss and perceptual loss formulas are applied to the super-resolution reconstructed image or defogging image c and the super-resolution reconstructed image or defogging image d. The setting of the loss function is consistent with that of the method provided by the present invention. The specific formula is as follows:

[0086] L total =L mc +λL pc +L md +λL pd ,

[0087] Where, L total Represents the total loss function during training, L mc With L pc They represent L1 loss and perceptual loss for super-resolution reconstructed image or dehazed image c, respectively. md With L pdwhere λ represents the L1 loss and perceptual loss applied to the super-resolution reconstructed image or dehazed image d, respectively, and the value of λ is set to 0.04. Subsequently, through the reverse regression and gradient descent mechanisms of the neural network, the first branch of the dual-branch network can focus on super-resolution reconstruction or dehazing of smooth areas, while the second branch focuses on super-resolution reconstruction or dehazing of edge areas. After sufficient training, the initial super-resolution reconstructed image a from the first branch has a super-resolution reconstruction effect but performs poorly in edge areas, while the initial super-resolution reconstructed image b from the second branch has significantly sharper edges, but the colors in smooth areas are not as accurate as those in the initial super-resolution reconstructed image a. Therefore, the fusion of the two images, c, will be significantly better.

[0088] Specifically, in this embodiment, the model application module uses a feature fusion method to generate an optimal super-resolution reconstructed or defogged image d, and outputs it as the final super-resolution reconstructed or defogged image.

[0089] After super-resolution reconstruction by this system, it is compared with the image of the same size obtained by the bicubic interpolation method, that is, the input image. Table 1 is a specific performance comparison of this system with the bicubic interpolation method and some deep neural network-based methods on the Set14 test set. It can be seen that after the dual-branch network used by this system is used to deal with the super-resolution reconstruction problem, compared with bicubic and some end-to-end deep neural network methods, the super-resolution reconstructed image obtained by this system has been significantly improved in visual indicators. Among them, the larger the SSIM, the better, and the smaller the PI and LPIPS, the better. Specifically, Table 1 is as follows:

[0090] Table 1

[0091]

[0092]

[0093] like Figure 9 (a) Figure 9 (b) Figure 9 (c) Figure 9 (d) Figure 9 (e) Figure 9As shown in (f), the comparison between the fog image and the real fog-free image after defogging by this system and the comparison with other methods shows that after defogging by this system, the restored defogging image is very close to the real fog-free image. As shown in Table 2, the present invention has great advantages in defogging ability. Compared with the traditional dark channel algorithm (DCP), it has a significant improvement in various indicators, and compared with some of the latest algorithms such as MSTN, the system provided by the present invention also has certain improvements and is quite comparable. In Table 2, the larger the PSNR and SSIM, the better, and the smaller the FADE, the better. Among them, GCANet, KDDN and MSTN use the indicator data in the original paper. Specifically, Table 2 is as follows:

[0094] Table 2

[0095] Dehazing Method PSNR SSIM FADE DCP 12.18 0.444 0.319 AOD 14.36 0.449 0.790 GCANet 17.58 0.594 - GDN 17.39 0.623 0.373 KDDN 17.39 0.589 - FFA 18.52 0.639 0.330 4K Dehazing 16.85 0.550 0.408 MSTN 18.42 0.630 - Method of the present invention 19.44 0.622 0.414

[0096] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. Image super-resolution reconstruction and dehazing method based on loss classification and dual-branch network, characterized by: The following steps are involved: S1. Prepare the image to be super-resolution reconstructed or defogged and its corresponding real reference image, crop the image to be super-resolution reconstructed or defogged and the real reference image to the same size and position, and perform data augmentation. S2. Build and train a dual-branch network model. The dual-branch network model uses an encoder-decoder network and is trained using a loss classifier and a constraint function. S3. Input the image to be super-resolution reconstructed or dehazed into the trained two-branch network model, and obtain the final result image by feature fusion; The construction process of the dual-branch network model is as follows: S211, the input image to be super-resolution reconstructed or defogged is passed through a feature extraction layer and a residual layer to obtain low-level features, and then the low-level features are down-sampled and encoded three times, wherein the first two down-sampling codes are a down-sampling layer followed by a residual layer, and the third down-sampling code is a down-sampling layer followed by a residual attention layer; S212, after the downsampling encoding, there are three upsampling decodings, where each upsampling decoding is an upsampling layer followed by a residual attention layer; during the upsampling encoding process, the result of each upsampling decoding is concatenated with the corresponding downsampling encoding output of the same size in the encoder in the channel dimension as the input of the next upsampling decoding; S213, concatenating the result of the third upsampling decoding with the initial low-level feature in the channel dimension, and inputting it into the feature recovery layer. The feature recovery layer outputs an image of dimension 3, which is used as the final output of this branch; In step S2, the loss classifier divides the output image into smooth areas and edge areas through L1 loss. The specific division process is as follows: S221, using the super-resolution reconstructed or dehazed image a obtained by the first branch, and performing L1 loss on the real reference image GT during the training process to obtain LossL1; S222. Calculate the absolute value of the difference between the image a and the true reference image GT pixel by pixel to obtain Loss (x, y, z). S223. After obtaining Loss(x,y,z), compare its value at position (x,y,z) with LossL1, and set 1 or 0 at the corresponding position to obtain Mask(x,y,z). If the value of Loss(x,y,z) is smaller than LossL1, it is set to 1, otherwise it is set to 0. The specific formula for smoothing the mask is as follows: Then the edge mask is expressed as 1-Mask(x,y,z); S224, applying a smoothing mask to image a, applying an edge mask to the preliminary super-resolution reconstructed or defogging image b, and adding the two to obtain a super-resolution reconstructed or defogging image c; S225, splicing the input of the feature recovery layer in each branch of the dual-branch network model with the output of the feature recovery layer, i.e., the preliminary restored image, in the channel dimension, and splicing the result of one branch splicing with the result of the other branch splicing again in the channel dimension, and finally passing through a feature fusion layer to obtain the best super-resolution reconstruction or dehazed image d; wherein the feature fusion layer refers to first passing through a residual attention layer and then passing through a convolution layer, and finally connecting to a Tanh activation layer; S226. After obtaining image c and image d, L1 loss and perceptual loss are performed on the two images and the true reference image GT, and the dual-branch network model is trained through the back propagation mechanism and gradient descent mechanism of the neural network, so that smooth areas and edge areas are obtained in the process of obtaining the final result image.

2. The image super-resolution reconstruction and dehazing method based on loss classification and dual-branch network according to claim 1, characterized in that: The dual-branch network model includes a first branch and a second branch, wherein each branch is composed of several feature extraction layers, residual layers, downsampling layers, upsampling layers, residual attention layers, and feature recovery layers.

3. The image super-resolution reconstruction and defogging method based on loss classification and dual-branch network according to claim 2 is characterized in that: The specific structure of each layer in the dual-branch network model is as follows: The feature extraction layer consists of a convolutional layer followed by a ReLU activation layer; The residual layer is a convolutional layer followed by a ReLU activation layer, and the ReLU activation layer is followed by a convolutional layer. The output of the convolutional layer is added to the input of the residual layer to form the residual; The downsampling layer consists of a convolutional layer followed by a ReLU activation layer; The upsampling layer consists of a sub-pixel upsampling layer followed by a convolutional layer, and the output of the convolutional layer is then passed through a ReLU activation layer. The sub-pixel upsampling layer is used to enlarge the size of the feature map; The residual attention layer is a convolutional layer followed by a ReLU layer. The output of the ReLU layer is added to the input of the convolutional layer to form a residual, which is then input into a convolutional layer. The output of the convolutional layer is input into a channel attention layer. The output of the channel attention layer is then input into a pixel attention layer. The output of the pixel attention layer is added to the input of the entire residual attention layer to form a large residual as the output of the entire residual attention. Among them, the channel attention layer is first an average pooling layer, which pools each feature map to a size of 1×1, followed by a convolutional layer to reduce the number of channels of the feature map. The convolutional layer is followed by a ReLU activation layer. Re The LU activation layer is followed by a convolutional layer to restore the number of channels of the feature map to its original number. The convolutional layer is followed by a Sigmoid activation layer to obtain the weight of each feature map. Finally, the weight is multiplied by the input feature map of the channel attention layer to obtain the feature map output with channel attention. The pixel attention layer is first a convolutional layer to reduce the number of channels of the feature map, followed by a ReLU activation layer. After the activation layer, another convolutional layer is followed to further reduce the number of feature maps to 1. Then, a Sigmoid activation layer is used to obtain the weight of each pixel. Finally, the weight is multiplied by the input of the pixel attention layer to obtain the feature map output with pixel attention. The feature recovery layer is a convolutional layer followed by a Tanh activation layer.

4. The image super-resolution reconstruction and defogging method based on loss classification and dual-branch network according to claim 1, characterized in that: The perceptual loss between image c and the true reference image GT will be calculated using the pre-trained VGG16 network. When calculating the perceptual loss, the network output image c and the true reference image GT are input into the VGG16 network, and the mean square error is calculated by taking the two feature maps before the 5th pooling layer and after the 4th convolution layer in the corresponding VGG network; The total loss function during training can be expressed as follows: L total =L mc +λL pc +L md +λL pd Where, L total Represents the total loss function during training; L mc With L pc They represent L1 loss and perceptual loss for image c respectively; L md With L pd They represent L1 loss and perceptual loss for image d respectively; λ is the weight parameter of perceptual loss.

5. The image super-resolution reconstruction and defogging system based on loss classification and dual-branch network according to claim 1, characterized in that: include: The preprocessing module is used to input the prepared image to be super-resolution reconstructed or dehazed and its corresponding real reference image, and perform preprocessing, cropping the image to be super-resolution reconstructed or dehazed and the real reference image to the same size and position, and performing data enhancement operations; The model training module is used to build and train a two-branch network model. The two-branch network model uses an encoder-decoder network and is trained through a loss classifier and constraint function. The model application module is used to input the image to be super-resolution reconstructed or dehazed into the trained two-branch network model and obtain the final result image through feature fusion.

6. The system according to claim 5, characterized in that The preprocessing module is divided into the preprocessing module in the model training process and the preprocessing module in the model application; During the model training process, the image preprocessing randomly crops the image into a fixed size of 512×512 to obtain the image to be super-resolution reconstructed or dehazed during training, and randomly flips it horizontally and vertically. The processing process of the preprocessing module for the super-resolution reconstruction task in the process of model application is to determine the super-resolution reconstruction multiple of the image to be super-resolution reconstructed, and to clarify whether the output super-resolution reconstructed image size is to maintain the original size, or to enlarge the pixel size by 2 times or 4 times. If it is the latter, the bicubic interpolation method is used to enlarge the image size by the corresponding multiple for the super-resolution reconstructed image first. If it is the former, it is not necessary; secondly, in the model application stage, it is also necessary to align the size of the input network to be super-resolution reconstructed, and crop several pixels of the super-resolution reconstructed image so that the length and width of the image are multiples of 8 respectively. The size of the cropped image can be expressed by the following formula: Width new =Width old -(Width old mod 8),Higth new =Higth old -(Higth old mod 8) Where, Width old , Higth old Respectively represent the width and height of the original image to be super-resolution reconstructed, Width new , Higth new They represent the width and height of the cropped image to be super-resolution reconstructed that is input to the dual-branch network, and mod represents the modulo operation; During the model application stage, the image preprocessing process for the dehazing task uses bicubic interpolation to resize it to 800×600.

7. The system according to claim 5, characterized in that The model training module inputs the pre-processed super-resolution reconstructed image or defogging image into the two-branch network to obtain preliminary super-resolution reconstructed images or defogging images a and b; uses the loss classifier to obtain the smoothing mask Mask and the edge mask 1-Mask based on the preliminary super-resolution reconstructed image or defogging image a and the real reference image GT; obtains the super-resolution reconstructed image or defogging image c by fusing the obtained smoothing mask and edge mask; during the training process, the L1 loss and perceptual loss formula are performed on the super-resolution reconstructed image or defogging image c and the super-resolution reconstructed image or defogging image d. The specific formula of the loss function is as follows: L total =L mc +λL pc +L md +λL pd , Where, L total Represents the total loss function during training, L mc With L pc They represent L1 loss and perceptual loss for super-resolution reconstructed image or dehazed image c, respectively. md With L pd They represent the L1 loss and perceptual loss for the super-resolution reconstructed image or the dehazed image d, respectively, and λ is the weight parameter of the perceptual loss.

Images