A visual dehazing method and system based on a three-branch neural network

Abstract

This invention discloses a visual dehazing method and system based on a three-branch neural network. The system includes: a feature extraction subsystem, comprising a parallel low-frequency branch network, a high-frequency branch network, and a semantic branch network; the low-frequency branch network is used to extract low-frequency image features from the original image and recover a first coarse dehazing result; the high-frequency branch network is used to extract high-frequency image features from the original image and recover a second coarse dehazing result; the semantic branch network is used to identify different objects and regions in the original image to obtain image semantic features and recover a third coarse dehazing result consistent with the semantics of the original image; and a fusion network subsystem is used to fuse the low-frequency image features, the first coarse dehazing result, the high-frequency image features, the second coarse dehazing result, the third coarse dehazing result, and the original image to generate a dehazed image.

Description

Technical Field

[0001] This invention relates to a visual dehazing method and system based on a three-branch neural network, belonging to the field of computer vision technology. Background Technology

[0002] With the rapid development of computer vision technology, image processing technology has been widely applied in many fields, including military reconnaissance, traffic monitoring, autonomous driving, aerospace, and environmental monitoring. However, in practical applications, the presence of particulate matter such as haze in the atmosphere often severely affects the quality of acquired images, leading to blurred image details and reduced contrast, which in turn affects subsequent image analysis, recognition, and understanding. Therefore, research on image dehazing technology has significant practical application value and scientific research significance.

[0003] In recent years, with the continuous innovation and optimization of neural network structures, multi-branch neural networks have gradually demonstrated their powerful feature extraction and learning capabilities. By introducing different paths and scales, multi-branch networks can capture multi-scale features in images, enhancing the network's robustness and generalization ability. In image dehazing tasks, multi-branch networks can better adapt to different haze concentrations and image scenes, improving the accuracy and stability of dehazing. However, existing multi-branch network structures still model the dehazing problem as an end-to-end image mapping problem, such as Two-branch Dehaze Networks and DWGAN. Due to the diversity of fog patterns and the complexity of scenes, the results obtained by directly feeding fogged images into end-to-end networks for training are still unsatisfactory. Existing methods treat the original image as a single input and directly output the dehazed image using an end-to-end network, which may lead to problems such as insufficient color recovery or blurred edges. Summary of the Invention

[0004] Considering that fog degrades different components of an image in various ways and to varying degrees, more effective dehazing algorithms need to separate and decompose the different components of the image into sub-problems, and design targeted solutions for the recovery of each component. To this end, this invention proposes a visual dehazing method and system based on a three-branch neural network. By introducing a three-branch network structure to separate and enhance different components of the image, and combining the advantages of deep learning and image processing, it achieves efficient and accurate image dehazing, improving image clarity and detail, and solving the problem of unsatisfactory results in existing multi-branch end-to-end dehazing methods.

[0005] According to one aspect of the present invention, a visual dehazing system based on a three-branch neural network is proposed, comprising: a feature extraction subsystem, including a parallel low-frequency branch network, a high-frequency branch network, and a semantic branch network; wherein the low-frequency branch network is used to extract low-frequency image features of the original image and recover a first coarse dehazing result, the high-frequency branch network is used to extract high-frequency image features of the original image and recover a second coarse dehazing result, and the semantic branch network is used to identify different objects and regions in the original image to obtain image semantic features and recover a third coarse dehazing result consistent with the semantics of the original image; and a fusion network subsystem, connected to the outputs of the low-frequency branch network, the high-frequency branch network, and the semantic branch network, for fusing the low-frequency image features, the first coarse dehazing result, the high-frequency image features, the second coarse dehazing result, the third coarse dehazing result, and the original image to generate a dehazed image.

[0006] According to another aspect of the present invention, a visual dehazing method based on a three-branch neural network is also proposed, comprising the following steps: constructing a low-frequency branch network to extract low-frequency image features of the original image and recover a first coarse dehazing result; constructing a high-frequency branch network to extract high-frequency image features of the original image and recover a second coarse dehazing result; constructing a semantic branch network to identify different objects and regions in the original image to obtain image semantic features and recover a third coarse dehazing result consistent with the semantics of the original image; and constructing a fusion network subsystem to generate a dehazed image by fusing the low-frequency image features, the first coarse dehazing result, the high-frequency image features, the second coarse dehazing result, the third coarse dehazing result, and the original image.

[0007] Compared with the prior art, the beneficial effects of the technical solution of the present invention are reflected in:

[0008] This invention utilizes a three-branch neural network structure (low-frequency, high-frequency, and semantic) to simultaneously process low-frequency, high-frequency, and semantic information of an image. This avoids interference between different components during the enhancement process of a single image input, improving the clarity and detail enhancement of the dehazed image and achieving more comprehensive and accurate dehazing. Compared to traditional methods or single-branch network dehazing techniques, this invention better preserves image details and textures, improves image clarity and contrast, and makes the dehazed image more closely resemble the real scene.

[0009] In addition, the present invention has good generalization ability. Since the three-branch neural network can learn image features under different fog conditions, the present invention can adapt to the defogging needs of different scenarios and weather conditions. Whether it is light fog or heavy smog, the present invention can effectively defog and obtain defogging images that are close to the real scene. Attached Figure Description

[0010] Figure 1 This is a flowchart of a visual dehazing method based on a three-branch neural network according to an embodiment of the present invention.

[0011] Figure 2 This is a schematic diagram of the defogging process of the visual defogging system based on a three-branch neural network according to an embodiment of the present invention.

[0012] Figure 3 This is a network architecture diagram of a visual dehazing system based on a three-branch neural network according to an embodiment of the present invention.

[0013] Figure 4 This is a schematic diagram of the defogging effect of the visual defogging system based on a three-branch neural network according to an embodiment of the present invention. Detailed Implementation

[0014] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments are provided for illustrative purposes only and are not intended to limit the scope of protection of the present invention in any way.

[0015] This invention proposes a visual dehazing method and system based on a three-branch neural network, which simultaneously processes low-frequency, high-frequency, and semantic information of images, achieving more comprehensive and accurate dehazing processing.

[0016] Please refer to Figure 1 The visual dehazing method based on a three-branch neural network proposed in this embodiment of the invention includes the following process steps Z1 to Z4:

[0017] Z1. A visible light sensing system for foggy scenes was built using a high-definition vision camera to acquire raw images in foggy environments. The built visible light sensing system was tested and calibrated. Its stability and reliability were verified through practical tests under different fog concentrations and weather conditions. Based on the test results, necessary calibrations and adjustments were made to ensure its accurate capture of image data in foggy environments. By building such a visible light sensing system for foggy scenes, high-quality raw image data is provided for subsequent defogging processing, laying a solid foundation for achieving efficient and accurate defogging results.

[0018] Z2. Construct an image processing subsystem. This subsystem extracts and fuses image information through low-frequency branch network subsystems, high-frequency branch network subsystems, semantic branch network subsystems, and fusion network subsystems, ultimately achieving visual dehazing. For details, please refer to [reference needed]. Figure 2The original foggy image is simultaneously input into three parallel subsystems for processing. The main purpose of the low-frequency branch network subsystem is to extract low-frequency features of the image, the main purpose of the high-frequency branch network subsystem is to extract high-frequency features of the image, the main purpose of the semantic branch network subsystem is to extract semantic features of the image, and the fusion network subsystem is responsible for fusing the features extracted by the three parallel branches to generate the final defogging result.

[0019] Z3. The high-performance transmission subsystem performs high-performance compression and network transmission of the original image and the clear image processed by the image processing subsystem.

[0020] Z4. The high-performance image decoder decodes the transmitted clear image and visualizes it on a high-definition display, while storing the decoding results in a high-performance disk array.

[0021] In some specific implementations, step Z2 may specifically include: constructing a low-frequency branch network to extract low-frequency image features from the original image and recover a first coarse dehazing result; constructing a high-frequency branch network to extract high-frequency image features from the original image and recover a second coarse dehazing result; constructing a semantic branch network to identify different objects and regions in the original image to obtain image semantic features and recover a third coarse dehazing result consistent with the semantics of the original image; and constructing a fusion network subsystem to fuse and optimize the low-frequency image features, the first coarse dehazing result, the high-frequency image features, the second coarse dehazing result, the third coarse dehazing result, and the original image to generate a clear dehazed image.

[0022] Another embodiment of the present invention proposes a system for implementing the above method, namely, a visual dehazing system based on a three-branch neural network, comprising:

[0023] (1) A visible light sensing system for fog scenes, used to capture raw images in fog environments;

[0024] (2) A feature extraction subsystem connected to the output of the visible light sensing system includes a parallel low-frequency branch network, a high-frequency branch network, and a semantic branch network; wherein, the low-frequency branch network is used to extract low-frequency image features of the original image and recover the first coarse dehazing result, the high-frequency branch network is used to extract high-frequency image features of the original image and recover the second coarse dehazing result, and the semantic branch network is used to identify different objects and regions in the original image to obtain image semantic features and recover the third coarse dehazing result consistent with the semantics of the original image;

[0025] (3) The fusion network subsystem is connected to the output of the low-frequency branch network, the high-frequency branch network and the semantic branch network. At the same time, the output of the visible light sensing system is also directly connected to the fusion network subsystem. The fusion network subsystem is used to fuse and optimize the low-frequency image features, the first coarse dehazing result, the high-frequency image features, the second coarse dehazing result, the third coarse dehazing result and the original image to generate a clear dehazed image.

[0026] (4) High-performance transmission subsystem, connected to the output end of the fusion network subsystem and the output end of the visible light sensing system, is used to encode and compress the dehazed image and the original image and transmit them efficiently and stably;

[0027] (5) Monitoring and storage subsystem, connected to the high-performance transmission subsystem, is used to decode and visualize the transmitted dehazed images in real time and store the decoded dehazed images.

[0028] In some exemplary embodiments, the visible light sensing system uses a high-performance, high-definition, and high-color-fidelity camera and lens to acquire original images containing scene texture and color information. In the dehazing scenario of this embodiment of the invention, the original hazy image is acquired.

[0029] The feature extraction subsystem and the fusion network subsystem together constitute the most critical part of the visual dehazing system—the image processing subsystem. It is responsible for performing deep processing on the acquired fog images to eliminate the effects of fog and improve image quality. This invention analyzes the degradation patterns of images caused by fog and constructs low-frequency branch networks, high-frequency branch networks, semantic branch networks, and a fusion network subsystem according to image processing requirements.

[0030] Fog affects different components of an image to varying degrees and in different ways. Specifically, fog makes image colors appear whiter; due to scattering effects, it diffuses light and dilutes the original colors in the image, making its impact on low-frequency components more multiplicative. The degradation mode of fog on high-frequency edge components mainly involves distorting the edges of objects in foggy images, exhibiting a convolutional degradation. Parts of objects may be completely invisible due to fog. However, semantic information can help infer the parts of objects obscured by fog. Existing methods treat the image as a single input and neglect the need to separate different image components, leading to mutual interference during the enhancement process of each component. Therefore, problems such as insufficient color restoration or blurred edges may occur. Considering that fog reduces different components of an image in various ways and to varying degrees, more effective dehazing algorithms need to separate and decompose the different components of the image into sub-problems and design targeted restoration for each component. Based on this, this invention constructs three branches: low-frequency, high-frequency, and semantic, each branch solving the degradation problem of one component of the image, which are then fused in the final step.

[0031] Low-frequency branching networks:

[0032] The low-frequency branch network is primarily responsible for extracting low-frequency information from the image, such as overall structure and color distribution. By training and optimizing the low-frequency branch network, it can accurately capture and retain the main features of the image, providing a foundation for subsequent dehazing processing. Considering the sensitivity of the self-attention mechanism to low-frequency information, this embodiment of the invention uses sliding window self-attention blocks as the basic building blocks of the low-frequency branch network. These blocks perform self-attention operations on local areas of the image through a sliding window pattern, which is particularly effective for extracting local information.

[0033] The low-frequency branch network is a U-shaped network. Its encoding part uses sliding window self-attention blocks as basic building blocks to construct two downsampling levels; its decoding part uses depthwise separable convolutions as basic building blocks, correspondingly constructing two upsampling levels. An exemplary implementation of the low-frequency branch network is as follows: Figure 3 As shown within the green dashed box, the encoder part begins with a 3×3 input convolution (Conv). Following the input convolution is a preprocessing block (S-TB), which stands for Swin-Transformer Block. This S-TB consists of multiple sliding window attention blocks (S-ABs). Next, there are two levels of downsampling, each consisting of a downsampler and an S-TB block connected sequentially. The sliding window attention block (S-AB) performs self-attention operations on local image regions using a sliding window pattern, which is particularly effective for extracting local information. Specifically, in the sliding window attention block, layer normalization is first performed, mimicking the Dehaze-Former approach. Then, the feature map is divided into small windows, and multi-head attention operations are performed within each window. A single-head attention operation is defined as follows:

[0034]

[0035] Where Q, K, and V represent the query, key, and value in the Transformer, respectively, all of which are generated by convolution; d k This is the dimension of the key vector; Softmax() operates in a windowed manner. This operation allows for efficient extraction of local information within each window, which is sufficient for local low-frequency feature extraction while maintaining manageable computational complexity.

[0036] Continue to refer to Figure 3The decoder section of the low-frequency branch network first consists of two upsampling layers. Each upsampling layer includes an upsampler, a fusion block, and a DW convolutional block connected sequentially. The DW convolutional block contains multiple depthwise separable convolutions. Then comes the output convolution 3×3 Conv. Simultaneously, the outputs of each encoder layer are connected to the decoder via skip connections. Specifically, the preprocessing S-TB module of the encoding section is connected to the fusion block of the second upsampling layer of the decoding section via a 1×1 Conv skip, and the S-TB of the first downsampling layer of the encoding section is connected to the fusion block of the first upsampling layer of the decoding section via a 1×1 Conv skip. Furthermore, the input of the low-frequency branch network (the original image I) is also connected to the output of the output convolution via a 1×1 Conv skip, fusing with the decoder output to generate the final output of the low-frequency branch network. This output consists of two parts: one part is the recovered low-frequency image features y. lf The main component is low-frequency color information, which can be effectively extracted using Gaussian blur; the other part is a rough dehazing result. lf-refined .

[0037] In some specific embodiments, the feature fusion module Fusion Block of the low-frequency branch network adopts SK-Fusion, and the downsampling and upsampling adopt PixShuffle and PixUnShuffle, respectively. However, this is only exemplary, and the present invention does not limit it. In addition, the implementation of the sliding window self-attention mechanism is a known technology in the art, and the present invention will not elaborate on it here.

[0038] High-frequency branching networks:

[0039] The high-frequency branch network is primarily responsible for extracting high-frequency detail information from an image. High-frequency information in an image typically manifests as details such as edges and textures, which are often lost or blurred during the dehazing process. The goal of the high-frequency branch network is to accurately extract and preserve these high-frequency details, ensuring that the dehazed image retains its sharpness and richness of detail.

[0040] The high-frequency branch network in this embodiment of the invention is also a U-shaped network, almost entirely composed of convolutional modules, because convolution is more sensitive to high-frequency information than self-attention; at the same time, the local dependence of high-frequency information is weaker than that of low-frequency information, but the global dependence is stronger, therefore the high-frequency branch network in this embodiment of the invention requires a large receptive field. Since convolution has a lower computational cost, it can build deeper networks, which perfectly meets this requirement. Thus, the high-frequency branch network in this embodiment of the invention includes more scale levels in the encoder and decoder.

[0041] An exemplary implementation of a high-frequency branch network is as follows: Figure 3 As shown within the blue dashed box, the encoder section has four downsampling levels, and correspondingly, the decoder section has four upsampling levels. Specifically, the encoder section first consists of two preprocessing multi-scale convolutional modules (MSCBs), whose internal network structure and functions are known technologies and will not be elaborated further. Then, there are four levels of downsampling, each of which includes a Discrete Wavelet Transform (DWT) downsampler and a multi-scale convolutional module (MSCB) connected in sequence. The DWT downsampler internally contains parallel DWT and convolution operations. The decoder section first consists of four levels of upsampling, each of which includes an upsampler, a fusion module (FusionBlock), and a multi-scale convolutional module (MSCB) connected in sequence. Finally, an output module (Output Block) is connected at the end. Similar to the low-frequency branch network, there are also skip connections between the encoder and decoder. Specifically, the first preprocessing MSCB is connected to the Fusion Block of the last (fourth) upsampling layer via a 1×1 Conv skip; the DWT Down Sampler of the first downsampling layer is connected to the Fusion Block of the third upsampling layer via a 1×1 Conv skip; the DWT Down Sampler of the second downsampling layer is connected to the Fusion Block of the second upsampling layer via a 1×1 Conv skip; and the DWT Down Sampler of the third downsampling layer is connected to the Fusion Block of the first upsampling layer via a 1×1 Conv skip. Furthermore, the input of the high-frequency branch network (i.e., the original image I) is also connected to the output of the decoder output OutputBlock via a 1×1 Conv skip, and is weighted and fused with the decoder output to generate the final output of the high-frequency branch network. This output consists of two parts: one part is the recovered high-frequency image features y. hf The main components are high-frequency detail information such as edges and contours, which can be effectively obtained using the Laplacian operator; the other part is a coarse dehazing result y. hf-refined It can be used to provide an initial estimate of the dehazed image.

[0042] Semantic branching networks:

[0043] Semantic branch networks (SBRs) learn and analyze the features of objects and regions, enabling more accurate identification and processing of different objects in images, providing a more targeted solution for dehazing. The output of a SBR is a coarsely restored but semantically consistent dehazed image. Similar to low-frequency and high-frequency branch networks, SBRs also include encoding and decoding parts. The encoder uses Res2Net pre-trained on ImageNet to extract abstract semantic information from the image, while the decoder employs a channel-level attention module. It performs global pooling on the input feature map, followed by processing through a multilayer perceptron and activation functions to generate a feature vector matching the channel dimension. This feature vector is then multiplied channel-wise with the feature map, ultimately yielding a coarse dehazed result, denoted as y, that is semantically consistent with the original image. sem Res2Net is a known network, and its internal structure and working principle will not be elaborated here.

[0044] An exemplary implementation of semantic branching networks is as follows: Figure 3 As shown within the red dashed box, the decoder section, from input to output, is sequentially connected to CP-AB (CP-Attention Block), PixShuffle, CP-AB, PixShuffle, CP-AB, PixShuffle, CP-AB, PixShuffle, CP-AB, and the output convolutional Conv Block. The output of the Res2Net encoder is further connected to the outputs of the second, third, and fourth CP-ABs for weighted fusion, with the fusion results serving as inputs to the subsequent PixShuffle. The semantic branch network ultimately outputs a coarsely dehazed result y via the Conv Block. sem Among them, CP-AB (CP-AttentionBlock) is a known network in this field, and its internal structure and working principle will not be described in detail here.

[0045] Converged network subsystem:

[0046] The fusion network subsystem is responsible for fusing and optimizing the outputs of the three branch networks. By formulating reasonable fusion strategies and optimization algorithms, it ensures that the advantages of each branch network are fully utilized while avoiding potential information conflicts and redundancy, thereby obtaining a high-quality dehazed image. The fusion network subsystem is designed following a progressive optimization principle from local to global to ensure that while integrating the three branch networks, it can also optimize and improve the image globally.

[0047] To enhance the global information processing capability of the fusion network subsystem, this invention introduces a Taylor attention module as the basic building block. This Taylor attention module uses a first-order approximation of Taylor expansion to approximate the self-attention operation across the entire feature map, possessing a broad global receptive field. This allows the fusion network subsystem to more sensitively capture long-range dependencies and global structural information in the image. Similar to the three branch networks mentioned above, it also includes encoder and decoder parts. An exemplary fusion network subsystem is shown below. Figure 3 As shown within the yellow dashed box, the encoder section begins with a 3×3 input convolution (Conv), followed by a preprocessing block (T-TB, TaylorTransformerBlock), and then three downsampling layers. Each downsampling layer consists of a downsampler and a T-TB module connected in sequence. Correspondingly, the decoder section comprises three upsampling layers, each consisting of an upsampler, a fusion block, and a T-TB module connected in sequence; finally, it connects to an output convolution (3×3 conv). The T-TB module, or Taylor Transformer Block, contains multiple Taylor attention modules. In the fusion network subsystem, the output of the T-TB preprocessing block connected after the input convolution is also connected to the Fusion Block of the last (third) upsampling layer of the decoder via a 1×1 Conv jump. The T-TB of the first downsampling layer of the encoder is connected to the Fusion Block of the second upsampling layer of the decoder via a 1×1 Conv jump, and the T-TB of the second downsampling layer of the encoder is connected to the Fusion Block of the first upsampling layer of the decoder via a 1×1 Conv jump. Furthermore, the input of the fusion network subsystem is also connected to the output of the 3×3 Conv output convolution via a 1×1 Conv jump for weighted fusion, generating the final output of the fusion network subsystem, i.e., the final dehazed image J. The outputs y of the aforementioned three branch networks are then... lf y lf-refined y hf y hf-refined y sem The original image I, after being stitched together along the channel dimension, is fed into the fusion network subsystem for feature fusion and global optimization to generate the final dehazed image J.

[0048] The Taylor attention module demonstrates significant advantages in long-distance feature interactions and global information modeling, as shown in the following formula:

[0049]

[0050] Here, the variables with subscripts i and j represent the vectors in the i-th and j-th rows of the matrix. The processing is similar to that of the low-frequency branch network. The feature map first undergoes layer normalization (RescaledLayer Norm), and then linear encoding is performed to generate the Q, K, and V of the Transformer. Subsequently, attention operations are performed. Finally, the final result is obtained through two layers of multilayer perceptron.

[0051] The fusion network subsystem design of this invention is adapted to the requirements of global modeling. Its internal linear approximation allows it to integrate global information under the constraints of linear storage requirements and computational complexity. This is crucial for the processing of two-dimensional image data in this invention. It enables the efficient fusion of global information under existing computational resource constraints, thereby serving to generate high-quality dehazed images.

[0052] The training process of the entire image processing subsystem is as follows:

[0053] ① For the low-frequency branch network, the output firstly consists of two parts: low-frequency image features and the preliminary dehazing result generated by this branch. In this embodiment, low-frequency image features are defined as the color features of the image, that is, the result obtained by applying Gaussian blur to the ground truth image. This process effectively eliminates edge and contour information, retaining only color information. Furthermore, since MSELoss is more sensitive to low-frequency features, this embodiment uses MSELoss to guide the training of the low-frequency branch network. Using L... lf The loss function for low-frequency branch networks is:

[0054] L lf =MSE(concat(Gaussian(GT),GT),y lf )

[0055] Where GT represents the ground truth for dehazing, concat represents the concatenation operation, and Gaussian represents the Gaussian blur operation. This method enables the low-frequency branch network of this invention to focus more on extracting and preserving color information during the dehazing process.

[0056] ② For the high-frequency branch network, the high-frequency information output by the high-frequency branch network in this embodiment mainly consists of image contours and edges, which correspond to the results obtained by applying the Laplacian operator to the ground truth image. This embodiment uses L1 loss to guide the training of this branch network because it is more sensitive to image edges. hf The loss function for high-frequency branching networks:

[0057] L hf =L1(concat(Laplace(GT),GT),y hf )

[0058] Here, Laplace represents the Laplace edge detection operation.

[0059] ③ For semantic branch networks, the output is a coarsely recovered but semantically correct dehazing result. The degree of semantic consistency with the fog-free ground truth (GT) is measured by the perceptual loss function. Using L... sem The loss function for the semantic branching network is expressed by the following formula:

[0060]

[0061] Among them, f i This represents the output of the i-th layer of VGG-16.

[0062] ④ For the fusion network subsystem, SmoothL1 loss and MS-SSIM loss are used for reduction training, with L... fusion The loss function is expressed in the following formula:

[0063] L fusion =SmoothK1(y,GT)+MSSSIM(y,GT)

[0064] Where y represents the final output of the fusion network subsystem.

[0065] Given the complexity of three-branch networks, directly training the entire network may present numerous difficulties and challenges. Therefore, this invention proposes a two-stage training method to progressively optimize network performance.

[0066] In the first stage, the three branch networks and the fusion network subsystem are each given their own input data and trained independently according to a loss function tailored to their output. That is, the low-frequency branch network only uses L... if To reduce the training frequency, the high-frequency branch network only uses L hf To reduce the training, the semantic information extraction branch only uses L sem To perform specification training, the fusion network subsystem uses L fusion This divide-and-conquer strategy allows each sub-network to focus on its specific task, thus maximizing its potential. However, it's important to note that the input required for the fusion network subsystem is the output of the three branch networks, which cannot be directly obtained in the first stage. To address this issue, this invention cleverly employs a surrogate input strategy. For low-frequency and high-frequency features, this invention obtains approximate image features by applying Gaussian blur and Laplacian edge detection to the ground truth image, which are then used as surrogate inputs to the fusion network subsystem. For coarse dehazing results, this invention directly uses the original hazy image as a surrogate input for initial training.

[0067] In the second stage, this invention relaxes the constraints on intermediate variables, retaining only the loss function of the fusion network subsystem for training, that is, using only L... fusion This strategy regulates the training of the entire network. It allows sub-networks to be fine-tuned in the second stage after fully learning their specific tasks in the initial training phase, resulting in better collaboration. This two-stage training method ensures that each part of the model is sufficiently optimized before being combined into a complete dehazing system. This not only improves the feasibility of training but also contributes to achieving superior dehazing performance.

[0068] During the training phase, this invention empirically sets the total number of iterations on the RESIDE-6K dataset to 500, and this applies to both phases. For other datasets, the number of iterations is set so that the total number of iterations is equal to that on the RESIDE-6K dataset. The batch size for the first phase of training is set to 8, and the batch size for the second phase of training is set to 3. The Adam optimizer is used. The initial learning rate is set to 2e-4, and training is performed using a cosine annealing strategy.

[0069] Furthermore, to ensure the stability and efficiency of the image processing subsystem, rigorous testing and verification of each subsystem are necessary. Through testing in real-world fog scenarios, the network structure and parameter settings are continuously adjusted and optimized to ensure the image processing subsystem can adapt to various complex fog environments and achieve efficient and accurate defogging.

[0070] In summary, by constructing an image processing subsystem that includes low-frequency branch networks, high-frequency branch networks, semantic branch networks, and fusion network subsystems, this invention can achieve comprehensive and accurate processing of fog images, providing high-quality input data for subsequent image analysis and applications. Figure 4 A schematic diagram of the defogging effect using the visual defogging system of this invention is provided. It can be seen that the present invention has a good defogging effect and can obtain a relatively clear defogging image.

[0071] In the visual dehazing system of this invention, the high-performance transmission subsystem is responsible for the efficient and stable transmission of the original image and the dehazed image processed by the image processing subsystem. The following are the specific construction steps and key points of the high-performance transmission subsystem.

[0072] First, select suitable high-performance network equipment and interfaces. This includes, but is not limited to, high-speed network cards, fiber optic interfaces, and dedicated image transmission equipment. These devices need to have high speed and low latency characteristics to ensure that image data can be transmitted in a short time to meet real-time requirements.

[0073] Secondly, an efficient image compression algorithm needs to be designed. Since both the original and dehazed images may contain a large amount of data, direct transmission would consume significant bandwidth and storage resources. Therefore, advanced image compression algorithms are required to efficiently compress image data, reducing the time and bandwidth required for transmission.

[0074] Meanwhile, to achieve stable transmission of image data, it is also necessary to consider the selection and optimization of network transmission protocols. Choosing a suitable network protocol, such as TCP / IP, and adjusting and optimizing parameters according to the actual application scenario, ensures the stability and reliability of image data during transmission.

[0075] Furthermore, to ensure the high performance of the transmission subsystem, the transmission process also needs to be monitored and managed. By monitoring indicators such as network status, transmission speed, and packet loss rate in real time, potential problems can be identified and resolved in a timely manner, ensuring the integrity and accuracy of image data.

[0076] Finally, to further improve transmission efficiency, parallel or multiplexing transmission methods can be considered. By utilizing multiple transmission channels or devices simultaneously, image data can be transmitted in parallel, further improving transmission speed and efficiency.

[0077] In summary, by selecting high-performance network devices and interfaces, designing efficient image compression algorithms, optimizing network transmission protocols, and implementing effective monitoring and management measures, this invention can construct a high-performance transmission subsystem, enabling fast and stable transmission of image data and providing strong support for the overall performance of the visual dehazing system.

[0078] In the visual dehazing system of this invention, the monitoring and storage subsystem is responsible not only for real-time decoding and visualization of the transmitted dehazed images, but also for persistently storing the decoded image data for subsequent analysis and processing. The following are the detailed construction steps and key functions of the monitoring and storage subsystem.

[0079] First, to address the real-time decoding and visualization requirements of dehazed images, the monitoring and storage subsystem is equipped with a high-performance image decoder. This decoder can quickly and accurately decode the transmitted compressed image data into visualized images, ensuring both real-time performance and clarity. Simultaneously, the decoder connects to a high-definition display, enabling real-time display of the dehazed image and providing monitoring personnel with intuitive and clear visual information.

[0080] Secondly, to meet the persistent storage requirements of image data, the monitoring and storage subsystem employs high-performance disk arrays as storage devices. These devices not only offer large storage capacity but also boast extremely fast read and write speeds, ensuring rapid writing and stable storage of image data. Furthermore, by adopting advanced storage technologies and management strategies, such as data redundancy and backup mechanisms, the reliability and security of the stored data are ensured.

[0081] In addition to basic decoding and storage functions, the monitoring and storage subsystem also has some extended capabilities. For example, it can automatically analyze and identify stored image data through integrated intelligent analysis algorithms, extracting key information or abnormal events to provide monitoring personnel with more advanced decision support. Furthermore, the subsystem supports remote access and control functions, allowing users to view and manage stored image data anytime, anywhere.

[0082] Finally, to ensure the stable operation and high performance of the monitoring and storage subsystem, regular system maintenance and optimization are necessary. This includes checking the operating status of hardware devices, updating software versions and drivers, and optimizing storage structure and management strategies. Regular maintenance and optimization ensure that the subsystem is always in optimal condition, providing reliable data support and assurance for the visual dehazing system.

[0083] In summary, by constructing a high-performance image decoder, employing advanced storage devices and technologies, integrating intelligent analysis algorithms, and implementing remote access and control functions, the monitoring and storage subsystem can achieve real-time decoding, visualization, and persistent storage of dehazed images, providing strong support for the stable operation and efficient application of the entire visual dehazing system.

[0084] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, several equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or purpose, should be considered within the scope of protection of the present invention.

Claims

1. A visual dehazing system based on a three-branch neural network, characterized in that, include: The feature extraction subsystem includes a parallel low-frequency branch network, a high-frequency branch network, and a semantic branch network. The low-frequency branch network is used to extract low-frequency image features from the original image and recover a first coarse dehazing result. The high-frequency branch network is used to extract high-frequency image features from the original image and recover a second coarse dehazing result. The semantic branch network is used to identify different objects and regions in the original image to obtain image semantic features and recover a third coarse dehazing result that is consistent with the semantics of the original image. A fusion network subsystem is connected to the output of the low-frequency branch network, the high-frequency branch network, and the semantic branch network. It is used to fuse the low-frequency image features, the first coarse dehazing result, the high-frequency image features, the second coarse dehazing result, the third coarse dehazing result, and the original image to generate a dehazed image. The fusion network subsystem uses the Taylor attention module as its basic building block and includes a fourth encoder and a fourth decoder; The fourth encoder includes three downsampling levels, each of which includes a downsampler and a T-TB module connected in sequence; the T-TB module is the Taylor Transformer Block. The fourth decoder includes three upsampling levels, each of which includes an upsampler, a fusion module, and a T-TB module connected in sequence. Each of the T-TB modules includes multiple Taylor attention modules; The output of each layer of the fourth encoder is connected to the fourth decoder via a skip connection; The fourth encoder encodes the low-frequency image features, the first coarse dehazing result, the high-frequency image features, the second coarse dehazing result, the third coarse dehazing result, and the stitching of the original image in the channel dimension as input. The input of the fourth encoder is also connected to the output of the fourth decoder through skip connections for weighted fusion to generate the dehazed image.

2. The visual defogging system as described in claim 1, characterized in that, Also includes: A visible light sensing system for fog scenes, used to capture the original image in a foggy environment.

3. The visual defogging system as described in claim 2, characterized in that, Also includes: A transmission subsystem, connected to the output of the fusion network subsystem and the output of the visible light sensing system, is used to encode, compress, and transmit the dehazed image and the original image.

4. The visual defogging system as described in claim 3, characterized in that, Also includes: A monitoring and storage subsystem, connected to the transmission subsystem, is used to decode and visualize the transmitted dehazed image in real time, and to store the decoded dehazed image.

5. The visual defogging system as described in claim 1, characterized in that, The low-frequency branch network is a U-shaped network, including a first encoder and a first decoder; The first encoder uses a sliding window self-attention block as the basic building block to construct two downsampling levels. Each downsampling level includes a downsampler and an S-TB module connected in sequence. The S-TB module includes multiple sliding window self-attention blocks. The sliding window self-attention blocks perform self-attention operations on local areas of the image through the sliding window pattern. The S-TB module is also known as the Swin-Transformer Block. The first decoder uses depthwise separable convolution as the basic building block to construct two upsampling levels. Each upsampling level includes an upsampler, a fusion module and a DW convolution module connected in sequence. The DW convolution module includes multiple depthwise separable convolutions. The output of each layer of the first encoder is connected to the first decoder via a skip connection; The input of the first encoder is connected to the output of the first decoder for weighted fusion to obtain the output of the low-frequency branch network.

6. The visual defogging system as described in claim 5, characterized in that, In each of the sliding window self-attention blocks, layer normalization is first performed, then the feature map is divided into multiple windows, and a multi-head self-attention operation is performed within each window. The self-attention operation for each head is as follows: in, Q , K , V These represent the query, key, and value in the Transformer, respectively, all generated by convolution. d k It is the dimension of the key vector; Softmax ( ) is accessed via a window; The self-attention operation allows for the extraction of local information within each window, enabling the extraction of local low-frequency features while maintaining manageable computational complexity.

7. The visual defogging system as described in claim 1, characterized in that, The high-frequency branch network is a U-shaped network, including a second encoder and a second decoder; The second encoder includes four downsampling levels, each of which includes a discrete wavelet transform downsampler and a multi-scale convolution module connected in sequence. The discrete wavelet transform downsampler includes parallel discrete wavelet transform operations and convolution operations. The second decoder includes four upsampling levels, each of which includes an upsampler, a fusion module, and a multi-scale convolution module connected in sequence. The outputs of the discrete wavelet transform downsamplers of the first three downsampling levels of the second encoder are connected to the third upsampling level, the second upsampling level, and the first upsampling level of the second decoder via skip connections, respectively. The input of the second encoder is connected to the output of the second decoder for weighted fusion to obtain the output of the high-frequency branch network.

8. The visual defogging system as described in claim 1, characterized in that, The semantic branching network includes a third encoder and a third decoder; The third encoder uses Res2Net pre-trained on ImageNet to extract abstract semantic information from the image; The third decoder employs a channel-level attention module to perform global pooling on the input feature map. Subsequently, it is processed by a multilayer perceptron and activation function to generate a feature vector that matches the channel dimension. This feature vector is then multiplied channel by channel with the feature map to finally obtain the third coarse dehazing result.

9. A visual dehazing method based on a three-branch neural network, characterized in that, Includes the following steps: A low-frequency branch network is constructed to extract low-frequency image features from the original image and recover the first coarse dehazing result; A high-frequency branch network is constructed to extract high-frequency image features from the original image and recover the second coarse dehazing result; A semantic branching network is constructed to identify different objects and regions in the original image to obtain image semantic features, and a third coarse dehazing result consistent with the semantics of the original image is recovered. A fusion network subsystem is constructed to generate a dehazed image by fusing the low-frequency image features, the first coarse dehazing result, the high-frequency image features, the second coarse dehazing result, the third coarse dehazing result, and the original image. The fusion network subsystem uses the Taylor attention module as its basic building block and includes a fourth encoder and a fourth decoder; The fourth encoder includes three downsampling levels, each of which includes a downsampler and a T-TB module connected in sequence; the T-TB module is the Taylor Transformer Block. The fourth decoder includes three upsampling levels, each of which includes an upsampler, a fusion module, and a T-TB module connected in sequence. Each of the T-TB modules includes multiple Taylor attention modules; The output of each layer of the fourth encoder is connected to the fourth decoder via a skip connection; The fourth encoder encodes the low-frequency image features, the first coarse dehazing result, the high-frequency image features, the second coarse dehazing result, the third coarse dehazing result, and the stitching of the original image in the channel dimension as input. The input of the fourth encoder is also connected to the output of the fourth decoder through skip connections for weighted fusion to generate the dehazed image.

Images