Lightweight image real-time restoration system and method thereof

Abstract

The invention relates to the technical field of vehicle-mounted image restoration, and provides a lightweight image real-time restoration system, which comprises a front image sensor, an embedded processing unit and a rear auxiliary driving decision module, the front image sensor is used for collecting a degraded image, the embedded processing unit is used for deploying a RepOneRestore network model, and the rear auxiliary driving decision module is used for receiving a restored image to carry out target identification. The problems of real-time performance and dynamic weather recovery capability are solved through RepViT lightweight design + DSCST frequency domain attention + SCCSF double-path fusion, the PSNR is increased by 1.9 dB while the attention calculation amount is reduced by 30% through frequency domain masks, and the downstream target detection false alarm rate is reduced by 35-45% through an SCCSF module.

Description

Technical Field

[0001] This invention relates to the field of vehicle image restoration technology, and more specifically, to a lightweight real-time image restoration system and method. Background Technology

[0002] An in-vehicle driver assistance system is an electronic system that uses various sensors (such as cameras, radar, lidar, and ultrasonic sensors) installed on a vehicle to perceive the surrounding environment, and combines this with vehicle status information. The system then uses algorithms to process and analyze the data, thereby providing assistance functions or proactive intervention for the driver in specific scenarios to improve driving safety and comfort.

[0003] Currently, existing vehicle-mounted driver assistance systems suffer from significant performance degradation in downstream tasks such as target detection and semantic segmentation due to image sensor degradation (e.g., blurring, occlusion, noise) in adverse weather conditions such as rain, snow, and fog, making it difficult to meet the real-time requirements of vehicle image usage.

[0004] To address the aforementioned issues, this application proposes a lightweight real-time image restoration system and method. Summary of the Invention

[0005] The purpose of this invention is to provide a lightweight real-time image restoration system and method, which solves the problems of real-time performance and dynamic image restoration in the prior art by combining the lightweight RepViT design with DSCST frequency domain attention and SCCSF dual-path fusion.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A lightweight real-time image restoration system includes a front-mounted image sensor and an embedded processing unit, as well as a rear-mounted driver assistance decision-making module;

[0008] The front-mounted image sensor is used to acquire degraded images, the embedded processing unit is used to deploy the RepOneRestore network model, and the rear-mounted driver assistance decision module is used to receive the restored images for target recognition.

[0009] As a further optimization of the present invention, the RepOneRestore network model includes a degraded image input unit, an encoder unit, an SCCSF module, a decoder unit, and a restored image output unit.

[0010] As a further optimization of the present invention, the encoder unit is connected to the SCCSF module by a RepViT branch and a DSCST branch. The RepViT branch is used to construct a multi-scale feature extraction layer, and the DSCST branch performs Haar wavelet transform on the input image to extract high-frequency components as degradation region indicators.

[0011] As a further optimization of the present invention, the SCCSF module includes a channel attention branch and a spatial attention branch, both of which are connected to a feature fusion weighting unit, and the feature fusion weighting unit is connected to a fused feature output unit.

[0012] A real-time image restoration method includes: constructing a U-shaped network architecture that combines CNN and Transformer in parallel;

[0013] Multi-scale low-level features are extracted using RepViT branch units, and cross-scale associations are established.

[0014] The DSCST branch unit of dynamic spatial channel scene transformation is used to selectively enhance the degraded region;

[0015] The SCCSF module, which uses skip-connection channel spatial feature fusion, is used for adaptive fusion of encoding and decoding features.

[0016] As a further optimization of the present invention, the RepViT branch unit achieves lightweight multi-scale feature extraction by reparameterizing the concatenation of MLP and depthwise separable convolution.

[0017] As a further optimization of the present invention, the DSCST module includes a frequency degradation region detection unit based on Haar wavelet variation, and also includes a spatial channel self-attention mechanism.

[0018] As a further optimization of the present invention, the input image is subjected to two-level Haar wavelet decomposition to extract high-frequency components in the horizontal and vertical directions and then threshold binarized.

[0019] As a further optimization of the present invention, the channel attention branch calculates the feature channel weights.

[0020] As a further optimization of the present invention, spatial attention branches generate spatial weights.

[0021] The beneficial effects of this invention are:

[0022] This invention achieves a breakthrough in overcoming the contradiction between real-time performance and accuracy by combining a lightweight architecture design with precise processing of degradation regions and cross-modal feature optimization.

[0023] RepViT ensures speed and basic capabilities, DSCST accurately restores the negative impact of severe weather in images, and SCCSF intelligently combines information from various parts. Ultimately, a lightweight model achieves fast and high-quality image restoration, significantly improving image processing speed and providing an efficient solution for tasks such as real-time vehicle image restoration. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the system module composition of the present invention;

[0026] Figure 2 for Figure 1 A schematic diagram of the SCCSF module structure. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Please see Figure 1-2 This invention provides an embodiment of a lightweight real-time image restoration system and method:

[0029] A lightweight real-time image restoration system includes a front-mounted image sensor and an embedded processing unit, as well as a rear-mounted driver assistance decision-making module;

[0030] The front-facing image sensor is used to acquire degraded images, and the embedded processing unit is NVIDIA Jetson AGXXavier with a processing latency of ≤0.015 seconds / frame. It is used to deploy the RepOneRestore network model. The rear-facing assisted driving decision module is used to receive the restored images for target recognition. It can be applied to systems such as autonomous driving, drone monitoring, or security patrol.

[0031] The RepOneRestore network model includes a degraded image input unit, an encoder unit, an SCCSF module, a decoder unit, and a restored image output unit.

[0032] In addition, the encoder unit is connected to the SCCSF module by the RepViT branch and the DSCST branch. The RepViT branch is a lightweight parallel path that combines the inverse residual block of MobileNetV3 with the reparameterized MLP of RepViT to construct a multi-scale feature extraction layer. The efficient discriminant module of the DSCST branch performs Haar wavelet transform on the input image and extracts high-frequency components as indicators of degradation regions. At the same time, it generates a dynamic weight matrix through 1×1 convolution to suppress the computational cost of non-degradation regions.

[0033] Furthermore, the SCCSF module has a channel attention branch and a spatial attention branch, both of which are connected to the feature fusion weighting unit to encode and fuse the outputs of the two branches. The feature fusion weighting unit is connected to the fused feature output unit, which is connected to the decoder unit.

[0034] A real-time image restoration method, comprising:

[0035] A U-shaped network architecture with parallel CNN and Transformer is constructed to achieve efficient end-to-end restoration and meet the real-time requirements of vehicle-mounted applications. Multi-scale low-level features are extracted through RepViT branch units and cross-scale correlations are established to improve the image restoration accuracy. DSCST branch units with dynamic spatial channel scene transformation are used to selectively enhance degraded regions.

[0036] An SCCSF module with skip-connection channel spatial feature fusion is used for adaptive fusion of encoding and decoding features;

[0037] Among them, the RepViT branch unit achieves lightweight multi-scale feature extraction by reparameterizing the concatenation of MLP and depthwise separable convolution, and reduces the number of parameters by depthwise separable convolution, thereby improving the speed of multi-scale feature extraction by 2 to 4 times.

[0038] In addition, the DSCST module includes a frequency degradation region detection unit based on Haar wavelet transforms, and also includes a spatial channel self-attention mechanism, whose attention weight matrix is ​​generated by the following formula:

[0039] Softmax is a learnable parameter. For frequency domain mask;

[0040] Frequency domain mask generation involves performing a two-level Haar wavelet decomposition on the input image, extracting high-frequency components in the horizontal and vertical directions, and then performing threshold binarization.

[0041]

[0042] Channel attention branch calculates feature channel weights.

[0043]

[0044] Spatial attention branches generate spatial weights.

[0045]

[0046] The final fusion feature calculation is as follows:

[0047] , Application channel attention weighting, where, It is element-wise multiplication. Apply spatial attention weighting; among which, It is element-wise multiplication, the operator Add the two weighted results element by element.

[0048] Understandably, this invention addresses the issues of real-time performance and dynamic weather recovery capabilities through RepViT lightweight design, DSCST frequency domain attention, and SCCSF dual-path fusion. It reduces the attention computation load by 30% while improving performance through frequency domain masking. Furthermore, the SCCSF module reduces the false alarm rate of downstream target detection by 35-45%.

[0049] In the description of this invention, unless otherwise stated, "a plurality of" means two or more; it should be understood that the terms "opening", "upper", "lower", "thickness", "top", "middle", "length", "inner", "around", etc., which indicate orientation or positional relationship, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the components or elements referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as limiting this invention.

[0050] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A lightweight real-time image restoration system, comprising a front-mounted image sensor and an embedded processing unit, and a rear-mounted driver assistance decision-making module, characterized in that: The front-mounted image sensor is used to acquire degraded images, the embedded processing unit is used to deploy the RepOneRestore network model, and the rear-mounted driver assistance decision module is used to receive the restored images for target recognition.

2. The lightweight real-time image restoration system according to claim 1, characterized in that: The RepOneRestore network model includes a degraded image input unit, an encoder unit, an SCCSF module, a decoder unit, and a restored image output unit.

3. The lightweight real-time image restoration system according to claim 2, characterized in that: The encoder unit is connected to the SCCSF module via a RepViT branch and a DSCST branch. The RepViT branch is used to construct a multi-scale feature extraction layer, and the DSCST branch performs Haar wavelet transform on the input image to extract high-frequency components as degradation region indicators.

4. The lightweight real-time image restoration system according to claim 3, characterized in that: The SCCSF module includes a channel attention branch and a spatial attention branch. The outputs of the channel attention branch and the spatial attention branch are weighted and then input into the fusion feature output unit.

5. A real-time image restoration method, used in the lightweight real-time image restoration system of claim 4, characterized in that, include: Construct a U-shaped network architecture that combines CNN and Transformer in parallel; Multi-scale low-level features are extracted using RepViT branch units, and cross-scale associations are established. The DSCST branch unit of dynamic spatial channel scene transformation is used to selectively enhance the degraded region; An SCCSF module with skip-connection channel spatial feature fusion is used for adaptive fusion of encoding-decoding features.

6. The real-time image restoration method according to claim 5, characterized in that: The RepViT branch unit achieves lightweight multi-scale feature extraction by cascading a reparameterized MLP with a depthwise separable convolution.

7. The real-time image restoration method according to claim 5, characterized in that: The DSCST module includes a frequency degradation region detection unit based on Haar wavelet variations, and also includes a spatial channel self-attention mechanism.

8. The real-time image restoration method according to claim 7, characterized in that: The input image is subjected to two-level Haar wavelet decomposition to extract high-frequency components in the horizontal and vertical directions and then thresholded and binarized.

9. The real-time image restoration method according to claim 7, characterized in that: The channel attention branch calculates the feature channel weights.

10. The real-time image restoration method according to claim 7, characterized in that: Spatial attention branches generate spatial weights.

Images