Image quality enhancement method and system and vehicle
By using the SwinTG network model in computer vision to enhance the quality of images collected in harsh environments, the problem of poor image imaging quality is solved, the accuracy of obstacle detection and environmental perception is improved, and the safety of autonomous driving is enhanced.
Patent Information
- Application Number
- CN202411840495.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-13
- Publication Date
- 2025-04-15
AI Technical Summary
In harsh environments, the image imaging quality of scene images is poor, affecting the accuracy of obstacle detection and vehicle body environment perception.
The SwinTG network model built on the window attention mechanism network structure and the GoogLeNet network structure is adopted to enhance the quality of the collected environmental images and remove noise in the image.
By removing noise, image quality is improved, obstacle detection and vehicle body environment perception are enhanced, and driving safety is improved during autonomous driving or unmanned driving.
Smart Images
Figure CN119831909A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing and provides an image quality enhancement method, system and vehicle. Background Art
[0002] With the development of technologies such as big data and deep learning, artificial intelligence technology has achieved breakthrough developments in many fields. Among them, computer vision technology is one of the most convenient and fastest technical means to perceive the external environment.
[0003] In the field of computer vision, object detection is one of the most basic and critical tasks, which aims to identify the category and location of objects in images and provide basic environmental information for scene analysis. The rapid advancement of intelligent connected vehicle technology is a direct product of the evolution of artificial intelligence.
[0004] The object detection algorithm in computer vision is applied to obstacle detection and vehicle environment perception in assisted driving technology or autonomous driving technology. However, when collecting scene images in harsh environments such as ice and snow, heavy rain, and dense fog, the imaging quality of the scene images is poor, which in turn affects the accuracy of obstacle detection and vehicle environment perception in the current environment. Summary of the invention
[0005] In view of this, the present application provides an image quality enhancement method, aiming to improve the above-mentioned problem.
[0006] Specifically, the following technical solutions are included:
[0007] On the one hand, an embodiment of the present application provides a method for enhancing image quality, and the method is specifically as follows:
[0008] The currently collected environment image is input into the image quality enhancement model, and the image quality enhancement model outputs a quality-enhanced environment image, where part of the noise in the currently collected environment image is removed from the quality-enhanced environment image.
[0009] In some embodiments of the present invention, the image quality enhancement model uses a SwinTG network model based on a window attention mechanism network structure and a GoogLeNet network structure. The SwinTG network model includes:
[0010] A shallow feature extraction module, a deep feature extraction module and a quality image reconstruction module connected in sequence;
[0011] The shallow feature extraction module is used to extract the shallow features of the environment image; the deep feature extraction module is used to extract the deep features of the environment image; and the quality image reconstruction module is used to reconstruct the image quality of the environment image.
[0012] In some embodiments of the present invention, the deep feature extraction module includes: six RSTB modules connected in series in sequence; and a 3×3 convolutional layer connected to the last RSTB module.
[0013] In some embodiments of the present invention, the RSTB module is composed of six Swin-Transformer modules connected in series in sequence, the last Swin-Transformer module is connected to the Inception model, the feature map is input into the first Swin-Transformer module, and is connected to the output residual of the Inception model.
[0014] In some embodiments of the present invention, the structure of the Swin-Transformer module is as follows:
[0015] The first LayerNorm layer and the W-MSA module, the feature map P1 is input into the first LayerNorm layer, the feature map P1 is connected with the output residual of the W-MSA module to obtain the feature map P2;
[0016] The second LayerNorm layer and the first MLP neural network model, the feature map P2 is input into the second LayerNorm layer, the feature map P2 is connected with the output residual of the first MLP neural network model to obtain the feature map P3;
[0017] The third LayerNorm layer and the SW-MSA module, the feature map P3 is input into the third LayerNorm layer, and the feature map P3 is connected with the output residual of the SW-MSA module to obtain the feature map P4;
[0018] The fourth LayerNorm layer and the second MLP neural network model, the feature map P4 is input into the fourth LayerNorm layer, and the feature map P4 is connected to the output residual of the second MLP neural network model.
[0019] In some embodiments of the present invention, the Inception model includes:
[0020] The first 1×1 convolutional layer;
[0021] A second 1×1 convolutional layer, a third 1×1 convolutional layer, a fourth 1×1 convolutional layer, and a two-dimensional average pooling layer connected to the first 1×1 convolutional layer;
[0022] The first 3×3 convolutional layer connected to the third 1×1 convolutional layer, the second 3×3 convolutional layer connected to the first 3×3 convolutional layer;
[0023] A 5×5 convolutional layer connected to the fourth 1×1 convolutional layer;
[0024] The fifth 1×1 convolutional layer connected to the average pooling layer; the sixth 1×1 convolutional layer connected to the second 1×1 convolutional layer, the second 3×3 convolutional layer, the 5×5 convolutional layer, and the fifth 1×1 convolutional layer;
[0025] GeLU activation function connected to the sixth 1×1 convolutional layer.
[0026] In some embodiments of the present invention, the SwinTG network model is trained through samples, and the sample construction process is specifically as follows:
[0027] Use a sliding window to crop an image patch from the noise-free image;
[0028] Performing geometric transformation on the image block, the transformation methods include: flipping the original image and rotating the image;
[0029] Randomly select a value from the noise level σ∈[0,75] range, generate Gaussian white noise with the noise level σ, and add the Gaussian white noise to the image block to obtain a noisy image, and use the noisy image as a sample.
[0030] In some embodiments of the present invention, the loss function L of the SwinTG network model is loss The details are as follows:
[0031]
[0032] in, is the denoised image output by the SwinTG network model for the i-th noise image sample, x i is the noise-free image corresponding to the i-th noise image sample.
[0033] On the other hand, an embodiment of the present application provides an image quality enhancement system, the system comprising:
[0034] An image acquisition unit, a processing unit connected to the image acquisition unit, and an output unit connected to the processing unit, wherein the processing unit is integrated with the above-mentioned image quality enhancement model;
[0035] Among them, the image acquisition unit is used to acquire environmental images, and the acquired environmental images are input into the image quality enhancement model in the processing unit. The image quality enhancement model enhances the image quality of the input environmental images, and outputs the environmental images with enhanced image quality through the output unit.
[0036] On the other hand, an embodiment of the present application provides a vehicle on which the above-mentioned image quality enhancement system is integrated.
[0037] When the vehicle is driving automatically or unmanned, the environmental image collected by the image acquisition unit (camera) is input into the image enhancement system, and the image quality enhancement system outputs the quality-enhanced image. The quality-enhanced image has most of the noise removed. The environmental image with image quality enhancement is conducive to accurate perception of the environment or more accurate identification of target objects in the environment, which greatly improves driving safety during automatic or unmanned driving. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0039] Figure 1 A schematic diagram of the structure of the SwinTG network structure provided by an embodiment of the present invention;
[0040] Figure 2 A schematic diagram of the structure of the RSTB module provided in an embodiment of the present invention;
[0041] Figure 3 A schematic diagram of the structure of a Swin-Transformer module provided in an embodiment of the present invention;
[0042] Figure 4 A schematic diagram of the structure of the Inception model provided in an embodiment of the present invention;
[0043] Figure 5 A schematic diagram of the structure of an image quality enhancement system provided by an embodiment of the present invention;
[0044] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0046] Unless otherwise defined, all technical terms used in the embodiments of the present application have the same meanings as commonly understood by those of ordinary skill in the art.
[0047] The image quality enhancement method provided by an embodiment of the present invention is specifically as follows: the currently collected environmental image is input into an image quality enhancement model, and the image quality enhancement model outputs a quality-enhanced environmental image. Compared with the collected original environmental image, the quality-enhanced environmental image removes most of the noise in the environmental image, which is beneficial to the perception of the environment or the recognition of target objects in the environment.
[0048] The image quality enhancement model is built based on the window attention mechanism network structure (Swin-Transformer) and the GoogLeNet network structure, referred to as the SwinTG network model. The SwinTG network model is trained, and the training process is as follows:
[0049] Constructing a sample set, in order to speed up the training speed of the model and the diversity of training data, first, use a sliding window to crop a small image block from a synthetic image (such as a fringe image, no noise); then, perform a random geometric transformation on the image block, including: flipping the original image and rotating the image, such as rotating 90°, 180° and 270°; finally, randomly select a value from the noise level σ∈[0,75] range, generate Gaussian white noise with the noise level σ, and add the Gaussian white noise to the image block to obtain a noisy image, which can ensure the diversity of data in each iteration, thereby improving the generalization ability and robustness of the model.
[0050] The noisy images are taken as samples, and the samples are divided into training samples and test samples. The SwinTG network model is trained based on the training samples. After the training of the SwinTG network model is completed, the SwinTG network model is tested based on the test samples. When the loss of the SwinTG network model reaches the set standard or the SwinTG network model is in a convergence state, the SwinTG network model can be used to enhance the quality of the scene image, and an image quality enhancement model is also formed.
[0051] The present invention uses ADAM optimizer to optimize the parameters (weights) of the SwinTG network model. Through the back propagation of the loss function, the weights of the SwinTG network model are continuously updated, so that the SwinTG network model reaches a convergent state. The loss function L of the SwinTG network model loss The expression is:
[0052]
[0053] in, is the denoised stripe image output by the SwinTG network model for the i-th noise stripe image sample, x iis the synthetic image (that is, noise-free image) corresponding to the i-th noise fringe image sample, and N is the number of samples.
[0054] Figure 1 A schematic diagram of the structure of the SwinTG network structure provided in an embodiment of the present invention, the network structure includes:
[0055] A shallow feature extraction module, a deep feature extraction module and a quality image reconstruction module are connected in sequence, wherein the shallow feature extraction module is used to extract shallow features of the environment image, and the shallow feature extraction module is composed of a 3×3 convolutional layer Conv; the deep feature extraction module is used to extract deep features of the environment image, and the deep feature extraction module is composed of several RSTB modules and a 3×3 convolutional layer. Specifically, the layer feature extraction module includes: six RSTB modules connected in series in sequence, and a 3×3 convolutional layer connected to the last RSTB module; the quality image reconstruction module is used to reconstruct the image quality of the environment image, and the quality image reconstruction module is composed of a 3×3 convolutional layer Conv.
[0056] Figure 2 A structural schematic diagram of the RSTB module provided in an embodiment of the present invention, wherein the RSTB module is composed of multiple Swin-Transformer (STL) modules and an Inception model. Specifically, the RSTB module is composed of six Swin-Transformer modules connected in series in sequence, the last Swin-Transformer module is connected to the Inception model, the feature map is input into the first Swin-Transformer module, and is connected to the output residual of the Inception model.
[0057] The Swin-Transformer module can be used to model long-range dependencies, which has the advantage of convolution for processing large-size images; image information and attention weights can interact and can be used as spatially varying convolutions.
[0058] Figure 3 FIG. 1 is a schematic diagram of the structure of a Swin-Transformer module provided in an embodiment of the present invention. The structure of the Swin-Transformer module is as follows:
[0059] The first LayerNorm layer and the W-MSA module, the feature map P1 is input into the first LayerNorm layer, the feature map P1 is connected with the output residual of the W-MSA module to obtain the feature map P2;
[0060] The second LayerNorm layer and the first MLP neural network model, the feature map P2 is input into the second LayerNorm layer, the feature map P2 is connected with the output residual of the first MLP neural network model to obtain the feature map P3;
[0061] The third LayerNorm layer and the SW-MSA module, the feature map P3 is input into the third LayerNorm layer, and the feature map P3 is connected with the output residual of the SW-MSA module to obtain the feature map P4;
[0062] The fourth LayerNorm layer and the second MLP neural network model, the feature map P4 is input into the fourth LayerNorm layer, and the feature map P4 is connected to the output residual of the second MLP neural network model.
[0063] Based on the Transformer's multi-headed self-attention (MSA) mechanism, the Swin-Transformer model limits the MSA calculation to local non-overlapping windows by moving the window, reducing the amount of model calculation. At the same time, the information of the upper and lower windows can be effectively interacted, thereby enhancing the global modeling ability of the network. Given an input image H×W×C, the Swin-Transformer model first divides the input image into M×M image blocks, and then flattens the image in the channel dimension to (H×W / M 2 )×M 2 ×C, then the data enters the Swin-Transformer module, the number of windows is H×W / M 2 , MSA calculation is performed for each window separately.
[0064] Figure 4 A schematic diagram of the structure of the Inception model provided in an embodiment of the present invention, wherein the Inception model includes:
[0065] The first 1×1 convolutional layer;
[0066] A second 1×1 convolutional layer, a third 1×1 convolutional layer, a fourth 1×1 convolutional layer, and a two-dimensional average pooling layer connected to the first 1×1 convolutional layer;
[0067] The first 3×3 convolutional layer connected to the third 1×1 convolutional layer, the second 3×3 convolutional layer connected to the first 3×3 convolutional layer;
[0068] A 5×5 convolutional layer connected to the fourth 1×1 convolutional layer;
[0069] The fifth 1×1 convolutional layer connected to the average pooling layer; the sixth 1×1 convolutional layer connected to the second 1×1 convolutional layer, the second 3×3 convolutional layer, the 5×5 convolutional layer, and the fifth 1×1 convolutional layer;
[0070] GeLU activation function connected to the sixth 1×1 convolutional layer.
[0071] The Inception model contains four parallel convolutional lines. The first three lines use convolutional layers with kernel sizes of 1×1, 3×3, and 5×5 to extract information of different spatial scales from image features. The two middle lines first perform a convolution operation with a kernel size of 1×1 on the input data to reduce the number of input channels of the data, thereby reducing the complexity of the model. The fourth line uses average pooling, followed by a convolutional layer with a kernel size of 1×1 to change the number of channels. In order to keep the output of the data consistent in the height (H) and width (W) dimensions, the convolutions in the four lines use appropriate padding, and finally the output data of the four lines are connected in the channel dimension.
[0072] Figure 5 The schematic diagram of the structure of the image quality enhancement system provided by the embodiment of the present invention, for the convenience of description, only the part related to the embodiment of the present invention is shown, the system comprises:
[0073] An image acquisition unit, a processing unit connected to the image acquisition unit, and an output unit connected to the processing unit, wherein the processing unit is integrated with an image quality enhancement model;
[0074] Among them, the image acquisition unit is used to acquire environmental images, and the acquired environmental images are input into the image quality enhancement model in the processing unit. The image quality enhancement model enhances the image quality of the input environmental images, and outputs the environmental images with enhanced image quality through the output unit. The present invention enhances the quality of the environmental images acquired in real time to remove noise in the environmental images as much as possible. The environmental images with enhanced image quality are conducive to accurate perception of the environment or more accurate identification of target objects in the environment.
[0075] An embodiment of the present invention also provides a vehicle, which is integrated with the above-mentioned image quality enhancement system. When the vehicle is automatically or unmanned, the environmental image captured by the image acquisition unit (camera) is input into the image enhancement system, and the image quality enhancement system outputs a quality-enhanced image. The quality-enhanced image has most of the noise removed. The environmental image with image quality enhancement is conducive to accurate perception of the environment or more accurate identification of target objects in the environment, which greatly improves driving safety during automatic or unmanned driving.
[0076] The SwinTG network model provided by the present invention has the following advantages:
[0077] (1) Swin-TransformerInception can model long-range dependencies and realize the interaction between image information. In order to make up for the insufficient local feature extraction capability of the Swin-TransformerInception network, the Inception network is used in the RSTB module. Through different convolutions, rich local image information can be obtained, thereby improving the local feature modeling capability of the model.
[0078] (2) SwinTG has a deep network structure, which is prone to the problem of vanishing network gradients, resulting in a decrease in network performance. In order to ensure that the network training accuracy does not decrease with the increase of network depth, multiple residual blocks are inserted into the overall network framework. The residual blocks map the image features of the shallow network to the deep network, ensuring that the deep network contains richer feature information than the shallow network, while being able to capture subtle changes in the network.
[0079] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the present application disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only.
[0080] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A method for enhancing image quality, characterized in that: The method is specifically as follows: The currently collected environment image is input into the image quality enhancement model, and the image quality enhancement model outputs a quality-enhanced environment image, where part of the noise in the currently collected environment image is removed from the quality-enhanced environment image.
2. The image quality enhancement method according to claim 1, characterized in that: The image quality enhancement model uses the SwinTG network model based on the window attention mechanism network structure and the GoogLeNet network structure. The SwinTG network model includes: A shallow feature extraction module, a deep feature extraction module and a quality image reconstruction module connected in sequence; The shallow feature extraction module is used to extract the shallow features of the environment image; the deep feature extraction module is used to extract the deep features of the environment image; and the quality image reconstruction module is used to reconstruct the image quality of the environment image.
3. The image quality enhancement method according to claim 2, characterized in that: The deep feature extraction module includes: six RSTB modules connected in series in sequence; and a 3×3 convolutional layer connected to the last RSTB module.
4. The image quality enhancement method according to claim 3, characterized in that: The RSTB module consists of six Swin-Transformer modules connected in series. The last Swin-Transformer module is connected to the Inception model. The feature map is input into the first Swin-Transformer module and connected to the output residual of the Inception model.
5. The image quality enhancement method according to claim 3, characterized in that: The structure of the Swin-Transformer module is as follows: The first LayerNorm layer and the W-MSA module, the feature map P1 is input into the first LayerNorm layer, the feature map P1 is connected with the output residual of the W-MSA module to obtain the feature map P2; The second LayerNorm layer and the first MLP neural network model, the feature map P2 is input into the second LayerNorm layer, the feature map P2 is connected with the output residual of the first MLP neural network model to obtain the feature map P3; The third LayerNorm layer and the SW-MSA module, the feature map P3 is input into the third LayerNorm layer, and the feature map P3 is connected with the output residual of the SW-MSA module to obtain the feature map P4; The fourth LayerNorm layer and the second MLP neural network model, the feature map P4 is input into the fourth LayerNorm layer, and the feature map P4 is connected to the output residual of the second MLP neural network model.
6. The image quality enhancement method according to claim 4, characterized in that: The Inception model includes: The first 1×1 convolutional layer; A second 1×1 convolutional layer, a third 1×1 convolutional layer, a fourth 1×1 convolutional layer, and a two-dimensional average pooling layer connected to the first 1×1 convolutional layer; The first 3×3 convolutional layer connected to the third 1×1 convolutional layer, the second 3×3 convolutional layer connected to the first 3×3 convolutional layer; A 5×5 convolutional layer connected to the fourth 1×1 convolutional layer; The fifth 1×1 convolutional layer connected to the average pooling layer; the sixth 1×1 convolutional layer connected to the second 1×1 convolutional layer, the second 3×3 convolutional layer, the 5×5 convolutional layer, and the fifth 1×1 convolutional layer; GeLU activation function connected to the sixth 1×1 convolutional layer.
7. The image quality enhancement method according to claim 2, characterized in that: The SwinTG network model is trained through samples. The sample construction process is as follows: Use a sliding window to crop an image patch from the noise-free image; Performing geometric transformation on the image block, the transformation methods include: flipping the original image and rotating the image; Randomly select a value from the noise level σ∈[0,75] range, generate Gaussian white noise with the noise level σ, and add the Gaussian white noise to the image block to obtain a noisy image, and use the noisy image as a sample.
8. The image quality enhancement method according to claim 2, characterized in that: The loss function L of the SwinTG network model loss The details are as follows: in, is the denoised image output by the SwinTG network model for the i-th noise image sample, x i is the noise-free image corresponding to the i-th noise image sample.
9. An image quality enhancement system, characterized in that: The system comprises: An image acquisition unit, a processing unit connected to the image acquisition unit, and an output unit connected to the processing unit, wherein the processing unit is integrated with the image quality enhancement model as claimed in claims 1 to 8; Among them, the image acquisition unit is used to acquire environmental images, and the acquired environmental images are input into the image quality enhancement model in the processing unit. The image quality enhancement model enhances the image quality of the input environmental images, and outputs the environmental images with enhanced image quality through the output unit.
10. A vehicle, characterized in that: The vehicle is integrated with the image quality enhancement system as claimed in claim 9.