A method of transmittance window based de-scattering imaging

Abstract

The application discloses a kind of based on the de-scattering imaging method of transparent window, comprising the following steps: S1, the light signal in the field of view of the transparent window is collected by the sensor arranged in the side of the transparent window;S2, the clear image of the field of view of the transparent window is obtained by removing the influence of scattered light from the light signal by the trained deep neural network.The present application can obtain a clear image in a low-visibility environment caused by scattering.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of de-scattering imaging, and in particular to a de-scattering imaging method based on a transparent window. BACKGROUND

[0002] Visibility refers to the maximum distance at which an object can be distinguished by the human eye in the horizontal direction under given meteorological conditions. Visibility is affected by atmospheric transparency, light intensity, weather conditions, and other factors. Generally, when the visibility is less than 1000 meters, it is considered to be in a low-visibility environment.

[0003] Infrared imaging technology uses infrared cameras to capture thermal radiation signals for target detection in low-visibility environments. However, infrared imaging technology can be affected by environmental temperature and weather conditions, resulting in limited imaging effects. In an environment with high temperature, the temperature difference between the background and the target can be small, resulting in poor infrared imaging results. Similarly, if the temperature of the target and the background is close, infrared imaging technology can have difficulty distinguishing between them. Moreover, weather conditions such as rain, snow, and fog can affect infrared imaging results. Water molecules and ice crystals can absorb and scatter infrared radiation, reducing the effective distance of infrared imaging under these conditions. SUMMARY

[0004] The present application aims to at least partially eliminate the problem of visual field blurring caused by scattering and provide a de-scattering imaging method based on a transparent window.

[0005] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0006] A de-scattering imaging method based on a transparent window, comprising the following steps: S1, collecting light signals in the field of view of the transparent window through a sensor arranged at the edge of the transparent window; S2, obtaining a clear image of the field of view of the transparent window without the influence of scattered light from the light signals through a trained deep neural network.

[0007] In some embodiments of the present application, the light signals collected by the sensor include direct light signals and scattered light signals, and the scattered light signals include scattered light in the surrounding environment, the effect of the transparent window on the scattered light in the surrounding environment, and the scattered light signal caused by the accumulation effect of the edge of the transparent window on the scattered light.

[0008] In some embodiments of the present application, the sensor includes one or a combination of visible light sensors and infrared sensors, and can optionally further include one or a combination of laser radar, millimeter wave radar, and ultrasonic sensor.

[0009] In some embodiments of the present application, step S1 includes denoising the collected light signal, specifically including: dividing the input light signal into a plurality of pixel blocks; calculating the local variance of light intensity for each pixel block; calculating the size of the denoising window of each pixel block according to the local variance; adaptively denoising the light signal of each pixel block according to the corresponding size of the denoising window; and recombining the processed pixel blocks into the denoised light signal.

[0010] In some embodiments of the present application, step S1 includes histogram equalization processing of the collected light signal, specifically including: dividing the input light signal into a plurality of pixel blocks; calculating the local contrast of each pixel block, and determining the contrast enhancement coefficient according to the local contrast; adaptively enhancing the contrast of each pixel block according to the corresponding contrast enhancement coefficient; and recombining the processed pixel blocks into the contrast-enhanced light signal.

[0011] In some embodiments of the present application, step S1 includes deep learning fusion of the light signals collected by the two or more sensors, specifically including: using a deep neural network to extract features of different types of light signals and converting them into a common feature space; applying a feature fusion strategy in the common feature space to fuse the features of different types of light signals; mapping the fused features back to the original signal spatial resolution; and converting the fused feature mapping back to the light signal to obtain the fusion output.

[0012] In some embodiments of the present application, in step S2, a transmittance estimation image corresponding to the type of light signal is generated from the collected light signal, the transmittance estimation image is input into a deep neural network based on U-Net, and an image with de-scattering effect is output from the deep neural network; wherein the transmittance estimation image describes the transmittance value of each pixel point in the image, indicating the degree of attenuation of light after passing through the scattering medium;

[0013] Preferably, the generation of the transmittance estimation image of the infrared light signal includes:

[0014] Obtaining the light intensity of each pixel point of the infrared image;

[0015] Determining the reference light intensity;

[0016] Calculating the ratio of the light intensity of the pixel point in the infrared image to the reference light intensity;

[0017] According to the ratio, obtaining the transmittance estimation image of the infrared light signal;

[0018] Preferably, the generation of the transmittance estimation image of the visible light signal includes:

[0019] Obtaining the light intensity of each pixel point of the visible light RGB image;

[0020] determining a reference light intensity;

[0021] calculating a ratio of the light intensity of a pixel in the visible light RGB image to the reference light intensity;

[0022] obtaining a transmittance estimation image of the visible light signal according to the ratio;

[0023] Preferably, the generating of the transmittance estimation image of the lidar signal comprises:

[0024] obtaining lidar point cloud data;

[0025] converting the point cloud data into a two-dimensional depth image;

[0026] determining a reference depth;

[0027] calculating a ratio of the depth of a pixel in the two-dimensional depth image to the reference depth;

[0028] obtaining a transmittance estimation image of the lidar signal according to the ratio;

[0029] More preferably, the method of converting the point cloud data into a two-dimensional depth image specifically comprises:

[0030] Step one: block processing

[0031] dividing the entire point cloud data into blocks of equal size, each block containing a predetermined number of points;

[0032] Step two: introducing an illumination model

[0033] When projecting the point cloud onto a 2D plane, the depth information and normal information of each point are retained simultaneously, obtaining a 2D image containing both depth information and illumination information;

[0034] Step three: Gaussian mixture model processing

[0035] For each block in the point cloud, use the Gaussian mixture model to model the depth of the points in each block, obtain multiple Gaussian distributions, and then select the mean of the Gaussian distribution with the largest distribution as the depth value of this block;

[0036] Step four: graph-based data fusion

[0037] Use the graph cut algorithm to post-process the depth image, so that the depth values of adjacent pixels are smoothed.

[0038] In some embodiments of the present application, in step S2, the deep neural network has one or more of the following features: introducing an attention module in the convolutional layers of the encoder and decoder of the U-Net; introducing a hybrid convolutional structure in the network; using a feature pyramid network (FPN) structure for multi-scale feature fusion in the decoder, fusing feature maps of different scales together through a top-down path and lateral connections; using an adaptive loss function to dynamically adjust the weights according to the characteristics of the input image; and using an unsupervised method for pre-training.

[0039] In some embodiments of the present application, in step S2, the method further comprises post-processing the image output by the deep neural network, the post-processing including removing residual scattering and enhancing image edges; the removing residual scattering includes calculating a weighted average value of each pixel point in the image, the weight being proportional to the similarity between adjacent pixels, and performing non-local mean (NLM) filtering according to the weighted average value; and the enhancing image edges includes performing Gaussian filtering on the image, calculating the gradient amplitude and direction of the image using a Sobel operator, performing non-maximum suppression on the gradient amplitude, and connecting the edges using a double-threshold method to obtain complete edge information.

[0040] The present application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method described above.

[0041] The present application has the following advantages:

[0042] The de-scattering imaging method based on a transparent window provided by the present application collects light signals in the field of view of the transparent window through sensors arranged on the side of the transparent window, utilizes the fact that the transparent window itself is equivalent to a large lens and has a large field of view, and obtains a clear image of the field of view of the transparent window that is free of the influence of scattered light from the collected light signals (including light signals directly obtained from the surrounding environment and light signals obtained after the light from the surrounding environment passes through the transparent window), thereby obtaining a clear image in a low-visibility environment and eliminating the decline in image quality caused by atmospheric scattering and the like. For example, when the present application is applied to a vehicle, the safety of driving in a low-visibility environment can be improved. However, the present application can be widely applied to various scenarios that use a transparent window and require improved visibility, and is not limited to vehicles.

[0043] In addition, in some embodiments, the present application also has the following advantages:

[0044] By introducing an attention mechanism, a hybrid convolutional structure, multi-scale feature fusion, an adaptive loss function, and unsupervised pre-training into a conventional U-Net, the de-scattering performance can be better.

[0045] The data obtained by combining various sensors is used to remove scattering imaging on the transparent window, and in the process of removing scattering, the special situation under low visibility environment is optimized, so that the effect of removing scattering is better and the adaptability is stronger.

[0046] In the post-processing process, the image quality can be further improved by using a non-local mean filter and a Canny edge detection algorithm.

[0047] Other beneficial effects in the embodiments of the present application will be further described below. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is a step flow chart of the transparent window-based scattering imaging method of the embodiments of the present application;

[0049] Figure 2a is a sensor layout strategy of uniform layout in the embodiments of the present application;

[0050] Figure 2b is a sensor layout strategy of non-uniform layout in the embodiments of the present application;

[0051] Figure 2c is a sensor layout strategy of ring layout in the embodiments of the present application;

[0052] The reference signs are as follows:

[0053] 1 is a transparent window, and 2 is a sensor. DETAILED DESCRIPTION

[0054] The present application will be further described below by comparing with the drawings and combining with the preferred embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0055] It should be noted that the left, right, up, down, top, bottom and other orientation terms in the present embodiments are only relative concepts or are referenced to the normal use state of the product, and should not be considered as limiting.

[0056] The following embodiments of the present application provide a transparent window-based de-scattering imaging method, comprising the following steps: S1, collecting light signals in the field of view of the transparent window through a sensor arranged at the edge of the transparent window; and S2, obtaining a clear image of the field of view of the transparent window free from the influence of scattered light from the light signals through a trained deep neural network. The transparent window-based de-scattering imaging method of the embodiments of the present application collects light signals in the field of view of the transparent window through a sensor arranged at the edge of the transparent window, uses the transparent window itself as a large lens, has a large field of view, and obtains a clear image of the field of view of the transparent window free from the influence of scattered light from the collected light signals (including light signals directly obtained from the surrounding environment and light signals obtained after the light from the surrounding environment passes through the transparent window), so that a clear image can be obtained in a low-visibility environment, and the image quality degradation caused by atmospheric scattering and the like can be eliminated. For example, when the present application is applied to a vehicle, the safety of driving in a low-visibility environment can be improved. However, the present application can be widely applied to various scenes using a transparent window and requiring improved visibility, and is not limited to vehicles.

[0057] In a preferred embodiment, the step S1 includes denoising processing of the collected light signals, specifically including: dividing the input light signals into a plurality of pixel blocks; calculating the local variance of light intensity for each pixel block; calculating the size of the denoising window of each pixel block according to the local variance; adaptively performing denoising processing on the light signals of each pixel block according to the corresponding size of the denoising window; and recombining the processed pixel blocks into the denoised light signals.

[0058] In a preferred embodiment, the step S1 includes histogram equalization processing of the collected light signals, specifically including: dividing the input light signals into a plurality of pixel blocks; calculating the local contrast of each pixel block, and determining the contrast enhancement coefficient according to the local contrast; adaptively performing contrast enhancement processing on each pixel block according to the corresponding contrast enhancement coefficient; and recombining the processed pixel blocks into the contrast-enhanced light signals.

[0059] In a preferred embodiment, the light signals collected by the sensor include direct light signals and scattered light signals, and the scattered light signals include scattered light in the surrounding environment, the effect of the transparent window on the scattered light in the surrounding environment, and the scattered light signal caused by the accumulation effect of the edge of the transparent window on the scattered light.

[0060] In a preferred embodiment, the sensor includes one or a combination of a visible light sensor and an infrared sensor, and optionally further includes one or a combination of a laser radar, a millimeter wave radar, and an ultrasonic sensor. The sensor has the following characteristics:

[0061] (1) Millimeter wave radar: can penetrate fog, rain, snow and other bad weather conditions, detect distant objects.

[0062] (2) Ultrasonic sensor: can measure the distance to the surrounding objects, commonly used in parking assistance and other scenarios.

[0063] (3) Laser radar (LiDAR): measures distance by emitting laser pulses and receiving returned reflection signals, achieving high-precision three-dimensional mapping.

[0064] Preferably, embodiments of the present application can combine multiple sensors (such as optical sensors, infrared sensors, etc.), realize multi-source data fusion, and improve the accuracy and stability of de-scattering imaging.

[0065] When performing data fusion, the multi-source data of each party needs to meet the following conditions:

[0066] a. Data timeliness: the data collected by each sensor should have similar time to ensure the accuracy of data fusion.

[0067] b. Data consistency: the data format and unit output by each sensor should be consistent to facilitate data processing and integration.

[0068] c. Data accuracy: the data collected by each sensor should be as accurate as possible to avoid introducing errors.

[0069] d. Data integrity: each sensor should cover as much information as possible in the perception range to avoid data loss or duplication.

[0070] In specific embodiments, the above conditions do not need to be met.

[0071] For the optical signals collected in step S1, embodiments of the present application also propose signal preprocessing and de-scattering algorithms.

[0072] Signal preprocessing

[0073] In low visibility environments, scattered light signals may be affected by various factors, such as noise, changes in lighting, etc. Noise mainly comes from the sensor itself, electronic devices and the environment. Noise can cause the intensity of scattered light signals to be unstable, so that the useful information in the signal is hidden. Noise can cause signal fluctuations, distortion, reduce the signal-to-noise ratio of the signal, and affect the reliability and accuracy of the signal. Changes in lighting conditions can cause changes in the intensity and distribution of scattered light signals. For example, at night or on cloudy days, the lighting intensity is low, while on sunny days, the lighting intensity is high. Lighting changes will affect the visibility, contrast and color of scattered light signals, making it more difficult to identify and process the signals. Therefore, before de-scattering processing, the original light signals captured by the sensor need to be preprocessed.

[0074] Signal preprocessing includes the following:

[0075] 1. Denoising: Use denoising algorithms such as bilateral filtering, non-local mean filtering or deep learning methods to denoise the captured signal, reduce the impact of noise on image quality.

[0076] The embodiment of the present application proposes an adaptive window scattering denoising algorithm (AWS-DA), which is used in the scene of transparent window side scattering imaging. The input is the scattering light signal captured by the sensor, and the output is the denoised scattering light signal. The denoising algorithm steps are as follows:

[0077] a) Divide the input scattering light signal into several small blocks (such as 8x8 pixels).

[0078] b) Calculate the local variance of the scattering light intensity for each small block.

[0079] c) Calculate the size of the denoising window according to the local variance. Higher local variance means larger structural changes and scattering information, so a smaller window is needed to preserve details; lower local variance means smaller structural changes, so a larger window can be used for smoothing.

[0080] d) Use adaptive window to denoise the scattering light signal.

[0081] e) Reassemble the processed small blocks into the denoised scattering light signal.

[0082] The denoising algorithm proposed in the embodiment of the present application can adaptively adjust the size of the denoising window according to the local characteristics of the image, so as to effectively reduce noise while preserving image details.

[0083] Improvements for this special scenario: Since the characteristics of the scattering light signal may differ greatly in different areas, AWS-DA can adjust the window size according to these characteristics, thereby preserving useful scattering light information while denoising.

[0084] 2. Histogram equalization: Perform histogram equalization on the denoised scattering light signal to enhance contrast and brightness, and improve the distinguishability of the scattering light signal in the image.

[0085] The embodiment of the present application proposes an adaptive local contrast enhancement algorithm (ALCEA). In the scenario of transparent window side scatter imaging, the input is the denoised scatter light signal, and the output is the contrast-enhanced scatter light signal. The histogram equalization steps are as follows:

[0086] a) The input scatter light signal is divided into several small blocks (such as 8x8 pixels).

[0087] b) Calculate the local contrast of each small block, and select the adaptive contrast enhancement coefficient according to the local contrast.

[0088] c) Apply the adaptive contrast enhancement algorithm to each small block to improve the distinguishability of the scatter light signal.

[0089] d) Reassemble the processed small blocks into the contrast-enhanced scatter light signal.

[0090] The histogram equalization method in the embodiment of the present application can adaptively adjust the histogram equalization parameters according to the distribution of the scatter light intensity in the local area, to improve the contrast and brightness of the image.

[0091] Improvements for this special scenario: In low-visibility environments, the distribution of scatter light intensity in the image may not be uniform. LRSI-AHE can adaptively adjust according to the scatter light intensity characteristics of the local area, so that the distinguishability of the scatter light signal in the entire image is improved.

[0092] 3. Sensor data fusion: Fuse signals captured by different types of sensors (such as optical, infrared, laser radar, etc.) to obtain more rich information, and improve the accuracy and robustness of signal processing.

[0093] The embodiment of the present application proposes a deep learning fusion algorithm (DLFA). In the scenario of transparent window side scatter imaging, the input is the scatter light signal captured by different types of sensors (such as optical, infrared, laser radar, etc.), and the output is the fused scatter light signal. The steps of sensor data fusion are as follows:

[0094] a) Preprocess the input different types of scatter light signals for deep learning fusion.

[0095] b) Use a deep neural network (such as a convolutional neural network) to extract features from the preprocessed scatter light signal. The deep neural network can automatically learn the features of different types of sensor signals and convert them into a common feature space.

[0096] c) In the common feature space, apply feature fusion strategies such as weighted average, maximum activation, etc. to fuse the features of different sensors. These strategies can be dynamically adjusted according to the scene and the performance of the sensors to achieve the best fusion effect.

[0097] d) Use deconvolution or other up-sampling operations to restore the fused feature map to the spatial resolution of the original signal.

[0098] e) Convert the fused feature map back to the scattered light signal to get the final fusion output.

[0099] The sensor data fusion method in the embodiment of the present application uses deep learning networks (such as convolutional neural networks, CNN) to fuse data from different sensors at multiple scales to extract more rich feature information.

[0100] Improvements for this special scenario: In low visibility environments, signals captured by different types of sensors have different characteristics. DL-MSFS can adaptively fuse these information at multiple scales to improve the accuracy and robustness of signal processing, thus achieving more reliable imaging results in complex low visibility environments.

[0101] De-scattering algorithm

[0102] The purpose of the de-scattering algorithm is to separate the scattered light signal and the direct light signal from the pre-processed signal, so as to obtain a clear and accurate image in a low visibility environment.

[0103] The following analyzes the advantages of the embodiment of the present application over the traditional technology from the perspective of system difference.

[0104] The embodiment of the present application is based on the imaging system in low visibility environment based on transparent window side edge scattering imaging, which utilizes the scattering characteristics of the transparent window side edge (the scattering characteristics refer to the scattering characteristics of the transparent window side edge, which refers to the interaction between the transparent window (such as the front windshield, side window, etc.) and the scattered light in the surrounding environment in low visibility environment. When light passes through particles such as fog, smoke, dust, etc. in the atmosphere, the light will scatter. The scattering characteristics of the transparent window side edge utilize the accumulation effect of these scattered light at the window edge) by installing sensors on the side edge to collect scattered light signals. The embodiment of the present application can utilize the existing vehicle window or other transparent window as an imaging device without additional complex optical system.

[0105] In contrast, existing imaging systems in low-visibility environments are usually designed for specific optical imaging systems (such as lidar, etc.), often requiring additional optical elements or adjusting the parameters of the imaging device to adapt to different scattering environments. For example, according to the characteristics of the scattering environment (such as fog, smoke, dust, etc.), adjust or add specific optical filters or polarizers to reduce the impact of scattered light on imaging quality. According to the light conditions and scattering degree of the environment, adjust the parameters of the imaging device (such as camera, lidar, etc.), such as exposure time, sensitivity, contrast, etc., to adapt to different scattering environments. Adjust the working wavelength of the imaging device: according to the characteristics of the scattering environment, select different imaging device working wavelength.

[0106] In step S2 of the embodiment of the present application, a scattering imaging removal algorithm based on a deep neural network (DNN) neural network model is proposed, which can be used in low-visibility environments. The algorithm mainly includes the following contents:

[0107] Scattering imaging removal using DNN: In the process of scattering imaging removal, the transmission estimation image is a key factor. Transmission represents the degree of light attenuation when light passes through a scattering medium (such as fog, smoke, snow, etc.). The transmission estimation image describes the transmission value of each pixel point, usually represented in the form of a grayscale image, where brighter pixels represent higher transmission and darker pixels represent lower transmission.

[0108] The transmission image is not directly collected, but is estimated according to the signals collected by different sensors. The specific method and description are as follows:

[0109] 1. Infrared sensor transmission estimation:

[0110] Step 1: Read the infrared image I IR;

[0111] Step 2: Calculate the local atmospheric light A IR. In a specific embodiment, the brightest part of the image can be used as an approximation of the atmospheric light: A IR = max(I IR);

[0112] Step 3: Calculate the ratio of I IR and A IR: R IR = I IR / A IR;

[0113] Step 4: Threshold R IR to obtain the transmission image T IR: T IR = 1-R IR.

[0114] 2. Optical sensor transmission estimation:

[0115] Step 1: Read the optical image I RGB;

[0116] Step 2: Convert the optical image from RGB color space to HSV color space, resulting in H, S, and V channels;

[0117] Step 3: Calculate the local atmospheric light A_V. In one specific embodiment, the brightest part of the image can be used as an approximation of the atmospheric light: A_V = max(V);

[0118] Step 4: Calculate the ratio of V to A_V: Calculate the ratio of V to A_V;

[0119] Step 5: Threshold R_V to obtain the transmittance image T_V: T_V = 1 - R_V.

[0120] 3. LIDAR transmittance estimation:

[0121] Step 1: Read the LIDAR point cloud data P;

[0122] Step 2: Convert the point cloud data to a two-dimensional depth image I_depth, where Z represents the distance values in the point cloud;

[0123] As a preferred embodiment, Step 2 proposes a method to convert point cloud data into a two-dimensional depth image, which is different from the existing method of directly mapping projection. This method includes the following steps:

[0124] Step 21: Block processing

[0125] The entire point cloud data is first divided into blocks of equal size, each block containing a certain number of points. This can help reduce computational complexity while also better preserving local information.

[0126] Step 22: Introduce the lighting model

[0127] When projecting the point cloud onto a 2D plane, we not only retain the depth information of each point, but also consider the normal information of each point. Specifically, the angle between the normal direction of each point and the direction of the light source is added to the depth image as a kind of "lighting" information (which can use the Lambert lighting model). In this way, a 2D image containing both depth information and lighting information can be obtained.

[0128] Step 23: Gaussian mixture model processing

[0129] For each block in the point cloud, we do not directly calculate the nearest distance of all points to the observer, but use a Gaussian mixture model to model these distances, resulting in multiple Gaussian distributions. Then, the mean value of the Gaussian distribution with the largest distribution is selected as the depth value of this block. In this way, we can better handle noise and outliers in the point cloud.

[0130] Step 24: graph-based data fusion

[0131] Finally, in order to ensure the continuity in the depth image, a graph cut algorithm is used to post-process the depth image, so that the depth values of adjacent pixels are as close as possible. In this way, not only can we get a smooth depth image, but also can better preserve the boundary information of the object.

[0132] Step 3: Calculate the local atmospheric light A_depth, in a specific embodiment, the brightest part of the image can be used as an approximation of the atmospheric light: A_depth = max(I_depth);

[0133] Step 4: Calculate the ratio of I_depth and A_depth: R_depth = I_depth / A_depth;

[0134] Step 5: Threshold processing of R_depth to obtain the transmittance image T_depth: T_depth = 1 - R_depth.

[0135] The embodiment of the application sends the pre-processed image into the DNN, and the input of the DNN is the transmittance estimation image corresponding to the pre-processed image. The DNN in the embodiment of the application uses a U-Net architecture to build, which includes feature extraction of the input and deconvolution operation of the output. Compared with the traditional U-Net, the embodiment of the application optimizes the structure and parameter setting of the network to better adapt to the imaging system in the low-visibility environment of the transparent window side scattering imaging. Specifically, the embodiment of the application uses some special convolution kernels and normalization methods to increase the stability and robustness of the network.

[0136] The traditional U-Net network structure and parameter description are as follows:

[0137] U-Net is a typical Encoder-Decoder structure, including compression path (downsampling) and expansion path (upsampling), and the basic structure of the U-Net network is as follows:

[0138] (1) Encoder (downsampling): including 4 convolution layers, each convolution layer is followed by a ReLU activation function and a 2x2 max pooling layer. The convolution kernel size is 3x3, and the step is 1.

[0139] (2) Decoder (upsampling): including 4 upsampling layers, each upsampling layer is followed by a 2x2 deconvolution layer, a convolution layer and a ReLU activation function. The convolution kernel size is 3x3, and the step is 1.

[0140] (3) Loss function: using pixel-level cross-entropy loss.

[0141] In the imaging system under the low-visibility environment of the transparent window side scattering imaging, the embodiment of the application proposes a neural network structure based on U-Net. The neural network structure of the embodiment of the application introduces attention mechanism, hybrid convolution structure, multi-scale feature fusion, adaptive loss function and unsupervised pre-training on the basis of the traditional U-Net, so as to realize better de-scattering performance. The specific content is described as follows:

[0142] 1. First, the attention module is introduced in the convolution layer of the encoder and decoder of the U-Net, so that the network can automatically learn and pay attention to the important areas in the input image, thereby helping the network to pay more attention to the key details in the low-visibility environment when processing the scattered light signal.

[0143] 2. Secondly, the embodiment of the application introduces a hybrid convolution structure, such as an Inception module, in the network. The structure can help the network to capture image details at different scales and improve the ability to process scattered light signals in a low-visibility environment.

[0144] 3. In the decoder part, the embodiment of the application uses a feature pyramid network (FPN) structure for multi-scale feature fusion. Through the top-down path and horizontal connection, the feature maps of different scales are fused together, thereby helping the network to better capture scattered light information at different scales.

[0145] 4. In order to make the model pay more attention to the key details in the low-visibility environment, the embodiment of the application uses an adaptive loss function. The loss function will dynamically adjust the weight according to the characteristics of the input image, so that the network can adaptively optimize the prediction of important areas.

[0146] 5. Since the imaging system under the low-visibility environment may face the problem of training data scarcity, the embodiment of the application uses an unsupervised method (such as an autoencoder or a GAN) for pre-training before training the deep network, which helps the network to learn more rich feature representation, thereby improving the de-scattering performance.

[0147] In summary, the neural network structure proposed by the embodiment of the application can better adapt to the imaging system under the low-visibility environment of the transparent window side scattering imaging by introducing the attention mechanism, the hybrid convolution structure, the multi-scale feature fusion, the adaptive loss function and the unsupervised pre-training. The above contents jointly make the neural network achieve better results in the de-scattering performance, thereby meeting the needs in specific scenarios.

[0148] In a preferred embodiment, the following contents are also included:

[0149] 1. Attention module introduction: the calculation formula of the attention module is as follows:

[0150] S = F(X, W) * sigma(G(X, V))

[0151] where S is the output feature map, X is the input feature map, F and G are convolution operations respectively, W and V are convolution kernels, and sigma is an activation function (such as a sigmoid function).

[0152] 2. Mixed convolution structure: taking the Inception module as an example, the calculation formula is as follows:

[0153] I_out = concat(conv_1x1(I_in), conv_3x3(I_in), conv_5x5(I_in), max_pool(I_in))

[0154] where I_out is the output feature map, I_in is the input feature map, concat is the concatenation operation, conv_1x1, conv_3x3 and conv_5x5 are convolution operations of different sizes, and max_pool is the pooling operation.

[0155] 3. Multi-scale feature fusion: using the feature pyramid network (FPN) structure, the calculation formula is as follows:

[0156] P_i = upsample(P_(i+1)) + conv_1x1(C_i)

[0157] where P_i and C_i are pyramid feature maps and original feature maps respectively, i is the level, and upsample is the up-sampling operation.

[0158] 4. Adaptive loss function: taking the weighted cross-entropy loss function as an example, the calculation formula is as follows:

[0159] L = -∑(w_p*y*log(sigma(x)) + w_n*(1-y)*log(1-sigma(x)))

[0160] where L is the loss value, y is the true label, x is the predicted value, sigma is the sigmoid function, and w_p and w_n are the weights of positive and negative classes respectively.

[0161] 5. Unsupervised pre-training: taking the autoencoder as an example, the loss function can be expressed as:

[0162] L = ||X-decode(encode(X))||2

[0163] where X is the input, encode and decode are the encoder and decoder respectively.

[0164] The de-scattering imaging method proposed in the embodiment of the application also includes post-processing.

[0165] The output of the DNN (which is an image that removes the scattering effect, preserves the target object information, reduces the influence of scattered light on the image, and thus improves the imaging quality) is post-processed by the embodiments of the present application, including removing residual scattering and enhancing edges. To remove residual scattering, the embodiments of the present application use a non-local mean (NOn-Local Mean, NLM) filter, which can effectively reduce noise and preserve details. To enhance edges, the embodiments of the present application use a Canny edge detection algorithm.

[0166] In the post-processing process, the embodiments of the present application use a non-local mean (Non-Local Mean, NLM) filter and a Canny edge detection algorithm. To adapt to the scenario of transparent window side scattering imaging, the embodiments of the present application improve these methods according to specific needs.

[0167] 1. Improved NLM filter

[0168] The NLM filter is a filtering method based on the similarity of image regions, which can effectively reduce noise while preserving image details. In the scenario of transparent window side scattering imaging, for the image output by the DNN, the embodiments of the present application first calculate the weighted average value of each pixel point in the image. The weight here is proportional to the similarity between adjacent pixels. Through this method, the embodiments of the present application can preserve the detailed information in the image while removing the residual scattering effect.

[0169] 2. Improved Canny edge detection algorithm

[0170] To adapt to the scenario of transparent window side scattering imaging, the embodiments of the present application optimize the Canny algorithm. First, the embodiments of the present application perform Gaussian filtering on the image output by the DNN to smooth the image and eliminate noise. Next, the embodiments of the present application use the Sobel operator to calculate the gradient magnitude and direction of the image. Then, the embodiments of the present application perform non-maximum suppression on the gradient magnitude to preserve edges and reduce the occurrence of false edges. Finally, the embodiments of the present application use a double-threshold method to connect the edges, thereby obtaining complete edge information.

[0171] Through these two post-processing steps, the embodiments of the present application can further optimize the image quality output by the DNN, remove residual scattering, and enhance edges. These improvements make the method of the embodiments of the present application more suitable for processing imaging systems in low-visibility environments under the scenario of transparent window side scattering imaging.

[0172] Compared with the prior art, the embodiment of the present application has higher despeckling effect and better adaptability. First, the embodiment of the present application designs a specific DNN structure and parameter setting for the imaging system in the low-visibility environment of the transparent window side scattering imaging. Second, the embodiment of the present application uses some special convolution kernels and normalization methods to increase the stability and robustness of the network. Finally, the embodiment of the present application uses NLM filter and Canny edge detection algorithm for post-processing of the output to further enhance the despeckling effect and edge information.

[0173] Using the transparent window-based despeckling imaging method of the embodiment of the present application, the embodiment of the present application can be used for the assistance of vehicle driving in low-visibility environment, and the clear image of the vehicle window field of view image obtained through despeckling processing facilitates the driver or automatic driving system to more accurately identify the objects, obstacles and pedestrians on the road.

[0174] In order to implement the method of the embodiment of the present application, the application system includes the following modules:

[0175] I. Sensor module

[0176] A plurality of sensors are installed on the side of the transparent window for collecting scattered light signals. The sensor types include photoelectric sensors, infrared sensors, etc., and appropriate sensor combinations can be selected according to different light conditions and scattering degrees, as shown in Table 1.

[0177] Table 1: Sensor combinations under different light conditions and scattering degrees

[0178]

[0179]

[0180] The overall scheme of the sensor module should include the consideration and configuration of multiple sensor types to achieve efficient scattered light signal capture. The following is the overall scheme and specific examples of the sensor module:

[0181] Sensor type selection: When designing the sensor module, different types of sensors should be considered to capture scattered light signals. For example, optical sensors (such as (Complementary Metal Oxide Semiconductor, CMOS) complementary metal oxide semiconductor, (charge coupled device, CCD) charge coupled device camera) can be selected to capture visible light scattering signals, infrared sensors (such as thermal imager) to capture infrared scattering signals, and laser radar sensors to capture laser scattering signals. Through the combination of multiple sensors, comprehensive capture of scattered light signals under various environmental conditions can be achieved.

[0182] Sensor layout strategy: Install sensors 2 on the side of the transparent window 1 (such as the front windshield, side window, etc.) to effectively capture the scattered light signal. Uniform layout (as shown in Figure 2a , non-uniform layout (as shown in Figure 2b , ring layout (as shown in Figure 2c or other suitable layout can be used to achieve the best capture effect of the scattered light signal. At the same time, considering the shape and size of the window, the sensors can be installed on the four sides or part of the edge of the window to improve the ability of the sensors to receive scattered light signals.

[0183] II. Data processing module

[0184] In the data processing module, signal preprocessing and despeckling algorithm are the key parts, and the specific method is as described above.

[0185] Compared with the above technology, the driving assistance system based on the side scattering imaging of the transparent window in the low visibility environment has the following advantages:

[0186] 1. Use existing car windows: The embodiment of the present application uses the side of the car window and other transparent windows for scattering imaging, without the need for additional complex equipment, reducing the burden on the system and reducing costs.

[0187] 2. Ultra-wide field of view: The car window itself is an imaging device with an ultra-wide field of view (the car window itself is equivalent to a large lens, which can have an ultra-wide field of view), which can provide a wider field of view than traditional imaging devices such as cameras and laser radars, improving the safety of the driver.

[0188] 3. Multi-sensor fusion: The embodiment of the present application can combine multiple sensors (such as optical sensors, infrared sensors, etc.) to achieve multi-source data fusion and improve the accuracy and stability of despeckling imaging.

[0189] 4. Real-time processing, combined with specially designed despeckling algorithms, such as deep learning-based image processing technology, can process the collected scattered light signals in real time and generate clear despeckling images.

[0190] The transparent window-based despeckling imaging method provided by the embodiment of the present application utilizes deep learning technology, introduces attention mechanism, hybrid convolution structure, multi-scale feature fusion, adaptive loss function and unsupervised pre-training on the basis of the traditional U-Net, so as to realize better despeckling performance. The data obtained by various sensors (such as infrared, optical and laser radar) are combined to remove the scattering imaging on the transparent window. In the despeckling process, the algorithm fully considers the special situation in the low-visibility environment and is optimized pertinently, so that the despeckling effect is better and the adaptability is stronger. At the same time, the non-local mean filter and the Canny edge detection algorithm are used in the post-processing part, so as to further improve the image quality.

[0191] The despeckling imaging method in the low-visibility environment provided by the embodiment of the present application has wide application prospects, such as driving assistance system, security monitoring, unmanned aerial vehicle and the like. Compared with the prior art, the embodiment of the present application has the advantages of low cost, wide field of view, better despeckling effect and stronger adaptability. By combining the information of various sensors and the deep learning technology, the embodiment of the present application can realize efficient scattering imaging removal and provide clear visual information for different scenes.

[0192] The above is a further detailed description of the present application in combination with a specific preferred embodiment, and the specific implementation of the present application cannot be limited to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, a number of equivalent substitutions or obvious modifications can be made, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present application.

Claims

1. A descattering imaging method based on a transparent window, characterized in that, The method comprises the following steps: S1, collecting light signals in the field of view of the transparent window through sensors arranged on the edges of the transparent window; the light signals collected by the sensors include direct light signals and scattered light signals, the scattered light signals including scattered light in the surrounding environment, the effect of the transparent window on the scattered light in the surrounding environment, and the accumulated effect of the edges of the transparent window on the scattered light; the light signals collected by the two types of sensors are fused by deep learning to obtain a fusion output; S2, obtaining a clear image of the field of view of the transparent window by removing the effect of scattered light from the light signals through a trained deep neural network, thereby achieving clear images in low-visibility weather conditions; wherein a transmittance estimation image corresponding to the type of light signal is generated from the collected light signal, the transmittance estimation image is input into a deep neural network based on U-Net, and an image with a scattering effect is output through the deep neural network; wherein the transmittance estimation image describes the transmittance value of each pixel point in the image, indicating the degree of attenuation of light after passing through a scattering medium.

2. The de-scattering imaging method of claim 1, wherein, The sensors include a combination of two or more of visible light sensors, infrared sensors, laser radars, millimeter wave radars, and ultrasonic sensors.

3. The de-scattering imaging method of claim 2, wherein, Step S1 includes denoising processing of the collected light signal, specifically including: segmenting the input light signal into a plurality of pixel blocks; calculating the local variance of light intensity for each pixel block; calculating the size of the denoising window for each pixel block according to the local variance; adaptively denoising the light signal of each pixel block according to the corresponding size of the denoising window; recombining the processed pixel blocks into a denoised light signal.

4. The de-scattering imaging method of claim 2, wherein, Step S1 includes histogram equalization processing of the collected light signal, specifically including: segmenting the input light signal into a plurality of pixel blocks; calculating the local contrast of each pixel block and determining the contrast enhancement coefficient according to the local contrast; performing adaptive contrast enhancement processing on each pixel block according to the corresponding contrast enhancement coefficient; recombining the processed pixel blocks into a contrast-enhanced light signal.

5. The de-scattering imaging method of claim 2, wherein, In step S1, the deep learning fusion of the light signals collected by the two types of sensors specifically includes: using a deep neural network to extract features of different types of light signals and converting them into a common feature space; applying a feature fusion strategy to fuse the features of different types of light signals in the common feature space; mapping the fused features back to the spatial resolution of the original signal; converting the mapped fused features back to light signals to obtain the fusion output.

6. The de-scattering imaging method of any one of claims 2 to 5, wherein, In step S2, the generation of the transmittance estimation image of the infrared light signal includes: obtaining the light intensity of each pixel point of the infrared image; determining the reference light intensity; calculating the ratio of the light intensity of each pixel point of the infrared image to the reference light intensity; obtaining the transmittance estimation image of the infrared light signal according to the ratio; the generation of the transmittance estimation image of the visible light signal includes: obtaining the light intensity of each pixel point of the visible light RGB image; determining the reference light intensity; calculating the ratio of the light intensity of each pixel point of the visible light RGB image to the reference light intensity; obtaining the transmittance estimation image of the visible light signal according to the ratio; The generation of the transmittance estimation image of the lidar signal comprises: obtaining lidar point cloud data; converting the point cloud data into a two-dimensional depth image; determining a reference depth; calculating the ratio of the depth of each pixel of the two-dimensional depth image to the reference depth; obtaining the transmittance estimation image of the lidar signal according to the ratio.

7. The de-scattering imaging method of claim 6, wherein, The method of converting the point cloud data into a two-dimensional depth image specifically comprises: Step 1: block processing Divide the entire point cloud data into blocks of equal size, each block containing a predetermined number of points; Step 2: introducing a light model When projecting the point cloud onto a 2D plane, the depth information and normal information of each point are retained to obtain a 2D image containing both depth information and light information; Step 3: Gaussian mixture model processing For each block in the point cloud, use the Gaussian mixture model to model the depth of the points in each block to obtain multiple Gaussian distributions, and then select the mean of the Gaussian distribution with the largest distribution as the depth value of this block; Step 4: graph-based data fusion Use the graph cut algorithm to post-process the depth image to smooth the depth value changes of adjacent pixels.

8. The de-scattering imaging method of claim 6, wherein, In step S2, the depth neural network has one or more of the following characteristics: Introducing an attention module in the convolutional layers of the encoder and decoder of U-Net; Introducing a hybrid convolution structure in the network; The decoder uses a feature pyramid network (FPN) structure for multi-scale feature fusion, which fuses feature maps of different scales together through a top-down path and horizontal connection; Using an adaptive loss function to dynamically adjust the weight according to the characteristics of the input image; Using an unsupervised method for pre-training.

9. The de-scattering imaging method of any one of claims 1 to 5, wherein, In step S2, the image output by the depth neural network is also post-processed, which includes removing residual scattering and enhancing image edges; The removal of residual scattering includes calculating the weighted average value of each pixel in the image, with the weight being proportional to the similarity between adjacent pixels, and performing non-local mean (NLM) filtering according to the weighted average value; The enhancement of image edges includes Gaussian filtering of the image, calculating the gradient amplitude and direction of the image using the Sobel operator, non-maximum suppression of the gradient amplitude, and connecting the edges using a double-threshold method to obtain complete edge information.

10. A computer readable storage medium storing a computer program, characterized in that, The computer program, when executed by a processor, implements the method of any one of claims 1 to 9.

Images