A vehicle navigation system to improve driving safety

Abstract

This invention discloses a vehicle navigation system for improving driving safety, comprising a sensor, an optical signal processing device, and a navigation device. The sensor is disposed near a vehicle window, which includes at least the windshield and optionally further includes side windows. The sensor collects optical signals within the window's field of view. The optical signal processing device processes the optical signals to obtain a clear image of the window's field of view after removing the effects of scattered light. The navigation device feeds navigation information to the driver or navigation data to an autonomous driving system based on the information from the image after removing the effects of scattered light. This invention can provide navigation information based on a clear image of the window's field of view in low-visibility environments, thereby improving driving safety in low-visibility conditions.

Description

Technical Field

[0001] This invention relates to the technical field of vehicle navigation systems, and in particular to a vehicle navigation system that improves driving safety. Background Technology

[0002] Visibility refers to the maximum distance at which the human eye can distinguish objects horizontally under given weather conditions. Visibility is affected by various factors, including atmospheric transparency, light intensity, and weather conditions. Generally, a visibility of less than 1000 meters is considered a low-visibility environment. In recent years, driving safety in low-visibility environments (such as fog, rain, and snow) has received widespread attention. This is because in low-visibility environments (such as fog, rain, and snow), microparticles such as water droplets and ice crystals in the air scatter light. Scattering obstructs light propagation, making the driver's field of vision blurry, and in severe cases, completely obscuring the road and obstacles ahead. In such situations, the driver's vision is limited, reaction time is shortened, and traffic accidents are more likely to occur.

[0003] Infrared imaging technology uses infrared cameras to capture thermal radiation signals for target detection in low-visibility environments. However, infrared imaging technology can be affected by ambient temperature and weather conditions, limiting imaging performance. In high-temperature environments, the temperature difference between the background and the target may be small, resulting in poor infrared imaging. Similarly, if the target and background temperatures are similar, infrared imaging technology may have difficulty distinguishing them. Moreover, weather conditions such as rain, snow, and fog can affect infrared imaging performance. Water molecules and ice crystals can absorb and scatter infrared radiation, reducing the effective range of infrared imaging under these conditions.

[0004] Existing technologies are still insufficient to effectively address driving safety issues caused by blurred driver vision in low-visibility environments (such as fog, rain, and snow). Summary of the Invention

[0005] The purpose of this invention is to solve the problem of visual blurring caused by scattering affecting vehicle driving to at least a certain extent, and to provide a vehicle navigation system that improves driving safety.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A vehicle navigation system for improving driving safety includes a sensor, an optical signal processing device, and a navigation device. The sensor is disposed near a vehicle window, which includes at least the windshield and optionally further includes side windows. The sensor collects optical signals within the window's field of view. The optical signal processing device processes the optical signals to obtain a clear image of the window's field of view after removing the effects of scattered light. Based on the information from the image after removing the effects of scattered light, the navigation device feeds navigation information to the driver or navigation data to an autonomous driving system. The navigation information includes image and / or voice information.

[0008] In some embodiments of the present invention, the light signals collected by the sensor include direct light signals and scattered light signals. The scattered light signals include scattered light from the surrounding environment, the effect of the window on the scattered light from the surrounding environment, and the scattered light signals resulting from the accumulation effect of the window edge on the scattered light.

[0009] In some embodiments of the present invention, the navigation device includes a display device for displaying the image; preferably, the display device is a HUD head-up display device, and the image can be displayed on the windshield.

[0010] In some embodiments of the present invention, the navigation device further includes an information fusion processing module, which fuses the real-time traffic information acquired by the vehicle communication system with the image after removing the influence of scattered light to form fused navigation image information and feeds it to the driver.

[0011] In some embodiments of the present invention, the sensor includes one or a combination of a visible light sensor and an infrared sensor, and optionally further includes one or a combination of a lidar, a millimeter-wave radar, and an ultrasonic sensor.

[0012] In some embodiments of the present invention, the optical signal processing device performs denoising processing on the collected optical signal, specifically including: dividing the input optical signal into several pixel blocks; calculating the local variance of the light intensity for each pixel block; calculating the size of the denoising window for each pixel block based on the local variance; adaptively performing denoising processing on the optical signal of each pixel block according to the denoising window of the corresponding size; and recombining the processed pixel blocks into a denoised optical signal; and / or, the optical signal processing device performs histogram equalization processing on the collected optical signal, specifically including: dividing the input optical signal into several pixel blocks; calculating the local contrast for each pixel block and determining the contrast enhancement coefficient based on the local contrast; adaptively performing contrast enhancement processing on each pixel block according to the corresponding contrast enhancement coefficient; and recombining the processed pixel blocks into a contrast-enhanced optical signal.

[0013] In some embodiments of the present invention, the optical signal processing device performs deep learning fusion of optical signals collected by two or more sensors, specifically including: using a deep neural network to extract features from different types of optical signals and converting them into a common feature space; applying a feature fusion strategy to fuse the features of different types of optical signals in the common feature space; restoring the fused feature mapping to the spatial resolution of the original signal; and converting the fused feature mapping back into an optical signal to obtain a fused output.

[0014] In some embodiments of the present invention, the optical signal processing device generates a transmittance estimation image corresponding to the type of optical signal from the collected optical signal, inputs the transmittance estimation image into a trained deep neural network, preferably a U-Net-based deep neural network, and outputs an image with descattering effect after passing through the deep neural network; wherein, the transmittance estimation image describes the transmittance value of each pixel in the image, representing the degree of attenuation of light after passing through the scattering medium;

[0015] Preferably, the generation of the transmittance estimation image of the infrared light signal includes:

[0016] Acquire the light intensity of each pixel in the infrared image;

[0017] Determine the reference light intensity;

[0018] Calculate the ratio of the light intensity of a pixel in an infrared image to the light intensity of a reference pixel;

[0019] Based on the ratio, an estimated image of the transmittance of the infrared light signal is obtained;

[0020] Preferably, the generation of the transmittance estimation image of the visible light signal includes:

[0021] Obtain the light intensity of each pixel in a visible light RGB image;

[0022] Determine the reference light intensity;

[0023] Calculate the ratio of the light intensity of a pixel in a visible light RGB image to the light intensity of a reference pixel;

[0024] Based on the ratio, an estimated image of the transmittance of the visible light signal is obtained;

[0025] Preferably, the generation of the transmittance estimation image of the lidar signal includes:

[0026] Acquire lidar point cloud data;

[0027] Convert point cloud data into a two-dimensional depth image;

[0028] Determine the reference depth;

[0029] Calculate the ratio of the depth of a pixel in a 2D depth image to the depth of a reference image;

[0030] Based on the ratio, a transmittance estimation image of the lidar signal is obtained;

[0031] More preferably, the method for converting point cloud data into a two-dimensional depth image specifically includes:

[0032] Step 1: Partitioning and Block Processing

[0033] The entire point cloud data is divided into equal-sized blocks, each containing a predetermined number of points;

[0034] Step 2: Introduce the lighting model

[0035] When projecting a point cloud onto a 2D plane, the depth information and normal information of each point are preserved simultaneously, resulting in a 2D image that contains both depth and lighting information.

[0036] Step 3: Gaussian Mixture Model Processing

[0037] For each block in the point cloud, a Gaussian mixture model is used to model the depth of the points in each block, resulting in multiple Gaussian distributions. Then, the mean of the Gaussian distribution with the largest distribution is selected as the depth value of this block.

[0038] Step 4: Graph-based data fusion

[0039] The graph cut algorithm is used to post-process the depth image, which smooths the changes in depth values ​​between adjacent pixels.

[0040] In some embodiments of the present invention, the deep neural network has one or more of the following features: an attention module is introduced into the convolutional layers of the encoder and decoder of the U-Net; a hybrid convolutional structure is introduced into the network; the decoder uses a feature pyramid network (FPN) structure to perform multi-scale feature fusion, fusing feature maps of different scales together through top-down paths and lateral connections; an adaptive loss function is used to dynamically adjust weights according to the characteristics of the input image; and unsupervised pre-training is performed.

[0041] In some embodiments of the present invention, the optical signal processing device further performs post-processing on the image output by the deep neural network. The post-processing includes removing residual scattering and enhancing image edges. Removing residual scattering includes: calculating a weighted average value for each pixel in the image, the weight of which is proportional to the similarity between adjacent pixels, and performing nonlocal mean (NLM) filtering based on the weighted average value. Enhancing image edges includes: performing Gaussian filtering on the image; calculating the gradient magnitude and direction of the image using the Sobel operator; performing nonmaximum suppression on the gradient magnitude; and connecting the edges using a double thresholding method to obtain complete edge information.

[0042] The present invention has the following beneficial effects:

[0043] The vehicle navigation system proposed in this invention for improving driving safety includes sensors, an optical signal processing device, and a navigation device. By placing the sensors near the vehicle windows, including the windshield, the sensors collect light signals within the window's field of view. Utilizing the window's characteristic of acting as a large lens with a large field of view, the optical signal processing device processes the light signals (including light signals directly obtained from the surrounding environment and light signals obtained after ambient light passes through the window) to obtain a clear image of the window's field of view, free from the influence of scattered light. This enables the acquisition of clear images in low-visibility environments. The navigation device then sends navigation information (clear images and / or voice information) to the driver or feeds navigation data to the autonomous driving system, thereby providing the vehicle with navigation information based on clear images of the window's field of view and improving driving safety in low-visibility environments.

[0044] This invention provides navigation information by acquiring clear images of the vehicle window's field of view, which can reduce the cost of the entire vehicle navigation system. By placing the sensor on the side of the vehicle window, the entire field of view of the window can be captured even in low-visibility environments, thus realizing a low-cost vehicle navigation system with a large field of view in low-visibility environments.

[0045] In addition, some embodiments also have the following beneficial effects:

[0046] By introducing attention mechanisms, hybrid convolutional structures, multi-scale feature fusion, adaptive loss functions, and unsupervised pre-training on the basis of traditional U-Net, better descattering performance can be achieved, resulting in clearer images and more accurate navigation information.

[0047] By combining data from multiple sensors, scattered images on the car window are removed. During the descattering process, optimizations are made for special situations in low visibility environments, resulting in better descattering effects and stronger adaptability, thus obtaining clearer images and making navigation information more accurate.

[0048] In the post-processing stage, the image quality can be further improved by using an improved nonlocal mean filter and the Canny edge detection algorithm.

[0049] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0050] Figure 1 This is a structural block diagram of the vehicle navigation system according to an embodiment of the present invention;

[0051] Figure 2a This is a uniform sensor layout strategy in the embodiments of the present invention;

[0052] Figure 2b This is a non-uniform sensor layout strategy in the embodiments of the present invention;

[0053] Figure 2c This is the sensor layout strategy with a ring layout in the embodiments of the present invention;

[0054] Figure 3 This is a schematic diagram of a vehicle navigation system in an embodiment of the present invention;

[0055] The attached figures are labeled as follows:

[0056] 1 is the car window, and 2 is the sensor. Detailed Implementation

[0057] The present invention will be further described below with reference to the accompanying drawings and preferred embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0058] It should be noted that the directional terms such as left, right, up, down, top, and bottom used in this embodiment are only relative concepts or are based on the normal use of the product, and should not be considered as restrictive.

[0059] The following embodiments of the present invention propose a vehicle navigation system to improve driving safety, including a sensor, an optical signal processing device, and a navigation device. The sensor is disposed near a vehicle window, which includes at least the windshield and optionally a side window. The sensor collects optical signals within the window's field of view. The optical signal processing device processes the optical signals to obtain a clear image of the window's field of view after removing the effects of scattered light. The navigation device feeds navigation information to the driver or navigation data to an autonomous driving system based on the information from the image after removing the effects of scattered light. The navigation information includes image and / or voice information. For example, the image of the window's field of view can be directly displayed to the driver as navigation information. Alternatively, corresponding voice information can be added based on image recognition technology (such as prompts for identified targets or obstacles in the image). Navigation prompts can also be given to the driver in voice form based solely on the image recognition results. Information obtained from an in-vehicle communication system (such as real-time traffic information) can also be further integrated to display and / or broadcast navigation information. The information from the image after removing the effects of scattered light can also be used as data for navigation and control by the in-vehicle autonomous driving system. Therefore, by providing navigation data information based on clear images of the vehicle's window field of view, the present invention can reduce the impact of scattering phenomena, improve the driver's visibility in adverse weather conditions, or provide accurate information about the surrounding environment for autonomous driving systems, thereby effectively improving driving safety in low visibility environments.

[0060] In a preferred embodiment, the light signals collected by the sensor include direct light signals and scattered light signals. The scattered light signals include scattered light from the surrounding environment, the effect of the window on the scattered light from the surrounding environment, and the scattered light signals resulting from the accumulation effect of the window edge on the scattered light.

[0061] In a preferred embodiment, the navigation device includes a display device for displaying the image; preferably, the display device is a HUD (Head-Up Display) and the image can be displayed on the windshield.

[0062] In a preferred embodiment, the navigation device further includes an information fusion processing module, which fuses the real-time traffic information acquired by the vehicle communication system with the image after removing the influence of scattered light to form fused navigation image information and feeds it to the driver.

[0063] In a preferred embodiment, the optical signal processing device performs denoising processing on the collected optical signal, specifically including: dividing the input optical signal into several pixel blocks; calculating the local variance of the light intensity for each pixel block; calculating the size of the denoising window for each pixel block based on the local variance; adaptively performing denoising processing on the optical signal of each pixel block according to the denoising window of the corresponding size; and recombining the processed pixel blocks into a denoised optical signal.

[0064] In a preferred embodiment, the optical signal processing device performs histogram equalization processing on the collected optical signal, specifically including: dividing the input optical signal into several pixel blocks; calculating the local contrast for each pixel block and determining the contrast enhancement coefficient based on the local contrast; performing adaptive contrast enhancement processing on each pixel block according to the corresponding contrast enhancement coefficient; and recombining the processed pixel blocks into a contrast-enhanced optical signal.

[0065] In a preferred embodiment, the optical signal processing device performs deep learning fusion of optical signals collected by two or more sensors, specifically including: using a deep neural network to extract features from different types of optical signals and converting them into a common feature space; applying a feature fusion strategy to fuse the features of different types of optical signals in the common feature space; restoring the fused feature map to the spatial resolution of the original signal; and converting the fused feature map back into an optical signal to obtain a fused output.

[0066] The vehicle navigation system for improving driving safety proposed in this invention includes the following components:

[0067] I. Sensors

[0068] Multiple sensors are installed on the sides of the vehicle windows to collect scattered light signals. Furthermore, the windows include at least the windshield, and optionally, side windows. The sensors include one or a combination of visible light sensors and infrared sensors, and optionally further include one or a combination of lidar, millimeter-wave radar, and ultrasonic sensors. These sensors have the following characteristics:

[0069] (1) Millimeter-wave radar: It can penetrate adverse weather conditions such as fog, rain, and snow to detect objects at a distance.

[0070] (2) Ultrasonic sensor: It can measure the distance to surrounding objects and is often used in scenarios such as parking assistance.

[0071] (3) LiDAR (Light Detection and Ranging): LiDAR measures distance by emitting laser pulses and receiving the reflected signals, achieving high-precision 3D mapping. It is suitable for various scattering scenarios, including light, moderate, and heavy scattering. Because LiDAR measures distance by emitting laser pulses and receiving the reflected signals, it is relatively unaffected by scattering. In low-visibility environments such as fog, rain, and snow, LiDAR can provide relatively stable measurement results.

[0072] (4) Infrared sensor: Suitable for moderate to heavy scattering scenarios, such as heavy fog, dense fog and severe air pollution. In these scenarios, visible light scattering is more severe, while infrared sensors can effectively detect targets by capturing thermal radiation signals, reducing the impact of scattering on imaging.

[0073] (5) Optical sensor: Suitable for light to moderate scattering scenarios, such as light fog, haze, and light air pollution. In these scenarios, visible light scattering has a relatively small impact on image quality, and the optical sensor can acquire relatively clear images.

[0074] Sensor types include photoelectric sensors, infrared sensors, etc. Appropriate sensor combinations can be selected according to different light conditions and scattering levels, as shown in Table 1.

[0075] Table 1: Sensor combinations under different lighting conditions and scattering levels

[0076]

[0077] The overall solution for the sensor module should include consideration and configuration of multiple sensor types to achieve efficient capture of scattered light signals.

[0078] Preferably, embodiments of the present invention can combine multiple sensors (such as optical sensors, infrared sensors, etc.) to achieve multi-source data fusion and improve the accuracy and stability of descattering imaging.

[0079] When performing data fusion, the multi-source data from all parties needs to meet the following conditions:

[0080] a. Data timeliness: Data collected by each sensor should be from similar times to ensure the accuracy of data fusion.

[0081] b. Data consistency: The data format and units output by each sensor should be consistent to facilitate data processing and integration.

[0082] c. Data accuracy: The data collected by each sensor should be as accurate as possible to avoid introducing errors.

[0083] d. Data integrity: Each sensor should cover all information within its sensing range as much as possible to avoid data loss or duplication.

[0084] In specific embodiments, not all of the above conditions need to be met.

[0085] Sensor Type Selection: When designing sensor modules, different types of sensors should be considered to capture scattered light signals. For example, optical sensors (such as Complementary Metal Oxide Semiconductor (CMOS) cameras, or Charge Coupled Device (CCD) cameras) can be selected to capture visible light scattered signals, infrared sensors (such as thermal imagers) to capture infrared scattered signals, and lidar sensors to capture laser scattered signals. By combining multiple sensors, comprehensive capture of scattered light signals under various environmental conditions can be achieved.

[0086] Sensor placement strategy: Sensor 2 is installed on the side of the vehicle window 1 (such as the windshield or side window) to effectively capture scattered light signals. A uniform layout (e.g., Figure 2a As shown), non-uniform layout (such as) Figure 2b As shown), circular layout (such as...) Figure 2c (As shown) or other suitable layout methods can be used to achieve the best capture effect of scattered light signals. At the same time, considering the shape and size of the window, the sensor can be installed around the perimeter or part of the edge of the window to improve the sensor's ability to receive scattered light signals.

[0087] Let's take the car's windshield as an example to explain the sensor layout strategy:

[0088] Uniform Layout: Sensors are evenly distributed along the four edges of the windshield. Assuming there are eight sensors, two sensors can be placed on each edge, with equal spacing between them. For example, two sensors can be placed on each of the left and right edges, and two on each of the top and bottom edges. This layout ensures that light is received evenly from all directions.

[0089] Non-uniform layout: Sensors are placed along the edges of the windshield based on actual needs and scattering characteristics, but the distance between the sensors is not equal. For example, more sensors can be placed in areas requiring higher resolution, while the number of sensors can be reduced in other areas. This layout method allows for optimization of sensor distribution based on specific scenario requirements.

[0090] Other layout options: More layout options can be designed based on specific application requirements and sensor characteristics. For example, spiral, ring, or other custom layouts can be considered. The key is to achieve the best capture effect for scattered light signals.

[0091] II. Optical Signal Processing Device

[0092] The optical signal processing device uses a trained deep neural network to obtain a clear image of the vehicle window field of view from the optical signal after removing the influence of scattered light. Signal preprocessing and descattering algorithms are key parts of the optical signal processing device, and the specific methods are described below.

[0093] Signal preprocessing

[0094] In low-visibility environments, scattered light signals can be affected by various factors, such as noise and changes in illumination. Noise primarily originates from the sensor itself, electronic equipment, and the environment. Noise can cause unstable intensity variations in the scattered light signal, thus masking useful information. Noise can also cause signal fluctuations and distortion, reducing the signal-to-noise ratio and affecting signal reliability and accuracy. Changes in illumination conditions can alter the intensity and distribution of the scattered light signal. For example, illumination intensity is low at night or on cloudy days, while it is high during bright sunshine. These illumination variations affect the visibility, contrast, and color of the scattered light signal, making it more difficult to identify and process. Therefore, preprocessing of the raw light signal captured by the sensor is necessary before descattering.

[0095] Signal preprocessing includes the following:

[0096] 1. Denoising: Use denoising algorithms, such as bilateral filtering, nonlocal mean filtering, or deep learning methods, to denoise the captured signal and reduce the impact of noise on image quality.

[0097] This invention proposes an Adaptive Window Scattering Denoising Algorithm (AWS-DA) for scenarios involving side-scattering imaging of a vehicle window. The input is the scattered light signal captured by the sensor, and the output is the denoised scattered light signal. The denoising algorithm steps are as follows:

[0098] a) Divide the input scattered light signal into several small blocks (e.g., 8x8 pixels).

[0099] b) Calculate the local variance of the scattered light intensity for each small block.

[0100] c) Calculate the size of the denoising window based on the local variance. Higher local variance implies greater structural changes and scattering information, thus requiring a smaller window to preserve details; lower local variance implies less structural changes, thus allowing for the use of a larger window for smoothing.

[0101] d) Use an adaptive window to denoise the scattered light signal.

[0102] e) Reassemble the processed small blocks into a denoised scattered light signal.

[0103] The denoising algorithm proposed in this invention can adaptively adjust the size of the denoising window according to the local characteristics of the image, so as to effectively reduce noise while preserving image details.

[0104] Improvements for this specific scenario: Since the characteristics of scattered light signals may vary greatly in different regions, AWS-DA can adjust the window size according to these characteristics, thereby preserving useful scattered light information while denoising.

[0105] 2. Histogram equalization: Histogram equalization is performed on the denoised scattered light signal to enhance contrast and brightness, and improve the recognizability of the scattered light signal in the image.

[0106] This invention proposes an Adaptive Local Contrast Enhancement Algorithm (ALCEA). In a scenario involving side-scattering imaging from a vehicle window, the input is the denoised scattered light signal, and the output is the contrast-enhanced scattered light signal. The histogram equalization steps are as follows:

[0107] a) Divide the input scattered light signal into several small blocks (e.g., 8x8 pixels).

[0108] b) Calculate the local contrast for each small block and select an adaptive contrast enhancement factor based on the local contrast.

[0109] c) Apply an adaptive contrast enhancement algorithm to each small block to improve the discernibility of the scattered light signal.

[0110] d) The processed small pieces are reassembled into a contrast-enhanced scattered light signal.

[0111] The histogram equalization method in this embodiment of the invention can adaptively adjust the histogram equalization parameters according to the distribution of scattered light intensity in a local area, so as to improve the contrast and brightness of the image.

[0112] Improvements for this specific scenario: In low-visibility environments, the distribution of scattered light intensity in an image may be uneven. LRSI-AHE can adaptively adjust based on the characteristics of scattered light intensity in local areas, thereby improving the recognizability of the scattered light signal across the entire image.

[0113] 3. Sensor data fusion: Fusion of signals captured by different types of sensors (such as optical, infrared, lidar, etc.) to obtain richer information and improve the accuracy and robustness of signal processing.

[0114] This invention proposes a Deep Learning Fusion Algorithm (DLFA) for a scenario involving side-scattering imaging of a vehicle window. The input consists of scattered light signals captured by different types of sensors (such as optical, infrared, and lidar sensors), and the output is a fused scattered light signal. The sensor data fusion steps are as follows:

[0115] a) Preprocess the different types of scattered light signals input to enable deep learning fusion.

[0116] b) Use deep neural networks (such as convolutional neural networks) to extract features from the preprocessed scattered light signals. Deep neural networks can automatically learn the features of signals from different types of sensors and convert them into a common feature space.

[0117] c) In a common feature space, feature fusion strategies, such as weighted averaging and activation maximization, are applied to fuse features from different sensors. These strategies can be dynamically adjusted based on the scene and sensor performance to achieve the best fusion effect.

[0118] d) Use deconvolution or other upsampling operations to restore the fused feature maps to the spatial resolution of the original signal.

[0119] e) Convert the fused feature map back to the scattered light signal to obtain the final fused output.

[0120] The sensor data fusion method in this embodiment of the invention utilizes deep learning networks (such as convolutional neural networks, CNN) to fuse data from different sensors at multiple scales in order to extract richer feature information.

[0121] Improvements for this specific scenario: In low-visibility environments, signals captured by different types of sensors have different characteristics. DL-MSFS can adaptively fuse this information at multiple scales, improving the accuracy and robustness of signal processing, thereby achieving more reliable imaging results in complex low-visibility environments.

[0122] Descattering algorithm

[0123] The purpose of descattering algorithms is to separate the scattered light signal from the direct light signal in the preprocessed signal in order to obtain a clear and accurate image in low visibility environments.

[0124] The advantages of the embodiments of the present invention compared with traditional technologies are analyzed from the perspective of system differences.

[0125] In low-visibility environments, the vehicle navigation system of this invention utilizes the scattering characteristics of the side of the vehicle window (scattering characteristics refer to the interaction between scattered light from the vehicle window (such as the windshield or side windows) and the surrounding environment in low-visibility conditions. When light passes through particles such as fog, smoke, and dust in the atmosphere, it is scattered. The scattering characteristics of the side of the window utilize the accumulation effect of these scattered light rays at the window edge) to collect scattered light signals by installing sensors on the side. In this embodiment of the invention, the system can utilize existing vehicle windows or other transparent windows as imaging devices without the need for additional complex optical systems.

[0126] In contrast, existing imaging systems for low-visibility environments are typically designed for specific optical imaging systems (such as LiDAR), often requiring additional optical components or adjustments to imaging device parameters to adapt to different scattering environments. For example, depending on the characteristics of the scattering environment (such as fog, smoke, dust, etc.), specific optical filters or polarizers are adjusted or added to reduce the impact of scattered light on image quality. Based on ambient light conditions and the degree of scattering, parameters of imaging devices (such as cameras, LiDAR, etc.), such as exposure time, sensitivity, and contrast, are adjusted to adapt to different scattering environments. Different operating wavelengths of imaging devices are selected based on the characteristics of the scattering environment.

[0127] This invention also proposes a scattering imaging removal algorithm based on a deep neural network (DNN) model, which can be used in low visibility environments. The algorithm mainly includes the following:

[0128] Using DNNs for scattering imaging removal: In the process of scattering imaging removal, the transmittance estimation image is a key factor. Transmission represents the degree of light attenuation when light passes through a scattering medium (such as fog, smoke, snow, etc.). The transmittance estimation image describes the transmittance value of each pixel, usually represented in grayscale, where brighter pixels indicate higher transmittance and darker pixels indicate lower transmittance.

[0129] The transmittance image is not directly acquired, but estimated based on signals collected by different sensors. The specific methods and descriptions are as follows:

[0130] 1. Estimation of infrared sensor transmittance:

[0131] Step 1: Read the infrared image I_IR;

[0132] Step 2: Calculate the local atmospheric light A_IR. In a specific embodiment, the brightest part of the image can be used as an approximation of the atmospheric light: A_IR = max(I_IR);

[0133] Step 3: Calculate the ratio of I_IR to A_IR: R_IR = I_IR / A_IR;

[0134] Step 4: Threshold R_IR to obtain the transmittance image T_IR: T_IR = 1 - R_IR.

[0135] 2. Optical sensor transmittance estimation:

[0136] Step 1: Read the optical image I_RGB;

[0137] Step 2: Convert the optical image from the RGB color space to the HSV color space to obtain the H, S, and V channels;

[0138] Step 3: Calculate the local atmospheric light A_V. In a specific embodiment, the brightest part of the image can be used as an approximation of the atmospheric light: A_V = max(V);

[0139] Step 4: Calculate the ratio of V to A_V;

[0140] Step 5: Threshold R_V to obtain the transmittance image T_V: T_V = 1 – R_V.

[0141] 3. LiDAR transmittance estimation:

[0142] Step 1: Read the lidar point cloud data P;

[0143] Step 2: Convert the point cloud data into a two-dimensional depth image I_depth, where Z represents the distance value in the point cloud;

[0144] As a preferred embodiment, step 2 proposes a method for converting point cloud data into a two-dimensional depth image. Unlike existing methods that directly perform mapping and projection, this method includes the following steps:

[0145] Step 21: Partitioning and Block Processing

[0146] The entire point cloud data is first divided into equal-sized blocks, each containing a certain number of points. This helps reduce computational complexity while also better preserving local information.

[0147] Step 22: Introduce the lighting model

[0148] When projecting a point cloud onto a 2D plane, we not only preserve the depth information of each point but also consider its normal information. Specifically, the angle between the normal direction and the light source direction of each point is added to the depth image as "lighting" information (which can be achieved using the Lambert lighting model). This results in a 2D image that simultaneously contains both depth and lighting information.

[0149] Step 23: Gaussian Mixture Model Processing

[0150] For each block in the point cloud, instead of directly calculating the nearest distance from all points to the observer, we use a Gaussian mixture model to model these distances, resulting in multiple Gaussian distributions. Then, we select the mean of the Gaussian distribution with the largest value as the depth value for that block. This allows us to better handle noise and outliers in the point cloud.

[0151] Step 24: Graph-based data fusion

[0152] Finally, to ensure continuity in the depth image, a graph cut algorithm is used for post-processing to make the depth values ​​of adjacent pixels as close as possible. This not only yields a smooth depth image but also better preserves the boundary information of objects.

[0153] Step 3: Calculate the local atmospheric light A_depth. In a specific embodiment, the brightest part of the image can be used as an approximation of the atmospheric light: A_depth = max(I_depth);

[0154] Step 4: Calculate the ratio of I_depth to A_depth: R_depth = I_depth / A_depth;

[0155] Step 5: Threshold R_depth to obtain the transmittance image T_depth: T_depth = 1 – R_depth.

[0156] In this embodiment of the invention, the preprocessed image is fed into a DNN, and the input to the DNN is the transmittance estimation image corresponding to the preprocessed image. The DNN in this embodiment uses a U-Net architecture, which includes feature extraction from the input and deconvolution of the output. Compared to the traditional U-Net, this embodiment optimizes the network structure and parameter settings to better adapt to imaging systems in low-visibility environments such as side-scattering imaging of vehicle windows. Specifically, this embodiment uses special convolutional kernels and normalization methods to increase the network's stability and robustness.

[0157] The traditional U-Net network structure and parameters are described below:

[0158] U-Net is a typical encoder-decoder architecture, including a compression path (downsampling) and an expansion path (upsampling). The basic structure of the U-Net network is as follows:

[0159] (1) Encoder (downsampling): It consists of 4 convolutional layers, each followed by a ReLU activation function and a 2x2 max pooling layer. The kernel size is 3x3 and the stride is 1.

[0160] (2) Decoder (Upsampling): It consists of 4 upsampling layers, each followed by a 2x2 deconvolution layer, a convolutional layer, and a ReLU activation function. The convolutional kernel size is 3x3, and the stride is 1.

[0161] (3) Loss function: use pixel-level cross-entropy loss.

[0162] This invention also proposes a U-Net-based neural network structure. Based on the traditional U-Net, this structure incorporates attention mechanisms, hybrid convolutional structures, multi-scale feature fusion, adaptive loss functions, and unsupervised pre-training to achieve better descattering performance. Specific details are described below:

[0163] 1. First, attention modules are introduced into the convolutional layers of the encoder and decoder of U-Net, enabling the network to automatically learn and focus on important regions in the input image, thereby helping the network to pay more attention to key details in low-visibility environments when processing scattered light signals.

[0164] 2. Secondly, embodiments of the present invention introduce hybrid convolutional structures into the network, such as the Inception module. This structure can help the network capture image details at different scales and improve its ability to process scattered light signals in low-visibility environments.

[0165] 3. In the decoder section, this embodiment of the invention uses a Feature Pyramid Network (FPN) structure to perform multi-scale feature fusion. Through top-down paths and lateral connections, feature maps of different scales are fused together, which helps the network to better capture scattered light information at different scales.

[0166] 4. To enable the model to focus more on key details in low-visibility environments, this embodiment of the invention uses an adaptive loss function, which dynamically adjusts the weights according to the characteristics of the input image, allowing the network to adaptively optimize the prediction of important regions.

[0167] 5. Since imaging systems in low-visibility environments may face the problem of scarce training data, this embodiment of the invention uses unsupervised methods (such as autoencoders or GANs) for pre-training before training the deep network, which helps the network learn richer feature representations, thereby improving descattering performance.

[0168] In summary, the neural network structure proposed in this invention, through the introduction of innovations such as attention mechanisms, hybrid convolutional structures, multi-scale feature fusion, adaptive loss functions, and unsupervised pre-training, can better adapt to imaging systems in low-visibility environments such as side-scattering imaging of vehicle windows. These innovations collectively enable the neural network to achieve better descattering performance, thereby meeting the needs of specific scenarios.

[0169] In a preferred embodiment, the following is also included:

[0170] 1. Introduction of the attention module: The calculation formula for the attention module is as follows:

[0171] S=F(X,W)*σ(G(X,V))

[0172] Where S is the output feature map, X is the input feature map, F and G are the convolution operations, W and V are the convolution kernels, and σ is the activation function (such as the sigmoid function).

[0173] 2. Hybrid Convolutional Structure: Taking the Inception module as an example, its calculation formula is as follows:

[0174] I_out=concat(conv_1x1(I_in),conv_3x3(I_in),conv_5x5(I_in),max_pool(I_in))

[0175] Where I_out is the output feature map, I_in is the input feature map, concat is the concatenation operation, conv_1x1, conv_3x3 and conv_5x5 are convolution operations of different sizes, and max_pool is the pooling operation.

[0176] 3. Multi-scale feature fusion: A Feature Pyramid Network (FPN) structure is used, and its calculation formula is as follows:

[0177] P_i=upsample(P_(i+1))+conv_1x1(C_i)

[0178] Where P_i and C_i are the pyramid feature map and the original feature map, respectively, i is the level, and upsample is the upsampling operation.

[0179] 4. Adaptive Loss Function: Taking the weighted cross-entropy loss function as an example, its calculation formula is as follows:

[0180] L=-∑(w_p*y*log(σ(x))+w_n*(1-y)*log(1-σ(x)))

[0181] Where L is the loss value, y is the true label, x is the predicted value, σ is the sigmoid function, and w_p and w_n are the weights of the positive and negative classes, respectively.

[0182] 5. Unsupervised pre-training: Taking an autoencoder as an example, its loss function can be expressed as:

[0183] L = ||X - decode(encode(X))||2

[0184] Where X is the input, and encode and decode are the encoder and decoder, respectively.

[0185] The optical signal processing device also performs post-processing on the images output by the deep neural network.

[0186] This invention post-processes the output of the DNN (a DNN-generated image that removes scattering effects, preserving target object information while reducing the impact of scattered light on the image, thus improving image quality. In this scenario, the DNN-output image is obtained by fusing and processing scattered light signals captured by optical, infrared, and lidar sensors.) to remove residual scattering and enhance edges. To remove residual scattering, this invention uses a non-local mean (NLM) filter, which effectively reduces noise and preserves details. To enhance edges, this invention uses the Canny edge detection algorithm (an edge detection operator proposed by computer scientist John F. Canny in 1986).

[0187] In the post-processing process, this embodiment of the invention employs a non-local mean (NLM) filter and the Canny edge detection algorithm. To adapt to the scenario of side scattering imaging of vehicle windows, this embodiment of the invention has improved these methods according to specific requirements.

[0188] 1. Improved NLM filter

[0189] The NLM filter is a filtering method based on image region similarity, which can effectively reduce noise while preserving image details. In the scenario of side scattering imaging from a car window, for the image output by the DNN, this embodiment of the invention first calculates the weighted average value of each pixel in the image. Here, the weights are proportional to the similarity between adjacent pixels. Through this method, this embodiment of the invention can preserve the detailed information in the image while removing residual scattering effects.

[0190] 2. Improved Canny edge detection algorithm

[0191] To adapt to scenarios involving side-scattering imaging from vehicle windows, this embodiment of the invention optimizes the Canny algorithm. First, the image output from the DNN is Gaussian filtered to smooth the image and eliminate noise. Next, the Sobel operator is used to calculate the gradient magnitude and direction of the image. Then, non-maximum suppression is applied to the gradient magnitude to preserve edges and reduce the occurrence of false edges. Finally, a double-thresholding method is used to connect the edges, thereby obtaining complete edge information.

[0192] Through these two post-processing steps, embodiments of the present invention can further optimize the image quality of the DNN output, remove residual scattering effects, and enhance edges. These improvements make the method of the embodiments of the present invention more suitable for imaging systems in low-visibility environments that process side-scattering imaging from vehicle windows.

[0193] Compared to existing technologies, the embodiments of this invention offer superior descattering performance and better adaptability. First, the embodiments of this invention design a specific DNN structure and parameter settings for imaging systems operating in low-visibility environments with side scattering from vehicle windows. Second, the embodiments of this invention employ special convolutional kernels and normalization methods to enhance the network's stability and robustness. Finally, the embodiments of this invention improve the NLM filter and the Canny edge detection algorithm for post-processing the output to further enhance the descattering effect and edge information.

[0194] III. Navigation Device

[0195] The navigation device feeds navigation information to the driver based on the image after removing the effects of scattered light. The navigation information includes image and / or voice information.

[0196] 1. The navigation device includes a display device for displaying the image; preferably, the display device is a HUD head-up display device, and the image can be displayed on the windshield.

[0197] The display device transmits the descattered image in real time to the display device (such as Augmented Reality (AR) glasses, in-vehicle displays, etc.) within the driver's field of vision, enabling the driver to obtain a clear view of an ultra-wide field of vision in low visibility environments.

[0198] In one specific embodiment, a head-up display (HUD) is installed on the inside of the car's windshield, projecting a clear image directly into the driver's field of vision through the transparent glass. This allows navigation, speed, warning, and other information to be displayed directly on the road ahead. This method allows the driver to focus their attention on the road ahead, while also reducing the frequency of eye movement and improving driving safety.

[0199] 2. The navigation device also includes an information fusion processing module, which fuses the real-time traffic information acquired by the vehicle communication system with the image after removing the influence of scattered light to form fused navigation image information and feeds it to the driver.

[0200] like Figure 3 As shown, the information fusion processing module is responsible for coordinating the information flow and processing between various modules to ensure the efficient operation of the system. Furthermore, this module can communicate with other vehicle systems (such as navigation and in-vehicle communication) to obtain real-time traffic information and driver operation commands. The process of obtaining real-time traffic information and driver operation commands is explained below:

[0201] 1. Scenario: The driver is driving in the rain, and there is a complicated intersection ahead. Visibility is affected by the rain.

[0202] 2. Real-time traffic information: The information fusion processing module receives traffic information through the vehicle communication system, indicating that there is a temporary construction area at the upcoming intersection, and some lanes have been closed. In addition, it receives the status information of nearby traffic lights, indicating that the current light is green.

[0203] 3. Driver's Operation Command: The destination has been set in the driver's navigation system. The driver issues a voice command: "Show best route," requesting the system to plan the best driving route based on current traffic information.

[0204] 4. After receiving this information, the information fusion processing module coordinates the various parts of the system for processing. First, it fuses the acquired real-time traffic information with the descattered image to highlight construction areas and traffic lights. Then, based on the driver's instructions, it marks the optimal driving route on a display device, such as displaying a guide line on AR glasses or an in-vehicle display screen.

[0205] In this way, the information fusion processing module ensures smooth information transmission and processing between various devices, improving the system's efficient operation. Simultaneously, it helps drivers better understand road conditions in low-visibility environments, enhancing driving safety.

[0206] The embodiments of the present invention also include the following parts:

[0207] IV. Power Supply

[0208] The power supply unit provides a stable and reliable power supply to the system to ensure normal operation under various working conditions.

[0209] By integrating and optimizing the above components, the embodiments of the present invention can provide drivers with a clear field of vision and real-time driving assistance information in low visibility environments, thereby improving driving safety and comfort.

[0210] Compared to the above-mentioned technologies, the vehicle navigation system for improving driving safety proposed in this invention has the following advantages:

[0211] 1. Utilizing existing vehicle windows: This embodiment of the invention utilizes the side of vehicle windows for scattering imaging, eliminating the need for additional complex equipment, reducing the burden on the system, and lowering costs.

[0212] 2. Extra-large field of view: The car window itself is an imaging device with an extra-large field of view (the car window itself is equivalent to an extra-large lens, and this extra-large lens can have an extra-large field of view). Compared with traditional imaging devices, such as cameras and lidar, it can provide a wider field of view and improve the driver's sense of security.

[0213] 3. Multi-sensor fusion: The embodiments of the present invention can combine multiple sensors (such as optical sensors, infrared sensors, etc.) to achieve multi-source data fusion and improve the accuracy and stability of descattering imaging.

[0214] 4. Real-time processing: Combined with specially designed descattering algorithms, such as deep learning-based image processing technology, the collected scattered light signals can be processed in real time to generate clear descattered images.

[0215] The vehicle navigation system proposed in this invention for improving driving safety organically integrates sensors, optical signal processing devices, and navigation devices, in addition to installing sensors on the sides of the vehicle windows and their subsequent descattering processing algorithms. In low-visibility environments, this vehicle navigation system can effectively provide drivers with clear and reliable visual information, reducing driving risks. By processing and presenting descattered images in real time, drivers can drive more safely in adverse weather conditions.

[0216] This invention has broad application prospects in improving driving safety, especially in the automotive industry under low visibility conditions. Compared with existing technologies, this invention offers advantages such as better visual effects, lower cost, a wider field of view, and stronger adaptability. This vehicle navigation system presents descattered images in real time within the driver's field of vision, effectively reducing driving risks and improving driving safety. Furthermore, this vehicle navigation system can communicate with other vehicle systems (such as navigation and in-vehicle communication) to obtain real-time traffic information and driver operation commands. The design and implementation of the entire method demonstrate its advanced nature and practicality.

[0217] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, several equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or purpose, should be considered within the scope of protection of the present invention.

Claims

1. A vehicle navigation system for improving driving safety, characterized in that, The system includes a sensor, an optical signal processing device, and a navigation device. The sensor is positioned near a vehicle window, which includes at least the windshield and optionally side windows. The sensor collects light signals within the window's field of view. The collected light signals include direct light signals and scattered light signals. The scattered light signals include scattered light from the surrounding environment, the effect of the window on the scattered light from the surrounding environment, and the accumulated effect of the window's edges on the scattered light. The optical signal processing device processes the light signals to obtain a signal with the scattered light removed, suitable for low-visibility environments. A clear image of the vehicle window's field of view; the navigation device feeds navigation information to the driver or navigation data to the autonomous driving system based on information from the image after removing the effects of scattered light; the navigation information includes image and / or voice information; the optical signal processing device generates a transmittance estimation image corresponding to the type of optical signal from the collected optical signals, inputs the transmittance estimation image into a trained U-Net-based deep neural network, and outputs an image with descattering effect after passing through the deep neural network; wherein, the transmittance estimation image describes the transmittance value of each pixel in the image, representing the degree of attenuation of light after passing through the scattering medium.

2. The vehicle navigation system as described in claim 1, characterized in that, The navigation device includes a display device for displaying the image; the display device is a HUD (Head-Up Display) and the image can be displayed on the windshield.

3. The vehicle navigation system as described in claim 1, characterized in that, The navigation device also includes an information fusion processing module, which fuses the real-time traffic information obtained by the vehicle communication system with the image after removing the influence of scattered light to form fused navigation image information and feeds it to the driver.

4. The vehicle navigation system as described in claim 1, characterized in that, The sensor includes one or a combination of a visible light sensor and an infrared sensor.

5. The vehicle navigation system as described in claim 1, characterized in that, The sensor also includes one or more combinations of lidar, millimeter-wave radar, and ultrasonic sensors.

6. The vehicle navigation system as described in claim 4, characterized in that, The optical signal processing device performs noise reduction processing on the collected optical signals, specifically including: The input optical signal is divided into several pixel blocks; Calculate the local variance of light intensity for each pixel block; The size of the denoising window for each pixel block is calculated based on the local variance. The light signal of each pixel block is denoised adaptively according to the denoising window of the appropriate size; The processed pixel blocks are recombined into a denoised light signal; And / or, the optical signal processing device performs histogram equalization processing on the collected optical signals, specifically including: The input optical signal is divided into several pixel blocks; Calculate the local contrast for each pixel block and determine the contrast enhancement coefficient based on the local contrast. Adaptive contrast enhancement processing is applied to each pixel block according to the corresponding contrast enhancement coefficient. The processed pixel blocks are then recombined into a contrast-enhanced light signal.

7. The vehicle navigation system as described in claim 4, characterized in that, The optical signal processing device performs deep learning fusion of optical signals collected by two or more sensors, specifically including: Deep neural networks are used to extract features from different types of optical signals and convert them into a common feature space. In the common feature space, a feature fusion strategy is applied to fuse the features of different types of optical signals; The fused feature maps are restored to the spatial resolution of the original signal; The fused feature maps are converted back into optical signals to obtain the fused output.

8. The vehicle navigation system as described in claim 4, characterized in that, The generation of the infrared light signal transmittance estimation image includes: Acquire the light intensity of each pixel in the infrared image; Determine the reference light intensity; Calculate the ratio of the light intensity of a pixel in an infrared image to the light intensity of a reference pixel; Based on the ratio, an estimated image of the transmittance of the infrared light signal is obtained.

9. The vehicle navigation system as described in claim 4, characterized in that, The generation of the transmittance estimation image for visible light signals includes: Obtain the light intensity of each pixel in a visible light RGB image; Determine the reference light intensity; Calculate the ratio of the light intensity of a pixel in a visible light RGB image to the light intensity of a reference pixel; Based on the ratio, an estimated image of the transmittance of the visible light signal is obtained.

10. The vehicle navigation system as described in claim 5, characterized in that, The generation of the transmittance estimation image for lidar signals includes: Acquire lidar point cloud data; Convert point cloud data into a two-dimensional depth image; Determine the reference depth; Calculate the ratio of the depth of a pixel in a 2D depth image to the depth of a reference image; Based on the ratio, a transmittance estimation image of the lidar signal is obtained.

11. The vehicle navigation system as described in claim 10, characterized in that, The method for converting point cloud data into a two-dimensional depth image specifically includes: Step 1: Partitioning and Block Processing The entire point cloud data is divided into equal-sized blocks, each containing a predetermined number of points; Step 2: Introduce the lighting model When projecting a point cloud onto a 2D plane, the depth information and normal information of each point are preserved simultaneously, resulting in a 2D image that contains both depth and lighting information. Step 3: Gaussian Mixture Model Processing For each block in the point cloud, a Gaussian mixture model is used to model the depth of the points in each block, resulting in multiple Gaussian distributions. Then, the mean of the Gaussian distribution with the largest distribution is selected as the depth value of this block. Step 4: Graph-based data fusion The graph cut algorithm is used to post-process the depth image, which smooths the changes in depth values ​​between adjacent pixels.

12. The vehicle navigation system as described in claim 1, characterized in that, The deep neural network has one or more of the following characteristics: An attention module is introduced into the convolutional layers of the encoder and decoder in U-Net; Introduce hybrid convolutional structures into the network; The decoder uses a Feature Pyramid Network (FPN) structure for multi-scale feature fusion, which combines feature maps of different scales through top-down paths and lateral connections. An adaptive loss function is used to dynamically adjust the weights based on the characteristics of the input image; Pre-training is performed using unsupervised methods.

13. The vehicle navigation system as described in claim 1, characterized in that, The optical signal processing device also performs post-processing on the image output by the deep neural network, including removing residual scattering and enhancing image edges; The removal of residual scattering includes: calculating a weighted average value for each pixel in the image, the weight of which is proportional to the similarity between adjacent pixels, and performing nonlocal mean (NLM) filtering based on the weighted average value; The image edge enhancement includes: performing Gaussian filtering on the image; calculating the gradient magnitude and direction of the image using the Sobel operator; performing non-maximum suppression on the gradient magnitude; and connecting the edges using a double thresholding method to obtain complete edge information.

Images