A 1:1 real-time augmented reality display driving assistance system and method based on color infrared imaging and AR HUD in a low-visibility environment

Abstract

This invention discloses a 1:1 real-time augmented reality display driving assistance system and method based on color infrared imaging and AR HUD in low-visibility environments. The system includes a color thermal imaging camera or multispectral infrared camera, an image processing unit, and an AR HUD display unit. The method includes: acquiring complete color infrared real-scene images, preprocessing, geometrically calibrating, and color enhancing the images, and then projecting them in real-time onto the windshield at a 1:1 scale using an AR HUD for overlay display with the real scene. This invention can provide drivers with real-time, clear color road views in low-visibility conditions such as nighttime, foggy, and rainy weather, eliminating the risk of missed detections and significantly improving driving safety.

Description

Technical Field

[0001] This invention relates to the field of vehicle safe driving technology, and in particular to an augmented reality driving assistance system and method under low visibility conditions. Background Technology

[0002] With the continuous increase in car ownership, road traffic safety issues are becoming increasingly prominent. Statistics show that the incidence of traffic accidents under low visibility conditions such as nighttime, foggy days, and rainy days is several times higher than during the day.

[0003] Currently, most in-vehicle night vision systems on the market use thermal imaging technology, which can only display black and white images and cannot distinguish key color information such as traffic lights, vehicle taillights, and traffic signs. Drivers still need to rely on their naked eyes to judge these important signals. Existing AR HUD systems are mainly used to display virtual information such as navigation and vehicle speed, but cannot project the real-time road view ahead, and cannot fundamentally solve the problem of obstructed driver vision under low visibility conditions.

[0004] Furthermore, traditional night vision systems typically display images on the central control screen or instrument panel, requiring drivers to frequently shift their gaze, which inherently poses a serious safety hazard. Even those few systems that attempt to project night vision images onto the windshield generally suffer from misalignment and disproportion between the image and the real scene, easily leading to driver misjudgment.

[0005] Patent CN109987025B discloses a vehicle driving assistance system and method for nighttime environments. This system uses an infrared camera to capture images, identify pedestrians, vehicles, and other objects ahead, and then displays augmented reality only the objects invisible to the driver. However, this system has the following inherent drawbacks: 1. It can only display a single recognized object and cannot display the overall road conditions. Drivers cannot obtain comprehensive environmental information such as potholes, road edges, and obstacles. 2. Using ordinary infrared cameras, only black and white images can be generated, and it is impossible to distinguish key color information such as traffic lights, vehicle taillights, and traffic signs; 3. It is highly dependent on object recognition algorithms and visibility judgment algorithms, which poses a security risk of missed detections or false detections. If the algorithm fails, it will lead to serious accidents.

[0006] Therefore, there is an urgent need for a driving assistance system and method that can provide drivers with a real-time, clear, color, and 1:1 accurate superimposed view of the real road ahead under low visibility conditions. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a 1:1 real-time augmented reality display driving assistance system and method based on color infrared imaging and AR HUD. It can provide drivers with real-time, clear color road images under low visibility conditions such as night, fog, and rain, and accurately overlay them with the real scene, significantly improving driving safety.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a 1:1 AR HUD low-visibility driving assistance system based on color infrared imaging, comprising: A color thermal imaging camera or multispectral infrared camera is installed at the front of the vehicle to capture a complete infrared real-view image and color information of the vehicle's direction of travel. An image processing unit, electrically connected to the color thermal imaging camera, is used to receive and process infrared grayscale images and convert them into high-fidelity visible light color images. The AR HUD display unit is electrically connected to the image processing unit and is used to project the processed full-color infrared real-scene image onto the vehicle's windshield at a 1:1 scale, overlaying it with the real road scene.

[0009] The system does not include an object recognition unit or a visibility determination unit; it directly projects the acquired complete road image.

[0010] Furthermore, the thermal imaging camera can simultaneously cover the visible light and infrared bands, and can collect and retain color information of traffic lights, vehicle taillights and traffic signs.

[0011] Furthermore, the projection field of view of the AR HUD display unit is the same as the shooting field of view of the thermal imaging camera.

[0012] Furthermore, the image processing unit includes an image preprocessing module, a geometric calibration module, a color enhancement module, and a projection control module.

[0013] Furthermore, it also includes a light sensor, electrically connected to the image processing unit, for detecting ambient light intensity.

[0014] Furthermore, the projection control module can automatically adjust the projection brightness of the AR HUD display unit according to the ambient light intensity detected by the light sensor.

[0015] Secondly, the present invention provides a 1:1 AR HUD real-time augmented reality display driving assistance method based on color infrared imaging, comprising the following steps: Real-time complete infrared images and color information of the vehicle's direction of travel are acquired using a color thermal imaging camera or a multispectral infrared camera. Denoising and distortion correction are performed on the acquired complete infrared real-scene images; Based on the camera's installation parameters and the vehicle's movement, the image is geometrically calibrated to ensure that the objects in the image correspond one-to-one with the objects in the real scene in terms of position and scale. The calibrated image is then subjected to color enhancement processing to improve its color saturation and contrast. The processed full-color infrared real-scene image is sent to the AR HUD display unit in real time; The AR HUD display unit projects images onto the vehicle's windshield in real time at a 1:1 scale, overlaying them with the real road scene.

[0016] The method does not include object recognition steps and visibility determination steps.

[0017] Furthermore, the geometric calibration described in step 3 specifically includes: Calculate the real-world coordinates of each pixel in the image based on the camera's installation height, pitch angle, and horizontal angle. Based on the calculation results, a perspective transformation is performed on the image to keep the straight lines in the image parallel to the straight lines in the real world. The image is dynamically compensated based on the vehicle's real-time speed and steering angle to eliminate image shift caused by vehicle movement.

[0018] Furthermore, it also includes the following steps: The ambient light intensity is detected using a light sensor; The AR HUD display unit automatically adjusts its projection brightness based on the detected ambient light intensity.

[0019] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described 1:1 AR HUD low-visibility driving assistance method based on color infrared imaging.

[0020] The beneficial effects of this invention are as follows: Using a color thermal imaging camera or a multispectral infrared camera, it can simultaneously capture images in the visible light and infrared bands. Under low visibility conditions, it can not only see the outline of objects, but also clearly distinguish key color information such as traffic lights and vehicle taillights. AR HUD projects a complete color infrared real-world image onto the windshield in real time at a 1:1 scale, accurately overlaying it with the real scene, allowing the driver to obtain comprehensive road information without having to look away. The unique geometric calibration method can dynamically calibrate the image based on the camera installation parameters and vehicle movement status, ensuring that the image is completely consistent with the real scene in position and scale, eliminating the sense of misalignment; It does not require object recognition or visibility judgment, eliminating the security risks of missed or false detections, and thus has higher system reliability; It can automatically adjust the projection brightness according to the ambient light intensity, ensuring good display effect under various lighting conditions. Attached Figure Description

[0021] Figure 1 This is a schematic block diagram of the 1:1 AR HUD real-time augmented reality display driving assistance system based on color infrared imaging according to the present invention.

[0022] Figure 2 This is a flowchart of the 1:1 AR HUD real-time augmented reality display driving assistance method based on color infrared imaging according to the present invention.

Claims

1. A 1:1 AR HUD low-visibility driving assistance system based on color infrared imaging, characterized in that, include: A color thermal imaging camera or multispectral infrared camera is installed at the front of the vehicle to acquire a complete infrared real-scene image and color information in the direction the vehicle is traveling; an image processing unit is electrically connected to the color thermal imaging camera to receive and process the infrared grayscale image and convert it into a high-fidelity visible light color image; an AR HUD display unit is electrically connected to the image processing unit to project the processed complete color infrared real-scene image onto the vehicle's windshield at a 1:1 scale and overlay it with the real road scene.

2. The system according to claim 1, characterized in that, The color thermal imaging camera or multispectral infrared camera covers both visible light and infrared bands, and can collect and retain color information from traffic lights, vehicle taillights, and traffic signs.

3. The system according to claim 1, characterized in that, The projection field of view of the AR HUD display unit is the same as the shooting field of view of the color thermal imaging camera.

4. The system according to claim 1, characterized in that, The image processing unit includes an image preprocessing module, a geometric calibration module, a color enhancement module, and a projection control module.

5. The system according to claim 1, characterized in that, It also includes a light sensor, which is electrically connected to the image processing unit, for detecting ambient light intensity.

6. The system according to claim 5, characterized in that, The projection control module can automatically adjust the projection brightness of the AR HUD display unit according to the ambient light intensity detected by the light sensor.

7. A 1:1 AR HUD low-visibility driving assistance method based on color infrared imaging, characterized in that, Includes the following steps: S1: Acquire real-time complete color infrared real-scene images of the vehicle's forward direction using a color thermal imaging camera or a multispectral infrared camera; S2: Perform noise reduction and distortion correction on the acquired infrared real-scene images; S3: Perform geometric calibration on the images based on the camera's installation parameters and the vehicle's motion state, so that the objects in the images correspond one-to-one with the objects in the real scene in terms of position and scale. S4: Perform color enhancement processing on the calibrated image to convert the infrared grayscale image into a high-fidelity visible light color image; S5: Send the processed complete color infrared real-scene image to the AR HUD display unit; S6: The AR HUD display unit projects images onto the vehicle's windshield in real time at a 1:1 scale, overlaying them with the real road scene.

8. The method according to claim 7, characterized in that, The geometric calibration described in step S3 specifically includes: calculating the real-world coordinates of each pixel in the image based on the camera's installation height, pitch angle, and horizontal angle; performing perspective transformation on the image based on the calculation results to keep the straight lines in the image parallel to the straight lines in the real world; and performing dynamic compensation on the image based on the vehicle's real-time speed and steering angle to eliminate image offset caused by vehicle movement.

9. The method according to claim 7, characterized in that, It also includes the following steps: The ambient light intensity is detected by a light sensor; the projection brightness of the AR HUD display unit is automatically adjusted according to the detected ambient light intensity.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 7-9.

Images