Vehicle infrared thermal imaging auxiliary system for assisting driver to drive safely

Abstract

The application discloses a vehicle infrared thermal imaging auxiliary system for assisting drivers to drive safely, relates to the field of infrared thermal imaging technology, and comprises a vehicle-mounted camera module, which is used for acquiring visible light video and infrared thermal imaging video of a specified direction outside a vehicle; an object similarity analysis module, which is used for determining the position of a current heat-emitting object in a picture according to the infrared thermal imaging video, obtaining position coordinate information and an object boundary OB1 of the heat-emitting object, and confirming the object boundary ob1 of the object at the corresponding coordinate position in the picture at the same moment in the visible light video according to the obtained position coordinate information; when the object boundary similarity between the object boundary OB1 and the object boundary ob1 is lower than a first similarity threshold, the heat-emitting object at the coordinate position is identified as a visible restricted object; and a video fusion module, which is used for obtaining a fusion video of the visible light video and a thermal image of the visible restricted object, so that it can be determined intelligently whether the thermal image of the heat-emitting object in the infrared thermal imaging needs to be displayed in the video.

Description

Technical Field

[0001] This invention relates to the field of infrared thermal imaging technology, specifically to a vehicle infrared thermal imaging assistance system to aid drivers in safe driving. Background Technology

[0002] Rainy and foggy weather is a high-risk period for traffic accidents, with accidents caused by obstructed visibility being commonplace. To enhance the intelligent perception capabilities of vehicles and improve driving safety, automotive driver assistance systems that utilize vehicle infrared thermal imaging technology have emerged.

[0003] From a technical perspective, infrared thermal imaging driving assistance systems have advantages over other sensors such as visible light cameras, lidar, and millimeter-wave radar, such as being less affected by rain, snow, fog, haze, and sandstorms. At the same time, infrared thermal imaging automotive night vision systems are very sensitive to the perception of living beings, making them an excellent choice and supplement for assisting driving at night or in inclement weather, or for increasing the safety and reliability of autonomous driving.

[0004] Dual-light fusion is a combination of "low-light + thermal imager" dual channels, namely visible light channel and infrared light channel.

[0005] For example, the Chinese invention with patent number 201410014024.2, "A method for real-time fusion of infrared images and visible light images".

[0006] For example, Chinese invention application No. 202210359925.X, entitled "A method and system for fusing infrared images and high-definition video," can display the infrared thermal image of a heat-generating object in a visible light image within the fused video, allowing users to spot the heat-generating object in a visible light image with better texture and thus react promptly.

[0007] Currently, dual-light fusion has also been applied to automotive driver assistance systems to address the limitations of images obtained from single-type sensors, resulting in richer visuals.

[0008] For example, Chinese invention application No. 201610363585.2, "Automotive Anti-Halo Video Image Processing Method Based on Fusion of Infrared and Visible Light Images".

[0009] In fused video, the thermal image of a heated object (such as a pedestrian or a car) in infrared thermal imaging can be displayed in a visible light image, enabling drivers to better detect potential dangers even when visibility is poor.

[0010] However, in the face of ever-changing driving conditions, if the video is always displayed as a fused video combining visible light and infrared thermal imaging, the presence of infrared thermal images may actually hinder the driver's ability to judge details in the video when the driver has good visibility.

[0011] For example, when reversing, visible light images should be used as auxiliary reversing images when visibility is good, while the appearance of infrared thermal images is not conducive to judging the reversing situation.

[0012] If users can only selectively turn infrared thermal imaging on or off, they often forget to turn it on while the car is in motion, making it difficult to make timely judgments when dangerous situations arise.

[0013] Therefore, it is becoming increasingly important to intelligently determine whether the thermal image of a heat-generating object in infrared thermal imaging needs to be displayed in the video. Summary of the Invention

[0014] To overcome the shortcomings mentioned above, this invention aims to provide a vehicle infrared thermal imaging assistance system that can automatically determine whether it is necessary to fuse the thermal image of a heated object with a visible light image, thereby selectively displaying the visible light image or the fused image of the thermal image of the heated object and the visible light image in the video, thus better assisting the driver in safe driving.

[0015] To achieve the above objectives, the present invention provides the following technical solution: a vehicle infrared thermal imaging assistance system to aid drivers in safe driving, comprising... The vehicle-mounted camera module is used to acquire visible light video and infrared thermal imaging video from a specified direction outside the vehicle; The object similarity analysis module determines the position of the current heat-generating object in the frame based on the infrared thermal imaging video, obtains the position coordinate information and object boundary OB1 of the heat-generating object, and confirms the object boundary ob1 of the object at the corresponding coordinate position in the visible light video at the same moment based on the obtained position coordinate information. When the object boundary similarity between object boundary OB1 and object boundary ob1 is lower than the first similarity threshold, the heat-generating object at that coordinate position is identified as a visible restricted object. The video fusion module is used to perform video fusion calculations on thermal images of visible objects in visible light video and infrared thermal imaging video to obtain fused video.

[0016] As a further aspect of the present invention: in the fused video, the image in the visible light video is used as the base image for display.

[0017] As a further aspect of the present invention: when the similarity between the object boundaries OB1 and ob1 is lower than a first similarity threshold and not lower than a second similarity threshold, the image of the visible restricted object in the infrared thermal imaging video is displayed in the fused video with a first transparency; when the similarity between the object boundaries OB1 and ob1 is lower than a second similarity threshold, the image of the visible restricted object in the infrared thermal imaging video is displayed in the fused video with a second transparency lower than the first transparency.

[0018] As a further aspect of the present invention: in response to receiving a vehicle headlight turn-on signal, the image of a restricted object visible in the infrared thermal imaging video is displayed in the fused video with a specified transparency.

[0019] As a further aspect of the present invention: in response to receiving the headlights on signal, the night assist mode is run, and the object similarity analysis module uses a third similarity threshold, which is different from the first similarity threshold, as the judgment standard; when the object boundary similarity between object boundary OB1 and object boundary ob1 is lower than the third similarity threshold, the heated object at that coordinate position is identified as a visible restricted object.

[0020] As a further aspect of the present invention: the third similarity threshold is higher than the first similarity threshold.

[0021] As a further aspect of the present invention: the vehicle light is an exterior lighting lamp.

[0022] As a further aspect of the present invention: the exterior lighting includes any one or any combination of low beam headlights, high beam headlights, side marker lights, front fog lights, and rear fog lights.

[0023] As a further aspect of the present invention: the vehicle headlight activation signal includes an activation signal for automatic vehicle headlight activation and / or a user-interactive input control signal for headlight activation.

[0024] As a further aspect of the present invention: the specified direction outside the vehicle includes the forward direction and / or the reverse direction.

[0025] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating the present invention; Figure 2 This is another schematic diagram of the process of the present invention; Figure 3 This is another flowchart of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Please see Figure 1-3 The vehicle infrared thermal imaging assistance system assists drivers in safe driving, including an on-board camera module, an object similarity analysis module, and a video fusion module.

[0030] The vehicle-mounted camera module is used to acquire visible light video and infrared thermal imaging video from a specified direction outside the vehicle. The vehicle-mounted camera module may have both an infrared thermal imaging camera unit and a visible light camera unit, each acquiring the corresponding type of video.

[0031] The object similarity analysis module determines the position of the current heat-generating object in the frame based on the infrared thermal imaging video, obtains the position coordinate information and object boundary OB1 of the heat-generating object, and confirms the object boundary ob1 of the corresponding coordinate position in the visible light video at the same moment based on the obtained position coordinate information. It compares the object boundary similarity between object boundary OB1 and object boundary ob1, and when the object boundary similarity between object boundary OB1 and object boundary ob1 is lower than the first similarity threshold, the heat-generating object at that coordinate position is identified as a visible restricted object.

[0032] The operations performed by the object similarity analysis module on infrared thermal imaging video and visible light video can be understood as extracting frame images from the acquired infrared thermal imaging video and visible light video respectively, obtaining infrared thermal imaging images and visible light images at the same time, and then performing corresponding operations on the infrared thermal imaging images and visible light images, such as determining the position of the current heated object in the infrared thermal imaging image, obtaining the position coordinate information and object boundary OB1 of the heated object, and confirming the object boundary ob1 of the object at the corresponding coordinate position in the visible light image at the same time based on the obtained position coordinate information.

[0033] The video fusion module performs video fusion calculations on thermal images of visible confined objects in visible light video and infrared thermal imaging video to obtain a fused video. Video fusion can be understood as fusing multiple frames of visible light images with the corresponding thermal images of visible confined objects, and then playing them sequentially to obtain a fused video.

[0034] It can be seen that the thermal image of a constrained object can be the thermal image defined by the object boundary OB1.

[0035] The object similarity analysis module automatically compares the similarity of object boundaries and determines whether to fuse the thermal image of the heated object with the visible light image based on a preset similarity threshold. When the similarity is high (not lower than the first similarity threshold), only the visible light image can be displayed to avoid affecting the driver's clear observation of details at the corresponding location. When the similarity is low, the thermal image of the heated object is fused onto the visible light image to help the driver promptly detect any hazards at the corresponding location and take timely obstacle avoidance or deceleration measures.

[0036] Preferably, in the fused video, the image in the visible light video is displayed as the base map.

[0037] More preferably, when the object boundary similarity between object boundary OB1 and object boundary ob1 is lower than a first similarity threshold and not lower than a second similarity threshold, the image of the visible restricted object in the infrared thermal imaging video is displayed in the fused video with a first transparency.

[0038] When the similarity between object boundary OB1 and object boundary ob1 is lower than the second similarity threshold, the image of the visible restricted object in the infrared thermal imaging video is displayed in the fused video with a second transparency lower than the first transparency.

[0039] The second similarity threshold is lower than the first similarity threshold.

[0040] When the similarity is closer to the first similarity threshold in the range below the first similarity threshold, it means that the heated object can be more easily found in the visible light image. When the similarity is significantly different from the first similarity threshold in the range below the first similarity threshold, it means that the heated object is more difficult to find in the visible light image. In this case, we distinguish between these situations. That is, after judging that the similarity is lower than the first similarity, we continue to judge whether the similarity is lower than the second similarity, so as to classify the similarity.

[0041] In the similarity grading where the heated object is more easily detected from the visible light image, the visibility of the image of the object with limited visibility is reduced in the fused image, allowing users to receive an alert while also better determining the actual outline of the object. In the similarity grading where the heated object is more difficult to detect from the visible light image, the influence of the image of the object with limited visibility in the fused image is increased, allowing users to detect potential dangers more promptly.

[0042] The specified directions outside the vehicle include the forward direction and / or the reverse direction.

[0043] When the specified direction outside the vehicle includes both forward and reverse directions, the system activates the front onboard camera module when the car is moving forward and the rear onboard camera module when the car is moving backward. The fused video from both directions can be displayed on a single screen, such as the central control display. The activation of the front and rear onboard camera modules can correspond to the car's current gear position; for example, the front camera module is activated when the car is in forward gear, and the rear camera module is activated when the car is in reverse gear.

[0044] Furthermore, in response to receiving a signal that the vehicle lights are on, the image of the restricted object visible in the infrared thermal imaging video is displayed in the fused video with a specified transparency.

[0045] Preferably, the vehicle light is an exterior lighting lamp.

[0046] Vehicle exterior lighting includes any one or any combination of low beam headlights, high beam headlights, side marker lights, front fog lights, and rear fog lights.

[0047] Vehicle light activation signals include automatic sensor-activated light activation signals and / or user-interactive input-controlled light activation signals.

[0048] In this embodiment, in response to receiving the headlights-on signal, the night assist mode is run, and the object similarity analysis module uses a third similarity threshold, which is different from the first similarity threshold, as the judgment criterion.

[0049] When the similarity between object boundary OB1 and object boundary ob1 is lower than the third similarity threshold, the heated object at that coordinate position is identified as a visible restricted object.

[0050] The third similarity threshold is higher than the first similarity threshold.

[0051] As a direct indication of poor ambient light, checking whether the headlights are on can better help the assistance system confirm the ambient light conditions.

[0052] After receiving the signal that the headlights are on, the judgment standard is changed from the first similarity threshold to a higher third similarity threshold, so that the assistance system can intervene earlier to provide a warning, avoiding the situation where the driver's night vision is not obvious and the thermal image of the heated object is difficult to find in the fused video.

[0053] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A vehicle infrared thermal imaging assistance system for assisting drivers in safe driving, characterized in that, include The vehicle-mounted camera module is used to acquire visible light video and infrared thermal imaging video from a specified direction outside the vehicle. The object similarity analysis module determines the position of the current heat-generating object in the frame based on the infrared thermal imaging video, obtains the position coordinate information and object boundary OB1 of the heat-generating object, and confirms the object boundary ob1 of the object at the corresponding coordinate position in the visible light video at the same moment based on the obtained position coordinate information. When the object boundary similarity between object boundary OB1 and object boundary ob1 is lower than the first similarity threshold, the heat-generating object at that coordinate position is identified as a visible restricted object. The video fusion module is used to perform video fusion calculations on thermal images of visible objects in visible light video and infrared thermal imaging video to obtain fused video.

2. The vehicle infrared thermal imaging assistance system for assisting drivers in safe driving according to claim 1, characterized in that, In the fused video, the image from the visible light video is used as the base map for display.

3. The vehicle infrared thermal imaging assistance system for assisting drivers in safe driving according to claim 2, characterized in that, When the similarity between object boundary OB1 and object boundary ob1 is lower than the first similarity threshold and not lower than the second similarity threshold, the image of the visible restricted object in the infrared thermal imaging video is displayed in the fused video with the first transparency. When the similarity between object boundary OB1 and object boundary ob1 is lower than the second similarity threshold, the image of the visible restricted object in the infrared thermal imaging video is displayed in the fused video with a second transparency lower than the first transparency.

4. The vehicle infrared thermal imaging assistance system for assisting drivers in safe driving according to claim 3, characterized in that, In response to receiving a signal that the vehicle lights are on, the image of a restricted object visible in the infrared thermal imaging video is displayed in the fused video with a specified transparency.

5. The vehicle infrared thermal imaging assistance system for assisting drivers in safe driving according to claim 1 or 4, characterized in that, In response to receiving the headlights-on signal, the night assist mode is activated, and the object similarity analysis module uses a third similarity threshold, which is different from the first similarity threshold, as the judgment criterion. When the similarity between object boundary OB1 and object boundary ob1 is lower than the third similarity threshold, the heated object at that coordinate position is identified as a visible restricted object.

6. The vehicle infrared thermal imaging assistance system for assisting drivers in safe driving according to claim 5, characterized in that, The third similarity threshold is higher than the first similarity threshold.

7. The vehicle infrared thermal imaging assistance system for assisting drivers in safe driving according to claim 4 or 6, characterized in that, The headlights mentioned are exterior lighting lights.

8. The vehicle infrared thermal imaging assistance system for assisting drivers in safe driving according to claim 7, characterized in that, The exterior lighting includes any one or any combination of low beam headlights, high beam headlights, side marker lights, front fog lights, and rear fog lights.

9. The vehicle infrared thermal imaging assistance system for assisting drivers in safe driving according to claim 7, characterized in that, Vehicle light activation signals include automatic sensor-activated light activation signals and / or user-interactive input-controlled light activation signals.

10. The vehicle infrared thermal imaging assistance system for assisting drivers in safe driving according to claim 1, 2, 3, 4, 6, 8, or 9, characterized in that, The specified directions outside the vehicle include the forward direction and / or the reverse direction.

Images