Systems and methods for low-visibility driving

Abstract

A method for low-visibility driving includes receiving image data from a visible light camera. The image data includes an image of an area in front of a vehicle. The method includes receiving sensor data from an object detection sensor. The object detection sensor is configured to detect an object in front of the vehicle. The sensor data includes information about the object in front of the vehicle. The method also includes detecting the object in front of the vehicle using the sensor data received from the object detection sensor and determining whether the object in front of the vehicle detected by the object detection sensor is not detectable by the visible light camera. The method also includes commanding a display to generate a virtual image using the sensor data to identify the object in front of the vehicle.

Description

Technical Field

[0001] This disclosure relates to systems and methods for driving in low visibility conditions. Background Technology

[0002] This introduction provides an overall overview of the background of this disclosure. To the extent described in this introduction, the work of the currently named inventors, and aspects of the description that may not conform to the prior art at the time of submission, are neither explicitly nor implicitly acknowledged as prior art to this disclosure.

[0003] Sometimes, vehicles travel in low-visibility conditions (e.g., fog, rain, snow, and night). When driving in low-visibility conditions, a vehicle's visible light cameras may not detect objects in front of the vehicle, such as animals or pedestrians. However, it is desirable to detect these objects. Therefore, it is useful to develop a method and system for displaying virtual images to highlight the presence and location of objects (e.g., animals or pedestrians) that are not obvious to vehicle users in low-visibility conditions (e.g., fog, rain, snow, and night). Summary of the Invention

[0004] This disclosure describes a method and system for displaying virtual images to highlight the presence and location of objects (e.g., animals or pedestrians) that are not readily apparent to a vehicle user under low visibility conditions (e.g., fog, rain, snow, and night). In one aspect of this disclosure, a method for driving in low visibility conditions includes receiving image data from a visible light camera. The image data includes an image of a region in front of the vehicle. The method includes receiving sensor data from an object detection sensor. The object detection sensor is configured to detect objects in front of the vehicle. The sensor data includes information about the objects in front of the vehicle. The method includes using the sensor data received from the object detection sensor to detect objects in front of the vehicle and determining whether the visible light camera cannot detect objects in front of the vehicle that are detected by the object detection sensor. The method also includes, in response to determining that the visible light camera cannot detect objects in front of the vehicle that are detected by the object detection sensor, instructing a display to generate a virtual image using the sensor data to identify the objects in front of the vehicle.

[0005] In one aspect of this disclosure, determining whether a visible light camera cannot detect an object in front of a vehicle includes determining a confidence level of an object detected by the visible light camera and detected by an object detection sensor in front of the vehicle; comparing the confidence level with a predetermined threshold to determine whether the confidence level is equal to or less than the predetermined threshold; and, in response to determining that the confidence level is equal to or less than the predetermined threshold, instructing a display to generate a virtual image using sensor data received from the object detection sensor to identify the object in front of the vehicle.

[0006] In one aspect of this disclosure, the method further includes preventing the generation of a virtual image for identifying objects in front of the vehicle in response to determining that the confidence level is greater than the predetermined threshold.

[0007] In one aspect of this disclosure, the object detection sensor is a thermal imaging camera configured to generate a thermal image of a region in front of the vehicle. The method also includes cropping the thermal image of the region in front of the vehicle to generate a cropped thermal image. The method further includes instructing a display to present the cropped thermal image. The cropped thermal image includes only thermal images of objects in front of the vehicle to highlight the location of those objects.

[0008] In one aspect of this disclosure, the virtual image includes a rectangle surrounding the entire cropped thermal image to highlight the location of objects in front of the vehicle.

[0009] In one aspect of this disclosure, the virtual image also includes an arrow adjacent to the rectangle to indicate the direction of movement of an object in front of the vehicle.

[0010] In one aspect of this disclosure, the object detection sensor is a night vision device configured to generate a night view image of a region in front of a vehicle. The method also includes cropping the night view image of the region in front of the vehicle to generate a cropped night view image. The method further includes instructing a display to present the cropped night view image. The cropped night view image includes only night view images of objects in front of the vehicle to highlight the position of the objects. A virtual image includes a rectangle surrounding the entire cropped thermal image. The virtual image includes arrows adjacent to the rectangle to indicate the direction of movement of the objects in front of the vehicle.

[0011] In one aspect of this disclosure, the method also includes determining the eye position of the vehicle operator, determining the position of an object in front of the vehicle, and determining the position of a virtual image based on the eye position of the vehicle operator and the position of the object in front of the vehicle.

[0012] In one aspect of this disclosure, commanding the display to generate a virtual image to identify objects in front of the vehicle includes generating the virtual image at the location of a previously determined virtual image.

[0013] In one aspect of this disclosure, the display is an augmented reality head-up display.

[0014] In one aspect of this disclosure, the display is part of a hybrid augmented reality (AR) head-up display (HUD) system. The vehicle includes a windshield. The windshield includes a polyvinyl butyral layer and an RGB phosphor embedded in the polyvinyl butyral layer. The hybrid AR HUD system includes a projector configured to emit a laser toward the windshield to cause the RGB phosphor to fluoresce.

[0015] In one aspect of this disclosure, a vehicle includes a display and multiple sensors. The multiple sensors include a visible light camera and an object detection sensor. The object detection sensor is configured to detect objects in front of the vehicle. The vehicle includes a controller that communicates with the multiple sensors and the display. The controller is configured to perform the methods described above.

[0016] Other areas of application of this disclosure will become apparent from the detailed description provided below. It should be understood that the specific embodiments and examples are for illustrative purposes only and are not intended to limit the scope of this disclosure.

[0017] The above-described features and advantages, as well as other features and advantages, of the currently disclosed systems and methods will become apparent from the detailed description, including the claims and exemplary embodiments, when taken in conjunction with the accompanying drawings. Attached Figure Description

[0018] This disclosure will be more fully understood from the detailed description and accompanying drawings, in which:

[0019] Figure 1 This is a block diagram depicting an embodiment of a vehicle including a system for driving in low visibility conditions;

[0020] Figure 2 yes Figure 1 A schematic front view of the vehicle's multi-focal-plane augmented reality display, highlighting Figure 1 The second image plane of the vehicle's multi-focal plane augmented reality display;

[0021] Figure 3 yes Figure 1 A schematic diagram of the second image plane of a multi-focal plane augmented reality display;

[0022] Figure 4 It is used in Figure 1 A schematic diagram of a part of a system that displays information on a vehicle's multi-focal-plane augmented reality display;

[0023] Figure 5 yes Figure 1 A schematic diagram of a vehicle's hybrid augmented reality display system;

[0024] Figure 6 yes Figure 5 A schematic side view of the windshield of a hybrid augmented reality display system;

[0025] Figure 7 yes Figure 1 A schematic front view of a portion of the vehicle, showing a virtual image displayed on the vehicle's monitor; and

[0026] Figure 8 This is a flowchart of a method for driving in low visibility conditions. Detailed Implementation

[0027] Reference will now be made in detail to several examples of this disclosure illustrated in the accompanying drawings. Wherever possible, the same or similar reference numerals are used in the drawings and description to refer to the same or similar parts or steps.

[0028] refer to Figure 1 The vehicle 10 generally includes a chassis 12, a body 14, front wheels, and rear wheels 17, and may be referred to as a vehicle system. In the depicted embodiment, the vehicle 10 includes two front wheels 17a and two rear wheels 17b. The body 14 is disposed on the chassis 12 and substantially encloses the components of the vehicle 10. The body 14 and the chassis 12 may together form a frame. The wheels 17 are rotatably coupled to the chassis 12 near corresponding corners of the body 14. The vehicle 10 includes a front axle 19 coupled to the front wheels 17a and a rear axle 25 coupled to the rear wheels 17b.

[0029] In various embodiments, vehicle 10 may be an autonomous vehicle and control system 98 may be integrated into vehicle 10. Control system 98 may be referred to as a system or a low-visibility driving system using one or more displays 29 (such as multi-focal plane augmented reality displays). Vehicle 10 is, for example, a vehicle that is automatically controlled to transport passengers from one location to another. In the illustrated embodiment, vehicle 10 is described as a pickup truck, but it should be understood that other vehicles may also be used, including trucks, sedans, coupes, sport utility vehicles (SUVs), recreational vehicles (RVs), etc. In one embodiment, vehicle 10 may be a so-called Level 2, Level 3, Level 4, or Level 5 automation system. Level 4 system means “high automation”, referring to the performance of the autonomous driving system in a specific driving mode for dynamic driving tasks, even if the human driver does not respond appropriately to intervention requests. Level 5 system means “full automation”, referring to the full-time performance of the autonomous driving system for dynamic driving tasks under many road and environmental conditions that a human driver can manage. In Level 3 vehicles, the vehicle system performs the entire Dynamic Driving Task (DDT) within the area it is designed to do so. If a malfunction occurs or the vehicle is about to leave its operational area, vehicle 10 essentially "requires" the driver to take over; only then is the vehicle operator expected to be responsible for DDT evacuation. In Level 2 vehicles, the system provides steering, braking / acceleration support, lane centering, and adaptive cruise control. However, even when these systems are activated, the driving vehicle operator must still drive and continuously monitor the automated features.

[0030] As shown in the figure, vehicle 10 generally includes a propulsion system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, at least one controller 34, and a communication system 36. In various embodiments, the propulsion system 20 may include an electric motor, such as a traction motor, and / or a fuel cell propulsion system. Vehicle 10 may also include a battery (or battery pack) 21 electrically connected to the propulsion system 20. Thus, the battery 21 is configured to store electrical energy and supply electrical energy to the propulsion system 20. In some embodiments, the propulsion system 20 may include an internal combustion engine. The transmission system 22 is configured to transmit power from the propulsion system 20 to the wheels 17 according to a selectable speed ratio. According to various embodiments, the transmission system 22 may include a stepped automatic transmission, a continuously variable transmission (CVT), or other suitable transmission. The braking system 26 is configured to provide braking torque to the wheels 17. In various embodiments, the braking system 26 may include a friction brake, a brake-by-wire brake, a regenerative braking system (such as an electric motor), and / or other suitable braking systems. The steering system 24 affects the orientation of the wheels 17 and may include a steering wheel 33. Although depicted as including a steering wheel 33 for illustrative purposes, in some embodiments contemplated within the scope of this disclosure, the steering system 24 may not include a steering wheel 33.

[0031] Sensor system 28 includes one or more sensors 40 (i.e., sensing devices) that sense observable conditions of the external and / or internal environment of vehicle 10. Sensors 40 communicate with controller 34 and may include, but are not limited to, one or more radars, one or more lidar sensors, one or more proximity sensors, one or more odometers, one or more ground-penetrating radar (GPR) sensors, one or more steering angle sensors, Global Navigation Satellite System (GNSS) transceivers (e.g., one or more Global Positioning System (GPS) transceivers), one or more tire pressure sensors, one or more front-view visible light cameras 41, one or more gyroscopes, one or more accelerometers, one or more inclinometers, one or more speed sensors, one or more ultrasonic sensors, one or more inertial measurement units (IMUs), and / or other sensors. Each sensor 40 is configured to generate a signal indicating the sensed observable conditions of the external and / or internal environment of vehicle 10. Because sensor system 28 provides data to controller 34, sensor system 28 and its sensors 40 are considered a source of information (or simply a source).

[0032] Visible light camera 41 is configured to collect visible light and convert it into electrical signals. Visible light camera 41 then organizes the information from the electrical signals to render images and video streams. Visible light camera 41 utilizes light with wavelengths from 380 nanometers to 700 nanometers (the same spectrum perceived by the human eye) by capturing light of red, green, and blue wavelengths (RGB) and creating images that replicate human vision. Therefore, in this disclosure, the term "visible light camera" refers to a camera configured to collect visible light and generate images and / or video streams from the collected visible light. The term "visible light" refers to light with wavelengths from 380 nanometers to 700 nanometers. In some embodiments, at least one of the visible light cameras 41 is a forward-looking camera configured to capture images of the area in front of vehicle 10.

[0033] Sensor system 28 also includes one or more object detection sensors 45 configured to detect objects 47 in front of vehicle 10. The term "object detection sensor" refers to a sensor specifically designed for detecting objects (such as pedestrians and animals) in front of vehicle 10 and does not include a visible light camera. Therefore, the term "object detection sensor" does not refer to a visible light camera. Object detection sensor 45 is configured to send sensor data to controller 34. The sensor data generated by object detection sensor 45 includes information about the object 47 in front of vehicle 10. For example, the sensor data may include information about the size, shape, and direction of movement of the object 47 in front of vehicle 10. As a non-limiting example, object detection sensor 45 may be or include one or more thermal imaging cameras, one or more night vision devices, one or more radars, and / or one or more lidars. The thermal imaging camera is configured to generate a thermal image of the area in front of vehicle 10. The night vision device is configured to generate a night vision image of the area in front of vehicle 10. In this disclosure, the term "night vision device" means "an optoelectronic device that allows the generation of a night vision image at near-complete darkness light levels." The term "night vision image" refers to an image generated by a night vision device 45 and converted from both visible and near-infrared light into visible light. Near-infrared light refers to light with wavelengths from 800 nanometers to 2500 nanometers.

[0034] The actuator system 30 includes one or more actuator devices 42 that control one or more vehicle features, such as, but not limited to, the propulsion system 20, the transmission system 22, the steering system 24, and the braking system 26. In various embodiments, the vehicle features may also include interior and / or exterior vehicle features, such as, but not limited to, doors, trunk, and cabin features such as air, music, and lighting.

[0035] Data storage device 32 stores data for automatically controlling vehicle 10. In various embodiments, data storage device 32 stores a defined map of the navigable environment. In various embodiments, the defined map may be predefined by and obtained from a remote system. For example, the defined map may be assembled by a remote system and transmitted to vehicle 10 (wirelessly and / or via wire) and stored in data storage device 32. Data storage device 32 may be part of controller 34, separate from controller 34, or part of controller 34 and a separate system.

[0036] Vehicle 10 may also include one or more airbags 35 communicating with controller 34 or another controller of vehicle 10. The airbags 35 include inflatable airbags and are configured to switch between a retracted configuration and a deployed configuration to cushion the effects of external forces applied to vehicle 10. Sensors 40 may include airbag sensors (such as IMUs) configured to detect external forces and generate signals representing the magnitude of such forces. Controller 34 is configured to command the airbags 35 to deploy based on signals from one or more sensors 40 (such as airbag sensors). Therefore, controller 34 is configured to determine when the airbags 35 have deployed.

[0037] The controller 34 includes at least one processor 44 and a non-transitory computer-readable storage device or medium 46. The processor 44 may be a custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among several processors associated with the controller 34, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, a combination thereof, or generally a device for executing instructions. The computer-readable storage device or medium 46 may include volatile and non-volatile storage, such as read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is persistent or non-volatile memory that can be used to store various operational variables when the processor 44 is powered off. The computer-readable storage device or medium 46 may be implemented using multiple storage devices, such as PROM (programmable read-only memory), EPROM (electrical PROM), EEPROM (electrically erasable PROM), flash memory, or other electrical, magnetic, optical, or combined storage devices capable of storing data, some of which represent executable instructions used by the controller 34 in controlling the vehicle 10. The controller 34 of vehicle 10 can be referred to as a vehicle controller and can be programmed to execute method 100. Figure 8 ), as described in detail below.

[0038] The instructions may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions. When executed by processor 44, the instructions receive and process signals from sensor system 28, execute logic, calculations, methods, and / or algorithms for automatically controlling components of vehicle 10, and generate control signals to actuator system 30 to automatically control components of vehicle 10 based on logic, calculations, methods, and / or algorithms. Although Figure 1 A single controller 34 is shown, but embodiments of vehicle 10 may include multiple controllers 34 that communicate via suitable communication media or combinations thereof and cooperate to process sensor signals, execute logic, calculations, methods and / or algorithms, and generate control signals to automatically control the features of vehicle 10. In various embodiments, one or more instructions of controller 34 are implemented in control system 98.

[0039] Vehicle 10 includes a user interface 23, which may be a touchscreen in the dashboard. User interface 23 may include, but is not limited to, alarms, one or more speakers 27 to provide audible sound, haptic feedback from vehicle seats or other objects, one or more displays 29, one or more microphones 31, and / or other devices suitable for providing notifications to vehicle users of vehicle 10. User interface 23 is in electrical communication with controller 34 and is configured to receive input from users (e.g., vehicle operators or vehicle passengers). For example, user interface 23 may include a touchscreen and / or buttons configured to receive input from vehicle user 11 (e.g., vehicle operator). Therefore, controller 34 is configured to receive input from users via user interface 23.

[0040] Vehicle 10 may include one or more displays 29 configured to display information to vehicle user 11 (e.g., vehicle operator or passenger) and may be an augmented reality (AR) display or a hybrid AR display. In this disclosure, the term "AR display" refers to a display that presents information to a user while still allowing the user to see the outside world. In some embodiments, display 29 may be configured as a head-up display (HUD) and / or a full-windshield display. Thus, display 29 may be an AR HUD or a full-windshield display. In an AR HUD, an image is projected onto the windshield 39 of vehicle 10. As described below, display 29 may be a multifocal plane AR display to facilitate manipulation of the virtual image 50 (e.g., size, position, and type).

[0041] Communication system 36 communicates with controller 34 and is configured to wirelessly communicate with other remote vehicles 48, such as, but not limited to, other vehicles (“V2V” communication), infrastructure (“V2I” communication), remote systems at remote call centers (e.g., General Motors’ ON-STAR), and / or personal electronic devices, such as mobile phones. In this disclosure, the term “remote vehicle” refers to a vehicle, such as an automobile, configured to transmit one or more signals to vehicle 10 without a physical connection to vehicle 10. In some embodiments, communication system 36 is a wireless communication system configured to communicate using the IEEE 802.11 standard or via a wireless local area network (WLAN) using cellular data communication. However, additional or alternative communication methods, such as Dedicated Short Range Communication (DSRC) channels, are also considered within the scope of this disclosure. A DSRC channel refers to a one-way or two-way short-to-medium range wireless communication channel specifically designed for automotive applications, along with a corresponding set of protocols and standards. Therefore, communication system 36 may include one or more antennas and / or communication transceivers 37 for receiving and / or transmitting signals, such as Cooperative Sensing Messages (CSM). Communication transceivers 37 may be considered as sensors 40. The communication system 36 is configured to wirelessly transmit information between vehicle 10 and another vehicle. Furthermore, the communication system 36 is configured to wirelessly transmit information between vehicle 10 and infrastructure or other vehicles.

[0042] refer to Figure 2 and Figure 3 The display 29 can be a multi-focal plane AR display as described above. In this case, the display 29 has a first image plane 58 and a second image plane 60. The first image plane 58 displays a view of the external world, and the second image plane 60 is for displaying a virtual image 50. Figure 7 The second image plane 60 spans multiple lanes, and the virtual image 50 appears at a position on the road surface 62 that is farther away than the first image plane 58. For example, as... Figure 3 As shown, the second image plane 60 covers the left lane 52, the center lane 54, and the right lane 56. As a non-limiting example, in the center lane 54, the second image plane 60 begins at a first predetermined distance D1 (e.g., 25 meters) from the vehicle 10 and ends at a second predetermined distance D2 (e.g., 90 meters) from the vehicle 10. Regardless of the specific distance, the second predetermined distance D2 is greater than the first predetermined distance D1 to help the vehicle user 11 see the virtual image 50. Figure 7In left lane 52 and right lane 56, a second image plane 60 is defined by an inclined boundary that begins at a first predetermined distance D1 from vehicle 10 and ends at a third predetermined distance D3 (e.g., 50 meters) from vehicle 10. The third predetermined distance D3 is greater than the first predetermined distance D1 and less than the second predetermined distance D2 to assist vehicle user 11. Figure 6 )See virtual image 50 ( Figure 7 As used herein, the term "multi-focal plane AR display" refers to an AR display that presents images in more than one image plane, wherein the image planes are located in different positions. It is desirable to use a multi-focal plane AR display in the currently disclosed system 98 to easily change the size, type, and / or position of the virtual image 50 relative to the view of the external world.

[0043] refer to Figure 4 System 98 includes a user tracker 43 (e.g., an eye tracker and / or a head tracker) configured to track the position and movement of the eyes 66 and / or head 69 of vehicle user 11. In the depicted embodiment, user tracker 43 may be configured as one or more cameras 41 of vehicle 10. As described above, camera 41 is considered as sensor 40 of vehicle 10. As sensor 40, user tracker 43 communicates with controller 34, which includes system manager 68. During operation of system 98, system manager 68 receives at least a first input 70 and a second input 72. The first input 70 represents the vehicle's position in space (i.e., the vehicle's location in space), and the second input 72 represents the position of the vehicle user in vehicle 10 (e.g., the position of the eyes 66 and / or head 69 of vehicle user 11 in vehicle 10). The first input 70 may include data such as GNSS data (e.g., GPS data), vehicle speed, road curvature, and vehicle steering. This data can be collected from sensors 40 of vehicle 10 and / or other remote vehicles 48 via vehicle 10's communication system 36. The second input 72 can be received from a user tracker (e.g., an eye tracker and / or a head tracker). System manager 68 is configured to determine (e.g., calculate) the type, size, shape, and color of the conformal graphic (i.e., virtual image 50) based on the first input 70 (i.e., the vehicle's position in space), the second input 72 (e.g., the position of the user's eyes and / or head in vehicle 10), and the sensed vehicle driving environment (which can be obtained via sensors 40). The type, size, shape, and color of the conformal graphic of virtual image 50 can be collectively referred to as virtual image features.

[0044] Continue to refer to Figure 4The system 98 also includes an image engine 74. The image engine 74 is part of the display 29 and may be an integrated circuit configured to generate virtual images 50. These generated virtual images 50 are then projected onto the windshield 39 (if the display 29 is a HUD) to display the virtual images 50 on a second image plane 60 along the road surface 62.

[0045] refer to Figure 5 and Figure 6 The display 29 may be part of a hybrid AR HUD system 49. In the hybrid AR HUD system 49, the display 29 is configured as an AR HUD and projects images onto the windshield 39 of the vehicle 10. As described below, the display 29 may be a multifocal plane AR display to facilitate the processing of the virtual image 50 (e.g., size, position, and type). In addition to the AR HUD, the hybrid AR HUD system 49 includes an RGB phosphor 53 embedded in the windshield 39. The windshield 39 may be made wholly or partially of polyvinyl butyral, and the RGB phosphor 53 may be embedded at random locations within the windshield 39. Therefore, the windshield 39 may include a polyvinyl butyral layer 55 and RGB phosphors 53 embedded in the polyvinyl butyral layer 55. The hybrid AR HUD system 49 may also include a projector 51, such as a microprojector, attached to the roof of the vehicle 10. Projector 51 is configured to emit a violet or ultraviolet laser to excite an RGB phosphor 53 embedded in a polyvinyl butyral layer 55 of the windshield 39. Therefore, the laser projected from projector 51 is specifically designed to excite the RGB phosphor 53 embedded in the polyvinyl butyral layer 55 of the windshield 39. Thus, laser-induced fluorescence will appear at each irradiated point on the windshield 39. Projector 51 can have a wide projection angle. Photons are generated at the phosphor location, and therefore, the image distance is at the windshield plane. The image generated by the AR HUD display 29 and the excited RGB phosphor may overlap.

[0046] refer to Figure 1 and Figure 7System 98 is configured to command display 29 (such as an AR HUD) to present virtual image 50 to detect the presence of one or more objects 47 (e.g., animals or pedestrians) and identify their locations, which are not apparent to vehicle user 11 under low visibility conditions (e.g., fog, rain, snow, and night). In doing so, display 29 displays information in a contextual manner by enhancing the road scene with conformal graphics. In this disclosure, the term "conformal graphics" refers to synthetically generated content (i.e., virtual image 50) presented as part of the external world. Therefore, display 29 is a conformal display. In this disclosure, the term "conformal display" refers to a display capable of presenting synthetically generated content (i.e., one or more virtual images 50) as part of the external world. Based on real-time vehicle sensing from sensor 40, display 29 uses display 29 (e.g., an AR HUD) to present to vehicle user 11 visual indications (i.e., virtual image 50) of the presence and real-time location of object 47 (e.g., pedestrians and / or animals). The system 98 of vehicle 10 uses one or more object detection sensors 45 and one or more visible light cameras 41 to perceive the road ahead. Sensor data from the object detection sensors 45 (e.g., night vision devices, thermal imaging sensors, lidar, and / or radar) is used to determine the three-dimensional position of the detected object 47 (e.g., pedestrians and / or animals). Furthermore, the sensor data is used in real time to plot highlighted areas (e.g., virtual image 50) to indicate the real-time position of the object 47 in the driving scene.

[0047] Continue to refer to Figure 1 and Figure 7Computer vision and machine learning are used to classify objects 47 in front of vehicle 10. When object detection sensor 45 detects an object 47 in front of vehicle 10 that was not detected by visible light camera 41, system 98 instructs display 29 to present a graphical indication (i.e., virtual image 50) at the actual location in the road scene, as well as a cropped and integrated live video presentation of object 47 (i.e., cropped live image 76). For example, thermal video images and / or night vision videos are cropped to contain only the identified object 47. The video images can be colored (e.g., red, yellow, green, etc.) to indicate the criticality of the path of object 47 that is consistent with the vehicle path. The cropped video image is then projected by display 29 (e.g., ARHUD) and highlighted with graphics (i.e., virtual image 50). By using virtual image 50, system 98 provides enhanced situational awareness of object 47, which is not obvious under low visibility conditions. As a non-limiting example, the virtual image 50 may include a rectangle 78 surrounding the entire cropped image 76 of the object 47 and an arrow 80 adjacent to the rectangle 78 to highlight the object 47. The rectangle 78 highlights the cropped image 76 of the object 47, and the arrow 80 indicates the direction of movement of the object 47 in front of the vehicle 10. In other words, graphical elements (such as the arrow 80 and / or color) are used to provide additional information about the speed, trajectory, and key properties of the detected object 47. In the vehicle 10 with the hybrid AR HUD system 49, when the object 47 is outside the field of view of the display 29 (AR HUD), the bounding box (i.e., rectangle 78) is displayed as an uncropped live image 76. In the vehicle 10 with the hybrid AR HUD system 49, when the object 47 is within the field of view of the display 29 (AR HUD) used for object recognition, the bounding box (i.e., rectangle 78) and the real-time cropped video image 76 are displayed.

[0048] Figure 8 This is a flowchart of a method 100 for low-visibility driving using a display 29 (such as a multi-focal plane augmented reality display and / or a hybrid AR HUD system 49). Method 100 begins at box 102. At box 102, the controller 34 uses signals generated, for example, by sensors 40, to determine that the vehicle 10 is being driven. For example, the controller 34 may receive data from one of the sensors 40 (such as a speed sensor) to determine that the vehicle 10 is moving. Method 100 then proceeds to box 104.

[0049] At box 104, object detection sensor 45 (e.g., night vision device, thermal imaging sensor, lidar and / or radar) and visible light camera 41 scan the road 61 in front of vehicle 10. Figure 7In response, the object detection sensor 45 and the forward-looking visible light camera 41 respectively send sensor data and image data to the controller 34. The image data includes one or more images of the road 61 in front of the vehicle 10 and may include information about one or more objects 47 in front of the vehicle 10. Therefore, the controller 34 receives image data from the visible light camera 41 and sensor data from the object detection sensor 45. The sensor data includes information about the objects 47 in front of the vehicle 10, such as the position, trajectory, heading, and direction of movement of the objects 47. Method 100 then proceeds to block 106.

[0050] At box 106, controller 34 uses sensor data from one or more object detection sensors 45 and / or image data from visible light camera 41 to identify (i.e., detect) and classify one or more objects of interest 47 in front of vehicle 10. Computer vision can be used to identify and classify one or more objects in front of vehicle 10. Machine learning can be used to classify the objects 47 in front of vehicle 10. Next, method 100 proceeds to box 108.

[0051] At block 108, controller 34 determines whether one or more visible light cameras 41 cannot detect an object 47 in front of vehicle 10 detected by one or more object detection sensors 45. If one or more visible light cameras 41 detect the object 47 in front of vehicle 10 detected by one or more object detection sensors 45, method 100 proceeds to block 110. To determine whether one or more visible light cameras 41 cannot detect the object 47 in front of vehicle 10 detected by one or more object detection sensors 45, controller 34 first determines the confidence level at which the visible light cameras detect the object 47 in front of vehicle 10 detected by one or more object detection sensors 45. Then, controller 34 compares this confidence level to a predetermined threshold, which can be determined by testing. If the confidence level at which the visible light cameras detect the object 47 in front of vehicle 10 detected by one or more object detection sensors 45 is greater than the predetermined threshold, controller 34 determines that the visible light cameras 41 are capable of detecting the object 47 in front of vehicle 10 detected by one or more object detection sensors 45. In response to determining that the visible light camera 41 is capable of detecting an object 47 in front of the vehicle 10 detected by one or more object detection sensors 45, method 100 continues to block 110.

[0052] At box 110, controller 34 does not perform any action. Therefore, display 29 prevents the generation of virtual image 50, which identifies and highlights object 47 detected by object detection sensor 45. After box 110, method 100 returns to box 104.

[0053] At block 108, if the visible light camera detects an object 47 in front of the vehicle 10 detected by one or more object detection sensors 45 with a confidence level equal to or less than a predetermined threshold, the controller 34 determines that the visible light camera 41 cannot detect the object 47 in front of the vehicle 10 detected by one or more object detection sensors 45, and method 100 proceeds to block 112.

[0054] At box 112, controller 34 determines the appropriate highlight and virtual image 50 to be presented on display 29. In other words, controller 34 determines one or more virtual images 50 to be displayed based on the movement, position, headway, and trajectory of object 47, as well as the position of vehicle 10 relative to object 47. For example, if object 47 in front of vehicle 10 is moving, controller 34 can determine the direction of movement and determine that an arrow 80 should be added to the adjacent rectangle 78 to indicate the direction of movement of object 47. Next, method 100 continues to box 114.

[0055] At block 114, controller 34 uses at least one input from user tracker 43 to determine in real time the position of the user's eyes 66 and / or head 69 in vehicle 10. As described above, user tracker 43 may be a camera 41 configured to track the movement of the user's head 69 and / or eyes 66. Then, controller 34 uses the input from user tracker 43 to continuously determine the position of the user's eyes 66 and / or head 69 in real time. Furthermore, at block 114, controller 34 uses sensor data to determine in real time the position of an object 47 detected by object detection sensor 45. Then, method 100 proceeds to block 116.

[0056] At box 116, controller 34 determines in real time the position, type, size, shape, and color of the virtual image 50 to be displayed on display 29 (e.g., a multifocal plane AR HUD display or hybrid AR HUD system 49) based on the position of the vehicle user 11's eyes 66 and / or head 69 and / or the position of vehicle 10 relative to object 47. As a non-limiting example, the position of the virtual image 50 in display 29 may change as the vehicle user 11 moves their head 69 and / or eyes 66. Furthermore, the size of the virtual image 50 may increase as object 47 moves closer to vehicle 10. Next, method 100 proceeds to box 118.

[0057] At frame 118, display 29 (e.g., AR HUD or full-windshield display) generates a virtual image 50 (e.g., a graphic) and a cropped live image of object 47 on windshield 39. As described above, object detection sensor 45 can be, for example, a night vision device and / or a thermal imaging sensor (e.g., a thermal imaging camera). The thermal imaging sensor can be a far-infrared (FIR) sensor. Far-infrared light refers to light with wavelengths from 8 micrometers to 14 micrometers (8k–14knm). Alternatively, the thermal imaging sensor can be a near-infrared (NIR) sensor, a short-wave (SWIR) sensor, and / or a mid-wave infrared (MWIR) sensor. NIR and SWIR sensors are used in conjunction with illumination. Object detection sensor 45 can generate a night view image or thermal image of the area in front of vehicle 10 (including object 47). Subsequently, controller 34 can crop the night view image and / or thermal image of the area in front of vehicle 10 to generate a cropped image 76 of the area in front of vehicle 10. This cropped image (i.e., the cropped night view image and / or the cropped thermal image) includes only an image of the object 47 in front of the vehicle 10 generated using the object detection sensor 45. As described above, the controller 34 also generates a virtual image 50. As a non-limiting example, the virtual image 50 may include a bounding box (i.e., rectangle 78) surrounding the entire cropped image and arrows 80 adjacent to rectangle 78. Arrows 80 indicate the direction of movement of the object 47 (e.g., a pedestrian and / or animal). The controller 34 then commands the display 29 to present the virtual image 50 on the windshield 39 at the location previously determined at frame 116 to identify and highlight the object 47 in front of the vehicle 10. Method 100 then proceeds to frame 120.

[0058] At box 120, controller 34 compares the confidence level of an object 47 in front of vehicle 10 detected by visible light camera 41 and detected by one or more object detection sensors 45. If the confidence level of the object 47 in front of vehicle 10 detected by visible light camera 41 and detected by one or more object detection sensors 45 is greater than a predetermined threshold, method 100 continues to box 122. At box 122, controller 34 commands display 29 to stop displaying virtual image 50 and cropped image 76. Then, method 100 returns to box 104. If the confidence level of the object 47 in front of vehicle 10 detected by visible light camera 41 and detected by one or more object detection sensors 45 is equal to or less than the predetermined threshold, method 100 returns to box 112.

[0059] While exemplary embodiments have been described above, this does not mean that these embodiments describe all possible forms contained in the claims. The language used in this specification is descriptive and not restrictive. It should be understood that various changes may be made without departing from the spirit and scope of this disclosure. As previously stated, features of different embodiments may be combined to form other embodiments of the currently disclosed systems and methods that may not be explicitly described or illustrated. Although various embodiments may have been described as providing an advantage or superiority over other embodiments or prior art implementations in one or more desired characteristics, those skilled in the art will recognize that one or more features or characteristics may be compromised to achieve desired overall system properties, depending on the particular application and implementation. These properties may include, but are not limited to, cost, strength, durability, lifecycle cost, merchantability, appearance, packaging, size, suitability, weight, manufacturability, ease of assembly, etc. Therefore, embodiments described as less desirable than other embodiments or prior art implementations in one or more features are not outside the scope of this disclosure and may be ideal for a particular application.

[0060] These figures are simplified and not to exact scale. For convenience and clarity only, directional terms such as top, bottom, left, right, upper, above, above, below, under, rear, and front may be used in the figures. These and similar directional terms should not be construed as limiting the scope of this disclosure in any way.

[0061] This document describes embodiments of the present disclosure. However, it should be understood that the disclosed embodiments are merely examples and other embodiments may take various alternative forms. The drawings are not necessarily drawn to scale; some features may be enlarged or reduced to show details of specific components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for teaching those skilled in the art to employ the currently disclosed systems and methods in different ways. As those skilled in the art will understand, various features illustrated and described with reference to any of the drawings may be combined with features illustrated in one or more other drawings to produce embodiments not explicitly stated or described. The combinations of features shown provide representative embodiments of typical applications. However, for a particular application or implementation, various combinations and modifications of features consistent with the teachings of this disclosure may be required.

[0062] This document describes embodiments of the present disclosure based on functional and / or logical block components and various processing steps. It should be understood that such block components may be implemented by multiple hardware, software, and / or firmware components configured to perform specified functions. For example, embodiments of the present disclosure may employ a variety of integrated circuit components (e.g., memory elements, digital signal processing elements, logic elements, lookup tables, etc.) that can perform multiple functions under the control of one or more microprocessors or other control devices. Furthermore, those skilled in the art will understand that embodiments of the present disclosure can be practiced in conjunction with multiple systems and the systems described herein are merely exemplary embodiments of the present disclosure.

[0063] For the sake of brevity, techniques related to signal processing, data fusion, signaling, control, and other functional aspects of the system (as well as the various operating components of the system) may not be described in detail herein. Furthermore, the connecting lines shown in the various figures included herein are intended to represent example functional relationships and / or physical connections between different elements. It should be noted that alternative or additional functional relationships or physical connections may exist in the embodiments of this disclosure.

[0064] This description is illustrative in nature and is by no means intended to limit this disclosure, its application, or its use. The broad teachings of this disclosure can be implemented in many forms. Therefore, although this disclosure includes specific examples, its true scope should not be so limited, as other modifications will become apparent upon examination of the drawings, description, and appended claims.

Claims

1. A method for driving in low visibility conditions, comprising: Image data is received from a visible light camera, wherein the image data includes an image of the area in front of the vehicle; Sensor data is received from an object detection sensor, wherein the object detection sensor is configured to detect objects in front of the vehicle, and the sensor data includes information about the objects in front of the vehicle. The object in front of the vehicle is detected using the sensor data received from the object detection sensor; Determine whether the visible light camera cannot detect the object in front of the vehicle, the object being detected by the object detection sensor; as well as In response to determining that the visible light camera cannot detect the object detected by the object detection sensor in front of the vehicle, the vehicle's display is instructed to generate a virtual image using the sensor data to identify the object in front of the vehicle. The display is a multi-focal-plane augmented reality (AR) display of the vehicle, having a first image plane and a second image plane located at different positions. The first image plane displays a view of the external world, and the second image plane is prepared for displaying the virtual image. As the vehicle travels along a road surface, the second image plane appears at a position on the road surface farther than the first image plane. The second image plane begins at a first predetermined distance from the vehicle and ends at a distance from the vehicle. The second predetermined distance is greater than the first predetermined distance. The second image plane includes a first linear boundary arranged horizontally relative to the road, the first linear boundary being located at a first predetermined distance from the vehicle. The second image plane is defined by an inclined linear boundary, the inclined linear boundary starting at a first predetermined distance from the vehicle and ending at a third predetermined distance from the vehicle, the inclined linear boundary being inclined at an angle relative to the first linear boundary, the third predetermined distance being greater than the first predetermined distance and less than the second predetermined distance. The second image plane also includes a second linear boundary arranged parallel to the road, the inclined linear boundary being inclined at an angle relative to the second linear boundary, the second linear boundary starting at a third predetermined distance from the vehicle and ending at a second predetermined distance from the vehicle.

2. The method of claim 1, wherein determining whether the visible light camera cannot detect the object in front of the vehicle comprises: Determine the confidence level at which the visible light camera detects an object in front of the vehicle, the object being detected by the object detection sensor; The confidence level is compared with a predetermined threshold to determine whether the confidence level is equal to or less than the predetermined threshold; as well as In response to determining that the confidence level is equal to or less than the predetermined threshold, the display is instructed to generate the virtual image using the sensor data received from the object detection sensor to identify the object in front of the vehicle.

3. The method according to claim 2, further comprising: In response to determining that the confidence level is greater than the predetermined threshold, the generation of the virtual image used to identify the object in front of the vehicle is prevented.

4. The method of claim 1, wherein the object detection sensor is a thermal imaging camera configured to generate a thermal image of the area in front of the vehicle, the method further comprising cropping the thermal image of the area in front of the vehicle to generate a cropped thermal image, the method further comprising commanding the display to present the cropped thermal image, and the cropped thermal image comprising only thermal images of the object in front of the vehicle.

5. The method of claim 4, wherein the virtual image comprises a rectangle surrounding the entire cropped thermal image.

6. The method of claim 5, wherein the virtual image further includes an arrow adjacent to the rectangle to indicate the direction of movement of the object in front of the vehicle.

7. The method of claim 1, wherein the object detection sensor is a night vision device configured to generate a night view image of a region in front of the vehicle, the method further comprising cropping the night view image of the region in front of the vehicle to generate a cropped night view image, the method further comprising commanding the display to present the cropped night view image, the cropped night view image including only the night view image of the object in front of the vehicle, the virtual image including a rectangle surrounding the entire cropped night view image, and the virtual image including arrows adjacent to the rectangle to indicate the direction of movement of the object in front of the vehicle.

8. The method according to claim 7, further comprising: Determine the eye position of the vehicle operator; Determine the position of the object in front of the vehicle; as well as The position of the virtual image is determined based on the eye position of the vehicle operator and the position of the object in front of the vehicle.

9. The method of claim 8, wherein instructing the display to generate the virtual image to identify the object in front of the vehicle includes generating the virtual image at a previously determined location of the virtual image.

Images