Vehicle display system for night driving
The image processing of vehicle camera systems in low-light conditions is enhanced through machine learning algorithms and computer vision technology, which solves the problem of low object recognition performance in existing systems at night, and realizes efficient object recognition and information provision in low-light conditions.
Patent Information
- Application Number
- CN202311397534.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-06-06
- Filing Date
- 2023-10-25
- Publication Date
- 2024-12-06
AI Technical Summary
Existing vehicle camera systems are difficult to provide high-resolution object detection and recognition performance under low light conditions, and night vision camera systems increase system complexity and resource usage.
Training using machine learning algorithms, the vehicle camera captures night images and enhances the images through computer vision algorithms, segments and recognizes the target object, and provides the occupants with the position and shape information of the target object through an augmented reality head-up display and transparent windshield display.
Improve occupants' perception and object recognition performance in low-light conditions, reduce system complexity and resource usage, and provide higher contrast and details.
Smart Images

Figure CN119091413A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to systems and methods for providing information to occupants of a vehicle. Background Art
[0002] To increase occupant perception and convenience, vehicles may be equipped with camera systems that are configured to detect and identify objects in the vehicle's surroundings. After identifying the object, the vehicle may utilize a display system to increase occupant perception by notifying the occupant of the presence of the object in the vehicle's surroundings. However, existing systems may not be suitable for low-light conditions (such as, for example, night driving scenarios). Night vision camera systems (such as, for example, infrared and / or thermal imaging camera systems) can be used to provide additional perception capabilities in low-light conditions. However, the use of night vision camera systems may increase weight, resource usage, and / or system complexity. In addition, the resolution of the images provided by the night vision camera system may be low, resulting in reduced object detection and recognition performance.
[0003] Therefore, while existing camera and display systems and methods achieve their intended purposes, a new and improved system and method is needed for providing information to occupants of a vehicle. Summary of the invention
[0004] According to several aspects, a method for providing information to an occupant of a vehicle is provided. The method includes: training a machine learning algorithm to receive a night image as an input and to produce an enhanced image as an output. The method also includes: using a camera of the vehicle to capture a night image of the vehicle's surroundings. The method also includes: enhancing the night image using the machine learning algorithm to produce the enhanced image. The method also includes: identifying a target object in the enhanced image using a computer vision algorithm. The method also includes: notifying an occupant of the vehicle of the presence of the target object.
[0005] In another aspect of the present disclosure, training the machine learning algorithm may further include: generating a training data set. The training data set includes a plurality of image pairs. Each of the plurality of image pairs includes a sample input image and a sample output image. Training the machine learning algorithm may further include: using the training data set to train the machine learning algorithm using supervised learning.
[0006] In another aspect of the present disclosure, generating a training data set may further include: capturing a plurality of image pairs using a camera. For each of the plurality of image pairs, the sample input image is a normal exposure image of the nighttime environment surrounding the vehicle, and the sample output image is a high exposure image of the nighttime environment surrounding the vehicle.
[0007] In another aspect of the present disclosure, generating a training data set may further include: capturing a plurality of image pairs using a camera. For each of the plurality of image pairs, the sample input image is a normally exposed image of the nighttime environment surrounding the vehicle, and the sample output image is a computer restored image of the nighttime environment surrounding the vehicle.
[0008] In another aspect of the present disclosure, capturing one of the plurality of image pairs may further include: capturing a sample input image using a camera. Capturing one of the plurality of image pairs may further include: restoring the sample input image by using at least one of a computer vision denoising algorithm and a computer vision defogging algorithm to generate a sample output image.
[0009] In another aspect of the present disclosure, generating a training data set may further include: capturing a plurality of image pairs using a camera. For each of the plurality of image pairs, the sample input image is a simulated night image of a daytime environment around the vehicle, and the sample output image is a normal exposure image of a daytime environment around the vehicle.
[0010] In another aspect of the present disclosure, capturing one of the plurality of image pairs may further include: capturing a sample output image using a first training camera. Capturing one of the plurality of image pairs may further include: capturing a sample input image using a second training camera and an inverse apodized neutral density filter configured to darken the sample output image.
[0011] In another aspect of the present disclosure, identifying the target object may further include: segmenting the enhanced image into a plurality of image segments. Each of the plurality of image segments represents one of the plurality of objects in the enhanced image. Identifying the target object may further include: identifying the target object, wherein the target object is one of the plurality of objects in the enhanced image.
[0012] In another aspect of the present disclosure, notifying an occupant of the vehicle may further include displaying the notification to the occupant of the vehicle using a display.
[0013] In another aspect of the present disclosure, the display is at least one of an augmented reality head-up display (AR-HUD) system and a transparent windshield display (TWD) system. Displaying the notification may also include: determining a position of the target object relative to the vehicle. Displaying the notification may also include: calculating a size, shape, and position of the graphic based on the position of the target object relative to the vehicle. Displaying the notification may also include: displaying the graphic on the windshield of the vehicle using at least one of the AR-HUD system and the TWD system based on the size, shape, and position.
[0014] According to several aspects, a system for providing information to an occupant of a vehicle is provided. The system includes a notification system including a camera, a display, and a vehicle controller in electrical communication with the camera and the display. The vehicle controller is programmed to capture a nighttime image of the vehicle's surroundings using the camera. The vehicle controller is also programmed to enhance the nighttime image using a machine learning algorithm to produce an enhanced image. The vehicle controller is also programmed to identify a target object in the enhanced image using a computer vision algorithm. The vehicle controller is also programmed to notify the occupant of the vehicle of the presence of the target object using the display.
[0015] In another aspect of the present disclosure, to identify the target object, the vehicle controller is further programmed to segment the augmented image into a plurality of image segments. Each of the plurality of image segments represents one of the plurality of objects in the augmented image. To identify the target object, the vehicle controller is further programmed to identify the target object, wherein the target object is one of the plurality of objects in the augmented image.
[0016] In another aspect of the present disclosure, the display is an augmented reality head-up display (AR-HUD) system in electrical communication with a vehicle controller. The AR-HUD system includes an occupant position tracking device and an AR-HUD projector. In order to notify the occupants of the vehicle, the vehicle controller is also programmed to determine the position of the target object relative to the vehicle. In order to notify the occupants of the vehicle, the vehicle controller is also programmed to determine the position of the occupants of the vehicle using the occupant position tracking device. In order to notify the occupants of the vehicle, the vehicle controller is also programmed to calculate the size, shape and position of the graphics based on the position of the occupants and the position of the target object relative to the vehicle. In order to notify the occupants of the vehicle, the vehicle controller is also programmed to display the graphics on the windshield of the vehicle using the augmented reality head-up display system based on the size, shape and position.
[0017] In another aspect of the present disclosure, the display also includes a transparent windshield display (TWD) system in electrical communication with the vehicle controller. The TWD system includes a TWD projector. In order to inform the occupants of the vehicle, the vehicle controller is also programmed to display graphics on the windshield of the vehicle using the TWD projector based on size, shape and position.
[0018] In another aspect of the present disclosure, the system also includes a training system, which includes a training camera system. The training camera system includes a beam splitter having a first port, a second port, and a third port. The training camera system also includes a first training camera optically connected to the first port of the beam splitter. The training camera system also includes a second training camera. The training camera system also includes a reverse apodization neutral density filter optically connected to the second training camera and the second port of the beam splitter. The training camera system also includes a relay lens optically connected to the third port of the beam splitter. The training camera system also includes a training controller in electrical communication with the first training camera and the second training camera. The training controller is programmed to generate a training data set using the training camera system. The training data set includes a plurality of image pairs. Each of the plurality of image pairs includes a sample input image and a sample output image. The training controller is also programmed to train a machine learning algorithm by using supervised learning using the training data set.
[0019] In another aspect of the present disclosure, in order to generate a training data set, the training controller is further programmed to capture a plurality of image pairs using the first training camera and the second training camera. For each of the plurality of image pairs, the sample input image is a simulated night image of the daytime environment around the vehicle, and the sample output image is a normal exposure image of the daytime environment around the vehicle.
[0020] In another aspect of the present disclosure, to capture one of the plurality of image pairs, the training controller is further programmed to capture a sample output image using the first training camera. To capture one of the plurality of image pairs, the training controller is further programmed to capture a sample input image using the second training camera.
[0021] According to several aspects, a system for providing information to an occupant of a vehicle is provided. The system includes: a notification system including a camera, an augmented reality head-up display (AR-HUD) system including an occupant position tracking device and an AR-HUD projector, and a vehicle controller in electrical communication with the camera and the AR-HUD system. The vehicle controller is programmed to capture night images of the vehicle's surroundings using the camera. The vehicle controller is also programmed to enhance the night images using a machine learning algorithm to produce an enhanced image. The vehicle controller is also programmed to employ a computer vision algorithm to identify a target object in the enhanced image. The vehicle controller is also programmed to utilize the AR-HUD system to notify the occupants of the vehicle of the presence of a target object.
[0022] In another aspect of the present disclosure, to notify an occupant of the vehicle, the vehicle controller is further programmed to determine the position of the target object relative to the vehicle. To notify an occupant of the vehicle, the vehicle controller is further programmed to determine the position of the occupant of the vehicle using an occupant position tracking device. To notify an occupant of the vehicle, the vehicle controller is further programmed to calculate the size, shape and position of a graphic based on the position of the occupant and the position of the target object relative to the vehicle. To notify an occupant of the vehicle, the vehicle controller is further programmed to display a graphic on the windshield of the vehicle using an augmented reality head-up display system based on the size, shape and position.
[0023] In another aspect of the present disclosure, the system also includes a training system, which includes a training camera system. The training camera system includes a beam splitter having a first port, a second port, and a third port. The training camera system also includes a first training camera optically connected to the first port of the beam splitter. The training camera system also includes a second training camera. The training camera system also includes a reverse apodization neutral density filter optically connected to the second training camera and the second port of the beam splitter. The training camera system also includes a relay lens optically connected to the third port of the beam splitter. The training camera system also includes a training controller in electrical communication with the first training camera and the second training camera. The training controller is programmed to generate a training data set using the training camera system. The training data set includes a plurality of image pairs. Each of the plurality of image pairs includes a sample input image and a sample output image. The training controller is also programmed to train a machine learning algorithm by using supervised learning using the training data set.
[0024] Further areas of applicability will become apparent from the description provided herein.It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.
[0026] Figure 1 is a schematic diagram of a system for providing information to an occupant of a vehicle according to an exemplary embodiment;
[0027] Figure 2 is a schematic diagram of an AR-HUD system for use by an exemplary occupant according to an exemplary embodiment;
[0028] Figure 3 is a schematic front view of a dual focal plane augmented reality display according to an exemplary embodiment, highlighting a second image plane of the dual focal plane augmented reality display;
[0029] Figure 4is a schematic diagram of a second image plane of a dual focal plane augmented reality display according to an exemplary embodiment;
[0030] Figure 5 is a schematic diagram of a machine learning training system according to an exemplary embodiment;
[0031] Figure 6 is a flow chart of a method for providing information to an occupant of a vehicle according to an exemplary embodiment;
[0032] Figure 7 is a flow chart of a first method for generating a training data set according to an exemplary embodiment;
[0033] Figure 8 is a flow chart of a second method for generating a training data set according to an exemplary embodiment;
[0034] Fig. 9 is a flowchart of a third method for generating a training data set according to an exemplary embodiment; and
[0035] Fig.10 is a schematic diagram of a driving scenario according to an exemplary embodiment. DETAILED DESCRIPTION
[0036] The following description is merely exemplary in nature and is not intended to limit the present invention, application, or uses.
[0037] Driving in low-light conditions may increase the risk of collision due to reduced visibility. Color camera systems may not be able to capture details and contrast in low-light conditions. Night vision camera systems (including infrared and / or thermal imaging camera systems) can be used to receive additional details and contrast in low-light conditions. However, the use of night vision camera systems can result in additional weight, resource usage, and / or system complexity. Accordingly, the present disclosure provides a new and improved system for providing information about target objects to occupants of a vehicle in low-light conditions using a color camera system.
[0038] refer to Figure 1 , a system for providing information to an occupant of a vehicle is shown, generally indicated by the reference numeral 10. The system 10 is shown with an exemplary vehicle 12. Although a passenger vehicle is shown, it should be appreciated that the vehicle 12 may be any type of vehicle without departing from the scope of the present disclosure. The system 10 generally includes a vehicle controller 14, a plurality of vehicle sensors 16, an augmented reality head-up display (AR-HUD) system 18, a transparent windshield display (TWD) system 20, and a human-machine interface (HMI) 22.
[0039] As described below, the vehicle controller 14 is used to implement a method 100 for providing information to an occupant of a vehicle. The vehicle controller 14 includes at least one processor 26 and a non-transitory computer-readable storage device or medium 28. The processor 26 can be a custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor of several processors associated with the vehicle controller 14, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, or a combination thereof, or generally a device for executing instructions. For example, the computer-readable storage device or medium 28 can include volatile and non-volatile storage in a read-only memory (ROM), a random access memory (RAM), and a keep-alive memory (KAM). KAM is a persistent or non-volatile memory that can be used to store various operating variables when the processor 26 is powered off. The computer readable storage device or medium 28 may be implemented using a variety of known storage devices, such as PROM (Programmable Read Only Memory), EPROM (Electrically Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory, or other electrical, magnetic, optical, or combination storage devices capable of storing data, some of which represents executable instructions used by the vehicle controller 14 to control various systems of the vehicle 12. The vehicle controller 14 may also be comprised of multiple controllers that are in electrical communication with each other. The vehicle controller 14 may be interconnected with additional systems and / or controllers of the vehicle 12, thereby allowing the vehicle controller 14 to access data such as, for example, the speed, acceleration, braking, and steering angle of the vehicle 12.
[0040] The vehicle controller 14 is in electrical communication with a plurality of vehicle sensors 16, an AR-HUD system 18, a TWD system 20, and an HMI 22. In an exemplary embodiment, the electrical communication is established using, for example, a CAN network, a FLEXRAY network, a local area network (e.g., WiFi, Ethernet, etc.), a serial peripheral interface (SPI) network, etc. It should be understood that various additional wired / wireless technologies and communication protocols for communicating with the vehicle controller 14 are within the scope of the present disclosure.
[0041] The plurality of vehicle sensors 16 are used to acquire information about an environment 30a surrounding the vehicle 12. In an exemplary embodiment, the plurality of vehicle sensors 16 include at least a camera system 32, a vehicle communication system 34, and a global navigation satellite system (GNSS) 36.
[0042] In another exemplary embodiment, the plurality of vehicle sensors 16 further include sensors that determine performance data regarding the vehicle 12. In an exemplary embodiment, the plurality of vehicle sensors 16 further include at least one of a motor speed sensor, a motor torque sensor, an electric drive motor voltage and / or current sensor, an accelerator pedal position sensor, a coolant temperature sensor, a cooling fan speed sensor, and a transmission oil temperature sensor.
[0043] In another exemplary embodiment, the plurality of vehicle sensors 16 also include sensors that determine information about the interior environment of the vehicle 12 (such as, for example, seat occupancy sensors, cabin air temperature sensors, cabin motion detection sensors, cabin cameras and / or cabin microphones, etc.).
[0044] In another exemplary embodiment, plurality of vehicle sensors 16 also includes sensors that determine information about the environment surrounding vehicle 12 (such as, for example, an ambient air temperature sensor, a barometric pressure sensor, and / or a photo and / or video camera positioned to view the environment in front of vehicle 12 ).
[0045] In another exemplary embodiment, at least one of the plurality of vehicle sensors 16 is capable of measuring distances in the environment surrounding the vehicle 12. In one non-limiting example, the plurality of vehicle sensors 16 include a stereo camera with distance measurement capabilities. In one example, at least one of the plurality of vehicle sensors 16 is fixed to the interior of the vehicle 12 (e.g., fixed to the roof lining of the vehicle 12) to obtain a view through the windshield of the vehicle 12. In another example, at least one of the plurality of vehicle sensors 16 is fixed to the exterior of the vehicle 12 (e.g., fixed to the roof of the vehicle 12) to obtain a view of the environment surrounding the vehicle 12. It should be understood that various other types of vehicle sensors (such as, for example, lidar sensors, ultrasonic ranging sensors, radar sensors, and / or time-of-flight sensors) are within the scope of the present disclosure. As described above, the plurality of vehicle sensors 16 are in electrical communication with the vehicle controller 14.
[0046] The camera system 32 is used to capture images and / or video of the environment 30a surrounding the vehicle 12. In one exemplary embodiment, the camera system 32 includes a photo and / or video camera positioned to view the environment 30a surrounding the vehicle 12. In one non-limiting example, the camera system 32 includes a camera mounted inside the vehicle 12 (e.g., mounted in the headliner of the vehicle 12) to obtain a view through the windshield 38. In another non-limiting example, the camera system 32 includes a camera mounted outside the vehicle 12 (e.g., mounted on the roof of the vehicle 12) to obtain a view of the environment 30a in front of the vehicle 12.
[0047] In another exemplary embodiment, the camera system 32 is a surround view camera system that includes a plurality of cameras (also referred to as satellite cameras) that are arranged to provide views of the environment 30a adjacent to all sides of the vehicle 12. In one non-limiting example, the camera system 32 includes a front camera (e.g., mounted in the front grille of the vehicle 12), a rear camera (e.g., mounted on the rear tailgate of the vehicle 12), and two side cameras (e.g., mounted below each of the two side mirrors of the vehicle 12). In another non-limiting example, the camera system 32 also includes an additional rear view camera mounted near the center high mounted brake light of the vehicle 12.
[0048] It should be understood that camera systems with additional cameras and / or additional mounting locations are within the scope of the present disclosure. It should also be understood that cameras with various sensor types (e.g., including charge coupled device (CCD) sensors, complementary metal oxide semiconductor (CMOS) sensors, and / or high dynamic range (HDR) sensors) are within the scope of the present disclosure. In addition, cameras with various lens types (e.g., including wide angle lenses and / or narrow angle lenses) are also within the scope of the present disclosure.
[0049] The vehicle controller 14 utilizes the vehicle communication system 34 to communicate with other systems external to the vehicle 12. For example, the vehicle communication system 34 includes the ability to communicate with vehicles ("V2V" communication), infrastructure ("V2I" communication), remote systems located at a remote call center (e.g., General Motors' ON-STAR), and / or personal devices. In general, the term vehicle-to-everything communication ("V2X" communication) refers to communication between the vehicle 12 and any remote system (e.g., vehicle, infrastructure, and / or remote system). In some embodiments, the vehicle communication system 34 is a wireless communication system configured to communicate via a wireless local area network (WLAN) using the IEEE 802.11 standard or by using cellular data communication (e.g., using GSMA standards (such as, for example, SGP.02, SGP.22, SGP.32, etc.)). Therefore, the vehicle communication system 34 may also include an embedded universal integrated circuit card (eUICC) configured to store at least one cellular connection configuration profile (e.g., an embedded subscriber identity module (eSIM) profile). The vehicle communication system 34 is also configured to communicate via a personal area network (e.g., Bluetooth) and / or near field communication (NFC). However, additional or alternative communication methods (such as dedicated short-range communication (DSRC) channels and / or mobile telecommunication protocols based on the 3rd Generation Partnership Project (3GPP) standard) are also considered to be within the scope of the present disclosure. A DSRC channel refers to a one-way or two-way short-range wireless communication channel specifically designed for automobiles and a corresponding set of protocols and standards. 3GPP refers to a partnership between several standard organizations that develop protocols and standards for mobile telecommunications. The 3GPP standards are structured as "versions." Therefore, communication methods based on 3GPP versions 14, 15, 16 and / or future 3GPP versions are considered to be within the scope of the present disclosure. Therefore, the vehicle communication system 34 may include one or more antennas and / or communication transceivers for receiving and / or sending signals, such as collaborative sensing messages (CSM). The vehicle communication system 34 is configured to perform wireless information communication between the vehicle 12 and another vehicle. In addition, the vehicle communication system 34 is configured to perform wireless information communication between the vehicle 12 and infrastructure or other vehicles. It should be appreciated that the vehicle communication system 34 may be integrated with the vehicle controller 14 (eg, integrated on the same circuit board as the vehicle controller 14 or be a part of the vehicle controller 14 ) without departing from the scope of the present disclosure.
[0050] GNSS 36 is used to determine the geographic location of vehicle 12. In an exemplary embodiment, GNSS 36 is a global positioning system (GPS). In a non-limiting example, GPS includes a GPS receiver antenna (not shown) and a GPS controller (not shown) in electrical communication with the GPS receiver antenna. The GPS receiver antenna receives signals from a plurality of satellites, and the GPS controller calculates the geographic location of vehicle 12 based on the signals received by the GPS receiver antenna. In an exemplary embodiment, GNSS 36 also includes a map. The map includes information about infrastructure such as municipal boundaries, roads, railways, sidewalks, buildings, etc. Therefore, map information is used to put the geographic location of vehicle 12 in context. In a non-limiting example, a wireless connection is used to retrieve the map from a remote source. In another non-limiting example, the map is stored in a database of GNSS 36. It should be understood that various other types of satellite-based radio navigation systems (such as, for example, Global Positioning System (GPS), Galileo (Galileo Satellite Navigation System), GLONASS (GLONASS Satellite Navigation System) and Beidou Navigation Satellite System (BDS)) are within the scope of the present disclosure. It should be appreciated that the GNSS 36 may be integrated with the vehicle controller 14 (eg, integrated on the same circuit board as the vehicle controller 14 or be a part of the vehicle controller 14 ) without departing from the scope of the present disclosure.
[0051] refer to Figure 2 , a system diagram of an AR-HUD system 18 for use with an exemplary occupant 40 is shown. Within the scope of the present disclosure, in one non-limiting example, the occupant 40 includes a driver, an occupant, and / or any other person in the vehicle 12. The AR-HUD system 18 is used to display AR-HUD graphics 42 (i.e., notification symbols that provide visual information to the occupant 40) on the windshield 38 of the vehicle 12. The AR-HUD system 18 includes an AR-HUD projector 44 and an occupant position tracking device 46. As described above, the AR-HUD system 18 is in electrical communication with the vehicle controller 14.
[0052] The AR-HUD projector 44 is used to project the AR-HUD graphics 42 onto the windshield 38 of the vehicle 12. It should be understood that various devices designed to project images (including, for example, optical collimators, laser projectors, digital optical projectors (DLP) and the like) are within the scope of the present disclosure.
[0053] The occupant position tracking device 46 is used to determine the position of the occupant 40 within the vehicle 12. For example, the occupant position tracking device 46 can track the position of the head 40a or the eyes 40b of the occupant 40. The position of the occupant 40 within the vehicle 12 from the occupant position tracking device 46 is used to locate the AR-HUD graphics 42 on the windshield 38 of the vehicle 12. In an exemplary embodiment, the occupant position tracking device 46 is one or more cameras disposed within the vehicle 12.
[0054] To operate the AR-HUD system 18, the vehicle controller 14 includes a plurality of software modules (including a system manager 48). During operation of the system 10, the system manager 48 receives at least a first input 50, a second input 52, and a third input 54. The first input 50 indicates the position of the vehicle 12 in space (i.e., the geographic location of the vehicle 12), the second input 52 indicates the position of the vehicle occupant 40 within the vehicle 12 (e.g., the position of the eyes and / or head of the occupant 40 within the vehicle 12), and the third input 54 is data related to target objects in the environment 30a surrounding the vehicle 12, which will be described in more detail below. The first input 50 may include data such as GNSS data (e.g., GPS data), vehicle speed, road curvature, and vehicle steering, and the data is collected from a plurality of vehicle sensors 16. The second input 52 is received from the occupant position tracking device 46. The third input 54 is data related to target objects in the environment 30a surrounding the vehicle 12. The system manager 48 is configured to determine (e.g., calculate) the type, size, shape, and color of the AR-HUD graphics 42 to be displayed using the AR-HUD projector 44 based on the first input 50 (i.e., the vehicle position in the environment 30a), the second input 52 (e.g., the position of the eyes 40b and / or the head 40a of the occupant 40 in the vehicle 12), and the third input 54 (i.e., the target object in the environment 30a surrounding the vehicle 12). The system manager 48 instructs the image engine 56, which is a software module or integrated circuit of the AR-HUD projector 44 or the vehicle controller 14, to display the AR-HUD graphics 42 using the AR-HUD projector 44. Based on the type, size, shape, and color of the AR-HUD graphics 42 determined by the system manager 48, the image engine 56 uses the AR-HUD projector 44 to display the AR-HUD graphics 42 on the windshield 38 of the vehicle 12. The AR-HUD graphics 42 are projected onto the windshield 38 by the AR-HUD projector 44 to display the AR-HUD graphics 42 along the road surface 58 .
[0055] In an exemplary embodiment of the present disclosure, the AR-HUD system 18 is a dual focal plane AR-HUD system. Figure 3 and Figure 4 , while continuing to refer to Figure 2, the AR-HUD system 18 has a first image plane 60 and a second image plane 62. The first image plane 60 displays a view of the outside world, while the second image plane 62 is reserved for displaying the AR-HUD graphics 42. The second image plane 62 spans multiple lanes, and the AR-HUD graphics 42 appear at a farther position on the road surface 58 relative to the first image plane 60. For example, Figure 3 and Figure 4 As shown, the second image plane 62 covers the left lane 64, the center lane 66, and the right lane 68. As a non-limiting example, in the center lane 66, the second image plane 62 starts at a first predetermined distance D1 (e.g., 25 meters) from the vehicle 12 and ends at a second predetermined distance D2 (e.g., 90 meters) from the vehicle 12. Regardless of the specific distance, the second predetermined distance D2 is greater than the first predetermined distance D1 to help the occupant 40 see the AR-HUD graphics 42 displayed by the AR-HUD projector 44. In the left lane 64 and the right lane 68, the second image plane 62 is defined by an inclined boundary that starts at the first predetermined distance D1 from the vehicle 12 and ends at a third predetermined distance D3 (e.g., 50 meters) from the vehicle 12. The third predetermined distance D3 is greater than the first predetermined distance D1 and less than the second predetermined distance D2 to help the occupant 40 see the AR-HUD graphics 42 displayed by the AR-HUD projector 44. As used herein, the term "dual focal plane AR-HUD" refers to an AR-HUD system that presents images in a first image plane and a second image plane, wherein the first image plane and the second image plane are located at different positions. It is necessary to construct the AR-HUD system 18 as a dual focal plane AR-HUD to facilitate the manipulation of the AR-HUD graphics 42 on the scene of the outside world. For example, by utilizing a dual focal plane AR-HUD, the size, position, and features of the AR-HUD graphics 42 can be changed based on, for example, the position of the eye 40b of the occupant 40.
[0056] Reference again Figure 1, the TWD system 20 is used to display images on the windshield 38 of the vehicle 12. In an exemplary embodiment, the AR-HUD system 18 can display AR-HUD graphics 42 in a predetermined area of the windshield 38 (for example, in the first image plane 60 and the second image plane 62). The TWD system 20 can display TWD graphics (not shown) in any area of the windshield 38. It should be understood that the terms AR-HUD graphics 42 and TWD graphics can be used interchangeably in the present disclosure, because the AR-HUD graphics 42 can be displayed using the TWD system 20, and the TWD graphics can be displayed using the AR-HUD system 18. These terms refer to any graphics displayed by the AR-HUD system 18 (AR-HUD graphics 42) or the TWD system 20 (TWD graphics). By jointly operating the AR-HUD system 18 and the TWD system 20, the vehicle controller 14 can display graphics in any area of the windshield 38. In an exemplary embodiment, the TWD system 20 includes a transparent phosphor (not shown) embedded in the windshield 38 and a TWD projector 70. As described above, the TWD system 20 is in electrical communication with the vehicle controller 14 .
[0057] Transparent phosphors are luminescent particles that emit fluorescence in response to being excited by the TWD projector 70. In an exemplary embodiment, the transparent phosphors are red, green, and blue (RGB) phosphors, thereby allowing full-color operation of the TWD system 20. The use of monochromatic and / or bichromatic phosphors is also within the scope of the present disclosure. When the excitation light is absorbed by the transparent phosphor, the transparent phosphor emits visible light. The excitation light can be, for example, violet light (ranging between about 380 nanometers and 450 nanometers) and / or ultraviolet light in the visible spectrum.
[0058] The TWD projector 70 is used to excite the transparent phosphors in a predetermined pattern to produce a TWD graphic on the windshield 38. In one exemplary embodiment, the TWD projector 70 is a violet / ultraviolet laser projector disposed near the headliner of the vehicle 12. The TWD projector 70 includes three lasers, each configured to excite one of the red, green, or blue transparent phosphors.
[0059] In one exemplary embodiment, in addition to the AR-HUD system 18 and the TWD system 20, the HMI 22 is used to display information to the occupants 40 of the vehicle 12. In another exemplary embodiment, as a substitute for the AR-HUD system 18 and / or the TWD system 20, the HMI 22 is used to display information to the occupants 40 of the vehicle 12. In the above exemplary embodiments, the HMI 22 is a display system located within the field of view of the occupants 40 and is capable of displaying text, graphics and / or images. It should be understood that HMI display systems (including LCD displays, LED displays, etc.) are within the scope of the present disclosure. Other exemplary embodiments in which the HMI 22 is disposed in the rearview mirror are also within the scope of the present disclosure. As described above, the HMI 22 is in electrical communication with the vehicle controller 14.
[0060] refer to Figure 5 , shows a schematic diagram of a machine learning training system 80. The system 80 is used to train a machine learning algorithm to receive nighttime images and output enhanced images, which will be described in more detail below. The system 80 includes a first training camera 82, a second training camera 84, a first filter 86, a second filter 88, a beam splitter 90, a relay lens 92, and a training controller 94.
[0061] The first training camera 82 and the second training camera 84 are used to capture images of the environment 30b surrounding the system 80. Within the scope of the present disclosure, the first training camera 82 and the second training camera 84 are referred to as "training" cameras because they are used in the machine learning training system 80 to capture training data, which will be described in more detail below. In an exemplary embodiment, the first training camera 82 and the second training camera 84 are similar in structure and function to one or more cameras of the camera system 32 described above. The first training camera 82 and the second training camera 84 are color (RGB) cameras. It should also be understood that cameras with various sensor types (for example, including charge coupled device (CCD) sensors, complementary metal oxide semiconductor (CMOS) sensors, high dynamic range (HDR) sensors, sensors with Bayer filter arrays, and / or sensors with X-Trans filter arrays) are within the scope of the present disclosure. In addition, cameras with various lens types (for example, including wide-angle lenses and / or narrow-angle lenses) are also within the scope of the present disclosure.
[0062] The first filter 86 is used to reduce the amount of light entering the second training camera 84 to create a simulated nighttime image. In an exemplary embodiment, the first filter 86 is a neutral density filter. Within the scope of the present disclosure, a neutral density filter is a filter that reduces the intensity of all incident wavelengths of light equally across the entire area of the neutral density filter. A neutral density filter can be characterized by optical density. Optical density is defined as:
[0063]
[0064] Where d is the optical density, I T is the intensity of light transmitted through the neutral density filter, and I0 is the intensity of light incident on the neutral density filter. Within the scope of the present disclosure, the first filter 86 has a constant optical density across the entire area of the first filter 86. Therefore, the image of the environment 30b captured during the day appears darkened. The first filter 86 is optically connected to the beam splitter 90 and the second filter 88.
[0065] Within the scope of the present disclosure, optical connectivity means that light can propagate through the space between components and interact with the optical properties of each component. Optical connectivity can be facilitated by positioning the components so that light is transmitted between the components due to the inherent reflective and / or refractive properties of the components and / or the properties of the medium (e.g., air) between the components. Optical connectivity can also be facilitated by employing optical waveguides (i.e., physical structures designed to confine and guide light along a specific path). It should also be understood that various additional methods for establishing optical connectivity between optical components (e.g., including planar waveguides, ridge waveguides, channel waveguides, hollow core waveguides, etc.) are within the scope of the present disclosure.
[0066] The second filter 88 is used to simulate the effect of the headlights of the vehicle 12, thereby making the simulated nighttime image more suitable for vehicle applications. In an exemplary embodiment, the second filter 88 is a reverse-apodized neutral density filter. Within the scope of the present disclosure, a reverse-apodized neutral density filter is a neutral density filter having a varying optical density over an area of the reverse-apodized neutral density filter. In a non-limiting example, the second filter 88 is a reverse-apodized neutral density filter having an optical density of approximately 0 near the center of the second filter 88 and having a gradually increasing optical density as it approaches the outer edge of the second filter 88 (such as, for example, a reverse-apodized filter sold by THORLABS, INC. as model NDYR20B). Therefore, the high-transmission area in the center of the second filter 88 simulates the enhanced light provided by the headlights of the vehicle 12 in a nighttime scene. The second filter 88 is in optical communication with the first filter 86 and the second training camera 84.
[0067] The beam splitter 90 is used to direct light from the relay lens 92 to the first training camera 82 and the second training camera 84. In an exemplary embodiment, the beam splitter 90 is an optical device configured to split a light beam into a transmitted light beam and a reflected light beam. In a non-limiting example, the beam splitter 90 has three ports that allow light to be transmitted into / out of the beam splitter 90. The first port 96a is optically connected to the first training camera 82. The second port 96b is optically connected to the first filter 86, the second filter 88, and the second training camera 84. The third port 96c is optically connected to the relay lens 92. In a non-limiting example, the beam splitter 90 includes two triangular glass prisms that are bonded together at their bottoms using a polyester, epoxy resin and / or urethane-based adhesive layer. The thickness of the adhesive layer is adjusted so that about half of the light incident through the third port 96c of the beam splitter 90 is reflected and emitted from the second port 96b, and about half of the light incident through the third port 96c is transmitted and emitted from the first port 96a.
[0068] Relay lens 92 is used to receive light from environment 30b and transmit the light to beam splitter 90. In an exemplary embodiment, relay lens 92 may be any type of lens configured to focus and / or transmit light for a camera system.
[0069] The training controller 94 is in electrical communication with the first training camera 82 and the second training camera 84. The training controller 94 includes at least one training processor 98a and a non-transitory computer-readable storage device or training medium 98b for training. The above description of the type and construction of the vehicle controller 14 also applies to the training controller 94. In one non-limiting example, as described above, the training processor 98a and the training medium 98b of the training controller 94 are similar in structure and / or function to the processor 26 and the medium 28 of the vehicle controller 14. In an exemplary embodiment, the training controller 94 may also include an electrical interface (not shown) for communicating with an external device (such as, for example, the vehicle controller 14). In one non-limiting example, the electrical interface is a serial or parallel port that allows digital transmission of information. In one non-limiting example, the electrical interface is a wireless communication system that allows wireless transmission of digital information. In one non-limiting example, the information includes training data, training statistics and / or machine learning model parameters, etc.
[0070] In some embodiments, the system 80 may be partially or completely integrated with the system 10. In other words, the system 80 may be fixed to the vehicle 12, and the training controller 94 may be in electrical communication with the vehicle controller 14. In other embodiments, the system 80 is used separately from the system 10, and the training results (e.g., model parameters) are transmitted from the training controller 94 to the vehicle controller 14 after training.
[0071] refer to Figure 6 , a flow chart of a method 100 for providing information to an occupant of a vehicle is provided. The method 100 begins at step 102 and proceeds to step 104. In step 104, the system 10 and / or the system 80 is used to generate a training data set, which will be described in more detail below. In an exemplary embodiment, the training data set includes a plurality of image pairs. Each of the plurality of image pairs includes a sample input image and a sample output image. After step 104, the method 100 proceeds to step 106.
[0072] In step 106, the system 10 and / or system 80 is used to train a machine learning algorithm using the training data set generated in step 104 to receive a night image and output an enhanced image. Within the scope of the present disclosure, a night image is an image of the environment 30a captured by the camera system 32 in a night driving scene with reduced ambient light. Due to the reduced light, the night image may lack details and contrast. The enhanced image is a restored (i.e., enhanced) version of the night image, which has higher contrast, details, and / or color accuracy.
[0073] In one non-limiting example, the machine learning algorithm includes multiple layers (including an input layer and an output layer) and one or more hidden layers. The input layer receives the night image as input. The input is then passed to the hidden layer. Each hidden layer applies a transformation (e.g., a nonlinear transformation) to the night image and passes the result to the next hidden layer until the last hidden layer. The output layer produces the final enhanced image.
[0074] To train the machine learning algorithm, a training data set of sample input images and their corresponding sample output images generated in step 104 is used. The machine learning algorithm is trained by adjusting the internal weights between nodes in each hidden layer to minimize the error. During training, an optimization technique (e.g., gradient descent) is used to adjust the internal weights to reduce the error. The training process is repeated for the entire data set until the error is minimized, and then the resulting trained model is used to enhance new input images.
[0075] After the machine learning algorithm is sufficiently trained, the algorithm is capable of enhancing nighttime images. By adjusting the weights between nodes in each hidden layer during training, the algorithm "learns" to recognize patterns in nighttime images that represent specific corresponding patterns, objects, and / or structures, etc. in the enhanced image. After step 106, method 100 proceeds to step 108.
[0076] In step 108 , the vehicle controller 14 utilizes the camera system 32 to capture a nighttime image of the environment 30 a surrounding the vehicle 12 . After step 108 , the method 100 proceeds to step 110 .
[0077] In step 110, the vehicle controller 14 utilizes the machine learning model to enhance the night image captured in step 108 to produce an enhanced image. As described above, in an exemplary embodiment, the input layer of the machine learning algorithm receives the night image as input. The input is then passed to the hidden layers. Each hidden layer applies a transformation (e.g., a nonlinear transformation) to the night image and passes the result to the next hidden layer until the last hidden layer. The output layer produces the final enhanced image. After step 110, the method 100 proceeds to step 112.
[0078] In step 112, the vehicle controller 14 segments the enhanced image generated in step 110 into a plurality of image segments. Each of the plurality of image segments represents an object of a plurality of objects in the enhanced image. In an exemplary embodiment, the vehicle controller 14 employs a computer vision segmentation algorithm to segment the enhanced image. In a non-limiting example, the segmentation algorithm employs methods such as, for example, edge detection, thresholding, clustering, and / or machine learning techniques such as convolutional neural networks. It should be understood that any method for segmenting objects in an image is within the scope of the present disclosure. After step 112, the method 100 proceeds to step 114.
[0079] In step 114, the vehicle controller 14 identifies a target object in the enhanced image. Within the scope of the present disclosure, a target object is an object related to the driving task. In a non-limiting example, the target object may include, for example, a pedestrian near the road, an animal near the road, a distant vehicle on or near the road, an object near the road, a traffic sign / traffic light, and / or a lane marking, etc. In an exemplary embodiment, the target object is one of the multiple objects identified from the enhanced image in step 112. After step 114, the method 100 proceeds to step 116.
[0080] In step 116, the vehicle controller 14 determines the position of the target object identified in step 114 relative to the vehicle 12. In an exemplary embodiment, the vehicle controller 14 utilizes the vehicle sensors 16 to detect the target object and determine the position of the target object relative to the vehicle 12. After step 116, the method 100 proceeds to step 118.
[0081] In step 118, as described above, the vehicle controller 14 calculates the size, shape, and position of the AR-HUD graphic 42. The position of the AR-HUD graphic 42 is selected so that from the perspective of the occupant 40, the AR-HUD graphic 42 appears to be overlaid on a target object in the environment 30a surrounding the vehicle 12. After step 118, the method 100 proceeds to step 120.
[0082] In step 120, the vehicle controller 14 utilizes the occupant position tracking device 46 to determine the position of the occupant 40, as described above, and utilizes at least one of the AR-HUD system 18 and the TWD system 20 to display the AR-HUD graphics 42 on the windshield 38 of the vehicle 12, as described above. In an exemplary embodiment, the vehicle controller 14 also utilizes the HMI 22 to provide an augmented view of the environment 30a surrounding the vehicle 12 (including the AR-HUD graphics 42). After step 120, the method 100 enters a standby state in step 122.
[0083] In an exemplary embodiment, the vehicle controller 14 repeatedly exits the standby state 122 and resumes execution of the method 100 at step 102. In a non-limiting example, the vehicle controller 14 exits the standby state 122 and resumes execution of the method 100 on a timer (eg, every three hundred milliseconds).
[0084] refer to Figure 7 , a flowchart of a first exemplary embodiment 104a of step 104 is shown. The first exemplary embodiment 104a is a method for generating a training data set using the system 10. The first exemplary embodiment 104a starts at step 702. In step 702, the vehicle controller 14 uses the camera system 32 to capture a normally exposed image of the nighttime environment around the vehicle 12. Within the scope of the present disclosure, the normally exposed image is an image exposed within a default exposure time (e.g., a shutter speed of 1 / 30 second) suitable for daytime image capture. The nighttime environment is, for example, an environment 30a with poor lighting due to lack of sunlight after sunset. The nighttime environment may also occur during bad weather, under an overpass / bridge, and / or in a tunnel, etc. After step 702, the first exemplary embodiment 104a proceeds to step 704.
[0085] In step 704, the vehicle controller 14 uses the camera system 32 to capture a high exposure image of the nighttime environment surrounding the vehicle. Within the scope of the present disclosure, a high exposure image is an image exposed within a long exposure time (e.g., a shutter speed of 30 seconds) suitable for image capture in extremely low light conditions. It should be understood that the images captured in steps 702 and 704 are captured for the same object, and the camera system 32 is not moved between the execution of steps 702 and 704. After step 704, the first exemplary embodiment 104a proceeds to step 706.
[0086] In step 706, the images captured in step 702 and step 704 are saved as an image pair in the medium 28 of the vehicle controller 14. In one non-limiting example, the image captured in step 702 is referred to as a sample input image, and the image captured in step 704 is referred to as a sample output image. After step 706, the first exemplary embodiment 104a proceeds to step 708.
[0087] In step 708, if the number of image pairs in the medium 28 of the vehicle controller 14 is less than the predetermined training data set threshold (e.g., one thousand image pairs), the first exemplary embodiment 104a returns to step 702 to capture more image pairs. If the number of image pairs in the medium 28 of the vehicle controller 14 is greater than or equal to the predetermined training data set threshold (e.g., one thousand image pairs), the first exemplary embodiment 104a ends and the method 100 continues as described above.
[0088] refer to Figure 8 , a flowchart of a second exemplary embodiment 104b of step 104 is shown. The second exemplary embodiment 104b is a method for generating a training data set using the system 10. The second exemplary embodiment 104b starts at step 802. In step 802, the vehicle controller 14 uses the camera system 32 to capture a normally exposed image of the nighttime environment around the vehicle 12. Within the scope of the present disclosure, the normally exposed image is an image exposed within a default exposure time (e.g., a shutter speed of 1 / 30 second) suitable for daytime image capture. The nighttime environment is an environment 30a with poor lighting due to lack of sunlight. The nighttime environment may also occur during bad weather, under an overpass / bridge and / or in a tunnel, etc. After step 802, the second exemplary embodiment 104b proceeds to step 804.
[0089] In step 804, the vehicle controller 14 restores the image captured in step 802 to create a computer restored image of the night environment. In an exemplary embodiment, in order to restore the image, the vehicle controller 14 uses a computer vision denoising algorithm, which is configured to reduce noise and enhance the visual quality of noisy images. In a non-limiting example, the computer vision denoising algorithm is a non-local means (NLM) algorithm. The NLM algorithm compares each pixel with adjacent pixels to identify similarities across the entire image, thereby retaining edges and small details while reducing noise. In another non-limiting example, the computer vision denoising algorithm is a denoising algorithm based on deep learning, such as a convolutional neural network (CNN) that has been trained based on multiple noisy-noise-free image pairs. It should be understood that any computer vision denoising algorithm can be used without departing from the scope of the present disclosure.
[0090] In another exemplary embodiment, in order to restore the image, the vehicle controller 14 uses a computer vision defogging algorithm, which is configured to restore visibility and enhance the visual quality of the blurred image. In a non-limiting example, the computer vision defogging algorithm is a dark channel prior (DCP) algorithm. The DCP algorithm relies on the observation that at least some pixels in the image have very low intensity values in at least one color channel (called a dark channel). The dark channel indicates the presence of haze and is used to identify and eliminate haze, thereby restoring visibility. In another non-limiting example, the computer vision defogging algorithm is a defogging algorithm based on deep learning, for example, a convolutional neural network (CNN) that has been trained based on multiple foggy-fog-free image pairs is used. It should be understood that any computer vision defogging algorithm can be used without departing from the scope of the present disclosure. After step 804, the second exemplary embodiment 104b proceeds to step 806.
[0091] In step 806, the image captured in step 802 and the image generated in step 804 are saved as an image pair in the medium 28 of the vehicle controller 14. In one non-limiting example, the image captured in step 802 is referred to as a sample input image, and the image generated in step 804 is referred to as a sample output image. After step 806, the second exemplary embodiment 104b proceeds to step 808.
[0092] In step 808, if the number of image pairs in the medium 28 of the vehicle controller 14 is less than the predetermined training data set threshold (e.g., one thousand image pairs), the second exemplary embodiment 104b returns to step 802 to capture more image pairs. If the number of image pairs in the medium 28 of the vehicle controller 14 is greater than or equal to the predetermined training data set threshold (e.g., one thousand image pairs), the second exemplary embodiment 104b ends and the method 100 continues as described above.
[0093] refer to Fig. 9 , a flowchart of a third exemplary embodiment 104c of step 104 is shown. Fig. 9 And also refer to Figure 5 , the third exemplary embodiment 104c is a method for generating a training data set using a machine learning training system 80. The third exemplary embodiment 104c begins at step 902. In step 902, the training controller 94 uses the first training camera 82 to capture a normally exposed image of a daytime environment surrounding the system 80. Within the scope of the present disclosure, the normally exposed image is an image exposed within a default exposure time (e.g., a shutter speed of 1 / 30 second) suitable for daytime image capture. A daytime environment is, for example, an environment 30b that is fully illuminated by sunlight and / or street lights. After step 902, the third exemplary embodiment 104c proceeds to step 904.
[0094] In step 904, the training controller 94 uses the second training camera 84 to capture a simulated night image of the daytime environment around the vehicle. Within the scope of the present disclosure, the simulated night image refers to the situation where the image captured by the second training camera 84 is darkened by the first filter 86 and the second filter 88, as described above in conjunction with Figure 5 As discussed. It should be understood that the images captured in step 902 and step 904 are captured for the same object. Step 902 and step 904 may be performed sequentially or simultaneously. By fixing the system 80 to a moving vehicle and performing steps 902 and 904 simultaneously, a large amount of training data may be quickly collected. After step 904, the third exemplary embodiment 104c proceeds to step 906.
[0095] In step 906, the images captured in step 902 and step 904 are saved as an image pair in the training medium 98b of the training controller 94. In one non-limiting example, the image captured in step 902 is referred to as a sample output image, and the image captured in step 904 is referred to as a sample input image. After step 906, the third exemplary embodiment 104c proceeds to step 908.
[0096] In step 908, if the number of image pairs in the training medium 98b of the training controller 94 is less than a predetermined training data set threshold (e.g., one thousand image pairs), the third exemplary embodiment 104c returns to step 902 to capture more image pairs. If the number of image pairs in the training medium 98b of the training controller 94 is greater than or equal to the predetermined training data set threshold (e.g., one thousand image pairs), the third exemplary embodiment 104c ends and the method 100 continues as described above.
[0097] refer to Fig.10 , an exemplary driving scene 1002 employing the system 10 and method 100 is shown. The exemplary driving scene 1002 includes the vehicle 12, a remote vehicle 1004, and the AR-HUD graphics 42 from the perspective of the occupant 40. In the exemplary driving scene 1002, the remote vehicle 1004 is the target object. Therefore, the AR-HUD graphics 42 indicates the presence and position of the remote vehicle 1004 to the occupant 40, thereby improving the occupant's perception.
[0098] The systems 10, 80 and methods 100 of the present disclosure provide several advantages. Using the system 10 and method 100, occupant awareness can be improved during nighttime driving by notifying the occupant 40 of the presence of a target object in the environment 30a surrounding the vehicle 12. Using the system 80, machine learning training data for the system 10 and method 100 can be efficiently collected under daytime conditions, thereby allowing a large amount of high-quality training data to be collected, improving the accuracy and performance of the machine learning model.
[0099] The description of the present invention is merely exemplary in nature, and changes that do not depart from the gist of the present invention are included within the scope of the present invention. These changes should not be regarded as departing from the spirit and scope of the present invention.
Claims
1. A method for providing information to an occupant of a vehicle, the method comprising: Training a machine learning algorithm to receive nighttime images as input and produce enhanced images as output; Using a camera of the vehicle to capture a nighttime image of the vehicle's surroundings; enhancing the nighttime image using the machine learning algorithm to generate the enhanced image; Using a computer vision algorithm to identify a target object in the enhanced image; as well as An occupant of the vehicle is notified of the presence of the target object.
2. The method of claim 1, wherein training the machine learning algorithm further comprises: generating a training data set, wherein the training data set comprises a plurality of image pairs, and wherein each of the plurality of image pairs comprises a sample input image and a sample output image; as well as By utilizing the training data set, supervised learning is employed to train the machine learning algorithm.
3. The method according to claim 2, wherein generating the training data set further comprises: The camera is used to capture the multiple image pairs, wherein for each of the multiple image pairs, the sample input image is a normal exposure image of the night environment around the vehicle, and the sample output image is a high exposure image of the night environment around the vehicle.
4. The method according to claim 2, wherein generating the training data set further comprises: The camera is used to capture the multiple image pairs, wherein for each of the multiple image pairs, the sample input image is a normally exposed image of the night environment around the vehicle, and the sample output image is a computer-restored image of the night environment around the vehicle.
5. The method of claim 4, wherein capturing an image pair of the plurality of image pairs further comprises: capturing the sample input image using the camera; as well as The sample output image is generated by restoring the sample input image by adopting at least one of a computer vision denoising algorithm and a computer vision defogging algorithm.
6. The method according to claim 2, wherein generating the training data set further comprises: The camera is used to capture the multiple image pairs, wherein for each of the multiple image pairs, the sample input image is a simulated night image of the daytime environment around the vehicle, and the sample output image is a normally exposed image of the daytime environment around the vehicle.
7. The method of claim 6, wherein capturing an image pair of the plurality of image pairs further comprises: capturing the sample output image using a first training camera; as well as The sample input image is captured using a second training camera and an inverse apodized neutral density filter configured to darken the sample output image.
8. The method according to claim 1, wherein identifying the target object further comprises: segmenting the enhanced image into a plurality of image segments, wherein each of the plurality of image segments represents one of a plurality of objects in the enhanced image; as well as The target object is identified, wherein the target object is one of a plurality of objects in the enhanced image.
9. The method of claim 1 , wherein notifying an occupant of the vehicle further comprises: A notification is displayed to an occupant of the vehicle using a display.
10. The method of claim 9, wherein the display is at least one of an augmented reality head-up display (AR-HUD) system and a transparent windshield display (TWD) system, and wherein displaying the notification further comprises: determining a position of the target object relative to the vehicle; calculating a size, shape, and position of a graphic based on a position of the target object relative to the vehicle; as well as Based on the size, shape, and position, the graphic is displayed on a windshield of the vehicle using at least one of the AR-HUD system and the TWD system.
