System and method for enhancing visibility while driving a vehicle
By processing environmental data in real time through vehicle sensors and augmented reality devices, and adjusting the image display, the problem of reduced driver visibility in adverse weather conditions is solved, providing a clear driving environment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- FORD GLOBAL TECH LLC
- Filing Date
- 2025-12-02
- Publication Date
- 2026-06-16
Smart Images
Figure CN122218950A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of visibility while driving a vehicle. Specifically, embodiments of this disclosure relate to systems and methods related to enhancing visibility when driving a vehicle in adverse weather conditions. Background Technology
[0002] Reduced visibility in adverse weather conditions can make driving difficult. Fog is a major culprit, forming a thick, opaque layer that significantly reduces the distance a driver can see ahead. Rain is another common problem, as torrential downpours can obscure roads and other vehicles, and water on the windshield can distort vision even when wiping objects are moving at full speed. Additionally, rain can cause glare from oncoming headlights or streetlights, further complicating visibility. Snow and ice present unique challenges. Snowfall can create a whitening effect, where the landscape becomes a uniform white, making it difficult to distinguish the road from its surroundings. Ice on the windshield can also obscure vision, and snow on the road can obscure lane markings and other key road features. In some cases, wind can blow debris, dust, or snow across the road, causing sudden visibility problems. In other cases, the windshield and windows of a vehicle may fog up from the inside, caused by the temperature difference between the inside and outside of the car in the presence of high humidity. This can be mitigated by using the defrost function, but it still presents temporary visibility challenges. Low light conditions (such as at dawn, dusk, or during periods of heavy cloud cover) can also reduce visibility, making it harder to see obstacles or other vehicles. Finally, glare from the sun, especially when the sun is low on the horizon, can be blinding. Wet roads often exacerbate this, as they can reflect sunlight directly into the driver's eyes. Each of these factors can significantly impair a driver's ability to see, requiring extra caution and slower driving speeds.
[0003] Traditional methods for cleaning vehicle windshields, such as windshield wipers and defrosters, are inadequate for heavy rain, snow, or ice. In heavy rain, wipers may not clear water quickly enough, reducing visibility. Snow and ice can accumulate much faster than wipers can handle. Defrosters operate slowly, especially in extremely cold conditions. Temperature differences between the vehicle's interior and exterior can cause persistent fogging in high humidity, which defrosters may not clear quickly enough. Enhanced solutions for severe weather are needed. Summary of the Invention
[0004] This disclosure describes systems and methods for enhancing driver visibility in adverse weather conditions.
[0005] Embodiments of this disclosure provide a method for enhancing driver visibility. The method includes displaying a first image of the environment surrounding a vehicle. The method further includes the vehicle determining a first environmental condition associated with the environment surrounding the vehicle, and determining that the first environmental condition is causing reduced driver visibility for the vehicle's driver. Thereafter, the method further includes the vehicle determining the type of the environmental condition and capturing first data from one or more sensors of the vehicle based on the type of the environmental condition. The vehicle then uses the first data to enhance the first image to generate a second image and displays the second image.
[0006] In another example, an augmented reality device is provided, comprising one or more processors and one or more sensors coupled to the processors. The augmented reality device also includes a memory device coupled to the processors, storing instructions that, when executed by the processors, cause the device to display a first image of the vehicle's surroundings, detect environmental conditions that reduce driver visibility, determine the type of the environmental conditions, capture first data associated with the environmental conditions based on the type, process the first data using a first algorithm to generate processed data, and use the processed data to enhance the first image. Therefore, embodiments of this disclosure provide methods and systems for context-aware enhancement of visual data associated with the external environment of a vehicle to enhance visibility while driving.
[0007] In yet another example, a vehicle is provided that includes one or more processors and one or more sensors coupled to the processors. The vehicle also includes a memory device coupled to the processors, storing instructions that, when executed by the processors, cause the vehicle to couple to an augmented reality device worn by the driver; send first image data to the augmented reality device, the first image data including information about the environment surrounding the vehicle; determine a current environmental condition associated with the environment; determine that the current environmental condition is causing reduced visibility for the driver; determine the type of the environmental condition; and determine first data based on the type of the environmental condition. The vehicle then processes the first data to determine one or more settings for the augmented reality device and transmits the one or more settings to the augmented reality device, wherein the one or more settings cause the augmented reality device to modify the first image data.
[0008] These and other advantages of this disclosure are provided in detail herein. Attached Figure Description
[0009] Specific embodiments are illustrated with reference to the accompanying drawings. The same reference numerals may be used to indicate similar or identical items. Various embodiments may utilize elements and / or components other than those shown in the drawings, and some elements and / or components may not be present in various embodiments. Elements and / or components in the drawings are not necessarily drawn to scale. Throughout this disclosure, singular and plural terms may be used interchangeably depending on the context.
[0010] Figure 1 An environment in which embodiments of the present disclosure may be implemented is shown.
[0011] Figure 2 A block diagram of a vehicle according to an embodiment of the present disclosure is shown.
[0012] Figure 3 A block diagram of an augmented reality device according to an embodiment of the present disclosure is shown.
[0013] Figure 4 A high-level flowchart of a process for enhancing driver visibility according to an embodiment of the present disclosure is shown.
[0014] Figure 5 Some exemplary use cases for enhancing driver visibility are shown according to another embodiment of this disclosure.
[0015] Figure 6 A block diagram of a server according to an embodiment of the present disclosure is shown. Detailed Implementation
[0016] The present disclosure will be described more fully below with reference to the accompanying drawings, which illustrate exemplary embodiments of the present disclosure and are not intended to be limiting.
[0017] Figure 1 An environment 100 in which embodiments of the present disclosure may be implemented is shown. Vehicle 102 may be any passenger or commercial vehicle, such as a car, truck, tanker, bus, etc. Environment 100 may also include a control server 104. Control server 104 may be part of a cloud-based computing infrastructure and may be associated with and / or include a Telematics Service Delivery Network (SDN) that provides digital data services to vehicle 102. References are made below. Figure 6 Provide details of control server 104.
[0018] Environment 100 may also include a user device 112. User device 112 may be a mobile phone, tablet, personal computer, smart key fob, etc. User device 112 may be associated with a user 110 of vehicle 102. User 110 may be the driver of vehicle 102 or a passenger in vehicle 102. User device 112 may receive information from vehicle 102 and / or control server 104. User device 112 may have a dedicated application installed thereon, which can interface with vehicle 102 to download and display various types of vehicle-generated information and other control data. In one embodiment, vehicle 102 may communicate directly with user device 112 to send and receive data without network 108 and / or server 104.
[0019] Environment 100 may also include network 108. Network 108 illustrates an example communication infrastructure in which connected devices discussed in various embodiments of this disclosure may communicate. Network 108 may be and / or include the Internet, a private network, a public network, or other configurations operating using any one or more known communication protocols such as Transmission Control Protocol / Internet Protocol (TCP / IP), Bluetooth, etc. ® Bluetooth ® Low Energy (BLE), Wi-Fi based on the IEEE 802.11 standard, Ultra Wideband (UWB), and cellular technologies such as Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), High Speed Packet Access (HSPDA), Long Term Evolution (LTE), Global System for Mobile Communications (GSM), and 5G are just a few examples.
[0020] Environment 100 may also include an augmented reality (AR) device 114. User 110 may use the AR device 114 while driving to enhance his / her awareness and assist driving. In some embodiments, the AR device 114 may provide user 110 with audio messages describing environmental conditions and / or audio information about captured images of the external environment of vehicle 102. The AR device 114 may communicate directly with vehicle 102 without any intermediate network or device. In embodiments, the AR device 114 may also communicate with server 104 via network 108. The AR device may exchange control, configuration, and / or user profile information with vehicle 102 and / or server 104. Refer to the following... Figure 3 Details of augmented reality device 114 are provided.
[0021] Vehicle 102 may include multiple units, including but not limited to an automotive computer, a vehicle control unit (VCU), and a detection unit. See below for reference. Figure 2Details of vehicle 102 are provided.
[0022] Figure 2 A block diagram of a vehicle 102 in which embodiments of the present disclosure may be implemented is shown. The vehicle 102 may include multiple units, including but not limited to an automotive computer 208, a vehicle control unit (VCU) 210, and an infotainment unit 238. The VCU 210 may include multiple electronic control units (ECUs) 214 configured to communicate with the automotive computer 208.
[0023] In some embodiments, a user device such as a mobile phone, laptop computer, smart key fob, etc., can be configured to connect to the vehicle computer 208. The user device can communicate via one or more wireless connections, and / or via near field communication (NFC) protocol, Bluetooth, etc. ® Protocols, Wi-Fi, Ultra-Wideband (UWB), and other possible data connectivity and sharing technologies can be used to directly connect to vehicle 102.
[0024] According to this disclosure, the vehicle computer 208 can be installed anywhere in the vehicle 102. The vehicle computer 208 may be or include an electronic vehicle controller having one or more processors 202, one or more memory devices 204, and one or more transceivers 206.
[0025] Processor 202 may be configured to communicate with one or more memory devices (e.g., memory 204 and / or memory 205) configured to communicate with a corresponding computing system. Figure 2 The processor 202 may communicate with one or more external databases (not shown in the diagram). The processor 202 may utilize the memory 204 to store programs and / or data in code form to perform operations according to this disclosure. The memory 204 may be a non-transitory computer-readable storage medium or memory storing vehicle control program code. The memory 204 may include any or a combination of volatile memory elements (e.g., dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), etc.) and may include any one or more non-volatile memory elements (e.g., erasable programmable read-only memory (EPROM), flash memory, electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), etc.). In some embodiments, the memory 204 may include modules 245 that may implement various embodiments of this disclosure. Modules 245 may include instructions that can be executed by the processor 202 to implement various embodiments of this disclosure.
[0026] The vehicle computer 208 may also include a transceiver 206. The transceiver 206 may be configured to receive information / input from one or more external devices or systems (e.g., user device 208, external server, etc.). Furthermore, the transceiver 206 may transmit notifications, requests, signals, etc., to external devices or systems. Additionally, the transceiver 206 may be configured to receive information / input from vehicle components (such as vehicle sensing system 232, one or more ECUs 214, etc.). Furthermore, the transceiver 206 may transmit signals (e.g., command signals) or notifications to vehicle components such as BCM 220, infotainment system 238, etc.
[0027] In some embodiments, VCU 210 may share a power and / or communication bus with vehicle computer 208 and may be configured and / or programmed to coordinate data between vehicle systems, connected servers, etc. VCU 210 may include or communicate with any combination of ECUs 214, such as BCM 220, Engine Control Module (ECM) 222, Transmission Control Module (TCM) 224, Telematics Control Unit (TCU) 226, Driver Assist Technology (DAT) Controller 228, etc. VCU 210 may also include and / or communicate with a Vehicle Sensing System (VPS) 230, which may connect to and / or control one or more vehicle sensing systems 232. The vehicle sensing system 232 may include one or more vehicle sensors, including but not limited to radio detection and ranging (LiDAR or “radar”) sensors configured to use radio waves to detect and locate objects inside and outside the vehicle 102, seating area latch sensors, seating area sensors, light detection and ranging (“LiDAR”) sensors, door sensors, proximity sensors, temperature sensors, wheel sensors, one or more ambient weather or temperature sensors, interior and exterior cameras, steering wheel sensors, etc. Sensors as part of the vehicle sensing system 232 may be coupled to the vehicle 102 at one or more locations in one or more ways. For example, various sensors of the vehicle sensing system 232 may be integrated into various subsystems of the vehicle 102 (such as doors, mirrors, roof, etc.) or attached to the vehicle 102 using suitable mounting mechanisms. In some embodiments, various sensors of the vehicle sensing system 232 may be located at the front, rear, sides, top, bottom, and underside of the vehicle 102. The location of the sensors may depend on their function. For example, sensors monitoring the area beneath the vehicle can be attached to the underside of vehicle 102, while sensors monitoring areas on either side of vehicle 102 can be mounted or integrated into the doors of vehicle 102. Vehicle sensing system 232 may also include one or more road noise sensors, such as accelerometers coupled to various mechanical components and / or systems of vehicle 102. Those skilled in the art will recognize that sensors can be coupled to the vehicle in various different ways and locations besides those mentioned above.
[0028] In some embodiments, VCU 210 can control vehicle operation aspects and implement one or more instruction sets received from server 104, user device 112, or from one or more instruction sets stored in memory 204.
[0029] TCU 226 can be configured and / or programmed to provide vehicle connectivity to wireless computing systems on and outside the vehicle 102, and may include a navigation (NAV) receiver 234 for receiving and processing GPS signals, BLE ® Module (BLEM) 236, Wi-Fi transceiver, UWB transceiver and / or may be configured to be used in vehicle 102 with other systems (e.g., vehicle key fob). Figure 2 Other wireless transceivers (not shown in the image), external servers, user devices, etc., for wireless communication (including cellular communication) between computers and modules. Figure 2 (Not shown in the image). TCU 226 can communicate with ECU 214 via a wired or wireless bus. In some respects, TCU 226 can be configured to determine the real-time vehicle geolocation, for example, via NAV receiver 234.
[0030] ECU 214 can control various aspects of vehicle operation and communication using inputs from the human driver, inputs from the vehicle computer 208, and / or wireless signal inputs received from other connected devices (such as server 206) via a wireless connection.
[0031] The BCM 220 typically integrates sensors, vehicle performance indicators, and variable reactors associated with vehicle systems. It may also include processor-based power distribution circuitry that controls functions associated with the vehicle body, such as lights, windows, safety devices, cameras, audio systems, speakers, wipers, door locks and entry controls, and various comfort controls. The BCM 220 can also operate as a gateway for bus and network interfaces to communicate with remote ECUs ( Figure 2 (Not shown in the image) Interaction.
[0032] The DAT controller 228 and / or the autonomous driving system 240 can provide Level 1 to Level 5 automated driving and driver assistance functionality, which may include features such as active parking assist, vehicle reversing assist, and / or adaptive cruise control. The DAT controller 228 can also provide various aspects of user and environmental inputs that can be used for user authentication.
[0033] In some embodiments, the vehicle computer 208 may be connected to the infotainment system 238 (or vehicle human-machine interface (HMI)). The infotainment system 238 may include a touchscreen interface portion and may include voice recognition features and biometric identification capabilities, which may identify the user based on facial recognition, voice recognition, fingerprint identification, or other biometric means. In other aspects, the infotainment system 238 may be further configured to receive user commands via the touchscreen interface portion and / or output or display notifications, navigation maps, etc., on the touchscreen interface portion. In some embodiments, the user device 112 may provide an HMI interface.
[0034] In one embodiment, vehicle 102 may further include a driver visibility system 242. The driver visibility system 242 may include algorithms that process specific types of data received from sensing system 232 and output one or more settings information that can be transmitted to augmented reality device 114. Augmented reality device 114 can then use the settings information to modify its behavior / configuration. In one embodiment, sensing system 232 may continuously monitor multiple parameters associated with the vehicle's external environment and provide this raw data to driver visibility system 242. Driver visibility system 242 may then select specific data from the raw data and process that data using one or more algorithms to generate settings information. For example, sensing system 232 may monitor one or more environmental parameters, such as the presence and amount of rain, glare, fog, smoke, snow, etc., that can affect driver visibility, and generate raw data associated with these parameters. This raw data can then be provided to driver visibility system 242. Based on the received data, driver visibility system 242 may determine which data from the raw sensor data will be used to mitigate the environmental parameters attributable to the reduction in driver visibility. The driver visibility system can then generate setup data for the augmented reality device 114. For example, if environmental parameters are related to glare caused by sunlight falling on the road, the driver visibility system 242 can generate setup data for manipulating the polarization of the lenses of the augmented reality device 114. The setup data can be sent to the augmented reality device 114, and the augmented reality device can modify its polarization to provide the user with a clear image without glare. Although the augmented reality device 114 is shown as a wearable device, it should be noted that the augmented reality device 114 can be implemented in other ways, such as a head-up display of vehicle 102 and / or as a vehicle windshield.
[0035] The computing system architecture of the automotive computer 208 and / or VCU 210 can omit certain computing modules. This should be easily understood. Figure 2 The computing environment depicted herein is an example of possible implementations according to this disclosure and should therefore not be considered limiting or exclusive.
[0036] In addition to the components mentioned above, vehicle 102 may also have numerous mechanical systems and subsystems. A chassis or frame may form the backbone of vehicle 102 and support the body and other components of vehicle 102. Vehicle 102 may include an engine that converts fuel into mechanical power to propel the vehicle forward. The engine includes various components such as the engine block, pistons, valves, and spark plugs. Vehicle 102 may also include a transmission system. The transmission system transmits power from the engine to the wheels. It includes a clutch, gearbox, drive shaft, differential, and other components. The transmission adjusts the power output to suit the vehicle's speed and load. Vehicle 102 may also include a suspension system. The suspension system absorbs shocks and maintains contact between the tires and the road, thus providing a smooth ride. It includes components such as springs, shock absorbers, and linkages. Vehicle 102 also includes a vehicle stopping system that allows the driver to decelerate or stop vehicle 102. It includes components such as pedals, master cylinders, lines, and bushings or shoes. Vehicle 102 also includes a steering system that allows the driver to guide the vehicle. The steering system includes components such as a steering wheel, steering column, rack and pinion, and tie rods. Vehicle 102 may also include an exhaust system for removing and filtering exhaust gases produced by the engine. It includes an exhaust manifold, catalytic converter, muffler, and exhaust tailpipe, among other components. Vehicle 102 also includes a cooling system to prevent overheating of the engine and / or battery. It includes components such as a radiator, water pump, thermostat, and coolant. Vehicle 102 also includes a cooling system for storing fuel and supplying fuel to the engine. It includes a fuel tank, fuel pump, fuel filter, and fuel injectors. The electrical system of vehicle 102 powers the vehicle's electrical components. It may include a battery, alternator, starter motor, and wiring. The heating, ventilation, and air conditioning (HVAC) system controls the temperature inside vehicle 102. It includes a heater core, blower motor, and air conditioning compressor. In some embodiments, the vehicle may be an electric vehicle (EV) or a hybrid vehicle, and in either case, some of the aforementioned components will be replaced by an electric motor and a high-voltage battery. All the mechanical components working together ensure optimal operation of vehicle 102.
[0037] Figure 3A functional block diagram of an augmented reality device 114 according to an embodiment of the present disclosure is shown. The augmented reality device 114 may include one or more hardware processors 302. Processor 302 may be a single-core or multi-core device (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof). Processor 302 manages data from sensors, executes AR applications, and seamlessly renders augmented visual effects onto the display of the augmented reality device 114. The augmented reality device 114 may also include multiple input / output devices 304. Input / output devices 304 may include displays that present augmented content to a user. Displays may be implemented using: optical see-through displays (e.g., transparent displays, such as those used in smart glasses, which allow users to see the real world directly while overlaying digital information), video see-through displays (e.g., those that use cameras to capture the real world and then display it on a screen with overlaid augmented elements), or head-mounted displays (HMDs) (e.g., those worn on the head and may include both optical see-through and video see-through technologies. HMDs provide an immersive augmented reality experience by overlaying augmented content onto the user's field of vision). In some embodiments, the enhanced input / output device 303 may also include a touch screen, a voice control system, a gesture recognition system, a handheld controller, etc.
[0038] Augmented reality device 114 may also include one or more sensors 306. Sensors 306 may include one or more cameras, gyroscopes, accelerometers, magnetometers, depth sensors, ambient light sensors, etc. Augmented reality device 114 may also include one or more memory devices 308. Memory devices 308 may include any or a combination of volatile memory elements (e.g., dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), etc.) and may include any one or more non-volatile memory elements (e.g., erasable programmable read-only memory (EPROM), flash memory, electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), etc.). In some embodiments, memory device 308 may include modules 312 that can implement various embodiments of the present disclosure. Modules 312 may include instructions and algorithms executable by processor 302 to implement various embodiments of the present disclosure. Memory device 308 may also include an operating system, development platform, applications, etc., associated with augmented reality device 114.
[0039] The augmented reality device 114 may also include a communication interface 310. The communication interface 310 can use any of the known communication protocols (such as Wi-Fi, Bluetooth). ®The augmented reality device (ARD) communicates with vehicle 102, server 104, user device 110, and / or network 108 via a cellular network. The ARD device also includes multiple mechanical components 314, such as a housing or frame that forms the outer shell or frame of the ARD device 114 and holds all internal components together; one or more lenses for focusing and projecting augmented content into the user's field of vision; mounting mechanisms, such as adjustable straps, headbands, or ear hooks, for securing the ARD device 114 to the user's head and maintaining the correct position of the display relative to the user's eyes; adjustable components, such as an interpupillary distance (IPD) adjuster, which allows the user to set the distance between the lenses to match their interpupillary distance, ensuring a comfortable and clear viewing experience; a cooling system, such as a small fan or heat sink, for dissipating heat and maintaining optimal operating temperature; buttons and controls, such as physical buttons, dials, or touch-sensitive areas on the device, allowing the user to interact with the ARD device 114; connectors and ports, such as USB ports, audio jacks, and charging connectors; and hinges and joints that allow for folding and adjusting the arms. All components of the ARD device can communicate via a communication bus 316.
[0040] As mentioned above, a driver's visibility can be affected by adverse weather conditions. In such situations, a clear view of the road and the surrounding environment will greatly enhance the driving experience. Figure 4A high-level flowchart of a process 400 for enhancing visibility according to an embodiment of this disclosure is shown. Considering a driver operating a vehicle and the weather outside is sunny and bright. At some point, considering a change in external weather, heavy rain begins. This reduces the driver's visibility of the road ahead and the external environment typically surrounding the vehicle. The driver can then use an augmented reality device 114 to enhance his / her visibility of the road and surrounding environment. When the driver wears the augmented reality device, the device is communicatively coupled to the vehicle. The augmented reality device can then use data received from its own sensors and / or data received from the vehicle's sensors to generate an image of the vehicle's surrounding environment and display that image on the augmented reality device's display. Thus, the driver is now able to see a real-time image of the road ahead and other parts of the external environment surrounding the vehicle. At step 402, the vehicle and / or the augmented reality device can detect environmental conditions affecting the driver's visibility (e.g., heavy rain obscuring road markings and other objects in the environment). Based on the type of weather conditions and the nature of adverse weather conditions (e.g., heavy rain in this example), the vehicle and / or augmented reality device can activate a specific set of sensors and / or collect data from a specific set of sensors (e.g., multispectral sensors such as RGB cameras, infrared sensors, and radar sensors) at step 404. Then, at step 406, the vehicle and / or augmented reality device can process the data using one or more specific algorithms to generate processed data including information about the location and type of road markings and objects in the environment. At step 408, the vehicle and / or augmented reality device can determine one or more setting data for one or more parameters of the augmented reality device. At step 410, the vehicle can send the setting data and / or processed data from the sensors to the augmented reality device. The augmented reality device can then use the setting information to change the configuration of one or more parameters. If the processed data is sent to the augmented reality device, the augmented reality device can then overlay the image of the surrounding environment with visual markings for road markings and obstacles. Therefore, the image provided by the augmented reality device will now present a more comprehensive and clearer view of the vehicle's surroundings, thus mitigating visibility problems caused by heavy rain. This enhances the driver's driving experience.
[0041] According to embodiments of the present invention, there are several ways in which augmented reality devices can enhance a driver's visibility. In one case, the augmented reality device may modify some of the parameters of its components based on received data to update / modify the image it displays. In other embodiments, the augmented reality device may add information to its displayed image based on received data to provide the driver with a more comprehensive picture. The methods or processes by which an augmented reality device enhances a driver's visibility may depend on the nature and type of weather conditions that cause reduced visibility. Depending on the type of weather or environmental conditions, the vehicle and / or the augmented reality device may select specific processes and / or combinations of processes to mitigate the impact of environmental conditions on the driver's visibility. Figure 5 Examples of various environmental conditions and processes performed by vehicles and / or augmented reality devices to mitigate the effects of environmental conditions and enhance driver visibility are shown.
[0042] The driver can begin driving the vehicle (step 502) and wear the augmented reality device (ARD) to assist driving. The ARD can be activated at step 504. In some embodiments, the driver can configure one or more parameters of the ARD, such as brightness, contrast, enhancement type, etc., based on the driver's personal preferences and / or driving style. As part of the activation process, the ARD can be communicatively coupled to the vehicle. Once coupled to the vehicle, the ARD and the vehicle can exchange some settings and capability data with each other. This informs the vehicle and the ARD what features each of them supports. In some embodiments, the ARD and / or the vehicle can also exchange user profile data. For example, the ARD can be personalized for the preferences of a first user. If the first user is now driving another vehicle (e.g., a rental vehicle), the ARD can transmit user profile and preference data to that other vehicle. The other vehicle can then send its capability data to the ARD (e.g., information about its sensors and the features it can support). Once the ARD and the vehicle are “synchronized,” the ARD can use data captured by its own sensors and / or the vehicle's sensors to generate and display a first image of the environment around the vehicle, at step 506. For example, one or more front-facing cameras of the vehicle (e.g., a dashcam, etc.) can be used to capture the first image. The first image may include a representation of the road and other objects in the environment, which may be outside the driver's field of vision (e.g., blind spots) if the driver is not wearing an augmented reality device.
[0043] As described above, the process by which augmented reality devices and / or vehicles mitigate the effects of weather conditions and enhance driver visibility depends on the nature of the weather conditions / phenomena. In the first example, it is considered that weather conditions cause glare from the sun reflected from a horizontal surface (such as a road) to impair driver visibility. In this case, the vehicle and / or augmented reality device can determine that glare from the sun causes reduced driver visibility (step 508). For example, the vehicle and / or augmented reality device's photometric sensors, cameras, or lidar systems can detect the presence and direction of glare. Photometric sensors measure the intensity of light, and by comparing the intensity of light from different directions, they can identify areas of excessive brightness, thus indicating glare. Cameras can capture images of the environment, and image processing algorithms can analyze these images to detect glare. These algorithms typically look for high-intensity areas and specific patterns associated with glare. Lidar systems can detect glare by measuring the reflection of a laser beam. At step 510, the vehicle and / or augmented reality device can determine first data associated with the environmental conditions. The first data may include information about the intensity and direction of the glare. Based on the first data, the vehicle and / or augmented reality device can determine the setting information for the polarization filter of the augmented reality device at step 512. For example, a 90-degree rule can be used to adjust the polarization filter of the augmented reality device. The setting information can be sent to the augmented reality device (step 514). The augmented reality device can then implement the new polarization setting at step 516 and display the modified image to the user at step 518. The modified image will now display the surrounding environment without the influence of glare, thereby providing enhanced visibility for the driver. It should be noted that the augmented reality device and / or vehicle continuously monitor glare in real time and adjust the polarization of the lens of the augmented reality device as needed. Therefore, the augmented reality device performs dynamic polarization adjustment while the vehicle is in motion, so that an image without glare is always presented to the driver.
[0044] In the second example, considering the weather conditions involving heavy rain that obstructs the driver's visibility, and the driver being unable to clearly see road signs or other objects in the environment, at step 520, the vehicle and / or augmented reality device can detect the presence of heavy rain. For example, an optical sensor mounted on the vehicle's windshield can be used to detect the presence and amount of rain. To mitigate the reduced visibility caused by heavy rain, at step 522, the augmented reality device and / or the vehicle can capture multispectral data from one or more sensors. For example, the multispectral data can include data in the visible and invisible spectra (e.g., infrared, radar, etc.). The vehicle and / or the augmented reality device can process this multispectral data (step 524) and determine the positions of road signs and other objects in the environment. At step 526, the augmented reality device can enhance the first image with data about the road signs and other objects determined from the multispectral data. The enhanced image is then displayed to the driver at step 528. Thus, the driver now has a clearer view of the road and the surrounding environment. In this case, while the driver is operating the vehicle, the vehicle and / or the augmented reality device also continuously monitors the presence and intensity of rain and the external environment in real time and updates the positions of road signs and objects in the environment.
[0045] In the third example, environmental conditions are considered in relation to the ambient light level inside the vehicle. At step 530, the augmented reality device can (e.g., using an ambient light sensor) continuously monitor the ambient light level inside the vehicle. At step 532, the augmented reality device determines whether the ambient light level is above or below a threshold. The threshold can be set by the driver according to his / her preferences. Based on whether the current ambient light level inside the vehicle is above or below the threshold, the augmented reality device can determine brightness and contrast settings at step 534. Then, at step 536, the augmented reality device can adjust the brightness and contrast settings accordingly. The augmented reality device continuously monitors the ambient light level inside the vehicle and adjusts the brightness and contrast settings to provide the driver with the best viewing experience.
[0046] In the fourth example, the environmental condition is considered to be fog that reduces the driver's visibility. In this case, at step 538, the vehicle and / or the augmented reality device can determine the presence of fog and identify an incomplete representation of the environment presented by the image currently displayed by the augmented reality device. In this case, the vehicle and / or the augmented reality device can capture data from infrared sensors that can penetrate the fog and use the infrared data to enhance the image presented to the user. Additionally, at step 540, the vehicle can also capture data from one or more external cameras of the vehicle to capture additional images. Then, at step 542, the augmented reality device can use these additional images to enhance the first image to present a more comprehensive view of the environment to the driver. This process can be performed continuously, and the enhanced image is displayed to the driver in real time.
[0047] In the fifth example, consider that the driver is driving at night and there are one or more other weather conditions (e.g., heavy fog / rain at night). In this case, the driver's visibility may be further reduced due to the low light or no light conditions at night. The augmented reality device can continuously monitor the ambient light conditions inside and / or outside the vehicle and adjust the brightness and contrast settings of the augmented reality device. For example, the presence or absence of streetlights on the road can change the ambient light conditions inside the vehicle. The augmented reality device can address this problem by adjusting the brightness and contrast settings in real time as the vehicle travels along the road. At step 546, the vehicle and / or the augmented reality device can detect that it is raining / foggy and that it is nighttime. The vehicle and / or the augmented reality device can infer that the rain / fog combined with the low light conditions at night causes reduced visibility for the driver. At step 548, the vehicle and / or the augmented reality device can capture data on the vehicle's external environment from one or more radar sensors, as radar may be best suited for capturing external environment data under given weather conditions. At step 550, the vehicle and / or the augmented reality device can detect the presence and location of objects in the captured radar data. Subsequently, at step 552, the augmented reality device can modify the first image to include visual representations of objects. The driver is now able to identify various objects in the external environment more clearly and accurately, thereby enhancing his visibility and driving experience. In some cases, the augmented reality device may also need to continuously adjust the polarization settings of its lenses to account for glare from headlights of oncoming vehicles, streetlights, or other strong light in the environment.
[0048] It should be noted that the environmental conditions described above are exemplary, and other environmental conditions may also occur. Similarly, in those cases, the vehicle and / or augmented reality device can determine the nature, type, and / or severity of the environmental condition, and determine which sensor data to use to mitigate its effects and enhance driver visibility, and how to process the sensor data, including which specific processing algorithm will be used. In other embodiments, instead of an augmented reality device, or in addition to an augmented reality device, the vehicle's head-up display may be used to display various images.
[0049] Figure 6 An example control server 600 is depicted according to one or more exemplary embodiments of the present disclosure, on which any of one or more technologies (e.g., methods) can be performed or on which the methods described above in conjunction with vehicle 102 can be performed. Figure 1A block diagram of the control server 104. In other embodiments, server 600 may act as a standalone device or may be connected to other servers (e.g., networked). In a networked deployment, server 600 may operate in a server-client network environment as a server machine, a client machine, or both. In the example, server 600 may act as a peer-to-peer (P2P) (or other distributed) network environment. Server 600 may be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, smart keychain, wearable computing device, network device, network router, switch, or bridge, or any machine capable of executing instructions (continuously or otherwise) specifying actions to be taken by the server (such as a base station). Furthermore, while only a single server is described, the term "server" should also be considered to include any collection of servers that individually or jointly execute a set (or more sets) of instructions for performing any one or more of the methodologies discussed herein, such as those configured for cloud computing, Software as a Service (SaaS), or other computer clusters.
[0050] The examples described herein may include logic or components, modules, or mechanisms, or may operate on logic or components, modules, or mechanisms. A module is a tangible entity (e.g., hardware) capable of performing a specified operation during operation. A module includes hardware. In the examples, the hardware may be specifically configured to perform a specific operation (e.g., hardwired). In another example, the hardware may include a configurable execution unit (e.g., a transistor, circuitry, etc.) and a computer-readable medium containing instructions that configure the execution unit to perform a specific task when in operation. The configuration may occur under the guidance of the execution unit or loading mechanism. Thus, when the device is in operation, the execution unit is communicatively coupled to the computer-readable medium. In this example, the execution unit may be a member of more than one module. For example, under operation, the execution unit may be configured at one point in time to implement a first module via a first set of instructions, and at a second point in time to be reconfigured at a second point in time to implement a second module via a second set of instructions.
[0051] Server (e.g., computer system) 600 may include a hardware processor 602 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), main memory 604, and static memory 606, some or all of which may communicate with each other via interconnect (e.g., bus) 608. Server 600 may also include a graphics display device 610, an alphanumeric input device 612 (e.g., a keyboard), and a user interface (UI) navigation device 614 (e.g., a mouse). In this example, the graphics display device 610, the alphanumeric input device 612, and the UI navigation device 614 may be a touchscreen display. Server 600 may additionally include a storage device (i.e., a drive unit) 616, a network interface device / transceiver 620 coupled to an antenna, and one or more sensors 628, such as a global positioning system (GPS) sensor, a compass, an accelerometer, or another sensor. Server 600 may include output controller 634, such as serial (e.g., Universal Serial Bus (USB)), parallel, or other wired or wireless (e.g., infrared (IR)), near field communication (NFC) connections, to communicate with or control one or more peripheral devices (e.g., printers, card readers, etc.).
[0052] Storage device 616 may include machine-readable medium 622 thereon storing one or more sets of data structures or instructions (e.g., software) that embody or utilize any one or more of the techniques or functions described herein. The instructions may also reside wholly or at least partially within main memory 604, static memory 606, or hardware processor 602 during execution of the instructions by server 600. In this example, one or any combination of hardware processor 602, main memory 604, static memory 606, or storage device 616 may constitute the machine-readable medium.
[0053] Although machine-readable medium 622 is shown as a single medium, the term "machine-readable medium" can include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store one or more instructions.
[0054] Various embodiments may be implemented wholly or partially in software and / or firmware. This software and / or firmware may take the form of instructions contained in or on a non-transitory computer-readable storage medium. Those instructions may then be read and executed by one or more processors to enable the performance of the operations described herein. The instructions may be in any suitable form, such as, but not limited to, source code, compiled code, interpreted code, executable code, static code, dynamic code, etc. Such computer-readable medium may include any tangible non-transitory medium for storing information in a form readable by one or more computers, such as, but not limited to, read-only memory (ROM); random access memory (RAM); disk storage media; optical storage media; flash memory; etc.
[0055] The term "machine-readable medium" can include any medium having the following properties: capable of storing, encoding, or carrying instructions executable by server 600; and causing server 600 to perform any one or more of the techniques disclosed herein; or capable of storing, encoding, or carrying data structures used by or associated with such instructions. Examples of non-limiting machine-readable media can include solid-state memory as well as optical and magnetic media. In examples, high-capacity machine-readable media includes machine-readable media having a plurality of particles having rest masses. Specific examples of large-scale machine-readable media can include non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM) or electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.
[0056] Instructions can also be transmitted or received on a communication network using a transmission medium via a network interface device / transceiver 620 that utilizes any of several transmission protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Exemplary communication networks may include local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), mobile phone networks (e.g., cellular networks), conventional telephone (POTS) networks, wireless data networks (e.g., the IEEE 802.11 series of standards known as Wi-Fi®, the IEEE 802.16 series of standards known as WiMax®), the IEEE 802.15.4 series of standards, and peer-to-peer (P2P) networks, etc. In the example, the network interface device / transceiver 620 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas for connection to the communication network. In the example, the network interface device / transceiver 620 may include multiple antennas to communicate wirelessly using at least one of the following: Single-Input Multiple-Output (SIMO) technology, Multiple-Input Multiple-Output (MIMO) technology, or Multiple-Input Single-Output (MISO) technology. The term "transmission medium" should be considered to include any intangible medium capable of storing, encoding, or carrying instructions executed by server 600 and comprising digital or analog communication signals or other intangible media to facilitate communication of such software. The operations and processes described and shown above may be implemented or performed in any suitable order as needed in various embodiments. Additionally, in some embodiments, at least a portion of the operations may be performed in parallel. Furthermore, in some embodiments, fewer or more operations than those described may be performed.
[0057] It should be noted that the vehicle implements and / or performs the operations described herein in accordance with the owner's manual and safety guidelines. Additionally, any action taken by the vehicle owner / driver based on recommendations or notices provided by the vehicle should comply with all rules specific to the vehicle's location and operation (e.g., federal, state, national, city, etc.). Recommendations or notices provided by the vehicle should be considered as advice and followed only in accordance with any rules specific to the vehicle's location and operation. In the foregoing disclosure, reference has been made to the accompanying drawings, which form part of the foregoing disclosure, illustrating specific implementations in which the present disclosure may be practiced. It should be understood that other implementations may be utilized and structural changes may be made without departing from the scope of the present disclosure. References to “an embodiment,” “embodiment,” “example embodiment,” etc., in this specification indicate that the described embodiment may include a particular feature, structure, or characteristic, but each embodiment may not necessarily include said particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Moreover, when features, structures, or characteristics are described in connection with embodiments, those skilled in the art will recognize such features, structures, or characteristics in conjunction with other embodiments, whether explicitly described or not.
[0058] Additionally, where appropriate, the functions described herein may be performed in one or more of the following: hardware, software, firmware, digital components, or analog components. For example, one or more application-specific integrated circuits (ASICs) may be programmed to implement one or more of the systems and programs described herein. Throughout the specification and claims, certain terms are used to refer to specific system components. As those skilled in the art will appreciate, components may be referred to by different names. This document is not intended to distinguish between components with different names but identical functions.
[0059] It should also be understood that the word “example” as used herein is intended to be non-exclusive and non-restrictive in nature. More specifically, the word “example” as used herein refers to one of several examples, and it should be understood that there is no undue emphasis or preference for any particular example described.
[0060] Computer-readable media (also known as processor-readable media) include any non-transitory (e.g., tangible) medium that contributes to providing data (e.g., instructions) that can be read by a computer (e.g., by the computer's processor). Such media can take many forms, including but not limited to non-volatile and volatile media. A computing device may include computer-executable instructions, wherein the instructions can be executed by one or more computing devices (such as those listed above) and stored on a computer-readable medium.
[0061] Regarding the processes, systems, methods, heuristics, etc., described herein, it should be understood that although the steps of such processes, etc., have been described as occurring in a certain ordered order, such processes can be practiced with the described steps performed in a different order than that described herein. It should also be understood that some steps may be performed simultaneously, other steps may be added, or some steps described herein may be omitted. In other words, the description of processes herein is provided for the purpose of illustrating various embodiments and should in no way be construed as limiting the claims.
[0062] Therefore, it should be understood that the above description is intended to be illustrative rather than restrictive. Many embodiments and applications beyond the examples provided will become apparent upon reading the above description. The scope should not be determined by reference to the above description, but rather by reference to the appended claims and the full scope of their equivalents. It is anticipated and expected that the techniques discussed herein will evolve in the future, and the disclosed systems and methods will be incorporated into such future embodiments. In conclusion, it should be understood that modifications and variations are possible with this application.
[0063] Unless explicitly indicated otherwise herein, all terms used in the claims are intended to be given their ordinary meaning as understood by one skilled in the art as described herein. Specifically, unless the claims explicitly limit the recitation to the contrary, the use of singular articles such as “a,” “the,” or “the” should be interpreted as one or more of the elements indicated by the recitation. Unless otherwise specifically stated or otherwise understood in the context of use, conditional language such as, in particular, “may,” “possibly,” “may,” or “can” is generally intended to express that some embodiments may include certain features, elements, and / or steps, while other embodiments may not include certain features, elements, and / or steps. Therefore, such conditional language is generally not intended to imply that one or more embodiments require each feature, element, and / or step in any way.
[0064] According to an embodiment, the first data includes data in the invisible spectrum, and wherein the one or more processors are operable to execute the instructions to further cause the vehicle to: use the first data to determine one or more objects in the current environment; generate a second image, the second image including the first image enhanced with a visual representation of each of the one or more objects, wherein the second image is displayed by the augmented reality device.
[0065] According to an embodiment, the one or more processors are operable to execute the instructions to further cause the vehicle to: capture second data, including one or more images of the current environment that are different from the first image data; and send the second data to the augmented reality device, wherein the augmented reality device displays the one or more images in addition to the first image data.
[0066] According to an embodiment, the one or more processors are operable to execute the instructions to further cause the vehicle to: receive user profile and preference data from the augmented reality device; and send vehicle capability data to the augmented reality device.
[0067] According to an embodiment, the environmental conditions include glare from sunlight, and the one or more settings include settings for the polarization value of the augmented reality device.
[0068] According to an embodiment, the one or more sensors include: a first sensor configured to capture third data in the visible spectrum; and a second sensor configured to capture fourth data in the invisible spectrum.
Claims
1. A method, the method comprising: The first image showing the environment around the vehicle; The vehicle determines the environmental conditions associated with the environment surrounding it. The vehicle determines that the environmental conditions are causing reduced driver visibility for the vehicle's driver. The type of environmental condition is determined by the vehicle. First data is captured from one or more sensors of the vehicle based on the type of environmental conditions. The first image is enhanced using the first data to generate a second image; as well as The second image is displayed.
2. The method of claim 1, further comprising: The first image and the second image are displayed on the augmented reality device worn by the driver.
3. The method of claim 1, wherein the one or more sensors comprise: A first sensor, configured to capture second data in the visible spectrum; as well as A second sensor is configured to capture third data in the invisible spectrum.
4. The method of claim 1, further comprising: Based on the environmental conditions, a first algorithm is selected to process the first data; The first data is processed using the first algorithm to generate the fourth data; as well as The first image is enhanced using the fourth data.
5. The method of claim 1, wherein the first data includes radar data, and the method further comprises: The vehicle uses the first data to determine the presence of one or more objects in the environment surrounding the vehicle; as well as The second image includes one or more visual symbols representing the one or more objects.
6. The method of claim 1, further comprising: One or more images are captured by the vehicle from one or more external cameras of the vehicle; as well as Enhancing the first image also includes displaying one or more images in addition to the first image.
7. The method of claim 1, wherein displaying the first image comprises displaying the first image on an augmented reality device worn by the driver, the method further comprising: The level of ambient light inside the vehicle is determined by the augmented reality device; as well as The brightness or contrast settings of the augmented reality device are adjusted based on the level of ambient light.
8. The method of claim 1, wherein the one or more sensors include a first sensor configured to capture fifth data in the invisible spectrum, and the first data includes infrared image data, the method further comprising: The vehicle uses the infrared image data to identify one or more objects in the environment; as well as The first image is enhanced using information about the one or more objects.
9. An augmented reality device, the augmented reality device comprising: One or more processors; One or more sensors, said one or more sensors being coupled to said one or more processors; A memory device coupled to the one or more processors, the memory device storing instructions that, when executed by the one or more processors, cause the augmented reality device to: Displays a first image of the environment surrounding the vehicle; Detect environmental conditions that reduce driver visibility; Determine the type of the environmental condition; First data associated with the environmental condition is captured based on the type of the environmental condition. The first data is processed using a first algorithm to generate the second data; as well as The first image is enhanced using the second data.
10. The augmented reality device of claim 9, wherein the one or more sensors comprise: A first sensor, configured to capture third data in the visible spectrum; as well as The second sensor is configured to capture fourth data in the invisible spectrum.
11. The augmented reality device of claim 10, wherein the augmented reality device is further operable to: The second sensor is used to capture fifth data about the environment; The fifth data is used to identify one or more objects in the environment; and The visual representation of the one or more objects is included in the first image.
12. The augmented reality device of claim 9, wherein the instructions further cause the augmented reality device to: Determine the level of ambient light inside the vehicle; The setting for the brightness or contrast value is determined based on the level of the ambient light; and Adjust the brightness value or the contrast value based on the settings.
13. The augmented reality device of claim 9, wherein the instructions further cause the augmented reality device to: Receive one or more images of the environment from the vehicle, the one or more images being different from the first image; and In addition to the first image, one or more other images are also displayed.
14. The augmented reality device of claim 9, wherein the environmental conditions include glare, and the first data includes information about the direction and intensity of the glare, wherein the instruction further causes the augmented reality device to: Based on the first data, the polarization setting of the lenses for the augmented reality device is determined; and The polarization of the lens is dynamically adjusted based on the first data.
15. A vehicle comprising: One or more processors; One or more sensors, said one or more sensors being coupled to said one or more processors; A memory device coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the vehicle to: Coupled with an augmented reality device worn by the driver of the vehicle; Send first image data to the augmented reality device, the first image data including information about the environment around the vehicle; Determine the environmental conditions associated with the environment; It has been determined that the aforementioned environmental conditions are causing reduced visibility for the driver; Determine the type of the environmental condition; The first data is determined based on the type of the environmental condition; The first data is processed to determine one or more settings for the augmented reality device; as well as The one or more settings are transmitted to the augmented reality device, wherein the one or more settings cause the augmented reality device to modify the first image data.