SYSTEMS AND METHODS FOR IMPROVING VISIBILITY WHILE DRIVING A VEHICLE

Abstract

Systems and procedures are provided to increase visibility while driving a vehicle. When driving in poor weather conditions, the driver's visibility of the road may be reduced. The driver can use an augmented reality device to assist their view of the road. The augmented reality device works in conjunction with the vehicle to determine the type of environmental / weather condition causing the reduced visibility. Based on this, the vehicle and / or the augmented reality device can use one or more of its sensors to determine additional information to augment the image displayed by the augmented reality device. The augmented image includes more details of the road and the environment surrounding the vehicle, thus providing the driver with improved visibility of the road and the surrounding environment.

Description

AREA OF TECHNOLOGY

[0001] The present disclosure relates to the field of visibility while driving a vehicle. In particular, embodiments of the present disclosure relate to systems and methods that improve visibility while driving a vehicle in adverse weather conditions. GENERAL STATE OF THE ART

[0002] Reduced visibility in poor weather conditions can make driving difficult. Fog is a major problem, as it creates a thick, opaque layer that significantly reduces the distance a driver can see ahead. Rain is another common issue, since heavy downpours can obscure the road and other vehicles, and water on the windshield can distort vision, even with wipers running at full speed. Furthermore, rain can cause glare from oncoming headlights or streetlights, further impairing visibility. Snow and ice present unique challenges. Snowfall can create a whiteout effect, where the landscape becomes a uniform white, making it difficult to distinguish the road from its surroundings.Ice on the windshield can also obstruct visibility, and snow accumulation on the road can obscure lane markings and other critical road features. In some cases, wind can blow debris, dust, or snow across the road, causing sudden visibility problems. In other cases, the windshield and windows of the vehicle can fog up from the inside, which can be caused by the temperature difference between the inside and outside of the car in the presence of high humidity. This can be mitigated by using the de-icing function, but it still presents a temporary visibility challenge. Low light conditions, such as at dawn, dusk, or under heavy cloud cover, can also reduce visibility, making it more difficult to see obstacles or other vehicles.Finally, glare from the sun, especially when it's low on the horizon, can be blinding. This is often exacerbated by wet roads, which can reflect sunlight directly into the driver's eyes. Any of these factors can significantly impair a driver's ability to see clearly, necessitating extra caution and slower driving speeds.

[0003] Conventional methods for cleaning a vehicle windshield, such as wipers and de-icer, struggle with heavy rain, snow, or ice. In heavy rain, wipers may not be able to remove the water quickly enough, resulting in reduced visibility. Snow and ice can accumulate faster than wipers can handle. De-icers may be slow to work, especially in extremely cold conditions. The temperature difference between the inside and outside of the vehicle, combined with high humidity, can cause persistent fogging, which de-icers may not be able to clear quickly enough. Improved solutions are needed for extreme weather conditions. SUMMARY

[0004] The present disclosure describes systems and methods for improving driver visibility in adverse weather conditions.

[0005] Embodiments of the present disclosure provide a method for improving driver visibility. The method includes displaying a first image of the environment surrounding a vehicle. The method further includes the vehicle determining a first environmental condition associated with the environment surrounding the vehicle and determining that the first environmental condition causes reduced driver visibility for a driver of the vehicle. The method then further includes the vehicle determining a type of environmental condition and, based on the type of environmental condition, acquiring initial data from one or more sensors of the vehicle. The vehicle then augments the first image using the initial data to generate a second image and displays the second image.

[0006] In another case, an augmented reality device is provided that includes one or more processors and one or more sensors coupled to the one or more processors.The augmented reality device further includes a storage device coupled to the one or more processors, and instructions are stored on the storage device which, when executed by the one or more processors, cause the augmented reality device to: display an initial image of an environment surrounding a vehicle, detect an environmental condition causing reduced driver visibility, determine the type of environmental condition, capture initial data associated with the environmental condition based on the type of environmental condition, process the initial data using an initial algorithm to generate secondary data, and augment the initial image using the processed data.Thus, embodiments of the present disclosure provide methods and systems for the context-aware enhancement of visual data associated with an external environment of a vehicle in order to enable improved visibility while driving the vehicle.

[0007] In yet another scenario, a vehicle is provided that includes one or more processors and one or more sensors coupled to the processor(s). The vehicle further includes a storage device coupled to the processor(s) and containing instructions that, when executed by the processor(s), cause the vehicle to: couple with an augmented reality device worn by the vehicle's driver; send initial image data to the augmented reality device, the initial image data containing information about an environment surrounding the vehicle; determine an environmental condition associated with the environment; determine that the environmental condition causes reduced driver visibility; determine the type of environmental condition; and determine initial data based on the type of environmental condition.The vehicle then processes the initial data to determine one or more settings for the augmented reality device and communicates the one or more settings to the augmented reality device, causing the augmented reality device to modify the initial image data.

[0008] These and other benefits of the present revelation are provided in detail in this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The detailed description is set forth with reference to the accompanying drawings. The use of the same reference numerals may indicate similar or identical elements. Different embodiments may use different elements and / or components than those illustrated in the drawings, and some elements and / or components may not be present in different embodiments. The elements and / or components in the figures are not necessarily drawn to scale. Throughout this disclosure, singular and plural expressions may be used interchangeably depending on the context. Fig. Figure 1 illustrates an environment in which embodiments of the present disclosure can be implemented. Fig. Figure 2 illustrates a block diagram of a vehicle according to an embodiment of the present disclosure. Fig. Figure 3 illustrates a block diagram of an augmented reality device according to an embodiment of the present disclosure. Fig. Figure 4 illustrates a higher-level flowchart of a process for increasing driver visibility according to an embodiment of the present disclosure. Fig. Figure 5 illustrates some exemplary use cases for increasing driver visibility according to another embodiment of the present disclosure. Fig. Figure 6 illustrates a block diagram of a server according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0010] The disclosure is described in more detail below with reference to the accompanying drawings, which show exemplary embodiments of the disclosure, and is not intended to be limiting.

[0011] Fig. Figure 1 illustrates an environment 100 in which the embodiments of the present disclosure may be implemented. The vehicle 102 may be any passenger car or any commercial vehicle, such as a car, truck, tanker, bus, or the like. The environment 100 may also include a control server 104. The control server 104 may be part of a cloud-based computing infrastructure and may be associated with and / or include a telematics service delivery network (Telematics SDN) that provides digital data services to the vehicle 102. Details of the control server 104 are given below with reference to Fig. 6 provided.

[0012] The environment 100 can also include a user device 112. The mobile user device 112 can be a mobile phone, tablet, personal computer, key fob, or the like. The user device 112 can be assigned to a user 110 of the vehicle 102. The user 110 can be a driver of the vehicle 102 or a passenger in the vehicle 102. The user device 112 can receive information from the vehicle 102 and / or the control server 104. A specialized application can be installed on the user device 112, which can interface with the vehicle 102 to download and display various types of vehicle-generated information and other control data. In one embodiment, the vehicle 102 can communicate directly with the user device 112 to send and receive data without requiring the network 108 and / or the server 104.

[0013] The environment 100 may further include a network 108. The network 108 illustrates an exemplary communication infrastructure in which the connected devices discussed in various embodiments of this disclosure can communicate. The network(s) 108 may be and / or include the Internet, a private network, a public network, or another configuration, operating using any one or more known communication protocols, such as Transmission Control Protocol / Internet Protocol (TCP / IP), Bluetooth, or similar technologies. ® Bluetooth ®Low Energy (BLE), Wi-Fi based on the Institute of Electrical and Electronics Engineers (IEEE) standard 802.11, Ultra-Wideband (UWB) and mobile communication technologies such as Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), High-Speed ​​Packet Access (HSPDA), Long-Term Evolution (LTE), Global System for Mobile Communications (GSM) and Fifth Generation (5G), to name a few examples.

[0014] The environment 100 may also include an augmented reality (AR) device 114. The user 110 can use the augmented reality device 114 while driving to enhance their awareness and assist them in driving. In some embodiments, the augmented reality device 114 can provide the user 110 with an audio message describing an environmental condition and / or provide audio information about a captured image of the vehicle 102's external environment. The augmented reality device 114 can communicate directly with the vehicle 102 without the need for intermediary networks or devices. In one embodiment, the augmented reality device 114 can also communicate with the server 104 via the network 108. The augmented reality device can exchange control, configuration, and / or user profile information with the vehicle 102 and / or the server 104.Details of the Augmented Reality Device 114 are provided with reference to . Fig. 3 provided.

[0015] Vehicle 102 may contain a variety of units, including, but not limited to, a vehicle computer, a vehicle control unit (VCU), and a detection unit. Details of Vehicle 102 are provided below with reference to Fig. 2 provided.

[0016] Fig. Figure 2 illustrates a block diagram of the vehicle 102 in which embodiments of the present disclosure can be implemented. The vehicle 102 can include a plurality of units, including, among others, a vehicle computer 208, a vehicle control unit (VCU) 210, and an infotainment unit 238. The VCU 210 can include a plurality of electronic control units (ECUs) 214 arranged in communication with the vehicle computer 208.

[0017] In some embodiments, a user device, such as a mobile phone, laptop computer, smart transponder, or the like, may be configured to connect to the vehicle computer 208, which can communicate via one or more wireless connections, and / or may establish this connection by using near field communication (NFC) protocols, Bluetooth ®- Connect protocols, Wi-Fi, Ultra Wideband (UWB) and other possible data connection and sharing techniques directly to the vehicle 102.

[0018] According to the disclosure, the vehicle computer 208 can be installed at any location in the vehicle 102. The vehicle computer 208 can be or include an electronic vehicle control unit comprising one or more processor(s) 202, one or more storage devices 204, and one or more transceivers 206.

[0019] The processor(s) 202 can be arranged in communication with one or more storage devices, which communicate with the respective computing systems (e.g., the memory 204 and / or one or more external databases located in Fig. (2 not shown) are arranged. The processor(s) 202 can / can use the memory 204 to store programs as code and / or data for performing operations according to the disclosure. The memory 204 can be a persistent, computer-readable storage medium or persistent, computer-readable memory in which program code for controlling vehicles is stored. The memory 204 can include any or a combination of volatile memory elements (e.g., dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), etc.) and any or more non-volatile memory elements (e.g., erasable programmable read-only memory (EPROM), flash memory, electronically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), etc.).In some embodiments, the memory 204 may include a module 245 that can implement the various embodiments of the present disclosure. The module 245 may contain instructions that can be executed by the processor 202 to realize the various embodiments of the present disclosure.

[0020] The vehicle computer 208 may also include a transceiver 206. The transceiver 206 may be configured to receive information / inputs from one or more external devices or systems, such as a user device 208, an external server, and / or the like. Furthermore, the transceiver 206 may transmit notifications, requests, signals, etc., to the external devices or systems. Additionally, the transceiver 206 may be configured to receive information / inputs from vehicle components, such as the vehicle sensor system 232, one or more ECUs 214, and / or the like. Furthermore, the transceiver 206 may transmit signals (e.g., command signals) or notifications to the vehicle components, such as the BCM 220, the infotainment system 238, and / or the like.

[0021] In some embodiments, the VCU 210 can share a power and / or communication bus with the vehicle computer 208 and can be configured and / or programmed to coordinate data between vehicle systems, connected servers, and / or the like. The VCU 210 can include or communicate with any combination of the ECUs 214, such as the BCM 220, an engine control module (ECM) 222, a transmission control module (TCM) 224, a telematics control unit (TCU) 226, a driver assistance technologies (DAT) controller 228, etc. The VCU 210 can also include a vehicle perception system (VPS) 230, which has connectivity with and / or controls and / or communicates with one or more vehicle sensor systems 232.The vehicle sensor system 232 may include one or more vehicle sensors, including, but not limited to, a radio detection and ranging sensor (RADAR or "radar" sensor) configured to detect and locate objects inside and outside the vehicle 102 using radio waves, seat belt buckle sensors, seat area sensors, a light detection and ranging sensor ("LIDAR" sensor), door sensors, proximity sensors, temperature sensors, wheel sensors, one or more ambient weather or temperature sensors, vehicle interior and exterior cameras, steering wheel sensors, etc. The sensors that are part of the vehicle sensor system 232 may be coupled to the vehicle 102 at one or more locations and in one or more ways. For example, the various sensors of the vehicle sensor system 232 may be integrated into the various subsystems of the vehicle 102, such as doors, mirrors, a roof, etc., or be attached to the vehicle 102 using a suitable mounting mechanism. In some embodiments, the various sensors of the vehicle sensor system 232 can be located on the front, rear, sides, top, bottom, and underside of the vehicle 102. The position of a sensor may depend on its function. For example, a sensor that monitors the area under the vehicle may be connected to a floor surface of the vehicle 102, while a sensor that can monitor an area on any side of the vehicle 102 may be mounted on or integrated into the doors of the vehicle 102. The vehicle sensor system 232 may also include one or more road noise sensors, such as accelerometers, which are coupled to various mechanical components and / or systems of the vehicle 102.It is understood by the expert that the sensors can be coupled to the vehicles in different ways and in positions other than those mentioned above.

[0022] In some embodiments, the VCU 210 can control operational aspects of the vehicle and implement one or more sets of instructions received from the server 104, the user device 112, or from one or more sets of instructions stored in the memory 204.

[0023] The TCU 226 can be configured and / or programmed to provide vehicle connectivity with wireless computing systems inside and outside the vehicle 102 and can include a navigation (NAV) receiver 234 for receiving and processing a GPS signal, a BLE ® -Module (BLEM) 236, a Wi-Fi transceiver, a UWB transceiver and / or other wireless transceivers (in Fig. 2 not shown) include those for wireless communication (which includes mobile communication) between the vehicle 102 and other systems (e.g. a vehicle radio key (in Fig. 2 (not shown), one or more servers, a user device, etc.), computers, and modules. The TCU 226 can communicate with the ECU 214 via a wired or wireless bus. In some aspects, the TCU 226 can be configured to determine a real-time vehicle geolocation, e.g., via the NAV receiver 234.

[0024] The ECUs 214 can control aspects of vehicle operation and communication using inputs from human drivers, inputs from the vehicle computer 208 and / or via wireless signal inputs received via the wireless connection(s) from other connected devices, such as, among others, the server 206.

[0025] The BCM 220 generally integrates sensors, vehicle performance indicators, and variable throttles assigned to the vehicle systems. It may also include a processor-based power distribution circuit capable of controlling functions associated with the vehicle body, such as lights, windows, security, cameras, audio systems, speakers, windshield wipers, door locks, access control, various comfort controls, etc. The BCM 220 can also function as a gateway for bus and network interfaces to communicate with remote ECUs (in Fig. 2 not shown) to interact.

[0026] The DAT 228 controller and / or the 240 autonomous driving system can provide automated driving and driver assistance functionality from Level 1 to Level 5, which may include, for example, active parking assistance, reversing assistance, and / or adaptive cruise control, among other features. The DAT 228 controller can also provide aspects of user and environmental input that can be used for user authentication.

[0027] In some embodiments, the vehicle computer 208 can connect to an infotainment system 238 (or a human-machine interface (HMI) of the vehicle). The infotainment system 238 can include a touchscreen interface section and can include speech recognition features and biometric identification capabilities that can identify users based on facial recognition, voice recognition, fingerprint identification, or other biological identification methods. In other aspects, the infotainment system 238 can also be configured to receive user instructions via the touchscreen interface section and / or to output or display notifications, navigation maps, etc., on the touchscreen interface section. In some embodiments, the user device 112 can provide the HMI interface.

[0028] In one embodiment, the vehicle 102 may further include a driver visibility system 242. The driver visibility system 242 may include algorithms that process specific types of data received by the sensor system 232 and output one or more setting information items that can be transmitted to the augmented reality device 114. The augmented reality device 114 can then use the setting information to modify its behavior / configuration. In one embodiment, the sensor system 232 may continuously monitor several parameters associated with the vehicle's external environment and provide this raw data to the driver visibility system 242. The driver visibility system 242 can then select specific data from the raw data and process this data using one or more of the algorithms to generate the setting information.For example, the sensor system 232 can monitor one or more environmental parameters, such as the presence and extent of rain, glare, fog, smog, snow, etc., which can affect driver visibility, and generate raw data associated with these parameters. This raw data can then be provided to the driver visibility system 242. Based on the received data, the driver visibility system 242 can determine which data from the raw sensor data should be used to mitigate the environmental parameters responsible for reduced driver visibility. The driver visibility system can then generate setting data for the augmented reality device 114.If the environmental parameter relates, for example, to glare caused by sunlight falling on the road, the driver visibility system 242 can generate setting data to determine the polarization of the lens of the augmented reality device 114. This setting data can be sent to the augmented reality device 114, and the device can modify its polarization to provide a clear, glare-free image for the user. Although the augmented reality device 114 is illustrated as a wearable device, it should be noted that it can be implemented in other ways, such as as an overhead display of the vehicle 102 and / or integrated into a vehicle windshield.

[0029] Certain computing modules may be omitted from the computer system architecture of the 208 vehicle computer and / or the VCU 210. It goes without saying that the in Fig. The computing environment shown in Figure 2 is an example of a possible implementation according to the present disclosure and should therefore not be considered restrictive or exclusive.

[0030] In addition to the components mentioned above, the Vehicle 102 may have numerous mechanical systems and subsystems. A chassis or frame may form the backbone of the Vehicle 102, supporting the body and other components. The Vehicle 102 may include an engine that converts fuel into mechanical power, thus propelling the vehicle forward. The engine includes various components, such as the engine block, pistons, valves, and spark plugs. The Vehicle 102 may also include a transmission system. The transmission system transfers the engine's power to the wheels. It includes, among other components, the clutch, gearbox, driveshaft, and differentials. The transmission adjusts the power output according to the vehicle's speed and load. The Vehicle 102 may also include a suspension system.The suspension system absorbs shocks and maintains contact between the tires and the road, providing a smooth ride. It includes components such as springs, shock absorbers, and linkages. The Vehicle 102 also includes a vehicle stopping system that allows the driver to slow down or stop the Vehicle 102. This includes components such as pedals, master cylinder, lines, and brake pads or shoes. The Vehicle 102 also includes a steering system that allows the driver to steer the vehicle. The steering system includes components such as the steering wheel, steering column, rack and pinion, and tie rods. The Vehicle 102 may also include an exhaust system that removes and filters the exhaust gases produced by the engine. This includes, among other components, the exhaust manifold, catalytic converter, muffler, and exhaust pipe.The vehicle 102 also includes a cooling system that prevents the engine and / or battery from overheating. It includes components such as the radiator, water pump, thermostat, and coolant. The vehicle 102 also includes a fuel storage and supply system. This includes the fuel tank, fuel pump, fuel filter, and fuel injectors. The vehicle 102's electrical system provides power to the car's electrical components. It may include the battery, alternator, starter motor, and wiring. The heating, ventilation, and air conditioning (HVAC) system controls the temperature inside the vehicle 102. It includes the heater core, blower motor, and air conditioning compressor.In some embodiments, the vehicle can be an electric vehicle (EV) or a hybrid vehicle, and in each case, some of the aforementioned components would be replaced by an electric motor and a high-voltage battery. When all the mechanical components work together, they ensure that the vehicle operates optimally.

[0031] Fig. Figure 3 illustrates a functional block diagram of the augmented reality device 114 according to an embodiment of the present disclosure. The augmented reality device 114 can include one or more hardware processors 302. The processor(s) 302 can be single-core or multi-core devices (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof). The processors 302 manage the data from sensors, execute AR applications, and seamlessly render the augmented visual information on the display of the augmented reality device 114. The augmented reality device 114 can also include a variety of input / output devices 304. The input / output device 304 can include a display that presents the augmented content to the user.The display can utilize transparent optical displays (e.g., transparent displays such as those used in smart glasses, which allow users to see the real world directly while overlaying digital information), transparent video displays (e.g., these use cameras to capture the real world and then display it on a screen with superimposed augmented elements), or head-mounted displays (HMDs) (e.g., these are worn on the head and can incorporate both transparent optical and transparent video technologies; HMDs provide an immersive augmented reality experience by covering the user's field of vision with augmented content). In some embodiments, the augmented input / output devices 303 may also include touchscreens, voice-controlled systems, gesture recognition systems, wearable controllers, and the like.

[0032] The augmented reality device 114 can also include one or more sensors 306. The sensors 306 can include one or more cameras, gyroscopes, accelerometers, magnetometers, depth sensors, ambient light sensors, etc. The augmented reality device 114 can further include one or more memory devices 308. The memory device 308 can include any volatile memory elements or a combination thereof (e.g., dynamic random-access memory (DRAM), synchronous dynamic random-access memory (SDRAM), etc.) and any one or more non-volatile memory elements (e.g., erasable programmable read-only memory (EPROM), flash memory, electronically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), etc.).In some embodiments, the storage device 308 may include a module 312 that can implement the various embodiments of the present disclosure. The module 312 may contain instructions and algorithms that can be executed by the processor 302 to realize the various embodiments of the present disclosure. The storage device 308 may further include the operating system, development platforms, applications, etc., associated with the augmented reality device 114.

[0033] The augmented reality device 114 can also include a communication interface 310. The communication interface 310 can use any of the known communication protocols, such as Wi-Fi or Bluetooth. ®or mobile networks to communicate with the vehicle 102, the server 104, the user device 110, and / or the network 108. The augmented reality device also includes a variety of mechanical components 314, such as a housing and frame that forms the outer shell or outer frame of the augmented reality device 114 and holds all internal components together; one or more lenses used to focus and project the augmented content onto the user's field of vision; mounting mechanisms, such as adjustable straps, headbands, or ear hooks, that secure the augmented reality device 114 to the user's head and maintain the correct positioning of the display relative to the user's eyes; and adjustable components, such as an interpupillary distance (IPD) adjustment device, which allows users toAdjusting the distance between the lenses to match the user's interpupillary distance to ensure a comfortable and clear viewing experience; cooling systems, such as small fans or heat sinks, to dissipate heat and maintain optimal operating temperatures; buttons and controls, such as physical buttons, rotary knobs, or touch-sensitive areas on the device, that allow users to interact with the augmented reality device 114; connectors and ports, such as USB ports, audio jacks, and charging connectors; and hinges and joints that allow the arms to be folded and adjusted. All components of the augmented reality device can communicate via a communication bus 316.

[0034] As noted above, a driver's visibility can be impaired by poor weather conditions. In such cases, a clear view of the road and the surrounding environment significantly improves the driving experience. Fig. Figure 4 illustrates a high-level flowchart of a process 400 for increasing visibility according to an embodiment of the present disclosure. Suppose a driver is operating a vehicle and the weather is clear and sunny. At some point, the weather is expected to change and it will begin to rain heavily. This reduces the driver's visibility of the road ahead and, more generally, of the external environment around the vehicle. The driver can then use the augmented reality device 114 to improve their visibility of the road and the surroundings. When the driver is wearing the augmented reality device, the device is communicatively coupled to the vehicle.The augmented reality device can then use data received from its own sensors and / or data received from the vehicle's sensors to generate an image of the vehicle's surrounding environment and display this image on a display of the augmented reality device. Thus, the driver is now able to see a real-time image of the road ahead and other parts of the external environment surrounding the vehicle. At step 402, the vehicle and / or the augmented reality device can detect the environmental condition that is affecting the driver's visibility (e.g., heavy rain obscuring road markings and other objects in the environment). Based on the type and nature of the adverse weather condition (e.g., heavy rain in this example), at step 404, the vehicle and / or the augmented reality device can use a specific set of sensors (e.g.,The vehicle and / or augmented reality device can activate multispectral sensors (such as RGB cameras, infrared sensors, and radar sensors) and / or collect data from them. In step 406, the vehicle and / or augmented reality device can then process the data using one or more specific sets of algorithms to generate processed data that includes information about the locations and types of road markings and objects in the environment. In step 408, the vehicle and / or augmented reality device can determine one or more setting data points for one or more parameters of the augmented reality device. In step 410, the vehicle can send the setting data and / or process data from the sensors to the augmented reality device. The augmented reality device can then use the setting information to change the configuration of one or more parameters.When the processed data is sent to the augmented reality device, the device can then overlay the image of the surrounding environment with visual markers for road markings and obstacles. Thus, the image provided by the augmented reality device now presents a more comprehensive and clearer view of the environment surrounding the vehicle, thereby mitigating the visibility problems caused by heavy rain. This improves the driver's driving experience.

[0035] According to the embodiments of the present invention, there are several ways in which an augmented reality device can improve the driver's visibility. In one instance, the augmented reality device can modify some of the parameters of its components based on received data in order to update / modify the image it displays. In other embodiments, the augmented reality device can add information to the displayed image based on received data to provide the driver with a more comprehensive picture. The method or process used by the augmented reality device to improve the driver's visibility may depend on the nature and type of weather condition that causes the reduced visibility.Depending on the type of weather or environmental condition, the vehicle and / or the augmented reality device can select a specific process and / or a combination of processes to mitigate the effects of the environmental condition on driver visibility. Fig. Figure 5 illustrates several examples of different environmental conditions and the processes carried out by the vehicle and / or the augmented reality device to mitigate the effects of the environmental conditions and improve driver visibility.

[0036] A driver can begin driving the vehicle (step 502) and wear the augmented reality device to assist with driving. The augmented reality device can be turned on at step 504. In some embodiments, the driver can configure one or more parameters of the augmented reality device, such as brightness, contrast, augmentation types, etc., based on personal preferences and / or the driver's driving style. As part of the turn-on process, the augmented reality device can be communicatively paired with the vehicle. Once paired, the augmented reality device and the vehicle can exchange some setting and capability data. This informs the vehicle and the augmented reality device about which features are supported by each.In some embodiments, the augmented reality device and / or the vehicle can also exchange user profile data. For example, the augmented reality device can be personalized with respect to the preferences of an initial user. If this initial user then drives a different vehicle (e.g., a rental vehicle), the augmented reality device can communicate the user profile and preference data to the other vehicle. The other vehicle can then send its capability data to the augmented reality device (e.g., information about its sensors and features it can support). Once the augmented reality device and the vehicle are "synchronized," the augmented reality device can then, at step 506, use data acquired from its own sensors and / or the vehicle's sensors to generate and display an initial image of the environment around the vehicle.For example, the first image can be captured using one or more of the vehicle's forward-facing cameras (e.g., a dashcam or similar device). This first image can include a representation of the road as well as other objects in the environment that might be hidden from the driver's view if they were not wearing the augmented reality device (e.g., blind spots).

[0037] As noted above, the process used by the augmented reality device and / or the vehicle to mitigate the effects of weather conditions and improve driver visibility depends on the nature of the weather condition / phenomenon. In a first example, it is assumed that the weather condition is such that glare from the sun, reflected off horizontal surfaces like the road, impairs driver visibility. In this case, the vehicle and / or the augmented reality device may determine that the glare from the sun is causing reduced visibility for the driver (Step 508). For example, photometric sensors, cameras, or LiDAR systems of the vehicle and / or the augmented reality device may detect the presence of glare and its direction.Photometric sensors measure light intensity and, by comparing light intensity from different directions, can identify areas of excessive brightness that indicate glare. Cameras can capture images of the environment, and image processing algorithms can analyze these images to detect glare. These algorithms often look for regions of high intensity and specific patterns associated with glare. LiDAR systems can detect glare by measuring the reflection of laser beams. At step 510, the vehicle and / or augmented reality device can determine initial data associated with the environmental condition. This initial data may include information about the intensity and direction of the glare.Based on the initial data, the vehicle and / or the augmented reality device can determine setting information for the polarization filters of the augmented reality device in step 512. For example, the polarization filters of the augmented reality device can be adjusted using the 90-degree rule. The setting information can be sent to the augmented reality device (step 514). The augmented reality device can then implement the new polarization settings in step 516 and display a modified image to the user in step 518. The modified image now shows the environment without the effects of glare, thus providing improved visibility for the driver.It should be noted that the augmented reality device and / or the vehicle continuously monitors glare in real time and adjusts the polarization of the augmented reality device's lenses as needed. Thus, the augmented reality device performs dynamic polarization adjustments while the vehicle is being driven, ensuring the driver always sees a glare-free image.

[0038] In a second example, it is assumed that weather conditions include heavy rain, which impairs the driver's visibility, making it impossible for the driver to clearly see road markings or other objects in the environment. The vehicle and / or augmented reality device can detect the presence of heavy rain at step 520. For example, an optical sensor mounted on the vehicle's windshield can be used to detect the presence and amount of rain. To mitigate the reduced visibility caused by heavy rain, the augmented reality device and / or the vehicle can collect multispectral data from one or more sensors at step 522. For example, the multispectral data can include data in the visible and non-visible spectrum (e.g., infrared, radar, etc.).The vehicle and / or augmented reality device can process this multispectral data (step 524) and determine the position of road markings and other objects in the environment. In step 526, the augmented reality device can augment the initial image with the data on the road markings and other objects determined from the multispectral data. The augmented image is then displayed to the driver in step 528. Thus, the driver now has a much clearer view of the road and the surroundings. Again, the vehicle and / or augmented reality device continuously monitors the presence and intensity of rain and the external environment, updating the positions of the road markings and objects in the environment in real time while the driver operates the vehicle.

[0039] In a third example, the environmental condition is assumed to refer to ambient light levels inside the vehicle. At step 530, the augmented reality device can continuously monitor the ambient light level in the vehicle (e.g., using an ambient light sensor). At step 532, the augmented reality device determines whether the ambient light levels are above or below a threshold. This threshold can be set by the driver according to their preference. Based on whether the current ambient light level inside the vehicle is above or below the threshold, the augmented reality device can determine the brightness and contrast settings at step 534. The augmented reality device can then adjust the brightness and contrast settings accordingly at step 536.The augmented reality device continuously monitors the ambient light levels inside the vehicle and adjusts the brightness and contrast settings to provide an optimal viewing experience for the driver.

[0040] In a fourth example, the environmental condition is assumed to be the presence of fog, which reduces the driver's visibility. In this case, at step 538, the vehicle and / or the augmented reality device can determine the presence of fog and that the image currently displayed by the augmented reality device is an incomplete representation of the environment. In this case, the vehicle and / or the augmented reality device can capture data from the infrared sensors, which can penetrate the fog, and use this infrared data to enhance the image displayed to the user. Additionally, the vehicle can also capture data from one or more of its external cameras to capture additional images at step 540.The augmented reality device can then, at step 542, enhance the initial image with these additional images to provide the driver with a more comprehensive view of the environment. This process can be continuous, and the enhanced image is displayed to the driver in real time.

[0041] In a fifth example, it is assumed that the driver is driving at night and one or more of the other weather conditions are also present (e.g., heavy fog / rain at night). In this case, driver visibility may be further reduced due to poor or no-light conditions at night. The augmented reality device can continuously monitor the ambient light conditions inside and / or outside the vehicle and adjust its brightness and contrast settings accordingly. For example, the presence or absence of streetlights on the road can vary the ambient light conditions inside the vehicle. The augmented reality device can account for this by adjusting its brightness and contrast settings in real time as the vehicle travels along the road.At step 546, the vehicle and / or the augmented reality device may detect that it is raining / fog and that it is nighttime. The vehicle and / or the augmented reality device may conclude that the rain / fog, combined with the poor lighting conditions at night, contributes to the reduced visibility for the driver. At step 548, the vehicle and / or the augmented reality device may collect data about the vehicle's external environment from one or more radar sensors, as radar may be best suited to collect this data under the given weather conditions. At step 550, the vehicle and / or the augmented reality device may detect the presence and location of objects in the collected radar data.Then, at step 552, the augmented reality device can modify the initial image to include visual representations of the objects. The driver can now identify the various objects in the external environment more clearly and accurately, thus improving their visibility and driving experience. In some cases, the augmented reality device may also need to continuously adjust the polarization settings of its lenses to compensate for glare from oncoming headlights, streetlights, or other bright lights in the environment.

[0042] It should be noted that the various environmental conditions described above are exemplary and that other environmental conditions may also occur. Even in these cases, the vehicle and / or the augmented reality device can determine the nature, type, and / or severity of the environmental condition and determine which sensor data to use to mitigate the effects of the environmental condition and improve driver visibility, and determine how to process the sensor data, including which specific processing algorithm to use. In other embodiments, an overhead display of the vehicle can be used to display the various images instead of, or in addition to, the augmented reality device.

[0043] Fig. Figure 6 shows a block diagram of an example control server 600 (e.g., control server 104). Fig.1) on which one or more arbitrary techniques (e.g., methods) can be performed, or which can perform the methods described above in connection with the vehicle 102, according to one or more exemplary embodiments of the present disclosure. In other embodiments, the server 600 can be operated as a standalone device or be connected (e.g., networked) to other servers. In a networked deployment, the server 600 can function as a server machine, a client machine, or both in server-client network environments. In one example, the server 600 can function as a peer server in peer-to-peer (P2P) (or other distributed) network environments.Server 600 can be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smart key fob, a wearable computing device, a web device, a network router, a switch or bridge, or any machine capable of executing instructions (sequentially or otherwise) specifying actions to be performed by this server, such as a base station. Furthermore, although only a single server is illustrated, the term "server" can also be understood to include any collection of servers that, individually or collectively, execute a set (or multiple sets) of instructions to perform one or more of any of the methodologies discussed in this document, such as cloud computing, software as a service (SaaS), or other computer cluster configurations.

[0044] Examples, as described in this document, may include or operate with logic or a set of components, modules, or mechanisms. Modules are tangible instances (e.g., hardware) capable of performing predefined operations during operation. A module includes hardware. In one example, the hardware may be specifically configured to perform a specific operation (e.g., hardwired). In another example, the hardware may include configurable execution units (e.g., transistors, circuits, etc.) and a computer-readable medium containing instructions, where the instructions configure the execution units to perform a specific task when operated. This configuration may occur under the guidance of the execution units or a loading mechanism.Accordingly, the execution units are communicatively coupled to the computer-readable medium when the device is operated. In this example, the execution units can be an element of more than one module. For instance, during operation, the execution units can be configured by a first set of instructions to execute a first module at one time and reconfigured by a second set of instructions to execute a second module at a second time.

[0045] The server (e.g., the computer system) 600 can include a hardware processor 602 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), main memory 604, and static memory 606, some or all of which can communicate with each other via a coupling (e.g., a bus) 608. The server 600 can further include a graphics display device 610, an alphanumeric input device 612 (e.g., a keyboard), and a user interface navigation device (UI navigation device) 614 (e.g., a mouse). In an example, the graphics display device 610, the alphanumeric input device 612, and the UI navigation device 614 can be a touchscreen display. The server 600 can additionally include a data storage device (i.e., a storage device).The server 600 may include a drive unit 616, a network interface device / transmitter 620 coupled to antenna(s), and one or more sensors 628, such as a global positioning system (GPS) sensor, a compass, an accelerometer, or another sensor. The server 600 may include an output control 634, such as a serial (e.g., Universal Serial Bus (USB)), parallel, or other wired or wireless (e.g., infrared (IR)), near field communication (NFC), etc., connection, for communicating with or controlling one or more peripheral devices (e.g., a printer, a card reader, etc.).

[0046] The data storage device 616 can include a machine-readable medium 622 on which one or more sets of data structures or instructions (e.g., software) are stored, embodying or utilizing one or more of any of the techniques or functions described herein. The instructions may also reside, wholly or at least partially, within the main memory 604, within the static memory 606, or within the hardware processor 602 during their execution by the server 600. In an example, any one or any combination of the hardware processor 602, the main memory 604, the static memory 606, or the data storage device 616 can constitute machine-readable media.

[0047] Although machine-readable medium 622 is illustrated as a single medium, the term "machine-readable medium" can include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) configured to store the one or more instructions.

[0048] Various embodiments can be implemented wholly or partially in software and / or firmware. This software and / or firmware can take the form of instructions contained in or on a non-transferable, computer-readable storage medium. These instructions can then be read and executed by one or more processors to enable the execution of the operations described herein. The instructions can be in any suitable form, including but not limited to source code, compiled code, interpreted code, executable code, static code, dynamic code, and the like.Such a computer-readable medium can include any tangible, non-transient medium for storing information in a form that is readable by one or more computers, such as read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory, etc.

[0049] The term “machine-readable medium” can include any medium capable of storing, encoding, or carrying instructions for execution by the Server 600, and which causes the Server 600 to perform one or more of any of the techniques of this disclosure, or which is capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-restrictive examples of machine-readable media can include semiconductor memory and optical and magnetic media. In one example, a machine-readable medium with mass includes a machine-readable medium with a plurality of particles that has a rest mass. Specific examples of machine-readable media with mass can include non-volatile memory, such as semiconductor memory devices (e.g.,electrically programmable read-only memory (EPROM) or electrically erasable programmable read-only memory (EEPROM) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0050] Instructions can also be transmitted or received via a communication network using a transmission medium through the network interface device / transmitter 620, utilizing any of a number of transmission protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Examples of communication networks include, but are not limited to, a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), mobile phone networks (e.g., cellular networks), analog telephone networks (POTS networks), wireless data networks (e.g., the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards group, known as Wi-Fi®, the IEEE 802.16 standards group, known as WiMAX®), and the 802.15 standards group.4 of the IEEE and peer-to-peer (P2P) networks. In one example, the network interface device / transmitter 620 can include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas for connecting to the communications network. In another example, the network interface device / transmitter 620 can include a plurality of antennas for wireless communication using at least one single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technique.The term "transmission medium" shall be understood to include any intangible medium capable of storing, encoding, or carrying instructions for execution by the Server 600, and including digital or analog communication signals or other intangible media to facilitate communication between such software. The operations and processes described and shown above may be performed or carried out in any suitable order in various implementations. Furthermore, at least some of the operations may be performed in parallel in certain implementations. In addition, fewer or more of the operations than described may be performed in certain implementations.

[0051] It should be noted that the vehicle implements and / or performs operations as described herein in accordance with the user manual and safety guidelines. Additionally, any action taken by the vehicle owner / driver based on recommendations or notifications provided by the vehicle should comply with all regulations specific to the vehicle's location and operation (e.g., federal, state, country, city, etc.). Recommendations or notifications provided by the vehicle should be treated as suggestions and followed only in accordance with any regulations specific to the vehicle's location and operation.The preceding disclosure refers to the accompanying drawings, which form part thereof and illustrate specific implementations in which the present disclosure can be practically implemented. It is understood that other implementations may be used and structural modifications made without deviating from the scope of the present disclosure. References in the description to "an embodiment," "an exemplary embodiment," etc., indicate that the described embodiment may include a specific feature, structure, or property, but not every embodiment necessarily includes that specific feature, structure, or property. Furthermore, such formulations do not necessarily refer to the same embodiment.Furthermore, if a feature, structure or property is described in connection with an embodiment, the person skilled in the art will recognize such a feature, structure or property in connection with other embodiments, whether this is expressly described or not.

[0052] Furthermore, the functions described in this document may be performed by one or more hardware, software, firmware, digital components, or analog components. For example, one or more application-specific integrated circuits (ASICs) may be programmed to execute one or more of the systems and procedures described in this document. Certain terms are used throughout the description, and patent claims refer to specific system components. It is obvious to those skilled in the art that the components may be designated by other names. This document does not distinguish between components that differ in name but not in function.

[0053] It is also understood that the word "example," as used in this document, is not intended to be exclusive or restrictive. In particular, the word "example," as used herein, indicates one of several examples, and it is understood that no undue emphasis or preference is placed on the specific example described.

[0054] A computer-readable medium (also called a processor-readable medium) comprises any non-volatile (e.g., physical) medium involved in providing data (e.g., instructions) that can be read by a computer (e.g., by a computer's processor). Such a medium can take many forms, including, without limitation, both non-volatile and volatile media. Computing devices can contain computer-executable instructions, the instructions being executable by one or more computing devices, such as those listed above, and being stored on a computer-readable medium.

[0055] With regard to the processes, systems, procedures, heuristics, etc., described in this document, it is understood that while the steps of such processes, etc., have been described as occurring in a certain sequence, such processes could, in practice, be implemented in a sequence that differs from that described in this document. Furthermore, it is understood that certain steps could be performed simultaneously, that other steps could be added, or that certain steps described in this document could be omitted. In other words, the descriptions of processes in this document serve the purpose of illustrating various embodiments and should in no way be interpreted as limiting the patent claims.

[0056] Accordingly, it is understood that the foregoing description is intended to be illustrative and not limiting. Many other embodiments and applications beyond the examples provided will become apparent from reading the preceding description. The scope should not be determined by reference to the foregoing description, but instead by reference to the attached claims, together with the entire range of equivalents to which these claims entitle. It is anticipated and intended that there will be future developments in the prior art discussed in this document and that the disclosed systems and methods will be incorporated into such future embodiments. Overall, it is understood that the application may be modified and varied.

[0057] All terms used in the patent claims shall have their general meaning as known to a person skilled in the art in the field of the technologies described in this document, unless expressly stated otherwise herein. In particular, the use of singular articles such as "a", "an", "a", "a", "a", "a", "a", "a", "a", "a", etc., shall be understood to refer to one or more of the elements indicated, unless a patent claim expressly limits this to the contrary.Phrases expressing conditional relationships, such as "may," "could," "might," or "might," are generally intended to convey that certain embodiments may include certain features, elements, and / or steps, while other embodiments may not, unless explicitly stated otherwise or implied by the context. Thus, such conditional phrases generally do not imply that features, elements, and / or steps are required in any way for one or more embodiments.

[0058] According to one embodiment, the first data includes data in a non-visible spectrum, and wherein the one or more processors are operable to execute instructions to further cause the vehicle to: identify one or more objects in a current environment using the first data; generate a second image, wherein the second image includes the first image augmented by a visual representation of each of the one or more objects, and wherein the second image is displayed by the augmented reality device.

[0059] According to one embodiment, the one or more processors can be operated to execute instructions to further cause the vehicle to: acquire second data, which includes one or more images of a current environment and which differ from the first image data; and send the second data to the augmented reality device, wherein the augmented reality device displays the one or more images in addition to the first image data.

[0060] According to one embodiment, the one or more processors can be operated to execute instructions to further cause the vehicle to: receive user profile and preference data from the augmented reality device; and send vehicle capability data to the augmented reality device.

[0061] According to one embodiment, the environmental condition includes glare from sunlight, and the one or more settings include a setting for a polarization value for the augmented reality device.

[0062] According to one embodiment, the one or more sensors include the following: a first sensor configured to capture third-party data in a visible spectrum; and a second sensor configured to capture fourth-party data in a non-visible spectrum.

Claims

[1] Procedure, encompassing: Displaying an initial image of the environment surrounding a vehicle; Determining an environmental condition, which is assigned to the environment around the vehicle, by the vehicle; Determine that the environmental condition causes reduced driver visibility for a driver of the vehicle; Determining a type of environmental condition by the vehicle; The vehicle acquires initial data from one or more of its sensors based on the nature of the environmental conditions; Extending the first image using the first data to generate a second image; and Displaying the second image. [2] The method of claim 1, further comprising: Displaying the first image and the second image on an augmented reality device worn by the driver. [3] Method according to claim 1, wherein the one or more sensors comprise: a first sensor configured to capture second data in a visible spectrum; and a second sensor configured to capture third data in a non-visible spectrum. [4] Method according to claim 1, further comprising: Selecting an initial algorithm to process the initial data based on the environmental conditions; Processing the first data using the first algorithm to generate fourth data; and Extending the first image using the fourth data. [5] The method of claim 1, wherein the first data includes radar data, the method further comprising: Determining the presence of one or more objects in the environment around the vehicle using the initial data; and where the second image includes one or more visual markers representing the one or more objects. [6] The method of claim 1, further comprising: Capturing one or more images from one or more external cameras of the vehicle by the vehicle; and where expanding the first image further includes displaying one or more additional images alongside the first image. [7] The method of claim 1, wherein displaying the first image comprises displaying the first image on an augmented reality device worn by the driver, the method further comprising: Determining the level of ambient light inside the vehicle using the augmented reality device; and Adjusting the brightness or contrast setting of the augmented reality device based on the level of ambient light. [8] Method according to claim 1, wherein the one or more sensors include a first sensor configured to capture fifth data in a non-visible spectrum, and the first data includes infrared image data, wherein the method further comprises: Identifying one or more objects in the environment using infrared image data; and Extending the first image using information about one or more objects. [9] Augmented reality device, comprising: one or more processors; one or more sensors that are coupled to one or more processors; a storage device coupled to one or more processors, wherein instructions are stored on the storage device which, when executed by the one or more processors, cause the augmented reality device to do the following: Displaying an initial image of the environment surrounding a vehicle; Detecting an environmental condition that causes reduced driver visibility; Determining a type of environmental condition; Collecting initial data associated with the environmental condition, based on the type of environmental condition; Processing the first data using a first algorithm to generate second data; and Extending the first image using the second data. [10] Augmented reality device according to claim 9, wherein the one or more sensors comprise the following: a first sensor configured to capture third-party data in a visible spectrum; and a second sensor configured to capture fourth data in a non-visible spectrum. [11] Augmented reality device according to claim 10, wherein the augmented reality device is further operable to: Fifthly, collecting environmental data using the second sensor; Identifying one or more objects in the environment using the fifth data; and Including a visual representation of one or more objects in the first image. [12] Augmented reality device according to claim 9, wherein the instructions further cause the augmented reality device to do the following: Determining the level of ambient light inside the vehicle; Determining a setting for a brightness value or a contrast value based on the level of ambient light; and Adjusting the brightness or contrast value based on the setting. [13] Augmented reality device according to claim 9, wherein the instructions further cause the augmented reality device to do the following: Receiving one or more images of the environment from the vehicle, wherein one or more images differ from the first image; and Displaying one or more additional images alongside the first image. [14] Augmented reality device according to claim 9, wherein the environmental condition includes glare and the first data includes information about a direction and intensity of the glare, wherein the instructions further cause the augmented reality device to do the following: Determining a polarization setting for a lens of the augmented reality device based on the initial data; and Dynamic adjustment of the lens polarization based on the initial data. [15] Vehicle, comprising: one or more processors; one or more sensors that are coupled to one or more processors; a storage device coupled to one or more processors and containing instructions which, when executed by the one or more processors, cause the vehicle to do the following: Pairing with an augmented reality device worn by a driver of the vehicle; Sending initial image data to the augmented reality device, where the initial image data includes information about an environment around the vehicle; Determining an environmental condition that is associated with the environment; Determine that the environmental condition causes reduced driver visibility; Determining a type of environmental condition; Determining initial data based on the type of environmental condition; Processing the initial data to determine one or more settings for the augmented reality device; and Communicating one or more settings to the augmented reality device, whereby the one or more settings cause the augmented reality device to modify the initial image data.

Images