Providing awareness of object located in path of vehicle

Abstract

A method for providing perception of an object located in a path of a vehicle. The method includes capturing data about an external environment using at least one perceptual sensor. The method also includes detecting one or more objects in the path of the vehicle based on the captured data. The method also includes estimating a position of the one or more objects in the path of the vehicle. The method also includes presenting one or more virtual objects using at least one display positioned inside the vehicle, wherein the one or more virtual objects represent the detected one or more objects.

Description

Background Art

[0001] The present disclosure generally relates to the field of automobiles. When a vehicle is in motion, even with the headlights on, due to dark conditions or poor visibility, a driver may not be able to see dangerous obstacles in the road. Dangerous obstacles can include, for example, animals wandering on the road, boulders, etc. A driver may also have difficulty seeing the lanes.

[0002] This introduction is provided only as background context and is not intended to be limiting in any way. It will be apparent to those of ordinary skill in the art that the concepts and principles of the present disclosure may be implemented equally in other applications and contexts. Summary of the Invention

[0003] The present disclosure relates to a system for providing awareness of objects located in a vehicle's path during low visibility. As described in more detail herein, embodiments enable the system to detect and identify objects located in the vehicle's path that a driver may not be able to see due to poor visibility. The system automatically uses perception sensors to detect, identify, and display such objects on a display to assist the driver in safe driving. As described in more detail herein, such objects may include potentially dangerous objects, where the system alerts the driver of such objects. Other objects can include road markings, such as road lines. As described in more detail herein, the system displays virtual representations of dangerous objects and road markings to better navigate the driver on a road with poor visibility.

[0004] In an illustrative embodiment, the present disclosure provides a computer-implemented method for providing awareness of an object in a path of a vehicle. The method includes: capturing data about an external environment using at least one sensing sensor; detecting one or more objects in the path of the vehicle based on the captured data; estimating the position of the one or more objects in the path of the vehicle; and presenting one or more virtual objects using at least one display located inside the vehicle, wherein the one or more virtual objects represent the one or more detected objects. Optionally, one or more objects in the path of the vehicle are hazardous obstacles. In some embodiments, one or more objects in the path of the vehicle are road markings. In some embodiments, the method further includes obtaining crowdsourced data about potential hazardous obstacles in the path of the vehicle. In some embodiments, the one or more virtual objects presented using at least one display are virtual hazardous obstacles. In some embodiments, the one or more objects presented using at least one display are virtual road lines. In some embodiments, the method further includes warning a driver of the vehicle if one or more of the detected objects are potential hazardous obstacles. In some embodiments, the at least one sensing sensor may include at least one of a camera, a radar detector, a light detection and ranging (Lidar) camera, or an ultrasonic camera.

[0005] In another illustrative embodiment, the present disclosure provides a component for a vehicle, the component comprising: at least one sensing sensor; at least one display positioned inside the vehicle; and a system that includes one or more processors and logic encoded on one or more non-transitory computer-readable storage media for execution by the one or more processors. The logic, when executed, is operable to cause the one or more processors to perform operations, the operations including: capturing data regarding an external environment using at least one sensing sensor; detecting one or more objects in a path of the vehicle based on the captured data; estimating positions of the one or more objects in the path of the vehicle; and presenting one or more virtual objects using at least one display positioned inside the vehicle, wherein the one or more virtual objects represent the one or more detected objects. Optionally, one or more objects in the path of the vehicle are hazardous obstacles. In some embodiments, one or more objects in the path of the vehicle are road markings. In some embodiments, the logic, when executed, is further operable to cause the one or more processors to perform operations including obtaining crowdsourced data regarding potential hazardous obstacles in the path of the vehicle. In some embodiments, the one or more virtual objects presented using at least one display are virtual hazardous obstacles. In some embodiments, the one or more objects presented using at least one display are virtual road lines. In some embodiments, the logic, when executed, is further operable to cause the one or more processors to perform operations including warning a driver of the vehicle if one or more of the detected objects are potential hazardous obstacles. In some embodiments, the at least one sensing sensor can include at least one of a camera, a radar detector, a light detection and ranging (Lidar) camera, or an ultrasonic camera.

[0006] In another illustrative embodiment, the present disclosure provides a non-transitory computer-readable storage medium having program instructions stored thereon. The program instructions, when executed by one or more processors, are operable to cause the one or more processors to perform operations, the operations including: capturing data regarding an external environment using at least one sensing sensor; detecting one or more objects in a path of the vehicle based on the captured data; estimating positions of the one or more objects in the path of the vehicle; and presenting one or more virtual objects using at least one display positioned inside the vehicle, wherein the one or more virtual objects represent the one or more detected objects. Optionally, one or more objects in the path of the vehicle are hazardous obstacles. In some embodiments, one or more objects in the path of the vehicle are road markings. In some embodiments, the instructions, when executed, are further operable to cause the one or more processors to perform operations including obtaining crowdsourced data regarding potential hazardous obstacles in the path of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Reference is made to the various drawings and descriptions of the present disclosure, in which like reference numerals are appropriately used to denote like components and / or system components and / or method steps.

[0008] Figure 1 is a side view block diagram of an example environment for providing awareness of an object located in the path of a vehicle.

[0009] Figure 2 is a flowchart for providing awareness of an object located in the path of a vehicle.

[0010] Figure 3 is a top view block diagram of an example external environment in which a vehicle is traveling on a path of the vehicle and there is an object on the path of the vehicle.

[0011] Figure 4 is a block diagram of an environment showing a view toward the front interior of the vehicle.

[0012] Figure 5 is a top view block diagram of an example external environment in which a vehicle is traveling on a path of the vehicle and there is an object on the path of the vehicle including road dashes and road solids.

[0013] Figure 6 is a block diagram of an environment showing a view toward the front interior of the vehicle.

[0014] Figure 7 is a block diagram of an example network environment of the present disclosure.

[0015] Figure 8 is a block diagram of an example computing system of the present disclosure. DETAILED DESCRIPTION

[0016] Figure 1 is a side view block diagram of an example environment 100 for providing awareness of an object located in the path of a vehicle. As shown, the assembly includes a system 102 of a vehicle 104 and at least one sensing sensor 106 having a lens 108. In some embodiments, the sensing sensor 106 and / or other sensing sensors may be located at an external portion of the vehicle. Locating at an external portion of the vehicle 104 means that at least a portion of the sensing sensor 106 (e.g., the lens 108) is exposed to the environment 100 or the external environment 100. In some embodiments, one or more of the sensing sensors may be located at an internal portion of the vehicle. For example, one or more of the sensing sensors may be located internally, observing through a window (e.g., behind a windshield, near a rearview mirror, etc.). Thus, the sensing sensor 106 and / or other sensing sensors capture various types of data regarding the external environment 100.

[0017] For ease of illustration, a perception sensor 106 is shown. In various embodiments, the system 102 may utilize multiple perception sensors and various types of perception sensors and sensors to capture data about the external environment 100. Any sensing method may be used, and the particular sensing method will depend on the particular implementation. For example, in various embodiments, one or more perception sensors may include one or more image sensing perception sensors or cameras, radar detectors, light detection and ranging (Lidar) cameras, and / or ultrasonic cameras or any combination thereof. The system may utilize image sensing perception sensors or cameras and / or infrared (IR) perception sensors or cameras and / or radar perception sensors or cameras. In various embodiments, the system may utilize any one or more of these perception sensors and / or other types of sensors and cameras to collect data. Such data collected may include data about any object outside the vehicle 104, including objects on the road. For example, such objects may include road surface features (e.g., bumps, potholes, etc.), environmental features (e.g., trash, live or dead animals, rocks, boulders, etc.). Such objects may also include other vehicles or people. The data may include Lidar data as well as images. The images may be a continuous series of images, which may include video. For ease of illustration, the perception sensor 106 is shown as being positioned on the bumper of the vehicle 104. In various embodiments, the perception sensor 106 and / or other cameras and sensors may also be positioned or mounted under the vehicle 104.

[0018] The perception sensor 106 may be referred to as a client device, which may communicate with the system 102. Such communication may be facilitated via any suitable communication network (not shown), such as a wired network, a Bluetooth network, a Wi-Fi network, etc., or any combination thereof.

[0019] In various embodiments, the data about the external environment 100 includes data about the ground of the vehicle path 110. In various embodiments, such data enables the system 102 to detect one or more objects in the vehicle's path or the vehicle path 110 based on the captured data. For example, the system 102 may detect an object 112 in the vehicle path 110, such as a deer or other animal or a potentially dangerous object. As described in more detail herein, the system 102 estimates the position of an object in the vehicle path 110 relative to the vehicle 104 based on the data collected. The system 102 also estimates the distance of such an object from the vehicle 104 based on the data collected.

[0020] As described in more detail herein, the system 102 uses one or more displays (not shown) located inside the vehicle to present one or more virtual objects, where the one or more virtual objects represent one or more detected actual or physical objects. As described herein in connection withFigures 3 to 6 Additional example embodiments are described in more detail that relate to the detection of objects and the presentation of virtual objects representing the objects.

[0021] For ease of explanation, Figure 1 One box is shown for each of system 102 and sensing sensor 106. Boxes 102 and 106 may represent multiple systems and sensing sensors. In other embodiments, environment 100 may not have all of the components shown and / or may have other components including other types of elements in place of or in addition to those shown herein.

[0022] Although system 102 performs the embodiments described herein, in other embodiments, any suitable component or combination of components associated with system 102 or any suitable one or more processors associated with system 102 may facilitate the performance of the embodiments described herein.

[0023] Figure 2 is a flowchart for providing awareness of an object located in the path of a vehicle. As described in more detail herein, the embodiments provide awareness of an object in the path of a vehicle during low visibility. Refer to Figure 1 and Figure 2 Both, the method is initiated at block 202, where a system such as system 102 uses one or more sensing sensors such as sensing sensor 106 to capture data regarding the external environment 100. As indicated herein, in various embodiments, one or more of the sensing sensors may be located at an external portion of the vehicle and / or an internal portion of the vehicle (e.g., observing through a window inside). In various embodiments, the data collected regarding the external environment 100 includes data regarding the ground on the vehicle path 110. In various embodiments, the data regarding the external environment may include image data, infrared data, and / or Lidar data. The specific type of sensing sensor for sensing sensor 106 may vary and will depend on the particular embodiment. Additionally, as described above, sensing sensor 106 may represent multiple sensors or cameras, which may include different types of sensing sensors.

[0024] At block 204, system 102 detects one or more objects in the path of vehicle 104 based on the captured data. In various embodiments, one or more objects in the path of the vehicle or vehicle path 110 are hazardous obstacles. For example, refer to Figure 1 and Figure 3 Both, the object 112 in vehicle path 110 is a stag, which is a hazardous obstacle because if vehicle 104 continues to travel on vehicle path 110, vehicle 104 will hit the stag.

[0025] Figure 3It is a top-down block diagram of an example external environment 100, where vehicle 104 is traveling on vehicle path 110 and object 112 is on vehicle path 110. For ease of illustration, sensing sensors 106 and system 102 are not shown. In this scenario, as vehicle 104 continues to move forward on vehicle path 110 towards object 112, if the vehicle does not bypass or stop before reaching object 112, vehicle 104 will hit object 112. Therefore, object 112 is referred to as a hazardous obstacle. Object 112 can represent other types of hazardous obstacles, including other types of animals, trash, rocks, boulders, trees, roadblocks, other vehicles, people, etc. These are objects that the driver of vehicle 104 may not see due to poor visibility or other factors such as being distracted by a mobile device or another person in vehicle 104.

[0026] In various embodiments, the system obtains crowdsourced data about potential hazardous obstacles in the path of the vehicle. For example, the system can obtain crowdsourced data from other drivers who report potential hazardous obstacles on the road. The crowdsourced data can be vehicle-to-vehicle (V2V) data. The system can also collect vehicle-to-infrastructure (V2I) data such as map data from the cloud and use global positioning system (GPS) technology to determine the vehicle's position on a given road with reported hazardous objects. In some embodiments, the system can be configured to report hazardous objects detected and identified by the system to a crowdsourcing application.

[0027] In various embodiments, the system can use artificial intelligence (AI) and machine learning to track known objects that may cause damage to a vehicle on a specific road or parking area. In some scenarios, some objects may not be hazardous obstacles by nature but are potentially dangerous objects based on the movement of the vehicle and the risk of driving into such obstacles. For example, such objects can also include permanently placed objects such as speed bumps or high curb transitions between a parking lot or driveway and a road.

[0028] Although some embodiments described in the context of objects in vehicle path 110 are hazardous obstacles, these embodiments and other embodiments can also be applied to non-hazardous objects. For example, in various embodiments, the path of the vehicle or one or more objects in vehicle path 110 are road markings. As described in more detail herein, the system helps the driver see, for example, road markings related to Figure 5 and Figure 6 related road markings.

[0029] In various embodiments, the system can use AI and machine learning to track known objects that contribute to safe driving. For example, as described above, in some scenarios, some objects may not be objects that are obstacles but are objects that contribute to safe driving, such as road markings, road lines, road signs, etc.

[0030] In various embodiments, the vehicle path 110 is a predicted path. In various embodiments, the system may predict the vehicle path 110 based on one or more techniques. For example, in various embodiments, the system may collect map data from the cloud and use GPS technology to determine the vehicle's position on a given road, and determine the vehicle path 110 based on the map and the vehicle's GPS position.

[0031] In various embodiments, the system may predict the vehicle path 110 based on a road detected by one or more sensing sensors of the vehicle 104. For example, the system may detect road markings or indicators (e.g., road lines, road signs, etc.) via one or more sensing sensors to predict the vehicle path 110. In some embodiments, the system may use AI and machine learning to determine the vehicle path 110 based on an image of the road shape captured by one or more sensing sensors in front of the vehicle 104.

[0032] In various embodiments, the system may also predict the vehicle path 110 based on real-time steering wheel activity. The system may determine the rotational position of the steering wheel of the vehicle 104 and also determine whether the steering wheel is rotating. Then, the system may predict the vehicle path 110 based on this information. For example, if the steering wheel is positioned such that the vehicle wheels are positioned to go straight ahead and the steering wheel is not turning, the system may predict that the vehicle path 110 goes straight ahead directly at that moment. If the system determines that the steering wheel rotates left or right to a certain degree, the system may predict that the vehicle path 110 bends towards the corresponding left or right direction by a corresponding amount. In various embodiments, the system may also determine the rate of rotation of the steering wheel and then predict the curvature of the vehicle path 110 based on the rate of rotation of the steering wheel.

[0033] Still referring to Figure 2 , at block 206, the system 102 estimates the position of one or more objects in the vehicle's path. In various embodiments, the system 102 may use Lidar technology to estimate the position of an object in the vehicle path 110. The system 102 also calculates the estimated position of a given object (such as object 112) relative to the front of the vehicle 104. As described in more detail herein, the system 102 warns or alerts the driver of any object in the vehicle path 110 that may be a dangerous obstacle, such as object 112.

[0034] At block 208, the system uses one or more displays located inside vehicle 104 to present one or more virtual objects. In various embodiments, the one or more virtual objects represent one or more actual or physical objects detected in the external environment. In various embodiments, the one or more virtual objects presented using the one or more displays are virtual hazardous obstacles. The terms present, display, depict, and visualize may be used interchangeably. Example embodiments relating to presenting virtual objects in one or more displays are described in more detail below in conjunction with Figure 4 Example embodiments involving presenting virtual objects in one or more displays are described in more detail.

[0035] Although steps, operations, or calculations may be presented in a particular order, the order may be changed in a particular implementation. Depending on the particular implementation, other orderings of the steps are possible. In some particular implementations, multiple steps shown as sequential in this specification may be performed simultaneously. Additionally, some embodiments may not have all of the steps shown and / or may have other steps that substitute or supplement those shown herein.

[0036] Figure 4 is a block diagram of environment 400, showing a view toward the front interior of the vehicle. This part of the vehicle may be the Figure 1 and Figure 3 corresponding part of vehicle 104 shown in. The instrument panel 402, windshield 404, steering wheel 406, infotainment display 408, heads up display 410, and object 112 are shown. In this example scenario, object 112 is visible through windshield 404. And the object shown is object 112 or stag 112 on the horizon 414. However, due to low or poor visibility or low lighting conditions, the driver may not see object 112. There may also be scenarios where the driver may not see object 112 due to distraction. For example, such a distraction may include the driver looking at the driver's mobile device rather than the vehicle path. Another exemplary distraction may be the driver looking at another person in the vehicle during a conversation, etc.

[0037] In various embodiments, the system presents or displays object 112 in infotainment display 408 and / or heads up display 410 in the form of corresponding virtual objects 432 and 422. For example, referring to infotainment display 408, the system displays virtual object 422 on virtual horizon 424, and also displays alert 426. Additionally, referring to heads up display 410, the system displays virtual object 432 on virtual horizon 434, and also displays alert 436. Both virtual objects 422 and 432 provide or increase the driver's visual awareness of object 112, which is a hazardous obstacle in this example scenario.

[0038] In some scenarios, the driver may visually and mentally focus on the infotainment display 408 rather than the road. The driver will see the virtual object 422, the virtual horizon 424, and the alert 426 on the infotainment display 408 and perceive the physical object 112 on the vehicle's path. In some scenarios, due to fog, rain, snow, or other weather conditions, the road visibility may be low or poor, and the driver may not see potential dangerous objects on the vehicle's path. The driver will see the virtual object 432, the virtual horizon 434, and the alert 436 on the head-up display 410 and perceive the physical object 112 on the vehicle's path. Thus, the virtual objects 422 and 432 and the alerts 426 and 436 are near-real-life views for the driver, which increases the awareness of the external environment for safer driving.

[0039] In various embodiments, the head-up display 410 provides an augmented reality (AR) windshield showing the actual physical road and enhances or overlays the road seen through the windshield 404 with virtual objects 432, the virtual horizon 434, and other information such as the alert 436. As described in more detail herein, the system can utilize AR technology to enhance features that are difficult to see in the driver's field of view. In other words, if the perception sensor detects a deer, the system can enhance the view of the actual deer in the driver's view in the windshield. The head-up display 410 can utilize all window surfaces via reflective projection or the like. In this way, when it is determined that the visibility of the actual object may be poor or obscured, the system enhances the driver's actual vision. As described in more detail herein, the system can consult a database to identify the deer and then decide how best to enhance what the driver is seeing. The system provides a predictive window to help the driver see what may be obscured.

[0040] In various embodiments, in the infotainment display 408 and / or the head-up display 410, the system can provide an enhanced version of the physical object. For example, the system can use one or more perception sensors to capture the object 112 in the form of one or more images or videos. The system can process the images to enhance or clarify the images for better visibility. For example, the object 112 may be obscured due to weather factors such as fog, rain, snow, etc. The system can apply digital photography filters to display an image of the object 112. For example, such filters can remove such weather factors from the image. As a result, the less visible actual physical object becomes more visible, enabling the driver to better distinguish what the actual object is.

[0041] In various embodiments, the system can similarly display different types of images of a given object such as object 112. Different types of images can include, for example, enhanced or filtered images of a physical object, created representations of a physical object, representations of a physical object obtained from the cloud, and the like. For clarity and ease of illustration, these different types of images are referred to as virtual to distinguish them from the actual physical objects that may be less visible. Thus, the system displays, in a clear and non-blurry manner, virtual object 432, which is a virtual version of physical object 112, on infotainment display 408 and / or head-up display 410.

[0042] In various embodiments, in infotainment display 408 and / or head-up display 410, the system can provide alternative representations of a physical object. For example, due to obscurity caused by weather factors such as fog, rain, snow, etc., the system may not be able to capture a clear image of object 112. In some cases, the vehicle's headlights may suddenly go out. In such scenarios, the system can present or display, in infotainment display 408 and / or head-up display 410, the contour shapes or silhouettes of object 112. In some embodiments, the system can obtain a representation of object 112, such as a contour or silhouette or animated image or icon, from an on-vehicle library or database and then display that representation in infotainment display 408 and / or head-up display 410. If a representation of the object cannot be obtained from the on-vehicle library or database, the system can obtain the representation from the cloud. In some embodiments, in the case where object 112 is moving, the system can display the object as a still frame of a captured video of the object for ease of identification. In some embodiments, the system can update the virtual image / object to the most recently captured image for higher accuracy. The system can also reposition or update the position of the virtual object on infotainment display 408 and head-up display 410 for higher accuracy.

[0043] In various embodiments, the system positions virtual objects 422 and 432 in infotainment display 408 and / or head-up display 410 at positions corresponding to the position of object 112 in windshield 404. For example, if object 112 is positioned in the middle of windshield 404, the system positions virtual objects 422 and 432 in the middle of infotainment display 408 and / or head-up display 410. If object 112 is located on the left or right side of windshield 404, the system will correspondingly position virtual objects 422 and 432 on the left or right side of infotainment display 408 and / or head-up display 410.

[0044] In various embodiments, if one or more of the detected objects are potential dangerous obstacles, alerts 426 and 436 warn the vehicle driver. For example, if the object is a dangerous obstacle, such as a stag, or other animal or object, etc. The system displays alerts 426 and 436 in the infotainment display 408 and / or the head-up display 410. These alerts provide additional indications to enhance the driver's awareness of the danger. Although the word "alert" is shown in the exemplary infotainment display 408 and the head-up display 410, the actual one or more words can vary and will depend on the specific implementation. For example, the system can display any warning words, such as "Warning", "Animal", "Boulder", etc. Additionally, the system can make the alert a predetermined color (e.g., red, etc.) to enhance the alert. Thus, the driver can slow down and turn to avoid hitting the object 112.

[0045] In various embodiments, the system also calculates the time to impact of the object 112 based on the estimated distance between the vehicle and the object 112 and based on the speed of the vehicle. In various embodiments, the system can use Lidar technology to estimate the distance between the object 112 and the vehicle. In some embodiments, the system can provide a visual alert progression based on the distance of the object 112. For example, if the object is 300 yards away, the system can display alerts 426 and 436 in yellow. If the object is 100 yards away, the system can display alerts 426 and 436 in orange. If the object is 50 yards away, the system can display alerts 426 and 436 in red. If the object is 25 yards away, the system can display alerts 426 and 436 in flashing red. In some embodiments, the system can issue an audio alert to further enhance the alert. For example, the system can activate a voice that describes the potential dangerous object (e.g., "Animal on the vehicle path!"). The system can also activate a voice that describes the distance of the potential dangerous object (e.g., "Animal on the vehicle path! 25 yards away!"). The specific color coding and audio alerts can vary and will depend on the specific implementation.

[0046] As described above, the system can track known objects that may cause damage to a vehicle on a specific road or parking area. Such known objects can include, for example, permanently placed objects, such as speed bumps, high curb transitions between a parking lot or driveway and a road, etc., and temporarily placed objects, such as street or parking lot barriers. Such known objects can be learned from crowdsourced data obtained from the cloud and / or from data about the environment that the system has detected, identified, and stored from previous trips. In this case, the system can warn the driver or suggest an alternative route to avoid such known objects.

[0047] In some embodiments, and in these scenarios, the system can implement automatic avoidance actions, such as automatically braking to slow down or stop the vehicle. The system can also control the steering of the vehicle to bypass the object for autonomous driving. In such a case of bypassing the object for autonomous driving, the system can also determine that the external environment is safe for driving around the object. For example, before driving around the object, the system can first determine that there are no other vehicles on the side of the vehicle where it might hit.

[0048] As described above, in various embodiments, one or more objects in the vehicle path can be road markings. Figure 5 and Figure 6 and the following related description describe example embodiments for road markings.

[0049] Figure 5 is a top - down block diagram of an example external environment 500, where vehicle 104 is traveling on vehicle path 110, and the objects include road dashed line 502 and road solid line 504 on vehicle path 110. For ease of illustration, the perception sensor 106 and the system 102 are not shown.

[0050] In this scenario, there are no dangerous obstacles on the vehicle path. The road dashed line 502 and the road solid line 504 are road markings to assist the driver in navigating the road. For example, the road dashed line 502 indicates to the driver the center road line in the road that separates vehicle 104 from oncoming traffic. The road solid line 504 indicates to the driver the side road line in the road that separates vehicle 104 from the road shoulder. In some scenarios, the driver of vehicle 104 may not be able to see the road dashed line 502 and the road solid line 504 due to poor visibility, or other factors such as poor lighting conditions, lack of reflectivity or faded road dashed line 502 and / or faded road solid line 504. In some scenarios, the road lines may not exist.

[0051] Figure 6 is a block diagram of environment 600, showing a view towards the front interior of the vehicle. This portion of the vehicle shown can be Figure 1 and Figure 5 the corresponding portion of vehicle 104 shown in. The instrument panel 402, windshield 404, steering wheel 406, infotainment display 408, and head - up display 410 are shown. The road dashed line 502 and the road solid line 504 are also shown, which are road markings to assist the driver in navigating the road.

[0052] In this example scenario, the road dashed line 502 and the road solid line 504 are visible through the windshield 404. The horizon 606 is also shown. However, due to low or poor visibility, the driver may not be able to see the road dashed line 502 and the road solid line 504. This low visibility may be caused by weather factors such as fog, rain, snow, mud, etc. Other factors that can cause poor visibility can include poor lighting conditions, lack of reflection of the lines, or faded road dashed line 502 and / or faded road solid line 504. In some scenarios, the road lines may not exist.

[0053] In various embodiments, the system presents or displays the road dashed line 502 and the road solid line 504 in the form of objects of virtual road lines (such as virtual road dashed line 612 and virtual road solid line 614) in the head-up display 410. The virtual road dashed line 612 and the virtual road solid line 614 respectively represent the physical road dashed line 502 and the road solid line 504. The system can also display a virtual horizon 616 that represents the actual horizon 606. In some scenarios where the road lines do not exist, the system can generate or obtain virtual road lines from an on-vehicle library or database and then display the virtual road lines in the head-up display 410 to assist the driver in navigating the road. Both the virtual road dashed line 612 and the virtual road solid line 614 provide or increase the driver's visual perception of the physical road dashed line 502 and the physical road solid line 504, which helps the driver navigate on the road, thereby increasing the perception of the external environment for safer driving.

[0054] In some embodiments, the system can enhance the display of virtual road lines where lanes suddenly merge or terminate. For example, the system can color-code the virtual merging or ending road lines in a predetermined color such as red to increase the perception of the changing road conditions. The system can also display an alert (e.g., "Lane Merge!" or "Lane End!") to increase the perception of the changing road conditions.

[0055] As described above, the head-up display 410 provides an augmented reality (AR) windshield that shows the actual physical road and enhances or overlays the road seen through the windshield 404 with the virtual road dashed line 612 and the virtual road solid line 614. In various embodiments, in the head-up display 410, the system can provide an enhanced version of the physical road dashed line 502 and the physical road solid line 504. For example, the system can use one or more perception sensors to capture an object 112 in the form of one or more images or videos. The system can process the images to enhance or clarify the images for better visibility. For example, the physical road dashed line 502 and the physical road solid line 504 may be blurred due to weather factors such as fog, rain, snow, mud, etc. The system can remove such weather factors from the images and display the physical road dashed line 502 and the physical road solid line 504 on the head-up display 410 in a clear and non-blurred manner.

[0056] In various embodiments, in the head-up display 410, the system may provide an alternative representation of physical objects. For example, due to blurring caused by weather factors such as fog, rain, snow, etc., the system may not be able to capture clear images of the physical road dashed lines 502 and the physical road solid lines 504. Or, the vehicle's headlights may suddenly go out. In such scenarios, the system may present or display in the head-up display 410 the shapes representing the physical road dashed lines 502 and the physical road solid lines 504. In some embodiments, the system may create or obtain a representation of the physical road dashed lines 502 and the physical road solid lines 504 and then display that representation in the head-up display 410.

[0057] In various embodiments, the system positions the virtual road dashed lines 612 and the virtual solid lines 614 in the head-up display 410 at positions corresponding to the positions of the actual physical road dashed lines 502 and the physical solid lines 504 in the windshield 404. This enables the driver of the vehicle to navigate the road based on the virtual road dashed lines 612 and the virtual road solid lines 614 in the head-up display 410.

[0058] In some scenarios, it may be difficult to distinguish what color the road lines are, making it difficult for the driver to know whether the lane is designed to travel in the same direction or the opposite direction on a given road. In some embodiments, the system may color-code the displayed virtual road lines accordingly.

[0059] In various embodiments, the system may utilize its sensing sensors and / or use AI and machine learning to track other known objects that contribute to safe driving. For example, in some scenarios, the system may track and present road signs, street signs, etc. and display these objects in the head-up display 410 to facilitate safe driving. When any dangerous obstacle appears, the system may display in the head-up display 410 any virtual object representing any dangerous obstacle.

[0060] In various embodiments, the system may generate virtual road lines based on prior knowledge, including the geographical location of the vehicle and the destination provided by the driver. For example, when the driver provides a destination, the system may calculate all areas where there are less visible road markings or no road markings. The system may prepare an enhanced view with virtual road lines and / or road markings before reaching a location with poor visibility or no road markings. When the driver is driving the vehicle, the system may display the virtual road lines and / or road markings in the head-up display 410 at an appropriate time before reaching that location.

[0061] In some embodiments, the system can enable a driver to manually turn on the presentation of virtual road markings. If the system determines that it would be safer to display such virtual road markings due to low lighting, absence of road markings, or other predetermined criteria, the system can automatically display the virtual road markings in the head-up display. In various embodiments, the system can obtain crowdsourced data on historical accidents and display appropriate warning symbols in areas where there have been a large number of traffic accidents historically.

[0062] In some embodiments, if a driver is driving a vehicle recklessly due to an unsafe condition, the system can initiate autonomous driving. For example, due to poor visibility of the road lines, the driver may be driving off the road or into oncoming traffic. Due to wet or slippery road conditions, the driver may be steering erratically. When an oncoming dangerous vehicle approaches, the driver may not be slowing down. The system can initiate autonomous driving and control the movement, speed, and direction of the vehicle in a safe manner in any of these situations or similar situations or a combination thereof.

[0063] The embodiments described herein have many benefits. For example, the embodiments automatically detect and identify objects located in the vehicle's path that the driver may not be able to see due to poor visibility. The embodiments also automatically use sensing sensors to detect, identify, and display such objects in a display to assist the driver in driving safely, where such objects can include potentially dangerous objects and / or road lines or other signs that contribute to safe driving.

[0064] Figure 7 is a block diagram of an example network environment 700 of the present disclosure. In some implementations, the network environment 700 includes a system 702, and the system 702 includes a server device 704 and a database 706. In various implementations, the system 702 can be used to implement Figure 1 the system 102, and execute the implementations described herein. The network environment 700 also includes client devices 710, 720, 730, and 740, which can communicate with the system 702 and / or can communicate with each other directly or via the system 702. The network environment 700 also includes a network 750 through which the system 702 and the client devices 710, 720, 730, and 740 communicate. The network 750 can be any suitable communication network, such as a Wi-Fi network, a Bluetooth network, the Internet, etc.

[0065] For ease of illustration, Figure 7A box is shown for each of system 702, server device 704, and network database 706, and four boxes are shown for client devices 710, 720, 730, and 740. Boxes 702, 704, and 706 may represent multiple systems, server devices, and network databases. Additionally, any number of client devices may exist. In other embodiments, environment 700 may not have all of the components shown and / or may have other elements including other types of elements in place of or in addition to those shown herein.

[0066] Although server device 704 of system 702 performs the embodiments described herein, in other implementations, any suitable component or combination of components associated with system 702 or any suitable one or more processors associated with system 702 may facilitate the performance of the embodiments described herein.

[0067] In various embodiments described herein, the processor of system 702 and / or the processors of any of client devices 710, 720, 730, and 740 may cause elements (e.g., information, etc.) described herein to be displayed on one or more display screens in a user interface.

[0068] Figure 8 is a block diagram of an example computing system 800 of the present disclosure. Computing system 800 may be used to implement Figure 7 the server of system 702 and / or Figure 1 system 102, and perform the embodiments described herein.

[0069] Computing system 800 generally includes at least one processing unit 802 and system memory 804. Depending on the particular configuration and type of the computing device, system memory 804 may be volatile, such as random access memory (RAM); non-volatile, such as read-only memory (ROM), flash memory, etc.; or some combination of volatile and non-volatile memory. System memory 804 generally maintains an operating system 806, one or more application programs 808, and program data 810. Operating system 806 may include any number of operating systems executable on a desktop or portable device, including but not limited to Linux, Microsoft Apple or

[0070] The computing system 800 may also have additional features or functionality. For example, the computing system 800 may also include additional data storage devices (removable and / or non-removable), such as, for example, magnetic disks, optical disks, magnetic tapes, or flash memories. Such additional storage devices may include a removable storage device 812 and a non-removable storage device 814. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. System memory 804, removable storage device 812, and non-removable storage device 814 are all examples of computer storage media. The available types of computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory (in removable and non-removable forms), or other memory technologies, CD-ROM, digital versatile disks (DVDs), or other optical storage, magnetic tape cartridges, magnetic tapes, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computing system 800. Any such computer storage media may be part of the computing system 800.

[0071] The computing system 800 may also have input devices 816, such as a keyboard, a mouse, a pen, a voice input device, a touchscreen input device, etc. Output devices 818 may also be included, such as a display, a speaker, a printer, a short-range transceiver such as a Bluetooth transceiver, etc. The computing system 800 may also include one or more communication connections 820 that allow the computing system 800 to communicate with other computing systems 822, such as via a wired or wireless network or via Bluetooth (the Bluetooth transceiver may be regarded as both an input / output device and a communication connection). One or more communication connections 820 are examples of communication media. The available forms of communication media generally carry computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transmission mechanism, and include any information delivery medium. The term "modulated data signal" may include a signal in which one or more of the characteristics of the signal are set or changed in a manner that encodes information in the signal. By way of illustrative example and not limitation, communication media may include wired media such as a wired network or a direct wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media. As used herein, the term computer-readable medium includes both storage media and communication media.

[0072] The computing system 800 may also include a positioning circuit 824. In various embodiments, the positioning circuit 824 may include circuitry that includes a Global Positioning System (GPS) circuit and / or a geolocation circuit. The positioning circuit 824 may automatically discern its location based on its relative position to multiple GPS satellites and / or using triangulation with a cellular carrier network and / or an IEEE standard 802.11 wireless (Wi-Fi) network (collectively referred to as "geolocation services") to determine the location based on multiple cellular communication facilities and / or multiple Wi-Fi networks. The positioning circuit 824, which includes a GPS circuit and / or a geolocation circuit, is often incorporated into smart phones and many other tablet computers or other portable devices. In various embodiments, the computing system 800 may not have all of the components shown and / or may have other elements including other types of components in place of or in addition to those shown herein.

[0073] Although the present disclosure has been illustrated and described herein with reference to the provided illustrative embodiments and specific examples, it will be apparent to those of ordinary skill in the art that other embodiments and examples may perform similar functions and / or achieve similar results. All such equivalent embodiments and examples are within the spirit and scope of the present disclosure and are intended to be covered by the following non-limiting claims for all purposes.

Claims

1. A computer-implemented method for providing awareness of an object located in a path of a vehicle, the method comprising: capturing data about an external environment using at least one sensory sensor; detecting one or more objects in a path of the vehicle based on the captured data; estimating a position of the one or more objects in a path of the vehicle; and One or more virtual objects are presented using at least one display positioned within the interior of the vehicle, wherein the one or more virtual objects represent the one or more detected objects.

2. The method according to claim 1, wherein: The one or more objects in the path of the vehicle are dangerous obstacles.

3. The method according to claim 1, wherein: The one or more objects in the path of the vehicle are road markings. 4 . The method of claim 1 , further comprising obtaining crowd-sourced data regarding potentially hazardous obstacles in the path of the vehicle.

5. The method according to claim 1, wherein: The one or more virtual objects presented using the at least one display are virtual hazardous obstacles.

6. The method according to claim 1, wherein: The one or more objects presented using the at least one display are virtual road lines.

7. The method according to claim 1, further comprising: If one or more of the detected objects are potentially dangerous obstacles, the driver of the vehicle is alerted.

8. The method according to claim 1, wherein: The at least one perception sensor may include at least one of a camera, a radar detector, a light detection and ranging (Lidar) camera, or an ultrasonic camera.

9. An assembly for a vehicle, the assembly comprising: at least one perception sensor; at least one display positioned within an interior of the vehicle; and A system comprising one or more processors and logic encoded in one or more non-transitory computer-readable storage media, the logic being operable to be executed by the one or more processors and, when executed, causing the one or more processors to perform operations comprising: capturing data about the external environment using the at least one sensory sensor; detecting one or more objects in a path of the vehicle based on the captured data; estimating the position of the one or more objects in the path of the vehicle; and One or more virtual objects are presented using the at least one display positioned within the interior of the vehicle, wherein the one or more virtual objects represent the one or more detected objects.

10. The assembly of claim 9, wherein the one or more objects in the path of the vehicle are dangerous obstacles.

11. The assembly of claim 9, wherein the one or more objects in the path of the vehicle are road markings.

12. The assembly of claim 9, wherein the logic, when executed, is further operable to cause the one or more processors to perform operations including obtaining crowd-sourced data regarding potentially hazardous obstacles in a path of the vehicle.

13. The assembly of claim 9, wherein the one or more virtual objects presented using the at least one display are virtual hazardous obstacles.

14. The assembly of claim 9, wherein the one or more objects presented using the at least one display are virtual road lines.

15. The component of claim 9, wherein the logic, when executed, is further operable to cause the one or more processors to perform operations comprising: If one or more of the detected objects is a potentially dangerous obstacle, the driver of the vehicle is alerted.

16. The assembly of claim 9, wherein the at least one perception sensor may include at least one of a camera, a radar detector, a light detection and ranging (Lidar) camera, or an ultrasonic camera.

17. A non-transitory computer-readable storage medium having program instructions stored thereon, the program instructions being operable when executed by one or more processors to cause the one or more processors to perform operations comprising: capturing data about an external environment using at least one sensory sensor; detecting one or more objects in a path of the vehicle based on the captured data; estimating a position of the one or more objects in a path of the vehicle; and One or more virtual objects are presented using at least one display positioned within the interior of the vehicle, wherein the one or more virtual objects represent the one or more detected objects.

18. The computer-readable storage medium of claim 17, wherein the one or more objects in the path of the vehicle are dangerous obstacles.

19. The computer-readable storage medium of claim 17, wherein the one or more objects in the path of the vehicle are road markings.

20. The computer-readable storage medium of claim 17, wherein the instructions, when executed, are further operable to cause the one or more processors to perform operations including obtaining crowd-sourced data regarding potentially hazardous obstacles in a path of the vehicle.

Images