Augmented reality door opening alarm

Abstract

A system for providing an alert to an occupant within a vehicle before the occupant leaves the vehicle includes: a plurality of on-board sensors adapted to collect information related to an environment external to the vehicle; an occupant monitoring system adapted to monitor and detect movement of an occupant within the vehicle; a display screen adapted to be viewed by an occupant in the vehicle; and a system controller adapted to detect when an occupant in the vehicle is about to leave the vehicle, detect a location of the occupant in the vehicle, display a view of an environment external to the vehicle on the display screen, determine whether a hazardous object is present within the environment external to the vehicle and classify the hazardous object, and provide an augmented reality alert within the display screen.

Description

Technical Field

[0001] The present disclosure relates to a system and method for providing a hazard warning when an occupant of a vehicle exits the vehicle. Background Art

[0002] Vehicles are equipped with many sensors to monitor the environment around the vehicle and provide warnings when objects are approaching, etc. Vehicles are also equipped with mirrors to allow vehicle occupants to see objects located near the vehicle that may be approaching the vehicle. Current sensor systems and mirrors are not suitable for actively monitoring the vehicle's surroundings for objects and hazards that may cause harm to occupants leaving the vehicle. This is particularly a problem for rear seat passengers, who typically do not have external mirrors to help them see the vehicle's surroundings before leaving the vehicle.

[0003] Therefore, while current systems and methods achieve their intended purposes, a need remains for a new and improved system and method for providing warnings to vehicle occupants of hazards detected in the vicinity before the occupants exit the vehicle. Summary of the invention

[0004] According to several aspects of the present disclosure, a method for providing a warning to an occupant in a vehicle before the occupant leaves the vehicle includes: detecting when an occupant in the vehicle is about to leave the vehicle using a system controller that communicates with multiple on-board sensors and an occupant monitoring system; detecting the position of the occupant in the vehicle using the system controller; displaying a view of the vehicle's external environment on a display screen adapted for viewing by the occupant using the system controller; determining whether a hazardous object is present in the vehicle's external environment using the system controller; classifying the hazardous object when the hazardous object is present in the vehicle's external environment; and providing an augmented reality alert in the display screen.

[0005] According to another aspect, the method further includes determining, using the system controller, whether a door has been opened; determining, using the system controller, whether an additional alarm is required when the door has been opened; and activating, using the system controller, the additional alarm if an additional alarm is required.

[0006] According to another aspect, detecting when an occupant in a vehicle is about to exit the vehicle using a system controller that communicates with a plurality of onboard sensors and an occupant monitoring system also includes determining, using the system controller, whether the vehicle is at least one of: stopped, parked, and not moving; determining, using the system controller via the occupant monitoring system, whether the occupant is reaching for a door latch in the vehicle; and actuating, using the system controller, an exterior camera on the vehicle and capturing, using the exterior camera, an image of an environment external to the vehicle when at least one of the following occurs: the vehicle is at least one of stopped, parked, and not moving, and the occupant is reaching for a door latch in the vehicle.

[0007] According to another aspect, displaying a view of the vehicle's exterior environment on a display screen adapted for viewing by an occupant using the system controller also includes at least one of: displaying a view of the vehicle's exterior environment on a dedicated display screen mounted to a vehicle door using the system controller, and displaying a view of the vehicle's exterior environment on a display screen of an in-vehicle infotainment system using the system controller.

[0008] According to another aspect, determining, using the system controller, whether a hazardous object is present in the environment external to the vehicle further comprises detecting, using the system controller and a plurality of onboard sensors, at least one moving object in the environment external to the vehicle.

[0009] According to another aspect, detecting at least one moving object in the external environment of a vehicle using a system controller and multiple on-board sensors also includes analyzing the speed, position, acceleration, trajectory and recent changes of any one of the speed, position, acceleration and trajectory of the at least one moving object relative to the vehicle.

[0010] According to another aspect, determining, using the system controller, whether a hazardous object is present in the environment external to the vehicle further comprises detecting, using the system controller and a plurality of onboard sensors, at least one travel hazard in the environment external to the vehicle.

[0011] According to another aspect, determining, using the system controller, whether a hazardous object is present within the vehicle's external environment also includes determining, using the system controller, a probability that at least one moving object will collide with at least one of a door and an occupant during the occupant's exit from the vehicle; and determining, using the system controller, a probability that at least one travel hazard will cause the occupant to at least one of trip, stumble, and fall during the occupant's exit from the vehicle.

[0012] According to another aspect, using a system controller to determine the probability that at least one moving object will collide with at least one of a door and an occupant during the occupant's exit from the vehicle, and using the system controller to determine the probability that at least one travel hazard will cause the occupant to trip, stumble, and fall, also includes: using a machine learning algorithm to probabilistically calculate the probability that at least one moving object will collide with at least one of a door and an occupant during the occupant's exit from the vehicle, and the probability that at least one travel hazard will cause the occupant to trip, stumble, and fall.

[0013] According to several aspects of the present disclosure, a system for providing a warning to an occupant in a vehicle before the occupant leaves the vehicle includes: a plurality of on-board sensors adapted to collect information related to the environment outside the vehicle; an occupant monitoring system adapted to monitor and detect movement of occupants in the vehicle; a display screen adapted for viewing by occupants in the vehicle; and a system controller that communicates with the plurality of on-board sensors, the occupant monitoring system, and the display screen, the system controller being adapted to detect when an occupant in the vehicle is about to leave the vehicle, detect the position of the occupant in the vehicle, display a view of the environment outside the vehicle collected by at least one of the plurality of sensors on the display screen, determine whether a dangerous object is present in the environment outside the vehicle, and when a dangerous object is present in the environment outside the vehicle, classify the dangerous object, and provide an augmented reality alert in the display screen.

[0014] According to another aspect, the system controller is further adapted to determine whether a vehicle door has been opened, and when the vehicle door has been opened, determine whether an additional alarm is required, and activate the additional alarm when an additional alarm is required.

[0015] According to another aspect, when detecting whether an occupant in the vehicle is about to leave the vehicle, the system controller is further adapted to determine whether the vehicle is at least one of: stopped, parked, and not moving; determine via the occupant monitoring system whether the occupant is touching a door latch in the vehicle; and when at least one of the following two situations occurs: the vehicle is at least one of stopped, parked, and not moving and the occupant is touching a door latch in the vehicle, actuate an exterior camera on the vehicle and capture an image of the vehicle's external environment using the exterior camera.

[0016] According to another aspect, when displaying a view of the vehicle exterior environment on the display screen, the system controller is further adapted to display at least one of the view of the vehicle exterior environment on a dedicated display screen mounted to a vehicle door and the view of the vehicle exterior environment on a display screen of an in-vehicle infotainment system.

[0017] According to another aspect, when determining whether a hazardous object is present within the vehicle exterior environment, the system controller is further adapted to detect at least one moving object in the vehicle exterior environment.

[0018] According to another aspect, when at least one moving object in the external environment of the vehicle is detected, the system controller is further adapted to analyze the speed, position, acceleration, trajectory and recent changes of any one of the speed, position, acceleration and trajectory of the at least one moving object relative to the vehicle.

[0019] According to another aspect, when determining whether a hazardous object is present within the vehicle exterior environment, the system controller is further adapted to detect at least one travel hazard in the vehicle exterior environment.

[0020] According to another aspect, when determining whether a hazardous object is present within the vehicle's external environment, the system controller is further adapted to determine a probability that at least one moving object will collide with at least one of a door and an occupant during the occupant's exit from the vehicle, and to determine a probability that at least one travel hazard will cause the occupant to at least one of trip, stumble, and fall during the occupant's exit from the vehicle.

[0021] According to another aspect, when determining a probability that at least one moving object will collide with at least one of a door and an occupant during the occupant's exit from the vehicle, and when determining a probability that at least one travel hazard will cause the occupant to at least one of trip, stumble, and fall, the system controller is further adapted to probabilistically calculate using a machine learning algorithm the probability that at least one moving object will collide with at least one of a door and an occupant during the occupant's exit from the vehicle, and the probability that at least one travel hazard will cause the occupant to trip, stumble, and fall.

[0022] Further areas of applicability will become apparent from the description provided herein.It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.

[0024] Figure 1 is a schematic diagram of a vehicle having a system for providing a hazardous object warning to an occupant before the occupant exits the vehicle according to an exemplary embodiment;

[0025] Figure 2 is a schematic diagram of a system according to an exemplary embodiment;

[0026] Figure 3 is a schematic diagram of the interior of a vehicle with occupants seated therein;

[0027] Figure 4 is an image displayed on the display screen including two persons on bicycles approaching the vehicle;

[0028] Figure 5 It is an image displayed on the display screen, including a moving object such as a car;

[0029] Figure 6 It is an image displayed on the display screen, including the travel hazard of a pothole next to the vehicle;

[0030] Figure 7 is an image displayed on a display screen, including a hazard for movement of the roadside kerb adjacent to the vehicle; and

[0031] Figure 8is a flow chart illustrating a method of providing a warning to an occupant in a vehicle before the occupant exits the vehicle according to an exemplary embodiment.

[0032] These figures are not necessarily to scale, and some features may be enlarged or reduced, such as to illustrate details of a particular component. In some cases, well-known components, systems, materials, or methods are not described in detail to avoid obscuring the present disclosure. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching those skilled in the art to use the present disclosure in various ways. DETAILED DESCRIPTION

[0033] The following description is essentially only exemplary and is not intended to limit the present disclosure, application or use. In addition, it is not desired to be bound by any express or implied theory presented in the aforementioned technical field, background technology, summary of the invention or the following specific embodiments. It should be understood that in all drawings, the corresponding reference numerals represent the same or corresponding parts and features. As used herein, the term module refers to any hardware, software, firmware, electronic control component, processing logic and / or processor device individually or in any combination, including but not limited to: application-specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated or grouped) and memories that execute one or more software or firmware programs, combinational logic circuits and / or other suitable components that provide the functions. Although the drawings shown in this article describe examples with certain element arrangements, additional intermediate elements, devices, features or components may be present in actual embodiments. It should also be understood that the drawings are merely illustrative and may not be drawn to scale.

[0034] As used herein, the term "vehicle" is not limited to automobiles. Although the present technology is described primarily in conjunction with automobiles herein, the present technology is not limited to automobiles. These concepts can be used in a variety of applications, such as those related to aircraft, ships, other vehicles, and consumer electronic components.

[0035] According to an exemplary embodiment, Figure 1 A vehicle 10 is shown with an associated system 11 according to various embodiments for providing a warning to an occupant 56 of the vehicle 10 of a hazardous object 64 in the environment outside the vehicle 10 before the occupant 56 leaves the vehicle 10. Typically, the system 11 works with other systems within the vehicle 10 to display various information and infotainment content to the occupants. The vehicle 10 generally includes a chassis 12, a body 14, front wheels 16, and rear wheels 18. The body 14 is disposed on the chassis 12 and generally substantially surrounds the components of the vehicle 10. The body 14 and the chassis 12 may together form a frame. The front wheels 16 and the rear wheels 18 are each rotatably coupled to the chassis 12 near respective corners of the body 14.

[0036] In various embodiments, the vehicle 10 is an autonomous vehicle, and the system 11 is incorporated into the autonomous vehicle 10. The autonomous vehicle 10 is, for example, a vehicle 10 that is automatically controlled to transport passengers from one location to another. The vehicle 10 is depicted as a passenger car in the illustrated embodiment, but it should be understood that any other vehicle including motorcycles, trucks, sport utility vehicles (SUVs), recreational vehicles (RVs), etc. may also be used. In an exemplary embodiment, the vehicle 10 is equipped with a so-called level 4 or level 5 automation system. The level 4 system represents "high automation" and refers to the performance of the autonomous driving system in a specific driving mode of all aspects of the dynamic driving task even if the human driver does not respond appropriately to the intervention request. The level 5 system represents "full automation" and refers to the full-time performance of the autonomous driving system in all aspects of the dynamic driving task under all road and environmental conditions that can be managed by the human driver. The novel aspects of the present disclosure are also applicable to non-autonomous vehicles.

[0037] As shown, the vehicle 10 generally includes a propulsion system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, a vehicle controller 34, and a wireless communication module 36. In an embodiment where the vehicle 10 is an electric vehicle, there may be no transmission system 22. In various embodiments, the propulsion system 20 may include an internal combustion engine, an electric machine such as a traction motor, and / or a fuel cell propulsion system. The transmission system 22 is configured to transmit power from the propulsion system 20 to the front wheels 16 and rear wheels 18 of the vehicle according to a selectable speed ratio. According to various embodiments, the transmission system 22 may include a stepped ratio automatic transmission, a continuously variable transmission, or other suitable transmission. The braking system 26 is configured to provide braking torque to the front wheels 16 and rear wheels 18 of the vehicle. In various embodiments, the braking system 26 may include friction brakes, line brakes, regenerative braking systems such as electric machines, and / or other suitable braking systems. The steering system 24 affects the position of the front wheels 16 and rear wheels 18. Although depicted as including a steering wheel for purposes of illustration, in some embodiments contemplated within the scope of the present disclosure, the steering system 24 may not include a steering wheel.

[0038] The sensor system 28 includes one or more sensing devices 40a-40n that sense observable conditions of the external environment and / or internal environment of the autonomous vehicle 10. The sensing devices 40a-40n may include, but are not limited to, radar, lidar, global positioning system, optical camera, thermal camera, ultrasonic sensor, and / or other sensors. The camera may include two or more digital cameras spaced apart from each other at a selected distance, wherein the two or more digital cameras are used to obtain stereo images of the surrounding environment in order to obtain a three-dimensional image or map. The plurality of sensing devices 40a-40n are used to determine information about the surrounding environment of the vehicle 10. In an exemplary embodiment, the plurality of sensing devices 40a-40n include at least one of a motor speed sensor, a motor torque sensor, an electric drive motor voltage and / or current sensor, an accelerator pedal position sensor, a coolant temperature sensor, a cooling fan speed sensor, and a transmission oil temperature sensor. In another exemplary embodiment, the plurality of sensing devices 40a-40n also include sensors for determining information about the environment surrounding the vehicle 10, such as an ambient air temperature sensor, an atmospheric pressure sensor, and / or a camera and / or video camera positioned to view the environment in front of the vehicle 10. In another exemplary embodiment, at least one of the plurality of sensing devices 40 a - 40 n is capable of measuring distances in the environment surrounding the vehicle 10 .

[0039] In a non-limiting example in which the plurality of sensing devices 40a-40n include cameras, the plurality of sensing devices 40a-40n measure distance using an image processing algorithm configured to process images from the cameras and determine the distance between objects. In another non-limiting example, the plurality of vehicle sensors 40a-40n include a stereo camera with distance measurement capabilities. In one example, at least one of the plurality of sensing devices 40a-40n is fixed to the interior of the vehicle 10, such as in the roof of the vehicle 10, with a field of view through the windshield of the vehicle 10. In another example, at least one of the plurality of sensing devices 40a-40n is a camera fixed to the exterior of the vehicle 10 (e.g., on the roof of the vehicle 10), which has a field of view of the environment surrounding the vehicle 10 and is adapted to collect information (images) related to the environment outside the vehicle 10. It should be understood that various other types of sensing devices, such as lidar sensors, ultrasonic ranging sensors, radar sensors, and / or time-of-flight sensors are within the scope of the present disclosure. The actuator system 30 includes one or more actuator devices 42 a - 42 n that control one or more features of the vehicle 10 , such as, but not limited to, the propulsion system 20 , the transmission system 22 , the steering system 24 , and the braking system 26 .

[0040] The vehicle controller 34 includes at least one processor 44 and a computer readable storage device or medium 46. The at least one data processor 44 can be any custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor of several processors associated with the vehicle controller 34, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, any combination thereof, or any device generally used to execute instructions. The computer readable storage device or medium 46 can include volatile and non-volatile storage devices such as read-only memory (ROM), random access memory (RAM), and keep alive memory (KAM). KAM is a persistent or non-volatile memory that can be used to store various operating variables when the at least one data processor 44 is powered off. The computer readable storage device or medium 46 can be implemented using any of a variety of known storage devices, such as PROM (programmable read-only memory), EPROM (electrically programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), flash memory, or any other electrical, magnetic, optical, or combination storage device capable of storing data, some of which represents executable instructions used by the controller 34 in controlling the vehicle 10.

[0041] The instructions may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions. When executed by at least one processor 44, the instructions receive and process signals from the sensor system 28, perform logic, calculations, methods and / or algorithms for automatically controlling components of the vehicle 10, and generate control signals to the actuator system 30 based on the logic, calculations, methods and / or algorithms to automatically control components of the vehicle 10. Although in Figure 1 Only one controller 34 is shown, but embodiments of the vehicle 10 may include any number of controllers 34 that communicate via any suitable communication medium or combination of communication mediums and collaborate to process sensor signals, execute logic, calculations, methods and / or algorithms, and generate control signals to automatically control features of the autonomous vehicle 10.

[0042] In various embodiments, one or more instructions of the vehicle controller 34 are included in the trajectory planning system and, when executed by at least one data processor 44, generate trajectory outputs that process kinematic and dynamic constraints of the environment. For example, the instructions receive process sensor and map data as input. The instructions execute a graph-based approach with a customized cost function to handle different road scenarios in both urban and highway roads.

[0043] The wireless communication module 36 is configured to wirelessly transmit information to and from other remote entities 48, such as, but not limited to, other vehicles ("V2V" communication), infrastructure ("V2I" communication), remote systems, remote servers, cloud computers, and / or personal devices. In an exemplary embodiment, the communication system 36 is a wireless communication system configured to communicate via a wireless local area network (WLAN) using the IEEE 802.11 standard or by using cellular data communications. However, additional or alternative communication methods such as dedicated short-range communication (DSRC) channels are also considered to be within the scope of the present disclosure. A DSRC channel refers to a one-way or two-way short-range to medium-range wireless communication channel dedicated to automotive use and a corresponding set of protocols and standards.

[0044] The vehicle controller 34 is a non-universal electronic control device having a pre-programmed digital computer or processor, a memory or non-transitory computer-readable medium for storing data such as control logic, software applications, instructions, computer code, data, look-up tables, etc., and a transceiver [or input / output port]. Computer-readable media include any type of media that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), hard drive, compact disk (CD), digital video disk (DVD), or any other type of memory. "Non-transitory" computer-readable media do not include wired, wireless, optical or other communication links that transmit temporary electrical or other signals. Non-transitory computer-readable media include media in which data can be permanently stored and media in which data can be stored and later rewritten, such as rewritable optical disks or erasable memory devices. Computer code includes any type of program code, including source code, object code, and executable code.

[0045] refer to Figure 2 , a schematic diagram of the system 11 is shown. The system 11 includes a system controller 34A in communication with a plurality of sensing devices (on-board sensors) 40a-40n. The system controller 34A may be the vehicle controller 34, or the system controller 34A may be a separate controller in communication with the vehicle controller 34. In addition to the plurality of on-board sensors 40a-40n, the system controller 34A communicates with an occupant monitoring system 50 adapted to monitor and detect movement of occupants within the vehicle, and a display screen 52 and a database 54 adapted to be viewed by occupants within the vehicle.

[0046] In an exemplary embodiment, the system controller 34A is adapted to detect when an occupant 56 in the vehicle 10 is about to exit the vehicle 10. The system controller 34A identifies conditions that indicate when an occupant 56 in the vehicle 10 is about to exit the vehicle 10. When determining whether the occupant 56 is about to exit the vehicle 10, the system controller 34A determines whether the vehicle 10 is at least one of stopped, parked, and not moving via access to a plurality of onboard sensors 40a-40n and onboard vehicle systems. In addition, see Figure 3 , the system controller 34A determines, via communication with the occupant monitoring system 50, whether an occupant 56 within the vehicle 10 is contacting a door latch 58 within the vehicle 10. The occupant monitoring system 50 includes at least one camera 60 adapted to capture data (images) related to the movement of the occupant 56 within the vehicle 10. Using image recognition techniques, the system controller 34A is adapted to recognize when the movement of the occupant 56 captured by the at least one camera 60 of the occupant monitoring system 50 indicates that the occupant 56 is contacting the door latch 58 within the vehicle 10, thereby indicating that the occupant 56 is about to leave the vehicle 10.

[0047] If the system controller 34A recognizes that the vehicle 10 is stopped, parked, or the engine of the vehicle 10 is not running, or if the system controller 34A detects via the occupant monitoring system 50 that the occupant 56 is reaching for the door latch 58 within the vehicle 10, the system controller 34A determines that the occupant 56 is about to leave the vehicle 10, activates at least one external camera on the vehicle 10 (at least one of the plurality of onboard sensors 40a-40n), and captures an image of the environment outside the vehicle 10 using the at least one external camera.

[0048] The system controller 34A is adapted to identify the position of the occupants 56 within the vehicle 10 via communication with the plurality of onboard sensors 40a-40n and to display a view of the environment outside the vehicle 10 collected by at least one of the plurality of sensors (at least one external camera) 40a-40n on the display screen 52. In an exemplary embodiment, the vehicle 10 includes a display screen 52 associated with each passenger position within the vehicle 10. More specifically, the vehicle 10 includes at least one display screen 52 for each passenger position within the vehicle 10 that is adjacent to a door 62 of the vehicle 10, wherein any occupant 56 within the vehicle 10 that is able to access the door 62 of the vehicle 10 can view an image of the environment outside the vehicle 10 on the display screen 52 associated therewith.

[0049] Reference again Figure 3When displaying a view of the environment outside the vehicle 10 on the display screen 52, the system controller 34A is also adapted to display at least one of a view of the environment outside the vehicle 10 on a dedicated display screen 52A mounted to a door 62 of the vehicle 10 and a view of the environment outside the vehicle 10 on a display screen 52B of an in-vehicle infotainment system. In an exemplary embodiment, the dedicated display screen 52A is mounted directly on the door 62 near the occupant 56 in the vehicle 10. The dedicated display screen 52A is mounted on the door 62 near the door latch 58 so that when the occupant 56 intends to leave the vehicle 10, the occupant 56 will naturally glance at the dedicated display screen 52A when reaching the door latch 58. In another exemplary embodiment, the system controller 34A displays a view of the environment outside the vehicle 10 on a display screen 52B of an in-vehicle infotainment system. Such an infotainment display screen 52B is located at a position that can be easily seen by the occupant 56 in the vehicle 10. As shown, the display screen 52B of the infotainment system is mounted in the headrest of the seat in front of the occupant 56. It should be understood that the infotainment display screen 52B can be of any known type and can be located at a location known in the infotainment display screen industry. It should also be understood that the system controller 34A can display a view of the environment outside the vehicle 10 on either or both of the dedicated display screen 52A on the door 62 and the display screen 52B of the infotainment system.

[0050] The system controller 34A is also adapted to determine if a hazardous object 64 is present within the environment external to the vehicle 10. A hazardous object 64 is any object within a predetermined distance of the vehicle 10 that could pose a threat to the occupant 56 during exit from the vehicle 10. A hazardous object 64 is any object that could collide with at least one of the door 62 of the vehicle 10 or the occupant 56 during exit from the vehicle 10, or an object (broken concrete, curb, pothole) or road condition (ice, water, debris) that could cause the occupant 56 to trip, stumble, or fall during exit from the vehicle 10.

[0051] In an exemplary embodiment, when determining whether a hazardous object 64 is present in the environment outside the vehicle 10, the system controller 34A is further adapted to detect at least one moving object 64A in the environment outside the vehicle 10. The system controller 34A uses computer vision and image recognition techniques applied to images of the environment outside the vehicle 10 captured by at least one external camera 40a-40n to identify any moving object 64A within a predetermined distance of the vehicle 10. Such an identified moving object 64A may be another vehicle, a pedestrian, a cyclist, a child playing nearby, or any other moving object 64A within a predetermined distance. The predetermined distance is a distance within which the moving object 64A is determined to pose a threat to the occupant 56 exiting the vehicle 10.

[0052] The system controller 34A also analyzes the speed, position, acceleration, trajectory, and recent changes in any of the speed, position, acceleration, and trajectory of each of the at least one moving object 64A relative to the vehicle 10. Therefore, if any of the at least one moving object 64A is moving toward the vehicle 10, away from the vehicle 10, moving erratically, moving at a high speed, etc., the system controller 34A can identify which moving object 64A it is to determine whether the at least one moving object 64A will collide with at least one of the door 62 of the vehicle 10 or the occupant 56 during the period when the occupant 56 leaves the vehicle 10, and therefore the moving object is a dangerous object 64.

[0053] In another exemplary embodiment, when determining whether a hazardous object 64 is present within the environment external to the vehicle 10, the system controller 34A is further adapted to detect at least one travel hazard 64B in the environment external to the vehicle 10. The travel hazard 64B is any object (concrete debris, curb, pothole) or road condition (ice, water, debris) that could cause the occupant 56 to trip, stumble, or fall during the occupant 56's exit from the vehicle 10.

[0054] In an exemplary embodiment, when determining whether a hazardous object 64 is present within the environment external to the vehicle 10, the system controller 34A is further adapted to determine a probability that the at least one moving object 64A will collide with at least one of the door 62 of the vehicle 10 and the occupant 56 during the occupant 56's exit from the vehicle 10, and a probability that the at least one traveling hazard 64B will cause the occupant 56 to trip, stumble, or fall during the occupant 56's exit from the vehicle 10. If the probability that the at least one moving object 64A will collide with at least one of the door 62 of the vehicle 10 and the occupant 56 during the occupant 56's exit from the vehicle 10 exceeds a predetermined threshold, then the at least one moving object 64A is classified as a hazardous object 64. Similarly, if the probability that the at least one traveling hazard 64B will cause the occupant 56 to trip, stumble, or fall during the occupant 56's exit from the vehicle 10 exceeds a predetermined threshold, then the at least one traveling hazard 64B is classified as a hazardous object 64.

[0055] For example, see again Figure 3, within the captured image of the external environment of the vehicle 10, the system controller 34A uses computer vision and image recognition technology to identify a moving object 64A (here, a car) that is approaching the vehicle 10 on the side where the occupant 56 is about to leave the vehicle 10. After analyzing the speed and trajectory of the identified moving object (vehicle) 64A, the system controller 34A determines that if the occupant 56 continues to leave the vehicle 10, the moving object (vehicle) 64A will have a 60% chance of colliding with the door 62 or the occupant 56. The predetermined threshold can be preset to, for example, 50%. Therefore, if the occupant 56 continues to leave the vehicle 10, the probability that the moving object (car) 64A will collide with the door 62 or the occupant 56 exceeds the predetermined threshold of 50%, and the moving object (car) 64A is classified as a dangerous object 64.

[0056] In an exemplary embodiment, when determining a probability that at least one moving object 64A will collide with at least one of a door 62 of the vehicle 10 and an occupant 56 during the occupant 56's exit from the vehicle 10, and when determining a probability that at least one travel hazard 64B will cause the occupant 56 to trip, stumble, and fall, the system controller 34A is further adapted to probabilistically calculate using a machine learning algorithm 70 the probability that at least one moving object 64A will collide with at least one of a door 62 of the vehicle 10 and an occupant 56 during the occupant 56's exit from the vehicle 10, and the probability that the at least one travel hazard 64B will cause the occupant 56 to trip, stumble, and fall.

[0057] A database 54 in communication with the system controller 34A is adapted to store data relating to past events in which the system controller 34A identified a hazardous object 64, as well as data relating to environmental aspects (weather, temperature, precipitation) and vehicle aspects (position, parked, stopped, engine off) during which such past events occurred.

[0058] The machine learning algorithm 70 is adapted to predict the probability that at least one moving object 64A will collide with at least one of the door 62 of the vehicle 10 and the occupant 56 during the occupant 56's exit from the vehicle 10, and the probability that at least one travel hazard 64B will cause the occupant 56 to at least one of trip, stumble, and fall during the occupant 56's exit from the vehicle 10, based on real-time data collected by multiple on-board sensors 40a-40n, analysis of captured images using computer vision and image recognition techniques, and data from past events from the database 54.

[0059] Various techniques are used to extract meaningful features from sensor readings and data, including time series analysis, frequency domain analysis, and space-time patterns. The machine learning algorithm 70 can be, but is not limited to, one of the following: long short-term memory (LSTM), gated recurrent unit (GRU), decision tree, random forest, support vector machine (SVM), neural network (NN), K nearest neighbor method (KNN), gradient boosting, and recurrent neural network (RNN). The machine learning algorithm 70 is trained with data collected from multiple different vehicles. Different data sets are collected from vehicles equipped with sensors such as GPS, accelerometers, cameras, radars, and LIDARs. The data contains various driving scenarios, including urban, highway, and off-road driving. Before feeding the data into the machine learning model, preprocessing steps are taken to remove noise, handle missing values, and standardize features. The basic step in driving behavior classification is to extract relevant features from the raw data. As described above, various techniques are used to extract meaningful features from sensor readings, including time series analysis, frequency domain analysis, and space-time patterns. Different types of machine learning algorithms can be used for probabilistic identification of dangerous objects. The machine learning algorithm 70 is trained on a labeled data set and evaluated using various performance metrics such as accuracy, precision, recall, F1 score, and confusion matrix. The hyperparameters of the model are adjusted to achieve optimal results. The machine learning algorithm 70 is also trained on the training data and will learn to map input features to corresponding probabilities.

[0060] Data from the system controller 34A identifying and classifying the occurrence of hazardous objects 64 within the vehicle 10 is saved to the database 54 and used to continuously update the machine learning algorithm 70. Thus, the system controller 34A uses the machine learning algorithm 70 and machine learning techniques to predict the probability that at least one moving object 64A will collide with at least one of the door 62 of the vehicle 10 and the occupant 56 during the occupant 56's exit from the vehicle 10, and to determine the probability that at least one traveling hazard 64B will cause the occupant 56 to trip, stumble, or fall during the occupant 56's exit from the vehicle 10, based on analysis of real-time data collected by the plurality of onboard sensors 40a-40n, analysis of captured images using computer vision and image recognition techniques, and data from past events from the database 54.

[0061] When the system controller 34A identifies a hazardous object 64 in the environment outside the vehicle 10, the system controller 34A further classifies the hazardous object 64 and determines a vulnerability / severity level for the hazardous object 64. For example, referring again to Figure 3, the system controller 34A identifies a dangerous object 64, here a moving object (car) 64A. Using computer vision and image recognition techniques, the system controller 34A identifies the moving object (car) 64A as a vehicle. Based on the identification of the moving object 64A, the vehicle, and other aspects (such as the speed and trajectory of the vehicle), the system controller 34A will determine the relative danger posed by the moving object (car) 64A. Reference Figure 4 , the system controller 34A identifies two dangerous objects 64, here a first bicyclist 64A1 and a second bicyclist 64A2. Using computer vision and image recognition techniques, the system controller 34A identifies the moving objects 64A1, 64A2 as bicyclists. Based on the identification of the objects 64A1, 64A2 and other aspects, such as the speed and trajectory of the bicycles, the system controller 34A will determine the relative dangers posed by the first bicyclist 64A1 and the second bicyclist 64A2. In this example, the severity of the potential danger posed by the vehicle 64A is much greater than the severity of the potential danger posed by the bicycles 64A1, 64A2. This determination of relative severity is used when determining the appropriate enhancement to provide within the displayed image of the environment outside the vehicle 10, as described below.

[0062] When a hazardous object 64 is identified by the system controller 34A within the environment external to the vehicle 10, the system controller 34A is also adapted to provide an augmented reality alert 72 within the display screen 52. The augmented reality alert 72 is adapted to draw the attention of the occupant 56 to the hazardous object 64 so that the occupant 56 can make a decision to continue exiting the vehicle 10 or wait until the hazardous object 64 passes.

[0063] Reference again Figure 3 , the system controller 34A overlays the augmented reality alert 72 on the image shown on the display 52. ​​As shown, the augmented reality alert 72 includes a graphic, here a warning triangle icon, overlaid on the moving object (car) 64A. As described above, the augmented reality alert 72 can be color-coded based on the relative severity level of the dangerous object 64. For example, Figure 3 The moving object (car) 64A in the image poses a relatively serious potential hazard, so the augmented reality alert 72 (graphic of a warning triangle) is displayed in red. Figure 5 , the moving object (car) 64A3 is identified as a dangerous object, but based on the distance from the vehicle 10, Figure 5 The moving object (car) 64A3 in the image poses less relative danger, so the augmented reality alert 72A (graphic of a warning triangle) is displayed in yellow.

[0064] In another example, again referring to Figure 4, two people riding bicycles 64A1, 64A2 are identified as dangerous objects 64 in the image of the environment outside the vehicle 10. Due to proximity to the vehicle 10, speed and / or behavior, the first bicycle rider 64A1 is identified as a dangerous object 64, which poses a relatively higher risk than the second bicycle rider 64A2. This can be based entirely on proximity to the vehicle 10 (the second bicycle rider 64A2 is farther away), or it can be based on the riding behavior of the first bicycle rider 64A1 and the second bicycle rider 64A2 (the first bicycle rider 64A1 moves quickly and erratically, and the second bicycle rider 64A2 moves slowly and steadily), or it can be a combination of both, thus causing the system controller 34A to overlay a red augmented reality alert 72B (warning triangle icon) on the image displayed above the first bicycle rider 64A1, and overlay a yellow augmented reality alert 72C (warning triangle icon) on the image displayed above the second bicycle rider 64A2.

[0065] In another exemplary embodiment, referring again to Figure 4 , even if both the first bicyclist 64A1 and the second bicyclist 64A2 have a probability of collision with either the door 62 of the vehicle 10 or the occupant 56 exceeding a predetermined threshold when the occupant 56 leaves the vehicle 10, the displayed augmented reality alerts 72B, 72C may be customized based on the calculated probabilities. For example, the machine learning algorithm 70 calculates a 90% probability that the first bicyclist 64A1 will collide with either the door 62 of the vehicle 10 or the occupant 56 when the occupant 56 leaves the vehicle 10, and calculates a 60% probability that the second bicyclist 64A2 will collide with either the door 62 of the vehicle 10 or the occupant 56 when the occupant 56 leaves the vehicle 10. Based on the relative calculated probabilities, the system controller 34A overlays a red augmented reality alert 72B (warning triangle icon) onto the image displayed above the first bicyclist 64A1, and overlays a yellow augmented reality alert 72C (warning triangle icon) onto the image displayed above the second bicyclist 64A2.

[0066] In another example, reference Figure 6 , the system controller 34A identifies a travel hazard 64B including a pothole 64B1 close to the vehicle 10, which may pose a tripping hazard to the occupant 56 when the occupant 56 leaves the vehicle 10. Due to the proximity of the vehicle 10, the system controller 34A overlays the augmented reality alert 72D (warning triangle icon) onto the image displayed on the pothole 64B1 to draw the occupant 56's attention to the pothole 64B1. The system controller 34A may use an external camera mounted on the side of the vehicle 10 to obtain Figure 6The view shown, or multiple cameras may be used to provide a top view of the vehicle 10, including the identified hazardous objects 64 (including the curb 64B2 adjacent to the vehicle 10) with an augmented reality alert 72E, such as Figure 7 shown.

[0067] The system includes a graphics processor 74 in communication with the system controller 34A. The graphics processor 74 processes information from the system controller 34A and determines the appropriate augmented reality alert graphic 72 to overlay onto the image displayed for the occupant 56.

[0068] In an exemplary embodiment, the system controller 34A is also adapted to determine whether a door 62 of the vehicle 10 has been opened, and if the door 62 has been opened, whether an additional alert 76 is needed. For example, if the system controller 34A displays and images on the display and provides the augmented reality alert 72 to draw the attention of the occupant 56 to the hazardous object 64, but the system controller 34A detects that the occupant 56 continues to open the door 72 despite the augmented reality alert 72, the system controller 34A will verify that the hazardous object 64 is still present and, if so, initiate the additional alert 76.

[0069] The additional alert 76 may include an audible chime, a tactile alert (vibrating seat), or an illuminated alert. Such additional alert 76 may be an alert already present in the vehicle 10 and adapted to provide an alert to the occupant 56, wherein the augmented reality alert 72 provided in the display 52 of the present disclosure is provided prior to such additional alert 76 to give the occupant 56 an opportunity to delay exiting the vehicle 10. The additional alert 76 is adapted to provide a further alert to the occupant 56 to warn that continuing to exit the vehicle 10 may be dangerous.

[0070] refer to Figure 8 , a method 100 for providing a warning to an occupant 56 in a vehicle 10 before the occupant 56 leaves the vehicle 10 includes starting at box 102, using a system controller 34A that communicates with multiple on-board sensors 40a-40n and an occupant monitoring system 50 to detect when an occupant 56 in the vehicle 10 is about to leave the vehicle 10; moving to box 104, using the system controller 34A to detect the position of the occupant 56 in the vehicle 10; moving to box 106, using the system controller 34A to display a view of the external environment of the vehicle 10 on a display screen 52 for the occupant 56 to view; moving to box 108, using the system controller 34A to determine whether there is a dangerous object 64 in the external environment of the vehicle 10; and when a dangerous object 64 is present in the external environment of the vehicle 10, moving to box 110, classifying the dangerous object 64; and moving to box 112, providing an augmented reality alert 72 in the display screen 52.

[0071] In an exemplary embodiment, the method 100 also includes moving to box 114 and determining, using the system controller 34A, whether the door 62 of the vehicle 10 has been opened. When the door 62 has not been opened, then moving to box 108, the method returns to detecting whether the hazardous object 64 is present. When the door 62 has been opened, then moving to box 116, the method 100 includes determining, using the system controller 34A, whether an additional alarm 76 is required. If an additional alarm 76 is required, then moving to box 118, the method 100 includes activating the additional alarm 76 using the system controller 34A, and moving to box 120, terminating the method 100. If an additional alarm is not required at box 116, then moving to box 120, the method 100 includes terminating the method 100.

[0072] In an exemplary embodiment, at box 102, when the system controller 34A in communication with multiple on-board sensors 40a-40n and the occupant monitoring system 50 is used to detect when the occupant 56 in the vehicle 10 is about to leave the vehicle 10, it also includes: starting at box 122, the occupant 56 is riding in the vehicle 10; moving to box 124, using the system controller 34A to determine whether the vehicle 10 is at least one of the following: stopped, parked, and not moving; moving to box 126, using the system controller 34A to determine via the occupant monitoring system 50 whether the occupant 56 is touching the door latch 58 in the vehicle 10; and moving to box 128, using the system controller 34A to actuate the external cameras 40a-40n on the vehicle 10, and when at least one of the following situations occurs, using the external cameras 40a-40n to capture images of the environment outside the vehicle 10: the vehicle 10 is at least one of stopped, parked, and not moving, and the occupant 56 is touching the door latch 58 in the vehicle 10.

[0073] In an exemplary embodiment, displaying a view of the environment outside the vehicle 10 on the display screen 52 for viewing by the occupant 56 using the system controller 34A also includes at least one of: moving to box 130, displaying a view of the environment outside the vehicle 10 on a dedicated display screen 52A mounted to the door 62 of the vehicle 10 using the system controller 34A; and moving to box 132, displaying a view of the environment outside the vehicle 10 on the display screen 52B of the in-vehicle infotainment system using the system controller 34A.

[0074] In an exemplary embodiment, determining whether a hazardous object 64 is present in the environment external to the vehicle 10 using the system controller 34A at box 108 also includes moving to box 134 and detecting at least one moving object 64A in the environment external to the vehicle 10 using the system controller 34A and a plurality of onboard sensors 40a-40n.

[0075] In another exemplary embodiment, detecting at least one moving object 64A in the external environment of the vehicle 10 using the system controller 34A and multiple on-board sensors 40a-40n also includes: moving to box 136, analyzing the speed, position, acceleration, trajectory and recent changes of any one of the speed, position, acceleration, trajectory of the at least one moving object 64A relative to the vehicle 10.

[0076] In another exemplary embodiment, determining whether a hazardous object 64 is present in the environment external to the vehicle 10 using the system controller 34A at box 108 also includes: moving to box 138, using the system controller 34A and multiple on-board sensors 40a-40n to detect at least one travel hazard 64B in the environment external to the vehicle 10.

[0077] In another exemplary embodiment, determining whether a hazardous object 64 exists within the external environment of the vehicle 10 using the system controller 34A at frame 108 further includes moving to frame 140, determining, using the system controller 34A, a probability that at least one moving object 64A will collide with at least one of the door 62 of the vehicle 10 and the occupant 56 during the occupant 56's exit from the vehicle 10, and determining, using the system controller 34A, a probability that at least one travel hazard 64B will cause the occupant 56 to at least one of trip, stumble, and fall during the occupant 56's exit from the vehicle 10.

[0078] In yet another exemplary embodiment, determining, at block 140, using the system controller 34A, a probability that at least one moving object 64A will collide with at least one of the door 62 of the vehicle 10 and the occupant 56 during the occupant 56's exit from the vehicle 10, and determining, using the system controller 34A, a probability that at least one travel hazard 64B will cause the occupant 56 to trip, stumble, and fall, further includes: moving to block 142, using the machine learning algorithm 70 to probabilistically calculate the probability that at least one moving object 64A will collide with at least one of the door 62 of the vehicle 10 and the occupant 56 during the occupant 56's exit from the vehicle 10. a probability of at least one of a door 62 and an occupant 56 colliding, and a probability that at least one travel hazard 64B will cause the occupant 56 to at least one of trip, stumble, and fall during the occupant 56's exit from the vehicle 10; and moving to box 144, when a hazardous object is identified, the method 100 proceeds to classifying the hazardous object 64 at box 110, and when no hazardous object 64 is identified, the method 100 returns to displaying a view of the external environment of the vehicle 10 on the display screen 52 for the occupant 56 to view at box 106 using the system controller 34A.

[0079] The disclosed system and method provide the advantage of providing a warning to a vehicle occupant of the presence of a hazardous object before the occupant exits the vehicle.

[0080] The description of the present disclosure is merely exemplary in nature, and variations that do not depart from the gist of the present disclosure are intended to be within the scope of the present disclosure. These variations should not be regarded as departing from the spirit and scope of the present disclosure.

Claims

1. A warning method for providing a warning to an occupant in a vehicle before the occupant leaves the vehicle, comprising: detecting, via a system controller in communication with a plurality of onboard sensors and an occupant monitoring system, when an occupant of the vehicle is about to exit the vehicle; detecting, by the system controller, a position of the occupant within the vehicle; displaying, by the system controller, a view of an environment external to the vehicle on a display screen adapted to be viewed by the occupant; determining, by the system controller, whether a dangerous object exists in the environment outside the vehicle; When there is a dangerous object in the environment outside the vehicle: classifying the hazardous objects; and An augmented reality alert is provided within the display screen.

2. The method according to claim 1, further comprising: Determining, by the system controller, whether a vehicle door has been opened; When the vehicle door has been opened, determining by the system controller whether an additional alarm is required; as well as If additional alarms are required, additional alarms are activated by the system controller.

3. The method according to claim 2, wherein: Detecting when the occupant in the vehicle is about to leave the vehicle by the system controller communicating with the plurality of onboard sensors and the occupant monitoring system further comprises: determining, by the system controller, whether the vehicle is in at least one of the following states: stopped, parked, and not traveling; determining, by the system controller via the occupant monitoring system, whether the occupant is accessing a door latch within the vehicle; and When at least one of the following situations occurs, an external camera on the vehicle is actuated by the system controller, and an image of the environment outside the vehicle is captured by the external camera: The vehicle is at least one of stopped, parked, and not moving, and The occupant is accessing the door latch within the vehicle.

4. The method according to claim 3, wherein: Displaying, by the system controller, a view of the environment external to the vehicle on a display screen adapted to be viewed by the occupant further comprises at least one of: displaying, by the system controller, the view of the environment outside the vehicle on a dedicated display screen mounted to the vehicle door; as well as The view of the environment outside the vehicle is displayed on a display screen of an in-vehicle infotainment system by the system controller.

5. The method according to claim 4, wherein: Determining by the system controller whether there is a dangerous object in the environment outside the vehicle further includes detecting by the system controller and the plurality of onboard sensors at least one moving object in the environment outside the vehicle.

6. The method according to claim 5, wherein: Detecting at least one moving object in the environment outside the vehicle through the system controller and the multiple vehicle-mounted sensors also includes: analyzing the speed, position, acceleration, trajectory and recent changes of any one of the speed, position, acceleration and trajectory of the at least one moving object relative to the vehicle.

7. The method according to claim 6, wherein: Determining, by the system controller, whether there is a dangerous object in the environment outside the vehicle further includes: detecting, by the system controller and the plurality of onboard sensors, at least one traveling hazard in the environment outside the vehicle.

8. The method according to claim 7, wherein: Determining by the system controller whether there is a dangerous object in the environment outside the vehicle further includes: determining, by the system controller, a probability that the at least one moving object will collide with at least one of the door and the occupant during the occupant's exit from the vehicle; and A probability is determined, by the system controller, that the at least one travel hazard will cause the occupant to at least one of trip, stumble, and fall during the occupant's exit from the vehicle.

9. The method according to claim 8, wherein: Determining, by the system controller, a probability that the at least one moving object will collide with at least one of the vehicle door and the occupant during the occupant's exit from the vehicle, and determining, by the system controller, a probability that the at least one travel hazard will cause the occupant to trip, stumble, and fall, also includes: probabilistically calculating, by a machine learning algorithm, a probability that the at least one moving object will collide with at least one of the vehicle door and the occupant during the occupant's exit from the vehicle, and a probability that the at least one travel hazard will cause the occupant to trip, stumble, and fall.

10. A warning system for providing a warning to an occupant of a vehicle before the occupant leaves the vehicle, comprising: a plurality of onboard sensors adapted to collect information related to an environment external to the vehicle; an occupant monitoring system adapted to monitor and detect movement of said occupant within said vehicle; a display screen adapted to be viewed by said occupants in said vehicle; as well as a system controller in communication with the plurality of onboard sensors, the occupant monitoring system, and the display screen, the system controller being adapted to: detecting when the occupant in the vehicle is about to leave the vehicle; detecting a position of the occupant in the vehicle; displaying on the display screen a view of the vehicle's external environment collected by at least one of the plurality of sensors; Determining whether there is a dangerous object in the environment outside the vehicle; as well as When there are dangerous objects in the environment outside the vehicle: classifying the dangerous objects; as well as An augmented reality alert is provided within the display screen.

Images