Vehicle emergency notification

Abstract

The invention relates to vehicle emergency notification. A computerized method for emergency notification of a vehicle is proposed. The vehicle includes external sensors and internal sensors, and the method includes predicting a likelihood of an accident of the vehicle based on vehicle data and data received from one or more of the external sensors and / or one or more of the internal sensors, and transmitting a pre-accident signal to the relay station in response to the likelihood of the occurrence of the accident being higher than an accident threshold, the pre-accident signal includes vehicle and passenger status information generated based on the pre-accident vehicle data, data received from one or more of the external sensors before an accident, and data received from one or more of the internal sensors before an accident.

Description

Technical Field

[0001] The present disclosure generally relates to safety improvements for vehicles, and more particularly to methods and systems for emergency notification of vehicles. Background Art

[0002] Intelligent vehicles such as smart cars, smart buses, etc. are on the path to significantly improving passenger safety. Such intelligent vehicles can be equipped with on-vehicle sensors capable of capturing the conditions inside and outside the vehicle. The outputs of these sensors can be used for different safety-related tasks.

[0003] Intelligent vehicles are now equipped with emergency call systems. These systems are designed to automatically initiate an emergency call to an emergency call center in the event of a traffic accident, providing critical vehicle information such as the accident time, location coordinates, travel direction, and vehicle characteristics. In addition to automatic activation, manual sending of the emergency call is also possible. Some systems also provide an optional voice connection between the emergency call center and the vehicle occupants.

[0004] Although some emergency call systems already include the vital signs of vehicle occupants in the emergency call, use collision patterns and occupant status to determine rescue, and improve the emergency response time by automatically sending location and sensor information to the emergency agency, these emergency call systems require the installation and / or use of additional sensors that are not typically provided in vehicles (such as biometric or infrared sensors for determining vital signs), can only determine the collision pattern in a very rough manner, and only send an emergency call after an accident has occurred.

[0005] Therefore, there is a need to provide improved methods and systems for emergency notification of vehicles. Summary of the Invention

[0006] More specifically, a computerized method for emergency notification of a vehicle is proposed. The vehicle includes external sensors and internal sensors, and the method includes predicting the likelihood of an accident occurring in the vehicle based on vehicle data and data received from one or more of the external sensors; and in response to the likelihood of the accident occurring being higher than an accident threshold, sending a pre-accident signal to a relay station, wherein the pre-accident signal includes vehicle and passenger status information generated based on the pre-accident vehicle data, data received from one or more of the external sensors before the accident, and data received from one or more of the pre-accident internal sensors before the accident.

[0007] In some embodiments, the method further comprises the steps of: in response to an accident having occurred, determining a severity level of the accident based on vehicle data, data received from one or more external sensors, and data received from one or more internal sensors; and in response to the severity level of the accident being higher than a severity threshold, sending a post-accident signal to a relay station, wherein the post-accident signal indicates confirmation of the accident and includes vehicle and passenger status information generated based on post-accident vehicle data, data received from one or more external sensors after the accident, and data received from one or more internal sensors after the accident.

[0008] In further embodiments, the method comprises the steps of: in response to the severity level of the accident being lower than the severity threshold, sending a no-severe-accident signal to the relay station to indicate that no severe accident has occurred, and deleting the pre-accident signal received at the relay station. In yet another embodiment, the no-severity-accident signal is manually issued by a passenger of the vehicle via a user interface on a display of the vehicle, or is automatically issued by an in-vehicle computing system of the vehicle in response to determining that the severity level is lower than the severity threshold.

[0009] In some embodiments, the method further comprises the steps of: in response to no accident having occurred, sending a no-accident signal to the relay station to indicate that no accident has occurred, and triggering deletion of the pre-accident signal received at the relay station. In further embodiments, the no-accident signal is manually issued by a passenger of the vehicle via a user interface on a display of the vehicle, or is automatically issued by an in-vehicle computing system of the vehicle in response to detecting that no accident has occurred based on data from at least one of an external sensor and an internal sensor.

[0010] In some embodiments, the vehicle data includes at least one of a speed, an acceleration, a steering direction, a tire orientation, and a geographical location of the vehicle. In some further embodiments, the external sensors include at least one of a radar sensor, a camera, and a lidar sensor that captures the environment around the vehicle at a specified time interval.

[0011] In some embodiments, the internal sensors include at least one cabin camera that captures the cabin of the vehicle at a specified time interval. In further embodiments, the pre-accident and post-accident vehicle and passenger status information is determined based on images captured by the cabin camera, wherein the determination is based on at least one of semantic segmentation of the image, depth information analysis of the image, three-dimensional pose estimation determination, and vital sign monitoring.

[0012] In some embodiments, the pre-accident vehicle and passenger status information includes at least one of the following: an estimated severity level of the accident; vehicle location; an estimated number of vehicles involved; an estimated type of the vehicles involved; object motion prediction; the number of passengers in the vehicle; the vital signs of the passengers in the vehicle; the location of the passengers in the vehicle; and the seat belt status of the passengers in the vehicle.

[0013] In some embodiments, the post-accident vehicle and passenger status information includes at least one of the following: the severity level of the accident; vehicle location; the number of vehicles involved; the type of the vehicles involved; vehicle deformation information; the accident location at the vehicle; object motion information; environmental hazard information; the number of passengers in the vehicle; the vital signs of the passengers in the vehicle; the location of the passengers in the vehicle; the injury severity information of the passengers in the vehicle; and the seat belt status of the passengers in the vehicle.

[0014] On the other hand, there is provided an emergency notification system including one or more internal sensors, one or more external sensors, and a computing and communication system configured to perform the methods as described herein.

[0015] On the other hand, there is provided a vehicle including the emergency notification system as described herein.

[0016] Finally, there is provided a computer program product containing instructions which, when executed on a computer, cause the computer to perform the methods as described herein.

[0017] These and other objects, embodiments, and advantages will become apparent to those skilled in the art from the following detailed description of the embodiments with reference to the accompanying drawings. The present disclosure is not limited to any particular embodiment. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The foregoing and further objects, features, and advantages of the subject matter will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings, in which like reference numerals are used to represent like elements, and in which:

[0019] Figure 1 is a high-level flowchart of the method for emergency notification described herein.

[0020] Figure 2 shows a flowchart of the method in the case of a severe accident according to an embodiment.

[0021] Figure 3 shows a flowchart of the method in the case of a non-severe accident according to an embodiment.

[0022] Figure 4Shows a flowchart of a method in the case of no accident according to an embodiment.

[0023] Figure 5A , Figure 5B , Figure 6A and Figure 6B Shows input images from an in-vehicle camera for different classifications and processing of vehicle and passenger status information according to an embodiment.

[0024] Figure 7 Is an overview of the process of emergency event notification according to an embodiment.

[0025] Figure 8 Shows a vehicle with external and internal sensors and a system for providing the functions described herein.

[0026] Figure 9 Shows the interior of a vehicle with possible positions of an in-vehicle camera.

[0027] Figure 10 Is an illustration of a computing system for implementing the functions described herein. Detailed Description

[0028] The present disclosure relates to methods and systems for emergency notification for vehicles.

[0029] Safety-related assistance systems (such as emergency notification systems, also known as eCall systems) are becoming increasingly relevant in modern vehicles (such as cars, buses, motorcycles, etc.), and vehicles may even be legally required to be equipped with such systems. It should be noted that hereinafter, the symbols "vehicle" and "car" should be understood interchangeably and refer to any vehicle in which an emergency notification system as described herein can be applied.

[0030] Emergency notification systems can significantly reduce emergency response time and save the lives of vehicle passengers. Therefore, it is beneficial if the emergency notification is reliable (i.e., provided in any case, including overall information about the vehicle and its passengers, and especially sent to the emergency call center as quickly as possible). An emergency notification system is described below, in which pre-accident signals enable a faster emergency response, and the overall data of the vehicle enables a more customized emergency response.

[0031] Figure 1is a high - level flowchart of a method for emergency notifications sent by a vehicle. The vehicle includes one or more external sensors and one or more internal sensors. In some embodiments, the internal sensors can include at least one cabin camera that captures images of the vehicle cabin at specified time intervals. Other internal sensors can include, for example, cabin radar sensors, cabin infrared sensors, etc. The cabin or in - vehicle cameras can also take different forms, such as described later with reference to Figure 9 described. The external sensors can include at least one of a radar sensor, a camera, and a lidar sensor that captures the vehicle's surrounding environment at specified time intervals. Other external sensors can be ultrasonic sensors, etc.

[0032] The method starts at block 11, where the likelihood of a vehicle accident (hereinafter also referred to as accident likelihood) is predicted based on vehicle data and data received from one or more external sensors. In some embodiments, determining the likelihood of an accident can additionally or alternatively be based on data received from one or more internal sensors (not shown). In some embodiments, the vehicle data can include at least one of the vehicle's speed, acceleration, steering direction, tire orientation, and geographical location. The geographical location of the vehicle can be determined based on the Global Positioning System (GPS) and can be represented by latitude and longitude values. The vehicle data can be provided to the emergency notification system described herein by a high - level driver assistance system described later with reference to Figure 8 described, which has collected the vehicle data. Thus, the vehicle data is general data related to the current state of the vehicle and can therefore be available at any time at the vehicle control system for other purposes, such as output to the driver via a user interface. Additionally, the data received from one or more external sensors for predicting the accident likelihood can include distance, speed, acceleration, and / or size information of other objects around the vehicle. Using an external vision algorithm that analyzes data from external sensors such as radar, camera, and lidar, it is possible to accurately predict whether an accident will occur and is inevitable.

[0033] In some embodiments, data from internal sensors can also be used to predict the likelihood of an accident. Additionally or alternatively, other vehicle systems involved in predicting the likelihood of an accident. For example, Adaptive Cruise Control (ACC) (also known as Adaptive Cruise Control) can support the emergency notification system described herein. ACC is a feature of a high - level driver assistance system (ADAS) typically found in modern vehicles and enhances the traditional cruise control system by combining sensors and technology for automatically adjusting the vehicle speed to maintain a safe following distance from the vehicle ahead. Thus, ACC has been periodically monitoring the vehicle's surrounding environment and predicting the movement of other objects (especially other vehicles).

[0034] Similar to traditional cruise control, ACC allows the driver to set a desired cruise speed for the vehicle, typically via the vehicle's dashboard controls. Thus, ACC also captures and controls the vehicle's current speed. ACC also receives inputs from external sensors as described above, which continuously monitor the road ahead and detect other vehicles in the same lane. These sensors measure, for example, the distance between the vehicle and the vehicle ahead. If the ACC system detects that the distance to the vehicle ahead is decreasing and the vehicle is approaching the vehicle ahead too quickly, the ACC system will automatically reduce the vehicle's speed by decelerating or applying the brakes. The system will do so until a safe following distance is re-established.

[0035] ACC allows the driver to set a preferred following distance, which can be a selected predefined distance setting, such as short, medium, or long, depending on the selected comfort level. Then, even if the vehicle ahead changes speed, ACC maintains this distance from the other vehicle. Thus, ACC can also control the vehicle's speed. For example, if the road ahead is clear or if the vehicle ahead accelerates, the ACC system will gradually increase the vehicle's speed up to the set cruise control speed, as long as it is safe to do so. Thus, ACC can provide a great deal of information to the emergency notification system.

[0036] The likelihood of a predicted accident can be based on a basic model, for example, by only determining the distance to an object or vehicle ahead and the speed of the object or vehicle ahead (e.g., obtained from ACC or from an external sensor and determined within the emergency notification system), and determining the vehicle's speed (i.e., speed and direction). The likelihood of an accident can be determined, for example, by how many different scenarios an accident would occur in. For example, consider three scenarios, namely, what might happen without braking, with the brakes pressed halfway, and with full braking. If the model determines that an accident will inevitably occur in the last two scenarios, the likelihood of an accident can be determined to be 66.6%.

[0037] In some embodiments, the likelihood of an accident can be determined by a more complex model based on trajectory prediction. Trajectory prediction is a procedure used in various fields including robotics, autonomous vehicles, and computer vision to predict the future path or motion of an object or entity based on its current state and past behavior. This involves making an educated guess about how an object will move in the future in order to make better decisions in applications such as autonomous driving navigation, collision avoidance, and tracking. However, trajectory prediction can also be used to predict accidents, as in the emergency notification system described herein.

[0038] Trajectory prediction typically starts with collecting data about the object of interest. Depending on the application, this data can come from various sensors in the vehicle, such as cameras, lidar, radar, or GPS. The data can include information about the vehicle's current position, speed, and acceleration, as well as historical data about its past motion. Then, the first step is to estimate the current state of the vehicle. This includes the position and speed of the vehicle at the current time. This estimation can be based on sensor measurements and can involve sensor fusion techniques for combining information from multiple sensors to obtain greater accuracy.

[0039] To predict the future trajectory, a motion model is used. A motion model is a mathematical representation of how the object of interest (i.e., the vehicle) is expected to move. The choice of motion model depends on the type of vehicle and the specific application. Common models include constant speed, constant acceleration, or more complex models for objects with specific behaviors. Using the estimated state and the motion model, a prediction algorithm is used to calculate the future path of the vehicle. The algorithm extrapolates the vehicle trajectory into the future based on the motion model. The length of the prediction horizon (i.e., how far into the future the prediction extends) is typically determined by the specific application and the required decision time frame.

[0040] Thus, in some embodiments, the trajectory prediction algorithm provides one or more predicted paths for the vehicle, but can also predict the paths of other vehicles or pedestrians, etc. The likelihood of an accident then depends on the trajectories (including speeds) of the vehicle and other objects. This likelihood can take any value between 0 and 1 or between 0% and 100%, and reflects how certain the model is about the accident. Additionally, the likelihood of an accident can also depend on the time of a possible collision. This time can be a factor in the prediction, but can also be preset. For example, the likelihood of an accident can reflect whether an accident is likely to occur within 3 seconds (or any other appropriate value). Note that the likelihood of an accident can be determined continuously, for example, at a predetermined time interval, such as every second. Other appropriate time intervals are also applicable.

[0041] The method then continues to determine whether the likelihood of an accident is higher than an accident threshold, as shown by diamond 12. For example, if the likelihood of an accident occurring within the next 3 seconds is determined to be 85% and the threshold is set at 75%, it is determined that an accident is likely to occur, which results in sending a pre-accident signal to a relay station in block 13. The relay station can be, for example, an emergency call center, a first responder line, etc. The pre-accident signal includes vehicle and passenger status information generated based on the following data: vehicle data at the time when it is determined that an accident is likely to occur (i.e., before the accident occurs), data received from one or more external sensors before the accident occurs, and data received from one or more internal sensors before the accident occurs. Thus, at the moment when the emergency notification system described herein determines that an accident is likely to occur, a pre-accident signal is sent to the relay station.

[0042] The pre-accident vehicle and passenger status information can be determined based on images captured by a cabin camera, wherein the determination is based on at least one of semantic segmentation of the image, analysis of depth information of the image, determination of three-dimensional pose estimation, and vital sign monitoring. The pre-accident vehicle and passenger status information can include at least one of the following: an estimated accident severity level; a vehicle location; an estimated number of vehicles involved; an estimated type of vehicles involved; object motion prediction; the number of passengers in the vehicle; the vital signs of passengers in the vehicle; the location of passengers in the vehicle; and the seat belt status of passengers in the vehicle. The pre-accident signal can also be labeled as a pre-accident signal such that the relay station can distinguish the pre-accident signal from other signals. An exemplary hypertext markup language (HTML) structure of such a pre-accident signal is shown in Table 1 below.

[0043] Table 1

[0044]

[0045] As can be seen from Table 1, the pre-accident signal in this example includes three main components, namely the MSD (Minimum Data Set, from row 2 to row 15), additional data (from row 15 to row 22), and data extension (from row 23 to row 29). It is obvious to those skilled in the art that in some embodiments, the additional data and data extension can be omitted from the pre-accident message. The MSD is similar to what has been defined in the current European eCall standard, which defines the message content of the accident (post) signal. It should be noted that the example pre-accident signal in Table 1 can be adapted to include more or fewer fields as needed. In some of the described embodiments, the pre-accident signal can particularly include the estimated collision severity (as shown in row 9 of Table 1) and the number of occupants (as shown in row 10 of Table 1). Additionally, the position of the occupants is indicated (as shown in row 11 of Table 1), the estimated injury severity (as shown in row 12 of Table 1), and the identification number of the pre-accident signal (as shown in row 13 of Table 1). As described below, the identification number can be used to quickly determine which pre-accident signal is also related to which post-accident signal, no-accident, and / or no-severe-accident signal.

[0046] As already pointed out in the example of the pre-accident signal in Table 1, it is possible to estimate the likelihood of the accident severity before the accident occurs. This determination can be based on the same information used to determine the likelihood of the accident, but additionally based on the information provided by internal sensors (such as on-board cameras). For example, it can be predicted where the accident will occur (e.g., the right rear door) and whether a person is sitting in the correct position. Additionally or alternatively, in order to predict the severity of the accident, seat belt status, passenger age, objects in the vehicle, etc. can be considered.

[0047] The pre-accident signal can generally notify the relay station to dispatch emergency services to ensure timely assistance. As described above, this pre-accident signal can be similar to the existing eCall system, but with additional information. It can include the accident location obtained from the internal camera using 2D and 3D body key points (explained later), the number of cars involved, the vital signs of all passengers, and the positions of all passengers inside the vehicle. Additionally, the system described herein can utilize advanced internal sensing features to double-check how many passengers are present in the vehicle and check whether the passengers are wearing seat belts correctly. For example, a vision algorithm can detect whether a passenger is using their seat belt incorrectly, such as under the arm instead of over it, which may increase the risk of injury in an accident.

[0048] This additional information can generate important insights for Emergency Medical Technicians (EMTs) to better assess the situation and provide appropriate care to passengers. Traditional seat-based occupant detection systems and seat belt buckle sensors cannot provide this information with absolute precision, as heavy objects on the seat may be mistaken for a person, and the misuse of seat belts cannot be detected at all. Using cabin awareness sensors and algorithms, the potential risks of (unsafe) cargo on the vehicle can be classified, and its movement can be predicted before a potential collision may occur. Using this information, the additional dangerous impact on the health and safety of passengers due to the behavior of an object in the event of a sudden deceleration (becoming a projectile) can be estimated and taken into account in the damage estimation in the pre-crash signal.

[0049] Now turning to Figure 2 , a flowchart of a method if a severe accident occurs according to an embodiment is shown. This can be determined by external sensors and / or internal sensors of the vehicle. For example, if the wheel sensors of the vehicle determine a high deceleration and / or an in-vehicle camera determines the deformation of the vehicle cabin, an accident can be determined, as shown by diamond 21. Then, the method continues to determine the severity level of the accident based on vehicle data, data received from one or more external sensors, and data received from one or more internal sensors, as shown in block 22. This determination can also be based on the deformation of the vehicle cabin, and also on additional information from internal sensors of the vehicle before and after the accident. For example, if a passenger is detected on the passenger seat and then a large deformation is detected at that position after the accident, a high severity can be determined. Other examples can be based on the comparison of the passenger postures before and after the accident.

[0050] The method then compares the determined severity with a severity threshold, which can also take a value between 0 and 1, where 0 is no severity and 1 is the highest severity, as shown by diamond 23. If it is determined that the severity is higher than the severity threshold (branch "yes"), the method proceeds to block 24 and sends a post-crash signal to a relay station. The post-crash signal indicates the confirmation of the accident and includes vehicle and passenger status information generated based on post-crash vehicle data, data received from one or more external sensors after the accident, and data received from one or more internal sensors after the accident.

[0051] The post-accident vehicle and passenger status information can be determined based on images captured by in-cabin cameras, wherein the determination is based on at least one of semantic segmentation of the images, analysis of depth information of the images, determination of three-dimensional pose estimation, and vital sign monitoring. The post-accident vehicle and passenger status information can include at least one of the following: the severity level of the accident; the vehicle location; the number of vehicles involved; the type of vehicles involved; vehicle deformation information; the accident location at the vehicle; object motion information; environmental hazard information; the number of passengers in the vehicle; the vital signs of the passengers in the vehicle; the location of the passengers in the vehicle; the injury severity information of the passengers in the vehicle; and the seat belt status of the passengers in the vehicle. The post-accident signal can also be marked as a post-accident signal so that the relay station can distinguish the post-accident signal from other signals (such as the pre-accident signal described previously regarding Figure 1 the pre-accident signal). An exemplary HyperText Markup Language (HTML) structure of such a post-accident signal is shown in Table 2 below.

[0052] Table 2

[0053]

[0054] As can be seen from Table 2, the pre-accident signal in this example also includes three main components, namely MSD (Minimum Data Set, from line 2 to line 19), additional data (from line 20 to line 26), and data extension (from line 27 to line 33). It is obvious to those skilled in the art that in some embodiments, the additional data and data extension can be omitted from the post-accident message. The MSD is similar to what has been defined in the current European eCall standard, which defines the message content of the accident (post) signal. Note that the example post-accident signal in Table 2 can be adapted to include more or fewer fields as needed. In some of the described embodiments, the post-accident signal can particularly include the accident severity (shown in line 9 of Table 2) and the number of occupants (shown in line 10 of Table 2). Additionally, the location of the occupants (shown in line 11 of Table 2), the injury severity (shown in line 12 of Table 2), and the identification number of the post-accident signal (shown in line 13 of Table 2) are indicated. This identification number can be used to quickly determine which post-accident signal is also related to which pre-accident signal, as described previously.

[0055] In addition, the post-accident signal can also indicate the vital signs of each passenger (as shown in line 14 of Table 2), the degree of cabin deformation (as shown in line 15 of Table 2), environmental hazards (if any) (as shown in line 16 of Table 2), and the location of the accident (as shown in line 17 of Table 2). This additional information can be determined based on the internal sensors of the vehicle, where the determination is based on at least one of semantic segmentation of images, analysis of depth information of images, determination of three-dimensional pose estimation, and vital sign monitoring. In addition, object detection and classification algorithms can be used, for example, to determine environmental hazards. In this or other embodiments, vital sign monitoring can be performed by connecting the passengers' smartwatches to the vehicle's on-board system. Additionally or alternatively, images captured by on-board cameras can be used to determine vital signs, for example, by generating a remote photoplethysmogram (RPPG) from the images by detecting facial expressions, skin color changes, and / or the movement of the passengers.

[0056] It will be apparent to those skilled in the art that the post-accident signal is sent only if the vehicle's electronic cabin is still functional after the accident. This also means that if the relay station does not receive the post-accident signal (and there is no other signal indicating that no accident or no serious accident has occurred), the relay station is still aware of the accident due to the pre-accident signal.

[0057] The post-accident signal (i.e., the confirmation signal) can also include the vital signs of all occupants after the accident, the position of each occupant through 3D body key points and the resulting medical conditions, information about any passengers who may be locked due to vehicle deformation, the movement of objects in the cabin, critical condition estimation, and external analysis such as the conditions of other vehicles involved and the presence of fire / smoke.

[0058] Figure 3 A flowchart of a method in the event of a non-serious accident according to an embodiment is also depicted. In this example, the severity level is determined (as shown in diamond 23) but is not higher than the severity threshold (i.e., the branch "no" is selected). Then, the method proceeds to block 34 (instead of block 24 as shown) and sends a no-serious-accident signal to the relay station to indicate that no serious accident has occurred and triggers the deletion of the pre-accident signal received at the relay station. This "delete previous message" ping (i.e., the no-serious-accident signal) is sent to prevent any personal information from being sent to the emergency services. This ensures the privacy and safety of the vehicle occupants because the initial signal is deleted and no unnecessary emergency response is triggered. As previously mentioned, the relay station waits for a predetermined time after receiving the pre-accident signal for another message that confirms the accident or cancels the indication / warning of the pre-accident signal. If such a message is not received within the time interval, an emergency response is triggered to ensure a quick response. Figure 2 shown in diamond 23) but not higher than the severity threshold (i.e., the branch "no" is selected). Then, the method proceeds to block 34 (instead of block 24 as Figure 2 shown) and sends a no-serious-accident signal to the relay station to indicate that no serious accident has occurred and triggers the deletion of the pre-accident signal received at the relay station. Sending this "delete previous message" ping (i.e., the no-serious-accident signal) prevents any personal information from being sent to the emergency services. This ensures the privacy and safety of the vehicle occupants because the initial signal is deleted and no unnecessary emergency response is triggered. As previously mentioned, the relay station waits for a predetermined time after receiving the pre-accident signal for another message that confirms the accident or cancels the indication / warning of the pre-accident signal. If such a message is not received within the time interval, an emergency response is triggered to ensure a quick response.

[0059] A no-severe-accident signal may need to be manually issued by a passenger of the vehicle through a user interface on a vehicle display to confirm that the accident is not severe. Alternatively, a no-severity-accident signal may be automatically issued by an in-vehicle computing system of the vehicle in response to determining that the severity level is below a severity threshold. In some additional embodiments, presenting the user interface to the passenger may be triggered by the in-vehicle computing system determining that the severity level is below the severity threshold.

[0060] In some additional embodiments, the emergency notification system includes an extension for handling minor accidents, i.e., having a low severity. The system may determine whether the accident is a low-speed impact and determine whether the driver has declined emergency service assistance (as a strict requirement), as previously described. In such a case, the system may record the damage inside and outside the vehicle and generate an accident report ready for an insurance claim. The report includes insurance information, vehicle information, and internal sensing identified for the driver, providing a comprehensive overview of the accident for insurance purposes. In accordance with possible data protection regulations, the system will collect this information in many cases (e.g., in a parking lot, at a traffic light, etc.) from nearby vehicles in advance. If the driver later manually reports damage that the system itself failed to register, this information may already be available. An exemplary hypertext markup language (HTML) structure for such a report is shown in Table 3 below.

[0061] Table 3

[0062]

[0063] As shown in Table 3, the report in this example is structured similarly to the pre-accident signal and the post-accident signal. However, the report clearly indicates that no emergency call was sent (as shown in row 12 of Table 3) and information about other vehicles (e.g., as shown in rows 14 and 20). In any embodiment, additional information may also be included in the report.

[0064] Figure 2 and Figure 3 relates to the situation where an accident has occurred. However, the predicted collision may not have occurred, for example, because both drivers were able to perform full braking. Therefore, Figure 4 shows a flowchart of the method if no accident has occurred according to these embodiments. Therefore, it is determined whether an accident has occurred (as shown in Figure 2 and diamond 21). If no accident has occurred, the method proceeds to block 42 (instead of block 22 in Figure 2 ), and sends a no-accident signal to the relay station to indicate that no accident has occurred and triggers the deletion of the pre-accident signal received at the relay station.

[0065] The no - accident signal and the no - serious - accident signal can be similar or identical. This is because both the no - accident signal and the no - serious - accident signal indicate to the relay station that the pre - accident signal will be deleted. The possible HyperText Markup Language (HTML) structure of such no - accident or no - serious - accident signals is shown in Table 4 below.

[0066] Table 4

[0067]

[0068] Generally speaking, the automatic emergency call system described in this article stands out because it combines external perception algorithms and internal perception algorithms to provide an overall view of pre - accident and post - accident situations. In some embodiments, the overall view of the situation includes the vital signs, location, and medical condition of the occupants, vehicle deformation, object movement, and external analysis, thus allowing emergency responders to make informed decisions and respond effectively to the situation. Additionally, the emergency notification system can also incorporate privacy measures by deleting unnecessary information in the absence of an accident and includes features for handling minor accidents and generating accident reports for insurance claims.

[0069] Figure 5A , Figure 5B , Figure 6A and Figure 6B show input images from an in - vehicle camera for different classifications and processing to determine vehicle and passenger status information according to an embodiment. This processed input image and the information obtained from it can also be used to determine whether an accident has occurred, the severity of the accident, and other data required for any of the processes described above.

[0070] Figure 5A shows an excerpt of an image from an in - vehicle camera to which semantic segmentation has been applied. Semantic image segmentation is a computer vision task that involves classifying each pixel in an image into a specific category or class. This means that semantic segmentation assigns a label to each pixel to indicate what object or region of the image the pixel belongs to. The purpose of this image processing is to segment the image into meaningful segments, where each segment corresponds to a specific object or region with semantic meaning. In the emergency notification system described in this article, semantic segmentation can be applied to determine relevant regions and exclude irrelevant regions from further processing. Thus, semantic segmentation improves the efficiency of determining the cabin state, such as the location of passengers, the location of objects present, etc.

[0071] In Figure 5AAmong them, the light pixels at the driver's head 50 are classified as "head", which is the relevant area for determining the severity of the injury. The light pixels at the stuffed animal toy 51 are classified as "unknown object", which may or may not be relevant to the severity determination. Therefore, in some embodiments, an "unknown object" can generally be regarded as resulting in a high severity, although in this example the stuffed animal toy 51 may not cause any injury. The pixels at the seat belt 52 are classified as "seat belt", which can also be used to determine the severity of the injury. For example, in this case, it can be determined that the area of the seat belt 52 is not optimally positioned, that is, under the passenger's left arm rather than on the shoulder. Therefore, if this is the last image before the accident, the estimated severity may be high due to the orientation of the seat belt.

[0072] Figure 5B Two depth images are depicted. A depth image, also known as a depth map or a depth perception image, is a two-dimensional representation of a three-dimensional scene, where each pixel in the image contains information about the distance or depth of the corresponding point in the real world. Depth images are commonly used in computer vision, robotics, and 3D modeling to capture and understand the spatial characteristics of a scene.

[0073] Figure 5B The upper image shows the situation in the vehicle before the accident. Three people are detected, namely, the driver 53, the co-driver 54, and the passenger 55 at the left rear seat. In the lower image, the cabin is deformed. Therefore, the deformation or the degree of deformation of the cabin can be determined by using the depth image. Here, a large bump 56 in the roof of the vehicle is depicted. If the depth (e.g., the distance from the bump 56 to the camera after the accident is similar to the depth of the head of the passenger 55 before the accident), a high severity of the accident can be determined.

[0074] Figure 6A The extraction of the input image provided by the in-vehicle camera (i.e., the internal sensor according to the present disclosure) is depicted. Two people can be detected, one at the driver's seat and one at the rear row. A body key point detector based on machine learning algorithms such as neural networks, random forests, support vector machines, boosted cascade algorithms, etc. has been applied to the image, and the body key points of the passenger are determined, for example, in the following example of the driver. The body key point detector has detected, for example, the shoulder key points of the right shoulder 60 and the left shoulder 61, the hip key points of the right hip 62 and the left hip 63, the center shoulder key point 64, the center hip key point 65, the elbow key point 66, and the wrist key point 67. The body key point detector can also identify Figure 6A and Figure 6BOther key points not shown, such as the knees, ankles, etc. For some passengers, not all key points can be detected, for example because the arm is behind another person or the seat. To implement a body key point detector, any image classifier capable of determining and classifying regions on a human body can be used.

[0075] It should be noted that the body key points of any person on one or more images captured by one or more in-vehicle cameras can be determined (for example, the body key points of a person sitting in the passenger seat or the rear seat can also be determined). The images captured by the in-vehicle cameras can also be subdivided into multiple regions (such as the seat region) to determine the corresponding body key points of each seat. In some embodiments, the images of multiple cameras can also be combined to determine a 3D image of the interior, where the body key points are determined. Based on the body key points, the pre-accident and post-accident postures of the passengers can be determined, so that the severity of the accident, injuries, vital signs of the passengers, etc. can be determined. Therefore, the skeleton of the body key points can provide additional information about the state of each passenger regarding vital signs, consciousness, and positioning or deformation before and after the accident.

[0076] Figure 6B An image with a determined object is shown. For example, an object detector based on a machine learning classifier such as a neural network, random forest, support vector machine, boosted cascade algorithm, etc. can detect objects, people, and / or seat regions in the images from the in-vehicle cameras and return a bounding box around the detected object and a classification of the object type. For example, as Figure 6B shown, the bounding box 68 around the child seat is determined and depicted. The respective corner pixel values (for example, the position of the pixel) can be associated with an object of the type 'child seat'. In addition, using the bounding box 69, the seat region of the left rear seat (in the driving direction) is determined and marked.

[0077] Based on machine learning-based object detection algorithms, a large number of different objects can be learned. Therefore, objects can be flexibly added to the training data. Such objects can also be inanimate objects, such as child seats, mobile phones, tablets, laptops, food, cups, books, and newspapers, or can be living objects or parts of living objects, such as animals, people, human hands, human heads, etc. The objects in the vehicle can be identified before an accident to provide the possibility of damage caused by unsecured loads in the vehicle. The presence and absence of passengers can be automatically obtained from a single monocular camera installed, for example, at the rearview mirror, with the help of object detection algorithms. With the help of object detection and / or body key points, the identification and classification of driver behavior can be added to the automatic insurance report as described above. For example, if the driver is distracted by a mobile phone, etc., it can be added.

[0078] Figure 7This is an overview of the process for emergency event notification according to an embodiment. On the left hand side, data sources that can be used by the emergency system described herein are depicted. Internal sensor data 70 (such as data from in-vehicle cameras and other internal sensors), external sensor data 71 (such as data from radar sensors, cameras, and lidar sensors that capture the vehicle's surrounding environment at specified time intervals), and vehicle data 72 (such as vehicle data from driver assistance systems, in-vehicle computing systems, etc.) can be fused to detect whether an accident is likely to occur.

[0079] For example, by using all three forms of information, it can be determined that the driver's distraction may lead to a situation where an accident is very likely to occur or a situation where the driver cannot respond to such situations. Suppose the vehicle is driving straight on a rural road. An oncoming car suddenly catches up and overtakes a truck, putting the oncoming car in the same lane as the vehicle. The external sensors detect this scenario, the vehicle data provides the vehicle's speed, and the internal sensors show that the driver of the vehicle is currently distracted by making a phone call. Even though the vehicle's emergency braking and audible signals for the driver can help reduce the severity of the accident, the system can estimate with relatively high accuracy that the accident cannot be completely avoided at this time.

[0080] If it is determined that the accident cannot be avoided, then a pre-accident signal 73 (as described above) is sent to a relay station, which includes vehicle and passenger status information generated based on pre-accident vehicle data 72, data received from one or more external sensors 71 before the accident, and data received from one or more internal sensors 70 before the accident. If no collision occurs (e.g., the in-vehicle computing system determines that there is no deceleration), in this example, a user interface 74 is provided for the vehicle's passengers to confirm that no collision has occurred. If the passengers confirm this, a no-accident signal 75 is sent to the relay station.

[0081] If an accident has occurred (e.g., as determined by the in-vehicle computing system that deceleration has occurred), then the severity is determined and checked, whether the accident is serious (shown in diamond box 76). In some embodiments, the severity is determined by considering data received from one or more internal sensors 70 and / or data received from one or more external sensors 71. For example, if the external sensors 71 detect that, for example, this vehicle or another vehicle is on fire, people around the vehicle are injured, etc., the severity can be determined to be high (or relatively high). Alternatively or additionally, if the internal sensors 70 detect that, for example, a passenger is in an unnatural position, has trauma, etc., the severity can be determined to be high (or relatively high).

[0082] If it is determined that the severity is high (as described above for other embodiments), i.e., the accident is severe, then an after-accident signal 77 (as described previously) is sent to the relay station. If it is determined that the severity is low (as described above for other embodiments), i.e., the accident is not severe, then in this example, the vehicle's passengers are provided with a user interface 79 to confirm that no severe collision has occurred. If the passengers confirm this, then a no-accident signal 75 is sent to the relay station. In this embodiment, the no-accident signal 75 also serves as a no-severe-accident signal. Additionally, although user interfaces 74 and 79 are shown as different, both can be presented on the same display in the vehicle.

[0083] Figure 8 An automobile having sensors and systems for providing information before and after an accident is shown. An on-vehicle camera 81 is described as a sensor for sensing the interior of the cabin. Such an on-board camera can have different positions, which will be described further with reference to Figure 9 The on-vehicle camera 81 provides raw or pre-processed image data to a computing system 82. A pre-processing unit (not shown) can be placed between the on-vehicle camera 81 and the computing system 82. The computing system 82 can also receive other raw or pre-processed data from other sensors, such as wheel (e.g., speed, rate, acceleration) sensors 83, door sensors 84, distance sensors 85, or any other internal or external sensors provided by the vehicle. The pre-processing unit can also pre-process the raw data from other sensors before providing it to the vehicle's computing system 82.

[0084] Additionally, the computing system 82 can receive additional data from an advanced driver assistance system 86, which is connected to multiple sensors, such as wheel sensors 83, front distance sensors 85, side distance sensors 87, parking sensors 88 (which can also act as distance sensors), other distance sensors (not shown), etc. Those skilled in the art will understand that Figure 8 this is merely illustrative and there can be more or fewer sensors included by the automobile.

[0085] The advanced driver assistance system 86 generally includes technologies for assisting the driver in operating the vehicle safely. The advanced driver assistance system 86 uses inputs from sensors to detect nearby obstacles or driver errors. Thus, the advanced driver assistance system 86 has stored a large amount of information that can be provided to the vehicle computing system 82 for determining vehicle and passenger status information.

[0086] In addition, the computing system 82 may also be connected to an in-vehicle infotainment system 89. The in-vehicle infotainment system 89 is a series of hardware and software in a vehicle that provides audio or video entertainment but also allows passengers to interact with the computing system 82 via a user interface. In some embodiments, a passenger may use the in-vehicle infotainment system 89 to issue a no serious accident signal or a no accident signal as described above.

[0087] Figure 9 The interior of a vehicle with possible positions of cameras is shown. The vehicle may include only one camera, but may include multiple cameras at different positions. Cameras that may be color, monochrome or grayscale cameras, infrared cameras, depth cameras, thermal cameras, event cameras (i.e., cameras that operate at a very high frame rate and only output the differences between frames (e.g., motion)), or time-of-flight cameras or combinations thereof may be placed in a central position to allow the entire cabin to be photographed, for example, placed in the rearview mirror area (positions 91 and 92) or placed in the upper center console (position 93).

[0088] Additionally or alternatively, one or more cameras may be located on the instrument cluster or steering column in front of the driver (as shown in position 94), or in the A-pillars (as shown in positions 95 and 96). Additional cameras may be located, for example, in the backrests of the front seats, B-pillars or ceiling to have better visibility on the second or third row seats. With two or more cameras, a depth image or a 3D image may be created based on multi-view geometry and the principle of triangulation (e.g., stereo cameras). Then, these cameras may take images, for example, at regular time intervals or if triggered by an application that requires vehicle safety monitoring as described herein. Applications that use the images may be executed on the in-vehicle computing system or at least partially remotely (e.g., in the cloud). Other sensors may also be or additionally present at Figure 9 the positions shown, such as an internal radar sensor and / or an internal lidar sensor.

[0089] Figure 10 is a diagram of the internal components of a computing system 100 that implements the functions described herein. The computing system 100 may be located in a vehicle and includes at least one processor 101, a user interface 102, a network interface 103, and a main memory 106 that communicate with each other via a bus 105. Optionally, the computing system 100 may also include a static memory 107 and a disk drive unit (not shown), which also communicate with each other via the bus 105. A video display, an alphanumeric input device, and a cursor control device may be provided as examples of the user interface 102.

[0090] In addition, the computing system 100 may further include a designated camera interface 104 to communicate with an in-vehicle camera of the vehicle. Alternatively, the computing system 100 may communicate with the camera via the network interface 103. The camera is used to capture images. The computing system 100 may also be connected to a database system (not shown) via the network interface, where the database system stores at least a portion of the images for the functions described herein.

[0091] Compared with existing systems such as the eCall system, the emergency notification system described herein provides faster pre-accident signals and additional information to emergency responders, including, for example, the accident location obtained from an in-vehicle camera using 2D and 3D body key points, the number of vehicles involved, the vital signs of all passengers, and the positions of all passengers within the vehicle. Based on the defined and determined severity levels, different recommendations may be included in the signals described herein. For example, the post-accident signal may request which emergency responders and how to request them. Additionally, the system may use external vision algorithms to analyze data from radar, cameras, and lidar to predict whether an accident is about to occur and to evaluate the possible severity of the accident. By sending a short signal before the accident, timely notification of the emergency services is ensured even if the vehicle's electronics are damaged in the accident. Overall, the emergency notification system described herein provides a more comprehensive and proactive approach to emergency response, ultimately leading to improved outcomes for those involved in an accident.

[0092] The main memory 106 may be a random access memory (RAM) and / or any other volatile memory. The main memory 106 may store program code for the vehicle monitoring computing software 108 and the communication software 109. Other modules required for the other functions described herein may also be stored in the memory 106. The memory 106 may also store additional program data 110 required to provide the functions described herein, such as for performing preprocessing, etc. A portion of the program data 110, the vehicle monitoring computing software 108, and / or the communication software 109 may also be stored in a separate memory such as cloud memory and executed at least partially remotely.

[0093] According to one aspect, a vehicle is provided. The methods described herein may be stored as program code 108, 109, or 110 and may be included at least partially by the vehicle. Portions of the program code 108, 109, or 110 may also be stored on a cloud server and executed on the cloud server to reduce the computational workload on the vehicle's computing system 100. The vehicle may also include one or more cameras connected, for example, via the camera interface 104, for capturing one or more images.

[0094] According to one aspect, a computer program comprising instructions is provided. When the computer executes the computer program, the instructions cause the computer to perform the methods described herein. The program code embodied in any of the systems described herein can be distributed, individually or jointly, in various different forms as a program product. Specifically, the program code can be distributed using a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to perform aspects of the embodiments described herein.

[0095] A computer-readable storage medium that is non-transitory in nature can include volatile and non-volatile, removable and non-removable tangible media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. The computer-readable storage medium can also include random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid state memory technologies, portable compact disc read-only memory (CD-ROM) or other optical storage, magnetic tape cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be read by a computer.

[0096] A computer-readable storage medium should not itself be construed as a transient signal (e.g., radio waves or other propagating electromagnetic waves, electromagnetic waves propagating through a transmission medium such as a waveguide, or electrical signals transmitted through a wire). The computer-readable program instructions can be downloaded from the computer-readable storage medium to a computer, another type of programmable data processing device, or another device, or downloaded to an external computer or external storage device via a network.

[0097] It should be understood that while specific embodiments and variations have been described herein, further modifications and substitutions will be apparent to those skilled in the relevant art. In particular, examples are provided by way of illustration of the principles, and a variety of specific methods and arrangements are provided for making those principles effective.

[0098] In certain embodiments, the functions and / or actions specified in the flowcharts, sequence diagrams, and / or block diagrams can be reordered, processed serially, and / or processed concurrently without departing from the scope of the present disclosure. Additionally, any flowchart, sequence diagram, and / or block diagram can include more or fewer blocks than shown in the embodiments according to the present disclosure.

[0099] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the embodiments of the present disclosure. It should also be understood that when used in this specification, the terms "comprising" and / or "including" specify the presence of the stated features, elements, steps, operations, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, elements, steps, operations, components, groups thereof, and / or combinations thereof. Further, insofar as the terms "comprising", "having", "has", "with", "consisting of", or variants thereof are used in the detailed description or claims, these terms are intended to be inclusive in a manner similar to the term "including".

[0100] Although the description of the various embodiments has set forth the method and although these embodiments have been described in considerable detail, it is not the intention of the applicant to limit or in any way restrict the scope of the appended claims to such detail. Additional advantages and modifications will be apparent to those skilled in the art. Accordingly, the present disclosure in its broader aspects is not limited to the specific details, representative apparatus and methods, and illustrative examples shown and described. The described embodiments are, therefore, to be considered as illustrative and provided for the purpose of teaching general features and principles and not as limiting the scope as defined by the appended claims.

Claims

1. A computerized method for emergency notification of a vehicle, wherein: The vehicle includes an external sensor and an internal sensor, and the method includes the following steps: predicting a likelihood of the vehicle being involved in an accident based on the vehicle data and data received from one or more of the external sensors; and In response to the possibility of the accident occurring being higher than an accident threshold, a pre-accident signal is sent to a relay station, wherein the pre-accident signal includes vehicle and passenger status information generated based on the pre-accident vehicle data, the data received from one or more of the external sensors before the accident, and the data received from one or more of the internal sensors before the accident.

2. The method according to claim 1, further comprising the steps of: In response to the accident having occurred, determining a severity level of the accident based on vehicle data, data received from one or more of the external sensors, and data received from one or more of the internal sensors; In response to the severity level of the accident being higher than a severity threshold, sending a post-accident signal to the relay station, wherein the post-accident signal indicates confirmation of the accident and includes vehicle and passenger status information generated based on post-accident vehicle data, data received from one or more of the external sensors after the accident, and data received from one or more of the internal sensors after the accident.

3. The method according to claim 2, further comprising the steps of: In response to the severity level of the accident being lower than the severity threshold, a no serious accident signal is sent to the relay station to indicate that no serious accident has occurred and to trigger deletion of the pre-accident signal received at the relay station.

4. The method according to claim 3, wherein: The no-severity accident signal is manually initiated by an occupant of the vehicle via a user interface on a display of the vehicle or automatically initiated by an onboard computing system of the vehicle in response to determining that the severity level is below the severity threshold.

5. The method according to any one of claims 1 to 4, further comprising the following steps: In response to the accident not occurring, a no-accident signal is sent to the relay station to indicate that the accident did not occur and to trigger deletion of the pre-accident signal received at the relay station.

6. The method according to claim 5, wherein: The no-accident signal is manually issued by a passenger of the vehicle via a user interface on a display of the vehicle, or is automatically issued by an onboard computing system of the vehicle in response to detecting that the accident has not occurred based on data from at least one of the external sensors and the internal sensors.

7. The method according to any one of claims 1 to 6, wherein: The vehicle data includes at least one of a speed, an acceleration, a steering direction, a tire orientation, and a geographic location of the vehicle.

8. The method according to any one of claims 1 to 7, wherein: The external sensor includes at least one of a radar sensor, a camera, and a lidar sensor that captures the surrounding environment of the vehicle at specified time intervals.

9. The method according to any one of claims 1 to 8, wherein: The interior sensor includes at least one cabin camera that photographs a cabin of the vehicle at designated time intervals.

10. The method according to claim 9, wherein: The vehicle and passenger status information before and after the accident is determined based on images taken by the cabin camera, wherein the determination is based on at least one of semantic segmentation of the image, depth information analysis of the image, determination of three-dimensional posture estimation, and vital sign monitoring.

11. The method according to any one of claims 1 to 10, wherein: The vehicle and passenger status information before the accident includes at least one of: an estimated severity level of the accident; vehicle location; an estimated number of vehicles involved; an estimated type of vehicles involved; object motion prediction; the number of passengers in the vehicle; vital signs of passengers in the vehicle; the location of passengers in the vehicle; and the seat belt status of passengers in the vehicle.

12. The method according to any one of claims 2 to 11, wherein: The vehicle and passenger status information after the accident includes at least one of the following: the severity level of the accident; the vehicle location; the number of vehicles involved; the type of vehicles involved; vehicle deformation information; accident location at the vehicle; object motion information; environmental hazard information; number of passengers in the vehicle; vital signs of passengers in the vehicle; location of passengers in the vehicle; injury severity information of passengers in the vehicle; and the status of seat belts of passengers in the vehicle.

13. An emergency notification system, the emergency notification system comprising: one or more internal sensors; one or more external sensors; as well as A computing and communication system configured to perform the method according to any one of claims 1 to 12.

14. A vehicle comprising an emergency notification system according to claim 13.

15. A computer program product comprising instructions which, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 12.

Images