COLLISION DETECTION IN MOBILE DEVICE

Abstract

A method characterized in that it comprises the following: Detection of a collision event at a collision device (1001, 1002, 1808, 2201, 2202) with at least one processor (301, 302, 1702, 1703, 2002, 2304); Extracting multimodal features (306) from sensor data generated by multiple sensing modalities of the collision device (1001, 1002, 1808, 2201, 2202) with at least one processor (301, 302, 1702, 1703, 2002, 2304); Computing multiple collision decisions with the at least one processor (301, 302, 1702, 1703, 2002, 2304), based on multiple machine learning models (307) applied to the multimodal features (306); and Determine that a severe vehicle collision has occurred, using at least one processor (301, 302, 1702, 1703, 2002, 2304) incorporating the collision device (1001, 1002, 1808, 2201, 2202) based on the multiple collision decisions and a severity model (2204).

Description

TECHNICAL AREA

[0001] This disclosure relates generally to the use of a mobile device to detect when a user is involved in a serious motor vehicle accident. BACKGROUND

[0002] If a driver or passenger is injured or otherwise impaired in a serious car accident, they may be unable to call emergency services using their mobile phone or car phone. This is particularly serious if the accident occurs in a location where there are no bystanders to assist the driver or passenger.

[0003] The state of the art document DE 10 2021 115 371 A1 discloses communication between autonomous vehicles and unprotected road users.

[0004] The prior art document DE 10 2021 203 354 B4 discloses a method for determining a future accident severity of a motor vehicle with an object by means of an assistance system of the motor vehicle.

[0005] The prior art document DE 10 2022 104 129 A1 discloses a method for determining at least one health effect on at least one occupant of a vehicle in a collision, a method for training a predictive function, a monitoring device, a vehicle comprising a monitoring device and a training device.

[0006] The prior art document DE 10 2022 111 037 A1 discloses a method for processing data relating to a vehicle event for a vehicle, which includes: receiving vehicle sensor data from one or more vehicle sensors relating to the vehicle event; and determining an evaluation of the vehicle event via a processor, including a fault or severity or both associated with the vehicle event, based on the vehicle sensor data.

[0007] The state of the art document DE 10 2022 120 111 A1 discloses an estimation of the severity of injuries by using an in-vehicle perception system. SUMMARY

[0008] The present invention is defined in the independent claims. Advantageous embodiments are specified in the dependent claims.

[0009] Embodiments for collision detection on one or more mobile devices (e.g., smartwatch and / or smartphone) are disclosed. In some embodiments, a method comprises: detecting a collision event on a collision device with at least one processor; extracting multimodal features from sensor data generated by multiple sensing modalities of the collision device with the at least one processor; calculating multiple collision decisions with the at least one processor, based on multiple machine learning models applied to the multimodal features; and determining that a severe vehicle collision has occurred with the at least one processor, involving the collision device, based on the multiple collision decisions and a severity model.

[0010] In some embodiments, the method further includes, in response to the determination of a severe collision, displaying a notification on a screen of the collision device, requesting a response from a user of the collision device.

[0011] In some embodiments, the method further includes determining whether the collision device is stationary for a predetermined period; in response to the fact that the collision device is stationary for the predetermined period, starting a timer or counter; determining that the timer or counter has reached a threshold time or count; and escalating the notification.

[0012] In some embodiments, the escalation process further includes determining that no response to the notification has been received after reaching the threshold time or number, and automatically contacting emergency services using one or more communication modalities of the collision device.

[0013] In some embodiments, the method comprises: sending at least one of the multimodal features, collision decisions, inference of a severe collision, or user interactions with the notification to a network server computer; receiving at least one update to at least one parameter of at least one machine learning model or the severity model from the network server; and updating the at least one parameter with the at least one update using the at least one processor.

[0014] In some embodiments, at least one of the multimodal features is a delay pulse signature present in acceleration data.

[0015] In some embodiments, at least one of the multimodal features is a sound pressure level of audio data captured by at least one microphone of the collision device.

[0016] In some embodiments, at least one of the multimodal features is a pressure change due to airbag deployment in the vehicle.

[0017] In some embodiments, at least one of the multimodal features is a decrease in the speed of the collision device.

[0018] In some embodiments, the method further comprises: receiving collision-related features from an accompanying device coupled to the collision device, with the at least one processor; calculating multiple collision decisions based on multiple machine learning models applied to the multimodal and crash-related features, with the at least one processor; and deducing that a severe vehicle collision has occurred, with the at least one processor involving the collision device, based on the multiple collision decisions and a severity model.

[0019] In some embodiments, the method further includes matching epochs for the multimodal features with epochs for the additional multimodal features to remove misalignment between epoch boundaries.

[0020] Other embodiments relate to a device, a system, and a computer-readable medium.

[0021] Certain embodiments described herein offer one or more of the following advantages. The disclosed collision detection embodiments enable a collision detection device (e.g., smartphone, smartwatch) to automatically detect when a user is involved in a vehicle collision, while also reducing the occurrence of false collision detections. When a collision is detected, the collision detection device presents a user interface that warns the user for a period of time. If the user is responsive, they can swipe the screen or provide other input to immediately call emergency services or dismiss the warning if they do not require emergency services. If there is no user interaction with the collision detection device after the period has elapsed, a countdown begins.When the countdown ends, emergency services and / or the user's emergency contact list will be automatically contacted via a phone call or text message. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 illustrates a smartwatch for collision detection according to one or more embodiments. Fig. 2 is a collision detection timeline according to one or more embodiments. Fig. Figure 3 is a block diagram of a collision detection system according to one or more embodiments. Fig. 4A is a table listing observed sensor signals and corresponding detection modes for sensors of a collision device according to one or more embodiments. Fig. 4B is a diagram of delay pulse and impact signatures in inertial sensor signals indicating a collision, according to one or more embodiments. Fig. 4C is a diagram of a loud audio signal indicating a collision, according to one or more embodiments. Fig. 4D is a diagram of sudden pressure change that indicates airbag deployment during a collision according to one or more embodiments. Fig. 4E is a diagram of a large drop in speed indicating a vehicle collision, according to one or more embodiments. Fig. 5A is a diagram of inertial acceleration according to one or more embodiments. Fig. Figure 5B is a diagram of the normalized horizontal acceleration according to one or more embodiments. Fig. 5C is a histogram of probability as a function of peak delay for aggregated delay pulses according to one or more embodiments. Fig. Figure 6A is a diagram of a nominal airbag pressure signature resulting from deployment during a collision according to one or more embodiments. Fig. Figure 6B is a probability diagram as a function of peak pressure for an aggregate peak pressure disturbance, showing broad mass-related false positives and true positives for a collision according to one or more embodiments. Fig. Figure 7A is a diagram showing a sound pressure level (SPL) during a collision according to one or more embodiments. Fig. Figure 7B is a probability diagram as a function of duration for a burst duration of SPL above 130 dB within 200 ms, showing broad mass-related false positives and true positives for a collision according to one or more embodiments. Fig. Figure 8 is a state machine diagram for triggering an emergency signal (hereinafter referred to as the “SOS signal”) according to one or more embodiments. Fig. 9A is a timeline for safely notifying a user after a detected collision according to one or more embodiments. Fig. 9B is a state machine for a warning escalation flow according to one or more embodiments. Fig. Figure 10 illustrates the use of multiple collision devices for detecting collisions according to one or more embodiments. Fig. Figure 11 illustrates a cross-section of a software stack for collision detection according to one or more embodiments. Fig. 12A and Fig. Figure 12B illustrates the epoch for time alignment for multimodal collision detection according to one or more embodiments. Fig. Figures 13A-13C illustrate the modeling of false positive risks due to time control according to one or more embodiments. Fig. 14A and Fig. Figure 14B illustrates the alignment of epochs via two collision devices according to one or more embodiments. Fig. Figure 15 illustrates a detection service on the collision device according to one or more embodiments. Fig. Figure 16 illustrates a delay buffer that causes the collision device to pause for a predetermined period before processing, according to one or more embodiments. Fig. Figure 17 illustrates a distributed and closed system for actively modifying the collision detection behavior according to one or more embodiments. Fig. 18A and Fig. Figure 18B illustrates an architecture for the collection and processing of data aimed at the general public according to one or more embodiments. Fig. Figure 19 is a state machine diagram illustrating anomaly detection processes according to one or more embodiments. Fig. Figure 20 is a block diagram of a large-scale backend data analysis system according to one or more embodiments. Fig. Figure 21 illustrates an adjustable system process according to one or more embodiments. Fig. Figure 22 is a flowchart of a collision detection process, as described in reference to Fig. 1-21 described. Fig. Figure 23 is a block diagram of a device architecture for implementing the features and processes referred to in the Fig. 1-22 are described according to one embodiment. DETAILED DESCRIPTION Collision Detection Overview

[0022] Fig. Figure 1 illustrates an exemplary collision detection device, a smartwatch 100, for collision detection according to one or more embodiments. Other examples of collision detection devices include a smartphone and a tablet computer. The disclosed collision detection embodiments enable a collision detection device, such as a smartwatch 100, to detect when a user is in a collision, while also reducing the occurrence of false collision detections. In some embodiments, when a collision is detected, the smartwatch 100 displays a user interface (UI) 101 with a well-being check for a predetermined period. If the user is responsive, the user can swipe over the "SOS Emergency Call" request 102 to contact emergency services or tap "Cancel" to dismiss the alert. However, if the user does not respond after the predetermined period has elapsed (e.g.,If there is no user interaction with UI 101, a countdown timer starts (e.g., counting down from 10). When the countdown ends, emergency services and / or the user's emergency contact list are automatically contacted via one or more communication methods (e.g., a phone call or a text message).

[0023] In some embodiments, when emergency services answer the SOS call, a digital assistant on the smartwatch 100 begins playing an audio message from a speaker on the smartwatch 100, indicating that the user has been in an accident. In some embodiments, the audio message plays on a loop with a predetermined number of seconds of pause between each playback. The digital assistant also transmits the user's estimated location (e.g., latitude and longitude of the smartwatch 100) and a search radius to emergency services. In some embodiments, the user's estimated location can be announced during the call, provided in a text message or email, and / or also announced through the smartwatch 100's speaker if wireless communications are operational at the accident site (e.g., no signal).

[0024] Fig. 2 is a collision detection event timeline 200 according to one or more embodiments. At t=0 seconds, the collision event 201 is detected by the smartwatch 100. At t=15 seconds, a notification 202 is displayed on UI 10 with a well-being check, as shown in Fig. Figure 1 shows that if the user does not respond for N seconds (e.g., N=30 seconds) after the notification is sent, a countdown is started with an audible warning (203). If there is still no response from the user after the countdown is complete (e.g., no interaction with UI 101), emergency services (204) are contacted (e.g., 911 is dialed). This example timeline (200) is for illustrative purposes only. Other timelines with different time limits or different numbers and / or types of alert escalations can also be used. Example collision detection system

[0025] Fig. Figure 3 is a block diagram of a collision detection system 300 for a mobile device according to one or more embodiments. The system 300 could, for example, be based on the in Fig. The smartwatch shown in the image will be implemented in 100.

[0026] The system 300 includes a low-power processor 301, an application processor 302 (also referred to herein as the "always-on processor" (AOP)), an SOS state machine 303, and collision detection clients 304. The low-power processor 301 consumes less power than the application processor 302. For this reason, at least, the low-power processor 301 runs continuously to detect triggers indicating a severe vehicle collision (hereinafter referred to as "collision event triggers"). The collision event triggers may be generated, for example, based on various observed signals from multiple sensors on the collision device. The detection service 305, running on the application processor 302, monitors the collision trigger event signals sent by the low-power processor 301.When a trigger event (see dashed line) is received, an epoch is started, and the detection service 305 extracts multimodal features 306 from sensor streams output by the sensors. For example, the detection service 305 retrieves samples of sensor data from an IMU buffer, audio buffer, barometer buffer, and GPS buffer for each epoch. Multimodal features 306 are extracted from the buffered sensor data (e.g., acceleration, rotational speed, pressure, velocity, audio snippets) and fed into machine learning models 307 to generate collision decisions, such as airbag deployment, severe collision, rollover collision, impact energy / directionality, and silence.In particular, Models 307 are applied to multimodal features to generate estimates of whether an airbag has deployed, whether a collision has occurred, whether a rollover collision has occurred, whether impact energy / directionality has occurred that indicates a collision, and a period of silence (e.g., no observation of existing signals) that indicates a collision (referred to purely as "collision decisions"). Models 307 can implement any suitable supervised or unsupervised machine learning models, including, but not limited to: regression models (e.g., linear, logistic), decision trees, random forests, support vector machines, neural networks (e.g., convolutional neural networks), classifiers, Naive Bayes, clustering, dimensionality reduction (e.g., principal component analysis (PCA)), etc.

[0027] The collision decisions issued by models 307 are input into the inference machine 308. The inference machine 308 uses a severity model, a mode detector, and collision features to infer whether a severe collision has occurred. In some embodiments, the inference machine 308 outputs a probability that a severe collision has occurred based on the accident decisions.

[0028] If the probability exceeds a specified threshold, a severe collision is predicted and sent to the SOS state machine 303, which performs a UI escalation as described in [reference to]. Fig. 8 described. If the escalation level rises to the level of contacting emergency services (e.g., unresponsive users after a period / countdown), the collision clients (e.g., a telephony application, a messaging application, an email application, a digital assistant) are notified with a 304 error so that emergency services and / or emergency contacts can be contacted, as described in section 8. Fig. 1 and Fig. 2 described. When the SOS state machine 303 de-escalates (e.g., the user touches the "Cancel" option 103 before the period / countdown expires), a de-escalation signal is sent to the low-power processor 301 to reset the triggering process.

[0029] Fig. Figure 4A is a table listing observed signals and corresponding acquisition modes for sensors of a collision device (e.g., Smartwatch 100) according to one or more embodiments. Sensors may include, but are not limited to, individual inertial sensors (e.g., accelerometer, gyroscope) or an inertial measurement unit (IMU) containing multiple inertial sensors; signals from a global navigation satellite system (GNSS), such as the Global Positioning System (GPS), BeiDou, etc.; pressure changes measured by a pressure sensor (e.g., a barometer); and audio signals acquired by one or more microphones of the collision device.Some examples of observed signals include, but are not limited to, pre-collision signatures captured in IMU and GPS data, impact magnitude determined from IMU data, rollover detection determined from IMU and audio data, airbag deployment determined from pressure and audio data, collision noises determined from audio data, and pre-collision signatures determined from IMU and GPS data.

[0030] Fig. Figure 4B is a diagram of deceleration pulse and impact signatures in inertial sensor data indicating a collision, according to one or more embodiments. The vertical axis represents acceleration, and the horizontal axis represents time (s). The acceleration is obtained from an IMU of the collision device. The signature is characterized by sharp pulses that exceed a specified magnitude for a specified duration. The signatures can be empirically derived from, for example, collision test data.

[0031] Fig. Figure 4C is a diagram of the loudness of an audio clip indicating a collision, according to one or more embodiments. The vertical axis represents loudness (dB) and the horizontal axis represents time (seconds). The audio can be obtained by sampling the output of one or more microphones of the collision device and storing the samples in a buffer. The audio signature is characterized by a rise in loudness level from an ambient audio baseline loudness over a specified duration (rise time), followed by a specified duration at the higher loudness level, followed by a return to the ambient audio baseline loudness level. The signature can be obtained empirically from, for example, collision test data.

[0032] Fig. 4D is a sudden pressure change diagram that indicates airbag deployment during a collision according to one or more embodiments. The vertical axis represents pressure (kPa) and the horizontal axis represents time (seconds). A pressure change can be obtained from a barometer in the collision device. The pressure signature is characterized by a sudden pressure increase from a baseline pressure (rise time), followed by a return to the baseline pressure over a specified duration. The signature can be obtained empirically from, for example, collision test data.

[0033] Fig. 4E is a graph of a large GPS speed drop indicating a vehicle collision, according to one or more embodiments. The vertical axis represents GPS speed and the horizontal axis represents time. The speed signature is characterized by a large speed drop from a rated speed to a lower speed over a specified duration. The signature can be obtained empirically from, for example, collision test data.

[0034] Fig. Figure 5A is a diagram of inertial acceleration according to one or more embodiments. The accelerometer in the IMU is used to measure a sudden deceleration during the impact of a collision. The normalized average deceleration is obtained from the horizontal components of the acceleration α, as shown in equation [1]. x , α y calculated: AveDeceleration=‖∑t〚(ax〛,ay)‖2

[0035] Fig. Figure 5B is a graph of normalized horizontal acceleration indicating a collision. The collision pulse duration shown in the figure indicates a collision.

[0036] Fig. Figure 5C is a histogram of probability as a function of peak deceleration for aggregated (e.g., broad-mass) deceleration pulses according to one or more embodiments. The vertical axis represents probability, and the horizontal axis represents peak deceleration. Broad-mass false positives and true positives are shown. The broad-mass data were collected from test setups to determine false positives (no collisions) and true positives (collisions). As can be observed, the deceleration pulse signature is a good indicator of a collision when the peak acceleration is greater than 5 g.

[0037] Fig. Figure 6A is a diagram of a nominal airbag pressure signature resulting from deployment during a collision according to one or more embodiments. This feature detects large pressure disturbances caused by airbag deployment and cabin deformation. The significance of this feature lies in the fact that airbags are three times more likely to deploy in a severe collision, making this feature a good indicator of a serious collision. The vertical axis represents pressure (kPa) and the horizontal axis represents time (ms). The nominal airbag pressure signature, characterized by a sudden pressure spike over a specified duration, is shown.

[0038] Fig. Figure 6B is a probability graph as a function of peak pressure for aggregated peak pressure disturbances. False positives (no collision) and true positives (collision) relative to the broad mass are shown. As can be seen, the airbag pressure signature is a good indicator of a collision when the pressure exceeds approximately 0.5 kPa.

[0039] Fig. Figure 7A is a diagram showing a sound pressure level (SPL) during a collision according to one or more embodiments. The vertical axis represents sound pressure level (dB) and the horizontal axis represents time (ms). The human pain threshold and a jackhammer SPL at a distance of 2 m are shown. During a collision, a burst SPL rises above 130 dB within a specified duration (e.g., 200 ms).

[0040] Acoustic characteristics can include airbag deployment and the sounds of metal and glass breaking. Acoustic characteristics are less sensitive to car windows closing than the air pressure signature, which is measured using reference to... Fig. 6C is described. The acoustic features exhibit a high data rate, offering numerous frequency band features. These acoustic features are also independent of the coupled or projectile-like state of the collision device.

[0041] Fig. Figure 7B is a histogram of probability as a function of SPL burst duration. False positives and true positives related to the general population are shown. This histogram demonstrates that an SPL burst duration above 130 dB within 200 ms is a good indicator of a collision.

[0042] Fig. Figure 8 is a state machine diagram for triggering a distress signal (hereinafter referred to as the “SOS signal”) according to one or more embodiments. When a collision is detected, the SOS state machine 303 ( Fig. 3) transitioned. The SOS state machine 303 synchronizes the flow with both local and paired device collision processes. In some embodiments, the SOS state machine 303 includes the states: INACTIVE 801, POTENTIAL 802, STAGING 803, NOTIFY 804, and PROCESS 805.

[0043] If the low-performance processor 301 ( Fig. 3) If a collision trigger event is detected, the SOS state machine 303 transitions to POTENTIAL 802. If the inference machine 308 assumes a true collision, the SOS state machine 303 transitions to STAGING 803; otherwise, the SOS state machine 303 returns to INACTIVE 801. If the SOS state machine 303 is in STAGING 803 and a paired device detects a true collision, the SOS state machine 303 transitions to PROCESSING 805. If the SOS state machine 303 is in INACTIVE 801 and a paired device detects a true collision, the SOS state machine 303 transitions to PROCESSING 805.

[0044] After waiting for a time interval (a window for coordinating a paired device) in STAGING 803, SOS state machine 303 moves to NOTIFY 804. After notifying the paired device of a correct collision, SOS state machine 303 moves to PROCESS 805. SOS state machine 303 moves to INACTIVE 801 when the SOS alert is complete, the system exceeds a timeout, or a collision decision is discarded by the SOS UI escalation.

[0045] Fig. 9A is a 900-minute timeline for safely notifying a user after a detected collision according to one or more embodiments. Any interaction with a collision detection device while driving can be disruptive. Thus, the UI notification is delayed by detecting that the collision detection device is stationary (also referred to as "stationarity"), and / or it is otherwise a safe time to display a notification. Stationarity can be determined by a number of detection modalities, including but not limited to: GNSS, Wi-Fi, altimeter, and integration with the vehicle's computer (e.g., speedometer).

[0046] In timeline 900, the UI escalation window 901 begins X seconds after a collision is detected and continues for a specified period. In the event of a collision, a user's vehicle is often not immediately stationary, and / or the vehicle does become stationary, but the user continues driving to reach a safe location (e.g., the hard shoulder of a motorway). In such scenarios, the system should wait for a sufficient duration of stationary status before presenting the notification to ensure the user is at a standstill. If stationary status cannot be determined by the detection modalities, a notification (e.g., a well-being check UI) is displayed on a collision device indicator.If a false positive collision is detected and the user continues driving at speed, the notification expires after waiting a duration sufficient to detect stationaryness.

[0047] Fig. 9B is an exemplary state machine 902 for controlling the UI escalation flow according to one or more embodiments. When a severe collision is detected, the state machine 902 waits for a predetermined duration (e.g., using a counter), as determined by one or a combination of detection modalities, for stationarity 903. When the stationarity count reaches a threshold (e.g., greater than or equal to a threshold value), the state machine 902 enters an escalation state 904, and a UI escalation window 901 is displayed on the collision device.

[0048] In some embodiments, a stationarity check, which must be satisfied before a UI notification, includes meeting a specified GPS speed threshold (e.g., 3 mph) and meeting a specified time interval (e.g., >= 30 seconds). A timer is reset when the GPS speed reaches a specified threshold (e.g., >5 mph). If the GPS speed is unavailable, the operation is not performed. In some embodiments, the stationarity check has multiple constraints. For example, the UI does not escalate before P seconds (e.g., 20 seconds), and if it does not escalate by Q seconds (e.g., 60 seconds), the GPS history is checked for the past M scans (e.g., 10 scans), and the UI is then escalated using a more relaxed stationarity check.An example of a relaxed stationarity check includes an equivalent stationarity sample, such as GPS speed, that satisfies a threshold or a missing GPS sample, and a number of equivalent stationarity samples that satisfy a specified threshold Z of W samples (e.g., 8 out of 12 samples).

[0049] Fig. Figure 10 illustrates the use of multiple collision detection devices according to one or more embodiments. During a vehicle collision, multiple devices 1001, 1002 in the vehicle detect collision signatures, including but not limited to smartwatches, smartphones, and tablet computers. Combining the information from the multiple devices 1001, 1002 increases the collision detection performance. Since the vehicle environment exhibits similar collision signatures in collision detection devices during a collision, some embodiments utilize all available collision detection devices and their respective detection modalities to combine collision data for collision detection with the highest confidence.In this embodiment, collision data is shared by the collision devices 1001, 1002, and if the collision data indicates that both collision devices 1001, 1002 detect collisions, the arbitration logic is used to preferably select one of the collision devices 1001, 1002 to raise the UI escalation.

[0050] With renewed reference to Fig. 10. Collision device 1001 (e.g., a smartwatch) and collision device 1002 (e.g., a smartphone) interact in bidirectional messaging to request and send collision-related features. In the example shown, collision device 1001 evaluates collision features by sensing modalities and sends the features to collision device 1002. Similarly, collision device 1002 evaluates collision features detected by its own sensing modalities and sends these features to collision device 1001. Each collision device 1001, 1002 combines its locally detected and received collision features to conclude that a severe collision has occurred.If a severe collision is suspected, the collision device 1001 initiates the UI escalation and the collision device 1002 de-escalates the UI escalation, as the UI escalation is handled by the collision device 1001.

[0051] Fig. Figure 11 illustrates a cross-section of a software stack 1100 for collision detection according to one or more embodiments. Referring to the left-hand figure, the low-power processor 301 aggregates raw sensor data from various sensor data streams, such as IMU data (acceleration, rotational speed, altitude), barometer (pressure change), audio (audio samples captured by microphone(s)), and GPS data (e.g., GPS speed). The sensor data is sent to the application processor 302 after a collision trigger event. If a companion device (e.g., a smartphone) is available, collision features are also provided to the application processor 302 via a wireless communication link (e.g., Bluetooth) between the collision device and the companion device.

[0052] The detection service 305, running on the application processor 302, implements a flow controller 1101 that manages a collision detection processing pipeline. This pipeline includes a multimodal feature layer 306, an estimation layer 307, and an inference layer 308. The inference layer 308 issues a severe-class decision based on collision decisions generated by machine learning models implementing an estimation layer 307 and sends the severe collision decision to the SOS state machine 303. The SOS state machine 303 initiates a UI escalation 1102 or a de-escalation using timers, countdowns, and whether the user responds to the UI escalation (e.g., by initiating or canceling an SOS call).

[0053] As in the lower part of Fig. As shown in Figure 11, the recognition service 305 combines the sensor data streams in a first n-second window (e.g., n=1 s) and sorts the streams for the flow control 1101. The flow control 1101 then divides the streams into N measurement / observation epochs (e.g., N=4). The streams are fed into the feature layer 306, which extracts the features from the sensor streams and outputs a feature vector for each epoch that includes all features.

[0054] The feature vector for each epoch is fed into estimation layer 307, which calculates collision decisions (e.g., collision, rollover, airbag deployment, impact energy, etc.) based on various sensor data using machine learning models. The output of estimation layer 307 is fed into inference layer 308, which predicts / infers that a severe collision has occurred based on a severity model. For example, inference layer 308 generates a probability that a severe collision has occurred if the probability meets (e.g., exceeds) a specified threshold probability.

[0055] Fig. Figure 12A illustrates a time alignment for multimodal collision detection for a nominal case on a single device according to one or more embodiments. In this example, four epochs are shown: P0, P1, P2, and P3, and there is one feature vector per epoch. In some embodiments, an epoch lasts n seconds, and the epochs overlap in time (e.g., 50% overlap). In the example shown, P1 overlaps P0 and P2, and P2 overlaps P1 and P3. In this example, the epochs containing the accident event are P1 and P2. In epochs P1 and P2, the delay pulse feature report is set to IMU=True (a delay pulse was detected), and the audio feature report is set to isAudio=True (audio of the collision was captured). In this example, the collision and rollover models in estimation layer 307 combine these features to make a collision decision (a collision was detected) or a rollover decision (a collision was detected).To make a rollover decision (the impact was a rollover collision, which is usually more severe).

[0056] Fig. Figure 12B illustrates the use of 4-second epochs with 50% overlap, which are fed into feature layer 306, which outputs one feature vector per epoch, which is then fed into estimation layer 308, which outputs collision and rollover collision decisions.

[0057] Fig. Figures 13A-13C illustrate the modeling of false positive risks due to time-based control according to one or more embodiments. In the example shown, the overlapping epochs P0, P1, P2, and P3 are the same as in Fig. Shown in 12A. Fig. 13A contains P1 and P2 an IMU event and an audio event if the events are delayed by a time t delay are separated by a time of less than 2 seconds. In Fig. 13B does not contain an epoch for both the IMU and the audio event if the events are delayed by a time t. delay are separated by a period greater than 2 seconds. Fig. Figure 13C illustrates that the probability of both events falling in the same epoch is 100% when t delay The time interval is less than 2 seconds. Therefore, the features must be separated by no more than 2 seconds to reduce false positives.

[0058] Fig. 14A and Fig. Figure 14B illustrates the synchronization of epochs across two devices according to one or more embodiments. The upper sections of Fig. 14A and Fig. 14B illustrates the epochs W0, W1 and W2 for a first collision device (e.g. a smartwatch), and the lower sections of Fig. 14A and Fig. Figure 14B illustrates the epochs P0, P1 and P2 for a second collision device (e.g. a smartphone).

[0059] To handle misalignments between epoch boundaries, a zone of held-back epochs is defined as: t(W_i−2≤(P_j) <t(W_i)+2, where the “2” values ​​from left to right in equation [2] are below referred to as lookBackTime and lookForwardTime respectively, and according to Fig. 13A-13C are derived. The epochs falling within the held-back epoch zone have an overlap of more than or equal to 50%. The held-back epoch zone extends the false positive range to + / - 6 seconds, as shown in Fig. Figure 14B shows that in some embodiments, the preview time lookForwardTime is achieved by introducing a delay using a buffer (delay buffer size delauBufferSize) on the first data processing device, thereby achieving the remote audio data time.

[0060] Fig. Figure 15 illustrates an identification service (e.g., Identification Service 305 in Fig. 3) according to one or more embodiments. In some embodiments, the detection service can delay execution on a first device 1500 (e.g., a smartwatch) to allow the second device 1501 (e.g., a smartphone) time to send audio results. Sensor data is buffered, and processing is delayed by N seconds (e.g., N=5). Flow control and processing are cleared when inactive and received audio frames are stored and applied to the next trigger event. The audio frames are stored and processed within a time window [lookBackTime, +lookForwardTime].

[0061] With renewed reference to Fig. 15. The first device 1500 (e.g., a smartwatch) observes a collision trigger event and triggers the second device 1501 (e.g., a smartphone 1501). In response, the second device 1501 sends an audio result every N seconds to the first device 1500, which is buffered by the first device 1501. The first device 1501 delays processing by N seconds (N=5), then initializes a trigger session and a flow controller 1101. The flow controller 1101 delivers the buffered audio results and begins epoching the audio received from the second device 1501.

[0062] Fig. Figure 16 illustrates a delay buffer 1600 that causes the collision device to pause for a predetermined period before processing, according to one or more embodiments. The upper section of Fig. 16 is the process flow on the first device 1500 (e.g., smartwatch), and the lower section of Fig. 16 is the process flow on the second device 1501 (e.g., smartphone). On the second device 1501, the recognition service 1601 starts the flow controller 1101, which processes the 4-second epochs with 50% overlap that are input to feature layer 306. The features are wirelessly sent to the first device 1500, where they are stored in the delay buffer 1600 for 5 seconds before being processed by the flow controller 1101 and input to feature layer 306. The output of feature layer 306 includes multimodal features received from the second device 1501 (e.g., isAudio), which are combined with other features (e.g., isIMU) generated on the first device 1500 to form a single feature vector. The feature vector is then entered into estimation layer 307, which generates collision decisions, as before with reference to Fig. 3 described.

[0063] Fig. Figure 17 illustrates a distributed and closed system 1700 for actively modifying collision detection behavior according to one or more embodiments. The system 1700 is a distributed closed collision detection system that enables dynamic modification of behavior based on field statistics. In some embodiments, the system 1700 includes collision detection devices 1701 (e.g., smartwatches, smartphones) that send detection events to the analysis processor 1702, which operates on one or more network server computers. The collision detection devices 1701 also send algorithm feature values ​​to the backend analysis processor 1703, which operates on the one or more network server computers.The analysis processor 1702 generates detection statistics which are sent to the OTA update processor 1704 of the air actuator, which pushes new algorithm parameters to the collision detection devices 1701, as referred to in . Fig. 18 described in more detail.

[0064] Fig. 18A and Fig. Figure 18B illustrates a software architecture 1800 for large-scale data collection and processing according to one or more embodiments. In particular, the system 1800 generates new algorithm parameters for collision detection algorithms that are executed on collision devices in the field.

[0065] Sensor streams (e.g., audio, axial head, GPS speed, pressure, acceleration, device movement, etc.) are acquired / sampled and stored in memory 1801 of the collision device(s). In some embodiments, the audio data is converted to SPL-by-SPL computation 1802 on the collision devices, and the SPL is stored in memory 1801. The IMU trigger 1803 generates collision trigger events based on device movement data. In some embodiments, the IMU trigger 1803 also receives driving state data from the activity classifier 1804, which runs on the collision devices. The driving state data shows, for example, based on a classifier applied to various sensor data (e.g., acceleration), that the collision device (and thus the user) is currently driving a vehicle.In some embodiments, if the driving state data indicates that the collision device is not in a vehicle, the trigger does not occur. When the IMU trigger 1803 detects a collision event trigger, a start daemon 1805 starts the anomaly detection daemon 1806.

[0066] The anomaly detection daemon 1806 runs continuously on the collision devices in the field monitoring for collision trigger events. In some embodiments, a check of the users' mobile profiles 1807 is performed to determine whether the users have consented to share collision data via their collision device 1808. If the users have consented, the daemon 1806 collects the collision data 1809 (e.g., features, estimates, inference, SOS UI escalations) and sends the collision data 1809 to the uploader 1810, which sends the collision data 1809 to the backup data processor endpoint 1811 (e.g., implemented on network server computers). The backup data processor endpoint 1811 is a large-scale data processing architecture that hosts a variety of virtual machines 1813 for simulating collision information collected by collision devices.For example, virtual machines 1813 (also known as "replay") simulate collision events using the collision data to determine whether new algorithm parameters (e.g., thresholds, weights, coefficients, etc.) need to be generated for the installed base of collision devices in the field. The new parameters are downloaded to the collision devices via Mobile Assets endpoint / server 1815. The parameters are then updated with the new algorithm parameters.

[0067] Fig. Figure 19 is a state machine diagram illustrating the uploading of collision data to the backup data processor endpoint 1811 according to one or more embodiments. The goal of daemon 1806 is to upload collision data to the backup data processor endpoint 1811 so that the data can be processed in parallel by memory / CPU-intensive algorithms to classify a batch of sensor data into collision detection and non-collision detection events, as described in [reference to Figure 19]. Fig. 20 described in more detail.

[0068] In some embodiments, the states of daemon 1806 are classified into three types: Inactive 1901, Detect 1902, and Upload 1903. When a collision device is turned on, application processor 302 is configured to start a service to download a configuration for the collision device from mobile asset endpoint 1813 (see Fig. 18) IDLE daemon 1901 (inactive) waits for messages (e.g., interprocess communication messages) from the low-performance processor 301. In response to the messages (trigger message), the application processor 302 issues a trigger event that starts the discovery service 305 ( Fig. 3) The collision data is spooled by the detection service 305 for upload to the backup data processor endpoint 1812. The upload daemon 1806 uploads the collision data to the backup data processor endpoint 1811.

[0069] Fig. Figure 20 is a block diagram of a backend data analysis system 2000 according to various embodiments. The system 2000 is configured to identify a false positive escalation rate, which is used to determine whether an improvement to the collision detection algorithm is needed. Starting from the left part of Fig. 20 is powered by the low-performance processor 301 ( Fig. 3) The generated collision trigger event is processed by the Enrichment Check 2001 to determine a probability of whether a given collision event trigger was actually a collision (pass) or not a collision (fail). The trigger is also sent to the Analysis Server 2002. The output of the Enrichment Check 2001 is sent to the Backup Server 2003. The Backup Server 2003 splits the data output from the Enrichment Check 2001 into separate pass / fail processing paths. The collision data is processed by the Accountability Checks 2005a and 2005b in the processing paths, and if the data passes the Accountability Checks 2005a and 2005b, the collision data is fed into the Collision Detectors 2006a and 2006b, which evaluate the collision events using the collision data for pass (f). pass ) and failure (f fail ) simulate.

[0070] The outputs of the collision detectors 2006a and 2006b are fed into the inverting probe 2007. The inverting probe 2007 samples the collision data until a pre-specified number of collision trigger events is observed. In some embodiments, the number of observed collision trigger events is predetermined, and the sampling rate is a random variable following a negative binomial distribution.

[0071] The output of the invert sampling 2007 (f) is divided by driving hours per trigger T, which is calculated by the analysis processor 2002, yielding the false positive rate T / f. The false positive rate can then be used to determine whether the collision detection algorithms running on the collision devices need to be improved, for example, with new tuning parameters.

[0072] Fig. Figure 21 illustrates an adjustable in-system algorithm according to one or more embodiments. Due to the large amount of collision data related to the mass, a tunable OTA operating point is used, as shown in Fig. Figure 21 shows that the OTA frequency is tuned to strike a balance between collisions of the lowest severity (most frequent and requiring the least assistance) and collisions of the highest severity (least frequent, requiring the most assistance). Fig. Figure 21 also shows a graph of FP (hours per detection) as a function of the OTA tuning parameter. Fig. Figure 21 also shows a graph of the probability of serious injury as a function of the OTA tuning parameter. It should be noted that the OTA tuning parameter is chosen to target only the most severe collisions and collisions with a lower probability of serious injury. Example process

[0073] Fig. Figure 22 is a flowchart of a process 2200 for collision detection, as referenced in Fig. described in 1-21. Process 2200 includes the steps: detecting a collision event on a collision device (2201); extracting multimodal features from sensor data generated by multiple sensing modalities of the collision device (2202); calculating multiple collision decisions based on multiple machine learning models applied to the multimodal features (2203); and determining that a severe vehicle collision has occurred, involving the collision device based on the multiple collision decisions and a severity model (2204).

[0074] Each of these steps was previously described with reference to Fig. 1-21 described in detail. Example device architecture

[0075] Fig. 23 is a block diagram of a collision device architecture 2300 for implementing the in Fig. Features and processes described in sections 1-22. The architecture 2300 can include a memory interface 2302, one or more hardware data processors, image processors and / or processors 2304, and peripheral interfaces 2306. The memory interface 2302, the one or more processors 2304, and / or the peripheral interface 2306 can be separate components or can be integrated into one or more integrated circuits. The system architecture 2300 can be included in any suitable electronic collision detection device, including, but not limited to: a smartwatch, a smartphone, a fitness band, and any other device that can be attached, worn, or held by a user.

[0076] Sensors, devices, and subsystems can be coupled to the 2306 peripheral interface to provide multiple functionalities. For example, one or more 2310 motion sensors, a 2312 light sensor, and a 2314 proximity sensor can be coupled to the 2306 peripheral interface to facilitate motion detection (e.g., acceleration, rotation rates), illumination, and proximity functions of the wearable device. A 2315 location processor can be connected to the 2306 peripheral interface to provide geo-positioning. In some implementations, the 2315 location processor can be a GNSS receiver, such as a Global Positioning System (GPS) receiver. An electronic magnetometer 2316 (e.g., a microchip) can also be connected to the 2306 peripheral interface to provide data that can be used to determine the direction of magnetic north.The electronic magnetometer 2316 can provide data to an electronic compass application. One or more motion sensors 2310 can include one or more accelerometers and / or gyroscopes designed to detect changes in speed and direction of motion. The barometer 2317 can be configured to measure atmospheric pressure (e.g., pressure changes within a vehicle). The Biosignal Sensor 2320 can be one or more of a PPG sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an electromyogram (EMG) sensor, a mechanomyogram (MG) sensor (e.g., piezoresistive sensor) for measuring muscle activity / contractions, an electrooculography (EOG) sensor, a galvanic skin response (GSR) sensor, a magnetoencephalogram (MEG) sensor, and / or one or more other suitable sensors configured to measure biosignals.

[0077] Communication functions can be facilitated by wireless communication subsystems 2324, which may include radio frequency (RF) receivers and transmitters (or transceivers) or optical (e.g., infrared) receivers and transmitters. The specific design and implementation of the communication subsystem 2324 may depend on the communication network(s) over which a mobile device is intended to operate. For example, the architecture 2300 may include communication subsystems 2324 designed to operate over a GSM network, a GPRS network, an EDGE network, a Wi-Fi™ network, and a Bluetooth™ network. In particular, the wireless communication subsystems 2324 may include hosting protocols, allowing the collision device to be configured as a base station for other wireless devices.

[0078] An audio subsystem 2326 can be coupled with a loudspeaker 2328 and a microphone 30 to support voice-enabled functions, such as voice recognition, voice replication, digital recording, and telephony functions. The audio subsystem 2326 can be configured to receive voice commands from the user. The audio subsystem 2326 can be used to capture audio during a collision and convert the audio into SPL for collision detection processing.

[0079] The I / O subsystem 2340 can include a touch-sensitive controller 2342 and / or one or more other input controllers 2344. The touch-sensitive controller 2342 can be coupled to a touch-sensitive surface 2346. The touch-sensitive surface 2346 and the touch-sensitive controller 2342 can, for example, detect contact and movement or interruption thereof using any of a plurality of touch-sensing technologies, including technologies based on capacitance, resistance, infrared, and surface acoustic waves, as well as other proximity sensor arrays or other elements for determining one or more points of contact with the touch-sensitive surface 2346. The touch surface 2346 can, for example, include a touchscreen or the digital crown of a smartwatch.The I / O subsystem 2340 can include a haptic machine or device for providing haptic feedback (e.g., vibrations) in response to commands from the processor 2304. In one embodiment, the touch-sensitive surface 2346 can be a pressure-sensitive surface.

[0080] One or more input controllers 2344 can be coupled with other input / control devices 2348, such as one or more buttons, toggle switches, a knurled wheel, an infrared port, and a USB port. The one or more buttons (not shown) can include a volume up / down button for volume control of the speaker 2328 and / or a microphone 2330. The touch-sensitive surface 2346 or other controllers 2344 (e.g., a button) can include or be coupled to fingerprint identification circuitry for use with a fingerprint authentication application to authenticate a user based on their fingerprint(s).

[0081] In one implementation, pressing the button for a first duration can unlock the touch-sensitive surface 2346; and pressing the button for a second duration longer than the first can turn the power supply to the mobile device on or off. The user may be able to customize the functionality of one or more of the buttons. The touch-sensitive surface 2346 can also be used, for example, to implement virtual or soft buttons.

[0082] In some implementations, the mobile device can play back recorded audio and / or video files, such as MP3, AAC, and MPEG files. In some implementations, the mobile device can include the functionality of an MP3 player. Other input / output and control devices can also be used.

[0083] The 2302 memory interface can be coupled to a 2350 memory. The 2350 memory can include high-speed random access memory and / or high-speed non-volatile memory, such as one or more magnetic disk storage devices, one or more optical storage devices, or flash memory (e.g., NAND, NOR). The 2350 memory can store the 2352 operating system, such as the iOS operating system developed by Apple Inc., Cupertino, California. The 2352 operating system can include instructions for handling basic system services and performing hardware-dependent tasks. In some implementations, the 2352 operating system can include a kernel (e.g., a UNIX kernel).

[0084] Memory 2350 can also store communication instructions 2354 to enable communication with one or more additional devices, one or more computers, or one or more servers, such as instructions for implementing a software stack for wired or wireless communication with other devices.Memory 2350 can include graphical user interface instructions 2356 to facilitate graphical user interface processing; sensor processing instructions 2358 to enable sensor-related processing and functions; telephone instructions 2360 to facilitate telephone-related processes and functions; electronic messaging instructions 2362 to enable processes and functions related to electronic messaging; web browsing instructions 2364 to enable processes and functions related to web browsing; media processing instructions 2366 to facilitate processes and functions related to media processing; GNSS / location instructions 2368 to facilitate GNSS and location-related processes and instructions; and collision detection instructions 2370, which are related to... Fig.Implement the collision detection processes described in 1-22. Memory 2350 further includes other application instructions 2372, including, but not limited to, instructions for applications that use a collision detection output.

[0085] Each of the instructions and applications mentioned above can correspond to a set of instructions for performing one or more of the functions described above. These instructions need not be implemented as separate software programs, sequences, or modules. Memory 2350 can accommodate additional or fewer instructions. Furthermore, various functions of the mobile device can be implemented in hardware and / or software, including one or more signal processing circuits and / or one or more application-specific integrated circuits.

Claims

[1] Procedure, characterized by , that it includes the following: Detection of a collision event at a collision device (1001, 1002, 1808, 2201, 2202) with at least one processor (301, 302, 1702, 1703, 2002, 2304); Extracting multimodal features (306) from sensor data generated by multiple sensing modalities of the collision device (1001, 1002, 1808, 2201, 2202) with at least one processor (301, 302, 1702, 1703, 2002, 2304); Computing multiple collision decisions with the at least one processor (301, 302, 1702, 1703, 2002, 2304), based on multiple machine learning models (307) applied to the multimodal features (306); and Determine that a severe vehicle collision has occurred, using at least one processor (301, 302, 1702, 1703, 2002, 2304) incorporating the collision device (1001, 1002, 1808, 2201, 2202) based on the multiple collision decisions and a severity model (2204). [2] The method of claim 1, further comprising: In response to the determination of a serious collision, a notification (202) is displayed on a screen of the collision device (1001, 1002, 1808, 2201, 2202) requesting a response from a user of the collision device (1001, 1002, 1808, 2201, 2202). [3] Method according to claim 2, further comprising: Determine whether the collision device (1001, 1002, 1808, 2201, 2202) is stationary for a predetermined period of time; as a reaction to the fact that the collision device (1001, 1002, 1808, 2201, 2202) is stationary for the predetermined period, Starting a timer or counter; Determine that the timer or counter has reached a threshold time or number; and Escalate notification (202). [4] The method of claim 3, further comprising: As a result of escalation, determine that no response to the notification (202) has been received after reaching the threshold time or number, and automatically contact emergency services using one or more communication modalities of the collision device (1001, 1002, 1808, 2201, 2202). [5] The method of claim 1, further comprising: Sending at least one of the multimodal features (306), collision decisions, inference of a serious collision or user interactions with the notification (202) to a network server computer; Receiving at least one update on at least one parameter of at least one machine learning model (307) or severity model (2204) from the network server; and Updating at least one parameter with at least one update with at least one processor (301, 302, 1702, 1703, 2002, 2304). [6] Method according to claim 1, wherein at least one of the multimodal features (306) is a delay pulse signature present in acceleration data. [7] Method according to claim 1, wherein at least one of the multimodal features (306) is a sound pressure level of audio data acquired by at least one microphone of the collision device (1001, 1002, 1808, 2201, 2202). [8] Method according to claim 1, wherein at least one of the multimodal features (306) is a pressure change due to airbag deployment in the vehicle. [9] Method according to claim 1, wherein at least one of the multimodal features (306) is a speed reduction of the collision device (1001, 1002, 1808, 2201, 2202). [10] Method according to claim 1, further comprising: Receiving collision-related features from an accompanying device coupled to the collision device (1001, 1002, 1808, 2201, 2202) with which at least one processor (301, 302, 1702, 1703, 2002, 2304) is connected; Computing multiple collision decisions based on multiple machine learning models (307) applied to the multimodal features (306) and crash-related features, using at least one processor (301, 302, 1702, 1703, 2002, 2304); and Inferring that a severe vehicle collision has occurred, using at least one processor (301, 302, 1702, 1703, 2002, 2304) incorporating the collision device (1001, 1002, 1808, 2201, 2202) based on the multiple collision decisions and a severity model (2204). [11] The method of claim 8, further comprising: Matching epochs for the multimodal features (306) with epochs for the additional multimodal features (306) to remove misalignment between epoch boundaries. [12] Institution, comprehensive: at least one motion sensor (2310); at least one processor (301, 302, 1702, 1703, 2002, 2304); a memory (1801, 2350) that stores instructions (2364) which, when executed by the at least one processor (301, 302, 1702, 1703, 2002, 2304), cause the at least one processor (301, 302, 1702, 1703, 2002, 2304) to perform characteristic operations that include the following: Detection of a collision event at least partially based on sensor data from at least one motion sensor (2310); Extracting multimodal features (306) from sensor data generated by multiple acquisition modalities of the facility; Computing multiple collision decisions based on multiple machine learning models (307) applied to the multimodal features (306); and Infer that a severe vehicle collision has occurred, taking into account the facility based on the multiple collision decisions and a severity model (2204). [13] Device according to claim 12, further comprising: In response to the determination of a serious collision, a notification (202) is displayed on a screen of the collision device (1001, 1002, 1808, 2201, 2202) requesting a response from a user of the facility. [14] Device according to claim 13, further comprising: Determining the stationary nature of the facility; Starting a timer or counter; Determine that the timer or counter has reached a threshold time or number; and Displaying the notification (202) on the facility's screen requesting a response from a user of the facility. [15] Device according to claim 14, further comprising: Determine that no response to the notification (202) was received from the user; and Automatic contacting of emergency services using one or more of the facility's communication modalities. [16] Device according to claim 12, wherein at least one of the multimodal features (306) is a delay pulse signature present in acceleration data. [17] Device according to claim 12, wherein at least one of the multimodal features (306) is a sound pressure level of audio data captured by at least one microphone of the device. [18] Device according to claim 12, wherein at least one of the multimodal features (306) is a pressure change due to airbag deployment in the vehicle. [19] Device according to claim 12, wherein at least one of the multimodal features (306) is a speed reduction of the device. [20] Device according to claim 12, further comprising: Receiving collision-related features from an accompanying device coupled to the installation; Computing multiple collision decisions based on multiple machine learning models (307) applied to the multimodal features (306) and crash-related features; and Infer that a severe vehicle collision has occurred, incorporating the collision device (1001, 1002, 1808, 2201, 2202) based on the multiple collision decisions and a severity model (2204). [21] Device according to claim 20, further comprising: Matching epochs for the multimodal features (306) with epochs for the additional multimodal features (306) to remove misalignment between epoch boundaries. [22] Device according to claim 12, further comprising: Sending at least one of the multimodal features (306), collision decisions, inference of a serious collision or user interactions with the notification (202) to a network server; Receiving at least one update to at least one parameter from at least one machine learning model (307) or the severity model (2204); and updating the at least one parameter with the at least one update.

Images