Method for determining tag of fall event

Abstract

A method for determining a true tag of a fall event is disclosed. The method comprises: receiving signals from one or more sensors configured to measure signals indicative of a characteristic of user movement from a distance; analyzing the received signal using a fall detection algorithm to determine a tag indicative of a fall event of the user; initiating a first user interface interaction mode of the user interface in which the user interface is configured to receive a first input from the user indicating a self-tag of the fall event; receiving a first input; and determining a level of mismatch between the self-tag and the determined tag. If the level of mismatch is above a threshold, the method further includes switching the user interface to a second user interface interaction mode in which the user interface is configured to receive a second input from the user indicating contextual information about the fall event; receiving a second input; and updating a self-tag of the fall event based on the received second input.

Description

Technical Field

[0001] The present invention relates to a method for determining a signature of a fall event, a controller for determining a signature of a fall event, and a system for determining a signature of a fall event. Background Art

[0002] Falls are a significant problem in elderly care that can contribute to morbidity and mortality in the elderly. Falls can be harmful to the elderly and, from a psychological perspective, often cause fear of falling, which in turn leads to social isolation and depression. With the growing aging population, there is an urgent need to develop fall detection and / or prevention systems. Due to the rapid development of sensor networks and advances in software technology (machine learning algorithms), fall detection systems can use various sensors (such as accelerometers, radar sensors, time-of-flight (ToF) sensors, Wi-Fi nodes, etc.) to detect signal patterns that are characteristic of falls, and thereby determine whether a fall event has occurred.

[0003] However, although current fall detection systems work well under laboratory conditions, it remains problematic to produce reliable results when these systems are applied to real-life conditions. Fall detection algorithms are typically pre-trained on training datasets consisting primarily of laboratory simulated fall data, using only a small amount of available real-world fall data. In order to improve the accuracy of fall detection systems, pre-trained fall detection algorithms need to be refined (updated) for specific elderly care facilities and / or specific elderly behaviors to better detect extreme cases of fall detection. Retrospective validation of fall time and type (retrospective labeling of fall events) is necessary to adapt fall detection algorithms to the specific details of the elderly / care facility and / or the specific details of the elderly's activities and motoric movements. Due to privacy regulations and other practical reasons, labeling of fall events under real-life conditions needs to be performed by the elderly (self-labeling) or one or more caregivers (or staff in the care facility). Summary of the Invention

[0004] The inventors have recognized that older adults often tend to underreport or intentionally lie about whether an incident constitutes a fall. This may be due to a fear of being deprived of their independence, because the fall was caused by their actions (e.g., such as getting up to go to the bathroom without seeking help from a caregiver), memory loss, etc. Consequently, the self-labeling provided by the older adult (e.g., via a user interface) to the fall detection system may be inaccurate or even intentionally false. False (inaccurate) labels can severely compromise the accuracy of the fall detection system. Underreporting of fall events (i.e., when an older adult intentionally labels a fall event as not a fall) can result in an increased number of false negatives in the fall detection system, including its ability to immediately report a fall. On the other hand, overreporting of fall events (i.e., when an older adult self-labels a non-fall event as a fall) can result in an increased number of false positives, leading to costly, unnecessary action by hospitals and / or nursing facilities and / or caregivers associated with the older adult at home.

[0005] Therefore, the aim is to provide a method for determining more accurate labels for fall events.

[0006] According to a first aspect, the object is achieved by a method for determining a label for a fall event. The method comprises the following steps: receiving a signal from one or more sensors configured to measure a signal indicating a characteristic of a user's movement; analyzing the received signal using a fall detection algorithm to determine a label indicating a fall event; starting a first user interface interaction mode of a user interface, wherein in the first user interface interaction mode, the user interface is configured to receive a first input from the user indicating a self-label for a fall event; receiving the first input; and determining a mismatch level between the self-label and the determined label. If the mismatch level is above a threshold, the method comprises switching the user interface to a second user interface interaction mode, wherein in the second user interface interaction mode, the user interface is configured to: receive a second input from the user indicating contextual information about the fall event; receiving the second input; and updating the self-label for the fall event based on the received second input.

[0007] Signals from one or more (remote) sensors (such as radar sensors, time-of-flight (ToF) sensors, Wi-Fi Doppler sensors, microphone sensors, etc.) can be used to determine movement characteristics (patterns) and / or audio patterns of a user that indicate a fall. The pattern can include the fall event itself as well as patterns before and after the fall. The fall detection and / or prevention algorithm analyzes the received sensor signals to determine a label indicating a fall event associated with the received signal. The fall detection algorithm may have been trained to determine whether a fall has occurred based on the received signal. For example, the determined label can indicate whether the received signal includes a fall event or does not include a fall event, and can indicate the type of fall event included in the received signal (e.g., a fall with injury, a fall without injury, a soft fall, a stroke fall, a near fall (an elderly person loses balance and does not fall to the floor), etc.). In order to improve the accuracy of the fall detection algorithm to identify fall events and / or classify the specific type of fall event, the user can be asked to self-label the fall event. User feedback (self-labeling of fall events) is particularly necessary in the case of remote sensing modalities with limited accuracy in determining fall events. In the first user interface interaction mode, the user provides self-labeling of the fall event. The self-labeling can indicate whether the fall event occurred or did not occur, and can indicate the type of fall event (e.g., fall with injury, fall without injury, etc.).

[0008] When the received signal data is labeled, it can be used to retrain (update) the fall detection algorithm to better identify movement patterns and / or audio characteristics of falls (or types of falls) in the future. However, for example, due to the reasons described above, the self-labeling of fall events provided by the user may be intentionally wrong and / or inaccurate. The method includes determining a mismatch level between the self-labeling provided by the user and the label of the fall event determined by the fall detection algorithm. If the mismatch level is above a threshold, the method includes switching the user interface interaction mode to a second mode, wherein the user interface is configured to receive a second input indicating contextual information about (associated with) the fall event. That is, the contextual data associated with the circumstances of the fall event contextualizes the fall event (provides a broader understanding of the fall event) and enables the veracity of the self-declaration of the fall event provided by the user to be judged. For example, the contextual data about the fall event may include the user's actions before the event was labeled as a fall. In another example, the user can be triggered / questioned to revisit his / her self-labeling of the event. In another example, the contextual data may include information supporting his / her self-labeling by the user. The user may have initially provided an erroneous and / or inaccurate self-label for the fall event. By being explicitly asked to provide additional (contextual) information about the event, the user is triggered (challenged) to revisit, reconsider his / her initial self-label, and provide an accurate (trusted) label for the event. This results in improved labeling and, thereby, can lead to more efficient updating (retraining) of the fall detection algorithm.

[0009] The second input may include verbal and / or non-verbal cues. The method may also include analyzing the verbal and / or non-verbal cues in the second input to determine a user intent score, the user intent score indicating the user's intent to deceive the fall detection system (regarding the self-label), and updating the self-label of the fall event based on the user intent score. For example, the user intent score may be a probability (likelihood) that the user has provided an incorrect self-label. Various machine learning (ML) models and techniques can be used to determine whether a user has deceptive intent based on verbal and non-verbal cues that indicate evidence of deception in the user's response. For example, speech parameters including non-verbal parameters (cues) (e.g., pitch, duration pattern, energy) and linguistic parameters (e.g., filled pauses in multiple audible responses, such as "um" or "ah") can be used as input to a speech ML model to determine the user's intent to deceive or not deceive when generating the audible response. Natural language processing (NLP) models (such as stylometric models) can be used to determine (classify) whether (part of) the text in the user's text response is deceptive based on the language (inconsistencies in the response in the second user input) and non-verbal (linguistic) cues in the text response. In another example, visual features in the video answer provided by the user in the second input can be used as input to an ML model (such as a support vector machine and a logistic regression model) to determine the user's intent to deceive or not deceive when generating the video answer. For example, by analyzing micro-expressions and eye movements that indicate deceptive behavior. By associating the user intent score with the self-label provided by the user, the self-label can be updated accordingly. For example, if the user intent score indicates that the user's intent is not to deceive, the self-label is updated based on the first user input. Alternatively, if the user intent score indicates that the user's intent is to deceive, the self-label is updated based on the contextual information. This further improves labeling and can thereby lead to more efficient updating (retraining) of the fall detection algorithm. The method can also include storing the user intent score along with the updated self-label in a training set and updating the fall detection algorithm based on the training set. Updating the fall detection algorithm to take into account the uncertainty associated with the user self-label can increase the accuracy and robustness of the fall detection algorithm, particularly for fine-tuning fall detection / fall prevention systems for the unique erratic behaviors of the elderly and the specific room settings of the elderly.

[0010] The method may also include receiving additional input indicative of a physiological parameter of the user during the time period in which the user provides the second input, and analyzing the additional input to determine a user intent score based on the user's physiological parameter. When people lie, their physiological reactions during their response may indicate a stress response that may be caused by lying. For example, a person who is lying may be more agitated, which may be reflected by an increased heart rate and breathing rate, or may sweat more (which results in a change in skin conductivity). By analyzing the user's physiological reactions (parameters) during the time period in which the user provides the second input, a better estimate of the user's intent to deceive may be achieved.

[0011] The method may also include obtaining a historical (past) user intent score for the user and determining a (current) user intent score based on the historical user intent scores. A person who is found to have intentionally misrepresented their previous self-labeling of fall events may be more inclined to provide a currently inaccurate labeling of a fall event. Thus, by taking into account the historical user intent scores, the current user intent score may be determined more accurately.

[0012] The step of receiving a second input indicating contextual information about a fall event includes receiving information about at least one of the following: the user's actions before the fall event, the user's supporting evidence regarding the fall event, the location data of the fall event, the time data of the fall event, the presence of another person (e.g., a nurse) during the fall event, and the light settings (intensity and spectrum) during and before the fall event. For example, although people may fall for different reasons under different conditions, the probability of falling is higher when walking or walking up / down stairs. A large percentage of falls are caused by improper sit-to-stand transitions. Similarly, due to temporary muscle weakness and balance disorders, the elderly are more likely to fall when they just wake up after sleeping. Therefore, the user's actions before the fall event can be a good indicator of a fall event. The time and location of the fall include important information to provide context for the fall event. For example, a large proportion of elderly people fall while using the toilet at night. The presence of another person during the fall may indicate that the elderly person may not have attempted to walk to the toilet alone, thereby making a trip and fall event less likely. The lighting (brightness level) at the fall location may also contribute to the fall event. Similarly, the lighting (light intensity and spectrum) that the elderly were exposed to in the day / hours before a fall may also contribute to the fall event. Studies have shown that users exposed to circadian lighting (lighting settings designed to promote circadian health) had a 40% reduction in fall rates. Challenging the user to provide supporting evidence about the fall may trigger the user to reconsider the self-label provided or expose inconsistent answers, which indicates that the user's self-claim is not true. Therefore, receiving contextual information (data) associated with the fall event enables a broader understanding of the fall event and triggers the elderly to confirm / de-confirm his / her initial self-label of the fall event. In addition, the contextual information (data) enables to expose inconsistent answers about the sensory data from the fall.

[0013] In the second user interface interaction mode, the user interface may be configured to select a question set from a set of default question sets and output the selected question set to the user. For example, the set of default question sets may include questions such as: "What did you do before the fall?", "What was the location of the fall?", "Are you injured?", etc. Outputting the selected question set to the user may facilitate providing him / her with contextual information about the fall event.

[0014] Additionally and / or alternatively, the user interface can be configured to determine a question setting based on a natural language processing (NLP) algorithm and output the determined question setting to the user. A powerful new class of large language models is making it possible for machines to generate text in natural human language. These large language models can generate a priori (non-existent) follow-up questions to the elderly in natural human language.

[0015] The user interface can be configured to determine a question setting based on the level of mismatch between the self-label and the determined label, and output the determined question setting to the user. For example, subsequent questions (settings) can be customized based on the level of mismatch between the self-label and the label determined (by the fall detection algorithm). The level of mismatch between the self-label and the label determined by the algorithm can indicate the user's intention to deceive or not to deceive. The question style wording (e.g., friendly rather than adversarial) affects people's response to questions. If the user's first input is truthful, a strictly adversarial question may cause user dissatisfaction, however, if the user's first input is deceptive, such a question will prompt the user to provide an accurate self-label of the fall event. Therefore, by optimizing the question style (selecting the question setting) based on the level of mismatch between the self-label and the determined label, a more accurate self-label can be determined.

[0016] Determining a label indicating a fall event may include determining the type of fall event, and when the determined fall event (the fall itself and the activities before and after the fall) is of a new type that has not been seen before, the first user interface interaction mode may be activated. The labeling of the fall event significantly improves the accuracy of fall detection. However, the elderly may be annoyed if he / she is often asked to provide self-labels of fall events. If the fall type is very common for this particular user (e.g., a near fall without injury), it may not be necessary to probe the user to provide a self-label. However, if the fall detection algorithm predicts a fall type that has not been seen before, the fall detection algorithm may have low confidence in such a prediction. Therefore, it is beneficial to activate the first user interface interaction mode only when a new (unseen for the elderly) fall type is determined. For fall prevention of future falls, it is also important to accurately understand the context that led to the fall event.

[0017] Alternatively, the second user interface interaction mode can be conditional on whether the determined fall event is of a new type. If the fall detection algorithm predicts a previously unseen fall type, more contextual information may be needed to correctly update the fall event label. Therefore, it is beneficial to activate the second user interface interaction mode only when a new (unseen for the elderly) fall type is determined.

[0018] The method may also include receiving input indicating one or more characteristics of the user, and determining a user interface input and / or output modality based on the one or more characteristics of the user. For example, the one or more characteristics of the user may include a health status, and an audio output modality through an audio assistant device or a virtual reality device may be used for users with visual impairments. In another example, the one or more characteristics of the user may include living conditions. A voice input modality with voice recognition may be used for users who live alone, while a keyboard input modality may be used for users who live in shared facilities. Adjusting the user interface input and / or output modality based on user characteristics and preferences enables the user to better communicate with the user interface.

[0019] The method may further include: receiving an input indicating one or more characteristics of the user; determining a time period for switching the user interface to a second user interface interaction mode, the time period being based on the one or more characteristics of the user and / or the user intent score; and switching the user interface to the second user interface interaction mode after the determined time period. The one or more characteristics of the user may include the user's psychological or physiological condition. For example, a user with dementia or memory loss issues may be more likely to forget details about a fall event after a long period of time has passed since the fall event. Therefore, for this user, it may be beneficial to initiate the second user interface mode immediately after the method has determined that the mismatch level is above a threshold. On the other hand, for some users, it may be beneficial to probe them at a later point in time to provide contextual information about the fall event (e.g., when they are less nervous / worried about a possible fall event or may be less excited by being asked about a false positive fall). Therefore, it is beneficial to adjust the time period for switching the user interface to the second user interface interaction mode based on the user's characteristics.

[0020] The method may also include obtaining data indicative of a user's psychological and / or physiological condition and determining the mismatch level based on the user's psychological and / or physiological condition. Certain illnesses (e.g., people with a history of stroke, Parkinson's disease) and injuries have been shown to have a strong correlation with falls. Additionally, people suffering from dementia and / or memory loss issues are more likely to inadvertently inaccurately label fall events. By obtaining data indicative of a user's psychological and / or physiological condition, the mismatch level can be more accurately determined.

[0021] The method may further include: if there is a discrepancy between the label determined by the fall detection algorithm and the updated self-label, determining whether the fall detection algorithm has provided a false positive and / or false negative indication; storing the received signal together with the false positive and / or false negative indication in a training set; and updating the fall detection algorithm based on the training set. The updated self-label provides a more accurate indication of whether a fall has actually occurred and / or an updated, more accurate label of the type of fall. Thus, the fall detection algorithm can be updated to reduce the incidence of false positives and false negatives. This enables the fall detection algorithm to adapt to the fall or activity characteristics of a specific user, thereby improving the overall accuracy of the fall detection algorithm. Retraining can be performed for specific elderly individuals and / or specific room layouts. This retraining can utilize single-shot learning or few-shot learning.

[0022] According to a second aspect, the object is achieved by a controller for determining a signature of a fall event, the controller being configured to:

[0023] - receiving signals from one or more sensors configured to measure signals indicative of characteristics of user movement;

[0024] - analyzing the received signal using a fall detection algorithm to determine a label indicative of a fall event for the user;

[0025] - initiating a first user interface interaction mode of the user interface, wherein in the first user interface interaction mode the user interface is configured to receive a first input from the user indicating a self-tag of a fall event;

[0026] - receiving a first input;

[0027] - determining the level of mismatch between the self-label and the determined label;

[0028] - if the mismatch level is above a threshold, switching the user interface to a second user interface interaction mode, wherein in the second user interface interaction mode the user interface is configured to receive a second input from the user indicating contextual information about the fall event,

[0029] - receiving a second input, and

[0030] - Updating the self-label of the fall event based on the received second input.

[0031] According to a third aspect, the object is achieved by a system for determining a signature of a fall event, the system comprising:

[0032] - one or more sensors configured to measure signals indicative of characteristics of user movement;

[0033] - A controller as described above.

[0034] According to a fourth aspect, the object is achieved by a computer program product for a computing device, the computer program product comprising a computer program code for performing a method for determining a signature of a fall event when the computer program product is run on a processing unit of the computing device.

[0035] It should be understood that the controller, system, and computer program product may have similar and / or the same embodiments and advantages as the lighting device described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and additional objects, features and advantages of the disclosed systems, apparatus and methods will be better understood through the following illustrative and non-limiting detailed description of embodiments of the apparatus and methods with reference to the accompanying drawings, in which:

[0037] Figure 1 An example of a system for determining a signature of a fall event is schematically shown;

[0038] Figure 2 schematically illustrates an example of a user interface in a personal device;

[0039] Figure 3 A method for determining a signature of a fall event is schematically illustrated.

[0040] All the figures are schematic, not necessarily to scale, and generally only show parts which are necessary in order to elucidate the invention, wherein other parts may be omitted or merely suggested. DETAILED DESCRIPTION

[0041] Figure 1An example of a system 100 for determining a signature of a fall event is shown. System 100 includes one or more sensors 102, 104 configured to measure signals 41, 42 indicative of characteristics of a user's movement. The one or more sensors 102, 104 may be, for example, radar sensors, Wi-Fi nodes, infrared (IR) sensors, acoustic sensors, and / or other sensors. In one example, the one or more sensors 102, 104 may be co-located with a lighting device (not depicted). After some processing, the signals 41, 42 from the one or more sensors 102, 104 may form a feature set. Exemplary features may include amplitude, spectral content, directional distribution, mean, variance, etc., but alternatively, the signal itself (i.e., a time series of sample values, such as a time series of values of channel state information (CSI) in a Wi-Fi signal) may be used as the feature set. For example, different motions and positions introduce different multipath distortions in the Wi-Fi signal and produce different patterns in the time series of values of the channel state information (CSI). Thus, the time series values of the channel state information (CSI) from the Wi-Fi nodes (signals 41 , 42 ) can be used to determine the characteristics of the pattern of the user's movement during the fall.

[0042] The system 100 also includes at least one data processor or controller 106. The controller 106 can be configured to receive signals 41, 42 from one or more sensors 102, 104. The controller 106 can connect and communicate with each sensor 102, 104 via a wireless connection, such as a radio frequency or optical communication link. For example, Wi-Fi, ZigBee, BLE, Lo-Ra, UWB, VLC, IR, Li-Fi, etc. The connection can alternatively be wired. Each sensor 102, 104 can include a transmitter (not depicted) for transmitting the corresponding signal 41, 42 or at least a subset of the extracted features to the controller 106 via a wired or wireless connection. The controller 106 can include a receiver (not depicted) for receiving each corresponding signal 41, 42 or feature extracted from the corresponding sensor 102, 104. The system 100 can also include at least one data repository or storage or memory 108 for storing computer program code instructions. The controller 106 can be communicatively coupled to a cloud 120. Alternatively, the system 100 may include a server. Each sensor 102, 104 may transmit its corresponding signal 41, 42 or extracted features to the server (or cloud), so that the server can obtain each corresponding signal 41, 42 or extracted features. The controller 106 may then be configured to retrieve (receive) each corresponding signal 41, 42 or extracted features from the server.

[0043] The controller 106 can be configured to analyze the received signals 41, 42 or the extracted features using a fall detection and / or prevention algorithm to determine a label indicating a fall event. For example, the determined label can indicate whether the received signals 41, 42 or the extracted features include a fall event or do not include a fall event, and can indicate that the received signals 41, 42 include the type of fall event (e.g., "fall with injury," "fall without injury," etc.). Additional fall event types can include "trip and fall" events (i.e., a rapid fall to the ground from a walking position), "fall into a chair" events, "soft fall" events (i.e., a user grabs furniture to slow the fall to the ground), "stroke fall" events (i.e., a fall from a standing position first onto one knee and then onto the ground), and "pick-up fall" events (i.e., a prolonged fall that occurs when a user attempts to pick something up from the ground). A trained fall detection and / or prevention algorithm can make such a determination because the algorithm may have been trained with inputs that may include instances or segments (time series data) of signals 41, 42 received from sensors 102, 104 (or extracted features), and output corresponding labeled instances of falls and / or non-fall incidents (events) and / or types of incidents (fall events). In particular, the fall detection algorithm can determine whether a fall event or a type of fall event has occurred by comparing the signals 41, 42 or a set of extracted features to a set of parameters for classifying whether a fall (or type of fall) has occurred. These parameters may include or be based on a set of features from known falls (types) (e.g., from a training set).

[0044] Figure 2An example of a user interface 230 in a personal device 240 is shown. The controller 106 may also be configured to initiate a first user interface interaction mode of the user interface 230, wherein in the first user interface interaction mode, the user interface 230 is configured to receive a first input 10 from the user 220 indicating a self-label of a fall event. The self-label may indicate whether the fall event occurred or did not occur, and may indicate the type of fall event (e.g., a fall with injury, a fall without injury, etc.). The user 220 may be asked via the user interface 230 in the personal device 240 to provide text and / or voice self-labels of the fall event and / or activities preceding the fall event. In one example, the personal device 240 may include a voice assistant device, and the user interface 230 in the personal voice assistant device 240 may ask the user 220 to provide text and / or voice self-labels of the fall event. In yet another example, the personal device 240 may include a virtual reality or augmented reality device (e.g., a virtual reality headset), and the fall event may be presented to the user 220 via the virtual reality or augmented reality device 240, and the user 220 may be asked to provide a text and / or voice self-label of the fall event. In another example, the user 220 may press a button, such as in the wearable device, to confirm / reject the label of the fall event. The button may be an alarm reset button that is connected to an alarm signal generated when the fall detection algorithm determines a fall. If the user presses the alarm reset button within a predetermined timeout period (which may be zero), the label of the event is non-fall. Otherwise, if the alarm reset signal is not received within the timeout period, the label is fall.

[0045] The controller 106 may be further configured to receive the first input 10. For example, the controller 106 may be connected and communicated with the user interface 230 via a wireless connection, such as a radio frequency or optical communication link. The connection may alternatively be wired. The controller 106 may be included in the same device 240 as the user interface 230. The device 240 may include a transmitter (not depicted) for transmitting the first user input 10 to the controller 106 via a wired or wireless connection. The controller 106 may include a receiver (not depicted) for receiving the first user input 10. Alternatively, the device 240 may transmit the first user input 10 to a server (or cloud 120), and the controller 106 may then be configured to retrieve (receive) the first user input 10 from the server.

[0046] The controller 106 may also be configured to determine a mismatch level between the user's 220 self-label and the label determined by the fall detection and / or prevention algorithm. For example, the controller 106 may apply a weighted average algorithm to the labels (by the user and by the fall detection algorithm) to determine the mismatch level. For example, if the fall detection algorithm predicts a 60% probability of falling, and the user's self-label indicates a non-fall (0% probability of falling), the mismatch level is determined (by the controller 106) to be 30%, assuming that the weights of the fall detection algorithm and the user's self-label are equal. In another example, assuming that the weight of the label predicted by the fall detection algorithm is higher, the determined mismatch level may be determined to be 50%. In another example, the controller 106 may determine the mismatch level by applying a confidence learning machine learning algorithm to the labels. Such confidence-based models for characterizing noisy labels and identifying mismatches between labels associated with the same event are known in the art of supervised learning and will not be discussed further in the context of this application.

[0047] If the mismatch level is above a threshold, controller 106 may be configured to switch user interface 230 to a second user interface interaction mode, wherein in the second user interface interaction mode, user interface 230 is configured to receive second input 20 from user 220 indicating contextual information regarding the fall event. The contextual data provided as second input 20 regarding the fall event may include the user's actions prior to labeling the event as a fall. In another example, the user may be triggered / challenged to revisit their self-labeling of the event. For example, this may be accomplished by stating to the user, "75% of users who were asked to clarify their self-declarations of trips and falls refined their responses after receiving additional information." In another example, the contextual data may include information provided by the user supporting their self-labeling. In another example, the contextual data may include location data of the fall event, such as GPS location data from a sensor device attached to user 220, and contextual location data from user 230, such as whether the fall event was located in the kitchen, bathroom, or living room. In yet another example, the contextual data may include time data, such as time of day data received by a sensor attached to the user 230 and / or time data received from the user 230. The contextual data may also include lighting settings (spectrum and / or intensity) during or before the fall. For example, the user may provide information about whether he or she had turned on (off) a light before the fall event.

[0048] In the second user interface interaction mode, the controller 106 may be configured to select at least one question setting from a set of default question settings and output one or more of the selected question settings to the user 220, for example, via the user interface 230 in the personal device 240. For example, the set of default question settings may include questions such as: "What did you do before the fall event?", "What was the location of the fall event?", "Are you injured?", "What is the date?", "Are you sure this is the correct label?", "What is wrong with your initial answer?", etc. The controller 106 may be configured to select one or more (or all) of the default (predetermined) question settings and output the selected question settings to the user 220. The default question settings may be stored in the memory 108 or the cloud 120.

[0049] Additionally and / or alternatively, the controller 106 can be configured to determine a specific user question setting customized in natural human language, for example, by using a natural language processing algorithm. For example, question settings, such as follow-up questions, can be customized based on a second user input (e.g., contextual information about a fall), historical data about an earlier fall event of the user, details of the user, etc. For example, it can be known (e.g., from a caregiver or from a camera image) that the first dementia patient likes to play with extension cords on the floor, and when bending forward for a long time to reach the cable, he (she) may become dizzy. However, the elderly person already knows that he (she) should not lower himself (herself) to the floor, and therefore, if caught, typically initially denies or even vehemently denies that he (she) has (again) lowered himself (herself) to the floor. Question settings can be customized to situations where the person intentionally lowers himself (herself) to the floor. Methods and techniques for generating text in natural human language are known in the art and will not be discussed in detail in the context of this application.

[0050] The controller 106 may also be configured to determine a question setting based on the mismatch level between the self-label and the determined label, and output the determined question setting to the user. The controller 106 may, for example, be configured to: when the mismatch level between the self-label and the determined label is high (above a threshold), select a more aggressive style of question setting, such as "What's wrong with your original answer?"; and when the mismatch level between the self-label and the determined label is moderate (below a threshold), select a more friendly style of question setting, such as "Are you sure this is the correct label?". The question setting may be selected from a set of default question settings that are classified according to the mismatch level and / or based on a conditional natural language processing algorithm that is conditioned on a question style based on the mismatch level.

[0051] The controller 106 may be configured to update the self-label of the fall event based on the second input 20 received via the user interface 230. For example, the controller 106 may be configured to analyze the received contextual data, e.g., using a natural language processing algorithm (NLP), to determine an updated (more accurate) self-label.

[0052] The second input may include verbal cues and / or non-verbal cues. The controller 106 may also be configured to analyze the verbal cues and / or non-verbal cues in the second input to determine a user intent score and update the self-label of the fall event based on the user intent score. Various machine learning models (ML) and techniques can be used to determine whether the user has deceptive intent based on verbal cues and non-verbal cues in the user's answer that are evidence of deception. For example, speech parameters including non-verbal parameters (e.g., pitch, duration pattern, energy) and language parameters (e.g., pauses filled in multiple audible answers, such as "um" or "ah") can be used as input to a speech ML model to determine the user's intent to deceive or not when generating an audible answer. A natural language processing (NLP) model (e.g., a stylistics model) can be used to determine (classify) whether (part of) the text in the user's text answer is deceptive or non-deceptive based on linguistic cues in the text answer (inconsistencies in the answer provided as the second input) and non-verbal features (such as the number of words, the number of words longer than 6 letters, etc.) (liars may use more simplified forms of language). For example, deceptive language styles are known to include fewer first-person singular pronouns, fewer third-person pronouns, fewer proper nouns, more negative emotion words, and more verbs of motion. In another example, visual features in a user's video answer can be used as input to an ML model (such as a support vector machine and a logistic regression model) to determine the user's intent to deceive or not deceive when generating the video answer. For example, by analyzing micro-expressions and eye movements that indicate deceptive behavior. Such ML models and techniques for analyzing text, voice, or video content to detect deception are known in the art and will not be discussed in detail in the context of this application. The controller 106 can be configured to update the self-label of the fall event based on the user intent score. For example, if the user intent score indicates that the user's intent is not to deceive (e.g., the user intent score for deception is below a threshold, such as 50%), the self-label is updated according to the first user input. The controller 106 can be further configured to store the user intent score together with the updated self-label in a training set and update the fall detection algorithm based on the training set. The training set can be stored in the memory 108 and / or the cloud 120. The stored information can be used, for example, to tune or retrain the algorithm, e.g., adjusting the algorithm's loss function to reflect the user intent scores in the updated training dataset.

[0053] The controller 106 may also be configured to obtain the user's historical (past) user intent scores (e.g., the user may have provided answers with questionable authenticity in the past) and determine the current user intent score based on the user's historical (past) scores. For example, the controller 106 may determine the current user intent score as a weighted average of the user's past (historical) user intent scores and the current user intent score. In one example, the weights may be equal. Alternatively, the controller 106 may determine the current user intent score by assigning a higher weight to the past user intent scores.

[0054] The controller 106 may be further configured to obtain data indicative of a user physiological parameter of the user, and further determine a user intent score based on the user's physiological parameter. For example, the controller 106 may be configured to receive input from one or more sensors monitoring the user's physiological parameter, such as an ECG (electrocardiogram) sensor, a PPG (photoplethysmography) sensor monitoring the user's heart rate, a radar sensor monitoring the user's heart rate and / or respiration rate, etc., during the period in which the user provides the second input 20. This input may be used as input to the ML model to determine whether the user has the intent to deceive. The ML model may make such a determination because it may have been trained to detect deception using known instances of sensor signals associated with deception in a training set.

[0055] The controller 106 can be configured to determine the type of fall event, such as a "fall with injury," "fall into a chair," "soft fall," etc., by analyzing the received signals 41 and 42 or features extracted from the received signals. The controller 106 can be configured to activate the user interface 230 according to the first user interface interaction mode when it is determined that the fall event is of a new type. In other words, the controller 106 can activate the first user interaction mode if the fall detection algorithm determines that the user has not previously experienced a fall type. In another example, the controller 106 can be configured to switch the user interface 230 to the second user interface interaction mode only when the fall detection algorithm determines a new type of fall event.

[0056] Controller 106 may also be configured to receive input indicating one or more characteristics of the user and determine a user interface input and / or output modality (single or multimodal) based on the one or more characteristics of the user. This input may include a user's medical health record indicating the user's physiological and / or psychological condition, the user's living conditions, signals from one or more sensors monitoring the user, and the like. One or more user interface output modalities may include visual (via computer graphics on a screen), audio, vibration, and the like. For example, one or more characteristics of the user may include the user's physiological and / or psychological condition. An audio output modality via an audio assistant device or a virtual reality device may be used for users with visual impairments. In another example, a visual output modality via a screen on a personal device may be used for users with hearing impairments, and the like. In another example, one or more characteristics of the user may include the user's stress state (e.g., based on heart rate monitoring). Compared to an audio assistant device (which may increase the user's stress level), a visual output modality via a screen on a personal device may be used for users under stress. One or more user interface input modalities may include keyboard input, a pointing device, a touch screen, and / or more complex modalities (such as computer vision, speech recognition, motion, orientation, and the like). For example, one or more characteristics of the user may include living conditions. The voice input modality with speech recognition may be used for users who live alone, while the keyboard input modality may be used for users who live in shared facilities.

[0057] The controller 106 may also be configured to determine a time period for switching the user interface 230 to the second user interface interaction mode; and switch the user interface 230 to the second user interface interaction mode after the time period. The time period may be based on one or more characteristics of the user. For example, the controller 106 may determine a shorter time period for a user with a medical condition associated with memory loss. In another example, the controller 106 may determine a later, longer time period for a user currently experiencing a psychological and / or physiological condition associated with stress.

[0058] Figure 3 An example of a method 300 for determining a signature of a fall event is shown, the method comprising the following steps:

[0059] - receiving ( 302 ) signals from one or more sensors 102 , 104 configured to measure signals 41 , 42 indicative of characteristics of the movement of the user 220 ;

[0060] - analyzing (304) the received signal using a fall detection algorithm to determine a signature indicative of a fall event for the user;

[0061] - initiating (306) a first user interface interaction mode of the user interface, wherein in the first user interface interaction mode the user interface is configured to receive a first input from the user indicating a self-label of a fall event;

[0062] - receiving (308) a first input;

[0063] - determining (310) a mismatch level between the self-label and the determined label;

[0064] - if the mismatch level is above a threshold, switching (312) the user interface to a second user interface interaction mode, wherein in the second user interface interaction mode the user interface is configured to receive a second input from the user indicating contextual information about the fall event,

[0065] - receiving (314) a second input, and

[0066] - Updating (316) the self-label of the fall event based on the received second input.

[0067] The method 300 may be performed by computer program code of a computer program product when the computer program product is executed on a processing unit of a computing device, such as the controller 106 .

[0068] In one example, the method 300 may further include the following optional steps: if there is a discrepancy between the label determined by the fall detection algorithm and the updated self-label, determining 318 whether the fall detection algorithm has provided a false positive and / or false negative indication; storing 320 the received signals 41, 42 together with the false positive and / or false negative indication in a training set; and updating 322 the fall detection algorithm based on the training set. During operation of the fall detection and / or prevention algorithm, instances of the received signals 41, 42 or features extracted from the received signals 41, 42 may be stored in the training set in the memory 108 and / or the cloud 120 to update the algorithm. The algorithm may use the (updated) training set to compare the current signal / feature with the signal / feature in the (updated) training set to determine the current label of the event. To improve the training of the algorithm, instances of the signal / feature may be stored together with a value indicating the performance of the algorithm. For example, in the case where the label determined by the fall detection algorithm indicates a fall, but the updated self-label indicates no fall, the received signals 41, 42 or the extracted features can be labeled as indicating a false positive (FP) and stored in the training set with the FP indication. In the case where the label determined by the fall detection algorithm does not indicate a fall, but the updated self-label indicates a fall, the received signals 41, 42 or the extracted features can be labeled as indicating a false negative (FN) and stored in the training set with the FN indication. The signals 41, 42 and the feature set (for which the updated self-label is the same as the label determined by the algorithm) can be stored as TP (true positive) or TN (true negative), respectively. The stored information can be used, for example, to adjust or train the algorithm to reduce the rate of false positives and false negatives.

[0069] It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims.

[0070] In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The use of the verb "comprise" and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements and by means of a suitably programmed computer or processing unit. In a device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0071] Aspects of the present invention can be implemented in a computer program product, which can be a set of computer program instructions stored on a computer-readable storage device that can be executed by a computer. The instructions of the present invention can be any interpretable or executable code mechanism, including but not limited to scripts, interpretable programs, dynamic link libraries (DLLs) or Java classes. The instructions can be provided as complete executable programs, partial executable programs, as modifications (e.g., updates) to existing programs, or as extensions (e.g., plug-ins) to existing programs. In addition, part of the processing of the present invention can be distributed across multiple computers or processors or even on a "cloud."

[0072] Storage media suitable for storing computer program instructions include all forms of non-volatile memory, including but not limited to EPROM, EEPROM and flash memory devices, magnetic disks such as internal and external hard drives, removable disks and CD-ROM disks. The computer program product may be distributed on such storage media or may be provided for download via HTTP, FTP, email or through a server connected to a network (such as the Internet).

Claims

1. A method for determining a signature of a fall event, the method comprising the following steps: - receiving signals from one or more sensors configured to measure signals indicative of characteristics of user movement; - analyzing the received signal using a fall detection algorithm to determine a label indicative of a fall event for the user; - initiating a first user interface interaction mode of the user interface, wherein in the first user interface interaction mode the user interface is configured to receive a first input from the user indicating a self-tag of a fall event; - receiving a first input; - determining the level of mismatch between the self-label and the determined label; - if the mismatch level is above a threshold, switching the user interface to a second user interface interaction mode, wherein in the second user interface interaction mode the user interface is configured to receive a second input from the user indicating contextual information about the fall event, - receiving a second input, and - Updating the self-label of the fall event based on the received second input.

2. The method of claim 1 , wherein the second input comprises a verbal prompt and / or a non-verbal prompt, and wherein the method further comprises: - analyzing verbal cues and / or non-verbal cues to determine a user intent score that indicates the user's intent to deceive, and -Update self-labeling of fall events based on user intent scores.

3. The method according to claim 2, wherein the method further comprises: - receiving further input indicative of a physiological parameter of the user; - analysing the further input to determine a user intent score based on a physiological parameter of the user.

4. The method according to claim 2, wherein the method comprises: - obtain the user's historical user intent score, and -Determine a user intent score based on the user's historical user intent scores.

5. The method according to any one of the preceding claims, wherein the step of receiving a second input indicating contextual information about the fall event comprises receiving information about one of: actions of the user prior to the fall event, supporting evidence about the fall event, time data, location data, presence of other people during the fall event, light settings during and before the fall event.

6. A method according to any one of the preceding claims, wherein in a second user interface interaction mode, the user interface is configured to select a question setting from a set of default question settings and to output the selected question setting to the user.

7. The method according to any of the preceding claims, wherein in the second user interface interaction mode, the user interface is configured to determine a question setting based on a natural language processing algorithm and output the determined question setting to the user.

8. A method according to any of the preceding claims, wherein in a second user interface interaction mode, the user interface is configured to determine a question setting based on a mismatch level between the self-label and the determined label, and output the determined question setting to the user.

9. The method of any one of the preceding claims, wherein determining a label indicative of a fall event comprises determining a type of the fall event, and wherein the first user interface interaction mode is initiated when the determined fall event is of a new type.

10. The method of any preceding claim, wherein determining a label indicative of a fall event comprises determining a type of the fall event, and wherein the second user interface interaction mode is conditional on whether the determined fall event is of a new type.

11. A method according to any preceding claim, wherein the method further comprises receiving input indicative of one or more characteristics of a user and determining a user interface input and / or output modality based on the one or more characteristics of the user.

12. The method according to claim 1 or 2, wherein the method comprises: - receiving input indicating one or more characteristics of a user; - determining a time period for switching the user interface to a second user interface interaction mode, the time period being based on one or more characteristics of the user and / or a user intent score from the self-label; - After the determined time period, switching the user interface to a second user interface interaction mode.

13. A controller for determining a signature of a fall event, the controller being configured to: - receiving signals from one or more sensors configured to measure signals indicative of characteristics of user movement; - analyzing the received signal using a fall detection algorithm to determine a label indicative of a fall event for the user; - initiating a first user interface interaction mode of the user interface, wherein in the first user interface interaction mode the user interface is configured to receive a first input from the user indicating a self-tag of a fall event; - receiving a first input; - determining the level of mismatch between the self-label and the determined label; - if the mismatch level is above a threshold, switching the user interface to a second user interface interaction mode, wherein in the second user interface interaction mode the user interface is configured to receive a second input from the user indicating contextual information about the fall event, - receiving a second input, and - Updating the self-label of the fall event based on the received second input.

14. A system for determining a signature of a fall event, the system comprising: - one or more sensors configured to measure signals indicative of characteristics of user movement; - A controller according to claim 13.

15. A computer program product for a computing device, the computer program product comprising computer program code for performing the method according to claims 1-12 when the computer program product is run on a processing unit of the computing device.

Images