Method for determining the label of a fall event

Abstract

A method for determining a truthful label of a fall event is disclosed. The method includes receiving signals from one or more sensors configured to remotely measure signals indicative of user movement characteristics, analyzing the received signals using a fall detection algorithm to determine a label indicative of a fall event by the user, and initiating a first user interface interaction mode of a user interface, wherein in the first user interface interaction mode, the user interface is configured to receive a first input from the user indicating a self-label of the fall event, receiving the first input, and determining a level of mismatch between the self-label and the determined label. If the level of mismatch exceeds a threshold, the method further includes switching the user interface to a second user interface interaction mode, wherein in the second user interface interaction mode, the user interface is configured to receive a second input from the user indicating contextual information about the fall event, receiving the second input, and updating the self-label of the fall event based on the received second input.

Description

Technical Field

[0001] The present invention relates to a method for determining a label of a fall event. The present invention further relates to a controller for determining a label of a fall event. The present invention further relates to a system for determining a label of a fall event.

Background Art

[0002] Falls are a serious problem in elderly care that can lead to morbidity and mortality in the elderly. Falls can not only cause injuries to the elderly, but also, from a mental aspect, falls often cause fear of falling, which in turn often leads to social isolation and depression. As the aging society progresses further, the development of fall detection and / or prevention systems has become an urgent task. Thanks to the rapid development of sensor networks and the progress of software technology (machine learning algorithms), fall detection systems use various sensors such as accelerometers, radar sensors, time of flight (ToF) sensors, Wi-Fi (registered trademark) nodes, etc. to detect the pattern of signals specific to falls, and thus can determine whether a fall event has occurred.

[0003] However, while current fall detection systems work well under laboratory conditions, they still have problems producing reliable results when applied to real-world conditions. Fall detection algorithms are typically pre-trained on training datasets that mainly consist of laboratory-simulated fall data, using only a small amount of available real-world fall data. To improve the accuracy of fall detection systems, pre-trained fall detection algorithms need to be refined (updated) for specific elderly care facilities and / or specific elderly behaviors to better detect corner cases of fall detection. Retrospective verification (retrospective labeling of fall events) of fall time and type is necessary to refine fall detection algorithms for specific elderly / care facility details and / or details of elderly activity and motor movement. Due to privacy concerns and other practical reasons, labeling of fall events in real-life situations must be done either by the older person (self-labeling) or by one or more caregivers (or staff at a care facility). [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] The inventors have found that older adults tend to underreport or deliberately lie about whether an event resulted in a fall. This may be due to fear of losing their independent living status because the fall was caused by their own actions (e.g., getting up in the middle of the night and going to the toilet alone without asking a caregiver), or due to memory loss. Thus, self-labels provided by older adults to the fall detection system (e.g., via a user interface) may be inaccurate or even deliberately false. Inaccurate labels can significantly impair the accuracy of the fall detection system. Underreporting of fall events, i.e., when older adults deliberately label fall events as no-falls, can lead to an increase in the number of false negatives in the fall detection system, even though the older adults are capable of reporting falls immediately. On the other hand, over-reporting of fall events (i.e., older adults self-labeling non-fall events as falls) can lead to an increase in the number of false positives, potentially resulting in costly and unnecessary actions by caregivers and / or care facilities and / or hospitals associated with older adults at home.

[0005] Therefore, the objective is to provide a method for determining more accurate labels for fall events. [Means for solving the problem]

[0006] According to the first aspect, the objective is achieved by a method for determining a label for a fall event. The method includes: receiving signals from one or more sensors configured to measure signals that characterize a user's movement; analyzing the received signals using a fall detection algorithm to determine a label that indicates a fall event; and initiating a first user interface interaction mode of a user interface, wherein in the first user interface interaction mode, the user interface is configured to receive a first input from the user indicating a self-label for a fall event; receiving the first input; and determining the level of mismatch between the self-label and the determined label. If the level of mismatch exceeds a threshold, the method switches the user interface to a second user interface interaction mode, wherein in the second user interface interaction mode, the user interface is configured to receive a second input from the user indicating contextual information about a fall event; receiving the second input; and updating the self-label for the fall event based on the received second input.

[0007] Signals from one or more (remote) sensors, such as radar sensors, time-of-flight (ToF) sensors, Wi-Fi® Doppler sensors, and microphone sensors, may be used to determine characteristic (patterns) and / or audio patterns of user movement indicating a fall. The patterns may include not only the fall event itself, but also patterns preceding and following the fall. A fall detection and / or prevention algorithm analyzes the received sensor signals to determine a label indicating a fall event associated with the received signal. The fall detection algorithm may be trained to determine whether a fall has occurred based on the received signal. For example, the determined label may indicate whether the received signal contains a fall event or not, and may indicate that the received signal contains a type of fall event, such as an injury fall, an injury-free fall, a soft fall, a stroke fall, or a near-fall (where an elderly person loses balance without falling to the floor). To improve the accuracy of the fall detection algorithm in recognizing and / or classifying specific types of fall events, the user may be asked to self-label fall events. User feedback (self-labeling of fall events) is particularly necessary in remote sensing modalities where the accuracy of determining fall events is limited. In a first user interface interaction mode, the user provides a self-label of a fall event. The self-label may indicate whether or not a fall event occurred, and may indicate the type of fall event, e.g., a fall resulting in injury, a fall without injury, etc.

[0008] If the received signal data is labeled, this may be used to retrain (update) the fall detection algorithm in the future to better recognize the motion and / or audio patterns characteristic of falls (or types of falls). However, self-labels of fall events provided by the user may be intentionally erroneous and / or inaccurate, for example, for the reasons mentioned above. The method includes determining the level of mismatch between the self-label provided by the user and the label of the fall event determined by the fall detection algorithm. If the level of mismatch exceeds a threshold, the method includes switching the user interface interaction mode to a second mode in which the user interface is configured to receive a second input indicating (relevant) contextual information about the fall event. That is, contextual data related to the circumstances of the fall event, which allows for determining the truthfulness of the self-report about the fall event provided by the user and for contextualizing the fall event (providing a broader understanding of the fall event). For example, contextual data about a fall event may include user actions preceding the event labeled as a fall. In another example, the user may be triggered / requested to reconsider their own self-label for an event. In yet another example, contextual data may include user-provided information that supports their own self-label. The user may have initially provided an incorrect and / or inaccurate self-label for a fall event. By being explicitly asked to provide further (contextual) information about the event, the user is triggered (requested) to reconsider and rethink their initial self-labeling and provide an accurate (confident) label for the event. This results in improved labeling, which in turn allows for more efficient updating (retraining) of the fall detection algorithm.

[0009] The second input may include verbal and / or non-verbal cues. The method may further include analyzing the verbal and / or non-verbal cues in the second input to determine a user intent score, the user intent score indicating the user's intent to deceive the fall detection system (about its own label), and updating the self-label of the fall event based on the user intent score. For example, the user intent score may be the probability (likelihood) that the user provided a false self-label. Various machine learning (ML) models and techniques may be used to determine whether or not there is a user's intent to deceive based on verbal and non-verbal cues present in the user's response as evidence of deception. For example, nonverbal parameters (cues) of multiple audible responses, such as pitch, duration pattern, and energy, and speech parameters, such as filled pauses including "um" or "ah," may be used as input to a speech ML model to determine whether the user intends to deceive or not when generating the audible response. Natural language processing (NLP) models, such as stylometry models, may be used to determine (classify) whether (part of) the text in a user's text response is deceptive, based on linguistic (inconsistencies in the response in the second user input) and nonverbal (linguistic) cues in the text response. In another example, visual features in a video response provided by the user in the second input may be used as input to an ML model, such as a support vector machine and a logistic regression model, to determine whether the user intends to deceive or not when generating the video response, for example, by analyzing micro-expressions and eye movements that indicate deceptive behavior. The self-labels provided by the user may be updated accordingly by associating them with a user intent score.For example, if the user intent score indicates that the user's intent is not to deceive, the self-label is updated according to the first user input. Alternatively, if the user intent score indicates that the user's intent is to deceive, the self-label is updated according to the contextual information. This further improves labeling, thereby enabling more efficient updating (retraining) of the fall detection algorithm. The method may further include storing the user intent score along with the updated self-label in a training set and updating the fall detection algorithm based on the training set. Updating the fall detection algorithm to account for uncertainties associated with the user self-label can improve the accuracy and robustness of the fall detection algorithm, in particular, to fine-tune the fall detection / fall prevention system to the unique quirky behavior and specific room setup of older adults.

[0010] The method may further include receiving additional input indicating the user's physiological parameters during the period in which the user provides a second input, and analyzing the additional input to determine a user intent score based on the user's physiological parameters. When a person lies, their physiological responses during the response may reflect stress responses that may arise from lying. For example, a person lying may be more agitated and show increased heart rate and respiratory rate, or may sweat more (which results in a change in skin conductivity). By analyzing the user's physiological responses (parameters) during the period in which the user provides a second input, a better estimation of the user's intent to deceive can be achieved.

[0011] The method may further include obtaining the user's historical (past) user intent score and determining the (current) user intent score based on the historical user intent score. Individuals who have been found to intentionally mislabel previous fall events may be more likely to provide inaccurate current labels for fall events. Thus, by considering the historical user intent score, the current user intent score can be determined more accurately.

[0012] The step of receiving a second input that provides contextual information about a fall event includes receiving information about at least one of the following: user behavior preceding the fall event, user corroborating evidence regarding the fall event, location data of the fall event, time data of the fall event, presence of other people (e.g., a nurse) at the time of the fall event, and lighting settings (intensity and light spectrum) at the time of and before the fall event. For example, people can fall under various conditions for various reasons, but people are more likely to fall when walking or going up / down stairs. The majority of falls are caused by improper sit-to-stand transfers. Similarly, older adults are more likely to fall when getting up due to temporary muscle weakness and balance problems. Thus, user behavior preceding a fall can be a good indicator of a fall event. The time and place of the fall contain important information that contextualizes the fall event. For example, the majority of falls among older adults occur when going to the toilet at night. The presence of another person at the time of a fall may indicate that the older adult was not attempting to walk to the toilet alone, and therefore less likely that a tripping and falling event occurred. Lighting (brightness level) at the location of the fall may also contribute to the fall event. Similarly, the lighting (light intensity and spectrum) to which the older adult was exposed on the day / hour prior to the fall may also contribute to the fall event. Studies suggest that users exposed to circadian lighting (lighting settings designed to promote circadian health) have a 40% reduction in fall rates. Requiring users to provide supporting evidence regarding falls may trigger them to reconsider their self-labeled statements or expose inconsistent responses indicating untrue self-reporting by the user. Therefore, receiving contextual information (data) related to fall events allows for a broader understanding of fall events and triggers older adults to confirm / deny their initial self-labeling of fall events. Furthermore, contextual information (data) can expose inconsistent responses regarding fall sensing data.

[0013] In the second user interface interaction mode, the user interface may be configured to select a question from a group of default question settings and to output the selected question settings to the user. For example, the group of default question settings may include questions such as "What did you do before the fall event?", "What is the location of the fall event?", and "Are you injured?". Outputting the selected question settings to the user can facilitate the user providing contextual information about the fall event.

[0014] Additionally, and / or alternatively, the user interface may be configured to determine question settings based on natural language processing (NLP) algorithms and output the determined question settings to the user. A powerful new class of large-scale language models is enabling machines to generate text in natural human language. These large-scale language models can generate deductive (non-existent) follow-up questions for older adults in natural human language.

[0015] The user interface may be configured to determine the question setting based on the level of mismatch between the self-label and the determined label, and to output the determined question setting to the user. For example, follow-up questions (settings) may be customized based on the level of mismatch between the self-label and the label determined (by the fall detection algorithm). The level of mismatch between the self-label and the label determined by the algorithm may indicate the user's intention to deceive or not. The wording of the question style (e.g., friendly rather than confrontation) influences how people respond to the question. A harsh, confrontational question may lead to user frustration if the user's initial input was true. However, if the user's initial input was deceptive, such a question may encourage the user to provide an accurate self-label for the fall event. Thus, by optimizing the question style (selecting the question setting) based on the level of mismatch between the self-label and the determined label, a more accurate self-label can be determined.

[0016] Determining a label for a fall event may include determining the type of fall event, and the first user interface interaction mode may be initiated when the determined fall event (the fall itself, as well as the activities preceding and following the fall) is a new type that has not been previously observed. Labeling fall events significantly improves the accuracy of fall detection. However, older adults may become frustrated if they are constantly asked to provide self-labels for fall events. If the fall type is common for the user (e.g., a near-fall without injury), it may not be necessary to probe the user for self-labeling. However, if a fall detection algorithm predicts a fall type that has not been previously observed, the algorithm may have low confidence in such predictions. Therefore, it is beneficial to initiate the first user interface interaction mode only when a new type of fall (not previously seen in this older adult) is determined. It is also important to accurately understand the context leading to the fall event in order to prevent future falls.

[0017] Alternatively, the second user interface interaction mode may be triggered by whether the determined fall event is of a new type. If a previously unseen fall type is predicted by the fall detection algorithm, more contextual information may be needed to correctly update the fall event label. Therefore, it is beneficial to initiate the second user interface interaction mode only when a new type of fall (not previously seen in this elderly person) is determined.

[0018] The method may further include receiving input representing one or more of the user's characteristics and determining user interface input and / or output modalities based on one or more of the user's characteristics. For example, one or more of the user's characteristics may include health status, and an audio output modality via an audio assistant device or virtual reality device may be used for a user with a visual impairment. In another example, one or more of the user's characteristics may include living status. A voice input modality with speech recognition may be used for a user living alone, and a keyboard input modality may be used for a user living in a shared facility. Adjusting user interface input and / or output modalities based on user characteristics and preferences enables better user engagement with the user interface.

[0019] The method may further include receiving input that represents one or more of the user's characteristics, determining a period for switching the user interface to a second user interface interaction mode, the period being based on one or more of the user's characteristics and / or a user intent score, and switching the user interface to the second user interface interaction mode after the determined period. One or more of the user's characteristics may include the user's psychological or physiological state. For example, a user with dementia or memory problems may be more likely to forget details about a fall event after a long period of time has passed since the fall event. Thus, it may be beneficial for such a user to initiate the second user interface mode immediately after the method determines that the level of mismatch exceeds a threshold. On the other hand, for some users (for example, if the user is not very stressed / worried about a possible fall event, or is unlikely to be very upset by being questioned about a false positive fall), it may be beneficial to prompt them to provide contextual information about the fall event at a later point in time. Thus, it is beneficial to adapt the time frame for switching the user interface to a second user interface interaction mode based on the user's characteristics.

[0020] The method may further include obtaining data indicating the user's psychological and / or physiological state, and determining the level of mismatch depending on the user's psychological and / or physiological state. Certain illnesses (e.g., Parkinson's disease, people with a history of stroke) and injuries have been shown to be strongly associated with falls. Furthermore, people suffering from dementia and / or memory problems are more likely to inadvertently and inaccurately label fall events. By obtaining data indicating the user's psychological and / or physiological state, the level of mismatch can be determined more accurately.

[0021] The method may further include determining whether the fall detection algorithm provided a false positive and / or false negative indication if there is a difference between the label determined by the fall detection algorithm and the updated self-label, storing the received signals along with the false positive and / or false negative indication in a training set, and updating the fall detection algorithm based on the training set. The updated self-label provides a more accurate indication of whether a fall actually occurred and / or an updated, more accurate labeling of the type of fall. Thus, the fall detection algorithm can be updated to reduce the incidence of false positives and false negatives. This allows the fall detection algorithm to be adapted to the characteristics of a particular user's falls or activities, thereby improving the overall accuracy of the fall detection algorithm. Retraining may be performed for a specific elderly person and / or a specific room layout. Retraining may utilize single-shot or multi-shot learning.

[0022] According to a second aspect, the objective is a controller for determining the label of a fall event, the controller being - Receives signals from one or more sensors configured to measure signals that represent the characteristics of the user's movements. - Analyze the received signal using a fall detection algorithm to determine a label indicating a user-induced fall event. - The user interface is started in a first user interface interaction mode, and in the first user interface interaction mode, the user interface is configured to receive a first input from the user indicating a self-label of a fall event. - Receives the first input, - Determine the level of mismatch between the self-labeled label and the determined label. - If the level of mismatch exceeds the threshold, switch the user interface to a second user interface interaction mode, and in the second user interface interaction mode, the user interface is configured to receive a second input from the user indicating context information regarding the fall event. - Receive the second input, and - Update the self-label of the fall event based on the received second input. This is achieved by a controller configured as described above.

[0023] According to a third aspect, the object is a system for determining a label of a fall event, the system comprising: - One or more sensors configured to measure a signal indicating a characteristic of the user's movement, - The controller described above, This is achieved by a system including the above.

[0024] According to a fourth aspect, the object is a computer program for a computing device, the computer program including computer program code for executing a method for determining a label of a fall event when the computer program is executed by a processing unit of the computing device. This is achieved by the computer program.

[0025] It should be understood that the controller, the system and the computer program may have the same and / or identical embodiments and advantages as the method described above.

Brief Description of the Drawings

[0026] The above and additional purposes, features and advantages of the disclosed systems, devices and methods will be better understood through the following exemplary and non-limiting detailed description of embodiments of the devices and methods with reference to the accompanying drawings. All figures are schematic and not necessarily to scale, and generally only the parts necessary to illustrate the invention are shown, with other parts omitted or merely suggested. [Figure 1] An example of a system for determining the label of a fall event is schematically shown. [Figure 2] A schematic example of a user interface in a personal device is shown below. [Figure 3] This outlines a method for determining the label of a fall event. [Modes for carrying out the invention]

[0027] Figure 1 shows an example of a system 100 for determining the label of a fall event. System 100 includes one or more sensors 102, 104 configured to measure signals 41, 42 that characterize the user's movement. One or more sensors 102, 104 may be, for example, radar sensors, Wi-Fi® nodes, infrared (IR) sensors, acoustic sensors, and / or other sensors. In one example, one or more sensors 102, 104 may be co-located with a lighting device (not shown). The signals 41, 42 from one or more sensors 102, 104 may, after some processing, form a feature set. Exemplary features may include magnitude, spectral components, directional distribution, mean, variance, etc., but alternatively, the signal itself, i.e., a time series of sampled values, for example, a time series of channel state information (CSI) in a Wi-Fi® signal, can function as a feature set. For example, different motions and positions result in different multipath distortions in the WiFi® signal, generating different patterns in the time-series values ​​of the Channel State Information (CSI). Thus, the time-series values ​​of the Channel State Information (CSI) from the Wi-Fi node (signals 41, 42) may be used to determine patterns characteristic of the user's movements during a fall.

[0028] The system 100 further includes at least one data processor or controller 106. The controller 106 may be configured to receive signals 41, 42 from one or more sensors 102, 104. The controller 106 may connect to and communicate with each sensor 102, 104 via a wireless connection, for example, via a radio frequency or optical communication link. Examples of such connections include Wi-Fi®, ZigBee®, BLE, Lo-Ra, UWB, VLC, IR, Li-Fi, etc. The connection may also be wired. Each sensor 102, 104 may include a transmitter (not shown) for transmitting at least a subset of their respective signals 41, 42, or extracted features, to the controller 106 via a wired or wireless connection. The controller 106 may include a receiver (not shown) for receiving their respective signals 41, 42, or extracted features from each sensor 102, 104. System 100 may further include at least one data repository or storage or memory 108 for storing computer program code instructions. Controller 106 may be communicatively coupled to Cloud 120. Alternatively, System 100 may also include a server. Each sensor 102, 104 may transmit its respective signal 41, 42, or extracted feature to the server (or cloud) so that the server may obtain its respective signal 41, 42, or extracted feature. In this case, Controller 106 may be configured to acquire (receive) its respective signal 41, 42, or extracted feature from the server.

[0029] The controller 106 may be configured to analyze the received signals 41, 42, or extracted features using a fall detection and / or prevention algorithm to determine a label indicating a fall event. For example, the determined label may indicate whether the received signals 41, 42, or extracted features include a fall event or not, and may indicate that the received signals 41, 42 include a type of fall event, such as "fall with an injury" or "fall without an injury". Further types of fall events may include “trip and fall” events (i.e., rapid falls from a walking position to the ground), “fall entering a chair” events, “soft fall” events (i.e., a user grabbing furniture to slow down a fall to the ground), “brain stroke fall” events (i.e., falling from a standing position first to one knee, then to the ground), and “pick-from-ground fall” events (i.e., a prolonged fall that occurs when a user attempts to pick something up from the ground). A trained fall detection and / or prevention algorithm may make such a determination because it has already been trained on inputs including instances or segments (time-series data) (or extracted features) of signals 41, 42 received from sensors 102, 104, and is capable of outputting corresponding labeled instances of fall and / or non-fall incidents (events) and / or incident (fall event) types. In particular, a fall detection algorithm may determine whether a fall event or a type of fall event has occurred by comparing the signal 41, 42, or the extracted feature set, with a set of parameters used to classify whether a fall (or type of fall) has occurred. These parameters may include, or be based on, a set of features from known falls (types), for example, from a training set.

[0030] Figure 2 shows an example of a user interface 230 in a personal device 240. The controller 106 is configured to initiate a first user interface interaction mode of the user interface 230, in which the user interface 230 is configured to receive a first input 10 from user 220 indicating a self-label of a fall event, which may indicate whether or not a fall event occurred, and may indicate the type of fall event, e.g., an injury-inducing fall, an injury-free fall, etc. User 220 may be asked to provide text and / or voice self-labels of the fall event and / or the activity preceding the fall event via the user interface 230 in the personal device 240. In one example, the personal device 240 may include a voice assistant device, and user 220 may be asked to provide text and / or voice self-labels of the fall event via the user interface 230 in the personal voice assistant device 240. In yet another example, the personal device 240 may include a virtual or augmented reality device, such as a virtual reality headset, and the user 220 may be presented with a fall event via the virtual or augmented reality device 240 and asked to provide a text and / or voice self-label of the fall event. In yet another example, the user 220 may affirm / negate the label of the fall event by pressing a button, for example, on a wearable device. The button may be an alarm reset button that is associated with an alarm signal generated when a fall detection algorithm determines that a fall has occurred. If the user presses the alarm reset button within a predetermined timeout period (which may be zero), the label of the event is non-fall. Otherwise, if the alarm reset signal is not received within the timeout period, the label is fall.

[0031] The controller 106 may further be configured to receive the first input 10. For example, the controller 106 may connect to and communicate with the user sensor 230 via a wireless connection, for example, via a radio frequency or optical communication link. The connection may alternatively be wired. The controller 106 may be included in the same device 240 as the user interface 230. The device 240 may include a transmitter (not shown) for sending the first user input 10 to the controller 106 via a wired or wireless connection. The controller 106 may include a receiver (not shown) for receiving the first user input 10. Alternatively, the device 240 may transmit the first user input 10 to a server (or cloud 120), and the controller 106 may then be configured to retrieve (receive) the first user input 10 from the server.

[0032] The controller 106 may further be configured to determine the level of mismatch between the user's self-label and the label determined by the fall detection and / or fall prevention algorithm. For example, the controller 106 may determine the level of mismatch by applying a weighted average algorithm to the labels (by the user and by the fall detection algorithm). For example, if the fall detection algorithm predicts a 60% probability of falling and the user's self-label indicates no fall (0% probability of falling), the level of mismatch is determined to be 30% (by the controller 106), assuming that the weights of the fall detection algorithm and the user's self-label are equal. In another example, the determined level of mismatch may be determined to be 50%, assuming that the label predicted by the fall detection algorithm has a higher weight. In yet another example, the controller 106 may determine the level of mismatch by applying a confidence learning machine learning algorithm to the labels. Such confidence-based models for characterizing noisy labels and identifying mismatches between labels associated with the same event are known in the field of supervised learning and will not be discussed further in the context of this application.

[0033] If the level of mismatch exceeds a threshold, the controller 106 is configured to switch the user interface 230 to a second user interface interaction mode, in which the user interface 230 may be configured to receive a second input 20 from user 220 that provides contextual information about a fall event. The contextual data provided as the second input 20 about the fall event may include user actions preceding the event labeled as a fall. In another example, the user may be triggered / requested to reconsider their own self-label of the event. This may be done, for example, by telling the user, "75% of the user's asked to clarify a trip and fall self-declaration refined their answer after receiving additional information." In a further example, the contextual data may include user information that supports their own self-label. In another example, context data may include location data of the fall event, e.g., GPS location data from a sensor device attached to user 220, and context location data from user 230, e.g., that the location of the fall event was the kitchen, bathroom, living room, etc. In yet another example, context data may include time data, e.g., time data received by a sensor attached to user 230, and / or time data received from user 230. Furthermore, context data may include lighting settings (spectrum and / or intensity) during or before the fall. For example, the user may provide information on whether the lights were on (off) before the fall event.

[0034] In the second user interface interaction mode, the controller 106 may be configured to select at least one question setting from a group of default question settings and output one or more selected question settings to the user 220, for example, via the user interface 230 on the personal device 240. For example, the group of default question settings may include questions such as "What did you do before the fall event?", "What is the location of the fall event?", "Are you injured?", "What is the date?", "Are you sure that this is the correct label?", and "What problems are there with your initial answer?". The controller 106 may be configured to select one or more (or all) of the default (predetermined) question settings and output the selected question settings to the user 220. The default question settings may be stored in memory 108 or the cloud 120.

[0035] Additionally, and / or alternatively, the controller 106 may be configured to determine customized question settings for a particular user in natural human language, for example, by using a natural language processing algorithm. For example, question settings, such as follow-up questions, may be customized based on a second user input (e.g., contextual information about falls), historical data about the user's previous fall events, user specifics, etc. For example, it may be known (e.g., from a caregiver or from camera footage) that a first dementia patient likes to play with extension cords on the floor and may become dizzy if they bend over for a long time to reach the cables. However, the elderly already know that it is wrong to self-lower and therefore will often initially, and even vehemently, deny it if they are found to have self-lowered. The question setting may be customized for such cases of intentional self-lowering. Methods and techniques for generating text in natural human language are known in the art and will not be detailed in the context of this application.

[0036] Controller 106 may further be configured to determine a question setting based on the level of mismatch between the self-label and the determined label, and to output the determined question setting to the user. Controller 106 may be configured to select, for example, a more aggressive style of question setting, such as "What problems are there with your initial answer?", if the level of mismatch between the self-label and the determined label is high (above a threshold), and a more friendly style of question setting, such as "Are you sure that this is the correct label?", if the level of mismatch between the self-label and the determined label is moderate (below a threshold). The question setting may be selected from a group of default question settings categorized according to the level of mismatch, and / or based on a conditional natural language processing algorithm in which the style of the question is conditioned based on the level of mismatch.

[0037] The controller 106 may be configured to update the self-label of the fall event based on a second input 20 received via the user interface 230. For example, the controller 106 may be configured to analyze the received contextual data, for example, using a natural language processing algorithm (NLP), to determine the updated (more accurate) self-label.

[0038] The second input may include verbal and / or nonverbal cues. The controller 106 may further be configured to analyze the verbal and / or nonverbal cues in the second input to determine a user intent score and to update the self-label of the fall event based on the user intent score. Various machine learning models (MLs) and techniques may be used to determine whether the user has an intent to deceive based on verbal and nonverbal cues present in the user's response as evidence of deception. For example, nonverbal parameters of multiple audible responses, such as pitch, duration pattern, and energy, and speech parameters, such as voiced pauses like "um" or "ah," may be used as input to a speech ML model to determine whether the user has an intent to deceive or not when generating the audible response. Natural language processing (NLP) models, such as stylometry models, may be used to determine (classify) whether (part of) the text in a user's text response is deceptive or not, based on linguistic cues (inconsistencies in the response provided as a second input) and non-verbal features such as word count and the number of words greater than six letters (people lying may use more simplified forms of language). For example, deceptive language styles are known to have fewer first-person singular pronouns, fewer third-person pronouns, fewer exclusive words, more negative sentiment words, and more motor verbs. In another example, visual features in a user's video response may be used as input to ML models, such as support vector machines and logistic regression models, to determine the user's intention to deceive or not when generating the video response. For example, by analyzing micro-expressions and eye movements that indicate deceptive behavior. Such ML models and techniques for analyzing text, voice, or video content to detect deception are known in the art and will not be detailed in the context of this application. The controller 106 may be configured to update the self-label of fall events based on the user intent score.For example, if the user intent score indicates that the user's intent is not to deceive, e.g., the user intent score for attempting to deceive falls below a threshold, e.g., 50%, the self-label is updated according to the first user input. The controller 106 may further be configured to store the user intent score along with the updated self-label in a training set and to update the fall detection algorithm based on the training set. The training set may be stored in memory 108 and / or cloud 120. The stored information may be used, for example, to adapt or retrain the algorithm, e.g., to adjust the algorithm's loss function to reflect the user intent scores in the updated training dataset.

[0039] The controller 106 may further be configured to obtain the user's historical (past) user intent score (for example, the user may have previously provided answers of questionable truthfulness) and to determine the current user intent score based on the user's historical (past) score. For example, the controller 106 may determine the current user intent score as a weighted average of the user's past (past) user intent score and the current user intent score. In one example, the weights may be equal. Alternatively, the controller 106 may determine the current user intent score by assigning a higher weight to the past user intent score.

[0040] The controller 106 may further be configured to obtain data indicating the user's physiological parameters and to determine a user intent score based on the user's physiological parameters. For example, the controller 106 may be configured to receive input from one or more sensors that monitor the user's physiological parameters during the period in which the user provides a second input 20, such as an ECG (Electrocardiography) sensor, a PPG (Photoplethysmography) sensor that monitors the user's heart rate, a radar sensor that monitors the user's heart rate and / or respiratory rate, etc. This input may be used as input to an ML model to determine whether or not the user has an intent to deceive. Such a determination may be made by the ML model being trained to detect deception using known instances of sensor signals associated with deception in the training set.

[0041] The controller 106 may be configured to determine the type of fall event, such as "fall resulting in injury," "fall into a chair," or "soft fall," by analyzing the received signals 41, 42, or features extracted from the received signals. The controller 106 may be configured to start the user interface 230 according to a first user interface interaction mode when it is determined that the fall event is of a new type. That is, the controller 106 may start the first user interaction mode on the condition that the fall detection algorithm has determined that a type of fall not previously seen for this user has occurred. In another example, the controller 106 may be configured to switch the user interface 230 to a second user interface interaction mode only when a new type of fall event has been determined by the fall detection algorithm.

[0042] The controller 106 may further be configured to receive inputs indicating one or more characteristics of the user and to determine user interface input and / or output modalities (unimodal or multimodal) based on one or more characteristics of the user. Inputs may include the user's medical health record indicating the user's physiological and / or psychological state, the user's living conditions, signals from one or more sensors monitoring the user, etc. One or more user interface output modalities may include visual (computer graphics via a screen), audio, vibration, etc. For example, one or more characteristics of the user may include the user's physiological and / or psychological state. An audio output modality via an audio assistant device or a virtual reality device may be used for a visually impaired user. In another example, a visual output modality via a screen on a personal device may be used for a hearing impaired user, etc. In yet another example, one or more characteristics of the user may include the user's stress level (e.g., based on heart rate monitoring). A visual output modality via a screen on a personal device may be used for a stressed user compared to an audio assistant device (which may contribute to an increased stress level for the user). One or more user interface input modalities may include keyboard input, pointing devices, touchscreens, and / or more complex modalities such as computer vision, speech recognition, motion, and orientation. For example, one or more user characteristics may include living conditions. A voice input modality with speech recognition may be used for users living alone, and a keyboard input modality may be used for users living in shared facilities.

[0043] The controller 106 may further be configured to determine a period of time for switching the user interface 230 to a second user interface interaction mode and to switch the user interface 230 to the second user interface interaction mode after the period. The period may be based on one or more characteristics of the user. For example, the controller 106 may determine a shorter period for a user who has a medical condition related to memory loss. In a further example, the controller 106 may determine a slower, longer period for a user who is currently experiencing a psychological and / or physiological condition related to stress.

[0044] Figure 3 shows a method 300 for determining the label of a fall event, the method being - Step (302) of receiving signals from one or more sensors 102, 104 configured to measure signals 41, 42 that represent the characteristics of the user 220's movements, - Step (304) of analyzing the received signal using a fall detection algorithm to determine a label indicating a fall event caused by the user, - Step (306) of starting a first user interface interaction mode of the user interface, wherein in the first user interface interaction mode, the user interface is configured to receive a first input from the user indicating a self-label of a fall event, - Step (308) of receiving the first input, - A step (310) to determine the level of mismatch between the self-label and the determined label, - Step (312) of switching the user interface to a second user interface interaction mode if the mismatch level exceeds a threshold, wherein in the second user interface interaction mode, the user interface is configured to receive a second input from the user indicating contextual information about a fall event. - Step (314) to receive a second input, - A step (316) to update the self-label of the fall event based on the second input received, This shows an example of a method that includes [this].

[0045] Method 300 may also be executed by the computer program code of the computer program if the computer program is executed on a processing unit of a computing device, such as the controller 106.

[0046] In one example, method 300 may further include an optional step 318 of determining whether the fall detection algorithm provided false positive and / or false negative indications if there is a difference between the label determined by the fall detection algorithm and the updated self-label; an optional step 320 of storing the received signals 41, 42 along with the false positive and / or false negative indications in a training set; and an optional step 322 of updating the fall detection algorithm based on the training set. During the operation of the fall detection and / or prevention algorithm, instances of the received signals 41, 42, or features extracted from the received signals 41, 42, may be stored in a training set in memory 108 and / or cloud 120 to update the algorithm. The algorithm may use the (updated) training set to determine the current label of an event by comparing the current signal / feature with those in the (updated) training set. To improve the training of the algorithm, instances of the signal / feature may be stored along with value indications of the algorithm's performance. For example, if the label determined by the fall detection algorithm indicates a fall, but the updated self-label indicates no fall, the received signals 41, 42 or extracted features may be labeled to represent false positive (FP) and stored in the training set along with the FP indication. If the label determined by the fall detection algorithm does not indicate a fall, but the updated self-label indicates a fall, the received signals 41, 42 or extracted features may be labeled to represent false negative (FN) and stored in the training set along with the FN indication. Signals 41, 42 and feature sets whose updated self-labels are the same as the labels determined by the algorithm may be stored as either TP (true positive) or TN (true negative), respectively.The stored information may be used, for example, to adapt or train an algorithm to reduce the rate of false positives and false negatives.

[0047] It should be noted that the embodiments described above are illustrative and not limiting to the present invention, and that those skilled in the art can design many alternative embodiments without departing from the scope of the appended claims.

[0048] In the claims, no reference numerals in parentheses should be construed as limiting the claim. The use of the verb “including” and its conjugations does not preclude the existence of elements or steps other than those described in the claims. The singular notation of an element does not preclude the existence of multiple such elements. The present invention may be implemented by hardware comprising several individual elements, and by a suitably programmed computer or processing unit. In a device claim listing several means, some of these means may be embodied by items of the same hardware. The mere fact that certain means are listed in different dependent claims does not imply that combinations of these means cannot be used advantageously.

[0049] Aspects of the present invention may be implemented in a computer program product, which may be a collection of computer program instructions stored in a computer-readable storage device that can be executed by a computer. The instructions of the present invention may include, but are not limited to, scripts, interpretable programs, dynamic link libraries (DLLs), or Java classes, and may be any interpretable or executable code mechanism. The instructions may be provided as a complete executable program, a partial executable program, a modification (e.g., an update) to an existing program, or an extension (e.g., a plug-in) to an existing program. Furthermore, some of the processing of the present invention may be distributed across multiple computers or processors, or across a "cloud."

[0050] Suitable storage media for storing computer program instructions include, but are not limited to, all forms of non-volatile memory, such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal and external hard disk drives, removable disks, and CD-ROM disks. Computer programs may be distributed on such storage media, or they may be made available for download via servers connected to a network such as HTTP, FTP, email, or the Internet.

Claims

1. A method for determining the label of a fall event, wherein the method is The steps include receiving signals from one or more sensors configured to measure signals that represent the characteristics of the user's movements, The steps include analyzing the received signal using a fall detection algorithm to determine a label indicating a fall event by the user, A step of initiating a first user interface interaction mode of a user interface, wherein in the first user interface interaction mode, the user interface is configured to receive a first input from the user indicating a self-label of the fall event; The step of receiving the first input, A step of determining the level of mismatch between the self-label and the determined label, If the level of the mismatch exceeds a threshold, the user interface is switched to a second user interface interaction mode, wherein in the second user interface interaction mode, the user interface is configured to receive a second input from the user indicating contextual information relating to the fall event. The step of receiving the second input, A step of updating the self-label of the fall event based on the second input received, Methods that include...

2. The second input includes verbal and / or non-verbal cues, and the method is Analyzing the verbal and / or nonverbal cues to determine a user intent score, wherein the user intent score indicates the user's intent to deceive, Updating the self-label of the fall event based on the user intent score, The method according to claim 1, including the method described in claim 1.

3. This method is The system receives further input indicating the physiological parameters of the user, Analyzing the further inputs to determine the user intent score based on the user's physiological parameters, The method according to claim 2, including the method described in claim 2.

4. This method is Obtaining the user's historical user intent score, Determining the user intent score based on the user's historical user intent score, The method according to claim 2, including the method described in claim 2.

5. The method according to claim 1, wherein the step of receiving a second input indicating contextual information relating to the fall event includes receiving information relating to user behavior prior to the fall event, supporting evidence relating to the fall event, time data, location data, the presence of further people at the time of the fall event, and one of the lighting settings at the time of the fall event and before the fall event.

6. The method according to claim 1, wherein in the second user interface interaction mode, the user interface is configured to select a question setting from a group of default question settings and to output the selected question setting to the user.

7. The method according to claim 1, wherein in the second user interface interaction mode, the user interface is configured to determine a question setting based on a natural language processing algorithm and to output the determined question setting to the user.

8. The method according to claim 1, wherein in the second user interface interaction mode, the user interface is configured to determine a question setting based on the level of mismatch between the self-label and the determined label, and to output the determined question setting to the user.

9. The method according to claim 1, wherein determining a label indicating the fall event includes determining the type of fall event, and the first user interface interaction mode is initiated if the determined fall event is of a new type.

10. The method according to claim 1, wherein determining a label indicating the fall event includes determining the type of fall event, and the second user interface interaction mode is conditional on whether the determined fall event is of a new type.

11. The method according to claim 1, comprising receiving an input representing one or more characteristics of the user, and determining user interface input and / or output modalities based on the one or more characteristics of the user.

12. This method is The system receives input that represents one or more characteristics of the user, Determining a period for switching the user interface to the second user interface interaction mode, wherein the period is based on the user intent score of one or more of the user's characteristics and / or the self-label. After the determined period, the user interface is switched to the second user interface interaction mode. The method according to claim 1 or 2, including the method described in claim 1 or 2.

13. A controller for determining the label of a fall event, wherein the controller It receives signals from one or more sensors configured to measure signals that represent the characteristics of the user's movements. The received signal is analyzed using a fall detection algorithm to determine a label indicating a fall event caused by the user. The user interface is started in a first user interface interaction mode, and in the first user interface interaction mode, the user interface is configured to receive a first input from the user indicating the self-label of the fall event. Receiving the first input, Determine the level of mismatch between the self-label and the determined label. If the level of the mismatch exceeds a threshold, the user interface is switched to a second user interface interaction mode, and in the second user interface interaction mode, the user interface is configured to receive a second input from the user indicating contextual information regarding the fall event. Receiving the second input, Based on the second input received, update the self-label of the fall event. A controller configured in such a way.

14. A system for determining the label of a fall event, the system is One or more sensors configured to measure signals that represent the characteristics of the user's movements, The controller according to claim 13, A system that includes this.

15. A computer program for a computing device, which, when the computer program is executed on a processing unit of the computing device, includes computer program code for performing the method described in claim 1.

Images