Sensor and system for monitoring

Abstract

A method and system for observing the presence, location, movement and / or posture of a person in a monitored area is provided. The system comprises: at least one sensor (101, 201, 301, 401, 601); and means for processing a measurement signal of the sensor, such as measurement electronics; and means for transmitting the measurements and / or data relating to said measurements for further processing. The system further comprises means (105, 205, 405, 605) for generating and capturing audio, and the system is configured to: determine, using the at least one sensor (101, 201, 301, 401, 601), that the security of the monitored person (206, 402, 602) is threatened; and based on the determined that the security of the monitored person (206, 402, 602) is threatened, initiating a voice conversation with the monitored person (206, 402, 602) using a means for generating audio, and listening to a reply from the monitored person (206, 402, 602) using a means for capturing audio. Based on the captured audio, the system is configured to evaluate a situation and determine whether an action, such as an alarm, needs to be taken, where the system is configured to make the voice conversation with the monitored person using a large language model (LLM).

Description

Technical Field

[0001] The present invention relates to a method and a system by which people in a monitored area can be observed, tracked and / or their condition and safety can be monitored. Background Art

[0002] Monitoring the condition of the elderly in the home environment is essential if one wishes to prolong the possibility that the aging population can cope properly in their home environment. Safety bracelet systems are widely used today for these types of applications. Their disadvantage is that the user must wear the bracelet continuously and must be able to press the alarm button in an emergency. There are also bracelets that check the user's health status, but they have the same problems as described above, and in addition, there is the further problem of false alarms.

[0003] Solutions have also been proposed to install a membrane made of piezoelectric material on the floor, wherein the membrane registers the pressure changes caused by movements on the surface of the floor. It is also known in the prior art to use sensors to be installed on or under the floor, which detect the presence and movement of people without requiring changes in pressure, but function with the help of capacitive sensors.

[0004] The prior art also proposes the possibility of using cameras, motion detectors based on, for example, infrared detection, or ultrasonic sensors to monitor the condition and status of the elderly. For example, WO2012164169 discloses a method and system for tracking an object based on ultrasonic technology.

[0005] Some prior art solutions using millimeter wave (MMW) radar to track people are known.

[0006] A disadvantage of prior art observation and monitoring systems is that they often interpret a person as needing help even when no external help is needed. In these cases, for example, it may be determined that a person has fallen and an alarm related to a fall may be issued even though the person has not fallen or is able to stand up without any help. These false alarms are costly to the company responsible for monitoring the safety of the person or to the monitored person himself, because the monitored person must be contacted (e.g., by a professional caregiver) to check whether the person needs help. For these reasons, there is a need for a reliable monitoring system that minimizes the number of false alarms.

[0007] Brief Description of the Invention

[0008] The problems of the state of the art are eliminated by using a system according to claim 1 and a method according to claim 9. The invention is characterized by what is disclosed in the claims.

[0009] The present invention relates to a system for observing the presence, location, movement and / or posture of people in a monitored area. The system comprises: at least one sensor; and means for processing measurement signals of the sensor, such as measurement electronics; and means for transmitting measurement results and / or data related to the measurement results for further processing. The system further comprises means for generating and capturing audio. The system is configured to: determine using the at least one sensor that the safety of the monitored person is threatened, based on the determined threat to the safety of the monitored person, initiate a voice conversation with the monitored person using the means for generating audio, and listen to a reply from the monitored person using the means for capturing audio, and assess the situation and / or determine whether action such as an alarm is required based on the captured audio.

[0010] In one embodiment of the present invention, the system is configured to use a Large Language Model (LLM) to conduct a voice conversation with the monitored person.

[0011] In one embodiment of the present invention, the system is configured to provide the large language model with activation prompts that are contextually relevant to the monitored person and / or monitored area.

[0012] In one embodiment of the present invention, the context-relevant prompt includes at least one of the following: information about the role and task of the large language model (LLM) in the speech conversation; information about the person being monitored, such as name, physical condition, and level of assistance required; the current situation as determined using the at least one sensor; conditions that require an alarm to be issued; instructions on how the LLM should interact with the rest of the system.

[0013] In one embodiment of the invention, the means for generating audio comprises a text-to-speech algorithm for converting text output from the system into audible speech for a person being monitored; the means for capturing audio comprises a speech recognition algorithm for converting audible speech from a person being monitored into text input for the system; and wherein the text output of the system is connected to the input of a large language model (LLM), and the text input of the system is connected to the output of a large language model (LLM).

[0014] In one embodiment of the invention, the system comprises at least one of the following one or more sensors: a radar sensor, a floor sensor, a motion detector and / or a camera.

[0015] In one embodiment of the invention, the means for generating audio is a speaker and / or the means for capturing audio is a microphone or a microphone array.

[0016] In one embodiment of the present invention, the system is configured to determine that the safety of the monitored person is threatened when the system has determined that the monitored person has: fallen; is lying on the floor; has spent too much time in certain parts of the monitored area, such as the bathroom or bed; has not eaten or exercised, and / or has not visited certain parts of the monitored area, such as not visiting a balcony during winter.

[0017] In one embodiment of the present invention, the voice conversation initiation includes questions or suggestions to the monitored person.

[0018] In one embodiment of the present invention, the system is configured to process the captured audio using a speech recognition algorithm to convert the captured audio into a text response.

[0019] In one embodiment of the present invention, the system is configured to compare the text response to a set of keywords to perform a situation assessment.

[0020] In one embodiment of the invention, the action includes sending a notification or alert containing a description of the situation and / or the content of the conversation.

[0021] The present invention also relates to a method for observing the presence, location, movement and / or posture of a person in a monitored area using a system, the system comprising: at least one sensor; and means for processing the measurement signal of the sensor, such as measurement electronics; means for transmitting the measurement results and / or data related to the measurement results for further processing; and means for generating and capturing audio. The method comprises: using the at least one sensor to determine that the safety of the monitored person is threatened, based on the determined safety of the monitored person, using the means for generating audio to initiate a voice conversation with the monitored person, and using the means for capturing audio to listen to a reply from the monitored person, and based on the captured audio, assessing the situation and determining whether action such as an alarm is required.

[0022] In one embodiment of the present invention, a large language model (LLM) is used to conduct a voice conversation with the monitored person.

[0023] In one embodiment of the present invention, a start prompt related to the context of the monitored person and / or the monitored area is provided to the large language model.

[0024] In one embodiment of the present invention, the prompt of contextual information includes at least one of the following items: information about the role and task of the large language model (LLM) in the speech conversation; information about the person being monitored, such as name, physical condition, and level of help required; the current situation as determined using the at least one sensor; conditions that require an alarm to be issued; instructions on how the LLM should interact with the rest of the system.

[0025] In one embodiment of the invention, the means for generating audio comprises a text-to-speech algorithm for converting text output from the system into audible speech for a person being monitored; the means for capturing audio comprises a speech recognition algorithm for converting audible speech from a person being monitored into text input for the system; and wherein the text output of the system is connected to the input of a large language model (LLM), and the text input of the system is connected to the output of a large language model (LLM).

[0026] With the above solution of the present invention, a monitoring system is provided which is able to provide reliable measurement results in different types of situations and is easy to install and maintain. One advantage is that the system can reliably observe the safety of people in the monitored area while minimizing the number of false alarms when asking people whether they need help through an automated voice dialogue.

[0027] Various other advantages will become apparent to those skilled in the art based on the following detailed description.

[0028] The terms “first,” “second,” and “third” are used herein to distinguish one element from another and do not particularly prioritize or rank them if not explicitly stated otherwise.

[0029] The exemplary embodiments of the invention presented herein should not be interpreted as limiting the applicability of the appended claims. The verb "to include" is used herein as an open limitation that does not exclude the presence of features that have not been enumerated. Unless explicitly stated otherwise, the features described in the dependent claims can be freely combined with each other.

[0030] The novel features which are believed to be characteristic of the invention are set forth with particularity in the appended claims. The invention itself, however, both as to its construction and its method of operation, together with further objects and advantages thereof, will be best understood from the following description of specific embodiments when read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present invention is illustrated by the following drawings, in which:

[0032] Figure 1 Components of an exemplary embodiment of the system of the present invention are presented in an area to be monitored,

[0033] Figure 2 The operation of an example embodiment of the system of the present invention is presented.

[0034] Figure 3 An exemplary embodiment of a sensor according to the solution of the invention is presented,

[0035] Figure 4 An exemplary embodiment of a system according to the invention is presented.

[0036] Figure 5 An example of a processing pipeline of a system according to one embodiment of the present invention is presented, and

[0037] Figure 6 An exemplary embodiment of a system according to the present invention is presented. DETAILED DESCRIPTION

[0038] In the solution of the present invention, the sensor can detect the presence and movement of an object in a monitored area. The monitored object can be, for example, an elderly person or some other person who would benefit from supervision. The sensor can be installed in an area to be monitored that is accessible to the object. In one embodiment of the present invention, the sensor can also be used to observe the vital functions of the monitored person, such as the person's breathing (e.g., breathing rate) and / or heart rate.

[0039] In the solution according to the invention, the system comprises at least one sensor and may further comprise measuring electronics that generate sensor observations by means of the sensor and a processor configured to process the sensor observations and / or a central unit comprising a memory, the central unit being, for example, a data processing device. For the purpose of this functionality, the central unit of the system may comprise the necessary software and information about the characteristic properties of the signal being detected. Typically, the measuring electronics and / or the central unit may infer information from the signal received via the sensor. The system may have a central unit that may manage one or more sensors or sensor groups. In one embodiment of the invention, a sensor group comprises, for example, sensors in the same space, such as in the same room.

[0040] The area to be monitored by the sensors may be the entire area where people are usually located or only a portion of an area. The area to be monitored may include, for example, one or more rooms, and some parts of the area, such as fixed facilities (such as cabinets), may remain outside the area to be monitored. In one embodiment in which a sleeping person is monitored, the sensors may be arranged in connection with the bed, above the bed and / or beside the bed, so that the monitoring area of ​​at least one sensor covers at least a portion of the bed or a person lying on the bed.

[0041] In the solution according to the invention, the sensor detects people in the monitored area and measures and detects the position, speed and / or shape of the monitored people. In one embodiment of the invention, the sensor is configured to observe objects based on signal strength and / or by filtering out possible erroneous measurements.

[0042] The sensor may include at least one of the following: a radar sensor, a floor sensor, a motion detector and / or a camera. In one embodiment, the system includes at least two sensors and is configured to detect and measure people in the monitored area based on measurement signals of the at least two sensors, which may monitor the same area and / or different parts of the monitored area. For example, the measurement areas of the sensors may overlap, for example, at a certain part of the area.

[0043] The components of the system may be integrated into a single unit, for example into a sensor assembly comprising components of the system. In one embodiment of the invention, the sensor assembly may include at least some of the following components in a single unit: a sensor; means for processing a measurement signal of the sensor, such as measurement electronics; means for transmitting the measurement results and / or data related to the measurement results for further processing; and means for generating and capturing audio.

[0044] The sensor can be mounted on a bracket, on a surface (e.g., a wall, door, floor or ceiling) and / or near a surface (such as, for example, a floor surface, a wall surface, a door surface or a ceiling surface of an area to be monitored that an apartment and / or an object can enter). In one embodiment of the invention, one or more sensors are installed in a corner of the space to be monitored, directly below the ceiling, and tilted toward the center of the space. A typical tilt angle can be, for example, 15 degrees, which enables the sensor to clearly observe over obstacles such as furniture. In one embodiment of the invention, one or more sensors are mounted on a wall or in a corner of the space to be monitored, typically above the horizontal plane of the floor, for example at a height of about 40 cm to 150 cm from the floor. The field of view of the sensor can be, for example, about 90 degrees on a horizontal plane.

[0045] The system further comprises at least one device for generating audio and at least one device for capturing audio. The device for generating audio may be, for example, a speaker, and / or the device for capturing audio may be, for example, a microphone or a microphone array.

[0046] In one embodiment of the present invention, the system may utilize common home audio equipment (such as Bluetooth speakers or wireless speakers) as a means for capturing and producing audio.

[0047] In one embodiment of the present invention, the system is configured to use a VoIP speakerphone as a means for generating and capturing audio to allow audio conversations to be conducted over the Internet.

[0048] The system may use the at least one sensor to determine that the safety of the monitored person is threatened. Based on the determination that the safety of the monitored person is threatened, the system may use the means for generating audio to initiate a voice conversation with the monitored person, and use the means for capturing audio to listen to a response from the monitored person. Based on the captured audio, the system may assess the situation and determine whether action is required, such as sending a notification or issuing or sending an alarm.

[0049] In one embodiment, sending an alert or notification includes sending an alert or message to a person and / or organization monitoring the health of the person, for example, as a message to a phone, to a nurse, to a relative, or to an emergency center. In one embodiment, the voice conversation initiation includes questions or suggestions to the monitored person, for example, to ask if the person is ok and / or needs help. In one embodiment of the invention, an alert or notification is sent if no response is received from the monitored person because the system cannot confirm that the person does not need help.

[0050] In one embodiment of the invention, the system may process the captured audio using a speech recognition algorithm to convert the audio into a text response. In one embodiment of the invention, the system may compare the text response to a set of keywords to make a situation assessment. A notification or alert may be sent containing a description of the situation and / or the content of the conversation.

[0051] In one embodiment of the invention, the system is configured to submit the captured audio to an external server or service, such as a cloud-based speech recognition engine, for speech-to-text translation.

[0052] In one embodiment of the invention, the system is configured to use an external (eg, cloud-based) text-to-speech service to generate the audio prompts.

[0053] In one embodiment of the invention, the sensor or system is configured to perform sentiment analysis on the captured audio and / or text responses for situation assessment. In one embodiment of the invention, the system is configured to perform sentiment analysis on the text responses using an external server or service, such as a cloud-based sentiment analysis service.

[0054] In one embodiment of the invention, a means for generating audio is used with a text-to-speech algorithm to convert text output from the system into speech audible to the person being monitored.

[0055] In one embodiment of the invention, a device for capturing audio is used together with a speech recognition algorithm to convert audible speech from the person being monitored into text input to the system.

[0056] In one embodiment of the invention, text output and input are connected to the input and output of a large language model (LLM). The large language model can be, for example, an artificial neural network trained to generate natural language based on a given context. The large language model (LLM) can be trained using self-supervised learning and / or semi-supervised learning (e.g., containing tens of millions to billions of weights). The large language model can work by inputting text and repeatedly predicting the next symbol or word.

[0057] In one embodiment of the present invention, a large language model (LLM) is used to conduct a voice conversation with a monitored person. The large language model (LLM) may be provided with special prompts to provide context, such as the context of the monitored person and / or the monitored space, to initiate a conversation. In one embodiment of the present invention, a conversation may be started by describing the current situation of the large language model (LLM) (e.g., based on observations from the at least one sensor).

[0058] In one embodiment of the invention, prompts direct the Large Language Model (LLM) to play the role of a virtual nurse or assistant whose responsibility is to check on the condition of the person being monitored when the system determines that the health of the person being monitored may be threatened. The prompts may include additional context, such as information about the condition of the person being monitored, the level of help required, and / or the conditions under which an alarm needs to be sounded. In addition, the prompts instruct the Large Language Model (LLM) to generate its responses in a structured manner, which responses can be used to drive or control the rest of the system. An illustration of the required syntax can be given as a few examples of the interaction.

[0059] In one embodiment of the present invention, the conditions under which an alarm needs to be issued include the following: the system has detected a fall, and when the large language model (LLM) asks the person being monitored whether he or she needs help, the person answers affirmatively or does not answer at all. If the system has detected a fall and the person being monitored answers that he or she does not need help, an alarm is not necessary and will not be issued.

[0060] In one embodiment of the invention, the structured response from the Large Language Model (LLM) includes commands to the rest of the system. Such commands could be, for example, "SAY <text>”, which triggers the text-to-speech system based on the command output from the large language model and the <text>" corresponding voice audio; and / or "ALARM", which triggers the sending of an alarm. These commands (presented above and below) are only examples of structured responses, and other predefined responses or commands and / or response protocols may be used, such as JSON messages or function calls.

[0061] In one embodiment of the invention, the system may determine that the safety of the monitored person is threatened when the system has determined that the monitored person: has fallen; is lying on the floor; has spent too much time in certain parts of the monitored area, such as the bathroom or bed; has not eaten or exercised, and / or has not visited certain parts of the monitored area, such as not visiting the balcony during winter. In one embodiment of the invention, the system may determine that the safety of the monitored person is threatened when the vital functions of the monitored person, such as the tracked heartbeat and / or the breathing of the monitored person, are not within predefined limits.

[0062] In one embodiment of the present invention, a sensor and / or a sensor system may be used by a user to determine fixed objects (such as a bed or a sofa) where a person may lie down, and the sensor does not determine that a person is lying in the area of ​​these fixed objects. In one embodiment of the present invention, the sensor may distinguish an object from an observed person by a determined elevation angle of the observed object, for example in such a way that when the determined elevation angle of the object is substantially always below a certain threshold elevation angle value, the object may be identified as not being a person.

[0063] In some applications, it is advantageous to first map unchanged areas, i.e. to map the measurement information of the sensors when mainly stationary and non-moving objects and structures are in place. This type of situation is, for example, in a residential apartment when the furniture is in place but no people, pets or robots are in the apartment. This mapped information can be recorded in the system, for example in a memory located in a central unit or in a memory device connected via a data network, wherein the memory device can be, for example, in a control center or a service center. For this purpose, the memory device can be integrated into the sensor or the system, so the memory device can be in the central unit or connected to the central unit via a data network.

[0064] According to one embodiment of the invention, the system maps unchanged areas continuously or at defined intervals, in which case the system is able to detect changes in areas caused, for example, by new furniture or by changes in the position of furniture. In this way, the system is able to gradually adapt to changes occurring in the area to be monitored.

[0065] Figure 1 Components of an example embodiment of a system in an area to be monitored are presented. The sensor 101 or multiple sensors to be used in the present invention are arranged to be connected to the area to be monitored in such a way that the area to be monitored can be monitored by using the sensor 101 or multiple sensors. The sensor can be mounted on top of a surface (e.g., a wall, floor or ceiling surface) and fastened to the surface, for example, with double-sided tape or with a sticker, in which case the sensor can be easily removed. The sensor 101 can be connected wirelessly or by wire to a gateway 104, which collects measurements obtained from the sensor 101 or status information formed by the sensor 101, such as detected objects, the health status of objects (such as people) and / or the movement and posture of objects.

[0066] The system may include at least one device 105 for generating audio (e.g., a speaker) and / or at least one device for capturing audio (e.g., a speaker microphone or a microphone array). The device for generating audio and / or the device for capturing audio may be integrated into the sensor 101, and / or the device for generating audio and / or the device for capturing audio may be separate units.

[0067] The gateway 104 can send information onward, for example to a control center or to another organization that oversees an area and / or objects (such as people) in the area. The transfer of information between the system and a certain recipient can be performed, for example, using a telephone connection, a wired broadband connection, or a wireless connection. It is advantageous to consider issues related to data security and privacy in data transfer, and many official regulations also address this issue.

[0068] In one embodiment of the invention, the sensor 101 or sensors comprise their own central unit and the central unit of the sensors is connected to the gateway 104. In a second embodiment of the invention, the sensor 101 or the central unit of the sensors is integrated into the gateway 104.

[0069] Some functions of the central unit or gateway 104 may be performed elsewhere (eg, in a central control room or service center, such as a server or cloud service) via a data network connection.

[0070] The system according to the present invention may also include a call button 102. After pressing the call button, the system may be connected to, for example, a caregiver, a security guard, or it may execute various alarm procedures. The call button may be wireless, and the call button may be adapted to function without batteries.

[0071] The notification and alarm procedures of the system according to the invention may include, for example, the activation of a local alarm, the transmission of an indication signal (such as a buzzer, light, siren, alarm clock, etc.), contacting an alarm center or service center, a care provider or a relative. In some cases, an alarm may also be sent directly to the person being monitored or to the user (e.g., by means of speech synthesis or speech recording). In order to perform these tasks, the sensor or system may include means for processing time data (such as, for example, a clock circuit).

[0072] According to one embodiment of the invention, in addition to the voice conversation, an alarm signal may be issued by the system in the space being monitored for a predetermined period of time. This alarm signal may be issued, for example, as a local alarm before sending an alarm or notification, and it may be issued via a light alarm unit and / or an audio alarm unit of the system. The light alarm unit and / or the audio alarm unit may be in each different part of the residence (e.g., a room). This functionality may also be integrated into the sensors, for example, into all sensors or only some sensors.

[0073] The system according to the invention may also include a fire detector 103, which may be connected to another system via a wired or wireless connection. If the fire detector 103 issues a fire alarm, an alarm procedure may be executed, for example by sending an alarm message to a control center or rescue authorities.

[0074] Figure 2 The operation of an embodiment of the system according to the invention is presented, wherein the health status or posture of a person 206 in a monitored area is monitored.

[0075] If the system's sensor 201 detects that the safety of the monitored person 206 is threatened, the system can initiate a voice conversation with the monitored person using a device for generating audio, and listen to a response from the monitored person 206 using a device for capturing audio. Based on the captured audio, the system can assess the situation and determine whether action is required, such as sending a notification or issuing or sending an alarm. Figure 2 In the example of , the means for generating audio and the means for capturing audio may be integrated into the sensor 201 .

[0076] In one embodiment of the invention, the system examines information measured by multiple sensors (e.g., by all sensors in an area being monitored) and sends a notification (e.g., a remote alarm) and / or initiates a voice conversation only if the sensors do not detect other people in the area.

[0077] exist Figure 2 In the case of the embodiment presented (where the system sends a message based on a voice conversation, based on a person falling down and / or due to a determined vital function of the person), the sensor 201 sends information about the situation to the gateway 204 of the system, and the gateway 204 sends the information and / or alarm onward to the server 201, for example via an Internet connection or via some other connection. The information and / or alarm from the server 207 is sent to an organization that monitors the health of the person, for example as a message to a mobile phone 202, as an alarm and / or, for example, to a nurse 203, a relative or an emergency center. In this way, for example, information about the person reaches the necessary person or organization, and the person who falls down gets help as soon as possible. In one embodiment of the invention, the system can send information directly from the gateway 204 to the organization or person that monitors the health of the monitored person. In this example, the monitored person can indicate in a voice conversation with the system received by the device 205 for capturing audio that she needs help, or that no response is received from the monitored person after initiating the voice conversation.

[0078] The processor, central unit and / or measuring electronics used in the solution of the invention may be integrated into the sensor, or they may be arranged separately or in a separate unit. In an embodiment of the invention, using software executed by a processor, the sensor or system may interpret the movement observed with at least one sensor and may issue an alarm if the alarm conditions defined for the program are met.

[0079] In one embodiment of the invention, only some of the sensors in the area to be monitored have a function that enables the emission of an alarm signal as described above. For example, only the sensors in some rooms (such as in a living room) may be provided with this function, and the sensors in other rooms send a notification forward immediately after detecting a fall and / or the measurement result of the monitored person is not within an acceptable and / or predefined range. In one embodiment of the invention, only some of the sensors in a space (such as in a room) include a function that enables the emission of an alarm signal as described above.

[0080] The system may also include a control center, and predetermined information about the presence, position, movement and / or posture of the object may be sent to the control center. The alarm terminology used by the system may be changed, for example, based on the presence information (which may be received, for example, from an RFID reader).

[0081] The system may also have a memory device, wherein the system is adapted to record the measurement signals or information derived from the measurement signals for observing the time dependency of the area being monitored and the behavior of people. By means of this, for example, if the person being monitored does not get up or go to the kitchen within a certain time, or if the person goes to the toilet too often, or if the vital functions of the observed person (such as breathing or heartbeat) have changed during a certain time, the system can issue an alarm or initiate a voice dialogue. The memory device also makes it possible to learn more common daily rhythms and detect anomalies occurring therein.

[0082] In one embodiment of the invention, the sensor and / or system may include a radio-based identification device for identifying a person. The radio-based identification device may be, for example, a device based on Bluetooth, Bluetooth Low Energy (BLE), or Zigbee. In this embodiment, the system may identify a person and a radio-based device carried by the person, such as a bracelet, watch, mobile device, tag, and the measurement results may be linked to a specific identified person. In this way, the system is able to know who is present in the monitored area and to whom the measurement results relate.

[0083] In one embodiment of the invention, the radio-based identification means may include an antenna array that enables more accurate association of an identification device with its carrier when more than one person and device are present.

[0084] In one embodiment of the invention, if the identification device detects a person (such as a nurse) in the monitored area, the alarm may be automatically disabled.

[0085] In one embodiment of the present invention, the alarm condition of the system may include the identity of the person. For example, when an unauthorized person enters a certain location, an alarm may be triggered.

[0086] In one embodiment of the invention, a radio-based identification device (e.g., a Bluetooth, Bluetooth Low Energy (BLE) or Zigbee-based device) may be used to locate a person or assist in locating a person. The sensor may include several antennas for the radio-based identification device, such as Bluetooth, BLE or Zigbee antennas for implementing direction finding technology (e.g., Zigbee, Bluetooth or Bluetooth Low Energy (BLE) direction finding technology), for example, according to the Bluetooth 5.1 specification. In one embodiment of the invention, if the radar of the sensor detects movement, but the radio-based identification device does not detect a remotely readable tag or device (such as a Bluetooth, BLE or Zigbee tag or device), the person detected by the radar may be considered a visitor. On the other hand, if the radar detects a remotely readable tag or device (such as a Bluetooth, BLE or Zigbee tag or device), the detected person may be identified and actions may be taken based on the identified person. In an example embodiment, when a resident is in a room and there is also an assisting person, the status of the person or room may be set in the system to "there is an assisting person in the room". In the same manner, an alarm made by a resident may also be acknowledged when the system recognizes that a person belonging to a non-resident of the room has entered the room. In this case, the alarm may be acknowledged automatically. In one embodiment, the alarm is not acknowledged automatically, but requires an active, identifiable event (e.g., from a user device).

[0087] In one embodiment of the invention, the identification of the detected person can be done by other devices, such as surveillance cameras, which are arranged in the corridor, for example. In this case, the radar-based sensor detects that someone is entering the room, and the system can check the information from the surveillance camera (for example, from a certain point in time from the surveillance record), where it can be seen that someone has entered the room. In one embodiment, this record can be linked to the room as an entry event, and if necessary, the entrant can be identified later by viewing the record. In this case, the identification can be automatic, but if it is not preferred, it is not necessary to implement automatic identification. If automatic identification from video is used, it can be implemented, for example, based on facial recognition technology. In one embodiment, facial recognition or video-based identification is not used if the user can be identified by other means. In one embodiment, video-based identification is only used when the person cannot be identified in any other way.

[0088] In one embodiment of the invention, in which a radio-based identification means is used, the necessary electronics and antenna may be integrated with the sensor. Figure 3 An example of an embodiment is presented in which a Bluetooth antenna array is integrated with the sensor 301. Figure 3 The Bluetooth antenna array includes four antennas 302 and the required electronics for controlling the operation of the identification device and the antennas. The antenna array can be used to measure and detect Bluetooth devices and tags and, for example, to locate people carrying Bluetooth devices (such as bracelets) using Bluetooth 5.1 direction finding technology. In one embodiment, for example, the data measured by the Bluetooth antenna array is combined with the data measured by the radar by the sensor to increase the position and positioning accuracy of the radar sensor. In this embodiment, an antenna or antenna array of the radar 303 is arranged at the center of the sensor and inside the area formed by the four Bluetooth antennas 302.

[0089] In one embodiment of the invention, the sensor according to the invention can be used, for example, in a ward or in a room where people are sleeping and need monitoring of the sensor. In this embodiment, the sensor can be arranged so that the sensor can measure and sense the person appearing on the bed. The sensor can be arranged in the room or connected to the room so that the monitored area of ​​one sensor covers at least a portion of a bed. In one embodiment of the invention, the sensor is arranged on the ceiling of the room, for example above each bed, for example one sensor above each bed. In one embodiment of the invention, the sensor is arranged on the wall of the room, for example next to each bed, for example one sensor next to each bed. With these embodiments, the sensor can measure and / or sense the presence of a person in bed and the vital functions of the person, such as movement, heartbeat and breathing. One of the advantages of these embodiments is that it can be monitored without disturbing a sleeping person, which is impossible with a wired sensor, for example. For example, in a hospital environment using this embodiment, it is also easy for staff and nurses to monitor people who should be asleep. With these embodiments, the sensor does not have to include its device for detecting the orientation of the sensor.

[0090] In one embodiment of the invention, at least one additional sensor according to the invention may be arranged in a monitored room or area where people are sleeping. This additional sensor is capable of sensing and monitoring people who have left their beds. In this case, the measurement area of ​​the additional sensor may be larger than the measurement area of ​​the sensor monitoring the bed. The measurement area may cover substantially the entire room (e.g., using a single or multiple additional sensors). This additional sensor may also be arranged in the room or connected to the room so that the measurement area of ​​one or more sensors covers the room and in particular covers the area outside the bed. In one embodiment, the additional sensors may be arranged on the ceiling, walls and / or corners of the room or on a bracket. With this embodiment, the room may be better monitored by staff, for example, if people are leaving their beds and / or disturbing other people who are trying to fall asleep, an alarm may be sounded. These additional sensors also enable monitoring of people who have left their beds, and, for example, if a person falls and / or if a determined vital function is not at a predefined and / or acceptable level, an alarm is generated. With these embodiments, the additional sensors do not have to include a device for detecting the orientation of the sensor.

[0091] In one embodiment of the invention, the sensor may comprise, for example, a millimeter wave (MMW) radar, which may, for example, operate under the MIMO radar principle. In an example embodiment, there may be, for example, three transmitter antennas and four receiver antennas. In this example, this forms a virtual antenna with 12 elements. With the sensor of the invention, the elevation, azimuth, movement and distance of an object may be observed very accurately. For example, FMCW (Frequency Modulated Continuous Wave) technology may be used for the radar.

[0092] The sensor may measure and detect movement of the monitored person, such as breathing rate, position, speed and / or shape. In one embodiment of the invention, the sensor may determine a state of the person, such as breaks or interruptions in the breathing of the monitored person, for example, to identify sleep apnea and / or restricted activity of the person. In one embodiment of the invention, the determined state of the person may include snoring of the person.

[0093] In one embodiment of the present invention, the system can track the precise location of a person using an FMCW radar operating in the millimeter wave band with an antenna array. The system can include a user interface or another configuration interface that can be used to specify the location of a bed, sofa, and other locations of interest, and save these locations in a room configuration data store. The CPU can receive the location of the person from the radar and negotiate with the room configuration to determine whether the occupant is in bed or another place for resting. When this happens, the CPU can send instructions to the radar to focus on the location of interest and begin monitoring fine movements. This focusing can be accomplished, for example, by using beamforming with an antenna array to amplify signals originating from the direction of interest. Large movements can be detected by observing changes in the range or Doppler spectrum of the signal. Small movements can be detected by observing changes in the phase angle of the signal.

[0094] In one embodiment of the invention, the radar-based sensor is configured to track the movement of the observed person in a first operating mode by analyzing the signals reflected from the person (e.g., Doppler frequency, range and angle of arrival of the signal). In one embodiment of the invention, the radar-based sensor is configured to track the heartbeat and / or breathing of the monitored person in a second operating mode by analyzing the phase of the measurement signal. In one embodiment of the invention, the scanning time of the sensor is longer in the second operating mode than in the first operating mode. In one embodiment of the invention, the second operating mode can be distinguished from the first operating mode only by the digital signal processing algorithm applied to the signal.

[0095] In one embodiment of the invention, in the second operating mode, tracking of a person is not performed. In one embodiment of the invention, one radar-based sensor may use the first operating mode and the second operating mode simultaneously, such that the first operating mode is always used, and the second operating mode is enabled when the second operating mode is needed and disabled when it is not needed. In one embodiment of the invention, one sensor may use the first operating mode and the second operating mode in an interleaved manner.

[0096] The radar-based sensor can be configured to enable the second operating mode based on detecting that the monitored person is not moving, has fallen, and / or the monitored person's speed is slower than a predefined threshold. The sensor can be configured to deactivate the second operating mode based on detecting that the monitored person is not determined to have fallen, the person is moving, and / or the monitored person's speed is higher than a predefined threshold.

[0097] In the second operating mode, the radar-based sensor may be configured to analyze the measurement signal in such a way that the phase of the measurement signal is determined in order to observe the movement of the person, such as heartbeat and / or breathing. In one embodiment of the invention, in the second operating mode, the radar-based sensor and / or the measurement electronics of the sensor are configured to analyze the measurement signal from the area and / or a certain distance around the area, which measurement signal is related to the determined azimuth, elevation and / or distance of the sensor from the person determined in the first operating mode.

[0098] In an embodiment of the invention, in an apartment or nursing home, there may be at least one sensor in each room. In this case, if corrective measures are not taken, the sensors (e.g., radars) will interfere with each other. In one embodiment, it is proposed to divide the mode and / or several sensors (e.g., radars) into specific time slots so that several sensors can be used simultaneously close to each other without causing interference. The transmission of the sensors can be synchronized and performed in a staggered manner, for example, in such a way that the sensors can observe the same person and / or the same room.

[0099] In one embodiment of the invention, different sensors may be in different operating modes, for example, some sensors determine stationary objects when the second operating mode is enabled, while other sensors use only the first operating mode to monitor movement of objects and search for stationary objects.

[0100] In one embodiment of the invention, the sensor or system is configured to detect a person falling and / or sitting down by a determined elevation angle of the person, such as such that a person can be determined to have fallen down when the elevation angle of the person is below a certain threshold elevation angle value. In one embodiment of the invention, the elevation angle of the person is tracked and filtered with a filter such as a Kalman filter or a low pass filter to prevent false alarms due to noisy measurements.

[0101] In the following, an example embodiment is described. In this example embodiment, the sensor is a radar-based sensor that includes two modes of operation. The first mode of operation of the sensor is used to track the presence and movement of people (e.g., in a single room). In this embodiment, tracking can be performed using measured point cloud data. The required Doppler range is given by

[0102]

[0103] In one example, if a person is moving at 1 m / s, the required Doppler range is +-400 Hz, and the maximum measurement interval at 60 GHz is 2.5 ms. Inhalation lasts about 2 seconds. If the corresponding movement is 5 mm, the required Doppler range is +-1 Hz, and the scan time is one second.

[0104] When the system observes that the person has stopped, the system may enable a second mode of operation in which the system is able to track the person's vital functions, such as heartbeat and / or breathing, such as breathing interruptions and / or frequency. In one embodiment of the invention, the system needs to perform measurements for a certain duration before it is able to detect a person's breathing cycle.

[0105] After determining the vital functions of the person, the system may deactivate the second mode of operation. In an example embodiment, the system may periodically determine the vital functions of the same person, for example, as long as the person remains stationary. If the system observes stationary persons, the system begins to determine the vital functions of these persons by using the second mode of operation.

[0106] In an example embodiment of the invention, operation in the second mode of operation may be implemented, for example such that when a stationary object has been detected, point cloud data around the area of ​​the detected object is saved and analyzed. The saved packets may be generated periodically (e.g. every 600ms). In an embodiment of the invention, the data may be transferred to a central control unit for analysis. Using the analysis of the signal (i.e. the point cloud data), information about smaller movements of the object may be observed, and thus the system is able to determine, for example, the breathing activity and / or the heartbeat of a person.

[0107] In one embodiment of the invention, the scanning time of the sensor is longer in the second operating mode, and due to this, a better signal-to-noise ratio can be achieved. In addition, more TX antennas can be utilized because more time is available for measurement. In this way, the angular resolution can be improved. To improve the range resolution, the frequency scanning range can be increased.

[0108] The Doppler frequency can be determined, for example, using a Fast Fourier Transform (FFT). Vital function activities (e.g., heartbeat and breathing activities) can be determined based on the determined Doppler frequency. In one embodiment of the present invention, more TX antennas are used in the second operating mode to increase the spatial resolution. Because the monitored person is not moving, signal processing can be performed on a smaller area.

[0109] Figure 4 At least a portion of an embodiment of a system is shown that can be used to monitor a person. In this embodiment, the sensor is an FMCW radar 401 that is configured to monitor a room 404 and track the person therein. When a person 402 enters a bed 410, a CPU 406 can instruct the radar to focus a beam 403 in the direction of the person, for example, to measure or monitor the person's health status. The CPU 406 can use a room configuration data store 407 to determine when a person is in bed. A user interface 408 that allows the location of a bed 409 to be specified can be used to enter the configuration into the data store 407. If the radar 401 or the system determines that the safety of the monitored person 402 is threatened based on the measurement data of the radar, a voice conversation with the monitored person can be initiated.

[0110] The CPU 406 and room configuration storage 407 may be integrated with or within the radar 401, or they may be located in a separate computer. The user interface may be a computer program or web-based application used with a web browser to remotely access the configuration storage. The CPU 406 may be a single CPU or it may include multiple CPUs, each running its own data processing pipeline tasks. Figure 4 In the example of , the means for generating audio and the means for capturing audio 405 for voice conversation may be integrated into the sensor 401 , or they may be separate units from the sensor 401 .

[0111] Figure 5 One embodiment of a data processing pipeline used by the system of the present invention is shown, which utilizes a radar-based sensor. The basic radar data processing pipeline 501 is responsible for determining and tracking the three-dimensional position of a person. The fine motion detection pipeline 502 detects finer movements below the threshold of the CFAR (Constant False Alarm Rate) detection block. Beamforming is first applied to increase the signal-to-noise ratio (SNR) of the range spectrum in the direction of interest. It then estimates the phase angle of the range spectrum within the range of interest. The phase angle can be high-pass filtered to see its changes due to movement. The magnitude of the change is evaluated in the spike detection block, and this signal is combined with the CFAR detection located within the region of interest in the motion detection block.

[0112] In one embodiment of the invention, the sensor and / or monitoring system is configured to provide a local alarm in the monitored area. In one embodiment of the invention, the local alarm includes: an audible alarm, such as via a speaker, headphones or hearing aid; a visual alarm, such as a light; and / or an alarm that causes vibration to the bed, mattress and / or to the monitored person. In one embodiment of the invention, the local alarm is an alarm on a wearable device such as a bracelet or watch, wherein the alarm is a vibration on the wearable device and / or there is an electric shock caused by the wearable device. In one embodiment of the invention, the local alarm can be provided simultaneously with a voice conversation and / or alternately with a voice conversation.

[0113] Figure 6 At least a portion of the components of an embodiment of a system is shown, which can be used to detect a health-related state of a person, such as a pressure ulcer. In this embodiment, the sensor can be a radar-based sensor 601, which is configured to monitor a room 604 and track the people in the room. The sensor and system will obtain measurement data from the sensor 601 and track the target (e.g., person) in the monitored area. Based on this, the sensor or system can evaluate the situation in the monitored area and determine when the safety of the monitored person 602 is threatened. Based on the safety of the determined monitored person being threatened, the system can use a device 605 for generating audio to initiate a voice conversation with the monitored person, and use a device for capturing audio to listen to the reply from the monitored person 602. The sensor or system can use speech synthesis and speech recognition in a voice conversation with a person. Based on the captured audio, the sensor or system can evaluate the situation and determine whether action such as an alarm is needed. If an alarm is needed, it can send the alarm to a specified recipient, such as a person and / or organization that monitors the health of the person.

[0114] Figure 7 An example of activation of a large language model (LLM) according to an example embodiment of the present invention is presented, wherein the large language model (LLM) is used to perform a voice dialogue with a monitored person. In this example embodiment, a prompt is presented, which is used to initiate a dialogue between the person being monitored (John) and the large language model (LLM). In the following examples, the name "John" can be replaced with the name of the monitored person, and / or the command "SAY" can be replaced with other commands as described above. The dialogue performed by the large language model (LLM) can be started by describing the current situation, for example based on observations from at least one sensor. In this example embodiment, the phrase "The fall detector sounded an alarm." is used to describe the situation for the large language model (LLM). Figure 7 An example startup prompt is as follows:

[0115] You are a virtual nurse. You take care of an elderly man named John who lives alone in an apartment equipped with a fall detector. When the fall detector sounds an alarm, you should ask John if he needs help.

[0116] If the answer is yes or no, you should sound the alarm and notify John that help is coming.

[0117] If the answer is no, you can tell John to move on.

[0118] Everything you say to John must be prefixed with the command SAY. For example:

[0119] SAY Do you need help?

[0120] You can use the ALARM command to sound an alarm.

[0121] The fall detector sounds an alarm.

[0122] In one embodiment of the invention, the sensor and / or system is configured to provide a local alarm before the person is determined to have moved, woken up, and / or started breathing again. In one embodiment of the invention, a remote alarm is provided if the person does not respond to the local alarm, for example, if the person does not respond to the voice dialogue, move, wake up, and / or start breathing in response to the local alarm after a predetermined time.

[0123] In one embodiment of the invention, the sensor and / or system is configured to provide a local alarm and / or a remote alarm or notification if: the system loses communication or electrical connection with the sensor; if the sensor is removed from its monitoring or installation location; and / or if the communication and / or electrical connection of the sensor is removed. In this way, the system and / or sensor can indicate, for example, that a monitored person has removed the sensor or that someone has attempted to steal the sensor.

[0124] In one embodiment of the present invention, the sensor is arranged on a bracket, floor, ceiling or wall of a room in a home environment or a hospital environment, for example next to or above a bed, so that the measurement area of ​​the sensor covers at least a portion of the bed and / or a person lying on the bed.

[0125] In one embodiment of the invention, the sensor includes means for detecting the orientation of the sensor, such as an accelerometer, and the sensor or system is configured to take the detected orientation of the sensor into account when determining the presence, position, movement and / or posture of the monitored person, for example by compensating measurements based on the detected orientation.

[0126] In one embodiment of the invention, the sensor comprises a battery configured to provide energy to the sensor. In one embodiment of the invention, the sensor comprises a mains power supply configured to provide energy to the sensor and / or the battery.

[0127] In one embodiment of the invention, the sensor comprises an attachment structure in which the sensor can be placed, wherein the attachment structure can be fixed to a stand, a wall or a ceiling. In one embodiment of the invention, the sensor can be removed from the attachment device without any tools, for example for charging the battery of the sensor. The sensor or the attachment structure for the sensor can be arranged on a stand, a wall, for example at a height of 1,5m from floor level or above.

[0128] In one embodiment of the invention, the sensor is configured to analyze the measurement signal by at least filtering the measurement signal in such a way that a phase of the measurement signal is determined in order to observe movements of the person, such as heartbeat and / or breathing.

[0129] In one embodiment of the invention, the sensor is configured to detect a person falling and / or sitting down by a determined elevation angle of the person, for example such that a person may be determined to have fallen down when the elevation angle of the person is below a certain threshold elevation angle value.

[0130] In one embodiment of the present invention, the system comprises at least two sensors of the present invention, and the system is configured to detect and measure people in the monitored area based on measurement signals of the at least two sensors, and the at least two sensors can monitor the same area and / or different areas.

[0131] In one embodiment of the invention, the sensor and / or sensor system comprises at least one light source, such as an LED light source, wherein the sensor is configured to enable the light source when the sensor observes a standing person (e.g., at certain times of the day) and / or when the light level in the monitored area is low. The sensor or system may include means for measuring the light level in the monitored area.

[0132] It is obvious to the person skilled in the art that different embodiments of the invention are not limited only to the examples described above and that they may therefore vary within the scope of the claims presented below.Characteristic features which may be presented in the description in combination with other characteristic features may also, if necessary, be used separately from each other.< / text> < / text>

Claims

1. A system for observing the presence, position, movement and / or posture of people in a monitored area, characterized in that The system comprises: at least one sensor (101, 201, 301, 401, 601); and means for processing measurement signals of the sensor, such as measurement electronics; and means for transmitting measurement results and / or data related to the measurement results for further processing, wherein the system further comprises means (105, 205, 405, 605) for generating and capturing audio, The system is configured to: using the at least one sensor (101, 201, 301, 401, 601) to determine that the safety of the monitored person (206, 402, 602) is threatened, Based on the determination that the safety of the monitored person (206, 402, 602) is threatened, using a device for generating audio to initiate a voice conversation with the monitored person, and using a device for capturing audio to listen to a response from the monitored person (206, 402, 602), and Assess the situation based on the captured audio and determine whether action, such as an alarm, is required, and The system is configured to use a large language model (LLM) to conduct the voice conversation with the monitored person.

2. The system according to claim 1, wherein the system is configured to provide the large language model with a start prompt that is relevant to the context of the monitored person and / or the monitored area.

3. A system according to claim 1 or 2, wherein the prompts related to the context include at least one of the following: information about the role and task of the large language model (LLM) in the speech conversation; information about the person being monitored, such as name, physical condition and level of help required; the current situation as determined using the at least one sensor; conditions that require an alarm to be issued; instructions on how the LLM should interact with the rest of the system.

4. A system according to any preceding claim, wherein the means for producing audio includes a text-to-speech algorithm for converting text output from the system into audible speech for the person being monitored, The means for capturing audio includes a speech recognition algorithm for converting audible speech from the person being monitored into text input for the system, and Wherein the text output of the system is connected to the input of the large language model (LLM), and the text input of the system is connected to the output of the large language model (LLM).

5. The system according to any preceding claim, wherein the system comprises at least one of the following one or more sensors (101, 201, 301, 401, 601): a radar sensor, a floor sensor, a motion detector and / or a camera.

6. A system according to any preceding claim, wherein the means for producing audio is a speaker and / or the means for capturing audio is a microphone or microphone array.

7. A system according to any preceding claim, wherein the system is configured to determine that the safety of the monitored person (206, 402, 602) is threatened when the system has determined that the monitored person (206, 402, 602) has: fallen; is lying on the floor; has spent too much time in certain parts of the monitored area, such as a bathroom or bed; has not eaten or exercised, and / or has not visited certain parts of the monitored area, such as a balcony during winter.

8. A system according to any preceding claim, wherein voice dialogue initiation comprises questions or suggestions to the monitored person.

9. A system according to any preceding claim, wherein the action comprises sending a notification or alert containing a description of the situation and / or the content of the conversation.

10. A method for using a system to observe the presence, location, movement and / or posture of a person in a monitored area, characterized in that The system comprises: at least one sensor (101, 201, 301, 401, 601); and means for processing measurement signals of the sensor, such as measurement electronics; and means for transmitting measurement results and / or data related to the measurement results for further processing, wherein the system further comprises means (105, 205, 405, 605) for generating and capturing audio, Wherein in the method: using the at least one sensor (101, 201, 301, 401, 601) to determine that the safety of the monitored person (206, 402, 602) is threatened, Based on the determination that the safety of the monitored person (206, 402) is threatened, using a device for generating audio to initiate a voice conversation with the monitored person, and using a device for capturing audio to listen to a response from the monitored person (206, 402, 602), and based on the captured audio, assessing the situation and determining whether action, such as an alarm, is required, A large language model (LLM) is used to conduct the voice conversation with the monitored person.

11. The system according to claim 10, wherein a start prompt related to the context of the monitored person and / or the monitored area is provided to the large language model.

12. A system according to claim 10 or 11, wherein the prompt of contextual information includes at least one of the following items: information about the role and task of the large language model (LLM) in the speech conversation; information about the person being monitored, such as name, physical condition and level of help required; current situation as determined using at least one sensor; conditions that require an alarm to be issued; instructions on how the LLM should interact with the rest of the system.

13. A system according to any of claims 10 to 12, wherein the means for generating audio comprises a text-to-speech algorithm for converting text output from the system into audible speech for the person being monitored; the means for capturing audio comprises a speech recognition algorithm for converting audible speech from the person being monitored into text input for the system; and wherein the text output of the system is connected to the input of the large language model (LLM), and the text input of the system is connected to the output of the large language model (LLM).

14. The method according to any of claims 10 to 13, wherein the system comprises at least one of the following one or more sensors (101, 201, 301, 401, 601): a radar sensor, a floor sensor, a motion detector and / or a camera.

15. A method according to any of claims 10 to 14, wherein the means for producing audio is a speaker and / or the means for capturing audio is a microphone or microphone array.

16. A method according to any of claims 10 to 15, wherein the safety of the monitored person (206, 402, 602) is determined to be threatened when the system has determined that the monitored person (206, 402, 602) has: fallen; is lying on the floor; has stayed in certain parts of the monitored area such as a bathroom or bed for too long; has not eaten or exercised, and / or has not visited certain parts of the monitored area, such as a balcony during winter.

17. A method according to any of claims 10 to 16, wherein voice dialogue initiation comprises questions or suggestions to the monitored person.

18. A method according to any of claims 10 to 17, wherein the action comprises sending a notification or alert containing a description of the situation and / or the content of the conversation.

Images