Monitoring Sensors and Systems

Abstract

A method and system for observing the presence, position, movement and / or posture of a person in a monitored area, the system comprising at least one sensor (101, 201, 301, 401, 601), means for processing measurement signals of the sensor, such as measurement electronics, and means for communicating the measurement results and / or data related to the measurement results for further processing. The system further includes a means for generating and capturing audio (105, 205, 405, 605), and is configured to determine, using at least one sensor (101, 201, 301, 401, 601), that the safety of the monitored person (206, 402, 602) is threatened, and based on the determined threat to the safety of the monitored person (206, 402, 602), to initiate an audio dialogue with the monitored person (206, 402, 602) using the means for generating audio, and to listen to a response from the monitored person (206, 402, 602) using the means for capturing audio. Based on the captured audio, the system is configured to assess the situation and determine whether an action, such as an alarm, is required, and in so doing, the system is configured to use a large-scale language model (LLM) to conduct the audio dialogue with the monitored person.

Description

[Technical Field]

[0001] The present invention relates to a method and system that can observe and track people within a surveillance area and / or monitor their condition and safety. [Background technology]

[0002] If it is desired to extend the possibility of elderly people being able to cope in the home environment, it is inevitable to monitor the condition of elderly people in the home environment. Emergency aid bracelet systems are currently widely used for this kind of application. Their weaknesses are that the user must wear the bracelet at all times and must be able to press the alarm button in case of an emergency. There are also bracelets that check the user's health condition, but they suffer from the same problems as above, and also have the problem of false alarms.

[0003] Another proposed solution is to install a film of piezoelectric material on the floor, which detects changes in pressure caused by movement on the floor. There is also known prior art technology that uses sensors installed on or under the floor, which do not require changes in pressure but instead use capacitance sensors to detect the presence and movement of people.

[0004] The prior art also introduces the possibility of using video cameras, motion detectors, for example based on infrared detection, or for example ultrasonic sensors, to monitor the physical condition and status of elderly people, for example WO 2012 / 164169 discloses a method and system for tracking objects based on ultrasonic technology.

[0005] Several prior art solutions are known that use millimeter wave (MMW) radar to track people. Summary of the Invention [Problem to be solved by the invention]

[0006] A drawback of conventional monitoring and surveillance systems is that they often interpret a person as needing help when no outside assistance is required. In such cases, for example, a fall may be detected and a fall-related alarm may be triggered even when the person has not fallen or is able to get up without assistance. These false alarms place a significant burden on the monitored person, who must contact the company responsible for monitoring the person's safety, or a professional caregiver to determine whether the monitored person needs help. For these reasons, there is a need for a reliable monitoring system that minimizes the number of false alarms. [Means for solving the problem]

[0007] The problems of the prior art can be overcome by using the system according to claim 1 and the method according to claim 9. The invention is characterized by what is disclosed in the claims.

[0008] The present invention relates to a system for observing the presence, position, movement, and / or posture of a person in a monitored area. The system includes at least one sensor, a means for processing a measurement signal from the sensor, such as measurement electronics, and a means for communicating the measurement results and / or data related to the measurement results for further processing. The system further includes a means for generating and capturing audio. The system is configured to: determine, using the at least one sensor, that the safety of a monitored person is at risk; initiate an audio dialogue with the monitored person based on the determined endangered safety of the monitored person using the audio generating means; listen to a response from the monitored person using the audio capturing means; and, based on the captured audio, assess the situation and / or determine whether an action, such as an alarm, is required.

[0009] In one embodiment of the present invention, the system is configured to use a large language model (LLM) to conduct a voice interaction with the monitored person.

[0010] In one embodiment of the present invention, the system is configured to provide a starting prompt to the large-scale language model that is relevant to the context of the monitored person and / or the monitored area.

[0011] In one embodiment of the present invention, the context-relevant starting prompt includes at least one of information regarding the role and task of the large language model (LLM) in the voice interaction, information about the person being monitored such as their name, physical condition and level of assistance needed, the current situation as determined by at least one sensor, conditions that warrant an alarm, and instructions on how the LLM should interact with the rest of the system.

[0012] In one embodiment of the invention, the means for generating audio comprises a text-to-speech synthesis algorithm that converts text output from the system into audible speech for the monitored person, the means for capturing audio comprises a speech recognition algorithm that converts audible speech from the monitored person into text input for the system, the text output of the system being connected to an input of a large scale language model (LLM) and the text input of the system being connected to an output of the large scale language model (LLM).

[0013] In one embodiment of the present invention, the system comprises at least one sensor or a plurality of sensors: a radar sensor, a floor sensor, a motion detector and / or a camera.

[0014] In one embodiment of the present invention, the means for generating sound is a speaker and / or the means for capturing sound is a microphone or a microphone array.

[0015] In one embodiment of the present invention, the system is configured to determine that the safety of the monitored person is at risk if the system determines that the monitored person has fallen, is lying on the floor, is staying in a particular part of the monitored area, such as the bathroom or bed, is not eating or exercising, and / or is moving to a particular part of the monitored area, such as a balcony in winter.

[0016] In one embodiment of the present invention, the initiation of the voice interaction consists of a question or suggestion to the monitored person.

[0017] In one embodiment of the present invention, the system is configured to process the captured speech with a speech recognition algorithm and convert the captured speech into a text response.

[0018] In one embodiment of the present invention, the system is configured to compare the text response to a set of situation assessment keywords.

[0019] In one embodiment of the present invention, the action includes sending a notification or alarm that includes a description of the situation and / or the interaction.

[0020] The present invention also relates to a method for observing the presence, location, movement and / or posture of a person in a monitored area using a system including at least one sensor, means for processing sensor measurement signals such as measurement electronics, means for communicating measurements and / or data related to the measurements for further processing, and means for generating and capturing audio. The method includes determining, using the at least one sensor, that a safety of a monitored person is at risk, initiating an audio dialogue with the monitored person based on the determined endangerment of the monitored person's safety using the audio generating means, listening to a response from the monitored person using the audio capturing means, and assessing the situation based on the captured audio to determine whether an action, such as an alarm, is required.

[0021] In one embodiment of the present invention, a large-scale language model (LLM) is used to conduct a spoken dialogue with the monitored person.

[0022] In one embodiment of the present invention, a starting prompt is provided to a large language model that is relevant to the monitored person and / or the context of the monitored area.

[0023] In one embodiment of the present invention, the contextual information prompts include at least one of information regarding the role and task of the large language model (LLM) in the voice interaction, information about the monitored person such as name, physical condition, level of assistance needed, the current situation as determined by at least one sensor, conditions that warrant an alarm, and instructions on how the LLM should interact with the rest of the system.

[0024] In one embodiment of the invention, the means for generating audio comprises a speech synthesis algorithm that converts text output from the system into audible speech for the monitored person, and the means for capturing audio comprises a speech recognition algorithm that converts audible speech from the monitored person into text input for the system, the text output of the system being connected to an input of a large scale language model (LLM) and the text input of the system being connected to an output of the large scale language model (LLM).

[0025] The above-mentioned solution of the present invention provides a monitoring system that can provide reliable measurement results under different circumstances and is easy to install and maintain. In particular, the system has the advantage of being able to reliably monitor the safety of people in the monitored area, while at the same time minimizing the number of false alarms by asking via automated voice dialogue whether they require assistance.

[0026] Various other advantages will become apparent to those skilled in the art from the following detailed description.

[0027] In this specification, the terms "first," "second," and "third," unless otherwise expressly stated, are used to distinguish one element from another and do not dictate any particular priority or ordering.

[0028] The exemplary embodiments of the present invention presented herein should not be construed as limiting the applicability of the appended claims. The verb "to comprise" is used herein as an open limitation that does not exclude the presence of similarly unrecited features. Features recited in dependent claims are mutually freely combinable, unless expressly stated otherwise.

[0029] The novel features which are believed to be characteristic of the invention are set forth with particularity in the appended claims. However, the invention itself, both as to its structure and method of operation, together with additional objects and advantages thereof, will best be understood from the following description of specific embodiments when read in connection with the accompanying drawings.

[0030] The present invention will now be described in detail with reference to the accompanying drawings. [Brief explanation of the drawings]

[0031] [Figure 1] 1 illustrates components in a monitored area of ​​an exemplary embodiment of the system of the present invention. [Figure 2] 1 illustrates the operation of an exemplary embodiment of the system of the present invention. [Figure 3] 1 shows an exemplary embodiment of a sensor according to the solution of the present invention; [Figure 4] 1 illustrates an exemplary embodiment of a system according to the present invention. [Figure 5] 2 illustrates an example processing pipeline for a system according to one embodiment of the present invention. [Figure 6] 1 illustrates an exemplary embodiment of a system according to the present invention. [Figure 7] 1 illustrates an example of starting a large language model (LLM) according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0032] In the solution of the present invention, sensors can detect the presence and movement of a subject within a monitored area. The monitored subject can be, for example, an elderly person or other person in need of monitoring. The sensors can be installed in the monitored area accessible to the subject. In one embodiment of the present invention, the sensors can also be used to monitor the monitored person's vital functions, such as respiration, e.g., respiratory rate, and / or heart rate.

[0033] In the solution according to the invention, the system may further comprise a central unit including at least one sensor, measurement electronics for generating sensor observations by the sensor, a processor configured to process the sensor observations, and / or a memory, the central unit being, for example, a data processing device. For this purpose, the central unit of the system may include necessary software and information on the characteristics of the detected signals. In general, the measurement electronics and / or the central unit may derive information from the signals received via the sensor. The system may have a central unit capable of managing one or more sensors or sensor groups. In one embodiment of the invention, a sensor group may include sensors located in the same space, for example, the same room.

[0034] The area monitored by the sensors may be the entire area where people normally reside, or only a portion of an area. The monitored area may consist of, for example, one or more rooms, and certain parts of the area, e.g., fixed fixtures such as cupboards, may be located outside the monitored area. In an embodiment for monitoring sleepers, sensors may be placed in relation to the bed, above the bed and / or next to the bed, such that the area monitored by at least one sensor covers at least a portion of the bed or the person lying in the bed.

[0035] In the solution according to the invention, the sensor detects people in the monitored area and measures and detects the position, velocity and / or shape of the monitored persons. In one embodiment of the invention, the sensor is configured to observe objects based on signal strength and / or by filtering out likely false measurements.

[0036] The sensors may include at least one of a radar sensor, a floor sensor, a motion detector, and / or a camera. In one embodiment, the system comprises at least two sensors configured to detect and measure people in a monitored area based on measurement signals of the at least two sensors, which may monitor the same area and / or different parts of the monitored area. For example, the measurement areas of the sensors may overlap, for example, in certain parts of the area.

[0037] The components of the system may be integrated into a single unit, e.g., a sensor device comprising the components of the system. In one embodiment of the present invention, the sensor device may include at least some of the following components in a single unit: a sensor, means for processing a measurement signal of the sensor (such as measurement electronics), means for communicating the measurement result and / or data related to the measurement result for further processing, and means for generating and capturing sound.

[0038] The sensor can be mounted on a stand or on a surface, such as a wall, door, floor, or ceiling, and / or on an apartment floor, wall, door, or ceiling, and / or near the monitored area where the subject has access. In one embodiment of the present invention, the sensor or sensors are installed in a corner of the monitored space directly below the ceiling, tilted toward the center of the monitored space. A typical tilt angle is, for example, 15 degrees, which provides the sensor with a good field of view over obstacles such as furniture. In one embodiment of the present invention, the sensor or sensors are installed on a wall or in a corner of the monitored space, typically above floor level, for example, approximately 40-150 cm above the floor. The sensor's field of view can be, for example, approximately 90 degrees in a horizontal plane.

[0039] The system further comprises at least one means for generating sound, which may be, for example, a speaker, and / or at least one means for capturing sound, which may be, for example, a microphone or a microphone array.

[0040] In one embodiment of the present invention, the system utilizes ordinary home audio equipment, such as Bluetooth® speakers or wireless speakers, as a means of capturing and generating sound.

[0041] In one embodiment of the present invention, the system is configured to use a VoIP speakerphone as a means of generating and capturing audio to enable voice interaction over the Internet.

[0042] The system may determine, via at least one sensor, that the safety of the monitored person is threatened. Based on the determination that the safety of the monitored person is threatened, the system may initiate a voice dialogue with the monitored person using a means for generating audio and listen to a response from the monitored person using a means for capturing audio. Based on the captured audio, the system may assess the situation and determine whether action is required, such as sending a notification or generating or sending an alarm.

[0043] In one embodiment, sending an alarm or notification includes sending a message to the person and / or to an organization monitoring the person's health, such as a message to a phone, a message to a nurse, a message to a relative, or a message to an emergency center. In one embodiment, initiating a voice interaction consists of a question or suggestion to the monitored person, such as asking the person if they are OK and / or if they need help. In one embodiment of the present invention, if there is no response from the monitored person, the alarm or notification is sent because the system cannot confirm that the monitored person does not need help.

[0044] In one embodiment of the present invention, the system can process the captured speech with a speech recognition algorithm to convert the speech into a text response. In one embodiment of the present invention, the system can compare the text response to a set of keywords for situation assessment. A notification or alarm can be sent along with a description of the situation and / or the content of the interaction.

[0045] In one embodiment of the present invention, the system is configured to send the captured speech to an external server or service, for example, a cloud-based speech recognition engine, for speech-to-text translation.

[0046] In one embodiment of the present invention, the system is configured to generate voice prompts using an external, e.g., cloud-based, speech synthesis service.

[0047] In one embodiment of the invention, the sensor or system is configured to perform sentiment analysis on the captured audio and / or text responses to provide a situation assessment. In one embodiment of the invention, the system is configured to perform sentiment analysis of the text responses on an external server or service, such as a cloud-based sentiment analysis service.

[0048] In one embodiment of the present invention, the means for generating speech is used in conjunction with a speech synthesis algorithm to convert text output from the system into audible speech for the person being monitored.

[0049] In one embodiment of the present invention, the means for capturing audio is used in conjunction with a speech recognition algorithm to convert audible speech from the person being monitored into text input for the system.

[0050] In one embodiment of the present invention, the text output and input are connected to the input and output of a large-scale language model (LLM). The large-scale language model can be, for example, an artificial neural network trained to generate natural language from a given context. The large-scale language model (LLM) can be trained using self-supervised and / or semi-supervised learning, for example, including tens of millions to billions of weights. The large-scale language model works by receiving input text and iteratively predicting the next token or word.

[0051] In one embodiment of the present invention, a large language model (LLM) is used to conduct a voice dialogue with a monitored person. The large language model (LLM) can be provided with special prompts to provide context for initiating the dialogue, for example, context of the monitored person and / or the monitored space. In one embodiment of the present invention, the dialogue can be initiated by describing a current situation for the large language model (LLM), for example, based on observations from at least one sensor.

[0052] In one embodiment of the present invention, the prompts guide the LLM to assume the role of a virtual nurse or assistant responsible for checking the monitored person's condition if the system determines that the monitored person's health may be compromised. The prompts can include additional context, such as information about the monitored person's condition, the level of assistance needed, and / or conditions that warrant an alarm. Furthermore, the prompts instruct the LLM to generate responses in a structured manner that can be used to drive or control the rest of the system. Indications of the desired syntax can be given as a few-shot example of the dialogue.

[0053] In one embodiment of the present invention, the conditions under which an alarm is justified include when the system detects a fall and the monitored person answers "yes" or does not answer at all when asked by a large-scale language model (LLM) if they need help. If the system detects a fall and the monitored person answers that they do not need help, no alarm is necessary and is not issued.

[0054] In one embodiment of the present invention, the structured responses from the large-scale language model (LLM) include commands to the rest of the system. Such commands can be, for example, "SAY <text>" to trigger a speech synthesis system to output speech corresponding to "<text>" based on commands from the large-scale language model, and / or "ALARM" to trigger the delivery of an alarm. These commands (presented above and below) are only examples of structured responses; other predefined responses or commands and / or response protocols, such as JSON messages or function calls, can be used.

[0055] In one embodiment of the present invention, the system may determine that the monitored person's safety is at risk if the system determines that the monitored person has fallen, is lying on the floor, is staying too long in a particular part of the monitored area, such as the bathroom or bed, is not eating or exercising, and / or has gone to a particular part of the monitored area, such as a balcony in winter. In one embodiment of the present invention, the system may determine that the monitored person's safety is at risk if the monitored person's vital functions, such as the tracked heart rate and / or breathing of the monitored person, are not within a predefined range.

[0056] In one embodiment of the present invention, a fixed object on which a person can lie, such as a bed or sofa, can be determined by the user using a sensor and / or sensor system, and the sensor will not determine that a person has fallen in the area of ​​these fixed objects. In one embodiment of the present invention, the sensor can distinguish the object from the observed person by the determined height of the observed object, for example, recognizing that the object is not a person if the determined object height is essentially always below a certain threshold height value.

[0057] In some applications, it may be advantageous to first map static areas, i.e., map sensor measurement information when primarily stationary or non-moving objects or structures are in fixed positions. Such situations include, for example, residential apartments where furniture is in fixed positions but no people, pets, or robots are present. This charted information can be recorded, for example, in a memory located in a central unit or in a memory means connected via a data network, which can be located, for example, in a control center or a service center. To do this, the memory means can be integrated into the sensor or the system, whereby the memory means can be located in the central unit or connected via a data network.

[0058] According to one embodiment of the present invention, the system graphs the unchanged area continuously or at defined intervals, in which case the system can detect changes in the area caused by, for example, new furniture or changes in furniture position. In this way, the system can gradually adapt to changes occurring in the monitored area.

[0059] 1 illustrates components of an exemplary embodiment of a system for monitoring a monitored area. A sensor 101 or sensors used in the present invention are positioned relative to the monitored area such that the monitored area can be monitored by the sensor 101 or sensors. The sensors can be installed on a surface, such as a wall, floor, or ceiling, and secured to the surface with, for example, double-sided tape or sticker strips, allowing for easy removal. The sensors 101 can be connected wirelessly or via a wire to a gateway 104, which collects measurements from the sensors 101 or status information generated by the sensors 101, such as detected objects, the health status of objects such as people, and / or the movement and posture of the objects.

[0060] The system may comprise at least one means for generating sound 105, e.g., a speaker, and / or at least one means for capturing sound, e.g., a microphone or a microphone array. The means for generating sound and / or the means for capturing sound may be integrated into the sensor 101 and / or the means for generating sound and / or the means for capturing sound may be separate units.

[0061] The gateway 104 can transmit the information to, for example, a control center or another organization that manages subjects such as an area and / or people therein. The information transfer between the system and the recipient can be performed using, for example, a telephone connection, a wired broadband connection, or a wireless connection. The data transfer advantageously takes into account data security and privacy issues that also address many official regulations.

[0062] In one embodiment of the present invention, the sensor 101 or sensors have their own central unit, which is connected to the gateway 104. In a second embodiment of the present invention, the central unit of the sensor 101 or sensors is integrated into the gateway 104.

[0063] Some of the functions of the central unit or gateway 104 may also be performed elsewhere via a data network connection, such as in a central control room or service center, on a server or cloud service, etc.

[0064] The system according to the present invention may also include a call push button 102. When pressed, the system may connect to, for example, nursing personnel, security personnel, or may execute various alarm procedures. The call push button may be wireless and may be adapted to function without a battery.

[0065] Notification and alarm procedures by the system of the present invention may include, for example, activation of a local alarm, emission of an indication signal (buzzer, light, siren, alarm clock, etc.), contacting an alarm or service center, a care provider or a relative, etc. In some cases, an alarm can also be sent directly to the monitored person or user, for example by voice synthesis or voice recording. To perform these tasks, the sensor or system may be equipped with the necessary means for processing time data, for example a clock circuit.

[0066] According to one embodiment of the present invention, in addition to voice interaction, an alarm signal may be provided by the system in the monitored space, lasting a predetermined time. This alarm signal may be provided, for example, as a local alarm before an alarm or notification is sent, and may be provided via a light and / or sound alarm unit of the system. The light and / or sound alarm units may be located in different parts of the premises, for example, in rooms. This functionality may also be integrated into the sensors, for example, in all sensors or only in some sensors.

[0067] The system according to the invention may also comprise a fire detector 103, which may be connected to other systems via wired or wireless connections. If the fire detector 103 issues a fire alarm, an alarm procedure may be executed, for example by sending an alarm message to a control center or rescue authorities.

[0068] FIG. 2 illustrates the operation of one embodiment of a system according to the present invention, which monitors the health or posture of a person 206 in a monitored area.

[0069] If the system's sensor 201 detects that the safety of the monitored person 206 is threatened, the system can use a means for generating audio to initiate an audio dialogue with the monitored person 206 and a means for capturing audio to listen to a response from the monitored person 206. Based on the captured audio, the system can assess the situation and determine whether action is required, such as sending a notification, creating or sending an alarm, etc. In the example of FIG. 2, the means for generating audio and the means for capturing audio can be integrated into the sensor 201.

[0070] In one embodiment of the present invention, the system examines information measured by several sensors, e.g., all sensors in the monitored area, and only if no other person is detected in the area by the sensors is a notification, e.g., a remote alarm, sent and / or a voice interaction initiated.

[0071] In the situation of the embodiment shown in FIG. 2 , the system sends a message, for example, based on a voice interaction, based on the subject's fall, and / or due to the subject's determined vital functions, while the sensor 201 sends information about the situation to the system gateway 204, which then sends the information and / or alarm to the server 201, for example, via an Internet connection or other connection. The information and / or alarm is sent from the server 207 to an organization monitoring the subject's health, for example, as a message to the mobile phone 202, and / or to, for example, a nurse 203, a relative, or an emergency center. In this way, for example, information about the subject can reach the necessary people or organizations, and the subject who has fallen can receive assistance as soon as possible. In one embodiment of the present invention, the system can send information directly from the gateway 204 to the organization or person monitoring the health of the monitored person. In this example, the monitored person may have mentioned needing help in a voice interaction with the system received by the voice capture means 205, or there may have been no response from the monitored person after the voice interaction began.

[0072] The processor, central unit and / or measurement electronics used in the solution of the present invention can be integrated into the sensor or can be separate or located in a separate device. In an embodiment of the present invention, software executed by the processor enables the sensor or system to interpret the behavior observed by at least one sensor and to issue an alarm if a program-defined alarm condition is met.

[0073] In one embodiment of the present invention, only a portion of the sensors in a monitored area have this capability to enable the generation of such an alarm signal. For example, only sensors in some rooms, such as a living room, may have this capability, while sensors in other rooms may send a notification immediately after a fall is detected and / or if the monitored person's measurements are not within acceptable and / or predefined ranges. In one embodiment of the present invention, only a portion of the sensors in a space, such as a room, have this capability to enable the generation of such an alarm signal.

[0074] The system may also include a control center, and predetermined information regarding the presence, location, movement, and / or attitude of the object may be transmitted to the control center. The alarm conditions used by the system may be altered based on presence information that may be received, for example, from an RFID reader.

[0075] The system may also have memory means adapted to record measurement signals or information derived therefrom for observing time-dependent relationships in the monitored area or human behavior, so that the system can sound an alarm or initiate a voice dialogue if, for example, the monitored person does not get out of bed for a certain period of time, does not go to the kitchen, goes to the toilet too frequently, or if the observed person's vital functions, such as breathing or heart rate, change within a certain period of time. The memory means may also enable the system to learn more general daily rhythms and detect abnormalities occurring within them.

[0076] In one embodiment of the present invention, the sensor and / or system may include a wireless-based identification means for identifying a person. The wireless-based identification means may be, for example, a Bluetooth®, Bluetooth® low energy (BLE) or Zigbee®-based means. In this embodiment, the system may recognize a person and a wireless-based device carried by the person, such as a bracelet, a watch, a mobile device, a tag, etc., and may associate measurements with a specific recognized person. In this way, the system may know who is present in the monitored area and to whom the monitoring results relate.

[0077] In one embodiment of the present invention, the radio-based identification means may comprise an antenna array that allows the identification device to be more accurately associated with its bearer when multiple people and devices are present.

[0078] In one embodiment of the present invention, if the identification means detects a particular person, such as a nurse, within the monitored area, the alarm may be automatically disabled.

[0079] In one embodiment of the present invention, the alarm conditions of the system can include the identity of a person, for example, an alarm can be set off if an unauthorized person enters a particular location.

[0080] In one embodiment of the present invention, a radio-based identification means, such as a Bluetooth®, Bluetooth® low energy (BLE), or Zigbee®-based means, can be used or aid in locating a person. The sensor can include multiple antennas for the radio-based identification means, such as a Bluetooth®, BLE, or Zigbee® antenna, to enable direction-finding technology, such as Zigbee®, Bluetooth®, or Bluetooth® low energy (BLE) direction-finding technology according to the Bluetooth® 5.1 specification. In one embodiment of the present invention, if the sensor's radar detects motion but the radio-based identification means does not detect a remotely readable tag or device, such as a Bluetooth®, BLE, or Zigbee® tag or device, the person detected by the radar can be considered a visitor. On the other hand, if the radar detects a remotely readable tag or device, such as a Bluetooth®, BLE, or Zigbee® tag or device, the detected person can be identified and action can be taken based on the identified person. In one example embodiment, if a resident is in a room and a caregiver is also present, the status of that person or room can be set in the system to "caregiver in room." Similarly, an alarm set off by a resident can be acknowledged as the system recognizing that someone other than the resident has entered the room. In this case, the alarm is automatically acknowledged. In one embodiment, alarms are not automatically acknowledged but require an active, identifiable event, for example, from the user device.

[0081] In one embodiment of the present invention, the identification of a detected person can be performed by other means, such as a surveillance camera located in a hallway. In this case, a radar-based sensor detects someone entering a room, and the system can review information from the surveillance camera, such as a surveillance recording from a specific point in time where a person can be seen entering the room. In one embodiment, this recording can be linked to the room as an entry event, and the entrant can be later identified by viewing the recording, if desired. In this case, identification can be performed automatically, but need not be performed if automatic identification is not desired. When automatic identification from video is used, it can be performed based on, for example, facial recognition technology. In one embodiment, facial recognition or video-based identification is not used if the user can be identified in other ways. In one embodiment, video-based identification is only used if the person cannot be identified in other ways.

[0082] In one embodiment of the present invention where radio-based identification is used, the necessary electronics and antennas can be integrated with the sensor. An example embodiment is shown in FIG. 3, where a Bluetooth® antenna array is integrated with the sensor 301. The Bluetooth® antenna array of FIG. 3 includes four antennas 302 and the necessary electronics to control the operation of the identification and antennas. This antenna array can be used to measure and detect Bluetooth® devices and tags, or to locate people carrying Bluetooth® devices, such as bracelets, using Bluetooth® 5.1 direction-finding technology. In one embodiment, data measured by the Bluetooth® antenna array is combined with radar data measured by the sensor, for example, to improve the position and location accuracy of the radar sensor. The radar 303 antenna or antenna array is, in this embodiment, located at the center of the sensor, inside the area formed by the four Bluetooth® antennas 302.

[0083] In one embodiment of the present invention, sensors according to the present invention can be used, for example, in a hospital room or other room where people are sleeping and need to be monitored. In this embodiment, the sensors can be positioned to measure and sense the presence of a person in a bed. The sensors can be positioned within or relative to a room such that the monitoring area of ​​one sensor covers at least a portion of one bed. In one embodiment of the present invention, the sensors are positioned on the ceiling of the room, e.g., above each bed, e.g., one sensor above each bed. In one embodiment of the present invention, the sensors are positioned on the walls of the room, e.g., next to each bed, e.g., one sensor per bedside. In these embodiments, the sensors can measure and / or sense not only the presence of a person in bed, but also their vital functions, such as their movement, heart rate, breathing, etc. One advantage of these embodiments is that they allow for undisturbed monitoring of the sleeping person, which is not possible with wired sensors, for example. This embodiment also makes it easier for staff and nurses to monitor sleeping people, for example, in a hospital environment. In these embodiments, the sensors do not need to include a means for detecting the orientation of the sensor.

[0084] In one embodiment of the present invention, at least one additional sensor according to the present invention can be placed in a monitored room or area where a person is sleeping. This additional sensor can detect and monitor a person leaving the bed. In this case, the measurement area of ​​the additional sensor can be larger than the measurement area of ​​the sensor monitoring the bed. The measurement area can cover substantially the entire room, for example, with one or more additional sensors. The additional sensor can also be placed in the room or relative to the room so that the measurement area of ​​the sensor or sensors covers the room and particularly the area outside the bed. In one embodiment, the additional sensor can be placed on the ceiling, walls, and / or corners of the room, or on a stand. This embodiment allows staff to better monitor the room, for example, to issue an alarm if a person leaves the bed and / or disturbs other people trying to sleep. These additional sensors can also monitor a person leaving the bed and issue an alarm, for example, if the person falls and / or if a measured vital function is not at a predefined and / or acceptable level. In these embodiments, the additional sensor does not need to include a means for detecting the orientation of the sensor.

[0085] In one embodiment of the present invention, the sensor may include, for example, a millimeter wave (MMW) radar, which may operate on the MIMO radar principle. In one example embodiment, there may be, for example, three transmit antennas and four receive antennas. In this example, a 12-element virtual antenna is formed. The sensor of the present invention allows for accurate observation of the elevation angle, azimuth angle, movement, and distance of an object. For example, FMCW (Frequency Modulated Continuous Wave) technology may be used for the radar.

[0086] The sensors may measure and detect movements, such as breathing rate, position, speed, and / or shape, of the monitored person. In one embodiment of the present invention, the sensors may determine a state of the monitored person, such as pauses or interruptions in the monitored person's breathing, for example, to recognize sleep apnea and / or immobility of the monitored person. In one embodiment of the present invention, the determined state of the person may include the person snoring.

[0087] In one embodiment of the present invention, the system can track the precise location of a person using an FMCW radar operating in the millimeter wave band with an antenna array. The system can include a user interface or another configuration interface that can be used to specify the location of a bed, sofa, or other location of interest and store it in a room configuration data store. A CPU can receive the person's location from the radar and examine the room configuration to determine whether the occupant is in bed or other resting position. In such a case, the CPU can instruct the radar to focus on the location of interest and begin monitoring for fine movement. This focusing can be achieved, for example, by using beamforming with the antenna array to amplify signals emanating from the direction of interest. Large movements can be detected by observing changes in the range or Doppler spectrum of the signal. Small movements can be detected by observing changes in the signal's phase angle.

[0088] In one embodiment of the present invention, the radar-based sensor is configured in a first mode of operation to track the movement of an observed person by analyzing signals reflected from the person, e.g., Doppler frequency, range and angle of arrival of the signal. In one embodiment of the present invention, the radar-based sensor is configured in a second mode of operation to track the heartbeat and / or breathing of the observed person by analyzing the phase of the measured signal. In one embodiment of the present invention, the sensor sweep time is longer in the second mode of operation than in the first mode of operation. In one embodiment of the present invention, the second mode of operation can differ from the first mode of operation only by the digital signal processing algorithm applied to the signal.

[0089] In one embodiment of the present invention, person tracking is not performed in the second operating mode. In one embodiment of the present invention, a single radar-based sensor can use the first and second operating modes simultaneously, such that the first operating mode is always used and the second operating mode is activated when needed and disabled when not needed. In one embodiment of the present invention, a single sensor can use the first and second operating modes in an interleaved manner.

[0090] The radar-based sensor can be configured to activate the second mode of operation based on detecting that the monitored person is not moving, has fallen, and / or that the monitored person's velocity is slower than a predefined threshold. The sensor can be configured to disable the second mode of operation based on determining that the monitored person is not falling, that the monitored person is moving, and / or that the monitored person's velocity is higher than a predefined threshold.

[0091] In the second mode of operation, the radar-based sensor can be configured to analyze the measurement signal such that a phase of the measurement signal is determined to observe the person's movements, such as heartbeat and / or breathing. In one embodiment of the present invention, in the second mode of operation, the radar-based sensor and / or the sensor's measurement electronics are configured to analyze the measurement signal from an area and / or a certain distance around the area related to the azimuth, elevation and / or distance of the person from the sensor determined in the first mode of operation.

[0092] In an embodiment of the present invention, an apartment or nursing home may have at least one sensor in each room. In this case, unless corrective measures are taken, the sensors (e.g., radars) will interfere with each other. In one embodiment, by dividing the modes and / or the number of sensors (e.g., radars) into specific time slots, multiple sensors can be used simultaneously near each other without causing interference. Sensor transmissions can be synchronized and performed in an interleaved manner, for example, so that the sensors observe the same person and / or the same room.

[0093] In one embodiment of the present invention, different sensors may be in different operating modes, for example, some sensors may determine stationary objects while the second operating mode is active, while other sensors may monitor object motion and search for stationary objects using only the first operating mode.

[0094] In one embodiment of the present invention, the sensor or system is configured to detect whether a person is falling and / or sitting by their determined height, such that the person can be determined to have fallen when their height is below a certain threshold height value. In one embodiment of the present invention, the person's height is tracked and filtered with a filter, such as a Kalman filter or a low pass filter, to prevent false alarms due to noisy measurements.

[0095] An example embodiment is described below. In this example embodiment, the sensor is a radar-based sensor with two operation modes. The first operation mode of the sensor is used to track the presence and movement of people, for example, in a room. In this embodiment, tracking can be performed using measured point cloud data. The required Doppler range is given by:

number

[0096] In one example, if a person is moving at a speed of 1 m / s, the required Doppler range is ±400 Hz, with a maximum measurement interval of 2.5 ms at a frequency of 60 GHz. Inhalation lasts approximately 2 seconds. If the corresponding movement is 5 mm, the required Doppler range is ±1 Hz, with a sweep time of 1 second.

[0097] When the system observes that the person has stopped, a second mode of operation can be activated, in which the person's vital functions, such as heart rate and / or breathing, for example, pauses and / or frequency of breathing, can be tracked. In one embodiment of the present invention, the system must perform measurements for a certain amount of time before it can detect periods of breathing for the person.

[0098] After the person's vital functions have been determined, the system can disable the second mode of operation. In one example embodiment, the system can periodically determine the same person's vital functions, for example, as long as the person is stationary. When the system observes people who are stationary, it begins to determine the vital functions of these people using the second mode of operation.

[0099] In one exemplary embodiment of the present invention, operation in the second operating mode can be implemented such that, for example, when a stationary object is detected, point cloud data surrounding the area of ​​the detected object is saved and analyzed. The saved package can be generated periodically, for example, every 600 ms. In one embodiment of the present invention, the data can be transferred to a central control unit for analysis. Analysis of the signal, i.e., the point cloud data, can provide information about small movements of the object, allowing the system to determine, for example, a person's breathing activity and / or heart rate.

[0100] In one embodiment of the present invention, the sensor sweep time is increased in the second mode of operation, which allows a better signal-to-noise ratio to be achieved. Also, since more time is available for measurements, more TX antennas can be utilized. In this way, the angular resolution can be improved. Range resolution can be improved by increasing the frequency sweep range.

[0101] The Doppler frequency can be determined, for example, by a Fast Fourier Transform (FFT). Based on the determined Doppler frequency, vital function activity, such as heart rate and respiratory activity, can be determined. In one embodiment of the present invention, more TX antennas are used in the second operating mode to improve spatial resolution. Because the monitored person is not moving, signal processing can be performed over a smaller area.

[0102] FIG. 4 illustrates at least some of the components of one embodiment of a system that can be used to monitor a person. In this embodiment, the sensor is an FMCW radar 401 configured to monitor a room 404 and track a person therein. When a person 402 enters a bed 410, a CPU 406 can instruct the radar to focus a beam 403 in the direction of the person, for example, to measure or monitor the person's health. The CPU 406 can use a room configuration data store 407 to determine when the person is in bed. The configuration can be entered into the data store 407 using a user interface 408 that can specify the location of a bed 409. If the radar 401 or the system determines, based on the radar measurement data, that the safety of the monitored person 402 is threatened, a voice interaction with the monitored person can be initiated.

[0103] The CPU 406 and room configuration store 407 can be integrated with or located within the radar 401, or can be located on a separate computer. The user interface can be a computer program or a web-based application used in a web browser to remotely access the configuration store. The CPU 406 can be a single CPU or can include multiple CPUs, each performing a task in the data processing pipeline. In the example of Figure 4, the speech generation and capture means 405 used for voice interaction can be integrated into the sensor 401 or can be a separate unit from the sensor 401.

[0104] Figure 5 shows one embodiment of a data processing pipeline used in a radar-based sensor-based system of the present invention. The basic radar data processing pipeline 501 is responsible for locating and tracking a person in three dimensions. The fine motion detection pipeline 502 detects fine motion below the threshold of the Constant False Alarm Rate (CFAR) detection block. First, beamforming is applied to increase the signal-to-noise ratio (SNR) of the range spectrum in the direction of interest. Next, the phase angle of the range spectrum bins within the region of interest is estimated. The phase angle can be high-pass filtered to detect changes due to motion. The magnitude of the change is evaluated in the spike detection block, and this signal is combined with the CFAR detection located within the region of interest in the motion detection block.

[0105] In one embodiment of the invention, the sensor and / or monitoring system is configured to provide a local alarm for the monitored area. In one embodiment of the invention, the local alarm comprises an audible alarm, for example, via a speaker, headphones, or hearing aid device; a visual alarm, such as a light; and / or an alarm that causes a vibration on the bed, mattress, and / or monitored person. In one embodiment of the invention, the local alarm is an alarm on a wearable device, such as a bracelet or watch, and the alarm may be a vibration on the wearable device and / or an electric shock triggered by the wearable device. In one embodiment of the innovation, the local alarm may be provided simultaneously with and / or alternating with voice interaction.

[0106] FIG. 6 illustrates at least some of the components of one embodiment of a system that can be used to monitor a person. In this embodiment, the sensor can be a radar-based sensor 601 configured to monitor a room 604 and track people within the room. The sensor and system obtain measurement data from the sensor 601 and track targets, e.g., people, within the monitored area. Based on this, the sensor or system can assess the situation within the monitored area and determine when the safety of the monitored person 602 is at risk. Based on determining that the monitored person's safety is threatened, the system can initiate a voice dialogue with the monitored person using a voice generation means 605 and listen to a response from the monitored person 602 using a voice capture means. The sensor or system can use voice synthesis and voice recognition in the voice dialogue with the monitored person. Based on the captured voice, the sensor or system can assess the situation and determine whether action, such as an alarm, is necessary. If an alarm is necessary, the alarm can be delivered to a designated recipient, e.g., an organization monitoring the person and / or their health.

[0107] FIG. 7 illustrates an example of a large language model (LLM) initiation according to an exemplary embodiment of the present invention, in which the LLM is used to conduct a voice dialogue with a monitored person. In this exemplary embodiment, a prompt is presented that is used to initiate a dialogue between the monitored person (John) and the large language model (LLM). In the example below, the name "John" can be replaced with the name of the monitored person, and the command "SAY" can be replaced with other commands, such as those described above. The dialogue performed by the large language model (LLM) can be initiated, for example, by describing the current situation based on observations from at least one sensor. In this example embodiment, the phrase "The fall detector makes an alarm" is used to describe the situation in the large language model (LLM). The example initiation prompt in FIG. 7 is as follows: You are a virtual nurse. You are caring for an elderly man called John who lives alone in an apartment with a fall detector. When the fall detector sounds an alarm, you ask John if he needs help. If the answer is yes, or if there is no answer, you need to sound an alarm and let John know help is on the way. If the answer is no, you need to tell John to keep going. Everything you say to John must be preceded by the command SAY. For example: SAY Need help? The ALARM command generates an alarm. The fall detector will sound an alarm.

[0108] In one embodiment of the present invention, the sensor and / or system is configured to provide a local alarm until it is determined that the person has moved, woken up, and / or started breathing again. In one embodiment of the present invention, a remote alarm is provided if the person does not respond to the local alarm, for example, if the person does not respond to the local alarm and engage in voice interaction after a predetermined time, does not move, does not wake up, and / or does not start breathing.

[0109] In one embodiment of the present invention, the sensor and / or system is configured to provide a local and / or remote alarm or notification if communication or electrical connection from the system to the sensor is lost, if the sensor is removed from its monitored or installed location, and / or if communication and / or electrical connection from the sensor is severed. In this way, the system and / or sensor can indicate, for example, a situation where a monitored person has removed the sensor or someone is attempting to steal the sensor.

[0110] In one embodiment of the present invention, the sensor is placed on a stand, floor, ceiling or wall of a room in a home or hospital environment, for example next to or above a bed so that the measurement area of ​​the sensor covers at least a portion of the bed and / or a person lying on the bed.

[0111] In one embodiment of the present invention, the sensor comprises means for detecting the orientation of the sensor, such as an acceleration sensor, and the sensor or system is configured to take the detected sensor orientation into account when determining the presence, position, movement and / or posture of the monitored person, such as by correcting measurements based on the detected orientation.

[0112] In one embodiment of the invention, the sensor comprises a battery configured to provide energy to the sensor. In one embodiment of the invention, the sensor comprises a mains power supply configured to provide energy to the sensor and / or the battery.

[0113] In one embodiment of the invention, the sensor comprises a mounting structure on which the sensor can be placed, the mounting structure being fixable to a stand, a wall or a ceiling. In one embodiment of the invention, the sensor is removable from the mounting means without tools, for example to charge the battery of the sensor. The sensor or the mounting structure for the sensor can be placed on a stand or a wall, for example at a height of 1.5 m or more above floor level.

[0114] In one embodiment of the present invention, the sensor is configured to analyze the measurement signal by at least filtering the measurement signal such that the phase of the measurement signal is determined in order to observe the person's movements, such as heartbeat and / or breathing.

[0115] In one embodiment of the present invention, the sensor is configured to detect a fall and / or sitting position of a person according to a determined height of the person, such that, for example, if the height of the person is below a certain threshold height value, it can be determined that the person has fallen.

[0116] In one embodiment of the present invention, a system comprises at least two of the above-mentioned sensors of the present invention, and the system is configured to detect and measure people in a monitored area based on measurement signals of the at least two sensors, which can monitor the same area and / or different areas.

[0117] In one embodiment of the invention, the sensor and / or sensor system comprises at least one light source, for example an LED light source, and the sensor is configured to activate the light source when the sensor observes a standing person, for example at certain times of day and / or when the light level in the monitored area is low. The sensor or system may comprise means for measuring the light level in the monitored area.

[0118] It will be clear to those skilled in the art that the different embodiments of the present invention are not limited to the examples described above, but can therefore vary within the scope of the claims presented below.Furthermore, characteristic techniques that may be presented in this specification in combination with other characteristic techniques can also be used separately from each other, if necessary.

Claims

1. 1. A system for observing the presence, position, movement and / or posture of a person in a surveillance area, comprising: at least one sensor (101, 201, 301, 401, 601), means for processing measurement signals of said sensor, such as measurement electronics, and means for communicating measurement results and / or data related to said measurement results for further processing, The system further comprises means (105, 205, 405, 605) for generating and capturing sound; The system comprises: Using at least one sensor (101, 201, 301, 401, 601), determine that the safety of the monitored person (206, 402, 602) is threatened; Based on the safety of the monitored person (206, 402, 602) determined to be threatened, using the means for generating audio to initiate an audio dialogue with the monitored person (206, 402, 602) and using the means for capturing audio to listen to a response from the monitored person (206, 402, 602); Evaluating the situation and determining whether an action, such as an alarm, is required based on the captured audio. It is configured as follows: The system is configured to use a large language model (LLM) to conduct the voice interaction with the monitored person. characterized in that system.

2. The system of claim 1 , wherein the system is configured to provide a starting prompt to the large-scale language model that is relevant to the context of the monitored person and / or the monitored area.

3. 3. The system of claim 1, wherein the context-related starting prompt includes at least one of information about the role and task of the large language model (LLM) in the voice interaction, information about the person being monitored such as name, physical condition and level of assistance needed, a current situation as determined by the at least one sensor, conditions that warrant an alarm, and instructions on how the LLM should interact with other parts of the system.

4. the means for generating speech includes a text-to-speech synthesis algorithm that converts text output from the system into audible speech for the monitored person; the means for capturing audio includes a speech recognition algorithm for converting audible speech from the person being monitored into text input for the system; 10. A system according to any one of the preceding claims, wherein the text output of the system is connected to an input of the large scale language model (LLM) and the text input of the system is connected to an output of the large scale language model (LLM).

5. 10. The system according to any one of the preceding claims, wherein the system comprises at least one sensor or a plurality of sensors (101, 201, 301, 401, 601) of the following: a radar sensor, a floor sensor, a motion detector and / or a camera.

6. 10. A system according to any one of the preceding claims, wherein the means for generating sound is a speaker and / or the means for capturing sound is a microphone or a microphone array.

7. The system of any one of the preceding claims, wherein the system is configured to determine that the safety of the monitored person (206, 402, 602) is at risk if the system determines that the monitored person (206, 402, 602) has fallen, is lying on the floor, is staying in a particular part of the monitored area, such as a bathroom or bed, is not eating or exercising, and / or is moving to a particular part of the monitored area, such as a balcony in winter.

8. 10. The system of claim 9, wherein the initiation of the voice interaction comprises a question or a suggestion to the monitored person.

9. 10. The system of any one of the preceding claims, wherein the action comprises sending a notification or alarm containing a description of the situation and / or the interaction content.

10. 1. A method of observing the presence, position, movement and / or posture of a person in a surveillance area using a system, comprising: said system comprising at least one sensor (101, 201, 301, 401, 601), means for processing measurement signals of said sensor, such as measurement electronics, and means for communicating measurement results and / or data related to said measurement results for further processing, The system further comprises means (105, 205, 405, 605) for generating and capturing sound; In the method, Using the at least one sensor (101, 201, 301, 401, 601), determine that the safety of the monitored person (206, 402, 602) is threatened; Based on the safety of the monitored person (206, 402) determined to be threatened, using the means for generating audio to initiate an audio dialogue with the monitored person (206, 402, 602), and using the means for capturing audio to listen to a response from the monitored person (206, 402, 602); assessing the situation based on the captured audio and determining whether an action, such as an alarm, is required; Conducting the spoken dialogue with the monitored person using a large language model (LLM). A method characterized by:

11. The method of claim 10 , further comprising providing the large-scale language model with a starting prompt that is relevant to the monitored person and / or the context of the monitored area.

12. 12. The method of claim 10 or 11, wherein the starting prompt associated with the context information includes at least one of information about the role and task of the large language model (LLM) in the voice interaction, information about the person being monitored such as name, physical condition and level of assistance needed, a current situation as determined by the at least one sensor, conditions that warrant an alarm, and instructions on how the LLM should interact with other parts of the system.

13. 13. The method of any one of claims 10 to 12, wherein the means for generating audio comprises a text-to-speech synthesis algorithm that converts text output from the system into audible speech for the person being monitored, and the means for capturing audio comprises a speech recognition algorithm that converts audible speech from the person being monitored into text input for the system, and the text output of the system is connected to an input of the large-scale language model (LLM), and the text input of the system is connected to an output of the large-scale language model (LLM).

14. The method according to any one of claims 10 to 13, wherein the system comprises at least one sensor or a plurality of sensors (101, 201, 301, 401, 601) of the following: a radar sensor, a floor sensor, a motion detector and / or a camera.

15. 15. The method of any one of claims 10 to 14, wherein the means for generating sound is a speaker and / or the means for capturing sound is a microphone or a microphone array.

16. 16. The method of claim 10, wherein the system determines that the safety of the monitored person is at risk if the monitored person falls, lies on the floor, stays in a particular part of the monitored area, such as a bathroom or bed, does not eat or exercise, and / or moves to a particular part of the monitored area, such as a balcony in winter.

17. The method of any one of claims 10 to 16, wherein the initiation of the voice interaction comprises a question or a suggestion to the monitored person.

18. The method of claim 10 , wherein the action comprises sending a notification or alarm containing a description of the situation and / or the interaction.

Images