Monitoring device and system

Abstract

To provide a monitoring device and system that detects abnormal conditions and various pathological symptoms in residents' daily lives at an early stage, and monitors individuals to ensure they can live safely and securely. [Solution] The system comprises an imaging device having an event sensor that outputs an event signal only when the brightness change of each pixel of the resident's image exceeds a predetermined threshold, a resident terminal means including an audio terminal device, a body movement processing means in a cloud server that detects the resident's body movements using the event signal, and an audio processing means in a cloud server that detects stress levels or depressive tendencies based on the resident's audio output acquired by the audio terminal device. The system monitors the resident's health and / or abnormal situations by detecting the resident's physical functional state using the body movement processing means and detecting their mental functional state using the audio processing means.

Description

Technical Field

[0001] The present invention relates to a monitoring device and its system for early detecting abnormal conditions and various pathological symptoms in the daily life of single-living persons, especially the elderly living alone, and for monitoring them so that individuals can live safely and securely.

Background Art

[0002] Continuously grasping the health status of single-living persons, especially the elderly living alone, by a monitoring device and detecting pathological symptoms at an early stage is not only useful for personal health management and accident prevention, but also contributes to the labor-saving and efficiency improvement of the staff, caregivers, relatives, security guards and managers who monitor in nursing / medical facilities and security companies. Further, if the video data output of such a monitoring device can be acquired remotely, remote monitoring and remote medicine can be performed more easily, and it can also be applied to pathological prediction and health diagnosis of single-living persons.

[0003] In response to such problems, various technologies for grasping the living status of residents by monitoring cameras, various sensors, etc. as technologies for monitoring the daily life status and sleeping status of the monitored persons are disclosed. In Patent Document 1, a technology for monitoring daily life by an interactive speaker through a network is disclosed. Further, in Patent Document 2, a technology for determining the sleep depth from the change in body movement in the sleeping state by a monitoring camera is disclosed.

[0004] Many of these conventional technologies focus on monitoring residents' physical functions, and continuous monitoring and data acquisition of mental functions, despite their interrelationship, are rarely performed. One reason for this is that monitoring mental function requires continuous monitoring and the collection of large amounts of data, which is a major obstacle. Continuous monitoring using video cameras requires the transmission of video data, and the sheer volume of data makes it unsuitable for remote centralized management and monitoring. Furthermore, most technologies are aimed at monitoring physical functions, such as detecting abnormal situations primarily through motion detection, and no technology has been demonstrated that comprehensively and continuously grasps and resolves abnormal situations in conjunction with mental function, including pathological prevention and prediction. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2022-089113 [Patent Document 2] Japanese Patent Publication No. 2018-164415 [Overview of the project] [Problems that the invention aims to solve]

[0006] This invention is provided in view of the above circumstances and primarily solves the following problems. (1) To enable early detection of abnormal situations and health conditions of residents, as well as prediction of pre-illness conditions. (2) Minimize the amount of image data required for monitoring residents, and facilitate continuous monitoring and remote monitoring through data transmission. (3) To comprehensively monitor residents' health by detecting their physical functional status as well as their mental functional status, such as their stress levels and depressive tendencies. [Means for solving the problem]

[0007] To solve these problems and achieve the above objectives, the monitoring device and system of the present invention is a monitoring device and system for monitoring a resident, wherein the monitoring device and system includes a resident terminal means including an imaging device that is placed near the resident and has an event sensor that outputs an event signal only when the brightness change of each pixel of the resident's image exceeds a predetermined threshold, and a body movement processing means in a cloud server that detects the physical functional state of the resident, including body movements, using the event signal, wherein the imaging device and the body movement processing means are connected via a communication network, and the health status and / or abnormal situations of the resident are monitored by the detection of the resident's physical functional state by the body movement processing means.

[0008] Furthermore, the monitoring device and system in the present invention is a monitoring device and system for monitoring a resident, and the monitoring device and system is configured to include a resident terminal means including an imaging device and an audio terminal device, which are placed near the resident and have an event sensor that outputs an event signal only when the brightness change for each pixel of the resident's image exceeds a predetermined threshold, an imaging device and an audio terminal device, a motion processing means in a cloud server that detects the resident's physical functional state, including body movements, using the event signal, and an audio processing means in a cloud server that detects the resident's mental functional state, including stress levels or depressive tendencies, based on the voice output of the resident acquired by the audio terminal device, wherein the imaging device, the audio terminal device, the motion processing means and the audio processing means are connected via a communication network, and the health status and / or abnormal situations of the resident are monitored by the detection of the resident's physical functional state by the motion processing means and the detection of the resident's mental functional state by the audio processing means.

[0009] Furthermore, the monitoring device and system in the present invention can also be configured such that the output data of the event signal indicates the coordinates of the pixels, the time, and the change in brightness.

[0010] Furthermore, the monitoring device and system in the present invention may also be configured such that the monitoring device and system for watching over the resident has monitoring terminal means including an image display device and an audio terminal device on the monitoring device's side, and the monitoring terminal means is connected to the motion processing means and audio processing means in the cloud server via the communication network, and the monitoring device is configured to monitor the resident's health condition and / or abnormal situations based on the detected physical and mental functional state of the resident.

[0011] Furthermore, the monitoring device and system in the present invention may also be configured such that the motion processing means generates an event image by summarizing the event signals in a predetermined integral time unit shorter than the period of one frame of the image display device, and based on the generated event image, detects the health status and / or abnormal situation based on changes in the resident's body movements, and if an abnormal situation occurs, an alert signal is generated to the monitoring terminal means.

[0012] Furthermore, the monitoring device and system in the present invention can also be configured to detect the resident's movements, daily physical activity, sleep duration, duration of physical activity, and duration of stillness based on the event signal, and to monitor the resident's health condition.

[0013] Furthermore, the monitoring device and system in the present invention may also be configured so that, based on the event signal, the motion processing means estimates an abnormal situation when the resident becomes stationary beyond a predetermined range or when the shape of a third party or object different from the predetermined shape of the resident is detected.

[0014] Furthermore, the monitoring device and system in the present invention may also be configured such that the voice terminal devices of the resident terminal means and the monitor terminal means are composed of two-way voice terminal devices, and the mental functional state of the resident and / or the monitor is detected based on the voice of the conversation between the monitor and the resident.

[0015] Furthermore, the monitoring device and system in the present invention may also be configured to include a storage and learning means for output data of the event signal and audio signal within the cloud server, and to store and learn the output data of the body motion processing means and audio processing means to detect the physical and mental functional state of the resident, or the mental functional state of the monitor. [Effects of the Invention]

[0016] According to the present invention, an event sensor camera is used to output and transmit an event signal only when the brightness change for each pixel exceeds a predetermined threshold, thereby enabling continuous monitoring of the resident's daily life with an extremely small amount of data. As a result, it becomes possible to acquire the resident's living status from a remote location using an event sensor camera and transmit the acquired event signals over a communication network for continuous monitoring and centralized management. Furthermore, by analyzing body movements using the acquired event signals, it is possible not only to understand daily movements and sleep patterns, but also to automatically detect abnormal situations such as falls, falling out of bed, and intrusion by third parties, and generate alerts.

[0017] Furthermore, by acquiring and analyzing the voice content of residents using voice terminal devices, it is possible to determine the mental functional state of residents, such as stress levels and depressive tendencies, enabling comprehensive monitoring in conjunction with their physical functional state. This two-way voice terminal device can also be used to analyze the stress levels and depressive tendencies of the monitor based on the content of conversations between residents and monitors. Moreover, by continuously acquiring, memorizing, and learning the physical and mental functional states of residents, it becomes possible to detect health problems and abnormal conditions in elderly individuals at an early stage. [Brief explanation of the drawing]

[0018] [Figure 1] This is an explanatory diagram showing an example configuration of the monitoring device and system of the present invention. [Figure 2] This is an explanatory diagram showing the event signal of the present invention. [Figure 3]It is an explanatory diagram showing the detection of the movement of a resident by an event signal of the present invention. [Figure 4] It is a block explanatory diagram regarding an example of a body movement processing process according to the present invention. [Figure 5] It is an application example for detecting the physical function state from the walking state of the present invention. [Figure 6] It is an explanatory diagram of an example of sleep state detection according to the present invention. [Figure 7] It is a block explanatory diagram regarding an example of an audio processing process according to the present invention.

Mode for Carrying Out the Invention

[0019] Hereinafter, embodiments of a monitoring device and a system according to the present invention will be described in detail with reference to the drawings. Note that any of the explanatory diagrams and drawings described in the following examples are drawn as schematic or schematic diagrams for explaining the present invention, and the actual dimensions and shapes are not particularly limited. Also, the system configuration, block diagram, dimensions, materials, shapes, relative arrangements, and usage examples used in the examples are not intended to limit the technical scope of the invention only to those unless otherwise specified.

[0020] FIG. 1 is an explanatory diagram showing a configuration example of a monitoring device and a system of the present invention. An imaging device 2 having an imaging range capable of observing the daily behavior state of a resident 1 and an audio terminal device 3 capable of collecting the audio of the resident 1 are installed. The imaging device 2 includes an event sensor that outputs an event signal (or event data) only when the luminance change of each pixel of the body movement image of the resident 1 exceeds a predetermined threshold value. The output of the imaging device 2, which will be described in detail later, captures the movement and movement of the resident 1 and other objects in the imaging range within the image as an event signal of independent pixel information in conjunction with the passage of time and outputs it.

[0021] The imaging device 2 and the voice terminal device 3 are connected to the resident terminal means 4 by wireless (short-range wireless communication, etc.) or wired connection. In other words, the resident terminal means 4 is composed of the imaging device 2 and the voice terminal device 3 and is connected to the supervisor terminal device 6 and the cloud server 7 via a communication network 5 such as the Internet WiFi. The supervisor terminal device 6 is equipped with an image display (monitor) device 9 and a voice terminal device 10 located near the supervisor 8. Preferably, the voice terminal device 3 on the resident side and the voice terminal device 10 on the supervisor 8 side are two-way voice terminal devices that allow the resident 1 and the supervisor 8 to communicate with each other.

[0022] This monitoring device and system will be described using an example configuration between a resident terminal means 4 including an imaging device 2 and an audio terminal device 3 according to the present invention, and a monitor terminal means 6 including an image display device 9 and an audio terminal device 10. However, it can also function as a monitoring device and system for the resident 1 only, without requiring the monitor terminal means 6.

[0023] Here, the monitoring terminal means 6 is assumed to be installed in a nurse's center or central control room that monitors Resident 1 away from Resident 1's residence. However, it may also be a monitoring terminal means 6 owned by a security company that monitors Resident 1, relatives, related parties, or the building manager, not just caregivers or medical personnel. Furthermore, this communication network 5 can be constructed using any form of means, whether wired or wireless.

[0024] The image sensor 2 is shown positioned on the ceiling (upper part), but it can also be positioned on a wall, or mounted on both the wall and the ceiling. Furthermore, it can be positioned anywhere as long as it can monitor the movements of occupant 1 within the room. It is also possible to install image sensor 2 in multiple locations to monitor even more subtle changes in body movement.

[0025] The audio terminal devices on the resident 1 side and the supervisor 8 side are preferably two-way speakers (also called conversational speakers or smart speakers), and must be positioned to pick up the voices of resident 1 and supervisor 8. It is also possible to integrate the audio terminal device 10 on the resident 1 side with the imaging device 2, or to place a small microphone such as a lavalier microphone together with resident 1 and transmit to the resident terminal means 4 via short-range wireless means.

[0026] The cloud server 7 is equipped with a monitoring platform 11. This monitoring platform 11 consists of a program that constantly analyzes and observes the movements and voice of resident 1, understands and records their sleeping and sleep status, and generates an alert if an abnormal situation occurs.

[0027] The monitoring platform 11 includes a motion processing means 12 and an audio processing means 14 that detect the movement of the resident 1 using the event signal output from the imaging device 2. The motion processing means 12 generates an event image by summarizing the event signal in a predetermined integration time unit. Here, the predetermined integration time unit is formed in units shorter than the duration of one frame of the monitor display image. Based on the generated event image, changes in the resident 1's movement are calculated and analyzed, and abnormal conditions are detected. If an abnormal situation occurs, an alert signal is issued to the monitor terminal means 6 by the alert means 13. This alert signal can also be configured to be issued to the resident terminal means 4.

[0028] Furthermore, the voice processing means 14 of the monitoring platform 11 has the function of analyzing and processing changes in the stress level and psychological tendencies of resident 1 and / or monitor 8, including changes in sound pressure, frequency changes, voice feature points, and abnormal keyword extraction, based on the voice content of resident 1 and / or monitor 8.

[0029] Furthermore, this monitoring platform 11 is equipped with memory and learning means 15, which stores and learns the physical movement characteristics and sleep characteristics of resident 1, as well as data on changes in psychological tendencies, including stress levels and depressive tendencies, of resident 1 and / or supervisor 8, processed by the voice processing means 14, and feeds this back to the physical movement processing means 12 and the voice processing means 14 to analyze and compare changes in resident 1's physical movement and psychological tendencies. The stored record logs are accumulated for the analysis of changes in resident 1's physical movement and sleep state, and for the analysis of the psychological tendencies of resident 1 and / or supervisor 8. By continuously storing and learning such physical and mental functional states, it is possible to grasp the correlation between mental functional states and behavioral abnormalities or changes, as well as the correlation with cognitive function.

[0030] In commonly used video cameras (or frame cameras), the camera acquires full-frame image data synchronized with a vertical synchronization signal, and then performs video processing by encoding inter-frame differences, compressing in real time, or extracting video changes at multiple frame intervals. However, the event camera in the present invention performs image processing that is completely different from such frame image synchronization cameras. The event data processing obtained from the event signal in the present invention captures the change in the brightness signal only of pixels where motion has occurred within the imaging range. Therefore, the output event signal is output as pixel coordinates (x, y), time (t), and brightness polarity or brightness change (p), as illustrated in Figure 2. In Figure 2, the coordinates (x1, y1) on the screen move from time t1 to coordinates (x4, y4) at time t4, and the brightness polarity at time t1 shows that the brightness signal is captured between (p1) and (p4).

[0031] As an event signal, only movements such as movement or changes in body movement of resident 1, or movement or vibration of any object are detected; stationary objects such as beds and backgrounds are not detected and do not transmit signals. If there is a change in the walking state, limb movements, turning over, head movement, or movement of bedding of caregiver 1, the movement trajectory of only the changed part is recorded. Here, the minimum high-speed time resolution t can be obtained down to about 1 μsec, but since detecting human body movement is sufficient, even at about 100 msec, it is possible to detect micro-movements caused by spasms of the mouth or limbs.

[0032] This event signal can be displayed as a movement trajectory image by grouping it into predetermined integration time units. Here, the integration time is set to be shorter than the duration of one frame, which corresponds to the frame rate.

[0033] The event signal outputs the pixel value of the brightness change at the moment movement occurs. Even if resident 1 is moving or sleeping, the background and bed are stationary and therefore do not acquire data, and only the brightness change due to resident 1's movement is extracted as output data. As a result, the amount of output data is significantly reduced compared to frame-based images (frame camera images). Transmitting event signals not only significantly reduces costs, but also eliminates the need for processing circuits such as bandwidth, memory capacity, noise processing, and image difference processing in video signal processing, allowing for a relatively simple circuit configuration. For example, compared to a frame camera image signal compressed by inter-frame difference processing in a frame image, the amount of data transmitted by the event data signal can be reduced to more than 1 / 100 to 1 / 500 (approximately 1 / 1000 of the frame image when detecting resident 1's body movement in a room with little background movement).

[0034] Figures 3(A) and 3(B) are explanatory diagrams illustrating examples of motion detection of resident 1 using event signals according to the present invention. Figure 3(A) shows an example of frame camera images (frame camera images) showing resident 1 moving from T1 to T2 to T3. Figure 3(B) is an example of a monitor image (event image) of an event signal acquired by the event camera of the present invention. Since the event camera detects only moving parts, the background and bed in Figure 3(A), which are stationary, are not detected, and only the movement of resident 1 is detected as a change in coordinates (x, y) and brightness (p) (grayscale image) over time (t). Figure 3(B) is an example of an event image when the temporal progression of the event signal is displayed on a monitor. This event image captures the movement of resident 1 from a horizontal imaging device 2', but the same can be achieved with an imaging device installed overhead.

[0035] Since this event image only shows the change in brightness of the part where movement occurred, it is possible to identify resident 1 based on body movement and physical characteristics, but it is difficult to identify them by face shape as in face recognition with a video camera (frame camera), making it a detection method that takes privacy into consideration. The event signal, which is the output of such an event camera that detects only the movement of objects, is transmitted as continuous data of time (t), coordinates (x, y), and brightness (p) to the body movement processing means 12 in the cloud server 7 via the communication network 5.

[0036] Figure 4 is a block diagram illustrating an example of the motion processing process of the motion processing means 12. First, the motion processing means 12 generates an event image by summarizing the event signals transmitted from the imaging device 2, which has an event sensor, in a predetermined integration time unit shorter than the duration of one frame of the monitor display image. This predetermined integration time unit depends on how precisely the movement and vibration velocity of the object to be detected are to be detected. A minimum high-speed time resolution of about 1 μsec can be obtained, but about 100 msec is sufficient for detecting human body movement. The event image summarized in this integration time unit can be further averaged over a predetermined rough time unit (for example, a few seconds) to be captured as the amount of body movement.

[0037] The event signal transmitted from the imaging device 2 is processed by the image processing means 41 to identify the overall shape of the person and the shape of each part of that person. In Figure 3(B), the overall shape of the person is shown as a frame A representing the shape of resident 1 moving as a whole, and each part is shown as a frame, with the face part a1, the feet part a2, and the hands part a3 each represented by a frame. This identification of the overall shape and part shapes is performed by identifying the necessary parts for the purpose of detecting body movement. The image processing means 41 captures the change in pixel coordinates for each identified part.

[0038] The change in pixel coordinates is used by the calculation means 42 to calculate the amount of body movement (momentum) per hour of the resident 1 as described above, and to measure the distance traveled, direction of movement, travel time, stopping time, walking speed, stride length, etc. In addition, the body movement of each part is calculated individually in the same manner as needed. These measurement points and calculations are performed by selecting the necessary items according to the purpose.

[0039] Each calculated data (body movement data), including the amount of body movement calculated by the calculation means 42, is compared in the comparison analysis means 43 with pre-learned average values ​​for normal conditions or healthy individuals, and the presence or absence of deviations or abnormal values ​​from the normal conditions, average values, or the body movement data of the person with a disability is calculated. The body movement data is sent to the memory learning means 44 and stored as a record in the database 45. This database 45 stores not only the measured body movement data of resident 1, but also average data for healthy individuals and characteristic data for each person with a disability, which are retrieved and stored from the learning data means 46.

[0040] The output from the comparative analysis means 43 is sent to the anomaly detection means 47. If the deviation from normal conditions is abnormal, or if an abnormal situation corresponding to disability data is detected, the alert generation means 48 issues an alert signal to the monitor terminal means 6 of the monitor 8. The alert signal is sent to the monitor terminal means 6 and communicated by light (flashing, color change) or sound (buzzer, voice). In addition, regardless of whether an abnormal situation has occurred, the output from the anomaly detection means 47 is sent as body movement data to the body movement data output means 49, which can be downloaded by the monitor terminal means 6 of the monitor 8 as needed or used as monitoring data.

[0041] The acquired body movement data can be used in various ways for monitoring resident 1, but the following will explain, as an example, (1) detection of walking difficulties, (2) detection of abnormal situations, and (3) monitoring of sleep state. In this invention, the physical functional state is not limited to the functional state shown in the above application examples, but refers to the change in the state of the resident, third parties, and moving objects detected using the event sensor.

[0042] This section describes the application of the present invention to gait disorder detection. It is medically known that detecting gait patterns can detect physical functional states and health conditions, as well as predict pre-disease states, such as locomotive syndrome, frailty, sarcopenia, Parkinson's disease, and progressive supranuclear palsy. Figures 5(A) and (B) show examples of applications for detecting physical functional states from gait. Resident 1 walks back and forth several times over a distance of approximately 5m, and this is captured by the event camera imaging device 2 and analyzed by the body motion processing means 12. Figure 5(A) shows an example of recording and observing walking over a specific section from above, showing how it deviated from the straight line of travel or the degree of lateral deviation. Figure 5(B) shows observation of vertical movement from a lateral position and walking posture. While it is preferable to regularly observe gait during morning meetings or exercise time, it is also possible to analyze gait by capturing movements in daily life without setting aside special observation time or measurement location.

[0043] To detect walking disorders, an event sensor camera acquires walking state data of resident 1, and the body motion processing means 12 acquires data on the time from the start of walking to the stepping of the foot, walking distance within a predetermined time, walking speed, vertical movement, lateral sway (asymmetry), vertical angle of the foot (shuffling detection), foot drop (toe lift angle), stride length, continuity, and acceleration using the calculation means 42. The acquired body motion data is compared and analyzed by the comparative analysis means 43 with data on walking disorders stored in the database 45, and monitoring is performed to determine whether there are any abnormalities in the physical function state or health state.

[0044] Next, we will explain how the present invention can be applied to anomaly detection. Examples of anomalies in monitoring in nursing and medical settings include fatal accidents, falls, falls from beds, and intrusion by third parties (or foreign objects such as animals). To detect these anomalies, first, the shape of resident 1 is memorized. Furthermore, it is desirable to additionally memorize resident 1's walking pattern. This walking pattern is memorized by storing the amount of change in characteristic points such as stride length, walking speed, forward lean, limb movements, and left-right and up-down sway as reference values, but this is not essential as the data is continuously accumulated by the memory learning means 44.

[0045] Among the abnormal situations, falls, accidents involving dropping, and resulting fatalities are of high risk. Early detection of abnormal situations is required not only in nursing and medical facilities, but also in cases of elderly people living alone or living by themselves. The body movement processing device 12 is configured to determine that an abnormal situation has occurred if the resident 1's body movement changes abruptly and remains still for a certain period of time (for example, 10-20 minutes or more), and an alert signal is issued. The monitor 8 checks on the resident 1's well-being through the two-way voice terminal device 3 on the resident 1's side using the two-way voice terminal device 10, or by visiting the resident 1 directly to check on their condition.

[0046] Furthermore, an alert signal can also be issued for an abnormal situation such as the intrusion of a third party or living organism such as an animal while the resident is asleep. In particular, if a third party enters the room in addition to resident 1 while the resident is asleep, it is possible to detect that someone else has entered in addition to resident 1 based on their approximate size and other characteristics, as people are always moving. Similarly, the intrusion of small animals such as animals or birds can also be detected because they are always moving. However, it is not possible to determine whether the person or animal other than resident 1 who has intruded constitutes an abnormal situation. Therefore, the detection of a third party's intrusion serves as a warning that there is a possibility of an abnormal situation, and if necessary, it is possible to activate two-way voice terminals 3 and 10 to determine whether the third party or animal constitutes an abnormal situation.

[0047] The event sensor camera according to the present invention can be used for various behavioral detection applications by taking advantage of its features: the amount of acquired image data is extremely small compared to frame cameras; it can acquire images with a temporal resolution of approximately 1 μsec, enabling high-speed detection; and it is difficult to recognize faces, thus preserving privacy. For example, it is suitable for monitoring multiple locations such as toilets, bathrooms, and bedrooms, where installation has been difficult despite the relatively high rates of falls and mortality. Furthermore, it is conceivable to install the event sensor camera according to the present invention in entrances and other locations to monitor abnormal behaviors such as repeatedly performing the same actions or wandering, which are characteristic of the elderly and those with cognitive impairments.

[0048] Furthermore, the system can also be used to monitor the sleep state of resident 1 by utilizing body movement detection. To understand the wakefulness, REM, and non-REM sleep states related to sleep quality, the amount of body movement at bedtime is detected and analyzed. To understand the amount of body movement of resident 1 using event sensor output, the calculation means 42 counts the total number of pixels for each predetermined period (T). While the American Academy of Sleep Medicine (AASM) international standard uses 30 seconds as one epoch period, an epoch period of 1 to 5 minutes is acceptable if detailed data is not required.

[0049] This body movement generally changes from an active state to an inactive state as the individual moves from a waking state to a REM state and then to a non-REM state. Since there are individual differences in body movement among people of all ages and genders, the body movement amount calculated from the event sensor output is stored as data by the memory learning means 44, and the quality of sleep in each sleep state is judged by comparing it with the average value of the stored data. In addition, the average of still time and the average of movement time during the same epoch period are obtained from the event data, and the sleep state is observed. The average of still time and the average of movement time are calculated by averaging the number of times the individual is still and the number of times they are moving, respectively, during the epoch period.

[0050] Figure 6 is a graph showing the changes in wakefulness, REM sleep, and non-REM sleep states during body movement, resting time, and movement time. From this graph, the trends for wakefulness, REM sleep, and non-REM sleep can be understood. As the sleep state progresses from wakefulness to REM sleep and then to non-REM sleep, body movement decreases, while the average resting time increases. Although there are many individual differences in this trend, generally, a sleep cycle tends to occur where the individual progresses from wakefulness to REM sleep and then to non-REM sleep, repeating this cycle two to three times, and then transitioning to non-REM sleep once before becoming awake again. In this way, based on event data, the average values ​​of body movement, resting time, and movement time can be obtained, and from the changes in these values, it is possible to determine whether a sleep cycle is occurring, whether a non-REM (deep sleep) state is being maintained, or whether the individual is in a light sleep state. This understanding of sleep states is used for daily health management, etc., by acquiring the output data from the body movement data output means 49 as daily recorded data.

[0051] Next, we will explain how the motion processing means 12 and the voice processing means 14 can be used in combination. The motion processing means 12 primarily detects the movements (body movements) of the resident 1, while the voice processing means 14 makes it possible to perform monitoring detection that takes into account the mental state of the monitor 8 in addition to the resident 1.

[0052] Monitoring Resident 1 requires proactively identifying and preventing mental functional disorders such as stress, depressive tendencies, delusions, and hallucinations, as well as sleep disorders, physical functional disorders such as sarcopenia, locomotive syndrome, frailty, and Parkinson's disease, and even combinations of these, resulting in verbal and physical aggression, wandering, and ultimately cognitive impairment (cognitive dysfunction). These mental disorders, sleep disorders, physical functional disorders, and cognitive dysfunctions are believed to be caused by changes in body movement as well as acquired organ damage or functional decline in the brain. From these findings, mental and physical functional states are closely related, and in particular, dementia and cognitive decline can be reliably and quickly identified by continuously acquiring data on both mental and physical functions. Therefore, this invention proposes early detection of abnormal situations and health conditions related to various monitoring situations, as well as prevention of illness, by incorporating both physical and mental functional state detection.

[0053] Figure 7 is a block diagram illustrating the configuration of a voice processing means 14 that detects the mental functional state, including psychological stress levels and depressive tendencies, based on the voices and conversation content of resident 1 and supervisor 8. To detect the mental state from the voices and conversation content, voice feature extraction and keyword extraction are performed. Voice features include changes in parameters such as voice pitch (frequency), pitch period, and voice power of conversations and speech, which are known to indicate a person's level of psychological stress. Keywords include words and phrases that represent emergency / abnormal situations, abusive language, abuse, etc. Here, mental functional state is illustrated by the example of detecting psychological stress levels and depressive tendencies, but is not limited to these and refers to various symptoms and functional declines / impairments detected by changes in the above voice parameters, including related cognitive decline tendencies.

[0054] Conversations and voices between resident 1 and monitor 8 are recorded by microphones in voice terminal devices 3 and 10 and input via the communication network 5 to the voice processing means 14 in the monitoring platform 11 stored in the cloud server 7. The signals transmitted via the communication network 5 are voice signals and do not have a large data capacity, but silent sections of voice may be removed or the voice data may be compressed before transmission. Furthermore, it is desirable to use two-way speakers, such as smart speakers (AI speakers), for the voice terminal devices 3 and 10.

[0055] The input audio signal is divided into fixed intervals (frames), and multiple audio frames are sequentially stored in the frame recording unit 71. The number of frames is desirable to be as large as possible depending on the clarity of the audio input and the recording capacity, but it is not particularly limited. Audio frames stored for a predetermined period are overwritten and updated when the storage capacity reaches the limit area. Silent portions, noise, background sounds, and other parts of the stored frames that do not contain speech are deleted or discarded, and only the speech portion is extracted and sent to the feature quantity extraction unit 72 and keyword extraction unit 73 of the audio processing means 14.

[0056] The feature extraction unit 72 extracts features for each input frame. These features include parameters such as pitch (frequency), pitch period, and voice power (sound pressure) of the voice input. These parameters are known to correlate with changes in emotions such as joy, anger, sadness, and happiness, and although there are individual differences, voice pitch, pitch period, and voice power change with emotional fluctuations. Therefore, by comparing these with features of a previously acquired sample of the subject's normal speech, the subject's emotional state can be determined. When resident 1 and supervisor 8 converse using the two-way voice terminal devices 3 and 10, it is desirable to ask the same questions and engage in conversation each time, such as during morning meetings or regular health checkups, and to ask content questions (for example, "What did you do yesterday?") that allow for as much of resident 1's conversation content as possible, rather than simply asking "yes" or "no" questions.

[0057] It is desirable to acquire voice samples of the subjects, such as resident 1 and supervisor 8, in advance and store their features in the feature database 74. However, it is also possible to acquire sample voices of the subjects while the system is running for a certain period, store them in the recording and learning means 75, extract features, and store them in the feature database 74 for use.

[0058] The voice features of the subject extracted by the feature extraction unit 72 are sent to the occurrence amount calculation unit 76 to calculate the trend (vertol) of the voice features. The features extracted for each predetermined frame (voice pitch, pitch period, voice power, etc.) are compared with the feature database 74, and at the same time, the difference between each frame is obtained to determine how they are changing over time. In other words, the amount of change is calculated to determine whether the voice pitch, volume (power), and pitch period are changing instantaneously (suddenly), or whether the vector change is significantly different from normal.

[0059] Meanwhile, the keyword extraction unit 73 extracts keywords correlated with emotions such as joy, anger, sadness, and happiness from the audio of the input frames. These keywords are primarily extracted from slanderous language, such as abusive language, insults, attacks on physical or psychological defects, prejudice, harassment, and venomous remarks, especially in the context of caregiving and nursing. However, they are not limited to these, and also include representative keywords for emotions such as joy, anger, sadness, and happiness. A language dictionary database 76 can also be used for this keyword extraction.

[0060] The language dictionary database 76 has pre-prepared speech patterns for specific languages, such as slanderous language, and languages ​​in areas specified by emotions such as joy, anger, sadness, and happiness. While such speech patterns can also be extracted using general-purpose standard speech patterns used in text-to-speech conversion, it is preferable to acquire a sample of the subject's speech beforehand and register it in the language dictionary for more reliable extraction. Furthermore, similar to the sample acquisition for feature extraction of the subject described above, it is also possible to acquire a sample of the subject's speech while the system is running for a certain period, store it in the memory learning means 75, extract related keywords, and store them in the language dictionary database 77 for use. The keyword extraction unit 73 extracts keywords from resident 1 and / or monitor 8 in specified areas such as slanderous language and emotions such as joy, anger, sadness, and happiness. The keywords extracted by the keyword extraction unit 73 are sent to the frequency calculation unit 78, where the frequency and percentage of each keyword are calculated for each area.

[0061] The output data from the feature occurrence calculation unit 76 and the output data calculated by the keyword frequency calculation unit 78 are sent to the likelihood calculation unit 79. The likelihood calculation unit 79 calculates the trend and frequency over a relatively long period using the feature changes and trends, which are the output data from the occurrence calculation unit 76, and the keyword frequency, which is the output data from the frequency calculation unit 78. The feature extraction and keyword extraction described above are performed for each frame and are instantaneous data, but since the emotions and stress levels of the subjects can rise and fall instantaneously (rapidly), it is necessary to grasp this frame-by-frame data over a predetermined period longer than the frame interval (for example, every few minutes). The likelihood calculation unit 79 extracts the data for each frame obtained by voice feature extraction and keyword extraction as data that has been accumulated or averaged over a predetermined period. The voice feature data and keyword data obtained in this way by the likelihood calculation unit 79 are sent to the stress level estimation unit 80.

[0062] The stress level estimation unit 80 levels the emotional fluctuations of the target person requiring monitoring or caregiver, the emotional differences from the normal care / nursing situation, and abnormal situations based on changes, levels, and trends in speech feature data and keyword frequency data, and estimates the degree of accumulated psychological stress or the likelihood of abnormal situations occurring. This leveling of psychological stress is reference data that can be used for comparative estimation by leveling the presence or absence of deviation from the psychological stress in normal care / nursing, and is not intended to perform a medical stress diagnosis.

[0063] The quantification of psychological stress according to the present invention indicates a deviation from normal conditions. By understanding the subject's stress, depressive tendencies, cognitive decline, and other emotional fluctuations in conjunction with keyword frequency, it becomes possible to reduce the likelihood of an abnormal situation occurring. For example, in a situation where psychological stress is heightened or emotional fluctuations change significantly during caregiving or nursing, if abusive language is frequently used as a keyword, some kind of abnormal situation can be inferred between resident 1 and monitor 8. In this case, an alert signal is sent from alert means 13 to monitor terminal means 6, and inquiries can be made via two-way voice terminal devices 3 and 10, or monitor 8 can be dispatched to monitor resident 1.

[0064] Furthermore, this monitoring system can be configured to issue an alert signal if an emergency keyword (for example, "Help!", "Someone come!", or "Screaming") is extracted using the keyword extraction function, even if the emotional fluctuations in the voice features do not indicate an abnormal situation. This suggests that an emergency may have occurred for the subject. By incorporating this emergency keyword detection function, it is possible to perform daily monitoring of residents such as those living alone or elderly people living alone.

[0065] In this invention, large-capacity databases such as the feature database 74, memory and learning means 75, and language dictionary database 77 in the voice processing means 14, as well as voice analysis processing programs that require high processing speed, are all stored on the cloud 7. Therefore, the two-way voice terminal devices 3 and 10 do not require any special specifications or programs on the terminal side, and simply by connecting to the monitoring platform 11 in the cloud, they can monitor the resident 1 and understand the emotional fluctuations, stress levels, and depressive tendencies of the resident 1 and the caregiver 8 during care and nursing.

[0066] The apparatus and system of the present invention detect abnormal situations such as falls (tripping), falls from beds, deaths in isolation, and intrusions by third parties or animals by detecting the physical function state of the resident, and grasp the resident's daily health status. At the same time, it can also grasp the mental function state, including stress levels and depressive tendencies, of the resident and the caregiver. Therefore, it becomes possible to not only monitor the behavior of the resident but also to comprehensively monitor and grasp their health status in conjunction with their mental state, enabling early detection of signs of dementia such as decreased rational thinking ability, sharpening of personality, wandering, sleep disorders, and paranoid delusions, as well as verbal and violent behavior during care and monitoring. Furthermore, by storing and accumulating this data, it can be applied to understand mental disorders, abnormal behavior, and predispositions and signs of cognitive dysfunction. [Industrial applicability]

[0067] Such comprehensive monitoring devices and systems are not only suitable for monitoring single people and elderly people living alone, but can also be widely applied to situations where relatives, related parties, security companies, or building managers of apartments, etc., perform nighttime monitoring and sleep monitoring, in addition to medical and care staff and caregivers in the medical and nursing fields. This opens up a wide range of potential uses in various fields such as nursing care, medical care, welfare, and the security industry. [Explanation of Symbols]

[0068] 1 resident 2. Imaging device 3. Voice terminal device 4. Resident terminal means 5. Communication Network 6. Monitor terminal means 7. Cloud (Server) 8 Overseer 9. Image display (monitor) device 10 Voice terminal device 11. Monitoring Platform 12 Body motion processing means 13, 48 Alert methods 14. Audio processing means 15, 44, 75 Memory and Learning Methods 41 Image processing means 47 Anomaly detection means 80 Stress level estimation unit

Claims

1. In monitoring devices and systems for monitoring residents, The monitoring device and system comprises a resident terminal means including an imaging device positioned near the resident and having an event sensor that outputs an event signal only when the brightness change of each pixel of the resident's image exceeds a predetermined threshold, and a motion processing means in a cloud server that detects the resident's physical functional state, including body movements, using the event signal. The imaging device and the motion processing means are connected via a communication network, and the monitoring device and / or system are characterized in that the motion processing means detects the physical functional state of the resident and / or monitors the resident's health condition and / or abnormal situations.

2. In monitoring devices and systems for monitoring residents, The monitoring device and system comprises: an imaging device positioned near the resident and having an event sensor that outputs an event signal only when the brightness change of each pixel of the resident's image exceeds a predetermined threshold; resident terminal means including an audio terminal device; body movement processing means in a cloud server that detects the resident's physical functional state, including body movements, using the event signal; and audio processing means in a cloud server that detects the resident's mental functional state, including stress levels or depressive tendencies, based on the resident's audio output acquired by the audio terminal device. A monitoring device and system characterized in that the imaging device, the voice terminal device, the motion processing means, and the voice processing means are connected via a communication network, and the health status and / or abnormal situations of the resident are monitored by the detection of the resident's physical functional state by the motion processing means and the detection of the resident's mental functional state by the voice processing means.

3. The monitoring device and system according to claims 1 and 2, characterized in that the output data of the event signal indicates the coordinates of a pixel, the time, and the change in brightness.

4. On the side of the monitor who oversees the aforementioned residents, a monitor terminal means including an image display device and an audio terminal device is provided. The monitoring terminal means is connected to the motion processing means and the voice processing means in the cloud server via the communication network. The monitoring device and system according to claim 2, characterized in that the monitor monitors the health status and / or abnormal situations of the resident based on the detected physical and mental functional status of the resident.

5. The motion processing means generates an event image by summarizing the event signals in a predetermined integral time unit shorter than the period of one frame of the image display device, and based on the generated event image, detects the health status and / or abnormal situation based on changes in the resident's body movements. The monitoring device and system according to claim 4, characterized in that an alert signal is generated to the monitoring terminal means when an abnormal situation occurs.

6. The monitoring device and system according to claim 4, characterized in that it detects the resident's movements, daily physical activity, sleep duration, duration of physical activity, and duration of stillness based on the event signal, and monitors the resident's health condition.

7. The monitoring device and system according to claim 4, characterized in that, based on the event signal, the motion processing means presumes an abnormal situation when the resident becomes stationary beyond a predetermined range or when the shape of a third party or object different from the predetermined shape of the resident is detected.

8. The voice terminal devices of the resident terminal means and the supervisor terminal means are configured as two-way voice terminal devices. The monitoring device and system according to claim 4, characterized in that it detects the mental functional state of the resident and / or the monitoring device based on the audio of the conversation between the monitoring device and the resident.

9. The aforementioned cloud server is equipped with means for storing and learning output data from the event signal and audio signal. The monitoring device and system according to claim 2, characterized in that it stores and learns the output data of the body motion processing means and the voice processing means, and detects the physical and mental functional state of the resident, or the mental functional state of the monitor.

Images