Method and system for monitoring a subject - Patents.com

Abstract

To provide a method and a system for monitoring a subject.SOLUTION: A method for monitoring a subject comprises the steps of: monitoring, from a first angle of detection, a space in which the subject is located to obtain data of the subject from a first perspective view in accordance with the first angle of detection; converting the obtained data of the subject into a three-dimensional data set; processing the three-dimensional data set to create an image of the subject from a second perspective view, the second perspective view being created at a second angle different from the first angle of detection; and analyzing the created image from the second perspective view to determine, calculate, or select one or more characteristics relating to activity of the subject being monitored.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to the field of subject monitoring. In particular, but not exclusively, the present invention relates to systems and methods for monitoring one or more subjects for the purpose of identifying potential risks. [Background technology]

[0002] Various monitoring systems have been developed for use in detecting and monitoring the movement and / or activity of a subject, particularly for monitoring patients, disabled and / or elderly people in medical settings such as hospitals, nursing homes, senior centers, or for home medical applications with the aim of improving the safety of patients or residents. Conventional monitoring systems are mainly equipped with a camera or video camera located within a predefined distance and / or direction from the subject in space to capture an image or video of the subject. Optionally, conventional systems often require the subject to be provided with a pressure mat, a sensor pad and / or a wearable sensor to increase the sensitivity in detecting the subject's movement, for example to detect movement out of bed. Other monitoring systems may include sensors based on invisible barriers such as infrared fences that operate by transmitting and receiving infrared beams, and an alarm can be triggered when a "break" to the infrared beam is detected. The academic paper “Night-Time Monitoring System 9eNightLog) for Elder Wandering Behavior,Sensors 2021,21,704,Cheung,JC-W.,et.al. discloses a remote sensor-based system for monitoring elderly wandering behavior. However, Cheung’s sensor requires sensors to be mounted on the ceiling and above the bed of the subject being monitored. This limitation is not very suitable for sensors being applied in shared locations, especially hospitals, since bed settings are often changeable depending on the room usage and occupancy.

[0003] Thus, existing technologies are known to suffer from a lack of sensitivity and therefore accuracy of their detection. Also, applications are limited by their mounting locations and the range of detection of the sensors. In many cases, this results in undesirable false alerts. The use of cameras or video cameras to monitor objects may also raise privacy concerns, and therefore is generally not acceptable for use in places such as toilets and bathrooms where falls and other accidents are likely to occur. Summary of the Invention [Problem to be solved by the invention]

[0004] It is an object of the present invention to provide a method and system for monitoring an object.

[0005] It is an object of the present invention to alleviate or obviate to some extent one or more of the problems associated with known monitoring systems, or at least to provide a useful alternative.

[0006] The above mentioned object is achieved by means of the combination of features of the independent claims, while the dependent claims disclose further advantageous embodiments of the invention.

[0007] Those skilled in the art will derive other objects of the present invention from the following description. Thus, the foregoing description of objects is not intended to be exhaustive, but rather serves only to illustrate some of the many objects of the present invention.

[0008] In a first main aspect, the present invention provides a method for monitoring an object, comprising the steps of: monitoring a space in which the object is positioned from a first angle of detection to acquire data of the object from a first perspective according to the first angle of detection; converting the acquired data of the object into a three-dimensional data set; processing the three-dimensional data set to generate an image of the object from a second perspective, the second perspective being generated at a second angle different from the first angle of detection; and analysing the image generated from the second perspective to determine, calculate or select one or more characteristics relating to the activity of the object during monitoring.

[0009] In a second main aspect, the present invention provides a system for use in monitoring an object, comprising a sensor adapted to monitor, from a first angle of detection, a space in which the object is positioned to acquire data of the object from a first perspective according to the first angle of detection, a processing module adapted to convert the acquired data of the object into a three-dimensional data set and to process the three-dimensional data set to create an image of the object from a second perspective, the second perspective being created at a second angle different from the first angle of detection, and an analysis module adapted to analyze the image created from the second perspective to determine, calculate or select one or more characteristics relating to the activity of the object during monitoring.

[0010] The summary of the invention does not necessarily disclose all of the features essential to defining the invention, and the invention may exist in any partial combination of the disclosed features. [Brief description of the drawings]

[0011] The above and further features of the invention will become apparent from the following description of preferred embodiments, which are given by way of example only in connection with the accompanying drawings, in which:

[0012] [Figure 1] 1 is a block schematic diagram illustrating a system for monitoring an object according to the present invention; [Diagram 2] 2 is a flow diagram illustrating a method of monitoring an object implemented in the system of FIG. 1. [Diagram 3] 1 is a depth image showing the pose or movement of an object from a first perspective view. [Figure 4] 11 is a depth image showing the pose or movement of a subject from a second perspective view. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The following description illustrates preferred embodiments by way of example only and without limitation to the combination of features required to put the invention into practice.

[0014] References herein to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with this embodiment is included in at least one embodiment of the invention. The phrase "in one embodiment" appearing in various places in this specification do not necessarily all refer to the same embodiment, nor are they mutually exclusive separate or alternative embodiments. Furthermore, various features are described that may be exhibited in some embodiments but not in other embodiments. Similarly, various features are described that may be requirements for some embodiments but not for other embodiments.

[0015] It should be understood that the elements illustrated in the figures may be implemented in various forms of hardware, software, or a combination thereof. These elements are preferably implemented in a combination of hardware and software on one or more appropriately programmed general-purpose devices, which may include a processor, memory, and input / output interfaces.

[0016] The present invention relates to a method and system for monitoring an object, such as a human or animal object. In particular, but not exclusively, the present invention relates to a remote sensor-based system for real-time detection and monitoring of one or more objects, which may be, for example, patients, disabled people, and / or elderly people, in locations such as, but not limited to, hospitals, nursing homes, senior centers, and / or residential areas. Multiple objects located at the same location may be detected and monitored simultaneously by a single sensor unit. Alternatively, multiple sensors may also be applied to synchronous detection to improve the sensitivity and accuracy of detection. The use of remote sensors negates the need for any cameras, video cameras, etc. to capture images or videos of the space, which are known to pose privacy concerns to the objects being monitored.

[0017] With reference to Figures 1 and 2, a system 10 and method for monitoring an object or objects in a space according to an embodiment of the present invention is shown. The system 10 comprises a sensor 20 for remotely monitoring a space, such as a room in a care home, an elderly care facility or a hospital ward, in which one or more objects are located, from a first angle of detection. In particular, the sensor 20 is adapted to obtain data of the object being monitored from a first perspective according to the first angle of detection. The sensor 20 may be attached, installed or generally located at any place or position in the space or room. For example, the sensor 20 may be attached at a horizontal level to the object, such as a side wall of the room, or at a high position on the side, such as a corner of the room. Thus, the sensor 20 is not limited to be provided at any particular place in the room, such as at the ceiling and above the object being monitored, as required by the prior art, to avoid obstruction or obscuration to the line of detection. Preferably, the sensor 20 provides detection in a range of about 0.1 meters to about 20 meters. The operation of the sensor 20 is not affected by the lighting conditions of the space under detection, and the sensor 20 is even capable of monitoring objects with high sensitivity and accuracy in dark environments, such as at night, or under high ambient light.

[0018] In one embodiment, the sensor 20 has an emitter 22 adapted to emit light pulses, e.g., electromagnetic radiation, at a first angle of detection towards the object, and a receiver 24 for receiving the electromagnetic radiation reflected from the object, thereby determining distance information of the object relative to the sensor 20. The sensor 20 will then process the distance information of the object into data of the object from a first perspective according to the first angle of detection. The electromagnetic radiation preferably comprises infrared (IR) radiation, but other radiation or combinations of radiation may also be applicable as long as they are reasonably suitable for the purpose of the present invention. In one embodiment, the sensor 20 may comprise a time-of-flight (TOF) sensor. In another embodiment, the sensor 20 may comprise a light detection and ranging (LIDAR) sensor, which may be, but is not limited to, a rotating LIDAR sensor or a solid-state LIDAR sensor. Preferably, the sensor 20 comprises a combined time-of-flight light detection and ranging (TOF-LIDAR) sensor as a single unit, which allows for a reduction in the size of the sensor 20 and the system 10.

[0019] In one embodiment, distance information of an object relative to the sensor 20 can be calculated by a time-of-flight measurement based on the following equation:

number

[0020] After measuring the time of flight, the sensor 20 may output the acquired data in the form of a depth image and / or a point cloud dataset. The data may be transferred from the sensor 20 via any known wire connection to a USB port or wirelessly via a network interface of the system 10. The data may further be communicated to a remote network 100 or a cloud-based database 110 via the communication module 50 of the system 10. It is important to note that all the acquired, transferred, processed and stored data is in the form of digital signals and that neither images nor videos are retained or stored in the system 10 to protect the privacy of the subjects being monitored.

[0021] The system 10 may further comprise a processing module 30 adapted to receive and convert the acquired data of the object into a three-dimensional data set. In one embodiment, the three-dimensional data set may comprise three-dimensional point cloud data (PCD), and the converting step comprises processing the data of the object from the first perspective into point cloud data from the first perspective. In the present invention, the term "point cloud data (PCD)" is given the meaning to refer to a set of data points in space, each data point representing a specific point position in X, Y and Z coordinates of the object being monitored. The point cloud data from the first perspective of the sensor 20 can be stored in the memory 32 of the processing module 30.

[0022] In another embodiment, the data of the object may comprise a depth image or a depth map from a first perspective. The depth image will then be converted into point cloud data (PCD) from the first perspective. The conversion from the depth image to point cloud data (PCD) may be performed based on the following equation:

number

number

number

[0023] In the context of the present invention, the term "depth image" or "depth map" generally relates to an image having pixel values ​​at x and y coordinates and containing depth values ​​represented by a brightness gradient proportional to distance with reference to the viewpoint. For example, a depth image may represent depth information of an object, with closer surfaces shown brighter and further away surfaces shown darker, or vice versa.

[0024] After transformation, the point cloud data from the first perspective will be stored in the memory 32 of the processing module 30. Optionally, background or reference data will be previously acquired from the first perspective and subtracted from the object data from the first perspective to remove background objects from the processing stage. In one embodiment, markers, which may be formed from a light reflective material, are used to provide a mark on the reference substrate or reference area of ​​space on or where the presence of the object is most likely to assist the detection of the object by the sensor 20.

[0025] The processing module 30 may further convert the point cloud data from the first perspective (PCD) into point cloud data from a second perspective, and process the point cloud data from the second perspective (PCD) into a depth image from the second perspective, where the second perspective is created at a second angle different from the first angle of detection. In one embodiment, the processing of the point cloud data from the second perspective (PCD) into a depth image from the second perspective may be performed by projecting the point cloud data onto an xy plane to form a depth image from the same perspective. The processing of the point cloud data from the second perspective (PCD) into a depth image from the second perspective is performed for the purpose of preparing the data into a format that can be processed by the analysis module 40, the steps of which are discussed further below.

[0026] In one embodiment, the second perspective view is preferably different from the first perspective view. For example, the second perspective view may be a top view of the object being monitored (see FIG. 4), whereas the first perspective view may be a horizontal view, a diagonal view, or a view from any angle based on the direction of detection of the sensor 20 (see FIG. 3). The top view may have a full view. In one further embodiment, the top view may have a wide-angle view for an expanded field of view.

[0027] The system 10 may further comprise an analysis module 40 adapted to analyze the created depth image from the second perspective view to determine, calculate or select one or more characteristics related to the activity of the subject during monitoring. The characteristics related to the activity of the subject may generally comprise one or more of the posture, movement, orientation, speed of movement and any general activity of the subject during monitoring. The characteristics may further be determined by processing based on a computer-implemented algorithm, for example, by calculating one or more depth values ​​of the detected subject with reference to one or more pre-set baselines or reference objects, which may be, for example, a bed surface, a floor surface, a chair or a bench top, thereby determining the movement or activity of the subject by inference, such as whether the subject is lying on a bed, sitting on a chair, eating, or lying on the ground. In one embodiment, the analysis module 40 may further comprise a machine learning module 42 for real-time analysis on the depth image based on artificial intelligence (AI) analysis. The analysis may include machine learning algorithms, which may be, but are not limited to, a convolutional neural network (CNN). The CNN can automatically select features from the depth image from the top view and recognize the posture, movement and / or activity of the subject based on continuous training. For example, the subject's activity predicted based on the determined, calculated or selected characteristics of the analysis image may include one or more of lying down, sitting up, standing up, walking, falling, getting out of bed, and leaving the monitored space. In one embodiment, the machine learning module 42 may include one or more of a Google Coral series computing module and / or an Nvidia Jetson series computing module.

[0028] In one embodiment, continuous data can be acquired by the sensor 20 and processed by the processing module 30 to create a series of depth images or video clips for machine learning, thereby enabling a more thorough analysis of activity based on both the content and context of the images and / or videos.

[0029] In response to various detected subject activities, the signaling module 60 of the system 10 may generate one or more alert signals to notify a caretaker of any potential risk. For example, a first alert signal may be generated when a relatively low-risk subject activity is detected or estimated, e.g., a subject's posture or movement is detected, e.g., changing from lying on a bed to sitting on a chair. A second alert may be generated when a relatively high-risk subject activity is detected, e.g., a subject's posture or movement is detected, e.g., changing from lying on a bed to lying on the floor, which is indicative of a fall incident, or when the subject leaves the monitored space. Different levels of the alert signal signify whether or not immediate caretaker attention or assistance is required. In one embodiment, a higher-risk second alert signal may be generated after a lower-risk first alert signal has been triggered and maintained for a predefined period, i.e., without human intervention, such as manually switching off and / or canceling the alert, implying that the subject's activity causing the first alert event has not yet been attended to.

[0030] The generated first and / or second alert signals can be outputted via the communication module 60 to one or more alarm devices 72 in one or more forms of visual and audible alerts or alarms. A second alert signal indicative of a higher possible risk to the subject may be provided in the form of a relatively louder, longer, and / or continuous beep accompanied by an alarm illumination, such as a red rotating light, to indicate that immediate attention is required, while a first alert signal indicative of a lower possible risk to the subject may be provided in the form of a relatively gentler, shorter, and / or intermittent beep accompanied by a gentler alarm illumination. The visible and / or audible alert signals may generally be provided by any known alarm device 72, such as a flashlight, speaker, alarm, etc., and the device is preferably located away from the room being monitored, for example at a caretaker's workstation, to avoid frightening the patient.

[0031] The communication module 50 may further communicate the alert signal wired or wirelessly to one or more computing devices 70 in any known form, such as a desktop computer, a laptop computer, a tablet computer, a smartphone, any portable or wearable electronic smart device, or other emergency service system. The communication module 50 may also transmit the alert signal to a network 100, either private or public, such as the Internet or a cloud-based server 110, to keep a record of any detected risk events or to exchange information or data. In one embodiment, the alert signal may be transmitted wirelessly between the system 10 and one or more of the computing device 70, the network 100, and the cloud-based server 110. For example, the first alert signal of lower risk may be maintained after a predefined period, e.g., 20 seconds, and a second alert signal of higher risk may be generated in a situation where the system 10 has not been able to detect any intervention or disturbance, such as, for example, a caretaker switching off the alarm. Simultaneously or subsequently, the alert, which may be provided in the form of a call or text message, may be transmitted wirelessly via communication module 50 to one or more pre-configured emergency contacts in system 10. In one other embodiment, the alert message may optionally be accompanied by details of the target's emergency contact and / or location which may be transmitted to an emergency system, for example.

[0032] The present invention is advantageous in that it provides a remote sensor-based monitoring system having a time-of-flight, light detection and ranging (TOF-LIDAR) sensor for detecting the posture, movement or activity of a subject. The system has a novel processing module adapted to convert data acquired from a first perspective view into a second perspective view, such as a top view, for subsequent artificial intelligence (AI) analysis. In contrast to the prior art, which requires the placement of a sensor or camera above the subject being monitored, such as on the ceiling of the room to avoid obstruction or obscuration by other objects or visitors in the room, the system of the present invention is essentially provided or located in any place or position in the room, for example horizontally relative to the subject, on a bench top, or mounted in a corner of the room. The system is therefore very versatile and particularly convenient for use in medical settings such as hospitals, nursing homes or senior living facilities, where bed placements often may change depending on the usage and level of occupancy of the room. Multiple sensors may further be provided to monitor a space in a synchronized manner to improve the sensitivity and therefore accuracy of detection. Furthermore, the system of the present invention negates the use of any image or video capturing devices, and all captured, processed and stored data is in the form of digital signals. Thus, the images or videos are retained by the system, thereby protecting the privacy of the subject being monitored. The use of the depth images for the data processing and analysis stage allows further benefits in protecting the privacy of the subject, since the representation of gradients or changes in brightness of the depth images allows the subject's personal identity, body features, facial and clothing features and / or other identifiable features or details to be substantially covered or hidden in a privacy-protecting manner. Thus, the system of the present invention is compact, versatile, and highly sensitive and accurate in detecting potential risks of the subject being monitored.

[0033] The present description illustrates the principles of the invention and it will thus be appreciated that those skilled in the art can devise various arrangements which, although not explicitly described or shown herein, embody the principles of the invention and are included within the spirit and scope of the invention.

[0034] Moreover, all statements herein reciting principles, aspects, and embodiments of the invention, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. In addition, such equivalents are intended to include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.

[0035] Although the present invention has been illustrated and described in detail in the drawings and the above description, it is understood that the present invention should be considered exemplary and not restrictive in character, and merely exemplary embodiments have been illustrated and described, and are not intended to limit the scope of the present invention in any manner. It can be understood that any feature described herein may be used with any embodiment. The exemplary embodiments are not mutually exclusive, nor are they exclusive of other embodiments not described herein. Thus, the present invention also provides embodiments comprising one or more combinations of the exemplary embodiments described above. Modifications and variations to the invention described herein may be made without departing from the spirit and scope of the invention. Therefore, only such limitations as are indicated by the appended claims should be imposed.

[0036] In the claims herein, any element expressed as a means for performing a specified function is intended to encompass any way of performing that function, such as a) a combination of circuit elements that perform that function, or b) any form of software, including firmware, microcode, or the like, in combination with appropriate circuitry for executing the software to perform that function. The invention defined by such claims resides in the fact that the functions provided by the various recited means are combined and brought together in the manner the claims call for. Any means that can provide those functions are therefore regarded as equivalent to the means shown in this specification.

[0037] In the following claims and in the foregoing description of the invention, unless the context requires otherwise, either by express words or necessary implication, the word "comprise" or variations such as "comprises" or "comprising" are used in their inclusive sense, i.e., to specify the presence of stated features, but are not used to exclude the presence or addition of further features in various embodiments of the invention.

[0038] Where any prior art publication is referred to herein, it should be understood that such reference does not constitute an admission that the publication forms part of the common general knowledge in the art.

Claims

1. 1. A method for monitoring a subject, comprising: monitoring a space in which the object is located from a first detection angle relative to a horizontal direction to obtain data of the object from a first perspective according to the first detection angle; converting the acquired data of the object into a three-dimensional data set; processing the three-dimensional data set to produce an image of the object from a second perspective, the second perspective being produced at a second angle different from the first detection angle; analyzing an image produced from the second perspective to determine one or more characteristics related to the subject's activity being monitored; A method for providing the above.

2. The method of claim 1 , wherein the monitoring step comprises detecting distance information of an object relative to a sensor performing the monitoring step, and processing the distance information into data of the object.

3. The method of claim 1 , wherein the monitoring step comprises a time-of-flight (TOF) measurement.

4. 2. The method of claim 1, wherein the three-dimensional data set comprises three-dimensional point cloud data (PCD), and the converting step comprises processing the data of the object from the first perspective view into point cloud data from the first perspective view.

5. The method of claim 4 , wherein the converting step further comprises converting the point cloud data from the first perspective view into point cloud data from the second perspective view.

6. The method of claim 5 , wherein the processing step further comprises processing the point cloud data from the second perspective into a depth image from the second perspective.

7. The method of claim 1 , wherein the first perspective view is different from the second perspective view, the second perspective view being a top view of the object being monitored.

8. The method of claim 1 , wherein the one or more characteristics related to the activity of the subject during monitoring comprise one or more of a posture, a movement, an orientation, and a speed of movement of the subject during monitoring.

9. 10. The method of claim 8, wherein the activities include one or more of lying down, sitting up, standing up, walking, falling, getting out of bed, and leaving the space being monitored.

10. The method of claim 1 , wherein the analyzing step comprises processing based on a machine learning algorithm of a computer-implemented algorithm.

11. 11. The method of any one of claims 1 to 10, further comprising generating one or more alert signals in response to the activity of the subject being monitored.

12. 1. A system for use in monitoring a subject, comprising: a sensor adapted to monitor a space in which the object is located from a first perspective according to a first detection angle with respect to a horizontal direction to obtain data of the object from the first perspective according to the first detection angle; a processing module adapted to convert the acquired data of the object into a three-dimensional data set and to process the three-dimensional data set to create an image of the object from a second perspective, the second perspective being created at a second angle different from the first detection angle; an analysis module adapted to analyze images generated from the second perspective to determine one or more characteristics related to the subject's activity being monitored; A system comprising:

13. 13. The system of claim 12, wherein the sensor has an emitter adapted to emit electromagnetic radiation at the first detection angle toward the target and a receiver adapted to receive electromagnetic radiation reflected from the target, thereby determining distance information of the target relative to the sensor, and the sensor is adapted to process the distance information of the target into data of the target.

14. The system of claim 12 , wherein the sensor comprises a time-of-flight (TOF) sensor.

15. The system of claim 12 , wherein the sensor comprises a light detection and ranging (LIDAR) sensor.

16. The system of claim 12 , wherein the data of the object comprises a depth image from a first perspective.

17. The system of claim 12 , wherein the three-dimensional data set comprises three-dimensional point cloud data (PCD).

18. The system of claim 12 , wherein the second perspective view is a top view of the object being monitored.

19. The system of claim 12 , wherein the analytical module comprises a machine learning module adapted to perform artificial intelligence analysis.

20. 20. The system of any one of claims 12 to 19, further comprising a signaling module for generating one or more alert signals in response to the activity of the subject being monitored.

Images