Systems and methods for continuously sharing the behavioral state of a living being - Patents.com

Abstract

To provide a system for continuously sharing action states of a creature.SOLUTION: The system comprises: a robot (12) configured to perform different actions; at least one sensing means (5), (6), (8), (9) for automatically sensing the creature; a determining means for continuously determining the state of the creature based on information derived from the at least one sensing means (5), (6), (8), (9); and a selecting means for selecting, from the different actions, an action that is assigned to the determined state; where the robot (12) is configured to perform the selected action.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to techniques for detecting human activities and other conditions and reporting them to a remote human, but is not limited to humans and may also be applied to other living beings, such as animals. [Background technology]

[0002] In elderly care, systems exist that detect undesirable conditions in people living alone and report them to a caregiver or allow the person to notify emergency services. For example, a device that detects when a person has fallen and reports it to a caregiver is described in US Patent Application No. 2009048540(A1).

[0003] US Patent Application Publication No. 20090322513(A1) describes a method for measuring physiological health parameters and reporting emergency cases, along with their location, to a caregiver.

[0004] However, in such monitoring systems, a small number of defined conditions are detected and reported in real time as single events.

[0005] Additionally, telepresence systems are known that are designed for active communication and / or to provide a sense of presence in a remote location through explicit interaction. US Patent No. 9,552,056 (B1) discloses a telepresence robotic device that mirrors and transmits the gestures of a remote operator to a recipient.

[0006] However, telepresence systems are not designed for communicating regular activities to remote people, especially in a continuous manner.

[0007] Kwangmin Jeong et al., "Fribo: A Social Networking Robot for Increasing Social Connectedness through Sharing Daily Home Activities from Living Noise Data," Proceedings of the 2018 ACM / IEEE International Conference on Human-Robot Interaction (2018), discloses a social robot that recognizes a user's activities by analyzing the resident's daily living sounds and shares the behavioral information with close friends through the respective robots. In particular, this document discloses a system that includes sensors located in a home or apartment that generate sensor data (behavioral data) of daily living sounds, and a social robot that receives the sensor data in real time, converts the sensor data into higher-level information, and transmits the information to another robot of the same kind that outputs the higher-level information to a second user using a speaker and a display.

[0008] However, because high-level information is only output as audio or display information, social robots stimulate only a few senses, require explicit attention to the device, and can only communicate events at specific times, thus giving little impression of a person's continuous presence. Summary of the Invention [Problem to be solved by the invention]

[0009] It is therefore an object of the present disclosure to improve the performance of systems and methods for continuously sharing the state of an organism. [Means for solving the problem]

[0010] The present disclosure provides systems and methods for continuously sharing the behavioral state of an organism.

[0011] The system according to the present disclosure for continuously sharing the behavioral state of a living being comprises a robot configured to continuously execute various behaviors, at least one sensing means for automatically sensing the living being, a determining means for determining the state of the living being based on information derived from the at least one sensing means, and a selecting means for selecting a behavior to be assigned to the determined state from the various behaviors, where the robot is configured to execute the selected behavior. To continuously communicate the state, the same behavior can be repeated until a new (next) state of the living being is determined by the determining means. The determination of the state of the living being is performed continuously.

[0012] A method according to the present disclosure for continuously sharing a behavioral state of a living being uses a robot configured to perform various behaviors and includes the following steps.

[0013] - automatically detecting a living being by at least one sensing means;

[0014] - continuously determining a state of the organism based on information derived from the at least one sensing means;

[0015] - selecting, from among the different behaviors, a behavior to be assigned to the determined state; and

[0016] - The robot executes the selected behavior.

[0017] The present disclosure will now be described in detail with particular reference to the accompanying drawings. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 illustrates a system according to an exemplary embodiment of the present disclosure. [Diagram 2] FIG. 2 is a block diagram of a first portion of the system shown in FIG. 1. [Diagram 3] FIG. 2 is a block diagram of a second portion of the system shown in FIG. 1. [Figure 4]1 is a simplified flow chart illustrating a method for performing autonomous or partially autonomous driving in accordance with the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] The system according to the present disclosure for continuously sharing the behavioral state of a living being comprises a robot configured to continuously execute various behaviors, at least one sensing means for automatically sensing the living being, a determining means for determining the state of the living being based on information derived from the at least one sensing means, and a selecting means for selecting a behavior to be assigned to the determined state from the various behaviors, where the robot is configured to execute the selected behavior. To continuously communicate the state, the same behavior can be repeated until a new (next) state of the living being is determined by the determining means. The determination of the state of the living being is performed continuously.

[0020] According to the present disclosure, a robot communicates the abstracted state of a remote creature in real time through its behavior, thereby enabling passive and continuous telepresence of the creature and revealing opportunities for shared actions and experiences. It should be noted that the term "real time" also encompasses delays caused by signal processing and transmission. Because the robot communicates through its behavior, it stimulates many senses, does not require explicit attention, and gives a good impression of the creature's continuous physical presence. An improvement over current technology is that the system according to the present disclosure allows for continuous state communication, which goes beyond the event nature of messages (i.e., the robot according to the present disclosure communicates something at any time, as opposed to systems that communicate only at specific points in time). As a result, the robot does not need to be attended to ("active") at that particular moment, and communication will have an effect even if attention to the robot occurs at any other time. In either case, observing and communicating the behavior of a remote creature enhances the impression of a real companion.

[0021] The creature can be a person or a pet, and the robot communicates with another person or pet, for example the status of the pet is shared with a person who is not allowed to keep pets.

[0022] To perform a selected behavior, the robot may be configured to move to a predetermined location within the robot's environment, move along a predetermined path, move with a predetermined speed profile, assume a predetermined pose, output a predetermined sound, and / or output light of a predetermined color and / or intensity.

[0023] Also, to perform different behaviors, the robot may be configured to move to different locations within the robot's environment, move on different paths, move with different speed profiles, assume different postures, output different sounds, and / or output light of different colors and / or different intensities.

[0024] The robot can operate within the user's environment, for example on the floor of the user's apartment, or in a miniature model of the apartment or the animal's cage. For example, the system can include a miniature model of the organism's environment, and the robot is configured to move within the miniature model (e.g., a small robot that uses at least in part external sensors installed within the model).

[0025] The system may operate in a form of telepresence, where the robot is located in the home of one user or group of users (the "host" user / location), whereas the corresponding sensor system is located in the space of another user (the "remote" user / location). The living being and the robot are located at different locations, and the system may comprise a transmitting means for transmitting, via a data network, information derived from at least one sensing means located at the "remote" user / location, a determining means located at the "host" user / location, a determined state from the determining means located at the "remote" user / location to a selecting means located at the "host" user / location, or a selected behavior from the selecting means located at the "remote" user / location to the robot located at the "host" user / location.

[0026] The detection means may use image processing methods (e.g., machine learning methods trained with labeled footage of people actually performing certain states) to classify between a predefined number of state classes, or it may directly use signals from sensors or specific devices, e.g., a television communicating its power state, or a refrigerator communicating whether the door is open, along with a predefined set of rules on how to map those signals to predefined states, i.e., "watching TV" or "eating".

[0027] The states judged by the judging means are as follows.

[0028] - Emotional states such as happy, excited, bored, relaxed, distracted, stressed;

[0029] - Physiological states such as asleep, awake, tired, active, cold / hot;

[0030] - actions performed, such as cooking, watching TV, talking on the phone, reading, knitting;

[0031] - Other specific cases, such as not being at home, not being able to make a phone call, not doing anything; and / or

[0032] - Emergency conditions like falls, arrhythmia / seizures, illness.

[0033] To this end, the determining means may be configured to determine a predetermined emotional state of the living being, a predetermined physiological state of the living being, a predetermined action performed by the living being, and / or a predetermined emergency state of the living being.

[0034] The determining means may also be configured to classify the state of the living being into different emotional states, different physiological states, different behaviours and / or different emergency states.

[0035] To ensure privacy protection, the system may be equipped with configuration means for the "remote" or "host" user to configure which states of the organism are not allowed to be shared and / or the time periods during which sharing of the organism's states is not allowed. In this way, the remote user or any operator configuring the system may specify in the privacy settings which states are willing to be shared and / or the user may actively enable / disable sharing for some period of time.

[0036] To sense the living being, fixed sensors, mobile sensors (e.g. sensors in a smart watch or smartphone) or signals indicative of a device being operated (e.g. appliances of a smart home system) may be used. In another exemplary embodiment, at least one of the sensing means is configured to detect an operating state of a device controllable by the living being.

[0037] The following sensor types may be used:

[0038] - Cameras like 3D cameras, active ("pan-tilt-zoom") cameras, IR cameras;

[0039] - Microphone

[0040] - Physiological sensors like skin conductivity sensors, EMG, EEG, electrodes, optical heart rate sensors, temperature sensors;

[0041] - magnetic switch sensors, proximity sensors (e.g. ultrasonic), motion sensors (e.g. optical), pressure sensors (e.g. piezoelectric); and / or

[0042] - Virtual sensors for on-screen selection etc.

[0043] In another exemplary embodiment, at least one of the sensing means is a camera, a microphone, a physiological sensor, a proximity sensor, a motion sensor, a pressure sensor, or a portable device configured to determine its location.

[0044] Alternatively or additionally, the system comprises at least one sensor robot equipped with at least one of the above sensing means, which may be configured to move in the environment of the organism and / or to guide at least one of the sensing means towards the organism.

[0045] The determining means may also be configured to determine a dialogue state in which the living being interacts with the sensor robot at the remote location, and the selecting means is configured to select a dialogue behavior when the dialogue state is determined by the determining means, for example, the robot moving to and / or moving around and / or interacting with a given object in the robot's environment, which dialogue behavior can then also be triggered in the robot at the host location to create a reflected impression and communicate the dialogue state at the remote location.

[0046] The movement of the robot may be controlled based on one or more external and / or internal sensors of the robot. In another exemplary embodiment, the robot includes at least one environmental sensor (e.g., a proximity sensor, a camera) that generates a signal indicative of a predetermined object, and the robot is configured to navigate to and / or around and / or interact with the predetermined object based on the signal derived from the at least one environmental sensor.

[0047] A method according to the present disclosure for continuously sharing a behavioral state of a living being uses a robot configured to perform various behaviors and includes the following steps.

[0048] - automatically detecting a living being by at least one sensing means;

[0049] - continuously determining a state of the organism based on information derived from the at least one sensing means;

[0050] - selecting, from among the different behaviors, a behavior to be assigned to the determined state; and

[0051] - The robot executes the selected behavior.

[0052] FIG. 1 illustrates a system according to an exemplary embodiment of the present disclosure, which shares the behavioral state of a first person 1 (a “remote” user) in a first apartment 2 with a second person 3 (a “host” user) in a second apartment 2. The system illustrated in FIG. 1 comprises in the first apartment 2 a camera 5 that captures an area in the living room, a pressure sensor 6 installed on a bed 7 in the bedroom, a microphone 8 installed in the kitchen, a smart watch 9 worn on the wrist of the first person 1, and a first processing device 10, such as a computer. In the second apartment 2, the system comprises a second processing device 11, e.g. another computer, connected to the first processing device 10 via a data network (not shown), and a robot 12 that communicates the abstracted state of the first person 1 through its own behavior.

[0053] The microphone 8 is configured to detect sounds generated from the first person 1's activities such as cooking, eating, walking, laughing loudly, and talking, and transmit the detected sounds to the first processing device 10 via radio frequency communication or cable. The camera 5 generates an image signal from the first person 1 in the living room and transmits the image signal to the first processing device 10 via radio frequency communication or cable. The pressure sensor 6 generates a pressure signal indicating whether the first person 1 is in bed 7 or not, and transmits the pressure signal to the first processing device 10 via radio frequency communication or cable. The smart watch 9 comprises multiple sensors such as an ambient light sensor, a three-axis accelerometer, a heart rate monitor, and a positioning system (indoor and / or GPS), and is configured to transmit data generated by the sensors to the first processing device 10 via radio frequency communication.

[0054] 2 shows a block diagram of a first processing device 10, which is connected to the camera 5, the pressure sensor 6, the microphone 8, the smartwatch 9 and the operating device 13 by first transmission / reception means 14 (e.g. Bluetooth or WLAN transmission device) and to a data network 15 by second transmission / reception means 16 (e.g. Internet router, residential gateway). The first processing device 10 comprises determining means 17 for determining a state of the first person 1 based on the sensor data received by the first transmission / reception means 14, and setting means 18 for setting states of the first person 1 for which sharing is not allowed and / or a time during which sharing of the state determined by the determining means is not allowed.

[0055] The determination means 17 continuously determines one or more states of the first person 1 based on sensor data automatically generated by the camera 5, the pressure sensor 6, the microphone 8 and the smart watch 9, and the determination means 17 detects in this sensor data certain characteristics that are assigned to a given state.

[0056] In detail, the emotional state "happy" can be estimated by image and voice processing methods, where a smiling face is detected in the image captured by the camera 5 and laughter is detected in the sound captured by the microphone 8, the physiological states "asleep" and "awake" can be detected based on the pressure signal of the pressure sensor 6 and the heart rate measured by the smartwatch 9, the states "watching TV", "on the phone", "reading" and "knitting" can be detected based on the image captured by the camera 5, the states "cooking" and "eating" can be detected based on the sound captured by the microphone 8, the emergency state "falling" can be detected based on the signal of the three-axis accelerometer of the smartwatch 9, and the states "excited", "engrossed" and "stressed" can be estimated by voice recognition. Voice recognition per se is known from the prior art, as is the use of image and voice processing methods for detecting certain events.

[0057] The accuracy of the state determination may be improved by detecting a set of characteristics, for example the state "not at home" may be set if there is no connection to the smartwatch 9, the pressure signal indicates "awake", the first person 1 is not captured by the camera 5 and no sound or voice of the first person 1 is detected for some period of time.

[0058] For each state (state class), one or more characteristics may be determined by a machine learning method trained with labeled sounds and / or images (video) of a person actually performing the respective state (state class). The determining means 17 or an external device may perform the machine learning method, where the states and their corresponding characteristics or learning data are input by the operating device 13 and stored in the determining means 17.

[0059] Furthermore, using the operating device 13, a user (e.g. the first person 1) can specify in the privacy settings which states are allowed to be shared and / or the user can actively enable / disable sharing for some time. This setting is stored in the setting means 18, which generates a graphical user interface for the privacy settings and filters the states where sharing is not allowed from the states determined by the determining means 17. The states where sharing is allowed are received from the second transmitting / receiving means 16, which transmits the detected states to the second processing device 11.

[0060] 3 shows a block diagram of the second processing unit 11 and the robot 12. The second processing unit 11 comprises a selection means 19 for selecting a behavior of the robot 12 based on the states allowed to be shared, and a transceiver means 20 (e.g., a WLAN router) connected to the data network 15 for receiving the states from the first processing unit 10 and transmitting the selected behavior to the robot 12. The robot 12 comprises a transceiver means 21 (e.g., a WLAN adapter) for receiving the selected behavior, a display 22 configured to display a symbolic representation of a face, a speaker 23 for outputting sound, a camera 24 for sensing the environment of the robot 12, motorized legs 25, motorized arms 26, a motorized tail 27, and a control means 28 (e.g., a microcontroller) for controlling the display 22, the speaker 23, the motorized legs 25, the motorized arms 26 and the motorized tail 27 to perform the selected behavior.

[0061] All the means described in FIGS. 2 and 3 may be realized as software modules executed on a processor of the respective first processing device 10, second processing device 11 or in the robot 12.

[0062] Exemplary behaviors of the robot 12 may be the following:

[0063] - sitting in a particular position, for example defined by a second person 3, an object such as a dog cage, or defined by a location relative to a room, for example a kitchen, an entrance door;

[0064] - Facial expressions, for example smiling, yawning, closing the eyes;

[0065] - Gestures, for example waving, or body postures, for example lying down or stretching;

[0066] - General behaviors such as walking around, jumping up, jumping down, turning around, dancing, etc.;

[0067] - Specific behaviors such as eating from a bowl, scratching a wall, playing with a ball;

[0068] - change the color, turn the display 22 on / off; and / or

[0069] - Acoustic expressions such as laughing, breathing heavily, snoring, gurgling, etc.

[0070] The selection means 19 stores, for each state that can be determined by the determination means 17, a behavior that represents an abstracted version of the respective state. In detail, if the first person 1 is awake and at home, a behavior is selected in which the eyes of the face displayed on the display 22 are open, if the first person 1 has left the apartment 2, a behavior is selected in which the robot moves to a specific location, for example next to the entrance door and stays there, if the first person 1 is in a happy state, a behavior is selected in which the robot 12 smiles and / or wags its tail, and if the first person 1 is detected together with a behavior that is interpreted as not doing anything, a behavior is selected in which the robot 12 changes the color of the displayed face.

[0071] The steps of determining the state of the first person 1, selecting a behavior to be assigned to the determined state, and communicating that state to the second person 3 via the selected behavior are performed automatically and continuously, without the need for any operational steps.

[0072] In certain implementations, a symbolic representation of the robot 12 and its behavior may be displayed on a portable device (e.g., a mobile phone or tablet with a companion app) so that state can be shared even if the second person 3 leaves the second apartment 4. Alternatively, the robot 12 or a part of the robot 12 is portable to exhibit behavior while carried inside or outside the second apartment 4, and / or the robot 12 is designed to operate within a miniature model of the second apartment 4 or within another environment with multiple interaction points, such as a doll's house, diorama or terrarium.

[0073] Alternatively or additionally, a sensor sensing the first person 1 may be installed in another (sensor) robot similar or identical to the robot 12 located in the second apartment 4. In this case, the interaction of the first person 1 with the sensor robot (looking, speaking and / or moving up to the sensor robot) may be detected, which triggers behavior in the sensor robot, which is then "reflected" to the robot 12.

[0074] Alternatively or additionally, signals from sensors or specific devices can be used to determine the state, e.g., a television communicating its power state, a refrigerator communicating whether the door is open, along with a set of predefined rules on how those signals map to expected states, i.e., "watching television" or "eating."

[0075] The functions of the determining means 17, the setting means 18 and / or the selecting means 19 described herein may be implemented using individual hardware circuits, using software working in conjunction with a programmed microprocessor or general purpose computer, using application specific integrated circuits (ASICs), and / or using one or more digital signal processors (DSPs). Furthermore, the determining means 17 or additionally the setting means 18 may also be part of the second processing device 11, or the selecting means 19 may be part of the first processing device 10, with the transmitting and receiving means 20 receiving the selected behavior and forwarding the selected behavior to the robot 12.

[0076] Alternatively or additionally, the robot 12 may comprise a robotic arm that has a fixed base position but multiple controllable joints, the robot 12 may be designed as a legged "animal" that is free to move around and can also express behaviors with its face, or as a wheeled robot with a screen.

[0077] FIG. 4 shows a simplified flow chart illustrating the individual steps performed by an implementation of the method described in detail above.

[0078] It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the present disclosure without departing from the scope or spirit of the disclosure. In view of the foregoing, it is intended that the present disclosure cover modifications and variations of this disclosure provided they fall within the scope of the following claims and their equivalents.

Claims

1. A system for continuously sharing a person's actions and / or behavioral states, comprising: a robot (12) configured to perform various behaviors; At least one sensing means (5), (6), (8), (9) for sensing said person; determining means (17) for determining said behavior and / or said state of said person based on information derived from said at least one sensing means (5), (6), (8), (9); a selection means (19) for selecting, from the various behaviors, the determined action and / or a behavior to be assigned to the state, the robot (12) is configured to execute the selected behavior; The actions determined by the determining means (17) include one or more actions that are interpreted as cooking, watching television, talking on the phone, reading, and knitting; The state determined by the determination means (17) includes one or more of the following: - emotional states indicating happy, excited, bored, relaxed, preoccupied, stressed; - Physiological state indicating whether asleep, awake, tired, or active; - other specific status indicating that you are not at home and are available to make a phone call; and - Falls, arrhythmia / seizures, emergency conditions indicating illness, A system comprising a setting means (18) for generating a graphical user interface for privacy settings, which sets states for which supply is not permitted and times for which sharing of said states is not permitted among the determination states determined by the determination means (17).

2. 2. The system of claim 1, wherein to perform the selected behavior, the robot (12) is configured to move to a predetermined position in an environment of the robot (12), move on a predetermined path, move with a predetermined speed profile, assume a predetermined posture, output a predetermined sound, and / or output light of a predetermined color and / or intensity.

3. 3. The system of claim 2, wherein to perform the different behaviors, the robot (12) is configured to move to different positions in the environment of the robot (12), move on different paths, move with different speed profiles, assume different postures, output different sounds, and / or output light of different colors and / or different intensities.

4. a miniature model of the human environment; The system of claim 1 , wherein the robot (12) is configured to move within the miniature model.

5. 2. The system according to claim 1, wherein the human and the robot (12) are located at different locations, and the system further comprises a transmission means (16) for transmitting the information derived from the at least one sensing means (5), (6), (8), (9) to the determination means (17), the determined action and / or the determined state to the selection means (19) or the selected behavior to the robot via a data network (15).

6. 2. The system according to claim 1, wherein the determining means (17) is configured to determine a predetermined emotional state of the person, a predetermined physiological state of the person, a predetermined action performed by the person, a predetermined emergency state of the person, and / or the other specific state indicative of not being at home and being able to make a phone call.

7. 7. The system according to claim 6, wherein the determining means (17) is configured to classify the state of the person (1) into different emotional states, different physiological states and / or different emergency states and to classify the behavior of the person into different behaviors.

8. 2. The system according to claim 1, wherein at least one of said sensing means (5), (6), (8), (9) is configured to detect an operational state of said human-controllable device.

9. 2. The system of claim 1, wherein at least one of the sensing means (5), (6), (8), (9) is a camera (5), a microphone (8), a physiological sensor, a proximity sensor, a motion sensor, a pressure sensor (6), or a handheld device (9) configured to determine its location.

10. The system according to claim 1, further comprising a sensor robot (12) equipped with at least one of said sensing means (5), (6), (8), (9).

11. The determination means (17) is configured to determine a dialogue state in which the person is interacting with the sensor robot (12); The system of claim 10, wherein the selection means (19) is configured to select, when the dialogue state is determined by the determination means (17), a dialogue behavior in which the robot (12) moves to and / or around a predetermined object in an environment of the robot (12).

12. the robot (12) comprises at least one environmental sensor that generates a signal indicative of the predetermined object; The system of claim 11, wherein the robot (12) is configured to navigate to and / or around the given object based on the signal derived from the at least one environmental sensor.

13. 1. A method for continuously sharing human actions and / or behavior states using a robot configured to perform various behaviors, comprising: - automatically detecting said person by at least one detection means (5), (6), (8), (9); - continuously determining said behaviour and / or said state of said person based on information derived from said at least one sensing means (5), (6), (8), (9); selecting, from the various behaviors, a behavior to be assigned to the determined action and / or state; and causing the robot (12) to execute the selected behavior; the activities determined by the determining step include one or more of activities interpreted as cooking, watching television, talking on the phone, reading, and knitting; The state determined by the determining step includes one or more of the following: - emotional states indicating happy, excited, bored, relaxed, preoccupied, stressed; - Physiological state indicating whether asleep, awake, tired, or active; - other specific status indicating that you are not at home and are available to make a phone call; and - Falls, arrhythmia / seizures, emergency conditions indicating illness, A method including a setting step of generating a graphical user interface for privacy settings, which sets states for which provision is not permitted among the determined states determined by the determining step, and times for which sharing of the states is not permitted.

Images