System for detecting motion process and / or vital sign parameters of person

Abstract

The invention relates to a system and a method for detecting a motion process of a person. The method includes: detecting, by a non-contact sensor, a plurality of images of a person during movement, wherein the plurality of images describe movement of a body element of the person; at least one skeletal model having limb positions is created for at least some of the plurality of images, and a motion process is calculated from the motion of the body element of the person by comparing changes in the limb positions in the at least one created skeletal model. Meanwhile, vital sign parameters and / or signal processing parameters about the person can be detected and analyzed.

Description

[0001] The present patent application is a divisional application of the patent application with the application date of 31 August 2020, the application number 202080076764.2, and the invention name “System for detecting a movement process and / or a vital parameter of a person”.

[0002] Cross-references to known patent applications

[0003] This patent application claims priority from the following German patent applications: DE 102019123304.6 filed on 30 August 2019; DE 102020102315.4 filed on 30 January 2020; and DE 102020112853.3 filed on 12 May 2020. The entire contents of the above-mentioned German patent applications are hereby incorporated by reference in their entirety into the present patent application. TECHNICAL FIELD

[0004] The present invention relates to a service robot for automatically performing geriatric tests.

[0005] Background of the invention

[0006] The medical health system has long been plagued by a severe lack of professionals. This lack of personnel results in less and less time being available for treating patients. The lack of time not only leads to dissatisfaction among patients and medical staff, but also to diseases not being treated sufficiently, which not only causes suffering for the patients, but also reduces the value created for the national economy. For these reasons, the necessity of documenting the condition of patients is increasing, so that a medical point of view can also be defended in the event of a claim for damages that can be attributed to the inadequate treatment. This obligation to document can in some cases have a self-reinforcing effect.

[0007] The service robot described in this document solves the problem that the geriatric tests currently carried out by medical staff are autonomously carried out by the service robot using a plurality of sensors. This service robot can in addition accurately document the completed exercises, whereby the health device of the service robot can fulfill the obligation to document and other compliance obligations associated therewith without having to arrange personnel for this purpose alone. A further effect is that the use of the service robot standardizes the evaluation of the tests, since the evaluation of the patients currently depends on the experience of the medical staff, which evaluation results differ from other medical staff for reasons of personal experience. Thus, the evaluation of the same exercises by medical staff can lead to different results, whereas the use of the service robot enables uniform evaluation results.

[0008] In addition to service robots in the field of geriatrics, such as those capable of taking the Barthel Index, performing the so-called "timed up and go" test and / or different features of simple psychological tests, service robots are in another aspect configured such that they can alternatively and complementarily solve other tasks in the clinic. These include, for example, the examination of a spectrometer, which can analyze different substances under or on the skin of a person. These analyses can be used, for example, to determine a delirium detection score.

[0009] In one aspect, the service robot is also configured to perform delirium identification and / or delirium monitoring. In this context, the service robot can in one aspect determine a possible presence of an attention disorder in a patient from the recognition of a sequence of acoustic signals. In an alternative and / or complementary aspect, the service robot can assess cognitive ability based on image recognition and / or indicate cognitive ability through the implementation of motor functions, such as by counting the fingers of a patient who reacts to a primarily visual request of the service robot. As an alternative and / or complementary solution, the service robot can determine the pain state of a patient. This can be achieved by recognizing emotions, detecting upper limb movements and / or taking the pain cries of a patient who is not receiving artificial respiration and / or who is receiving artificial respiration. In one aspect, the service robot can determine the blood pressure of a patient, its respiratory rate and, in addition to the purposes of initial diagnosis and / or treatment, use this information to manipulate hardware and software components of itself.

[0010] In addition to this, the service robot can be configured to detect manipulation attempts, for example, in the scope of the detection data. The service robot can also check whether a user is mentally and / or physically affected by an adverse influence that can affect the quality of a test to be performed or its results. Furthermore, in one aspect, the service robot can adjust its signal processing quality to environmental influencing factors and also adjust the signal output. Here, this includes adjustments to the input and output, user dialog, etc.

[0011] Furthermore, the use of service robots also significantly relieves medical personnel from having to do work that is sometimes time-consuming and monotonous but has no direct influence on the health of patients, thus preventing these personnel from implementing measures that directly improve the health of patients. State of the art

[0012] The person of ordinary skill in the art is familiar with different kinds of service robots in the field of health or geriatrics. Thus CN108422427 describes a rehabilitation robot which is able to deliver a tablet to a patient for taking. Similarly, the service robot described in CN206833244 is able to distribute material in a hospital. In the field of hospitals, Chinese patent applications CN107518989 and CN101862245 relate to a service robot which is able to transport a patient, similar to a wheelchair. CN205950753 describes a robot which identifies a patient by means of a sensor mechanism and provides navigation for the patient in a hospital. In CN203338133 a robot is described which is used to provide support for nursing staff and is able to accompany a patient in a hospital in dealing with everyday matters. Conversely, CN203527474 relates to a robot which provides support for an elderly person by means of a robotic arm.

[0013] CN108073104 relates to a care robot which is able to care for patients who are infected, by means of delivering medication or administering medication to these patients, massaging the patients, feeding them, communicating with them, etc. Here, the care robot is able to reduce the risk of infection for medical staff because it reduces the number of times that medical staff come into contact with the patients. CN107598943 describes a robot for accompanying an elderly person. This robot has several monitoring functions, but its basic function is floor cleaning.

[0014] CN106671105 relates to a mobile service robot which takes care of an elderly person. This service robot monitors human parameters such as body temperature by means of a sensor mechanism and is also able to monitor facial expressions. The service robot is also able to identify whether a person has fallen and is able to alert help accordingly via a network.

[0015] Similar prior art also includes CN104889994 and CN204772554, in which a service robot in the medical field is able to identify a heart rate, provide oxygen to a patient and also includes voice recognition and a multimedia module for entertainment purposes. The product described in CN105082149 is also able to detect blood oxygen. CN105078445 relates to a service robot which records, inter alia, an electrocardiogram of an elderly person, measures the oxygen content in the blood. Similarly, CN105078450 has an electroencephalogram measurement function.

[0016] Certain health robots are dedicated to performing exercises or tests on patients. CN108053889 describes a system for performing exercises on patients based on saved information in a relatively brief manner. CN108039193 describes a system for automatically generating health reports in a robot. CN107544266 describes detecting motoric / adaptive exercises by means of a robot, recording and saving data for analysis thereof and transmitting the data to an external system. At the same time, the robot is able to monitor the taking of medication by means of different sensors.

[0017] CN106709254 describes a robot for medical diagnosis of patients, which likewise formulates a treatment plan on the basis of the diagnosis. To this end, the robot analyzes voice and image information and compares it with saved information in a memory. To this end, a neural network is used.

[0018] CN106407715 describes a service robot which uses voice processing and image recognition to perform a patient's anamnesis. In addition to the questioning by means of voice input and output means, it is also possible to refer to a tongue photo taken by a robot camera during the anamnesis by means of a touchpad.

[0019] CN105078449 describes a service robot equipped with a tablet and communication means, by means of which cognitive function training or cognitive psychological tests can be performed in order to find out whether a patient suffers from Alzheimer's disease. To this end, the tablet records a telephone conversation between the patient and a child according to a specific procedure, from which the conversation record it is determined whether the patient suffers from Alzheimer's disease.

[0020] Service robots analyze gestures in one aspect on the basis of the folding of a paper. In the prior art, it is essentially the gesture itself that is recognized. However, the challenge lies in particular in recognizing and tracking the fingers. US10268277, as well as US9372546 or US9189068, for example, describe a general gesture recognition system. In US9690984, for example, a hand is recognized by means of a machine learning algorithm on the basis of a camera by means of a skeleton model. These solutions essentially involve empty hands. In contrast, US9423879 involves recognizing and tracking objects in the hand and proposes using a thermal sensor to distinguish between the hands and fingers (by means of the emitted heat) and other objects (which tend to be cooler).

[0021] In the prior art, only two documents are found which deal with the recognition of paper or objects similar to paper in the hand of a user. Thus US 9117274 describes how a depth camera is used to recognize a piece of paper held in the hand of a user, whereas in a next step an image is projected onto this piece of paper which represents a flat surface and the user is able to interact with the image. The paper is recognized by comparing its corners with quadrangles which are stored in a memory and which are spatially rotated within the space. In contrast thereto, US 10242527 describes how a gaming table (in a casino) is monitored by automatic recognition of gestures, which includes gaming chips or playing cards which have a certain degree of similarity to paper. It is not described how the recognition is carried out, but only for what purpose such an analysis is carried out. Furthermore, the corners of the playing cards are rounded, whereas the corners of the paper are usually not rounded.

[0022] From the analysis of the cognitive state of a person, various approaches are also described in the prior art which have an influence on the control of a robot. For example, US 20170011258 shows how a robot is to be operated on the basis of the emotional state of a person, wherein this state is basically analyzed by facial expression of the person, which is also detected by a histogram of gradients analysis. The emotional state of the person can be determined by a classification method on the basis of a clustering evaluation, but also by a neural network. For example, US 2019012599 generally describes how a multi-layer convolutional neural network is used to generate weights on the basis of a video of a captured face, which multi-layer convolutional neural network has at least one convolutional layer and at least one hidden layer, the last layer of which describes the emotion of a person, which multi-layer convolutional neural network determines the weights of the input variables of at least one layer, calculates the weights in at least one feed-forward-process and updates them in the scope of a backpropagation.

[0023] In the prior art, there are different examination results in the detection of the mental state of a person. For example, US 9619613 uses a special instrument which works by means of vibrations to analyze the mental state of a person. For example, US 9659150 uses an acceleration sensor to carry out a timed up and go test. In US 9307940, stimuli are triggered to test the mental capacity of a patient by outputting a defined length of a stimulus sequence and recording the reaction of the patient. US 8475171 uses virtual reality technology to show different images to a patient and to diagnose, for example, Alzheimer's disease by the recognition of these images by the patient. US 10111593 uses motion analysis to recognize mental confusion, for example. In contrast, CN 103956171 attempts to draw conclusions about a simple mental test score on the basis of the speech of a patient.

[0024] The service robot is configured in such a way that, by means of its sensor means, such as by means of a camera, it can detect further medical parameters, including the measurement of blood pressure in a contactless manner. This latest technology for determining blood pressure by means of camera-based analysis is generally in the research phase. Zaunseder et al. (2018) describe the general situation of blood flow color analysis methods. A review article by Rouast et al. (2018) also deals with this. Karylyak et al. (2013) or Wang et al. (2014), for example, specifically deal with the examination of analysis algorithms that can determine blood pressure on the basis of signal data, while McDuff et al. (2014) deal with the examination of the determination of the points in time of cardiac contraction and cardiac relaxation pressure, and Bai et al. (2018) examine the effectiveness of new signal filters. Parati et al. (1995), for example, find a conventional approach to determining blood pressure from detected measurement values. Liu et al. (2018) deal with specific implementation means of color analysis, again comparing individual subareas of the face, as does Verkruysse et al. (2008), while Lee et al. (2019) describe specific implementation means based on facial movements. In contrast, Unakafov (2018) compares different approaches based on freely disposable data items. The approach by Pasquadibisceglie et al. (2018) has already entered the practical application phase, integrating a color analysis method into a mirror. In contrast, Luo et al. (2019) use a smartphone for recording color data. The approach by Wei et al. (2018) has already entered the specific implementation phase, which already has the features of a clinical examination. The approach by Ghijssen et al. (2018) is in contrast. They emit light through a finger by means of a laser, which is detected on the opposite side by means of a sensor, the outgoing light having a speckle pattern, whereby on the one hand periodic vessel blood flow can be detected and on the other hand, as in the previous approaches, periodic vessel expansion of the blood vessel can be detected.

[0025] Reference sources

[0026] Zaunseder et al. Cardiovascular analysis by imaging photoplethysmography - a review. Biomedical engineering / Biotechnik 2018; 63(5): 617-634, DOI: 10.1515 / bmt-2017-01.

[0027] Kurylyak et al. Estimation of blood pressure from PPG signals, 2013 IEEE International Instrumentation and Measurement Technology Conference (I2MTC). DOI: 10.1109 / I2MTC.2013.6555424.

[0028] McDuff et al. Remote detection of peak photoplethysmographic systolic and diastolic blood pressure using a digital camera. IEEE Transactions on Biomedical Engineering, vol. 61, no. 12, 12 December 2014, DOI: 10.1109 / TBME.2014.2340991.

[0029] Bai et al. Real-time robust non-contact heart rate monitoring using camera. IEEE Access VOLUME 6, 2018. DOI: 10.1109 / ACCESS.2018.2837086.

[0030] Pasquadibisceglie et al. Non-contact analysis of cardiovascular parameters for personal healthcare systems. 2018 AEIT International Conference. DOI: 10.23919 / AEIT.2018.8577458.

[0031] Wang et al. Cuffless blood pressure analysis using pulse transit time and heart rate. 2014 12th International Conference on Signal Processing(ICSP). DOI: 10.1109 / ICOSP.2014.7014980.

[0032] Luo et al. Smartphone-based blood pressure measurement using transdermal optical imaging technology. Circ Cardiovasc Imaging. 2019; 12:e008857. DOI: 10.1161 / CIRCIMAGING.119.008857.

[0033] Wei et al. Transdermal optical imaging reveals basal pressure through heart rate variability analysis: a new method that can rival electrocardiography. Front Psychol 9:98. DOI: 10.3389 / fpsyg.2018.00098.

[0034] Parati et al. Role of spectral analysis of blood pressure and heart rate variability in the assessment of cardiovascular regulation. Hypertension. 1995; 25: 1276-1286. DOI: 10.1161 / 01.HYP.25.6.1276.

[0035] Rouast et al. Remote heart rate measurement using low-cost RGB facial video: a technical literature review. Front. Comput. Sci., 2018, 12(5): 858-872. DOI: 10.1007 / s11704-016-6243-6.

[0036] Lee et al. Vision-based heart rate measurement using unsupervised clustering for spherical electrocardiogram head motion. Sensors 2019, 19, 3263. DOI: 10.3390 / s19153263.

[0037] Liu et al. Transcutaneous optical imaging reveals different spatiotemporal modalities of facial cardiovascular activity. Scientific Reports, (2018) 8: 10588. DOI: 10.1038 / s41598-018-28804-0.

[0038] Unakafov. Pulse rate analysis using imaging photoplethysmography: a general framework and method comparison on public data sets. Biomedical Physics & Engineering Express 4 (2018) 045001. DOI: 10.1088 / 2057-1976 / aabd09.

[0039] Verkruysse et al. Remote plethysmographic imaging using ambient light. Optics Express, Vol. 16, No. 26, 22 December 2008, pp. 21434-21445. DOI: 10.1364 / OE.16.021434.

[0040] Ghijssen et al. Biomedical Optics Express, Vol. 9, Issue 8, pp. 3937-3952 (2018). DOI: 10.1364 / BOE.9.003937.

[0041] Yamada et al 2001 (DOI: 10.1109 / 6979.911083)

[0042] Roser and Mossmann (DOI: 10.1109 / IVS.2008.4621205)

[0043] US20150363651A1

[0044] McGunnicle 2010 (DOI: 10.1364 / JOSAA.27.001137)

[0045] Espy et al. (2010) (DOI: 10.1016 / j.gaitpost.2010.06.013)

[0046] Senden et al. (DOI: 10.1016 / j.gaitpost.2012.03.015)

[0047] Van Schooten et al. (2015) (DOI: 10.1093 / gerona / glu225)

[0048] Kasser et al. (2011) (DOI: 10.1016 / j.apmr.2011.06.004)

[0049] Furthermore, the service robot can determine substances on or in the skin, partly in contact with the skin, partly in a non-contact manner. Here, mainly spectrometer solutions are employed. Spectrometers or spectrometer-like solutions are described, for example, in US6172743, US6008889, US6088605, US5372135, US20190216322, US2017146455, US5533509, US5460177, US6069689, US6240306, US5222495, and US8552359. BRIEF DESCRIPTION OF DRAWINGS

[0050] Each figure shows:

[0051] Figure 1 A schematic structure of a service robot is shown;

[0052] Figure 2 A top view of a wheel of a service robot is shown;

[0053] Figure 3 A management system of a service robot is shown;

[0054] Figure 4 A seat is recognized by a 2D laser radar;

[0055] Figure 5 A person on a seat is recognized by a 2D laser radar;

[0056] Figure 6 A method is shown to move a person to sit down autonomously;

[0057] Figure 7 A person is navigated to a seat that meets certain criteria;

[0058] Figure 8 A door is recognized, in particular by a laser radar;

[0059] Figure 9 A fixed marking in front of an object is recognized;

[0060] Figure 10 Tagging of motion data from a stand-up and walk test is shown;

[0061] Figure 11 A repeated speech sequence is recognized;

[0062] Figure 12 Detection and analysis of a fold in paper is shown;

[0063] Figure 13 A sentence written down is analyzed by a service robot;

[0064] Figure 14 Identification of possible manipulation of the service robot by a third person is shown;

[0065] Figure 15 Manipulation or assistance by a third person is shown;

[0066] Figure 16 Calibration of the service robot is shown taking into account user obstacles;

[0067] Figure 17 Movement of the service robot towards the patient is shown;

[0068] Figure 18 Passage of the robot through a door is shown;

[0069] Figure 19 Determination of the risk of dementia and post-operative monitoring of the surgical patient by the service robot is shown;

[0070] Figure 20 Preparation of data for therapy recommendations by the service robot is shown;

[0071] Figure 21a Determination of a measurement area on the patient is shown;

[0072] Figure 21b Measurement and analysis of a spectroscopic examination is shown;

[0073] Figure 22 Output and analysis of the patient's reaction to a tone sequence is shown;

[0074] Figure 23 Analysis of the patient's image recognition for diagnostic purposes is shown;

[0075] Figure 24 Ensuring sufficient visibility of the service robot display is shown;

[0076] Figure 25 Gesture recognition by looking at displayed numbers is shown;

[0077] Figure 26 Display of two fingers and detection of the patient's reaction is shown;

[0078] Figure 27 Analysis of the mood by the service robot is shown;

[0079] Figure 28 Analysis of the patient's upper limb activity is shown;

[0080] Figure 29 Detection of the patient's cough is shown;

[0081] Figure 30 Determination of blood pressure is shown;

[0082] Figure 31 Self-learning to identify moisture on a surface is shown;

[0083] Figure 32 Navigation while detecting moisture on a surface is shown;

[0084] Figure 33 Analysis of a fall event is shown;

[0085] Figure 34 Monitoring of vital sign parameters while exercising / testing is shown;

[0086] Figure 35 Analysis of a person's gait progression in terms of fall risk is shown;

[0087] Figure 36 Progression of a mobility test is shown;

[0088] Figure 37 Determination of sitting balance is shown;

[0089] Figure 38 Determination of rising is shown;

[0090] Figure 39 Determination of rising attempt is shown;

[0091] Figure 40 Determination of standing balance is shown;

[0092] Figure 41 Determination of standing balance and distance between feet is shown;

[0093] Figure 42 Determination of standing balance / impact is shown;

[0094] Figure 43 Gait start classification is shown;

[0095] Figure 44 Determination of step length is shown;

[0096] Figure 45 Determination of step height is shown;

[0097] Figure 46 Determination of gait symmetry is shown;

[0098] Figure 47 Determination of gait continuity is shown;

[0099] Figure 48 Determination of stride deviation is shown;

[0100] Figure 49 Determination of trunk stability is shown;

[0101] Figure 50A turn around measurement is shown;

[0102] Figure 51 A turn around measurement is shown;

[0103] Figure 52 A sit down autonomously measurement is shown;

[0104] Figure 53 A signal to noise ratio optimization at skeleton model evaluation is shown;

[0105] Figure 54 Adjusting image segments when probing sensor motion is shown;

[0106] Figure 55 Navigation to detect a person on the side is shown;

[0107] Figure 56 Determining a training program configuration is shown;

[0108] Figure 57 An architectural view is shown;

[0109] Figure 58 Manipulation recognition based on audio signals is shown;

[0110] Figure 59 A scoring measurement system in combination with a rise from seat / sit down autonomously is shown;

[0111] Figure 60 A system for synchronizing actions between a person and a service robot is shown;

[0112] Figure 61 A system for collecting and evaluating origami training is shown;

[0113] Figure 62 A system for recognizing manipulations is shown;

[0114] Figure 63 A spectrometer system is shown;

[0115] Figure 64 An attention analysis system is shown;

[0116] Figure 65 A system for cognitive analysis is shown;

[0117] Figure 66 A system for determining a pain state is shown;

[0118] Figure 67 A system for blood pressure measurement is shown;

[0119] Figure 68 A system for measuring a substance is shown;

[0120] Figure 69 A system for wetness assessment is shown;

[0121] Figure 70 A system for fall detection is shown;

[0122] Figure 71 A system for detecting vital sign parameters is shown;

[0123] Figure 72 A system for determining a fall risk score is shown;

[0124] Figure 73 A human balance determination system is shown;

[0125] Figure 74 A foot position determination system is shown;

[0126] Figure 75 A turn action classification system is shown;

[0127] Figure 76 A gait classification system is shown;

[0128] Figure 77 A system for changing an optical signal of a sensor is shown;

[0129] Figure 78 An image segment adjustment system is shown;

[0130] Figure 79 A system for enabling side shooting is shown;

[0131] Figure 80 An iterative classifier generation system for a large number of joints is shown;

[0132] Figure 81 A surface wetness assessment procedure is shown;

[0133] Figure 82 A path planning when detecting wetness on the ground is shown;

[0134] Figure 83 A method for determining a foot position is shown;

[0135] Figure 84 A method for determining a turn action is shown; and

[0136] Figure 85 A method for capturing a human action along a straight line is shown. DETAILED DESCRIPTION

[0137] The term user refers to the person using the service robot 17, who in this case is mainly analyzed by the service robot 17 by means of the described means essentially in a sensor-like manner. Here, the user can refer to an elderly person, on whom the service robot 17 carries out geriatric tests, but also relatives or third parties assisting the elderly person here can interact with the service robot 17 or carry out tests on the elderly person.

[0138] Figure 1 A possible service robot specification sketch is shown, which can be referred to as mobile service robot 17. The service robot 17 has a laser scanner (lidar) 1 to scan the surroundings of the service robot 17. As an alternative and / or complementary solution, other sensors are also possible here, such as a camera (2D and / or 3D) 185, ultrasound and / or radar sensors 194.

[0139] The service robot 17 has at least one display 2, which in one aspect refers to a touch panel. In Figure 1 The shown aspect, the service robot 17 has two touch panels. The touch panels also have, for example, a microphone 193 and a loudspeaker 192, which can allow a voice communication with the service robot 17. The service robot 17 also has at least one sensor 3 for contactless three-dimensional detection of movement data of a patient. In a non-limiting example, the sensor is a Microsoft Kinect instrument. As an alternative, an Orbecc Astra 3D camera can also be used. Such a 3D camera has a stereo camera system for recognizing depth, which can analyze a skeletal model of a patient and usually also has an RGB camera to recognize colors. In one alternative aspect, a conventional monochrome camera can be employed. Here, the technology that can be used in the 3D camera is a time-of-flight sensor (ToF) or a Speckle sensor.

[0140] There are pressure-sensitive push rods 4 around the service robot 17 housing at a distance of, for example, 5 cm, at least in the area in which the service robot 17 can travel. A processing unit 9 is connected to the pressure-sensitive push rods 4 and recognizes a collision of the service robot 17 with an object. Upon a collision, the drive unit 7 is immediately stopped.

[0141] The service robot 17 has two drive wheels 6 in one aspect, which are centered to each other and arranged in parallel (see Figure 2The robot caster wheel is shown in a plan view. It has two or three further support wheels 5 around it, for example on a circular track. This arrangement of support wheels 5 enables the service robot 17 to be rotated into position by countersteering the drive wheel 6. The axes of the two or three support wheels 5 are supported in such a way that they can be rotated 360 degrees about a vertical axis. In the case of two support wheels 5, the distance of the drive wheel is greater than Figure 2 the distance shown, whereby the service robot 17 is prevented from tipping over too easily.

[0142] The service robot 17 additionally has a power supply 8 for powering the drive unit and the processing unit 9, the sensor arrangement (laser scanner 1, sensors 3 and push bar 4) and the input and output unit 2. The power supply 8 is a battery or accumulator. Alternative energy sources, such as fuel cells, are of course also conceivable, including direct methanol fuel cells or solid oxide fuel cells.

[0143] The processing unit 9 has at least one memory 10 and at least one interface 188, such as a WLAN, for exchanging data. Here, in an optional aspect there is included (not shown in the figure) a device for reading out mobile memories, such as transponders / RFID tags. In another aspect the mobile memories are also writable. In one aspect, the interface or a further interface 188, such as a WLAN, allows wireless communication with a network. The service robot 17 has rules for carrying out analyses saved in the memory 10, as described in later parts of this text. As an alternative and / or in addition, the rules can also be saved in a memory of a cloud 18, which the service robot 17 accesses via the at least one interface 188, such as a WLAN. This need not be mentioned in detail in other places, but is included in this disclosure.

[0144] The sensors 3 identify the person and their actions and create a skeleton model on the basis of the person's movements. Here, the sensors 3 can in one aspect also identify a walking aid / underarm support (UAGS). In addition, the service robot 17 optionally has one or more microphones 193, which can be implemented independently of the touchpad, in order to record the person's speech and analyse it in the processing unit.

[0145] Figure 57An architectural view is shown, but it hides the applications described in the following sections. On the software level there are different modules with basic functions of the service robot 17. Contained in the navigation module 101 are, for example, different modules. Here, a 2D or 3D environment detection module 102 is included, which analyzes, for example, environment information on the basis of different sensor data. The path planning module 103 allows the service robot 17 to determine its own path to be taken. The motion planner 104 uses the path planning results of the path planning module 103, for example, and calculates the best route for the service robot taking into account or optimizing different cost functions. In addition to data of the path planning, data from obstacle avoidance, preferred driving direction, etc., such as the expected direction of motion of a monitored person, can also act as cost functions. Aspects of the motion dynamics also play an important role here, such as speed adaptation when turning. The self-localization module 105 enables the service robot 17 to determine its own position on a map, for example, by means of distance measurement data, by comparing the environment parameters detected by the 2D / 3D environment detection device with the environment parameters saved in the map of the map module 107, etc. The mapping module 106 allows the service robot 17 to be able to map its environment. The created map is saved in the map module 107, for example, but it can also contain other maps that were not created by the service robot 17 itself. The charging module 108 is used for automatic charging. In addition, a database containing room data 109 can be provided, which contains, for example, information on which space should be analyzed by a person, etc. The motion analysis module 120 contains, for example, a motion process extraction module 121 and a motion process evaluation module 122. These contain rules for motion analysis, respectively, which will be described in detail in the following sections. The person recognition module 110 contains, for example, a person identity recognition module 111, which contains rules for determining whether it is a person or another object on the basis of the detected sensor data, for example. The visually based person tracking module 112 for virtual person tracking is based essentially on camera data as input variables, the laser-based person tracking module 113 correspondingly uses the laser radar 1. The person identity recognition module 114 can classify a person detected later as belonging or not belonging to a previously tracked person, for example, when the tracking process is interrupted. The seat recognition module 115 allows, for example, to detect a seat. The service robot 17 furthermore has a person-service robot interaction module 130, which includes a graphical user interface 131, a speech synthesis unit 133 and a speech analysis module 132. In addition, there is an application module 125, which can contain a large number of applications, such as applications for practicing and testing a person, which will be described in detail below.

[0146] On the hardware level 180 there is a odometry unit 181, an interface for communication with an RFID transponder, a camera 185, operating elements 186, an interface 188, such as WLAN, a power supply charging control system 190, a motor control system 191, a loudspeaker 192, at least one microphone 193, a radar and / or ultrasonic sensor 194, a probe 195 (which will be described in detail elsewhere), and a spectrometer 196 and, for example, a projection device 920. The laser radar 1, the display 2 and the drive device 7 have already been described.

[0147] Figure 3 The service robot 17 is shown connected via the interface 188 with a cloud 18. The treating physician can access a patient management module 160 provided in the cloud 18 using the computer 161, which is connected with a memory 162, via the terminal 13.

[0148] The medical staff can save patient data in the patient management module 160 or, in one aspect, import them from other systems via the interface 188, such as a WLAN. The other systems essentially include a hospital management system (HIS) and / or a patient data management system, which are usually used in hospitals or clinics. The patient data include, in addition to the name and the patient room number, if available, also information about the general health status. The computer 161 generates an ID for each person in the patient management module 160, which is saved together with the patient data in the memory 162. The medical staff can define the tests to be performed. The management system is connected via the cloud 18 with a regulatory authority 150, which consists of a processor 151 and a memory 152. The regulatory authority 150 provides the rules for performing and analyzing the exercises, which are consistent with the rules of the service robot 17 and are, for example, centrally maintained in the regulatory authority and are then distributed to the plurality of service robots 178.

[0149] In the regulatory authority 150 there are saved classifications of objects and movements, but also combinations of both, in order to analyze the observations on the test level. For example, there are saved positions of the double legs, the upper limbs, the arms, the hands, etc. on the basis of a skeleton model. In addition, it is possible to identify objects which are to be analyzed within the scope of the test. The regulatory authority 150 can initially be created by a person of ordinary skill in the art on the basis of templates, i.e. the boundary values of the individual limbs are specified. For the boundary values, it is also possible to use fuzzy algorithms. Alternatively, it is also possible for the medical staff to label individual images or image sequences, which are converted into a skeleton model, for example, again on the basis of images of people, and then to specify the classifications mapping the boundary values by means of machine learning algorithms, including neural networks.

[0150] In one aspect there is also a cloud-based navigation module 170 with a navigation computer 171 and a navigation memory 172.

[0151] The service robot 17 can be associated with cloud applications in the cloud 18. The treating physician can assign a mobile memory unit, such as a tag, to the person for whom the test is to be performed. The tag contains the patient ID and / or another tag ID assigned to the person or its own ID. Using this tag or serial number and / or ID, the person's identity can be recognized on the service robot 17. The identity can also be recognized in other ways, such as by entering registration data in a screen-guided menu, but also by biometric features, such as a face scan or software recognition on a mobile device, which already contains the code input or read into the service robot 17. The service robot 17 now downloads the test saved by the medical staff, but not including its personal related data, by the person ID assignment, from the cloud 18 through the interface 188, such as WLAN, respectively. After the test is completed, the service robot 17 loads the test data into the patient management module 160 after encryption - by the person ID assignment. The data is decrypted in the patient management module 160 only (see below). The medical staff can then analyze the data, as detailed by the respective examples below.

[0152] In another aspect, the medical staff can transmit a request for performing the test or its subcomponents onto a storage medium, such as a transponder in the form of an RFID tag, which the person acquires in order to recognize the identity on the service robot 17, for which the service robot has an RFID interface 183. Here, data will be transmitted from the storage medium to the service robot 17, including the person ID specified by the patient management module 160. After the test is completed, the service robot 17 retransmits the data to the storage medium, whereby the medical staff can transmit the data to the patient management module 160 when reading the storage medium. In a supplementary and / or alternative aspect, the data can also be transmitted to the patient management module 160 encrypted via a wireless or wired interface 188, such as WLAN.

[0153] The above-described solutions or data exchange can also be combined by the storage medium (or transponder).

[0154] The service robot has a sensor arrangement consisting of the camera 185, the laser radar 1 and radar and / or ultrasonic sensors 194, which are used not only for navigation purposes but also for person detection and tracking, so that these sensors in combination with corresponding software modules form a person detection and tracking unit 4605 in terms of hardware, in which other sensor arrangements can also be used, such as in combination with inertial sensors 5620, which are located on the person to be detected and / or tracked. In terms of person detection and person tracking, a person recognition module 110 can first be used, by means of which a person can be recognized from sensor data, and different sub-modules can also be used. This includes, for example, a person identity recognition module 111, which can recognize the identity of a person. Here, for example, characteristic features of the person are stored. A person can be recognized again in the person identity recognition module 114, for example after a person tracking interruption, which can be achieved by means of a visual person tracking module 112, for example by analyzing the data of the camera 185, or a laser-based person tracking module 113, for example by analyzing the data of the laser radar 1. A person can be recognized again in the person identity recognition module 114 by means of a modality comparison, in which, for example, the modalities are derived from the stored person features. A motion analysis module 120 allows different motions to be analyzed. The detected motions are first preprocessed in a motion process extraction module 121, i.e. motion features are extracted which are classified and evaluated in a motion process evaluation module 122, for example in order to recognize special motions. In terms of detecting and analyzing the motions of a person, a skeleton model can be created in a skeleton creation module 5635, which determines the joint nodes at the joints of a person and the directional vectors between the joint nodes. Feature extraction can take place on the basis of the joint nodes, for example in a skeleton model-based feature extraction module 5460. A number of special feature extraction modules and a number of feature classification modules are listed in this text, which can be placed on the feature extraction modules mentioned. In one aspect, this includes a walking feature extraction module 5605, a walking feature classification module 5610 and a gait progression classification module 5615, which also use the data of the skeleton creation module 5635.

[0155] A clarification is required in terms of the terms used: In one aspect, for example, a mentioned hand joint represents a position at which a hand can be used, for example when it comes to analyzing a person's grip on an object, depending on the analysis result, and can also include a finger joint, as long as the fingers can be analyzed within the detection distance. The following mentions are made of a person and a user. Here, a person can be understood relatively broadly, while a user usually means a person whose identity is recognized on the service robot 17. However, the terms can be used as synonyms in many places, the difference being particularly important when it comes to recognizing a manipulation.

[0156] In the context of threshold comparisons, the document sometimes refers to exceeding a threshold, which later leads to a specific evaluation of a case. Furthermore, different calculation methods can be used, which can partially lead to the opposite interpretation of the analysis result. An example is the comparison of two modalities for the re-identification of a person. If for this a similarity coefficient, such as a correlation, is calculated, for example, a high correlation above a certain threshold means that the two persons are the same person. But if there is a difference between the individual values, then a high difference value means the opposite, namely a high dissimilarity. But such alternative calculations are to be regarded as synonyms of the primary calculation of the correlation.

[0157] The use of machine learning methods makes it superfluous to determine explicit thresholds, for example for a movement process, which facilitates the modal analysis. In other words, instead of a threshold comparison of the dedicated distances of the joints from the skeleton model, a modal comparison is made which simultaneously analyzes several joints. If in the following threshold comparisons are mentioned, in particular in the context of movement processes, a modal comparison approach can also be found when using machine learning algorithms. As a basis for such a modal comparison, it is possible, for example, to detect within a time course whether a body posture of a movement process is correct or incorrect and to analyze it continuously. On the basis of the extracted features, such as the joints, a classifier can be created, which is then compared with other detected body postures specified as correct or incorrect and the resulting course of the joints derived therefrom.

[0158] Acquisition of the Barthel index

[0159] One of the tests that the service robot 17 can perform is to take the Barthel index or to perform the Barthel test. With this Barthel test, an estimate of the basic self-care abilities or the care needs, such as eating and drinking, personal hygiene, mobility and seat / urinary control, is made on the basis of behavioral observations. For this purpose, the service robot 17 is configured in such a way that it can ask the user questions about these subject areas via the communication means. The user can mean the person to be evaluated. As an alternative and / or complementary solution, other persons or relatives can likewise ask questions about these topics via the communication means. Here, the questions are asked via a menu guide of the display 2 of the service robot 17 or via a voice interface. As an alternative or complementary solution to the display 2 and / or the microphone 193 installed in the service robot 17, a separate display 2, such as a tablet, which can be connected to the service robot 17 via an interface 188, such as a WLAN, can also be used, which the person can hold in his hand or place on a table, whereby the answering and completion of the exercises can be facilitated. The differentiation between the person to be evaluated and, for example, a relative is made by means of the question dialog. As an alternative and / or complementary solution, a differentiation can also be made in a different way, which will be explained in detail in the chapter on the recognition of the manipulations.

[0160] Identification of a chair for a patient who should be tested for timed sit-to-stand and walk tests

[0161] One of the tests performed by the service robot 17 is the so-called "timed getting up and walking" test. In this test, the person to be assessed sits in a wheelchair, gets up and then walks three meters in order to return and sit down again. The time taken is detected, and the time is converted into a score on the basis of a table.

[0162] The service robot 17 scans the space in which it is located using the laser scanner 1, calculates the distance to the walls and creates a virtual map within the mapped range by means of the mapping module 106, which reflects the contours of the space, but also records objects located between the laser scanner 1 and the walls in the XY plane. The created map is saved in the map module 107. As soon as the laser scanner 1 no longer has a 360° view, the service robot 17 performs a movement so that the service robot 17 can scan its surroundings almost 360°. The service robot 17 performs this scan, for example, from different positions in the space, in order to recognize, for example, obstacles standing apart. If the service robot 17 has scanned the space once and created a virtual map, the service robot 17 can recognize the space again by rescan- ning a part of the space. The more precise the other scans, the more of the space can be scanned. Here, the service robot 17, for example, records the distance traveled and measures the distance here, whereby the service robot 17 can determine its position in the space. In addition, the distance traveled can also be measured by analyzing the rotational movement of the wheels, in combination with their circumference. If a camera 185 is used instead of a laser scanner to create the map, it is easier to determine the position, since not only characteristic dimensions can be recognized in the XY plane, but also characteristic dimensions in the Z plane, whereby unique dimensions in the space can be recognized more quickly than in a two-dimensional map.

[0163] More than one sensor can also be used when mapping the space by means of the mapping module 106, for example the combination of the laser radar 1 and a sensor 3, which is an RGB camera, for example, which can detect the coloring in the space and assign a color value to each point in the XY plane recorded by the laser radar 1. For this purpose, the processing unit of the service robot 17 performs image processing in such a way that it first assigns a Z-axis coordinate to each point in the XY plane, which is represented by the inclination of the laser radar and its height above the ground. The RGB camera, in turn, has a known relative position to the laser radar, as well as a known alignment angle and a known shooting angle, whereby it is known, for example, at which distance horizontal straight lines are in the picture, which are 2 m away and 50 cm from the ground. With these parameters, it is possible to assign a pixel in the RGB picture to each spatial coordinate determined by the laser radar 1, whereby the color value of the pixel is also determined.

[0164] By means of the laser radar 1 it is possible to determine the position of the expected seat in space. Figure 4 The recognition method is described in the middle. The seat usually has up to four legs, wherein a single-legged chair is an office swivel chair, which is not very suitable for people who can be disabled and older, since it can be rotated around the Z-axis. More likely are two-legged or four-legged chairs, wherein a two-legged chair usually means a so-called cantilever chair. The protruding feature of the chair legs is that they stand spaced apart in the XY plane, whereby the laser radar 1 can recognize the objects standing spaced apart in step 405. In addition, the chair legs have a uniform cross-sectional area in the XY plane at a constant Z-axis (step 410). Here, the diameter of the objects, i.e. the potential chair legs, is between 0.8 cm and 15 cm, or between 1 cm and 4 cm, and this is determined in step 415. The distance in the XY plane between the objects that can be spread as chair legs is typically about 40 cm, step 420. The chair legs of a four-legged chair are mainly arranged in the form of a rectangle (step 425). This means that two objects of the same diameter indicate a cantilever chair with two chair legs (step 430). If the front and rear legs of the seat have the same cross-sectional area as each other, it can belong to a four-legged chair (step 435).

[0165] The service robot 17 can now base the assignment of the "seat" attribute to these objects on this feature (two or four objects standing spaced apart, which have a roughly symmetrical cross-section, a distance of about 40 cm, and are roughly arranged in a rectangle), and determine the positions of the seats in step 440 in the virtual map created by the laser radar 1 and / or one or more further sensors, in which positions there is likely to be one or more seats. In addition, a spatial orientation is assigned to each recognized seat in step 445. The seat is usually roughly parallel to the wall, usually at a distance of between 2 cm and 20 cm from the wall, wherein this distance refers to the back of the seat. Therefore, the line between the two chair legs that are parallel to the wall and are usually 40-70 cm from the wall is assigned the "seat front" attribute, step 450, and the two areas that are orthogonal are identified as the "back" of the seat in step 455. As an additional and / or alternative solution, it is also possible to identify the face of a four-legged chair that is further away from the wall that is close as the front.

[0166] It is also possible to use a 2D or 3D camera 185 instead of the laser radar 1 to recognize the seat. In this case, the processing unit sends the images to a network service in the cloud 18 via an interface 188, such as a WLAN, and an API, which is set up to classify the images, or the processing unit employs an image classification algorithm saved on the memory 10 of the service robot 17, which is able to recognize the images created in the 2D or 3D camera 185 as a seat, including a seat with an armrest. There are a large number of algorithms that are able to perform such a classification and create models, which can then be used in the network service of the cloud 18 or in the memory 10 of the service robot 17 on the 2D or 3D camera 185 of the service robot 17, where these algorithms also include neural networks, such as convolutional neural networks.

[0167] The service robot 17 can save the seat positions independently of the seat recognition method, for example in its own memory 10, which is integrated with the navigation module 101 of the service robot 17, step 465. In addition, the service robot 17 detects the number of seats in the space 470 and counts the number of seats in a clockwise order. As an alternative to this, there can also be other orders. The seats are given a number in this order, which is saved as an object ID, step 475.

[0168] The service robot 17 can draw up a space map on the basis of the described processing method, including the seats present, i.e. determining the position of the seats in the space, including their orientation. However, for the purpose of performing the timed getting up and walking test, a person has to be sitting on one of the seats, who can also have a walking aid in the vicinity of the seat if necessary. In the case of the service robot 17 being configured in this way, so that the service robot 17 recognizes the identity of the person using the laser radar 1, the following method is employed: Figure 5 The position and orientation of the seat in the space is recognized in step 505 by the aforementioned method in the space 470. In order to recognize the legs and possibly a walking aid and to distinguish the objects designed in cross-section in the XY direction, the service robot 17 navigates around the position of the seat in step 510 by at least 20°, for example at least 45°, ideally at least 90°, step 510, where the laser radar 1 and / or one or more further sensors are oriented in the direction of the seat (step 515). Here, the service robot 17 maintains a distance of more than 50 cm, for example more than 80 cm, step 520. As a result, the prediction accuracy can be improved in order to recognize the chair legs to the service robot 17 and to allow the conclusion to be drawn that a person is sitting on the seat.

[0169] If someone is sitting on the chair, two further objects are optionally arranged in the vicinity of the chair, step 525, which are roughly circular, step 530, with a diameter of slightly less than 4 cm, such as less than 3 cm, step 535. They are likely to have a distance to each other and / or to the chair legs which is significantly different from the distance of approximately 40 cm which is present between the chair legs, step 540. Furthermore, these distances are likely to be located laterally of the chair legs, step 545. Taking this information into account, the processing unit 9 in the service robot 17 is able to identify the recognized objects as walking aids, step 550. If the majority of these features are not detected, a walking aid is not identified, step 585. For this purpose, a Naive Bayes estimation method can be used, for example. Since not every person who is supposed to be tested must have a walking aid, steps 525-550 are optional and / or not necessary in order to identify the identity of the person sitting on the chair by means of the laser radar 1. Spatially, there can be the legs of a person sitting on the chair around the front chair legs.

[0170] One or two legs can be placed in front of the front chair legs, between the front chair legs or behind the front chair legs. Thereby, a slightly funnel-shaped area is formed which extends slightly radially from the center of the chair forwards up to a maximum of approximately 50 cm beyond the connecting line of the two chair legs, step 555. The data detected by the laser radar 1 are analyzed as follows in order to ensure that two (step 560) roughly circular to elliptical objects (step 565) with a diameter of 6-30 cm, such as 7-20 cm, step 570, are identified in this area. Two legs can also be placed between the front chair legs or even behind the chair legs. The more densely objects are on the line between the two front chair legs, the more circular the shape of the objects is, step 575. If these criteria are met to the greatest extent possible, the control mechanism 150 saved in the service robot 17 recognizes a person sitting on the chair on the basis of the laser radar data, step 580. Alternatively, no person sitting on the chair is identified, step 590.

[0171] As an alternative to the laser radar 1, a person and a walking aid can also be identified by means of conventional image classification, as has been explained slightly above, for example. Here, the service robot 17 likewise improves the prediction accuracy by the service robot 17 determining the orientation of the chair from a plurality of positions, whereby a 2D or 3D camera 185 can take pictures of the chair, as described in the previous section. Instead of conventional image classification, methods which are well described in the prior art (SDK, such as Kinect SDK, Astra Orbbec SDK, OpenPose, PoseNet of Tensorflow, etc.) can also be used in order to identify a person by means of a 2D or 3D camera 185 on the basis of recognizing a skeleton.

[0172] The service robot 17 identifies the identity of the person sitting on the chair on the basis ofFigure 6 The identity of a person in the space can also be recognized, step 605, for which different alternative and / or complementary solutions can be employed: for this purpose, on the one hand, the laser radar 1 can be used to recognize two cross sections of at least 5 cm diameter, for example at least 7 cm, which are not precisely circular and whose static distance is at least 3 cm, or at least 5 cm. As a complementary or alternative solution, the identity of a person can be recognized by a 2D or 3D camera 185 on the basis of image classification, wherein, for example, the SDK mentioned in the previous section can be used. If a change in position in the space occurs over time, there is a high probability in one aspect of classifying this as a person. The service robot 17 additionally uses algorithms from the prior art, which create a skeletal model of a person by means of the SDK of the sensor, for example the camera 185, and / or third-party provider software, and track this over time, step 610, for example by means of the visually based person tracking module 112 and / or the laser-based person tracking module 113. If a person who is not seated on a seat is recognized, step 615, the service robot 17 requests this person to sit down, for example acoustically and / or optically, step 620. Here, the service robot 17 tracks the movement of this person in the direction of the seat, step 625. If the service robot 17 does not detect movement in the direction of the seat, step 630, the service robot 17 changes its position, step 635. The background to this measure is that the service robot 17 can be in the way of the person, or the sensor has not detected the person correctly, if necessary. If a person is incorrectly detected, the process is interrupted, and instead the process continues (not shown in Figure 6 step 640. Here, the service robot 17 tracks the movement again, step 645, for example by means of the visually based person tracking module 112 and / or the laser-based person tracking module 113. If the service robot 17 does not find movement in the direction of the seat, step 650, the service robot 17 requests the person to sit down again, but with a higher degree of request, for example by means of an increased volume of the voice output, an alternative voice output, a visual signal, etc., step 655. The person is tracked again in terms of movement in the direction of the seat, step 660. If no movement of the person in the direction of the seat is assumed thereafter, step 665, the service robot 17 sends a message to the computer via the interface 188, for example WLAN, which interacts with the medical staff via the display 2 and requests the medical staff to move to the service robot 17 and to provide support for the service robot 17, step 670. In an alternative and / or complementary aspect, the service robot 17 can detect the degree of detection of the person, i.e. the service robot 17 uses internal rules to determine the quality of the detection, for example a deviation from a detection threshold, on the basis of which the number of requests issued by the service robot 17 to the person is determined.

[0173] Since there are more than one seat in the space that can be occupied by a person and not every seat has enough space in front of it to pass the necessary 3m distance, the service robot 17 has a correction mechanism. This correction mechanism provides that the service robot 17 identifies from the number of seats identified (step 705) whether there is a free space in front of the seat facing the vertical direction of the seat front with a length of at least 3.5m, such as at least 4m, step 710. If there is a corresponding free space in front of the seat for performing the test, this feature is saved as an attribute in the memory 10 of the service robot 17, step 715. These information is used when navigating the user to the seat or by ensuring that the seat is occupied by a person, which is suitable for performing the test through the sufficient space in front of the seat. As a complementary and / or alternative solution, the seats can also be identified by floor markings, such as in some of the following chapters.

[0174] For this, the service robot 17 can indicate the appropriate seat to the standing person in step 720 when requesting the person to sit down. The service robot 17 can also request the person to get up again and then sit down to another seat, step 725. In one aspect, the seat is identified in detail in step 730. For this, the service robot 17 uses the object ID and the order of locating the seats. In addition, there is information in the memory 10 of the service robot 17, for example, that a person is sitting in seat 6, but seats 4 and 7 are suitable for performing the test, since there is enough space in front of them. The service robot 17 can implement in the request sent to the person the information that the person can sit down to the seat or two positions on the left side of the service robot 17 or one position on the right side of the service robot 17. In this case, the service robot 17 can correct such information based on the orientation in the space occupied by the person and the service robot 17 so that the output information is related to the orientation or perspective of the person. In the example described, they are two positions on the right side or one position on the left side of the person. Similarly, the service robot 17 can also use the coordinates of the standing person and the coordinates of the appropriate seat in order to direct the standing person to sit down to the seat or to direct in the manner "please sit down to the seat on your oblique left side", if necessary, specifying the distance. In a complementary and / or alternative aspect, the color information of the seats can also be included here, which can be obtained by the RGB camera, for example, before.

[0175] If the seat is empty and the service robot 17 does not recognize a person in the space, but the service robot 17 has received an instruction to test a person using the seat, the service robot 17 will position itself, for example, at a distance of more than one meter from the seat. In one aspect, the service robot 17 has information via its navigation module 101 which indicates from which direction the patient approaches. These information can be saved explicitly in the system on the one hand.

[0176] In one additional or alternative aspect, the service robot 17 can recognize a door or a passage. In the case of a closed door, there is a misalignment of at least 1 cm in the wall which is oriented perpendicularly to the wall surface, step 805, and the misalignment in the perpendicular direction is more than 77 cm, for example more than 97 cm, but less than 120 cm, step 810. As a complementary solution or alternative to this, it belongs to a double misalignment of a few cm, wherein the inner distance reaches the mentioned about 77 cm, preferably about 97 cm. With the aid of these information, in particular a closed door can be recognized by means of the laser radar 1. In the case of an open door, the service robot 17 recognizes by means of the laser radar 1 a plane which approaches one of the edges in the XY direction over a length of about 77-97 cm (step 815) and an angle of 1-178° to the edge (step 820), with the possibility of a variable angle over the course of time (step 825), and / or a distance of at least 90 cm behind the recognized opening without further restrictions of the XY surface in the map which is drawn by the service robot 17, for example by means of the laser radar 1, step 830.

[0177] In the case of a 2D or 3D camera 185, on the one hand algorithms based on characteristic door feature curves can be used again. On the other hand, it is also possible to process the information from the Z direction and, if necessary, to combine it with the data from the XY direction, if the region which is recognized as a possible door or passage in the XY surface has a height limit of 1.95-2.25 m, these information help to recognize a passage with a greater probability. As a complementary solution and / or alternative to this, it is also possible to include object information which relates to a door handle.

[0178] For the case that the seat is not occupied, the service robot 17 creates a direct path between the seat and the door or the passage which is not blocked by an obstacle, for example by determining the Euclidean distance, based on the determined position of the door or passage and the position of the seat, by means of its navigation module 101. The service robot 17 positions itself outside this path, for example so spatially that its sensors can inform about the seat and / or the door / passage.

[0179] If the service robot 17 recognizes a person entering the space, the service robot 17 will request him to sit on the seat as described above.

[0180] If there is now a person on the seat, the service robot 17 will request the person to get up, walk three meters straight ahead and then return to the seat again, by means of an output unit, such as a loudspeaker 192, as an alternative and / or complementary solution, but also by means of the display 2.

[0181] The service robot 17 can recognize the distance marker on the floor by means of a 2D or 3D camera 185, for which common modal recognition methods are used. The service robot 17 uses the recognized position information of the seat in a first step. To make sure that it is a distance marker and not, for example, a normal floor pattern, the service robot 17 first determines the position in the space that is 3 m away from the front face of the seat, approximately perpendicular to it, by means of its navigation module 101. Subsequently, the floor area located approximately at this position is scanned to recognize such a marker. Other floor areas are scanned to recognize whether the pattern is unique or repeated. If it is unique, or saved in the memory 10 of the service robot 17, if applicable, the pattern is used as a 3 m point marker.

[0182] One disadvantage of a fixed floor marker is that a person sitting on the seat, etc. can move the seat to a varying extent when cleaning work is being carried out in the space. Therefore, in an alternative and / or complementary aspect, the service robot 17 is equipped with a projection device 920 for projecting a marker from the front face of the seat, approximately perpendicular to it, at a distance of 3 m. The XY coordinates of the seat and the 3 m point are again determined by, for example, the navigation module 101, which is updated by the service robot 17 in advance on the basis of the seat position. For this purpose, light sources, such as lasers or LEDs, can be used, which can again be focused by means of lenses or tools with a similar function. The projection device 920 can project a plane onto the floor, a bar, if necessary with characters that signal that the person should move there. The projection device 920 is in one aspect (see the top view in Fig. 9) arranged movably independently of the service robot 17, whereby the service robot 17 can, for example, by means of a rotational movement, always be positioned in the front face in the direction of the person 910 (as indicated by the line 940), while the projected marker 915 is always located in the same position perpendicular to the front face 905 of the seat. Here, the light source of the projection device 920 is in one aspect arranged movably, in another aspect a mirror, such as a micro mirror or a microstructured element, ensures such control of the light, so that the light remains in the same position, for example, when the service robot 17 is moving, for example, rotating. As a result, the angle between the time points a) and b) changes when the person 910 moves in the direction of the marker 915, which includes the angle between the lines 925 and 935 and 930 and 935. In an alternative and / or complementary aspect, the service robot 17 can also move parallel to the direction of travel of the person. Figure 9 a) to b)) independently of the service robot 17, whereby the service robot 17 can, for example, by means of a rotational movement, always be positioned in the front face in the direction of the person 910 (as indicated by the line 940), while the projected marker 915 is always located in the same position perpendicular to the front face 905 of the seat. Here, the light source of the projection device 920 is in one aspect arranged movably, in another aspect a mirror, such as a micro mirror or a microstructured element, ensures such control of the light, so that the light remains in the same position, for example, when the service robot 17 is moving, for example, rotating. As a result, the angle between the time points a) and b) changes when the person 910 moves in the direction of the marker 915, which includes the angle between the lines 925 and 935 and 930 and 935. In an alternative and / or complementary aspect, the service robot 17 can also move parallel to the direction of travel of the person.

[0183] In another aspect, Figure 9c) to d)), the light source is able to project onto a face of the floor which, from the perspective of the service robot 17, has a width of more than 3 m. Thereby, the projection device 920 covers the section of the path that the person should walk from the seat. Here, the central axis of the projection device 920 is rotated by an angle of 10° to 60°, such as 20° to 50°, from the central axis of the camera 185 with respect to the Z-rotation axis of the service robot 17, i.e. in the direction in which the person should move from the service robot 17. If, for example, there is a person on the seat which is located on the left side of the service robot 17 and the 3 m point is located on the right side of the service robot 17, the projection mark, such as a bar, is on the right edge of the projection face from the perspective of the service robot 17. After the service robot 17 is rotated towards the fixed 3 m point, the projection mark will move towards the left edge of the projection face. Here, for example, a projection device 920 can be used which is located in a conventional (LCD) data and video projector, in which the matrix is manipulated in software in such a way that different areas of the projection face are illuminated to different degrees. If the seat with the person is on the right side of the service robot 17 and the 3 m point is on the left side, the orientation is correspondingly reversed. Figure 9 c) In c), a person 915 is sitting on a seat 905. The projection device 920 is able to illuminate a face which is composed of the dashed rectangle. The 3 m point mark 915 is in the right area. By rotating the service robot 17, the projection face moves clockwise together with the service robot 17, here keeping the 3 m mark at a fixed XY coordinate, which means that the 3 m mark moves towards the left area of the projection face. Figure 9 d)) Here, the illustration assumes a fixed projection direction 920. In an alternative and / or complementary aspect, the projection device 920 is arranged movably (the effect is not shown in detail).

[0184] As an alternative to this, the service robot 17 is not rotated and the complete section that the person has to pass through is detected by means of the laser radar 1 and / or the 2D or 3D camera 185 (see Figure 9 e), wherein the smallest area detected by the sensor is at least the dashed line. In another aspect, the 2D or 3D camera 185 is arranged adjustably and the projection device 920 or the light source is arranged rigidly or likewise adjustably (not shown separately).

[0185] The processor in the service robot 17 calculates the projection face on the basis of the coordinates of the navigation module 101 of the service robot 17, the inclination of the projection device 920 and its height, whereby the positions of the seat, the 3 m point and the service robot 17 can be determined in advance in order to project a bar which appears to the best extent possible undistorted to the person who is completing the exercise, which is approximately parallel to the front of the seat. Depending on the embodiment, the shape of the mark can also be different.

[0186] As described in the prior art, the person can be tracked by means of the visually person tracking module 112 and / or the laser-based person tracking module 113. To this end, the service robot 17 likewise detects the posture of the upper limbs in order to recognize when the person starts to stand up. From this point in time, the time required by the person to complete the test is likewise detected. The timing ends when the person rotates again and sits back down in the chair after returning to the chair. The rotational movement is recognized as a modality by means of an algorithm, whereby a skeletal model can be generated by means of the person's joint nodes, the arms in the rotational movement can be approximately parallel to a plane, which coincides with the path to be covered. In one aspect, the joint nodes of the arms are analyzed in the time course, and it is determined that the joint nodes present symmetrically change by more than 160° with respect to a straight line connecting the start position and the reversal position.

[0187] In an alternative and / or supplementary aspect, the service robot 17 is configured in such a way that the service robot 17 determines the distance covered by the patient when walking a path. Since the start position and the reversal position are 3 m apart, the length of the path walked, which begins and ends at the chair at the start position, is 6 m, and it is also the end position, the reversal position is 3 m from the chair. Here, the service robot 17 does not have to detect markings on the ground. The distance of the path walked can be determined in different ways, including the addition of the step length. This can be based on the distance of the ankle joints or ankle joints, which are recognized by means of the 2D or 3D camera 185 in combination with the analysis framework used here, which assigns their points in three-dimensional space, which are determined by the service robot 17, for example, in vectorial terms. As an alternative and / or supplementary solution, the path walked by the patient can also be determined by adding the Euclidean geometric distances between the coordinate points covered by the patient, which can also be determined from a map of the surroundings in which the patient and the service robot 17 are located, wherein the coordinates of the patient can be determined with respect to a reference position. Here, the distance to the recognized spatial limits or the position of the service robot 17 can be determined by self-localization (self-localization module 105).

[0188] During the tracking of the patient, for example, by means of the visually person tracking module 112 and / or the laser-based person tracking module 113, the service robot 17 calculates the distance walked, which is set in relation to the total distance that the patient has to walk. The service robot 17 can feedback to the patient by means of output means, such as the display 2 and / or the speech synthesis unit 133, how far he still has to walk, how many steps he still has to take, when the patient can rotate, etc.

[0189] The service robot 17 transmits the score on the basis of the detected time on the basis of the reference data saved in the memory 10 of the regulating authority 150. The service robot 17 can transmit the score and / or the detected time to the patient management module 160 in the cloud 18 by means of the interface 188, such as a WLAN.

[0190] As Figure 10 illustrated, in one aspect, the service robot 17 is able to detect movements with its sensors 3 in step 1005, record these movements as a video in step 1010, save in step 1015, and transmit in step 1030 to a cloud storage in the cloud 18 via an interface 188, such as a WLAN, which is located within the regulatory authority 150. The data transmission is carried out in encrypted form. The facial features to be evaluated are previously masked, whereby the person remains anonymous, step 1025. The video material is provided within the regulatory authority 150 for the purpose of tagging in order to continue to improve the reference data of the regulatory authority 150 with the aid of self-learning algorithms. For this purpose, the saved data can be accessed via a terminal 1030, whereby a medical staff member is able to identify the video taken and tag it, step 1035. Here, tagging means classifying the body posture of the person manually, such as the person sitting into a seat, the person standing straight, the person moving forward or backward, the person rotating, etc. For these events derived from the video sequence, a tag can be assigned to the point in time. Here, for example, the start or end point of a movement is time-stamped, and at the same time the movement, such as the body posture describing the orientation of a limb part, for example, in the course of time, is categorized / classified. Subsequently, the data tagged in this way, for example, is saved in a database, in which the library data is also saved, step 1040. Subsequently, the regulatory authority 150 is able to improve the classification rules autonomously by means of algorithms, such as neural networks. The improvement is basically achieved in two ways: a) detecting situations that have not been described before, as they occur occasionally, and b) increasing the number of events. Thereby, both ways enable a more precise weight estimation in the range of the performed classification, step 1045. Here, the vector space resulting from the body posture of the patient, its movements, etc. is assigned accordingly, whereby the posture of the person to be evaluated can be estimated more easily. Here, this includes getting up from a seat, walking, rotating, and sitting down again by oneself. For better classification, the new weights are saved 1050 in the regulatory authority 150 and transmitted in an upgrade manner to the service robot 17 via the interface 188, such as a WLAN.

[0191] Figure 59The system for detecting and analyzing the movement of getting up and sitting down on a chair is summarized as follows: The system comprises a processing unit 9, a memory 10 and at least one sensor for contactlessly detecting the movement of a person, wherein the memory 10 of the system has a chair detection module 4540, an output device (such as a loudspeaker 192) and / or a display 2 for transmitting instructions, a duration-path module for determining the duration of a path 4510 traveled and / or a speed-path module 4515 for determining the speed of the detected person on the path, and a duration-path evaluation module 4520 for evaluating the speed of the person on the path and / or the duration of the time taken to travel the path. In addition, the system can also have a hearing test unit 4525 for performing a hearing test, a vision test unit 4530 and / or a mental ability test unit 4535. The system can be a service robot 17. In one aspect, the system has a projection device (920), for example in order to project a marker of the point of rotation and / or the starting point. In one aspect, the system has a person recognition module 110, a person identity recognition module 111, a tracking module (112, 113), a motion analysis module 120, a skeleton creation module 5635 and / or a feature extraction module based on a skeleton model 5640.

[0192] Simple psychological test

[0193] Simple mental test: conversation exercise

[0194] In addition, the service robot 17 is configured in such a way that it can perform a simple psychological test. The goal of the simple psychological test is to identify cognitive limitations, such as senile dementia. Within the scope of the test, questions are put to the patient by means of the communication means (voice input and output, display 2) of the service robot 17, which the patient can answer by means of the communication means of the service robot 17 (for example in the form of voice input, with a screen on which there are selected answers, with manual input, such as date, place of residence, etc.). For this execution, on the one hand, the display 2 of the service robot 17 can be used, and on the other hand, a separate display 2, such as a tablet, which is connected to the service robot 17 via an interface 188 (such as WLAN), can be used, which the person can hold in his hand or place on a table, whereby it is possible to easily answer and complete the exercises.

[0195] The service robot 17 is configured in such a way that it can communicate with a person, as in Figure 11As disclosed in the method described above. To this end, in one aspect, the service robot 17 can be aligned in space such that the display 2 of the service robot 17 is approximately parallel to an axis extending through the user's shoulders, hips and / or knees and extending through the skeleton model, which axis is identified by means of the 2D or 3D camera 185 and its SDK. Thereby, the service robot 17 is aligned with the user, step 1105. Within the scope of the interaction with the user, at least one speech sequence saved in the memory 10 is played back by means of the loudspeaker 192, and the user can be requested to repeat the played speech sequence by means of the display 2 and / or speech output, step 1110. Upon making said request, the service robot 17 records the user's sound signal by means of the microphone 193, step 1115, and records, in particular as long as the outputted repeated speech sequence, step 1120. As with the following steps, this is effected by means of the speech analysis module 132. The service robot 17 analyzes the amplitude of the signal within the time range, step 1125. If the amplitude drops to zero or near zero, such as less than 90% of the maximum amplitude, for more than 1 second, such as more than 2 seconds, the recording is ended, step 1130. Sampling is continued, wherein the sampling width is defined by the phase in which the amplitude nears zero for more than 1 second and is at least 70% of the length of the speech sequence the user should repeat and which is saved in the service robot 17, step 1135. In this way, it can be ensured that multiple speech attempts of the user are detected and analyzed separately. The service robot 17 compares the samples in the time domain or in the frequency domain and calculates a similarity value, step 1140, taking into account methods customary in the field of audio technology, in particular cross-correlation. If, here, for example, the similarity value is below a threshold value, step 1145, the service robot 17 can, in one alternative or supplementary aspect, request the user to repeat the speech sequence again (connection 1145=>1110). If the similarity value exceeds a certain threshold value, the service robot 17 modifies the values relating to the user in the database in the memory 10 of the service robot 17, step 1150. The recorded speech signal emitted by the user is saved, step 1155, and is transferred to the patient management module 160, step 1160, together with the modified values from the database by means of the interface 188, such as WLAN. In one supplementary or alternative aspect, only the highest similarity value of the recorded speech sequences and modal sequences by the service robot 17 is saved. The system furthermore calculates the number of repetition attempts and, if the number of attempts exceeds a certain threshold value, interrupts the recording of the relevant repetition attempt and proceeds to the next speech sequence to be repeated. Likewise, the multiple repetition attempts or failed repetition attempts of the user are also noted in the database.

[0196] Simple mental test: folding exercise

[0197] The exercises of the simple psychological test include taking paper, folding of the paper and putting down the paper / letting the paper fall down, such asFigure 12 The collection and evaluation of the paper folding procedure is illustrated. For this purpose, the mobile service robot 17 has an optional device which contains the paper to be evaluated, which the person to be evaluated can take up in the test, for example, on request of the service robot 17. Alternatively, the mobile service robot 17 can also remind the person to be evaluated to have such paper in the space in which the test is to be performed. For this purpose, the output unit of the voice output and / or the display 2 is correspondingly configured, step 1205.

[0198] The service robot 17 is thus configured in such a way that the user's two hands can be recognized, detected and tracked by means of a sensor 3 which is a 3D camera, for example, an embodiment of a time-of-flight camera (TOF), i.e. in a first step the two hands are recognized, step 1210, and in a second step the hands are tracked while the person is folding the paper, step 1215. It is also possible to use a solution based on individual 2D cameras to recognize the two hands (step 1210), to track the (hand) movements (step 1215) instead of the ToF camera, in order to recognize the respective pose or the folded paper, step 1220. Here, the weighting is derived, for example, from a model which classifies by means of conventional machine learning methods, for example, regression methods and / or neural networks, such as by means of a convolutional neural network. For this purpose, a large number of folding movements are filmed beforehand, labeled and learned from the conventional algorithms. As an alternative and / or complementary solution, it is also possible to create a skeleton model based on the architecture by means of 2D cameras, for example, by combining OpenPose or PoseNet with Tensorflow.

[0199] The movements are detected in this time, for example, by means of a visual human tracking module 112 and / or a laser-based human tracking module 113. Here, in a first step the two hands are recognized, step 1210, and segmented from the overall filming. In a second step the objects in the two hands are recognized by means of the segmentation, step 1220, for example, by means of a fault-tolerant segmentation algorithm which allows modal recognition, for example, a RANSAC architecture. The movements in the time course are detected by means of tracking methods which are well described in the prior art, step 1215. Here, at the beginning there is no paper in the hands. Then, the user takes a piece of paper, folds it and then moves the paper in the negative Z-axis direction, i.e. in the direction of the floor. At the last movement of the paper, one or both hands of the user do not necessarily have to intervene. The paper is determined, for example, by means of paper classification, i.e. two- or three-dimensional data of the camera 185 which is created by filming the paper and labeling the images beforehand. The term paper encompasses paper and materials which have an equivalent effect and / or similar dimensions in the exercise and possibly the same characteristics as a piece of paper.

[0200] The service robot 17 requests the user to pick up a piece of paper at the beginning of the exercise, step 1205. For this, a speech output is made, for example, using the loudspeaker 192 of the service robot 17. As a complementary or alternative solution, a display on the display 2 can also be used or a combination of both. From the point in time of the request, object recognition is used, for example, for the paper. The service robot 17 similarly requests the user to fold the paper, for example, in the middle, step 1225. Here, the service robot 17 observes the folding process and after the end of the folding process, the service robot 17 requests the user to put down the paper or let it fall. As a complementary or alternative solution, the information about the folding and / or putting down or letting it fall can also be issued directly after the previous request for the same exercise.

[0201] In one aspect, a 3D camera is used, for example, a Kinect or Astra Orbbec. The challenge in recognizing the individual elements of the hand, i.e. the fingers and the finger tracking derived therefrom, step 1230, is that the line of sight of the camera 185 can be obscured by the individual fingers and thus a direct estimation is not possible. The same is true for gestures performed when there is no object in the hand. In another aspect, if the folding of the paper is performed with one or more hands, depending on the type of folding process, a part of the fingers is also obscured. The folding process can be recognized therefrom or classified as a folding process, for example, depending on the finger movement, step 1235, for example, at least one thumb and at least one, up to several fingers of the same hand touch at the height of the fingertips, step 1240, i.e. for example, at least two fingers are detected and tracked. Alternatively, one or more fingers of one hand touch one or more fingers of another hand, for example, in the area of the fingertips, step 1245. In any case, the paper is engaged by at least one finger, step 1250. For example, there is paper between these fingers, which is recognized as described in the next section.

[0202] The system and method are an alternative and / or complementary solution to the recognition of a piece of paper and its shape change (step 1252), wherein the paper is in contact with or engaged with at least one finger. Here, in one aspect, the four corners of a piece of paper in the user's one or two hands are recognized, step 1255. Here, each corner is tracked individually over time, step 1260, and the distance between the corners is determined, step 1265. Here, a successful folding is recognized, for example, by the following: a) the distance of two corners each in three-dimensional space is reduced by more than 90%, such as more than 98%, step 1270. As an alternative and / or complementary solution, the distance between two opposite sides of the paper can also be tracked, and if the distance is less than a specified value, the folding process is recognized. As a complementary and / or alternative solution to this, (b) the surface of the paper is tracked according to its curvature, step 1275. To this end, the folding module determines the center between two corners 1277, respectively, and monitors (tracks) the curvature of the paper, for example, in this area, step 1279. In this case, a successful folding is recognized, step 1280, by the following: namely, the curvature in this area increases over time, step 1282, while the paper sides / edges near the corners are slightly parallel, step 1284 (especially the folded corners), and the distance of the paper sides decreases significantly, step 1285, such as to a distance of less than 2 mm, whereby a separate detection of two approximately equally sized paper parts is usually no longer necessary, since the depth resolution of the camera 185 cannot detect two stacked papers due to the paper being thinner. As a complementary and / or alternative solution to this, (c) the area of the paper in three-dimensional space is also detected over time, wherein a depth of less than 2 mm of the paper cannot be detected or can only be detected poorly. A folding of the paper is determined by a decrease in the area of the paper over time of more than 40%, such as about 50%. This solution can also be implemented without explicitly analyzing and tracking the fingers, for example. As an alternative and / or complementary solution to this, (d) the distance of the two ends of the paper edges that are parallel to each other is detected and analyzed, step 1293, and a folding is recognized when the distance of the two ends of the paper to each other is less than 20 mm, step 1294. By a combination of two or these three detection variants, the detection accuracy can be improved overall. If this exercise is successfully completed, i.e. the paper is folded, and then moved in the direction of the center of the earth, step 1295, or alternatively placed on a flat surface, step 1297, this is noted in the database, step 1299, in particular in a database that saves test results.

[0203] In summary, Figure 61A system for recognizing a folding exercise is shown, such as a service robot 17: the system comprises a processing unit 9, a memory 10 and a plurality of modules in its memory 10 for non-contact detection of a person's motion, such as 2D and / or 3D cameras 185, laser radar 1, radar and / or ultrasonic sensors 194. This includes a paper detection module 4705, a folding motion recognition module 4710 for determining a folding motion of a paper, a skeleton creation module 5635 for creating a skeleton model of a person, a paper distance corner edge module 4720 for detecting a distance of an edge and / or corner of a paper, a paper shape change module 4725 for detecting a shape change of a paper, a paper bend module 4730 for detecting a bend of a paper, a paper size module 4740 for detecting a size of a paper and / or a paper edge orientation module 4745 for detecting an orientation of a paper edge. Furthermore, the memory 10 comprises a fingertip distance module 4750 for detecting a distance of at least one fingertip and a paper detection module 4705, such as consisting of a paper segmentation module 4755 for detecting a paper and / or a module 4760 for paper classification. Furthermore, the system has an output device, such as a loudspeaker 192 and / or a display 2 for transmitting instructions and an interface 188 to a terminal 13. In one aspect, the system has a person recognition module 110, a person identity recognition module 111, a tracking module (112, 113), a motion analysis module 120, a skeleton creation module 5635 and / or a skeleton model based feature extraction module 5640. From the beginning of a process, at least one hand of a person is detected, recognized and tracked; a paper is detected, recognized and tracked and the detected size, shape and / or motion of the paper and the hand elements are recognized together as a folding process. In one aspect, the paper is also recognized by a fault-tolerant segmentation algorithm and a classification of the folding process, such as a paper classification and / or a classification based on two- or three-dimensional modalities, including shape modalities and / or motion modalities.

[0204] Simple mental test: sentence exercise

[0205] Within the scope of the test, the service robot 17 is also able to request the user to spontaneously come up with a sentence, the spelling and grammar of which are irrelevant in the analysis thereof, but which must contain at least a subject and a predicate. To this end, the service robot 17 requests the person to be evaluated to autonomously come up with a spontaneous sentence, step 1305, by means of the communication means (display 2; loudspeaker 192), and to write this sentence by means of a finger to the touchpad of the service robot 17, step 1320. This can take place by means of a display output 1310 or a speech output 1315. In a second aspect, the service robot 17 provides a pen or a similar object to this end, step 1320. In a third aspect, a pen and paper are made available to the person to be prepared, in order to write down the sentence, step 1325, and the service robot 17 requests the person to place the written paper in front of the camera 185 of the service robot 17, step 1330, to record this and to save it to the memory 10 of the service robot 17. To this end, the sensor means (2D, 3D camera 185) track the movement of the user 1335, for example by means of a visual person tracking module 112 and / or a laser-based person tracking module 113, with internal object recognition of the paper (see the aforementioned solution) and the recognition that the user places the paper in front of the 2D camera of the service robot 17, step 1340, and the service robot 17 recognizes the paper (step 1345) that the service robot 17 takes a picture of (step 1350).

[0206] In the next process step, the sentence contained in the image is subjected to OCR, step 1355. Here, the processor of the service robot 17 uses a corresponding database that has been created in order to process images or texts that can be subjected to OCR. Depending on the specific aspect, such data processing can take place in the cloud. In the next step, a natural language parser 1360 is used in order to determine whether a subject and a predicate are present in the sentence. To this end, the detected sentence is broken down into individual words (tokenization) in a first step, step 1365. Subsequently, the base form of the words is formed (stemming and / or lemmatization), step 1370. Subsequently, POS tagging (part of speech) takes place, by means of which the words are classified as subject, predicate, object, etc., step 1375. In this context, a solution based on neural networks can also be used. To this end, toolkits such as NLTK or SpaCy can be used. The result is saved to the memory in step 1380, and in the next step 1385, it is compared whether a subject and a predicate are present in the sentence given by the user. In this case, the successful completion of the exercise is saved in the database (step 1390).

[0207] Simple mental test: pentagon exercise

[0208] Another test element comprises drawing two overlapping pentagons. For this, in one aspect the person to be assessed is enabled to draw on a display 2, which is located on the service robot 17. In a second aspect, the display 2 is freely movable within the space in which the service robot 17 and the user are located, but is wirelessly connected to the service robot 17 via an interface 188, such as a WLAN. Here, the drawing can be made with the fingers or by means of a pen suitable for tablets. In a third aspect, the drawing can be done on a piece of paper, wherein the person who has drawn is requested by the service robot 17 via a communication means to place the drawn picture in front of a camera 185 of the service robot 17. The camera 185 takes a picture. In this regard, these processes are similar to the processes described in 1305 to 1350 in Figure 13

[0209] The detected picture is compared by the computer with pictures saved in a database, wherein the features of the picture are compared with the features of the classified pictures using the regulating mechanism 150, and an assignment is made on the basis of probabilities. For the classification mechanism, methods can be used which have been described in the prior art and which have been created on the basis of previous automated exercises, essentially using neural network methods. Alternatively, it is also possible to use a classification mechanism which is created without exercises, which classifies the features of the pentagons on the basis of characteristic features of the pentagons, which are determined in the form of defined rules, such as the number of corners and lines, for overlapping pentagons. This takes into account, for example, rounded edges, non-straight line courses and lines which do not form a closed pentagon. Within the scope of this analysis, it is possible to use a smoothing scheme to simplify the classification. If a certain threshold value is reached in the similarity comparison, such as the degree of association, between the comparison model recorded by the service robot 17 and saved in the regulating mechanism 150 or the recognition rules of the two overlapping pentagons, it is saved in the database that the exercise has been successfully completed.

[0210] Manipulation recognition

[0211] ​The service robot 17 comprises a function to recognize a manipulation by a third person when the exercise is completed. For this purpose, the sensor means for analyzing the user and his activities are able to detect whether there is another person in the space, step 1405. Here, it is analyzed whether a person, including the user, is positioned in the space during the test so that he can manipulate the service robot 17, i.e. whether he is within a so-called "critical distance" from the service robot 17, step 1410. Manipulation also includes in one aspect the input of data on the display 2 of the service robot 17. Here, the distance of the person from the service robot 17 is determined, and it is then determined whether the person is positioned close enough to the service robot 17 to be able to input, if necessary, on the display 2, by at least one of the following three ways: a) a total distance value is assumed, such as 75 cm. If this value is less, the service robot 17 assumes that the display 2 can be manipulated, step 1415. As an alternative and / or complementary solution, the distance of the person's hand and / or fingers from the service robot 17 can also be detected, wherein a distance that is shorter than the distance at which the person is located is assumed to be a possible manipulation. b) the arm length of the person is determined by means of a skeleton model, step 1420, in particular by determining the distance between the shoulder joint and the hand joint or the finger joint. If this distance is less, the service robot 17 assumes that it can be manipulated, step 1425. c) an average arm length is derived from the size of the person determined by the service robot 17, step 1430, step 1435 (for example saved in the memory 10), and once this distance is less, manipulation / ability to manipulate is assumed, step 1425. As a complementary solution to these three solutions, the service robot 17 can calculate the positioning of the person in the space with respect to the position of the display 2, step 1440. If the alignment of the shoulders, hip joints, etc. or the front of the person derived therefrom is approximately parallel or at an angle of less than 45° to the display 2 and this person has been aligned in the direction of the display 2, for example derived from the initial movement direction of the person, the arm, head, knee, foot posture and facial features, etc., the probability of interaction with the display is increased. Depending on the orientation of the sensor means of the service robot 17, this solution can also be implemented for other elements of the service robot 17 instead of the display 2, such as the off button. In this case, the screen of the display 2 in relation to the person is not considered, but a virtual screen orthogonal to the symmetry axis of the operating element 186, oriented towards the center of the service robot 17. In an optional second step, the sensor means analyze whether an input or manipulation of the service robot 17 by the user or a third person is made, step 1450.As generally described in the prior art, for this the service robot 17 tracks people in its surroundings based on characteristic features, such as based on height, limb size, gait characteristics, such as the color and texture of their clothing surface, etc., step 1445, for example by means of a virtual person tracking module 112 and / or a laser-based person tracking module 113. The user and third person are distinguished by the recognition taking place at the service robot 17, wherein the recognized person is assumed to be the user. This is achieved by observing the input content on the display 2 by means of optical sensors of the service robot 17. In summary, here the orientation of a person relative to the service robot 17 can be determined by measuring the angle between the front of the person and an axis lying vertically on the operating element 186 of the service robot 17, respectively projected into the horizontal plane, and comparing the measured angle with a threshold value, wherein below the threshold value an increased probability of manipulation is detected. In one aspect, the person can be logged on at the service robot 17 and the identity features of the person detected and saved, for example after which the person can be detected and tracked, the identity features of the person detected, the detected identity features compared with the identity features of the person saved at the time of login and with a threshold value, wherein here the similarity of the comparison is compared and the threshold value implies a minimum similarity. Here, if the value is below the threshold value, an increased probability of manipulation is detected, and if the threshold value is exceeded, a lower probability of manipulation is detected. Finally, the determined probabilities of manipulation can be multiplied in order to determine a manipulation score, which is saved together with the detected person and the analysis result, for example at the time of or after the analysis by the robot. Depending on the comparison manner, other interactions can also be made, as described at the outset.

[0212] Figure 62One aspect of a system for recognizing manipulations is shown. The system or service robot comprises a processing unit 9, a memory 10 and sensors for contactlessly detecting movements of at least one person, such as a 2D and / or 3D camera 185, a laser radar 1, a radar and / or an ultrasound sensor 194. The system comprises a module whose memory 10 contains rules. This comprises, for example, a manipulation attempt detection module 4770 which detects a manipulation attempt by at least one person detected in the surroundings of the system, a person identity recognition module 111, a person-robot distance determination module 4775 for determining the distance of at least a person from the service robot 17, a size-arm length orientation module 4780 for determining the size, arm length and / or orientation of at least one person, and / or an input registration comparison module 4785 for comparing whether it is the person recognized by the system as having an identity who inputs on the system, for example by operating an element 186. Furthermore, the system also comprises, for example, output means, such as a loudspeaker 192 for transmitting instructions, a display 2 and / or an interface 188 to a terminal 13. In one aspect, the system has a person recognition module 110, a tracking module (112, 113), a motion analysis module 120, a skeleton creation module 5635, a feature extraction module based on a skeleton model 5640 and / or a motion planner 104.

[0213] In order to rule out that a third person is merely inputting in accordance with the user's instructions, the spoken exchange between the people is analyzed by means of the existing microphone 193 (in Figure 14In this case, the voice signal is recorded by at least one integrated microphone 193 within the surroundings of the service robot 17, step 1560. The recognition of the voice source is implemented in two alternative or complementary ways, for example also in the voice analysis module 132. For this purpose, on the one hand, a visual analysis of the lip movements can be carried out, which are first recognized, step 1565, then tracked, step 1570, and then time-synchronized with the voice signal recorded by the service robot 17, step 1575. For recognizing the speaking movements of the lips, image recognition and tracking methods from the prior art are used. Thereby, this enables the service robot 17 to recognize who the logged voice comes from, whether they are in agreement with the user for whom the exercise should be performed, wherein for example the voice of the user can be detected when the service robot 17 recognizes the user identity. In the opposite case, there can be a manipulation, step 1580. A disadvantage of this solution is that it is only difficult to detect or even impossible to detect the lip movements of a third person whose body posture is not directed towards the service robot 17, if necessary. A second way of circumventing this problem comprises a tonal analysis of the multiple microphones 193 installed at different positions on the service robot 17 and recording through multiple channels in the time course and frequency course, step 1480, wherein the processor of the service robot 17 carries out a runtime analysis, step 1485, and from the time offset of the signal occurrence, it is calculated from which person the signals come, step 1490. As an alternative and / or complementary solution, it is also possible to use the microphones 193, wherein in this case a triangulation can be carried out by changing the position of the service robot 17. For this purpose, for example, the time offset calculated by triangulation, the time course, determines the origin in the room, which can be done two-dimensionally or three-dimensionally. This origin is then matched with the localization of the person in the room determined by the 2D or 3D camera 185 or the laser radar 1, whereby the service robot 17 can determine in this way which person has just spoken, step 1495. If a third person is involved to whom the voice signal belongs (and not the user), there can be a manipulation, step 1498. The value is then adjusted in the memory and in one aspect a request or error message is generated when the user is talking.

[0214] Now, other people can only provide input assistance for the user, i.e. not enter themselves, but only the speaker, the recorder, etc. enter into the service robot 17 via the display 2 or the microphone 193. To check whether this is possible, the recorded word sequence is analyzed, ensuring that it can be assigned to the respective person by at least one of the ways shown in the previous chapters, step 1505. Figure 15The basic points of this processing method are shown. As a complementary solution or alternative, it is possible to record the speech in the surroundings of the service robot 17, step 1510, and to distinguish the speakers, step 1515, according to different speech features / speech characteristics, in particular in the speech analysis module 132, which basically include the speech sequence, in particular the fundamental frequency, different speech intensities and / or different speech speeds. This method is combined with Figure 14 The method described uses the speech signal of the person according to the speech signal propagation through the lip tracking or localization, so that it is possible to correspond the recognized speech signal to the person, without having to determine the lip movement and / or the spatial position of the speaker anew each time and without having to compare it, if necessary, with the 2D / 3D person tracking result. After matching the person to the speech characteristics, step 1520, it is possible to record the speech and at the same time to track the user, step 1525. The sequences recorded accordingly are analyzed in terms of content and saved in the memory 10 of the service robot 17, i.e. the "predictive behavior" is searched, i.e. it is checked whether these text segments or speech segments / speech modalities occur in time one after the other several times, step 1530, and from different persons, step 1535, which is achieved by tagging the modalities and the speech characteristics corresponding to the person, such as the fundamental frequency (as an alternative and / or complementary solution, it is also possible to use the speech intensity and / or the speech speed). Figure 14The text segment, speech segment or speech modality refers, for example, to the same words and / or word sequence. Here, when evaluating these sequences to assist the user or the manipulation of the service robot 17, it is important to know which person first mentioned the relevant sequence. If it was first mentioned by the user, step 1565, then it is assumed that it was not a manipulation, but an assisting activity by a third person, step 1570. If the sequence first came from a third person, then it is assumed to be a manipulation, step 1575. To this end, it is checked in a first step whether a speech segment was first recorded by a person who is not the user and then this speech segment was repeated by the user. For this purpose, an association is made, in particular in the temporal range, in order to search for the same words. Here, in particular, it is checked whether more than one word that occurs individually in succession is repeated. As a supplement or alternative to the association analysis of the speech sequence, it is also possible to carry out a lexical analysis by means of natural language processing, step 1545, in which the words from the previous segments, for example, are analyzed and the sequences of tagged words are compared on the basis of the part-of-speech tagging, for example, by means of spaCy or NLTK in Python. This processing method can in addition also check whether the "predictions" were not reproduced acoustically by the user for recording by the service robot 17, but were directly input into the service robot 17 by the user. Since only these repeated speech segments / modality are of interest, the speech segments / modality are also detected by the service robot 17 in the scope of the test and analyzed in terms of content, for example, in the form of a table of questions for the user, step 1540. For this purpose, the text input ("arbitrary text") in the service robot 17 is compared to the menu-guided input (options) accordingly also by means of natural language processing (step 1550), alternatively by means of the saved speech signals which reflect the menu options, step 1555, are analyzed and the third-person speech recordings are compared to the user input content, step 1560. If it is found in the described sequences for manipulation recognition that the input or recording of the user was made by a third person or the "predictions" were made by a third person, then it is noted in the memory 10 of the service robot 17 that a manipulation has taken place, step 1580.

[0215] In summary, the method for determining the probability of a manipulation comprises detecting and tracking at least one person in the surroundings of the robot by means of the non-contact sensor, determining the position of the person in the surroundings of the robot, recording and analyzing the audio signal, locating the origin of the audio signal, comparing the determined position of the person and the position of the origin of the audio signal, and comparing the position difference with a threshold value, and determining the probability of a manipulation of the robot on the basis of the comparison of the position difference with the threshold value. Here, the origin of the audio signal is located by means of the detection of the direction of the audio signal, for example also by means of a transformation of the position of the service robot 17 or by means of a second microphone, by means of triangulation of the determined direction by means of at least one microphone. Locating the origin of the audio signal comprises triangulation of the direction of the audio signal detected by means of the microphone, the position of the at least one person determined by means of the non-contact sensor, and the detected direction of the audio signal. Further, for example, the face is analyzed, the lip movement in the time course is detected, the detected audio signal is compared with the detected lip movement in terms of time, for example by means of a correlation analysis, with a threshold value, and the detected audio signal is assigned to the detected person if the threshold value is exceeded. The method can likewise comprise logging in of the person on the robot (as a user) and detecting and saving the identity features of the person (as a user), wherein the identity features comprise the frequency, the intensity and / or the spectrum of the audio signal of the person, for example, further detecting and tracking the person, detecting the identity features of the person, comparing the detected identity features with the identity features of the person saved on the robot in the context of the logging in of the person, and comparing with a threshold value (i.e. with a minimum similarity), logging in the input content of the person on the operating element (186), and classifying whether the input on the operating element (186) was made by the logged-in person (user). For example, an increased probability of a manipulation of the robot is determined if the input on the operating element (186) of the robot was made by a person who is not logged in. The method can further comprise, for example, determining the words and / or the word sequence in the detected audio signal or audio sequence, assigning the determined words and / or the word sequence to the detected person, and determining an increased probability of a manipulation of the robot if the comparison of the determined word sequence results in a word and / or word sequence difference which exceeds a threshold value, i.e. does not reach a minimum correlation. Furthermore, the method can determine, for example, the words or the word sequence input by the person by means of the operating element (186), determine the words and / or the word sequence in the detected audio signal, assign the determined words and / or the word sequence in the detected audio signal to the detected person, and determine an increased probability of a manipulation of the robot if a minimum similarity of the words and / or the word sequence is found when comparing the word sequence input by means of the operating element (186) with the word sequence determined from the detected audio signal and at the same time a minimum similarity of the identity features of the detected person and the identity features detected and saved at the time of the logging in is found.

[0216] Figure 58An architectural view of a system for manipulating recognition based on audio signals is shown. This comprises a processing unit 9, a memory 10 and sensors for contactlessly detecting movements of a person detected in the surroundings of the system, at least one microphone 193, a person position determination module 4415 for determining the position of a person in space, an audio source position determination module 4420 for determining the spatial origin of an audio signal, a module 4025 for correlating two audio signals, an audio signal person module 4430 for associating an audio signal with a person and / or a speech analysis module 132. In addition, there is also an input registration comparison module 4785 for comparing whether the person whose identity is recognized by the system is making an input on the system. The system additionally also has an audio sequence input module 4435 for comparing an audio sequence, i.e. a sequence of sounds such as the repetition of a word, with a sequence of letters of a haptic input. In addition, there are also output means such as a loudspeaker 192 and / or a display 2 for transmitting instructions. A connection can be established with a terminal via an interface 188, such as a WLAN. The sensors for contactlessly detecting movements of a person refer to 2D and / or 3D cameras 185, laser radars 1, radars and / or ultrasonic sensors 194. The system has, in one aspect, a person recognition module 110, a person identity recognition module 111, a tracking module (112, 113), a motion analysis module 120, a skeleton creation module 5635 and / or a feature extraction module based on a skeleton model 5640.

[0217] Checking the user for impairments

[0218] The user, in particular the elderly, can suffer from hearing and vision impairments in addition to cognitive impairments that can exist and should be checked by means of the processing described in this patent application, which can falsify the test results. In order to improve the accuracy of the test results, the service robot 17 is, in one aspect, configured such that the service robot 17 can perform a short hearing test on the user before starting the exercise, as a complementary solution or as an alternative, a short vision test on the user. Basically, in Figure 16 The method steps carried out here are shown in Fig. 16. The service robot 17 can first optionally request this problem situation in which possible comprehension problems can occur from the user by means of a screen output and / or a sound output, so that the service robot 17 needs to be calibrated depending on the user. The vision and / or hearing test belong to this kind of calibration. Then, the service robot 17 prompts the user to participate in the calibration, step 1605.

[0219] In the context of a short-term hearing test, the service robot 17 prompts the user to press a corresponding key in a menu of the display 2 when the user hears a specific sound. As a complementary or alternative solution, the user can also be provided with a voice input as described in the prior art, which can again be analyzed in the voice analysis module 132 by natural language processing methods. Subsequently, the service robot 17 plays a sequence of different frequencies and different volumes, but which individually have essentially constant frequency and volume, step 1610, and each time "asks" the user whether he heard the sound, which can be achieved, for example, by the service robot 17 showing the user the display 2 with input options by means of which the user can state to what extent he heard the sound, step 1615. In one aspect, here the sound is lower in volume and higher in frequency over the course of time, step 1620. But different sequences in connection therewith can also be considered. The user's answer is detected. Subsequently, a score is determined, step 1625, which states in which range of the user the sound can be heard. If the user, for example, does not reach a certain threshold within the hearing behavior, i.e. the answer of the service robot 17 guided by its screen menu or voice menu and analyzed accordingly does not reach a predefined boundary value of a positive answer, for example, only three of seven sounds are recognized, a corresponding score value can be determined therefrom. In one aspect, this score is saved in a database in the service robot 17, step 1630, for example, together with the user information characterizing the clinical state of the person. As a complementary or alternative solution, the service robot 17 can also determine whether the user needs an increased volume of the signal output by the service robot 17 by the volume of the answer of the user, for example, relative to the ambient noise level 1635 recorded by at least one further microphone 193. In a complementary and / or alternative aspect, the output volume of the sound signal of the service robot 17 is adjusted accordingly, for example, when it is determined by at least one of the ways described that the user is hard of hearing, the volume is increased.

[0220] In the context of a short-term vision test, when the user is able to recognize certain letters or other types of symbols, the service robot 17 prompts the user to press the corresponding key in a menu on the display 2, step 1650. As a complementary or alternative solution, the user can also be provided with a voice input, as described in the prior art, which can also be analyzed by means of a natural language processing method. Subsequently, the service robot 17 plays a series of characters or pictures on the display 2, step 1655. Here, the user signals in step 1660 whether the user has recognized the characters or which characters the user has recognized. Here, in one aspect, the symbols or pictures are made smaller in the course of time (step 1665). However, different sequences in connection therewith can also be considered. As a complement and / or complementation thereto, different color patterns can also be used to identify possible color blindness of the user. The user's answers are detected. The test result is displayed in the form of a score, step 1670. If, for example, the user has not reached a certain vision threshold, or color blindness is found, i.e. a certain number of objects / patterns (such as three out of seven) cannot be recognized, which affects the score, this can be saved in a database in the service robot 17 in one aspect, step 1675. In a complementary and / or alternative aspect, the size of the letters is adjusted accordingly when outputting text elements on the display 2 of the service robot 17, if necessary also the menu design, so that the corresponding menu items can be displayed with larger letters, step 1680. Furthermore, in a complementary aspect, the color of the display 2 can also be adjusted in order to better recognize the display menu in the case of color blindness. In a complementary and / or alternative aspect, the service robot 17 can change the distance to the user, for example closer to the user if the user has a visual impairment, step 1695. For this purpose, a parameter value defining the regular distance between the user and the service robot 17 will be temporarily modified in the navigation module 101, step 1690. Finally, it is also possible to adjust the contrast and / or brightness of the display 2 of the service robot 17 in terms of environmental conditions, taking into account the user's vision, step 1685.

[0221] Improving signal processing quality by adapting to environmental influencing factors

[0222] On the other hand, taking into account not only vision and hearing tests but also user deficiencies, the service robot 17 can adjust its input and output units according to the environment, thereby enabling operation under varying brightness and / or noise background conditions. The service robot 17 has a commercially available brightness sensor near the display to determine how much light falls on the display 2. Simultaneously, it adjusts the brightness of the display 2 according to the environment, particularly increasing the brightness of the display 2 when there is strong incident light and decreasing the brightness of the display 2 when the brightness value is low. As a supplementary or alternative, the service robot 17 can measure background noise using one or more microphones 193. In one aspect, this may result in the service robot 17's sound output volume being similarly higher when the noise level is high and lower when the noise level is low. In a supplementary or alternative aspect, at least another microphone 193 records the background noise and uses a noise cancellation method (a phase shift of the input signal around the recorded background noise) to improve the signal quality of the sound input signal, thereby enabling better speech processing and thus avoiding issues such as data detection errors, requiring the service robot 17 to repeat questions or prompts.

[0223] Furthermore, as a measure to improve the accuracy of measurement results, service robot 17 will also inquire whether the person being assessed is in pain, including the intensity of the pain. For this purpose, interaction between service robot 17 and the person being assessed is achieved via a communication device already described elsewhere. This information is stored in the user's database records.

[0224] As another measure to improve the accuracy of measurement results, the service robot 17 obtains information from the patient management module 160 about when the patient entered the hospital, the test was conducted in the hospital, and calculates the current length of stay to account for cognitive decline caused by prolonged hospitalization. Simultaneously, the patient management module 160 also checks whether the patient has been diagnosed with a disease. This information is also considered when displaying the results of the simplified psychological test and is stored in the user's database record.

[0225] The service robot 17 transmits the saved results of the aforementioned test tasks to the patient management module 160 via an interface 188 (such as WLAN), making them available to medical personnel, while also recording the results.

[0226] Spectral measurements of patients

[0227] In one respect, the service robot 17 is configured to determine whether a patient has specific secretions on their skin, which in turn indicate a specific disease, enabling diagnosis. Thus, the service robot 17 can, for example, determine whether a patient is sweating in bed, and if necessary, determine the severity of the sweating. Here, for example, a spectrometer 196, such as a near-infrared spectrometer, can be used; alternatively, a Raman spectrometer can also be used. Figure 21a The procedure for measuring secretions 2100 is shown in the diagram. Here, measurements can be taken separately for different body parts. The treatment of three parts is illustrated exemplarily: measurements on the hands, forehead, and torso, particularly on a bed sheet. Detecting sweat from these areas is, for example, part of a mental disorder detection score in another test to demonstrate the presence of mental disorder in the patient.

[0228] The service robot 17 is configured such that it can record the patient in the bed via a 3D sensor, such as a 3D camera. For this purpose, the sensor is positioned on the service robot 17, allowing it to be positioned at a height of at least 80 cm, for example, at least 1.2 m, and in a manner that allows it to rotate and / or tilt.

[0229] The service robot 17 can identify the bed based on object recognition, step 2105. To this end, in one aspect, the service robot 17 can scan the space by means of a 2D or 3D sensor, such as a laser radar 1, from which it can infer that there is a bed in the space. As an alternative and / or in addition, the dimensions can also be determined by means of a map saved in the memory 10, which map comprises room information such as the width and depth of the space, for example. In this way the method analyzes the dimensions of the space, step 2110. The service robot 17 can additionally also determine the dimensions of the measured object, step 2115, such as a triangulation (step 2120) in combination with an implemented odometry unit 181, which can determine the positional deviation of the service robot 17. The dimensions of the measured object in relation to the room information in the space are determined, step 2122, for which it is not necessary to resort to the odometry function. The determined dimensions, in particular the outer dimensions of the bed, are classified based on rules saved in the memory 10, whereby it is determined whether the object belongs to a bed, step 2124. In one aspect, this comprises the dimensions occupied by the bed. In a supplementary and / or alternative aspect, the object recognized by the laser radar 1 and / or the 2D and / or 3D camera 185 can also be classified based on the characteristic features of a bed being explicitly recognized, step 2125. Here, the design of the wheels of the bed and / or the lifting device for adjusting the height of the bed can be involved. But it is also possible to use classification rules created by learning typical bed features based on machine learning and / or neural network methods. As an alternative and / or in addition, the bed can also be equipped with a sensing device and / or a barcode, step 2130, such as an RFID or Bluetooth transmitter, which can identify the bed.

[0230] In one aspect, the orientation of the bed in space can be determined by positioning sensors on the bed, step 2140, such as by using backscattered signals that are reflected differently on the bed frame and by running time and / or phase differences to determine the orientation of the bed in space. The barcode can likewise be fixed on the service robot 17 in such a way that this reading allows the spatial orientation of the bed to be determined. The code saved in the sensor and / or barcode is read by the service robot 17 and compared with the code saved in the memory 10 of the service robot 17 and assigned to the bed, whereby the service robot 17 can determine that the read sensor and / or barcode has been assigned to a bed. As an alternative and / or in addition, especially when the bed is identified on the basis of its dimensions (step 2124), the orientation of the bed in space is determined by the bed dimensions, which in one aspect can also be identified by comparing the position to the nearest wall, step 2135: That is, the service robot 17 determines the orientation of the bed, especially the head of the bed, on the basis of prior knowledge that the bed has a substantially rectangular shape with the shorter sides being the head or the foot of the bed. Here, the shorter side is identified as the head of the bed, which is closer to the wall of the room, for example. In one alternative aspect, the service robot 17 can identify a person on the bed, especially the head and the arms, which can be analyzed, for example, in the context of a skeletal model.

[0231] Subsequently, in an optional step 2145, the service robot 17 determines in which position the service robot 17 is relatively closer to the head of the patient. To this end, the service robot 17 determines in the next step on which side of the bed the service robot 17 can walk as far as to the head of the bed. If the distance to the wall at the head of the bed on one side of the bed is less than 1 m, the service robot 17 walks along this side of the bed. If the distance exceeds 1 m, the service robot 17 determines the distance to the wall on the other side of the bed and then walks as far as possible to the wall near the head of the bed, i.e. on the side where the service robot 17 can walk as far as possible to the wall near the head of the bed. In one alternative aspect, the service robot 17 first checks the depth of both sides as described before and then walks along the side that is oriented toward the head of the bed, on which side the service robot 17 travels the farthest toward the wall near the head of the bed.

[0232] As a next step, the service robot 17 determines a candidate region for the head, step 2150. To this end, the service robot 17 is positioned such that it is instructed to face in the direction of the predicted head position. This can be achieved, for example, by rotating the service robot 17 into position, wherein the service robot 17 has a rotation angle of 25° to 90° measured with respect to the longer side of the bed. The service robot 17 detects the surface of the bed, in particular in the region to the head of the bed, by means of 2D or 3D sensors. As an alternative and / or in addition, the service robot 17 calculates a candidate region in which the head is usually located and which is located at least 25 cm from the longer side of the bed, at least 10 cm from the head of the bed, and at most 60 cm from the head of the bed.

[0233] As an alternative and / or in addition, it is also possible to save the width interval. If the distance to the longer side of the bed (determined in relation by comparing the bed sides and / or by a predefined length interval) is less than a defined threshold value, for example 20 cm, it is assumed that the bed extends along a wall, and the service robot 17 moves along the longitudinal side of the bed until there is sufficient space. Subsequently, the service robot 17 uses the methods already described for determining a candidate region for the head, the service robot 17 instead scans the entire bed by means of the camera 185, and the pictures thereof are analyzed by means of a commercially available architecture which enables head recognition.

[0234] The service robot 17 can determine the forehead on the basis of the head features, step 2152. In one aspect, this is achieved by defining a region which is limited by the following facial features: about 4 cm above the line connecting the eye center points on both sides of the hairline, which can be recognized by the color contrast of the patient's skin. As an alternative and / or in addition, the head shape can also be used here, wherein the forehead plane is limited by rounding off the head. Here, for example, the histogram of oriented gradients scheme can be used, which employs, for example, the OpenCV or Scikit-image architecture. Here, as a limit, it is also possible to use the angle, which is formed by the beam of the head sensor and the perpendicular at the point where the beam hits the surface of the head. After recognizing the patient's forehead, the service robot 17 tracks the position of the head, step 2172, for example by means of the visually based human tracking module 112 and / or the laser-based human tracking module 113.

[0235] If the service robot 17 encounters problems in recognizing the patient's forehead or eyes, in one aspect, the service robot can swap the bed side to ensure that the patient's back of the head does not turn towards it. As an alternative and / or in addition, the service robot 17 can request the patient to move the head, step 2154, for example to look towards the service robot, by means of its output unit, for example the display 2 and / or the speech synthesis unit 133. After such a request has been issued, the recognition of the head or the forehead is reattempted.

[0236] The service robot 17 is also able to use other classification algorithms to recognize the patient's hands. Here, in one aspect the same method as for recognizing the patient's head is used (i.e. two candidate regions for the hands are determined approximately in the center of the longer side of the bed, at a depth of about 30 cm parallel to the shorter side of the bed, step 2157), as an alternative and / or in addition, an algorithm from the SDK of the RGB or RGB-D camera 185 can be used to create a (partially assigned) skeleton model of the patient, in which here in particular the arms and the hands, i.e. their joints, are recognized, while the connections between the joints can be displayed as directional vectors. If the service robot 17 does not recognize the arms or the hands, the service robot 17 can request the patient to move the arms or the hands, step 2159, e.g. to stretch them out from under the bed sheet, by means of its output unit, such as the display 2 and / or the speech synthesis unit 133. After such a prompt has been issued, the recognition of the arms or the hands is reattempted. Similarly to the forehead, the service robot 17 can also recognize the hand surface, such as the back of the hand and / or the palm. Here, in order to improve the positioning, as an alternative and / or in addition, the joints from the skeleton model can also be taken into account, in which the focus region of the hand is between the hand joint and the finger joints. As an alternative and / or in addition, the palm can be recognized by picture classification, in which the classification algorithm is trained with the aid of images showing the palm.

[0237] Another focus body region is the patient's upper body, which is defined beforehand by a candidate region pointing from the head downwards, which extends over a length of about 1.5 times the head height, half the head height starting from below the head, and has a width of about 2 times the head width. As an alternative and / or in addition, this region is defined starting from about 10 cm below the patient's head, over a width of about 45 cm and a height of about 50 cm, or alternatively approximately centered on the bed, at a distance of about 50 cm from the head of the bed. As an alternative and / or in addition, the classification can also be carried out in three dimensions. Here, in one aspect the width of the bed is scanned as a function of the height, and in the region in which the axis is parallel to the longer side of the region in the middle of the bed, the direction is towards the head of the bed. The portion located below the candidate region of the head is selected along the ridge line generated in this region. In this way the candidate region of the upper body can be determined, step 2160. Then, the protrusions relative to the level of the mattress detected by the 3D sensor means of the service robot 17 are scanned, and in the case of a protrusion detected in the candidate region, this region is recognized as the upper body, step 2162.

[0238] In order to enable the service robot 17 to detect the target areas identifying the patient: the forehead, the hand surface / hand back and the upper part of the torso. These can be so identified in space by means of a sensor device, such as by means of the RGB-D camera 185, whereby their surfaces can be displayed in a three-dimensional coordinate system accordingly. Here, for example, a tracking is also carried out, step 2170, in particular a tracking of the head of the patient, step 2172, and in one variant also a tracking of the hands, step 2174, and optionally a tracking of the upper body, for example by means of the visually human tracking module 112 and / or the laser-based human tracking module 113. Here, for example, the images created by the sensor device are segmented in order to determine the body areas by classification, enabling the spectrometer (196) to be aligned to these body areas. The respective classification can be saved in the memory 10. The service robot 17 can also save the areas on which measurements should be carried out in an application for controlling the spectrometer, for example.

[0239] Before the measurement, the service robot 17 tracks the movement of the hands or the head, optionally also the movement of the upper body, in a defined time period, step 2170. If no movement is found for a duration which exceeds a defined threshold value, for example 5 seconds, or only movements of the hands / head are found which do not exceed a defined threshold value, for example 5 mm, step 2180, the detected data is measured and analysed, step 2185.

[0240] During the measurement using the spectrometer, the head of the patient or the hand on which the measurement is being carried out is continuously tracked by means of the RGB-D camera 185 in the scope of the safety check 2178. If a movement is found, such as a rotational movement of the head, a lowering or raising movement of the head which exceeds a defined threshold value, the measurement is immediately interrupted. The service robot 17 continues to track the area on which the measurement should be carried out and, when the movement of the head is less than the defined threshold value, a renewed measurement attempt is started again.

[0241] The service robot 17 additionally has, for example, a near-infrared spectrometer 2186 for carrying out a material analysis, which can be arranged rotatably and pivotably and can be electronically adjusted in relation thereto. The service robot 17 can so align the spectrometer 196 by means of this arrangement that the path of the radiation determined by the spectrometer 196 reaches the coordinates of the target area in the three-dimensional space and also that the reflected radiation of the spectrometer 196 can be detected again, step 2187. Although an infrared diode with a focusing lens is used as the light source, in one aspect an infrared laser can be used.

[0242] The measurement will be carried out, i.e. the signal of the spectrometer 196 will be analyzed and classified 2189 according to a parameter database, which contains reference spectra and from which it is possible to determine what 2188 is in or on the target area in terms of quality or quantity 2190. As an alternative and / or in addition, it is also possible to save directly the classification rules for determining the substance from the measured spectrum, which work, for example, on the basis of a correlation assessment. Here, in one aspect, characteristic signals are determined, i.e. essentially the spectral profile of the sweat is determined, step 2191, which consists of the individual spectra of water, sodium and / or chlorine and which, for example, occur on the skin of the patient, for example on the forehead or the hand. From the detection of the patient's bed linen in the torso target area, it is determined how wet the bed linen is, i.e. here the classification used for analyzing the signal is considered in terms of the material of the bed cover.

[0243] By scanning different partial areas of the bed linen, the amount of water that has come out as sweat is also estimated, for example, by classification by means of a reference database.

[0244] The database has, in another aspect, reference spectra by means of which the concentration of other substances can be determined, including different drugs 2192, such as 9-tetrahydrocannabinol (THC), or other substances 2193, such as glucose, lactic acid, uric acid, urea, creatinine, cortisol, etc.

[0245] The service robot 17 has a further reference database which allows classification of the determined measurement values on the basis of combinations of different substances and / or their concentrations, from which it is possible to associate different illnesses with the measured spectra, step 2194. Here, concentration thresholds or measured substance amounts, ratios of substance amounts and / or concentrations to one another and their combinations form an integral part of this classification. One example is the combination of urea, uric acid and creatinine, in which the concentration of uric acid is greater than 0.02 mmol / l, the concentration of creatinine is 0.04 mmol / l (at lower temperatures the concentration will be higher) and the urea concentration is > 15 mmol / l (at low temperatures) or > 100 mmol / l (at high temperatures). The service robot 17 takes the ambient temperature into account in this classification by means of a thermometer located in the service robot 17, the season or the outdoor temperature and, in the latter case, is equipped with an interface 188, such as WLAN, from which the outdoor temperature at the location where it is located can be determined by means of the cloud 18, i.e. the service robot 17 can detect further data in order to improve the analysis or transmit them to other databases 2196 by means of other sensors 2195 and / or by means of the interface 188, such as WLAN.

[0246] The measurement results are saved in a database located in the service robot 17, step 2197 and / or can be transmitted to a server in the cloud 18 via an interface 188, such as a WLAN, and saved there, step 2198. Subsequently, the measurement results can be output via the display 2 and / or speech output, step 2199, for example by a terminal that the service robot 17 and / or the medical staff can access at the time of analysis.

[0247] In Figure 63 The spectrometer system (of the service robot 17, for example) is shown in overview in Fig. 48: it comprises a processing unit 9, a memory 10 and sensors for contactless detection of a person (such as a 2D and / or 3D camera 185, a laser radar 1, a radar and / or an ultrasonic sensor 194), a spectrometer 196 and a spectrometer alignment unit 4805 for aligning the spectrometer 196 with a body region of a person, which is similar to the flip unit. In addition, the system can also have a thermometer 4850 for measuring the ambient temperature and / or an interface 188 to a terminal 13. In the memory 10 there are a body region detection module 4810 for detecting a body region, a body region tracking module 4815 for tracking a body region before and / or during a spectral measurement on the body region, a spectrometer measurement module 4820 for monitoring (including interrupting and / or continuing a spectral measurement based on the movement of the body region on which the measurement is being made), a visual person tracking module 112 and / or a laser-based person tracking module 113. The system has access to a reference spectrum database 4825 and / or has a saved disease image database 4830 with disease images associated therewith for comparing the measured spectrum and determining the measured substance, which is located in the cloud 18 and / or in the memory 10. In addition, there is a sweat module 4835 for determining the sweat flowing off, a delirium detection score determination module 4840 for determining a delirium detection score and / or a cognitive ability assessment module 4845 for determining the cognitive ability in the memory 10 or the cloud 18. In one aspect, the system has a person recognition module 110, a person identity recognition module 111, a tracking module (112, 113) and / or a motion analysis module 120.

[0248] Delirium recognition and monitoring on the basis of several tests

[0249] As an alternative to simplified psychological testing, a testing method for identifying mental confusion, now mastered by medical professionals, has emerged in clinical diagnosis. Mental confusion is at least a temporary disorder. It specifically employs the term CAM-ICU, where CAM stands for "Confusion Assessment Method" and ICU for "Intense Care Unit." The assessment conducted by medical professionals includes attention deficit, which can be assessed through auditory and / or visual tests, but also through tests of disordered thought requiring motor responses.

[0250] Analysis of the attention disorder of a patient based on recognition of a sequence of acoustic signals

[0251] Service robot 17 is configured in one respect (see Figure 22 Thus, the service robot 17 can output pulse sequences (e.g., tone sequences) of different sound signals through a speaker 192 with a pulse frequency of, for example, 0.3-3 Hz, or about one hertz, step 2205. Simultaneously, the service robot 17 can detect signals from at least one tactile sensor 4905 (step 2210) and synchronize with the output signals, step 2220. Furthermore, a value can be assigned to each sound signal in the memory 10. There is a certain time delay 2215 between the output sound signal and the signal detected by the tactile sensor 4905, i.e., a maximum of half the pulse length, tracking the phase shift of the pulse signal. Here, the signal from at least one tactile sensor 4905, registered with possible phase shifts, is analyzed to determine whether it appears within a defined spectral range stored in the memory 10, step 2225, i.e., comparing whether a detected signal appears after the defined tone sequence. If this is the case, a counter in the memory 10 is incremented by an integer value, step 2230, or alternatively, not incremented, step 2235. Subsequently, the measured counter values ​​are categorized in this manner, such that a diagnosis corresponding to the counter value is assigned to each measured counter value, step 2240. A sound is output to the patient to, for example, check their cognitive abilities. The larger the value, the lower the adverse impact on the patient's cognitive abilities. The diagnostic results are stored in the memory 10 of the service robot 17, step 2245, and optionally, the diagnostic results are transmitted to a cloud-based cloud storage within the cloud 18, and optionally provided to medical personnel via a terminal.

[0252] The tactile sensor 4905 is a piezoelectric, piezoresistive, capacitive or resistive sensor. However, as described by Kappassov et al. in 2015 (DOI: 10.1016 / j.robot.2015.07.015), other types of sensors can also be used. In one aspect, there is a tactile sensor 4905 on an effector 4920 of the service robot 17, which has at least one joint and can be positioned such that the service robot 17 can reach the patient's hand, i.e. the tactile sensor 4905 is positioned at a distance to the hand that is less than a threshold value, which is stored in the memory, for example. In one aspect, the sensor is integrated in the robot arm. In an alternative or supplementary aspect, the sensor is mounted on the surface of the service robot 17. To this end, the service robot 17 uses at least one camera 185 to identify the patient's identity, track him and determine the position of his hands, for example the position of the right hand.

[0253] Application example:

[0254] The service robot 17 outputs a string of letters corresponding to a word via the loudspeaker 192 in a test for detecting the patient's attention. Each letter is output at an interval of about one second. The service robot 17 requests the patient to perform a pressing movement with one of his hands when a specific letter is recognized. These pressing movements are analyzed by the tactile sensor 4905 described and the frequency of recognizing a specific letter is counted. The higher the recognition rate, the lower the adverse effect on the patient.

[0255] As shown in Figure 64 , the attention analysis system consists of a system, such as the service robot 17, which comprises a processing unit 9, a memory 10, an output unit for sound signals, such as the loudspeaker 192, a tactile sensor 4905 and a tactile sensor analysis unit 4910 for analyzing signals by the tactile sensor and a tactile sensor output comparison module 4915 for comparing whether the detected signal occurs after a defined output. The system can likewise comprise an effector 4920, such as a robot arm, and a camera 185. The tactile sensor 4905 is positioned on the effector 4920, for example. In the memory 10 there is an effector positioning unit 4925, which positions the tactile sensor 4905 in the vicinity of a person's hand by the effector 4925 and determines the cognitive ability of the person by the person identity recognition module 111 and / or the hand recognition module 4930 and the cognitive ability assessment module 4845. The system has a person recognition module 110, a tracking module (112, 113), a movement analysis module 120, a skeleton creation module 5635 and / or a feature extraction module based on a skeleton model 5640 in one aspect.

[0256] Analysis of the cognitive capacity of a patient based on image recognition

[0257] In an alternative or complementary aspect of the service robot 17, the service robot 17 is configured such that the recognition of the image can be analyzed and classified by the patient in order to thereby assess the cognitive abilities of the patient, in particular its attention. Figure 23The procedure used herein is exemplarily illustrated. Here, the service robot 17 indicates to the patient by means of the speech synthesis unit 133 that the service robot 17 should memorize a plurality of pictures, step 2305. After the speech output, a series of pictures will be displayed on the display of the service robot 17, step 2310, for example five pictures each for three seconds. Then, the patient is informed by means of the speech synthesis unit 133 that the service robot 17 wants to signal whether the patient recognizes the displayed pictures by means of a movement of the patient’s head, i.e. the service robot 17 wants to make a classification related thereto in step 2315. Here, a shaking of the head is evaluated as a rejection and a nodding as a confirmation. Ten pictures are played on the screen of the service robot 17 each for three seconds (step 2320). Here, five pictures are repeated compared to the first series of five pictures, but each picture is played only once. In one aspect, the order of the pictures can be determined by means of a random generator and / or the new pictures can be distinguished in comparison to the displayed pictures, step 2325. The service robot 17 saves whether the sequence of displayed pictures has already been displayed, step 2330, and detects the head movement of the patient during the display (or at most for one second). For this purpose, the service robot 17 has at least one sensor, for example the RGB-D camera 185, which can recognize and track the head of the patient, step 2335, wherein the analysis takes place, for example, by means of the visually human tracking module 112 and / or the laser-based human tracking module 113. This includes a turning of the head and / or a nodding. Here, the service robot 17 can detect the feature points of the face by means of a classification method, wherein the eyes, eye sockets, mouth and / or nose are included. Solutions in this regard are known from the prior art (for example DOI: 10.1007 / 978-3-642-39402-7_16; 10.1007 / s11263-017-0988-8), which use histograms of oriented gradients. As a next step, the head movement of the patient is classified in order to identify a shaking and / or a nodding, step 2340. For this purpose, architectures from the prior art can likewise be used. The shaking or nodding movement thus identified is synchronized with the displayed pictures accordingly, step 2345. The sequence of displayed pictures is then encoded depending on whether the patient has correctly recognized the pictures after they have been displayed to him or her again or for the first time, step 2350. The service robot 17 optionally saves the result of the comparison of the values, for example together with the date of execution, to a database, which likewise saves the displayed picture sequences, for example. A counter is incremented each time a picture is correctly recognized by the patient, step 2355. The score resulting from the incrementation is a criterion for measuring whether the patient is suffering from a cognitive impairment. For this purpose, the determined score is classified and assigned a medical annotation, step 2360. The score and its medical annotation are saved in a database, step 2365, if necessary in a cloud storage within the cloud 18, step 2370, and made available to medical staff by means of a terminal for analysis, step 2375.

[0258] In an alternative aspect, the service robot 17 is able to determine the position of the patient's eyes 2410 and the position of the display 2 in three-dimensional space (step 2405). The service robot 17 uses this data in one aspect to check whether there is an obstacle between the line of sight of the eyes and the display 2. For example, when the patient is in bed, a fall prevention device can be such an obstacle. To this end, the service robot 17 first calculates the coordinates of the line of sight, step 2415, and checks by means of the 3D camera whether the coordinates of the line of sight are associated with a detected obstacle, step 2420. If the service robot 17 recognizes an obstacle, the display inclination is repositioned, step 2450, and as an alternative and / or in addition, the service robot 17 is repositioned in the XY plane, step 2455. In an alternative and / or additional aspect, the service robot 17 is configured such that the service robot 17 is able to determine the angle between the patient's eyes and the display surface with the aid of the spatial coordinates of the display 2 (step 2425), for example the display corners, in order to ensure that at least one angle is within a certain interval, step 2430, which can be device-specific. In this way it can be ensured, for example, that a reflective surface of the display 2 does not result in the patient not being able to recognize the display 2 sufficiently well because the display inclination would cause the patient to recognize a strong reflection. To this end, the service robot 17 is able to adjust the display inclination 2450 and / or to reposition the service robot 17 in space accordingly. As an alternative and / or in addition, the font size and / or other symbols on the display 2 can also be adjusted in accordance with the distance between the patient and the display 2. To this end, the service robot 17 first calculates the Euclidean distance between the eyes and the display 2, compares this with a reference value stored in the memory 10 of the service robot 17, checks whether this distance is acceptable for recognition, and in an additional aspect can incorporate data on the patient's eyesight in order to adjust the reference value if necessary. In this way, the service robot 17 is able to adjust the display size of the display 2 (i.e. the size of the objects and symbols displayed) and / or the positioning of the service robot 17 in space in the XZ plane (i.e. the floor plane) in order to ensure that there is sufficient distance for recognizing the content of the display.

[0259] Based on the repositioning of the service robot 17 in the XZ plane, the tilt of the display 2, and / or the adjustment of the display size, the service robot 17 is able to determine the tilt probability of the display 2 and / or the display size of the display 2 by scanning its surrounding environment and possible extended or alternative unobstructed line of sight. This may determine its position, tilt, and / or display size in the XZ plane, so that the patient does not perceive any obstruction between their eyes and the display 2, and the display 2 is positioned in such a spatial manner that it is as non-reflective as possible and / or the display size is sufficient for the patient's vision.

[0260] In an alternative and / or supplementary aspect, the service robot 17 has a control device for the tilt of the display and a dialogue function in the display 2, or is configured as a voice interface. With this dialogue function, the patient can be given feedback on the extent to which the display is sufficiently identifiable. If the patient is dissatisfied with the effect, the service robot 17 can change the alignment of the display 2. This can be achieved in one aspect by repositioning the service robot 17 relative to the patient. This can be achieved by rotating into place, or by occupying other positions (e.g., defined by the area of ​​the ground covered by the service robot 17). In an alternative and / or supplementary aspect, the tilt of the display 2 can be adjusted, wherein the tilt axis can be aligned horizontally and / or vertically.

[0261] After repositioning the display 2 and / or service robot 17 in the XZ plane, the described process is repeated to check whether the patient is able to properly recognize the display 2.

[0262] The robot counts the number of fingers of a hand

[0263] The service robot 17 is configured in one respect to be able to recognize or track fingers via a camera 185, such as an RGB-D camera 185, in order to analyze the gestures displayed by the hands based on the displayed numbers, for example with the aid of a visual human tracking module 112 and / or a laser-based human tracking module 113. Figure 25 This process is illustrated. For this purpose, the depth image 2505 generated by the 3D depth camera is converted into a 3D scatter plot, in which a spatial coordinate is assigned to each pixel of the camera, step 2510, thereby enabling the skeleton to be identified via a third-party provider, such as NUITrack's camera SDK or software, step 2515. Here, joints, including hand joints and finger joints, are identified accordingly.

[0264] As a next step joint selection 2520 is performed, i.e. only the necessary joint nodes are continued for the following calculations to be performed. Subsequently, angle calculation 2525 is performed, e.g. the angles between the third and second phalange, the second and first phalange and the first phalange and the metacarpus are calculated. (Herein generally the third phalange is referred to as the phalange with the finger tip). Since the thumb generally does not have a second phalange, it is here referred to the angles between the third and first phalange, the first phalange and the metacarpus and in one aspect the angle between the metacarpus and the carpus. Herein each phalange or each metacarpus is shown as a directional vector starting from the observed joint node. As a next step feature extraction 2530 is performed, wherein e.g. the angles of the joint nodes of each finger are jointly analyzed. In the scope of feature classification 2535, which is implemented based on determined rules, e.g. an extended index finger is defined as an angle of 180° between the first and second phalange and the second and third phalange. In the scope of feature classification threshold values can be defined, which can deviate slightly from the condition of 180° and e.g. for the angle between the third and second phalange 150° to 180° and for the angle between the first and second phalange 120° to 180° and for the angle between the metacarpus and the first phalange 90° to 180°. For the thumb the angle between the third and first phalange is 120° to 180° and the angle between the first phalange and the metacarpus is 150° to 180°. In the scope of gesture classification 2540 the respective fingers and their joint angles are jointly observed. Thereby in one aspect a value of 2 displayed by the fingers can be detected based on manually defined values 2545 by extending the thumb and the index finger, the index finger and the middle finger or a combination of two of them and / or other fingers, while the other fingers, especially the angle between the second and third phalange, has an angle of less than 120°, e.g. less than 90°. If the thumb is not extended but two other fingers are extended, then here the angle between the third and first phalange of the thumb is less than 120°, e.g. less than 100°, e.g. finally recognized as 2. The angle between the metacarpus is optionally less than 145°, e.g. less than 120°.

[0265] In one aspect the feature extraction, the feature classification and the gesture classification can be trained by predefined rules, e.g. angles of the respective joint nodes and combinations thereof, or also by a machine learning approach 2550, such as a support vector model, in which specific angle combinations are correspondingly labeled, i.e. combinations of angles of the respective phalanges can be explained to each other, thereby e.g. showing two fingers, which corresponds to a value of 2.

[0266] Within the scope of the patient's cognitive ability test, in one aspect first the output of the service robot 17 is triggered, by means of the speech synthesis unit 133, by means of the loudspeaker 192 and / or by means of text and / or pictures to the screen output of the service robot 17. This speech output requests the patient to show two fingers 2605. Here, the camera 185 recognizes the patient's hands, the fingers thereof and tracks the finger movements. Here, the service robot 17 analyzes this within the scope of gesture classification in order to determine how many fingers are shown 2610. Here, in one alternative aspect, it is considered whether the finger gesture shown by the patient can be associated with a code 2665, as described below. Subsequently, the service robot 17 saves a value which indicates whether the patient has shown two fingers 2615, i.e. the result of the comparison of the evaluated finger gesture with the numerically output value is evaluated.

[0267] The service robot 17 has in one alternative and / or complementary aspect at least one effector 4920, such as a robot arm with at least one joint, which additionally has at least one robot hand 2620 with at least two fingers which can imitate human fingers, but at least five fingers, wherein one corresponds to the thumb in terms of its arrangement and they have as many knuckles as a human hand. Here, the service robot 17 is able to display numbers by means of these fingers, wherein the extended fingers and finger gestures are derived from the angle of the knuckles, which have been classified before according to the use of the camera 185 for recognizing the knuckles. Thereby, the service robot 17 is also able to display the value 2, step 2670, by means of extending, for example, the thumb and the index finger on the robot hand, i.e. the angle between the first three knuckles is, for example, close to 180°, while the angle of the other knuckles and between the knuckles and the metacarpal bone is less than 120°. The service robot 17 is configured in such a way that the service robot 17 is able to synchronize the gesture control of the robot hand by means of the speech output and / or the display 2 in such a way that the robot hand displays the value 2, while the patient is requested to show as many fingers as the robot hand displays by means of the display 2 and / or the speech synthesis unit 133, step 2675. Then the hand recognition, hand tracking and gesture classification are performed in order to recognize two fingers of the patient, step 2610, in order to save a value when it is determined that the patient has shown the number two, step 2615. Here, in one aspect after the service robot 17 has requested the patient to show the numerical value or has displayed the corresponding number by means of the robot hand by means of its output units, such as the loudspeaker 192 and / or the display 2, the gesture shown by the patient is analyzed within a time window of, for example, 3 seconds.

[0268] The knowledge gained within the scope of this test allows to assess to which extent the patient is affected by disorganized thinking and thus gives a test method for the identification and monitoring of mental disorders.

[0269] In an alternative and / or complementary aspect, the service robot 17 is configured to be able to display numbers with fingers based on cultural and / or national differences. As an alternative and / or complementary aspect, the service robot 17 can also facilitate the recognition of the displayed number when analyzing the hand gestures by taking these differences into account. As a result, for example, a patient from Germany is more likely to display the number 2 with the thumb and index finger, while an American displays the number 2 with the index and middle finger. To do this, the service robot 17 contains in the memory 10 codes for different hand gestures representing the same number, step 2650, and they are accordingly specific to each country / culturally conditioned. The patient data kept by the service robot 17 in its memory 10 can also contain one of these codes, step 2652, which accordingly indicates the patient's national / cultural background. Thus, in the memory 10 there are saved multiple gestures for each number, in particular multiple combinations of fingers. As a next step, the codes are compared in order to determine the gesture that the patient prefers to use, step 2655. Just in case the patient's possible cognitive impairment is also taken into account, the test reliability can be improved that the patient displays a number with the hand and / or finger gestures that he is familiar with. The service robot 17 can thus display a hand gesture and / or finger gesture to the patient, for example the number 2, which corresponds to the patient's cultural / national background, step 2660, which is done by the robot hand of the effector 4920, step 2670. As an alternative and / or complementary aspect, this information about the patient's respective coded cultural / national background can also be used to better recognize two fingers displayed by the patient. This finger output and / or recognition is carried out taking into account such codes as an alternative embodiment.

[0270] The service robot 17 is also configured to align the robot hand in space by the actuators 4920 using at least one joint, so that the robot hand can be recognized by the patient in step 2638. To this end, the service robot 17 detects the head of the patient and its orientation in space in that the service robot 17 uses a facial pose recognition method 2625 which is already well established in the prior art, such as included in the OpenPose architecture. Here, in one aspect a histogram of gradients scheme can also be implemented, for example in the OpenCV or Scikit-image architecture. The service robot 17 determines the orientation of the head in space and calculates a line of sight range for the eyes by this architecture. This means in particular a cone angle with an opening angle of 45°, such as 30° (measured from the perpendicular) - in the following this is referred to as "good recognizability" - which is aligned perpendicularly to the front of the head. Thus, the service robot 17 has a cone angle recognition device 2630. Here, the service robot 17 detects its position in space and the position of the actuators 4920, in particular the position of the robot hand, and determines whether this position is within the cone angle 2632. If the position is not within the cone angle, the service robot 17 calculates which angle settings of the joints of the actuators 4920 are necessary to position the robot hand within the cone angle. Here, the service robot 17 calculates a three-dimensional region in space which has a minimum distance to the patient and which changes, for example, depending on the body region of the patient. This minimum distance is saved in the memory 10 of the service robot 17 in step 2636. By recognizing the patient on the bed, for example his head and body, a "permitted region" is calculated in which the robot hand is allowed to move, in which the distance to the head is greater than the distance to the torso or arms. In one aspect, the distance to the head is 50 cm and the distance to the remaining body parts of the patient is 30 cm. The service robot 17 determines in step 2638 in which part of the permitted region of the service robot 17 the robot hand can be positioned in order to enable the "good recognizability" of the hand between the two cone angles. Then, the service robot 17 aligns the robot hand by the actuators 4920 in step 2638 so that the "good recognizability" of the hand by the patient is enabled. If this positioning cannot be achieved, the service robot 17 requests the patient in step 2640 to look at the service robot 17 by means of an output unit, for example a display 2 and / or a speech synthesis unit 133 by means of a loudspeaker 192. Then, it is checked again in step 2642, i.e. steps 2630-2640 are carried out. If the alignment of the determined cone angles is not changed, the service robot 17 cancels the test step 2644 and transmits information to the medical staff in step 2646, for example by means of an interface 188, such as a WLAN, to a server and / or a mobile terminal. As an alternative and / or additional solution, the service robot 17 can request the patient again and / or wait longer.If the robot hand is aligned in a manner that the patient can recognize well, then in step 2670 two fingers can be shown with one hand and the process continues as described before. This is an optional aspect when aligning the robot hand based on facial pose recognition considering "allowed areas".

[0271] Figure 65 A system for cognitive analysis is shown in Fig. 5. The system, such as a service robot 17, comprises a processing unit 9, a memory 10, an output unit outputting values in the memory by a value output module 4940, a person detection and tracking unit (4605) with a camera (185) and a person recognition module (110). The output unit refers to a sound generator, such as a loudspeaker 192, a display 2 and / or an effector 4920, such as a robot arm, in one aspect with a robot hand 4950. The system has in the memory 10 a gesture detection module 4960 for detecting a person gesture, a finger pose generation module 4955 for generating a finger pose (4950) of a robot hand, wherein the finger pose represents a value, for example. The system has additionally a cognitive ability assessment module 4845 for assessing the cognitive ability of a detected person. The system is in one aspect connected with a patient management module 160. The system has rules for determining the cognitive ability of a detected person as already described elsewhere. The system has in one aspect a person identity recognition module 111, a tracking module (112, 113), a motion analysis module 120, a skeleton creation module 5635 and / or a skeleton model based feature extraction module 5640.

[0272] Determination of the state of pain

[0273] The service robot 17 is in one aspect configured such that it can perform a test of the patient's pain sensation, which is realized by the observation of the behavior of the patient by the service robot 17. Here, a processing method of the pain expression indicators of the pain assessment table created in medicine is employed. Such a test is also performed in the context of delirium monitoring. Here, in a first step the facial expression of a patient lying in a bed is analyzed. Some approaches are shown elsewhere in the context of the present invention, such as the recognition of the face of a patient by the service robot 17 when the patient is lying in a bed, the tracking of the patient if necessary, which includes the navigation of the service robot 17 associated therewith. In one aspect the bed is detected by a sensor and the images generated here are analyzed by a modal comparison in order to assess whether it is a bed. In one aspect, these approaches can also be used here. Figure 21a

[0274] State of pain: recognition of the emotion

[0275] ​The service robot 17 analyzes the emotions expressed by the patient through his facial expressions in a first part test. To this end, the service robot 17 has access, in one aspect, to a facial classification database in which classification rules within and across multiple facial candidate regions are saved which, based on facial features, are able to draw conclusions about the emotional state of the patient, which is described in detail below. This two-stage approach deviates from the prior art as described as a one-stage approach in US20170011258 or US2019012599. Within the scope of this embodiment, a histogram of oriented gradients approach can be used, for example, which is implemented within the OpenCV or Scikit-image framework. In recognizing the emotions, basically, emotions are detected which express the level of tension of the patient, from relaxed to extreme tension, which are reflected in the facial distortions.

[0276] Within the scope of the method, the head of the patient is first identified in step 2705, for example, by the OpenPose or similar framework. Such analysis can be performed by a 2D or 3D camera 185. Here, for example, a candidate region of the face is first identified 2710, and then, based on the histogram of gradients algorithm, feature extraction is performed in at least one candidate region in step 2715 which is required for analyzing the emotional state, whereby, for example, the movement of the mouth or the eyebrows can be evaluated. In step 2720, feature classification is performed based on the acquired histogram of gradients data, which feature classification employs existing feature classification, by means of mature clustering methods, such as K-Means, support vector machines and / or by means of labeling facial expressions based on weights acquired by means of a trained neural network, for example, a multi-layer convolutional neural network with backpropagation. In the next step, step 2725, the classification performed on the candidate regions is classified across the candidate regions, again by means of mature clustering methods based on machine learning, such as by means of K-Means, support vector machines and / or the convolutional neural network already mentioned. Here, for example, the movement of the mouth and the eyebrows is jointly analyzed.

[0277] In step 2730 the recognition algorithm can be filtered in different aspects, i.e. the patient age that the service robot 17 has retrieved from a database via an interface 188, such as a WLAN, is corrected in step 2735, provided that the emotion is analyzed directly on the service robot 17. As an alternative and / or complementary solution, the patient head image created by the camera 185 for the recognition of the emotion can also be transmitted to the cloud 18 via an interface 188, such as a WLAN, and analyzed there. In this case, possible age information is transmitted from a cloud storage in the cloud 18 to the module that performs the emotion recognition. Another filter, in one aspect, determines whether the patient is intubated with a tracheal tube (step 2740), which ensures that the patient is artificially ventilated through the mouth. The classification algorithm for the emotion assessment is created, for example, in one aspect, by using training data of pictures of patients who are artificially ventilated using a corresponding tracheal tube. Details on the detection of the intubation are described below and can also be used within the scope of the processing method described herein, in one aspect.

[0278] In step 2745, the emotion is assessed in the range of the determined score on a scale of 1 to 4, for which the determined emotion is compared to emotions saved in the memory 10 and assigned with a scale value. The value "1" indicates a facial expression that is classified as normal, while the level of tension rises on the scale up to the highest value 4, which suggests a facial distortion. There is one matrix diagram for the classification across the candidate regions, which shows the respective score for different facial expressions.

[0279] In one aspect, these values are detected over the course of hours or days, for example when the patient is in a relaxed state at the beginning of a series of emotion measurements by the service robot 17, which can simplify the analysis of the emotional state of the patient if necessary, which can be saved to the memory 10, for example by medical staff, such as via a terminal and menu configuration, which the service robot 17 can access. This includes information about the health status, for example that the patient is pain-free at the beginning of the measurement. Thereby, the facial expressions and emotions can be detected and analyzed in a pain-free and possibly in a pain- accompanied state, whereby the classification features of the pain-free state can be considered for the assessment of the pain- accompanied state and used as a filter. The classification quality is improved by the dynamic classification of the facial expressions (step 2750), since the classification can be achieved based on the facial differences observed at multiple points in time. Thereby, for example, a retrospective classification can also be made, in which, for example, only the extracted features are saved together with a timestamp that characterizes the detection time and reclassified. To achieve this, the facial recordings are saved. In summary, a person is detected individually and / or in association, a face is recognized, candidate regions are selected within the face, features of the surface curvature of the candidate regions are extracted, and the surface curvature of the candidate regions is classified, whereby this classification describes the pain state.

[0280] State of pain: detection of upper limb activity

[0281] The second part of the test focuses on the movement of the upper extremities, such as the upper arm, lower arm, hand and fingers. Here, the service robot 17 tracks the movements recognized by the service robot 17 as described above within the time course, either by means of a 2D camera and an OpenPose or similar architecture or a 3D camera, if necessary by means of an RGB-D camera 185, wherein the analysis takes place, for example, by means of a visually human tracking module 112 and / or a laser-based human tracking module 113. The procedure on the RGB-D camera 185 is to convert the 3D image into a scatter plot in step 2805, to assign a spatial coordinate to each point in step 2810, to perform skeleton model recognition in step 2015 by means of the camera architecture or other software tools of the prior art, wherein the joints of the skeleton can be recognized. Subsequently, joint selection is performed in step 2820, i.e. the joints are identified here with the aim of identifying, for example, shoulder joints, elbow joints, hand joints and finger joints. The angles of these joints are calculated in step 2825, the angles being defined, for example, by means of directional vectors, which take the joint as their origin. In step 2830, the angles of these limbs are detected within the time course in the context of feature extraction. The limbs are then classified in such a way that the number of angle changes per time unit, the speed, i.e. the angular velocity, etc. are taken as a measure of the intensity of the movement. Here, the service robot 17 classifies the movements in step 2835 on a scale of 1 to 4 and saves the value. The value 1 corresponds to no movement within the tracked time. The value 2 corresponds to little arm movement and / or slow movement, 3 to finger movement and 4 to strong finger movement, which are defined, for example, by the number of finger movements per time unit and / or their speed and which are related to threshold values.

[0282] State of pain: vocalization of pain

[0283] The third part of the test focuses on the patient's vocalization of pain, which can be performed in two essentially different procedures, which are aimed at different scenarios and which are represented by two evaluation variants. The first scenario involves a patient on artificial respiration, wherein the vocalization is evaluated on the basis of coughing. In the second scenario, the patient is not on artificial respiration and the typical pain sound is evaluated. Here, the processing method is described in detail in Figure 29 .

[0284] State of pain: vocalization of pain of a patient on artificial respiration

[0285] In a first variant, patients are involved who are on artificial respiration. They either have a tracheal tube inserted, which ensures artificial respiration through an opening in the neck, or a tracheal tube inserted, which enables artificial respiration with the mouth. With the aid of image recognition algorithms, the service robot 17 can identify patients on artificial respiration in step 2901. To this end, the head and neck region of the patient is recorded as a 2D or 3D image, which is used as a candidate region in a first step, wherein the neck is used as a candidate region if a tracheal tube is used and the mouth is used as a candidate region if a tracheal tube is used. The candidate region is identified in step 2905, for example in combination with a face recognition based on a histogram of gradients (HoG), and the candidate region derived therefrom, such as the mouth and the neck, is identified in step 2910. Both regions are analyzed accordingly. Here, a model assumption is made, namely the shape which such a tube typically has (step 2915). Subsequently, the pixels taken by the camera 185 are analyzed by an optional real-time and fault-tolerant segmentation algorithm 2920 to identify such a tube. Thereby, the service robot 17 can detect the tube.

[0286] As an alternative and / or in addition, a database-based identification can be carried out in step 2902, wherein the service robot 17 queries information about the patient's respiration from a cloud-based database in step 2925 via an interface 188, such as a WLAN, and the patient information in the cloud 18 in step 2927, and / or these information are saved together with other patient data in the memory 10 of the service robot 17 (step 2929).

[0287] The service robot 17 determines in how far the patient breathes normally or even coughs 2930 for both ventilation scenarios. The determination process can be carried out in different ways. In one scenario, the service robot 17 uses data from the ventilator and / or the adapter between the endotracheal tube and the ventilator in step 2935. The service robot 17 accesses the ventilator used in step 2936 in one aspect via an interface 188, such as a WLAN, and detects the analysis curves of the breathing cycles carried out by the ventilator, which are detected by pressure and / or flow sensors. The recorded curve progression is compared to thresholds in step 2941, which are typical for different ventilation scenarios, such as pressure or volume-controlled ventilation, and which occur in these ventilation scenarios when coughing. As an alternative and / or in addition, these cases can be labeled, for example, by medical staff, to identify atypical ventilation modalities, such as coughing, which can be classified as coughing in step 2942 by a machine learning algorithm and / or neural network. As an alternative and / or in addition, the curve progression over time (pressure, volume, flow) can be analyzed and deviations in the time course are identified, which no longer occur before and after the breathing cycle, are classified as coughing by a machine learning method and / or neural network. Here, not only the anomaly before and after the cycle can be compared directly in step 2942, but also a chain of several cycles can be detected accordingly, such as a coughing episode involving several coughing events. In the case of a ventilator that supports coughing by adjusting the ventilation, the respective modality of the ventilator can also be identified by the service robot 17, as an alternative to the ventilation curve derived therefrom (pressure / flow over time), and taken into account when classifying the patient's ventilation in step 2944. As an alternative and / or in addition, the service robot 17 can also obtain information from the ventilator that it is in a coughing support mode or triggers coughing, from which the system can detect a coughing event. As an alternative and / or in addition, the analysis results of the ventilator can be accessed via an interface 188, such as a WLAN, and the adapter can also be accessed in step 2937, which can measure the pressure and / or flow in the delivery hose between the endotracheal tube and the ventilator by pressure and / or flow sensors and transmit the signals wirelessly to the service robot 17 via the interface 188, after which the respective analysis results of the respective ventilation scenario are created, which can be analyzed as described above.

[0288] As an alternative and / or in addition, detection can be carried out by at least one sensor located on the patient's body, step 2950. This includes, for example, inertial sensors 2952 (such as with a magnetometer) used in the chest, neck or cheek area, strain sensors 2954 (such as a strain gauge belt placed on the patient's skin), contact microphones 193 (step 2956) which are likewise placed on the patient's skin, in one aspect on the bone immediately below the skin, which can detect coughing sounds, thermistors 2958, for example on or in the nose, and which are each connected wirelessly to the service robot 17 via an interface 188 (such as WLAN). Here, both the direct connection to the sensor devices and the access to the data generated by the sensor devices and saved in the memory 10, such as a hospital information system, are possible. Here, the data in terms of coughing signals can be analyzed or analyzed within the service robot 17.

[0289] Drugman et al., 2013, "Objective evaluation of sensor correlations for automated cough detection, IEEE Journal of Biomedical and Health Informatics, vol. 17 (3), May 2013, pages 699-707 (DOI: 10.1109 / JBHI.2013.2239303) has shown that the detection of coughing by means of a microphone 193 has the best results compared to the sensors used in the previous section, wherein only un-tubed patients were included in the observation, although in this case the patients had a tracheostomy or a tracheostomy was possible. On the basis thereof, in one aspect at least one microphone 193 (step 2960) is used, which is located on the patient and / or on other locations in the patient's room and is connected directly (or indirectly to a data document in the memory 10 which the service robot 17 has access to (in one variant the archive also includes data analyzed in terms of coughing signals)) to the service robot 17, such as the integration 2962 of the at least one microphone 193 in the service robot 17, the recording of the noise in the patient's surroundings in the memory 10 of the service robot 17 and the subsequent classification of the sound signal as to whether a cough occurred. Here, for example, machine learning and / or neural network algorithms are used which have been trained with the recorded cough noises.

[0290] In order to create such a classification, in one aspect a system can be trained which has at least one processor for processing audio data and at least one audio data memory in which audio data is saved, which in one aspect can also exist as spectral data and is labeled accordingly.

[0291] In an alternative and / or complementary aspect, the 3D sensor of the service robot 17, such as a 3D camera, is able to determine the movement around the mouth, however also including the movement of the chest and / or neck, i.e. the candidate region 2974 is analyzed by a fault-tolerant segmentation algorithm 2976. It has been described elsewhere how the mouth can be detected. The candidate region 2974 is determined from the detection of the chest and abdomen, such as by the distance of the shoulder joints above the skeleton model, and the same distance is determined in the direction of the feet, orthogonal to this line, to identify the candidate region of the torso under the bed sheet, consisting of the chest and abdomen, both of which move when breathing, so that they can be analyzed. As an alternative and / or complementary solution, the candidate region 2974 can also be determined, for example, by extending twice the head height, about 2.5 times the head width, from the chin downwards. In both cases, the head is identified here as an initial step and is thus used as a reference 2972, from which the candidate region can be identified therefrom. As an alternative and / or complementary solution, the dimensions of the bed can also be used as a reference 2972 to identify the candidate region 2974. As an alternative and / or complementary solution, it is also possible to use the 3D camera to detect the protrusions of the bed sheet surface, wherein in one aspect a gradient histogram evaluation is carried out for analysis on a classification basis, which is trained by a system which contains 2D or 3D images of the bed as input variables, after which it is identified whether there is a patient on the bed and they are analyzed by classification methods from the field of machine learning and / or neural networks, while the analysis result constitutes a classification which, inter alia, enables the detection of the upper body of the patient.

[0292] In step 2978, within the scope of feature extraction, the movements of the mouth, cheeks, neck and upper body detected by the 3D camera are analyzed over the time course. Here, in accordance with Martinez et al. 2017, "Respiratory rate monitoring during sleep by depth camera in real conditions", 2017 IEEE Winter Conference on Computer Vision (WACV), 24-31 March 2017, (DOI: 10.1109 / WACV.2017.135), interference reduction 2980 takes place on the basis of motion detection, which is covered by clothing or a bed sheet, in particular the upper body / abdomen. By determining the power density spectrum, interference is eliminated which arises from the detected bed sheet movements and which itself exacerbates the difficulty of breathing movement detection, which contributes to the detection of the movements of the thorax. Here, for example, in step 2982, the power density spectrum is determined for each pixel in the three-dimensional space detected over the time course, for example by means of a fast Fourier transform (FFT), after which the power density spectra of all pixels are aggregated in step 2984 and the maximum value is determined by quadratic interpolation in step 2986, wherein the position of the maximum value in step 2988 indicates the respiratory frequency. Subsequently, they are monitored for frequency changes, which are represented as coughs in step 2990. Subsequently, the frequency of the detected limbs is determined in step 2990. Here, for example, the use of a gradient histogram calculation is taken into account in step 2990. Subsequent feature classification takes place in step 2992 on the basis of the classification generated by the cough movement and non-cough movement recordings, for which standard classification methods and / or neural networks can be used, as described elsewhere in this text. If no cough is detected by the above-described approach, this criterion is assessed as 1 point. If a cough is detected, it is assessed as 2 points.

[0293] A summary of this procedure is described as follows: a person is detected, the face and neck are recognized, the face and neck area are analyzed in terms of the modalities described for the device for performing artificial respiration, a value is saved upon detection of the modalities described for the device for performing artificial respiration, wherein the device for performing artificial respiration describes a state of pain.

[0294] State of pain: vocalization of pain of a patient without artificial respiration

[0295] If the service robot 17 does not recognize a cannula on the patient by means of the implemented image recognition and / or no information about artificial respiration is saved in the database with patient information, another variant of the third part of the test is carried out, in which the service robot 17 analyzes the noises emitted by the patient by means of the microphone 193. These noises are classified by means of a machine learning algorithm and / or a neural network algorithm using an algorithm trained on the basis of labeled noise data, whereby different characteristic pain cries can be recognized by means of the noises. If no pain cries are found, the criterion is assessed with the value 1. If pain cries are detected with a frequency of less than three times per minute for a duration of less than 3 seconds, the criterion is assessed with the value 2. An increasing frequency or a longer duration is assessed with the value 3, while the detection of verbal pain cries, which can also be determined in one aspect by means of a conversation with the patient, is assessed with the value 4.

[0296] The scores of the three-part test are finally added together. The results are saved in a database, which in one variant can be transmitted to a server in the cloud 18 by means of an interface 188, such as a WLAN, and saved there. In both variants, the medical staff has access to the analysis results, whereby the test results (partial results and overall results) can also be observed in detail and displayed visually by means of a terminal. In one alternative and / or supplementary aspect, the individual test parts can also be carried out independently of one another.

[0297] In summary, the method for determining the pain state on the basis of pain cries of a patient without artificial respiration is summarized as follows: recording a sound signal, analyzing the sound signal by means of a pain classification in order to determine whether the recorded sound signal represents a pain cry, evaluating the sound signal classified as a pain cry by means of a pain intensity classification, wherein the pain intensity classification comprises assigning scale values to the recorded sound signal and each scale value represents a pain state. In one aspect, the following steps are additionally carried out: determining the location of origin of the sound signal, determining the location for determining the pain state of a person, adjusting the determined location by comparison with a threshold value, i.e. in terms of the lowest similarity of the location values, and saving a value if the determined pain state is below the threshold value compared to the determined pain state.

[0298] In Figure 66The system for determining the pain state of a person is summarized as follows: The system or service robot 17 comprises a processing unit 9, a memory 10 and sensors for contactless detection of a person, such as 2D and / or 3D cameras 185, laser radar 1, radar and / or ultrasound sensors 194. Depending on the configuration of the pain state determination, there can be different modules in its memory 10. In one aspect, the system has a person recognition module (110), a visually person tracking module (112), a face recognition module for recognizing faces 5005, a face candidate region module for selecting in candidate regions in faces 5010, an emotion classification module for classifying the surface curvature of the candidate regions of the expression 5015 and an emotion evaluation module for determining the emotion scale value 5020. For example, the system has a bed recognition module for recognizing beds 5025 and / or an upper limb analysis module 5035 for detecting the upper limbs of a person, tracking the upper limbs over time and analyzing the intensity of the angle change per unit of time between the torso and the upper arm, the upper arm and the lower arm and / or the finger joints and the metacarpal bones, their speed and the number of angle changes, and for example, a pain state calculation module 5040 for determining the scale value of the pain state. In one aspect, the system comprises a microphone 193 for recording sound signals, for example, an audio source position determination module (4420) for analyzing the source position of the sound signals and an audio signal-person module (4430) for correlating the audio signals with a person. The system can have, for example, a pain vocalization module (5055) for classifying the intensity and frequency of the sound signals and determining a scale value representing the pain vocalization. In one aspect, the system comprises an artificial respiration device recognition module 5065 for recognizing devices for artificial respiration, i.e. selecting candidate regions of artificial respiration, analyzing the candidate regions of artificial respiration by object recognition and object classification means for recognizing intubations, such as tracheal intubations or tracheal intubations. Furthermore, movements, air flows and / or sounds can be detected and / or classified in terms of pain manifestations by a pain sensitivity analysis module 5085 for analyzing sensors fixed on the person's body, such as inertial sensors, strain sensors, contact microphones and / or thermistors. In one aspect, the system has a person identity recognition module 111, a motion analysis module 120, a skeleton creation module 5635 and / or a feature extraction module based on a skeleton model 5640.

[0299] Determination of the blood pressure and determination of other cardiovascular parameters

[0300] The service robot 17 is also equipped with a system which is able to detect repetitive activities of the human body which are associated with the blood output from the heart to the main blood vessels with each heartbeat. Here, changes in the movement which are generated by the movement of the main blood vessels and which propagate for example in the form of a wave in the body and / or which are generated by the arterial movement in the skin are detected. The latter are more resistant to fluctuations in the illumination of the body part and / or different shades of the skin. As an alternative and / or in addition, changes in the blood volume or blood flow in the skin which are associated with the heartbeat are detected for example in the time course. Figure 30 The process of data detection and analysis is shown.

[0301] In step 3005 the body region and the plurality of partial regions are identified. For example, the body region comprises the face, wherein the analysis is carried out for example by means of the camera 185. The system uses algorithms from the prior art, such as OpenCV, OpenPose or dlib architecture, to detect and track the face (also alternatively other body regions), wherein the analysis is carried out for example by means of the visually human tracking module 112 and / or the laser-based human tracking module 113. Here, at least the forehead, the cheek or the chin are detected as partial regions, for example a plurality of body regions are detected jointly, which are then analyzed separately and / or separately according to the steps explained below. Here, for example candidate regions, i.e. partial regions of the face, which are relevant for the analysis, can be selected, for which segmentation methods known from the prior art (such as RANSAC) can be used. These partial regions can be tracked, also including the tracking of the body region tracked in step 3010 in the time course by means of the architecture described.

[0302] In an optional step 3015, the camera 185 is aligned as parallel as possible to the area that shall be tracked. To this end, the detection angle of the face can be minimized, which detection angle is derived from the axis perpendicular to the camera detecting the face and the axis perpendicular to the sagittal plane of the face. Here, the system determines, for example, a plane in which a face can extend through and which is substantially parallel to the top view of the face and, for example, coincides with the sagittal plane. Based on, for example, a histogram of oriented gradients (HoG), the system has a classification method that describes deviations from this top view and, in turn, can detect the inclination of the face in three-dimensional space. In one aspect, the system evaluates with this processing method to what extent the face looking into the camera 185 of the system is aligned parallel to the camera lens. If deviations occur, the system can in one aspect adjust the inclination of the camera 185 in three-dimensional space by means of corresponding mechanical means, such as a tilting unit 5130, for example by actuating two tilting axes arranged orthogonal to each other, which are driven by a servo motor. The term tilting unit thus refers herein to a tilting unit having at least two axes, which on the one hand enables an inclined position of the horizontal plane and on the other hand enables a rotation about a vertical axis. As an alternative and / or complementary solution, the wheels of the service robot 17 are actuated such that the service robot 17 rotates towards the person in order to reduce the determined deviation of the camera plane from the plane of the face. In an alternative and / or complementary aspect, such a deviation triggers a voice output of the service robot 17, which gives the patient an instruction to align his face accordingly. Here, for example, a rule is stored that requests an alignment in the XY plane when a deviation in the XY plane is found. For example, the face inclination is calibrated and the tilting mechanism is actuated, the orientation of the service robot 17 is determined and / or a voice output is made depending on the patient for as long as the angle between the camera plane and the plane of the face reaches a minimum. As an alternative and / or complementary solution, the detection angle of the camera 185 with respect to the sagittal plane and, if necessary, also with respect to the transverse plane can be minimized, which for example also includes the driving mobility of the robot 17.

[0303] In an optional aspect, the system is configured in step 3020 to illuminate the body part, such as the face. That is, the face (or other body part) is illuminated with a light that illuminates the body part of the patient to be photographed during the photographing by the camera 185. Here, at least one light is used, which is located, for example, in the vicinity of the camera 185. This light is ideally located below and / or above the camera 185, i.e. perpendicular to the camera 185. The emitted light is scattered, for example, to ensure that the face to be photographed is illuminated as uniformly as possible. Depending on the position of the face and its size, the side arrangement of the camera 185 is such that the nose can produce a shadow projection that hits the cheek located next to the nose and whose recording can provide a signal-to-noise ratio that is above average, which can cause a deterioration in the analysis quality, if necessary.

[0304] The camera 185 used provides at least one color channel for the analysis, wherein for example at least one green signal channel is included, since here the emitted light can be absorbed particularly well by hemoglobin. The camera 185 can also provide color channels in orange and / or cyan tones in another aspect. The color depth of each channel is for example at least 8 bits, and the frame rate is for example 30 pictures per second. In one aspect, the camera 185 can be an RGB-D camera 185, which in addition to color recognition can also provide depth recognition, for example based on a time-of-flight sensor or a Speckle pattern, in order to detect regular vascular blood flow and regular vascular distension.

[0305] In a first step 3025, signal extraction is carried out. For this purpose, the input signal is first selected on the basis of the video signal of the region being tracked, which can either be the motion caused by the pumping regularity of the heart and / or the color change resulting from the flow of blood, in particular hemoglobin, from which the regular vascular blood flow and / or regular vascular distension can be detected.

[0306] In a second step, a color channel analysis is carried out on the basis of the raw data with the inclusion of known information, determining which features are mapped by which color channel, provided that this relates to the detection of blood flow. This refers in particular to the channel weighting in step 3030. Here, in particular the green color channel, the green and red color channels (for example to observe the difference between the green and red channels), combinations of green, cyan and orange, etc. can be analyzed. In order to detect motion, the position resolution can be determined as a complementary or alternative solution. That is to say, the vertical and / or horizontal motion of the detected facial attributes is tracked, for example the facial position and partial regions thereof in the temporal course are detected and analyzed. This includes both the motion of the head and also the individual facial parts.

[0307] The subsequent signal determination in a first partial step (preprocessing 3035) uses for example at least one filter. This can include trend adjustment (for example by scaling and / or standardization); smoothed mean observation, high-pass filtering; band-pass filtering, if necessary designed as adaptive band-pass filtering; amplitude selection filtering; Kalman filtering; and / or continuous wavelet transformation. As an alternative and / or complementary solution, a linear polynomial approximation of the second power can also be used.

[0308] Subsequently, a signal separation method 3040 is used in order to improve the signal-to-noise properties and to reduce the number of observed feature dimensions. Here, for example principal component analysis or independence analysis can be used, in one aspect also machine learning methods.

[0309] The signal processing 3045 comprises determining the pulse rate and, if necessary, other quantities in the range of a Fourier transform (fast or discrete Fourier transform, in particular for determining the maximum power spectral density), an autoregressive model (for example by means of the Burg method), the use of bandpass filters associated with detecting maxima, such as identifying wave crests, a continuous wavelet transform and / or a machine learning model, in particular an unsupervised learning. As an alternative and / or in addition, a discrete cosine transform can also be used.

[0310] In the range of the post-processing 3050, different methods can also be used in order to compensate for errors, for example due to head movements, for which a Kalman filter, a (adaptive) bandpass filter, an outlier detection, a smoothed average, a Bayesian fusion and / or a machine learning method can also be used.

[0311] The processing steps carried out so far have already reflected a part of the medically relevant parameters, such as the pulse rate, the pulse rate variability, the pulse wave propagation time, the pulse wave shape, etc. In step 3055, medically relevant parameters are continued to be calculated, for example on the basis of different approaches described in the prior art, by means of which the systolic and diastolic blood pressure can be determined, for which linear or non-linear prediction methods can be used.

[0312] The machine learning methods mentioned, such as neural networks, for example convolutional neural networks, can identify hidden and partly unknown features in the data and take them into account in the analysis, for example in the range of a cluster analysis carried out. Here, for example, the weights of a classification or linear and non-linear prediction model are generated on the basis of training data, which can then be used in the range of the described flow in the production run.

[0313] In one aspect, the determined pulse rate, pulse rate variability values and, if necessary, other values are compared with the values saved in the memory 10 after the post-processing has been carried out in one step, on the basis of which the pulse rate, the pulse rate variability or other quantities, in particular including the systolic and diastolic blood pressure, are determined.

[0314] Filtering is performed in the context of pre- and post-processing of the to-be-detected quantities. In one aspect a bandpass filter can be used for the pulse amplitude, which covers a spectrum of 0-6 Hz, such as at least 0.7-4.5 Hz. In one aspect, the pulse signal can also be sampled in a smaller range within this frequency range, such as in a window of 0.1 Hz. Subsequently, smoothing can be performed by a low pass filter. For example, the pulse rate, heart rate or pulse frequency or heart frequency can be processed by a bandpass filter with a width of 0.7 to 4 Hz. For determining the pulse rate difference, a bandpass filter with a window of 0 to 0.4 Hz can also be used, in one aspect, sampled at intervals of 0.02 Hz. The pulse wave propagation time can be obtained by comparing the values of at least two regions, wherein the analysis can be performed in a bandwidth of 0.5 to 6 Hz, in one aspect, sampled at intervals of 0.1 Hz. The pulse wave propagation time can be determined by comparing the values of multiple regions. The pulse shape is characterized by the unsampled trend in a frequency spectrum of about 0-6 Hz, such as by the area under the curve, the height and / or the width. The first derivative of these values yields the pulse energy.

[0315] Thereby, the blood pressure can be determined by a linear model, for example, by the pulse wave propagation time and the pulse rate or heart rate and the previous blood pressure value, and can be used in a linear regression model or a neural network. The shape of the measured pulse can also be analyzed, for example, by determining the difference between the pulse curve and the perpendicular line drawn through the maximum, thereby replacing the previous blood pressure value.

[0316] As in Figure 67As shown in the middle, a diagram of a system for determining blood pressure is as follows: The system for determining cardiovascular parameters of a person in one aspect, namely the service robot 17, comprises a processing unit 9, a memory 10 and a camera 185, such as a 2D and / or 3D camera, and furthermore comprises a body region detection module 4810, a body region tracking module 4815, a face recognition module 5005, a face candidate region module 5010 and a cardiovascular activity module 5110 for detecting motion resulting from cardiovascular activity. The camera 185 provides at least one 8-bit green color channel. The system additionally has a light 5120 to illuminate the face during recording by the camera 185, which light is located, for example, above and / or below the camera 185. The system has a blood pressure determination module 5125 capable of determining systolic and diastolic blood pressure 5125 and / or a flipping unit 5130 to minimize the detection angle of the camera 185 relative to the sagittal plane by the largest possible amount. To this end, the system has a number of rules in order to, for example, place a perpendicular line between the eyes of the person being detected, thereby dividing the head into two halves. The face is segmented, wherein a histogram of gradients is placed over each segment. If they have a similarity below a certain threshold (mirror-opposite), the face is perceived as being in a state of vertical monitoring. Now, the camera 185 can be manipulated by the flipping unit 5130 such that during the manipulation, the mirror-opposite halves of the face are compared by the histogram of gradients and the camera is positioned such that the threshold of the histogram of gradients is ensured to be below. In one aspect, the system has a person recognition module 110, a person identity recognition module 111, a visually person tracking module 112, a motion analysis module 120, a skeleton creation module 5635, a feature extraction module based on a skeleton model 5640 and / or a motion planner (104).

[0317] Detection of a substance on or under the skin surface

[0318] In one aspect, the service robot 17 is also equipped with a detector 195, which is located, for example, on the patient-facing side of the service robot 17. The detector 195 is, in one aspect, fixedly integrated into or on the surface of the service robot 17. In an alternative and / or supplementary aspect, the detector 195 is mounted on an actuator 4920, such as a robotic arm, and is subsequently aligned with the patient's body surface identified by the service robot 17, as exemplarily described for aligning the spectrometer 196 with the patient's skin, and in one aspect, touches the patient's skin in this manner. As an alternative and / or supplementary aspect, the service robot 17 can also request the patient to touch the detector 195, for example, with a finger. Here, in one aspect, the service robot 17 can verify whether the patient has actually touched the detector 195. In one aspect, verification can be performed by trial measurement, wherein the detected value is compared with a measurement interval stored in the memory 10 to assess whether the patient has actually placed their finger on the detector 195. However, in this approach, it cannot be ruled out that the measurement results may be affected by the orientation of the finger on the sensor. Therefore, in an alternative and / or supplementary aspect, finger tracking is based on a camera, wherein analysis is performed, for example, by means of a visual human tracking module 112 and / or a laser-based human tracking module 113. Such tracking has been described elsewhere in this document. As an alternative and / or supplementary approach, a dialogue-based method can be used, wherein the service robot 17 asks the patient whether the service robot 17 has correctly positioned the finger, which can be done via a display 2 and / or voice output.

[0319] The surface of detector 195 is composed of crystals, such as those with a cubic lattice structure (e.g., diamond), a hexagonal lattice structure, or a quadrilateral lattice structure. The refractive index of the crystal is 1-4, for example 1.3-1.4, 2.2-2.4, or 3.4-4.1. The spectral width of the crystal is spaced between 100 nm and 20000 nm, for example, between 900 nm and 12000 nm. The measurement method of detector 195 here utilizes the deflection of the material on the crystal surface caused by the laser-induced excitation of an analytical laser 5205, which is excited by another laser 5210 on and / or within the patient's skin. Here, the surface excited by the other laser 5210 is, for example, coupled to detector 195 at the position where the analytical laser 5205 is deflected on the crystal surface. For analysis, feature extraction is performed, where features include the wavelength change of the other laser 5210 and the resulting deflection of the analytical laser 5205 detected by a sensor based on the photoelectric effect. Here, it is possible to use... Figure 30The steps shown in Fig. 3, in particular 3025-3050, have been described in detail elsewhere. Subsequently, the features are classified by comparison with the feature classes saved in the memory 10. Here, a specific substance and its concentration are assigned to, for example, the specific wavelength and / or the wavelength change of the further laser 5210 and the analysis of the deflection of the laser 5205 based thereon. Subsequently, the determined classification is saved and output via the display 2 and / or saved in the patient management module 160.

[0320] In an alternative and / or supplementary embodiment, a camera-based system is used, which is aligned to the skin surface of the patient and is able to perform the measurement. Here, in one aspect, the system is fixedly installed on the service robot 17 or is able to align three-dimensionally as such, whereby the skin surface of the patient can be detected without the patient needing to move. Here, as described for the detection of the mood, the service robot 17 detects, for example, the patient area of the patient whose skin surface should be detected.

[0321] The skin surface to be detected is illuminated by at least one camera 185, for which in one aspect an LED lamp is used, which forms a spectrum in the range of 550-1600 nm, for example at least in the range of 900-1200 nm, and thus belongs to the infrared waveband. Here, the sensor of the at least one camera 185 is a gallium arsenide indium or lead sulfide-based sensor, which in one aspect is supplemented by a silicon-based sensor, which is integrated in another camera 185 if necessary. In one aspect, the LED lamp can be replaced by a laser. The light source is thus operated such that the wavelength of the light source changes over time. Here, the at least one camera 185 detects the emission of the substances on or in the skin excited by the light. During the measurement, feature extraction is carried out, which determines the phase and frequency of the emission of the substances on and in the skin, in one aspect also taking into account the frequency of the emitted light. Here, pre- and / or post-processing can be carried out, in which different filters are used, such as bandpass filters and / or lowpass filters. In general, it is also possible here to carry out a Fourier analysis of the signals, in which the frequency of the signals is determined. Figure 30 The steps 3025 to 3050 shown in Fig. 3 and described in detail elsewhere are carried out. Subsequently, the concentration of the substance is determined based on the feature extraction.

[0322] As in Fig. 3, the steps 3025 to 3050 shown in Fig. 4 and described in detail elsewhere are carried out. Subsequently, the concentration of the substance is determined based on the feature extraction. Figure 68As shown in the middle, a diagram of a system for substance measurement is as follows: A system for measuring a substance on and / or in a person's skin in one aspect comprises a probe 195 with an analysis laser 5205 and another laser 5210, wherein the analysis laser deflects from a crystal surface, for example, and the other laser 5210 excites the substance by a change in wavelength, wherein the excited substance region engages the medium 5215, such as a crystal, at the location of the analysis laser 5205 deflection, and further the system comprises a laser change module 5225 for feature extraction and feature classification of the wavelength change of the other laser 5210 and a laser deflection analysis module 5220 for analysis of the deflection of the analysis laser. The system has, for example, sensors for contactless detection of a person, a motion analysis module (120) for analysis of the person's motion over time and / or a finger positioning recognition module 5230 for automatic recognition of the positioning of a finger on the medium 5215 and performing a measurement after the finger is placed on the medium. A system for measuring a substance on and / or in a person's skin in one aspect comprises, for example, a service robot 17, a probe 195 with a medium 5215, which has a cubic, hexagonal or tetragonal lattice structure, a refractive index of 1-4 and a spectral width interval of 100 nm-20,000 nm. The system further comprises an analysis laser 5205 and another laser 5210, wherein the analysis laser 5205 deflects from a crystal surface, for example, and the other laser 5210 excites the substance by a change in wavelength, wherein the excited substance region engages the medium 5215 at the location of the analysis laser 5210 deflection. In addition, the system can comprise a laser change module 5225 for feature extraction and feature classification of the wavelength change of the other laser 5210 and a laser deflection analysis module 5220 for analysis of the deflection of the analysis laser 5205. The analysis laser is analyzed by a sensor based on the photoelectric effect 5250. The system can additionally contain an interface for transmitting data to a patient management system 160. The probe 195 can be positioned on an effector 4920 and the system has a module containing rules in order to be able to position the probe 195 on a person's skin, for example, by balancing the effector 4920 and the location on which the effector is to be positioned and manipulating the effector such that the distance between the effector 4920 and the location on which the effector 4920 is to be positioned falls to at least near zero. Furthermore, the system has sensors for contactless detection of a person, such as 2D and / or 3D cameras 185, laser radar 1, radar and / or ultrasound sensors 194. In one aspect, the system has a body area detection module 4810 and a body area tracking module 4815 for tracking the measurement area.In one aspect, a system for measuring substances on and / or in human skin is equipped with a camera 185 and a tilting unit (5130) capable of horizontally and / or vertically aligning the camera 185, with a body area detection module (4810) and a body area tracking module (4815) for identifying a person and tracking the body area of the person over time (in one aspect the same as the person identification module 111 and the tracking modules 112 and 113), and with at least one light source 5270 capable of illuminating the skin of the person to be examined, wherein the system has a wavelength change unit 5275 for changing the wavelength of the light emitted by the at least one light source and a wavelength change analysis unit 5280 for analyzing the wavelength change of the detected signal. The at least one light source 5270 can be a laser (in one aspect the same as the lasers 5205 and / or 5210) and / or a plurality of LED lights with different spectra, which can be controlled accordingly. The wavelength of the emitted light can be 550 nm to 1600 nm, such as 900 to 1200 nm. The camera 185 can have a gallium arsenide indium or lead sulfide material light detector, for example. In one aspect, the system has a further camera 185 for detecting light with a spectrum of 400-800 nm. The system can have a material classification module 5295 for feature extraction and feature classification of the detected data and comparison of the classified data with a substance classification, i.e. by comparing the analyzed features with saved features, for example at least the detected light. In one aspect, the system has a person identification module 110, a person identity identification module 111, a tracking module (112, 113), a motion analysis module 120, a skeleton creation module 5635 and / or a feature extraction module based on a skeleton model 5640.

[0323] Humidity recognition and robot navigation

[0324] It can occur within the surrounding environment in which the service robot 17 moves that the ground on which the service robot 17 and the person tracked by the service robot 17 move becomes wet, for example due to a cleaning operation or a liquid spill. Depending on the person being guided for exercise by the service robot 17, such a wet surface can constitute a threat with an increased risk of falling. In order to reduce the risk of injury to the person, the service robot 17 has in one aspect corresponding sensor means in order to recognize the wetness of the ground. Different sensor technologies used here are described in the prior art:

[0325] Yamada et al. "Discrimination of road conditions to understand the driving environment of a vehicle", IEEE Transactions on Intelligent Transportation Systems, vol. 2(1), March 2001, 26-31 (DOI: 10.1109 / 6979.911083) describes a method for detecting the humidity on the ground by means of the polarization of the incident light. Here, the Brewster angle (53.1°) is used as the angle of inclination in order to set the reflection to 0 in the horizontal polarization plane, while the vertical polarization exhibits a stronger reflection. On the basis of the measured intensity relationship of the horizontal and vertical polarization, it is determined how great the humidity on the measured surface is.

[0326] In contrast, Roser and Mossmann, "Weather condition classification on monochrome images", 2008 IEEE Intelligent Vehicles Symposium, June 2-6, 2008 (DOI: 10.1109 / IVS.2008.4621205) propose a solution which has no polarization filter and is based on picture parameters such as contrast, brightness, sharpness, hue and saturation which are extracted from the picture as features. In this case, the brightness is taken into account by means of the Koschmieder model which is established during the image processing, wherein the brightness is related, inter alia, to the attenuation and scattering of the light. The contrast can be determined by means of the difference of the local brightness extrema, wherein here the brightest and darkest pixels in the observed region are compared with one another. In terms of sharpness, this solution is based on the Tenen degree criterion which is established during the image processing. The hue and saturation are determined by means of defined pixel groups. For each detected region, a histogram with 10 regions is generated for each feature, and from this a vector is derived which contains the results of the individual features. These vectors are classified by means of machine learning / artificial intelligence methods, including k-NN, neural networks, decision trees or support vector machines. Here, initially data which is labeled in advance before the training algorithm is provided, wherein the determined classification allows an assessment of future taken ground pictures in order to determine how wet the ground is.

[0327] In contrast, the US patent application No. 2015 / 0363651 Al analyzes the surface texture taken by the camera in order to check its humidity, and here two images taken at different points in time are compared, feature extraction being carried out. The features include the spatial proximity of the pixels within a region (i.e. search for recurring modalities), edge detection and its spatial orientation, the similarity of the gray scale between the images, the defined Laws texture energy measure, the autocorrelation and power density model, and the texture segmentation (based on regions and based on boundaries, i.e. edges between pixels with different textures).

[0328] In contrast, McGunnicle, 2010 "Detecting wet surfaces using near-infrared illumination", Journal of the Optical Society of America A, Vol. 27(5), 1137-1144 (DOI: 10.1364 / JOSAA.27.001137) uses infrared diodes with a spectrum of approximately 960 nm and records the light emitted by these diodes via an RGB (CCD) camera in order to analyse the spectrum accordingly. Here, McGunnicle describes that wet surfaces emit a characteristic spectrum, whereby the wetness on the surface can be detected.

[0329] Another approach is to use light in the range of visible or invisible radar waves, in particular ultra-wideband radar waves, for substance analysis. The reflected signal can be analysed (i.e. classified) as described in the prior art, wherein characteristic features can be identified when measuring the humidity on a surface, whereby the type of humidity can be detected.

[0330] In any case, the sensor is arranged on the service robot 17 such that it can detect at least the surface in front of or below the service robot 17, in one aspect also sideways or pointing backwards.

[0331] In one aspect, an algorithm for determining humidity is stored in the memory 10 of the service robot 17, for example as values in a database, whereby it is possible to detect the spectrum of the infrared waveband based on the spectral analysis or the radar waves emitted by the service robot 17 and reflected by the surface. The service robot 17 has, in one alternative or supplementary aspect, a self-learning system 3100 (see Figure 31), in order to distinguish between wet and dry ground. This self-learning system can for example help an optical method for determining the texture and / or reflection of a surface. For this purpose, the service robot 17 drives over surfaces 3110, which the service robot 17 usually moves over, in a dry state. Here, the service robot 17 records the surface by means of at least one integrated sensor, step 3120, carries out feature extraction, step 3130, for example according to the Roger and Mossmann approach or according to the teaching of the US patent application with the application number 2015 / 0363651 Al. This is preferably carried out at different times of day in order to take into account different light conditions (daylight, artificial lighting and / or a combination of both). The recorded measurement values are assigned a value by an input device which is connected to the service robot 17 in one aspect by means of the interface 188 (such as WLAN), which characterizes the recorded surface as dry (labeling 3140). This value is saved in the memory 10 together with the recorded measurement values in step 3145. In a further step 3150, the service robot 17 partially drives over the previously driven-over surface, but in which the previously driven-over surface is wet. For this purpose, the input device which is connected to the service robot 17 in one aspect by means of the interface 188 (such as WLAN) is used to assign a value to the recorded measurement values, which characterizes the recorded surface as wet (labeling 3140). The order in which the dry or wet surface is driven over first (or even the order is reversed) has little influence on the effectiveness of this method. Subsequently, the features recorded by the sensors are classified 3160 by means of a machine learning / artificial intelligence method, as shown by Roger and Mossmann. The result is that the surface is detected as wet or dry in step 3170. The result of the feature classification, i.e. whether the surface is wet or dry, is saved in the memory of the service robot 17 in step 3180.

[0332] When the service robot 17 drives in the future, the service robot 17 has access to the saved classifications (for example for radar reflections, infrared red spectrum absorption, light reflection or recording of the texture by means of a camera) and analyzes by means of the saved classifications whether the detected surface is wet or dry by means of the measurement values detected by its sensors.

[0333] Figure 32Navigation of the service robot 17 is shown in the case of humidity on the surface 3200 of the floor. If the service robot 17 is navigated in the company of a person 3210, for example to accompany a patient, the service robot 17 records the surface characteristics of the floor over which the service robot 17 moves, step 3120. Feature extraction 3130, feature classification 3160, and consequently humidity detection 3170 are carried out. For this purpose, in one aspect, the service robot 17 detects the width of the dry or rather wet area 3230, for example by means of a rotational movement 3220 around the vertical axis of the service robot 17, in an optional step, depending on the sensors used or the analysis algorithms implemented. In one aspect, this embodiment can be saved in the motion planner 104. Depending on the sensor type and implementation, instead of a rotational movement of the service robot 17, a flipping unit 5130 can be used, or the detection angle of the sensor is wide enough to detect the surface in the direction of travel even without movement. Here, the width is determined, for example, in the case of a right angle to the direction of travel. The width of the dry (alternatively wet) area is compared 3240 to the value saved in the memory. In one optional aspect, the relationship of the width of the wet area detected to the width of the space in which the service robot 17 moves is determined. If the width of the dry area detected is smaller than the width saved in the memory, the service robot 17 does not move into the wet area, but rather stops and / or turns in step 3250 as saved in the motion planner 104. In one optional aspect, an output is carried out by the output unit (display 2, loudspeaker 192, if necessary also including the projection device 920 / alarm light), which indicates the surface identified as wet. The service robot 17 in one optional aspect sends a message to the server and / or terminal in step 3260 via the interface 188, such as WLAN. If, on the contrary, the dry area detected is wider than the threshold value, the service robot 17 is navigated through the dry area in step 3270 as saved in the motion planner 104. Here, the service robot 17 maintains a minimum distance to the surface detected as wet as saved in the motion planner 104, step 3280. The service robot 17 in one optional aspect can indicate the wet surface to the accompanying person by means of the output unit (display 2, loudspeaker 192, if necessary also including the projection device 920 / alarm light), step 3290.

[0334] In one aspect, the classification of humidity also includes humidity. Thus, for example, there can be a very thin film of moisture on a surface that is perceived as dry in itself, but which has little influence on the friction that an object can experience on the surface.

[0335] In summary, the method of detecting and assessing humidity on a surface is summarized as follows: detecting a surface, such as a floor, classifying surface features for detecting humidity on the surface, segmenting the detected surface into wet and non-wet areas, measuring the width of the detected area, and comparing the measured width of the detected area to at least one stored value in order to assess the width of the detected area.

[0336] An alternative procedure is summarized below as shown in Figure 81 : detecting a surface 6005, classifying the surface to detect wetness 6010, segmenting the surface into wet and non-wet areas 6015, storing the wet areas into a map 6020, measuring the smallest dimension of the face 6025 and outputting it based on it via output means 6030, transmitting a message 6035 and / or modifying values in the memory 10 (step 6040). As an alternative and / or complementary solution, the path planning and / or motion planning 6045 can be modified. Figure 82 A part of the procedure is also shown. The service robot 17 is passing through a corridor 6071 (see Figure 82 a) moving, wherein there are multiple segments of wetness detected by the service robot on the floor 6070. The service robot 17 plans a new path 6073 in the path planning module 103, which is calculated as an obstacle based on the wetness. The service robot 17 compares the width between the segments of the face measured as wet, stored as an obstacle, such as from a map stored in the map module (107) 6074, and keeps a safe distance from the segments of the face detected as wet and follows the newly calculated path (see Figure 81 c). As can be seen in Figure 81 d), the width of the segments of the face detected as wet can ensure that the service robot does not navigate around it, as the width between the area classified as wet and the walls of the corridor is smaller than the width of the service robot 17, so the service robot 17 stops in front of it.

[0337] As can be seen in Figure 69As shown in the middle, the system for detecting humidity is described as follows: it comprises a sensor for contactless detection of a surface (such as a camera 185 or a radar sensor 194), a segmentation module 5705 for segmenting the detected surface, a humidity determination module 5305 for classifying the segments in terms of humidity on the surface, and a humidity evaluation module 5310 for evaluating the size of the classified segments of the surface. In addition, it can also comprise a map module 107 containing obstacles in the surroundings of the system and the segments classified in terms of humidity. In one aspect, the system comprises a motion planner 104 and / or a path planning module 103, such as an output unit (2 or 192) and an output content saved in the memory 10 for indicating the surface detected as wet. The system can refer to a service robot 17, such as an escort.

[0338] The system for detecting the location of humidity on a surface (such as a service robot 17, which in one aspect is an escort) comprises a unit for determination (such as a camera 185), a humidity determination module 5305 for classifying the detected and segmented surface in terms of humidity on the surface, and a humidity evaluation module 5310 for evaluating the size of the classified segments on the surface. Here, for example, the classified surface is analyzed such that the humidity width is evaluated slightly perpendicular to the motion direction of the system, and dry and / or wet areas are determined within the width range thereof. The system can have a motion planner 104 containing rules, for example, so as to be able to move through a dry area when the width thereof determined exceeds a value saved in the memory 10. The motion planner 104 can have rules for determining the minimum distance to a wet area, for example, by entering the area classified as wet into a map and comparing its own position with the map. The system has an output unit (2 or 192) and rules saved in the memory 10 for indicating that the detected area is wet and / or issuing a warning. The motion planner 104 can have rules for directing the system, for example, when the width of the wet area determined exceeds a certain threshold value or the width of the dry area determined is below a certain threshold value, the motion thereof towards the prescribed target direction of travel can be interrupted, similar to the rules of the prior art for mobile systems moving towards obstacles. In addition, the system also has a unit for sending messages to a server and / or terminal 13.

[0339] Method for classifying a fallen person

[0340] The service robot 17 has a fall recognition function in one aspect, i.e. the service robot 17 is configured such that the service robot 17 can detect a fall of a person directly or indirectly. In Figure 33 The analysis process 3300 for a fall event is shown in the middle. Indirectly means that the service robot 17 accesses external sensing means, while directly means that the analysis is carried out by the service robot's own sensing means.

[0341] In one aspect, the person is equipped with a sensor unit for fall detection, i.e. the service robot 17 is connected via an interface 188, such as a WLAN, to an external fall sensor which is located on the person 3310 to be monitored. The sensor unit comprises at least a control unit, a power supply, possibly a memory, an interface 188, such as a WLAN, and at least one inertial sensor 3315, such as an acceleration sensor, for detecting the motion of the person. In one aspect, the signals of the inertial sensor in the sensor unit are analyzed in step 3325, in one alternative aspect the signals are transmitted to the service robot 17 in step 3320, whereby the analysis of the signals 3330 can be implemented in the service robot 17. Next, the detected measurements are classified in step 3335, whether the person has fallen or not. The classification can be performed, for example, by measuring an acceleration which exceeds a defined threshold. Subsequently, a notification is issued in step 3345 based on the fall detection via the interface 188, such as a WLAN, i.e. for example a notification to an alarm system and / or triggering an alarm, such as an alarm sound. If the classification of the detected motion is performed within the sensor unit, the notification and / or the alarm is issued by the sensor unit (via the interface 188, such as a WLAN). If the service robot 17 performs the classification of the motion, the notification and / or the alarm is triggered.

[0342] The sensor unit is designed in one aspect such that the sensor unit detects the motion of the person for detecting the severity of the fall, for which measurements are taken and the measurements are classified directly within the sensor unit and / or by the service robot 17 in step 3340. This specifically refers to how far the person equipped with an acceleration sensor continues to move after the detection. For this purpose, the acceleration and / or orientation data of the sensor unit can be analyzed. For this purpose, rules are saved in the memory of the sensor unit and / or the service robot 17 which trigger different notifications based on the measured motion data. For example, when a fall is detected, the sensor unit can determine motion information which exceeds a defined threshold and / or classify them such that the person who has fallen can get up again, the notification and / or the alarm is modified, such as the priority of the notification is reduced. Conversely, if no further motion and / or change in position of the person who has fallen is detected by the sensor unit after the fall event, the notification and / or the alarm can be modified, such as the priority of the notification is increased. In one aspect, the notification or the alarm is only issued after the motion behavior of the person after the fall has been analyzed, i.e. if necessary after a few seconds of the actual fall, whereby the notifications related to the fall event can be reduced if necessary.

[0343] In one aspect, the service robot 17 is equipped with a wireless sensor unit in order to be able to determine a fall event of a person in step 3350. The sensor unit refers to a camera 185 or 3D camera, a radar sensor and / or an ultrasound sensor 194 or a combination of at least two sensors. Here, the sensor unit is used to identify the person's identity and / or to track the person in a time course in step 3355, for example by means of a visual person tracking module 112 and / or a laser-based person tracking module 113. Thereby, the service robot 17 can be equipped with a Kinect or Astra Orbbec, i.e. a RGB-D camera 185, which is able to create a skeleton model of the detected person in step 3360, in which the body joints are displayed as nodes and limbs connecting the nodes, for example as directional vectors, by means of methods described in the prior art, for example by camera SDFK, NUITrack, OpenPose, etc. Here, different orientations of the directional vectors are determined in the scope of feature extraction 3365, for example the distance of the nodes from the detected surface on which the detected person moves. In the scope of feature classification 3370, the service robot 17 analyzes whether the detected person is standing, walking, sitting or possibly falling. Here, the rules of the feature classification can be fixedly prescribed in one aspect, in an alternative and / or complementary aspect they can be learned by the service robot 17 itself. In the learning, photos of falling persons are taken, as well as photos of persons who do not fall, which are analyzed, wherein it is determined by tagging which case the photos belong to. On the basis of this, the service robot 17 is able to classify by means of machine learning / artificial intelligence methods, whereby in the future photos of persons can be classified as falling or not falling.

[0344] Here, fall detection is performed, for example, on the basis of the extraction of the following features in step 3367, in which the body parts are analyzed for a fall, i.e. classified: the distance from the ground and the distance change, or a joint acceleration derived from the distance or distance change in one direction (for example over a defined minimum holding period), the vertical component of which is greater than the horizontal component, wherein the vertical component preferably points towards the center of the earth. For example, if the distance of the hip joint from the ground is less than 20 cm, the distance change is for example greater than 70 cm to less than 20 cm, or the acceleration of the hip joint is likewise directed towards the ground, this is classified as a fall event, wherein the acceleration is for example below a defined duration (for example 2 seconds). As an alternative and / or additional solution, it is also possible to orient at least one directional vector in space (as a connection between two joints) or to classify a change in the directional vector in space as a fall event. This also applies, in particular, to a change in the orientation after a change in orientation (for example from a substantially vertical direction to a substantially horizontal direction), for example the orientation of the substantially horizontal spine and / or legs is counted as a fall event, which is optionally performed over a defined period of time. In an alternative and / or additional aspect, the height of the person can be determined by means of the 3D camera. If the height of the person is below a defined height, this is detected as a fall event. As an alternative and / or additional solution to the height of the person, it is also possible to determine the area occupied by the person on the ground. To this end, in one aspect the area is determined by means of the vertical projection of the person being tracked onto the ground.

[0345] After the occurrence of a classified fall event, the service robot 17 triggers a notification and / or an alarm in step 3345, for example by means of the interface 188 (such as WLAN), by means of the loudspeaker 192 as a sound alarm, etc.

[0346] For the detection of a person by the service robot 17 by means of radar and / or ultrasound 194, the external dimensions of the person are primarily detected, including the height thereof. If a height drop is detected here, or a height accelerated drop, this is classified as a fall event, if necessary in combination with a drop below a threshold value. As an alternative and / or additional solution, the service robot 17 can also classify the area occupied by the person on the ground (in the example the vertical projection).

[0347] The service robot 17 also detects the position of the head of the person in step 3369 in one aspect. The position is tracked in relation to the ground and / or the position of the detected obstacles. That is to say, for example, a detected wall is detected by means of the sensor unit (camera 185, radar and / or ultrasound sensor 194). As an alternative and / or additional solution, the position of the wall can also be determined by means of the laser radar 1. The service robot 17 compares the horizontal position of the head of the person with the (horizontal) position of the wall and / or other obstacles in space.

[0348] In an alternative and / or complementary aspect, also the vertical position is considered here. Thereby, for example, the camera 185 can also analyze the distance of the tracked head from objects, such as a table, in three-dimensional space. The laser radar 1 (two-dimensional, essentially horizontally aligned) for example can recognize the table legs, but not necessarily the position in space of the table top which the head can touch in case of a fall. In contrast, the camera-based analysis allows for a three-dimensional detection of the head and other obstacles in space and for determining the distance between the head and the other obstacles, which are analyzed in step 3374 in the context of the classification. If the service robot 17 detects a distance value between the head and one of the other obstacles which is below a value stored in the memory of the service robot 17, the value is modified in the memory of the service robot 17 and a separate notification or a separate alarm is triggered, if necessary.

[0349] The service robot 17 additionally tracks the person after the fall and detects to which extent he stands up again or attempts to stand up, i.e. fall after movement detection and classification in step 3340. The extent is for example the distance to the ground, the acceleration in the vertical direction opposite to the ground, the orientation body part or limb vector, the analysis of the height and / or (projected) area of the person. In an aspect, also the extent of the change in position of the joints is analyzed. Subsequently, the movement or even the extent of the attempt to get up of the person is classified. Thereby, the value in the memory of the service robot 17 is adjusted, in an aspect the extent of the notification is modified by the interface 188 (such as WLAN) / alarm. This actually means that different alarms and / or notifications can be made depending on the extent of the fall. It is also possible to combine the fall classification by sensors worn by the person and the sensor data analysis by the camera 185, radar and / or ultrasound 194 within the service robot 17. In summary, the evaluation procedure of a fall event is as follows: detection and tracking of the person's movement, detection of the fall event by feature extraction and classification of the orientation of the person's limbs and / or torso, detection and classification of the person's movement after the fall and evaluation of the severity of the fall event.

[0350] In an aspect, the service robot 17 can also detect vital parameters of the fallen person in step 3380 by its sensor means. As explained elsewhere, for this purpose integrated radar sensors can be used, such as ultra-wideband radar. Here, in an aspect the body parts of the person which are not covered by clothing can be detected by radar and / or camera-based methods and the pulse of the person can be measured in these areas, for example by radar. These information can be considered in the classification of the notification and / or alarm, in an aspect the vital parameters, such as the pulse, can be transmitted with the notification.

[0351] In Figure 70A system for fall classification is shown in detail. Thus, the system for detecting a fall of a person comprises first a service robot 17, a memory 10, at least one sensor capable of detecting the movement of a person non-contact in the course of time, a person identity recognition module 111 and a person tracking module 112 or 113, a fall detection module 5405 for extracting features from the sensor data and classifying the extracted features as a fall event, a fall event assessment module 5410 for classifying the severity of the fall event. For the purpose of transmitting messages, the system can additionally have an interface 188 to a server and / or a terminal 13. The fall detection module 5405 can have for example a skeleton creation module 5635 for creating a skeleton model of the person. The fall detection module 5405 can comprise a classification for determining the distance or the change of distance of the joints from the skeleton model to the ground, the acceleration of the joints in the vertical direction, the orientation of the direction vectors resulting from connecting at least two joints, the change of orientation of the direction vectors, the height and / or the change of height of the person, for example by means of a person size analysis module 5655, which determines the height of the person by means of the vector difference of two direction vectors, which extend from a common origin to at least one foot and at least the head of the person, the area on the ground occupied by the person projected along the vertical direction, and / or the position of the head of the person relative to looking at the ground and / or relative to looking at a detected obstacle. The system additionally has a vital sign parameter detection unit 5415 for detecting vital sign parameters of the person, for example a camera 185, a laser radar 1, a radar and / or an ultrasound sensor 194, and a vital sign parameter analysis module 5420 for analyzing the vital sign parameters of the detected person. The system has in one aspect a person recognition module 110, a motion analysis module 120 and / or a skeleton model based feature extraction module 5640.

[0352] Preventing a fall

[0353] The service robot 17 detects in one aspect vital sign parameters of the person during the execution of the test and / or the exercise, as in Figure 34The person is identified and tracked, step 3400, as described elsewhere herein. For this purpose, the service robot 17 identifies and tracks the person, e.g. by means of the visual person tracking module 112 and / or the laser-based person tracking module 113 in combination with the camera 185 or the laser radar 1. For this purpose, the person is identified and tracked, step 3355, for which the person identification module 111 can be used. The system is (optionally) positioned in front of the person, step 3420, and (optionally) moved in front of the person, step 3430. The body region of the person is identified and tracked, step 3440, for which the body region is required to perform the exercise and / or test, in order to detect vital sign parameters by measuring next to or on the body region, step 3450. It is considered to detect as body region e.g. the face, the hands and the chest region of the person. Corresponding methods of detecting such body regions are described elsewhere herein and / or in the prior art. The vital sign parameters detected comprise e.g. the pulse rate, the pulse rate variability, the systolic and diastolic blood pressure or yet the respiration of the person, such as the respiration rate. It is described elsewhere herein how the service robot 17 determines these exemplary mentioned vital sign parameters. Yet other methods can be used to determine the vital sign parameters. The vital sign parameters are detected using at least one sensor, such as the camera 185 and / or a radar sensor (such as a microwave pulse radar, a distance control radar, a Doppler radar, a continuous wave radar, an ultra-wideband radar) and / or a combination thereof, which detects the body region and the vital sign parameters of the mentioned person, preferably over time.

[0354] In one aspect, the motion of the person's skin and / or clothing above and / or below is determined using the sensors used herein. In an alternative and / or complementary aspect, the motion of the person's skin surface and / or clothing is analyzed with respect to the person's motion towards the service robot 17, i.e. the detected body region signal is corrected with the person's motion, step 3460. For this purpose, the service robot 17 detects at least one further body region in addition to the body region analyzed for determining the vital sign parameters, and determines the distance of the at least one further body region from the service robot 17. Herein, the motion detection for analyzing the vital sign parameters of the analyzed body region is synchronized with the motion detection for determining the relative motion of the detected person's body region. The body region can be detected for determining the relative motion e.g. by the laser radar 1, e.g. by the camera 185, e.g. the RGB-D camera 185, an ultrasound and / or radar sensor 194.

[0355] The measurements taken by the service robot 17 can be continuous or non-continuous measurements, i.e. for example every 10 seconds. The measured vital sign parameters are saved in the memory of the service robot 17 in step 3470 and can be transmitted to other systems via the interface 188, such as a WLAN. The service robot 17 compares the determined vital sign parameters with threshold values saved in the memory in step 3480. The values saved in the memory can be saved fixedly and / or derived dynamically from past values of the detected vital sign parameters, for example as an average of previously recorded values analyzed over a time interval. If it is identified in step 3490 that the detected vital sign parameters exceed or fall below the threshold values, the service robot 17 can modify the values in the memory, for example. Such a modification can at least trigger one of the following events: in step 3492 an output unit (display 2, loudspeaker 192, projection device 920, etc.) will be triggered, i.e. for example a voice output will be initiated. The service robot 17 can thus in one aspect request the person to reduce his speed. In an alternative and / or complementary aspect, the service robot 17 can request the person to sit down. In addition or independently thereof, the service robot 17 can maneuver a defined location, step 3498. Here, at least one seat can be involved, which is assigned coordinates in the map of the service robot 17. Subsequently, the service robot 17 can maneuver the seat. The seat can mean a chair. The service robot 17 can identify the chair directly in its surroundings by means of the implemented sensor devices (as described elsewhere herein), as an alternative and / or complementary solution, the service robot 17 can also save it on the map of the service robot 17 within the map module 107. Upon detecting a deviation of the vital sign parameters, the service robot 17 can trigger a notification in step 3494, i.e. for example send a notification and / or trigger an alarm via the interface 188, such as a WLAN. In addition, the service robot can in one aspect reduce its speed in step 3496.

[0356] In one application example, the service robot 17 accompanies a person in a walking exercise, for example by means of a lower arm support. Here, the service robot 17 moves in front of the person, who follows the service robot 17. The service robot 17 detects the person here by means of at least one sensor and performs feature extraction, feature classification and gait progression classification in order to analyze the gait progression of the person. Here, a camera 185 mounted on the service robot 17 detects the face and determines the systolic and diastolic blood pressure in the course of time, which are saved in a blood pressure memory in the service robot 17, respectively. The determined blood pressure measurements are analyzed in the course of time and optionally saved and compared with the values saved in the blood pressure memory. As an alternative and / or in addition, the measurements are compared with the measurements determined before a defined duration t, for example t = 10 seconds. If the systolic blood pressure drops by at least 20 mmHg and / or the diastolic blood pressure drops by more than 10 mmHg, which for example indicates an increased risk of falling, the blood pressure values are modified in the memory of the service robot 17. As a result, a voice output is made by the service robot 17, telling the person that he should reduce his walking speed. By reducing the speed, the risk of the person fainting and falling here is reduced. The service robot 17 reduces its speed and sends a notification to a server via an interface 188, such as WLAN, which in turn alerts the people in the surroundings of the service robot 17 and calls for rescue. The service robot 17 optionally detects a chair in its surroundings, i.e. within a defined distance relative to its position. If the service robot 17 detects a chair, the service robot 17 slowly navigates the person to the chair and requests the person to sit down by outputting.

[0357] In a similar example, the service robot 17 detects the respiratory frequency of the person in the course of time when completing the walking exercise by analyzing the movement of the chest and / or abdominal region of the person, which is achieved by means of an ultra-wideband radar sensor mounted on the service robot 17. The detected measurements are likewise saved in the memory and compared with the measurements saved in the memory. As an alternative and / or in addition, the measurements are compared with the measurements determined before a defined duration t, for example t = 10 seconds. If the amplitude of the change in respiratory frequency exceeds a certain threshold value, the steps already described in the previous section for the deviation of the blood pressure measurements are performed.

[0358] According to Figure 71The system for detecting vital sign parameters is described as follows: The system for detecting vital sign parameters of a person, such as a service robot 17, comprises a processing unit 9, a memory 10, at least one sensor capable of detecting the movement of a person non-contact in a time course, such as a camera 185, a laser radar 1, an ultrasound and / or radar sensor 194, a person identification module 111 and a person tracking module (112, 113) for detecting and tracking a person, a vital sign parameter analysis module 5420. In addition, the system comprises a body area detection module 4810 and a body area tracking module 4815 for tracking the detection area of a vital sign parameter and a vital sign parameter detection unit 5415 for detecting vital sign parameters of a person non-contact and / or contact-based in a time course. The vital sign parameter analysis module 5420 is capable of comparing the detected vital sign parameters with at least one threshold value saved, for example, and issuing a system notification via an interface 188, output via an output unit (2 or 192), modifying the speed change of the system (such as reducing the speed) and / or initiating a maneuver to the target position of the system. The latter is implemented, for example, by the navigation module (110), such as by adjusting the path planning to a seat, such as a chair, which is, for example, within a defined minimum distance of the system. The threshold value used in the vital sign parameter analysis module 5420 can be determined dynamically from the vital sign parameters detected previously, such as based on the average value of the vital sign parameters detected within a defined time interval. The vital sign parameter analysis module 5420 can also detect the body movement of a person and analyze the detected vital sign parameters by comparing the detected body movement. The detected vital sign parameters can be the pulse rate, the pulse rate variability, the systolic and diastolic blood pressure and / or the respiratory rate. In one aspect, the system has an interface 188 capable of detecting the data of a vital sign parameter sensor 5425 fixed on the body of a person and analyzing it in the vital sign parameter analysis module 5420. The application module 125 has rules for performing at least one exercise, such as the exercises saved as an example herein. In one aspect, the fall risk, such as a severe fall risk, can be determined by the vital sign parameters detected and analyzed, in which a fall is expected to occur within a time interval of only a few minutes. The system has a person recognition module 110, a person identification module 111, a tracking module (112, 113), a movement analysis module 120, a skeleton creation module 5635 and / or a feature extraction module based on a skeleton model 5640 in one aspect.

[0359] Recognizing an increased fall risk of a person

[0360] Older people generally have an increased risk of falling. In the prior art a number of factors have been examined which have a significant influence on the risk of falling. Espy et al. (2010) in the experimental paper "Independent effects of step speed and step length on stability and fall risk", Gait & Posture, July 2010, vol. 32(3), pp. 278-282, (DOI: 10.1016 / j.gaitpost.2010.06.013) describe, for example, that slow walking leads to an increased risk of falling, but that this risk can be reduced by shorter step lengths. Senden et al. "Accelerometer-based gait analysis, an additional objective method to filter subjects at risk of falling", Gait & Posture, June 2012, vol. 36(2), pp. 296-300, (DOI: 10.1016 / j.gaitpost.2012.03.015) also describe, by means of an acceleration sensor, that long steps and fast walking are associated with an increased risk of falling (determined by means of the standardized Tinetti test), which corresponds to the square root of the vertical acceleration of the body. A lower symmetry in the gait progression determined by means of an acceleration sensor predicts a preceding fall. Van Schooten et al. (2015) "Gait and fall risk analysis: quantity and quality of daily life walking predict falls in older adults", Journal of Gerontology: A Biological Sciences and Medical Sciences, vol. 70(5), May 2015, pp. 608-615, (DOI: 10.1093 / gerona / glu225) likewise use an acceleration sensor and show that an increased variance in the double step / walk cycle in the walking direction and a lower amplitude when walking in the vertical direction are associated with an increased risk of falling. Kasser et al. (2011) "Prospective analysis of balance, gait, and strength to predict falls in women with multiple sclerosis", Archives of Physical Medicine and Rehabilitation, vol. 92(11), November 2011, pp. 1840-1846, (published online 16 August 2011) (DOI: 10.1016 / j.apmr.2011.06.004) likewise report that an increase in the asymmetry in the gait progression is a significant predictor of the risk of falling.

[0361] In one aspect, the service robot 17 is configured such that it is able to analyze the risk of falling of a human gait progression, as in Figure 35In one (optional) aspect, the person is able to log in to the service robot 17 by means of an input unit, an RFID transponder, a barcode, etc., step 3510. Subsequently, the service robot 17 performs person identification by means of its person identification module and subsequently tracks the person, step 3355, for example by means of a visual person tracking module 112 and / or a laser-based person tracking module 113. For tracking, sensors are used which are able to detect the person non-contact, such as a camera 185, an ultrasonic sensor and / or a radar sensor 194. The service robot 17 requests the person whose fall risk is to be assessed to follow the service robot 17 by means of an output 3520 of an output unit. The service robot 17 (optionally) positions itself in front of the person in step 3420, (optionally) moves to the person in step 3525 and (optionally) detects its speed in step 3530. In one aspect, this is achieved by detecting the distance to the person identified by means of the synchronization and by determining the speed of the service robot 17, whereby the relative speed of the person to the service robot 17 can be determined and the speed of the person by the speed of the service robot 17 itself. The speed of the service robot 17 itself is determined by means of an odometer unit 181 of the service robot 17 and / or by tracking obstacles saved in a map of the service robot 17 and the relative movement of the service robot to these obstacles.

[0362] The service robot 17 performs feature extraction in step 3365 in order to extract features from the skeleton model in step 3360, such as the position of the joints 3541, the direction vectors 3542 connecting the joints to each other, the plumb line of the person, etc. As an alternative and / or in addition thereto, features can also be extracted from inertial sensors or the like fixed to at least one limb of the person, such as the instantaneous acceleration 3543, the direction of the acceleration 3544, etc. After the feature extraction 3365, a feature classification 3370 is performed, in which a plurality of features are evaluated in combination. One example of this is the speed of the person, which can be determined as a feature from the detected data of the service robot 17 as an alternative and / or in addition to the above-mentioned method, wherein the respective classified features can be, for example, the step length 3551 and / or the double step 3552 of the person detected by the service robot 17 with respect to time, and the speed is determined with respect to the step length within each detected time unit, wherein in one aspect a plurality of steps are analyzed. Here, the step length is extracted in one aspect by the position of the ankle joint in the skeleton model within the scope of the feature extraction 3365, wherein the skeleton model is created in step 3360 by analyzing the camera recordings of the person. When analyzing the data of inertial sensors, for example, installed on the foot or the shin, the time points and the time between the time points will be detected / extracted, i.e. the time during which the sensor starts a circular movement with its radius pointing towards the ground, i.e. the direction vector of the acceleration is analyzed for this, step 3544. The instantaneous acceleration is preferably determined or extracted in the sagittal plane in step 3543, and the distance covered within the scope of the feature classification is determined in step 3543 with respect to the instantaneous acceleration and the duration of time between the time points, which is then displayed in step 3551. Here, the extracted features are considered individually, each of which is referred to in such a combination classified. In one alternative and / or additional aspect, the ankle joint is detected by a radar sensor and / or an ultrasonic sensor 194. As an alternative and / or in addition thereto, the knee joint can be determined by the position of the knee joint, the direction vector from the knee joint to the parallel orientation of the shank, and the height of the knee joint above the ground in the case where the direction vector passes through the plumb line, wherein the height of the knee joint above the ground when the direction vector passes through the plumb line indicates the distance at which the ankle joint is observed from the knee joint. The double step 3552 mentioned is determined by the distance between the detected ankle joints, wherein in one aspect the individual step lengths 3551 are summed up.

[0363] In an alternative and / or complementary aspect, the service robot 17 analyzes the respective length of each double step and / or the duration of the single steps within the double steps and establishes a correlation of length and duration in step 3553. The service robot 17 adds the values detected for more than one double step in order to determine an average value for more than one double step. The service robot 17 analyzes the bending and / or stretching 3554, i.e. the angle of the thigh with respect to the vertical, in one aspect.

[0364] The service robot 17 then analyzes the speed, the step length and the double step length and the pace of the detected person. In an alternative and / or complementary aspect, the standing time of at least one foot, such as both feet, is also analyzed in step 3555, and the standing time is analyzed, for example, with respect to a plurality of steps, respectively. As an alternative and / or complementary solution, the track width can also be analyzed in step 3556, wherein the distance between the ankle joints is analyzed. Furthermore, the service robot 17 detects further joints in the skeletal model of the person, such as the head, shoulder joint, hip joint, etc., and detects their position in space, for example, three-dimensionally, and analyzes the parameters in the time progression. This analysis includes the size of the lift-off of these parameters in one aspect, however, also their movement in the sagittal plane, both horizontally and vertically. In one aspect, the acceleration of at least one of the mentioned points in the skeletal model is also analyzed for this purpose, step 3557.

[0365] The service robot 17 saves the detected values in step 3570, classifies the gait progression in step 3580 by means of the classified features, and compares it in step 3585 to gait progression classifications saved in the memory of the service robot 17 (or available through an interface 188, such as a WLAN). For this purpose, at least one of the mentioned classified features is analyzed, preferably a plurality of them are analyzed (jointly), and compared to the features in the memory. The service robot 17 determines a score on the basis of the comparison of the service robot 17, which score reflects the fall risk in step 3590, such as the probability of the detected person falling within a defined duration of time. In one aspect, the classification contains values for the determined speed, pace (steps per minute) and step length in relation to the person parameters, such as the person size. If the person has a walking speed of approximately 1 m / s, a pace of less than 103 steps per minute and a step length of less than 60 cm, it is an indication that this person has an increased risk of falling, for example. As an alternative and / or in addition, the detected accelerations in the three-dimensional space are analyzed, and the harmonics are formed by means of a discrete Fourier transform. Then, the ratio of the cumulative sum of the amplitudes of the odd harmonics to the cumulative sum of the amplitudes of the even harmonics is formed. Values of the vertical acceleration of less than 2.4, values of the acceleration in the walking direction in the sagittal plane of less than 3 and values of the lateral acceleration in the frontal plane of less than 1.8 are an indication of an increased risk of falling. The respective analysis results are analyzed in the walking feature classification module 5610. Here, a plurality of parameters, such as accelerations, step length, pace, etc., are analyzed simultaneously, for example.

[0366] In summary, a system, such as the service robot 17, for determining a score describing the fall risk of a person, such as the service robot 17, as in Figure 72The system for determining a score describing the fall risk of a person comprises a processing unit 9, a memory 10 and a sensor for detecting the motion of a person in a time course, including a gait course, such as a camera 185, a lidar 1, an ultrasound and / or radar sensor 194, a motion course extraction module 121 and a motion course evaluation module 122, which in one aspect is configured for determining a fall risk score within a fall risk determination module 5430, such as by analyzing accelerations in horizontal and / or vertical planes, strides, velocities and / or derived quantities therefrom, etc. The motion course extraction module 121 can be a walking feature extraction module 5605 for extracting gait course features, a motion course evaluation module 122, a walking feature classification module 5610 for classifying the gait course based on the extracted features, such as the joint nodes of a skeleton model of the detected person, directional vectors between the joint nodes of the skeleton model, accelerations or directional vectors of the joint nodes, positions of the joint nodes relative to each other in space and / or angles derived from the directional vectors, and a gait course classification module 5615 for classifying the gait course, such as including the length of the steps, the length of the double steps, the walking velocity, the relationship of the steps in the double steps, the bending and / or stretching, the standing duration, the stride and / or the distribution (position) and / or distance of the joint nodes relative to each other and / or the acceleration of the joint nodes, wherein the classification such as includes comparing the detected gait course to gait courses saved in the memory and determining a fall risk score. The gait course classification module 5615 can comprise a person velocity module 5625 for determining the velocity of the person, wherein the velocity of the person relative to the detection and analysis unit or the system is determined by the distance the person travels per time unit, wherein the odometry unit 181, the obstacles detected in the map and / or the relative position to the obstacles detected in the map are included. Furthermore, the system comprises a person identity recognition module 111 and a person tracking module (112 or 113) and components for the person to log in to the system (such as 2, 186), wherein for example visual features of the person are saved and used in the scope of the person identity re-recognition module (114). The system can acquire sensor data of an inertial sensor 5620 via the interface 188 and analyze these sensor data in the motion course extraction module 121. The sensor can be worn on the person, such as on the lower limbs, or on a walking aid used by the person, such as on a shoulder or lower arm brace, and detect the motion of the walking aid.

[0367] The system has in one aspect a person recognition module 110, a motion analysis module 120, a skeleton creation module 5635 and / or a feature extraction module based on a skeleton model 5640. From the flowchart it is shown that the process of determining a fall risk score in one aspect describing the concealment (non-seriousness) of a fall risk is as follows: detecting a person's gait progression (such as by the mentioned sensor non-contacting a person), extracting features of the detected gait progression, classifying the extracted gait progression features, comparing the classified at least two features of the gait progression with a gait progression classification saved in a memory and determining a fall risk score.

[0368] Mobility test (Tinetti test) carried out by the service robot

[0369] The service robot 17 is in one aspect configured such that the service robot 17 is able to analyze different body positions of a person when sitting, standing, walking and / or the motion of the person, as Figures 36-52 is shown, in order to have an overall view of the person's mobility. A series of method steps are embodied in a plurality of steps, which are therefore exemplary summarized in Figure 36 Here, in step 3525 the service robot 17 is in one aspect able to walk in front of the person when walking, in one alternative aspect behind the person. As described elsewhere herein, for this the person is able to log in the service robot 17 in step 3510, and in step 3355 a person identity recognition is carried out and a person tracking is carried out by the service robot 17, such as by means of a visual person tracking module 112 and / or a laser-based person tracking module 113 in combination with a laser radar 1 and / or a camera 185. In another aspect, the service robot 17 requests in step 3521 the person to perform a specific action, such as getting up, running, etc., wherein the output can be carried out by means of a display 2, a voice output, etc. This step 3521 is an optional step, or related to the respective analysis. In one aspect it is preferred in a time progression, analyzing why a defined time interval is used. For this, the service robot 17 uses in one aspect information from a skeleton model, which is created by means of a photo of the person in step 3360 by at least one 3D sensor and / or camera 185 and is able to be realized by means of an SDK from the state of the art. In step 3365 a feature extraction is carried out, which comprises for example in step 3541 the joint nodes, and in step 3542 the directional vectors between the joint nodes. Subsequently, in step 3370 a feature classification is carried out, which is in particular related to the analysis task. In step 3370 (optionally) the results of the feature classification are saved and a classification is carried out, which is again related to the task, so that this method step is in step 3700 in Figure 36The classification is referred to as continuation. Here, in one aspect a threshold comparison is made. Subsequently, a score is determined for each task. The detected data can be (temporarily) saved, such as data generated by the complete or partial analysis and / or classification.

[0370] In one aspect, when the feature classification is made in step 3370, the joint angles from the feature classification are analyzed synchronously in step 3365, but not determined in detail from the joint angles. Rather, a position can be determined based on the classifier in three-dimensional space (estimating the position in three-dimensional space based on two-dimensional data can be helpful), in which the body posture is recorded, which describes the posture identified as correct or incorrect, and the classifier is determined thereafter. Alternatively, the body posture is prescribed which describes the correct procedure, and analyzed for the position in time and the distribution of the joint angles. Here, the distribution of the joint angles can be analyzed, such as based on the demonstration of the body posture, respectively, and a classifier is created therefrom, which is then compared to the distribution of the detected, prescribed correct body postures and the distribution of the joint angles in space derived therefrom, wherein the classifier is then recreated, taking into account all existing joint angle distribution data. For this, the Dagger algorithm in Python can be used, for example. In this way, a classifier is created, such as by a neural network, which can recognize the correct movement, and then also the movement not done correctly. Here, the analyzed and classified body postures include (not finally) the sitting balance, getting up, attempting to get up, standing balance in different situations, starting to walk, gait symmetry, step continuity, course deviation, trunk stability, rotating 360°, autonomous sitting or putting down, using the lower arm support, etc., mentioned in the following sections.

[0371] Sitting balance

[0372] In the scope of the analysis, the service robot 17 detects the person and analyzes to what extent the person sitting is leaning to one side, sliding off the seat or sitting stably or rather stably. Here, the features of the skeleton model will be extracted, such as the joint angles of the knees, hips, shoulders, head, etc., and the orientation of the body parts / limbs of the person will be detected and analyzed using the directional vectors between the individual joint angles. In one aspect, the directional vector between at least one shoulder joint angle and at least one hip joint angle is analyzed (preferably at the position of one half of the body, respectively; and / or parallel to the spine), and its deviation from the perpendicular / plumb line is analyzed in Figure 37 step 3601.

[0373] In another aspect the orientation of the person is analyzed, i.e. in this case at least one direction vector between the shoulder joints, the hip joints, etc. and the knee, etc. is detected in step 3603. Here, preferably more than one direction vector is detected. This direction vector can be used for example to determine the frontal plane of the person in step 3602, which extends parallel to the direction vector. In another aspect the position of the hips in space is detected and in step 3604 a deviation over time in the transversal plane is analyzed. Furthermore, it is determined how much this person slides for example back and forth on his seat.

[0374] In the context of the sitting balance classification in step 3710, in step 3711 the deviation of at least one shoulder joint and at least one hip joint from the perpendicular line / plumb line and / or the direction vector inclination in the frontal plane is analyzed. Furthermore, in step 3712 the change (amplitude, frequency, etc.) of the position of the shoulder joints in the transversal plane is determined. In step 3713 a threshold value and / or a comparison with a modality, such as a motion modality, is made by means of these two steps 3711 and 3712. If at least one of the determined values is greater than the threshold value (such as 1.3 m), the measurement is classified as low sitting balance in step 3714, otherwise it is classified as high sitting balance in step 3715. For this purpose, in step 3716 a score is respectively specified which is saved in a sitting value memory.

[0375] Standing up

[0376] The service robot 17 in one aspect analyzes to which extent the person is able to get up (see also Figure 38 ) In the context of extracting features, the service robot 17 in step 3545 identifies objects and / or obstacles as described in the prior art. The service robot 17 here will extract for example a scatter plot in the vicinity of the tracked hand joints and segment the scatter plot, whereby it is possible to distinguish the hands of the objects. The segmentation is preferably performed in real time (for example at 30 fps). In one aspect, it is also possible to compare the detected scatter plot with scatter plots saved in a memory, for example to assign an object thereto, in order to establish a correlation between the objects detected in a sensoric manner and their symbolic meaning, whereby it is possible to classify certain objects as more relevant than others, whereby for example a chair with armrests or a walking aid is classified in relation to for example a vase.

[0377] In the context of the classification of features, standing is determined in step 3610. To this end, the distance between the head and the ground is measured in step 3611, for example on the basis of the position of the head joint and at least one ankle joint. In step 3614, these values are compared with values held in the memory, if necessary, and / or with threshold values and / or modalities. If the determined height is greater than a threshold value, for example, in this case 1.4 m, the person is classified as standing in step 3616, otherwise as sitting in step 3617. As an alternative and / or in addition to the height of the person, the orientation of the directional vectors between at least one ankle joint and at least one knee joint, between at least one knee joint and at least one hip joint and between at least one hip joint and at least one shoulder joint is also analyzed in step 3612, wherein the three directional vectors are substantially parallel to one another, as indicated by a threshold comparison 3615 and / or a modality comparison, for example, on the basis of which a threshold value is calculated as a deviation from parallel lines. As an alternative and / or in addition, the orientation of the directional vector between at least one knee joint and at least one hip joint can be analyzed next, determining to what extent this directional vector is substantially perpendicular. If the deviation from the parallel and / or perpendicular line is classified as being below a threshold value, the service robot 17 detects the features as standing in step 3616, otherwise as sitting in step 3617.

[0378] Furthermore, it is detected in step 3620 whether a hand is using an auxiliary tool, wherein an auxiliary tool is understood broadly here as a walking aid, armrest of a chair, a wall, etc., i.e. all tools that a person can use to help stand up when getting up. The distance between at least one hand joint and at least one of the objects retrieved is determined in step 3621. If at least one hand is at a distance of less than a certain threshold value 3622, for example 8 cm, from one or more objects or obstacles, it is classified as using an auxiliary tool in step 3623, otherwise as not using an auxiliary tool in step 3624. Here, in one aspect, the minimum distance to the body of the person being observed, i.e. to the directional vectors of the joints and / or the interconnecting joints, is presupposed.

[0379] Within the scope of the getting-up classification 3720, the service robot 17 is able to classify in the following way: If the person stands up 3721 after a defined time, or an input 3722 by the person, in particular an input that the person is unable to get up (independently), this case is classified in step 3723 as a case in which the person needs help. If the person stands up within a defined time in step 3724 and the person uses an aid in step 3623, the person is classified in step 3725 as a person who needs an aid to get up 3725. A third case in this classification is that the person does not need an aid in step 3624 and is able to achieve standing up in step 3724 within a defined duration, whereby the person is able to get up without an aid in step 3726. A getting-up score 3727 is determined on the basis of steps 3723, 3725 and 3726.

[0380] Attempt to stand up

[0381] In a variant for detecting getting up in the past, the getting-up attempt is determined as an alternative and / or complementary solution. Figure 39 Therefore, within the scope of the feature classification, as a complement to the feature classification in Figure 38 the knee-hip direction vector, i.e. to what extent the service robot 17 is parallel to the transversal plane, is analysed.

[0382] Within the scope of the getting-up attempt feature classification 3730, the following steps are completed: If no standing up is detected in step 3731 within a defined time on the basis of the getting-up attempt feature classification information 3370, or an input is made by the person in step 3732 (compared to steps 3731 and 3732), no aid is detected in step 3624, the person is classified in step 3733 as a person who is unable to get up without help. If no aid is detected in step 3624 and the local maximum is not equal to the global maximum and the number of local maxima is greater than 1, a multiple getting-up attempt is detected in step 3735. To this end, the distribution of the joint defining the standing up is analysed in the time course and / or the angle or the angle change of the direction vector between the hip and the knee is analysed with the horizontal line (as an alternative: the perpendicular line) as a reference, which describes the horizontal line on the transversal plane. If, for example, an angle change from about 0° (transversal plane) to about 30° (change in the direction of rotation) is detected twice, but then a change in the other direction occurs (for example, back to 30°) and then an angle change >> 30°, for example 90°, is detected, a three-time getting-up attempt is detected (with the last one being successful). In contrast, if no aid is detected in step 3624, the standing up 3616 and the case in step 3736 is classified as a case in which the person does not need an aid. A getting-up attempt score 3737 is given overall on the basis of steps 3733, 3735 and 3736.

[0383] Standing balance

[0384] In an alternative and / or complementary aspect, the service robot 17 analyzes a person's standing balance, such as Figure 40 As shown. Complementing the previous analysis, balance determination 3630 is performed in feature classification 3370. To this end, the amplitude, orientation, and / or frequency of positional changes of at least one shoulder joint, at least one hip joint, or at least one ankle joint in the lateral plane 3631 are analyzed over a time period (e.g., 5 seconds), and a threshold comparison and / or comparison with a modality, such as a motion modality, is performed in step 3632. In this context, stride length and / or the presence of a step can also be analyzed based on the ankle joint. If the frequency of amplitude, orientation, and / or positional changes is below the threshold 3632 (e.g., 10 cm of lateral fluctuation) and / or does not conform to the modality, stability is assumed 3635; otherwise, instability is assumed 3636. As an alternative and / or supplementary approach, the amplitude, orientation, and / or frequency of at least one direction vector of deviation from the vertical and / or sagittal plane and / or frontal plane (foot, knee, or hip and at least one joint located above) can be analyzed in steps 3633 and 3631 over a time period (e.g., 5 seconds). In addition to at least one shoulder joint, the upper joint also includes a head joint. Based on the threshold comparison in step 3634, deviations below the threshold and / or modality are defined as stable 3635, otherwise as unstable 3636. If a person stands 3616 but is unstable 3636, then in step 3740, the standing balance is classified as unstable 3741. If a person stands 3616, uses an assistive device 3623, and is in a stable balance state 3635, then the person is assigned the classification "Standing stably with assistive device". If a person stands 3616, does not use an assistive device 3624, and is stable 3635, then it is assumed that they can stand stably without using an assistive device 3743. A standing balance score 3744 is given based on this classification.

[0385] Standing balance with feet close together

[0386] As an alternative and / or supplement to previous standing balance analysis (see Figure 41), preferably after output 3521 by the service robot 17, the person's feet are requested to be close together while standing, and the foot distance 3640 is determined within the scope of the feature classification 3370. To this end, the ankle joint and / or the knee joint are used in accordance with the position of the extracted joint nodes 3541, in one aspect also the hip joint and the knee joint and / or the orientation of the directional vector 3542 between the knee and ankle joint. On the basis of these data, the distance 3641 of the ankle joint is determined, in one aspect within the frontal plane. Subsequently, a threshold comparison 3642 and / or a modal comparison classifies whether the feet are far apart 3643 or close together (i.e. the distance 3643 is small), wherein for example a threshold of 12 cm (from one joint center to the other) can be employed.

[0387] The standing is classified into three classes upon the subsequent standing balance foot distance classification 3745: in a first class (standing instability 3746) the person is grouped as standing 3616 and unstable balance 3636. In a second class, the standing person 3616 is classified as standing 3616, standing balance 3635, using an aid 3623 or foot distance wide 3644. The standing person 3616 is classified in a third class, in which he is in stable balance 3635, does not use an aid 3624 and has a foot distance low 3643. This classification is embodied in the standing balance-foot distance score 3749.

[0388] In one aspect, the ankle joint cannot be directly obtained from the data of the SDK that extracts the skeleton model if necessary, but instead the ankle joint is obtained via the knee joint. Here, the position of the knee joint can be determined by the directional vector from the knee joint parallel to the shank and the height of the knee joint above the ground when the directional vector (between the hip joint and the knee joint or between the knee joint and th...

Claims

1. A computer-implemented method for classifying rotational movements of a person, comprising: • detecting a person over a time course; • creating a skeletal model of the detected person; • extracting features of the joints of the skeletal model and / or of the direction vectors between the joints of the person; • feature classification, the feature classification comprising determining rotational movements of the direction vectors over the time course.

2. The computer-implemented method according to claim 1, further comprising: • determining an angle of rotation from at least one rotational movement of the direction vectors; • accumulating the angles and / or the angle changes; and • comparing the sum to a threshold value and / or to a modality. The rotational movements are determined by the angle changes of at least one direction vector between two joints of the person over the time course, which is projected into a transversal plane, and / or wherein the rotational movements are determined by the angle changes of at least one direction vector, which connects a shoulder joint, a hip joint, a knee joint, an ankle joint, an arm joint.

3. The computer-implemented method of claim 1, wherein, 4. The computer-implemented method according to claim 1, further comprising analyzing the balance of the person. The balance is determined by analyzing the amplitude, the orientation and / or the frequency of the position changes of the joints in a transversal plane and comparing them to threshold values and / or modalities saved in a memory (10), or wherein the balance is determined on the basis of the deviation of the direction vectors between the joints of the person from the plumb line of the person.

5. The computer-implemented method of claim 1, wherein, A processing unit (9), a memory (10) and at least one sensor for detecting the movements of the person over a time course, a skeletal model based feature extraction module (5640) for extracting features based on the joints and / or the direction vectors between the joints of the person and / or the direction vectors between at least two joints, and a rotational movement feature classification module (5680) for the feature classification of rotational movements.

6. A system for classifying human rotational movements, comprising: The rotational movement feature classification module (5680) comprises an angle analysis module (5682) for accumulating the detected angles and / or for analyzing the angle changes.

7. The system of claim 6, wherein, 8. A computer-implemented method for determining the balance of a person, comprising: • detecting a person non-contact over a time course; • creating a skeletal model of the detected person; • extracting features of the joints of the skeletal model and / or of the direction vectors between the joints; • analyzing the amplitude, the orientation and / or the frequency of the position changes of the joints in a transversal plane. The stride of the detected person is determined in the frontal plane over the time course with respect to the distance of the ankle joint, wherein the balance is determined when the stride has fallen below a threshold value.

9. The computer-implemented method of claim 8, further comprising:

10. The computer-implemented method according to claim 8, comprising: • detecting objects in the surrounding of the person; • detecting the position of the person and / or the position of at least one hand joint of the person; • determining the distance between the at least one hand joint and at least one object in the surrounding of the person; • modifying a value in a memory (10) when the distance falls below a threshold value. The determination of the balance is performed for a person standing, sitting or walking.

11. The computer-implemented method of claim 8, wherein, 11. A computer program product for classifying rotational movements of a person, comprising program code means for causing a computer to carry out the steps of the computer-implemented method according to claim 1 when said computer program product is carried out on a computer.

12. A system for determining the balance of a person, having a sensor for detecting the person non-contact in a time course, a skeleton creation module (5635) for creating a skeleton model of the person, a skeleton model based feature extraction module (5640) for extracting features based on the joint nodes and / or directional vectors between the joint nodes of the person, a transverse joint analysis module (5645) for analyzing the position changes of the joint nodes in the transverse plane for their amplitude, orientation and / or frequency and comparing the detected values with threshold values and / or modalities saved in a memory (10).

13. The system of claim 12, comprising: a plumb line joint analysis module (5650) for determining the deviation of the directional vectors from the plumb line of the person using threshold values and / or modalities saved in a memory (10), a stride-step module (5675) for determining the stride and / or step of the person by the distance in the frontal plane in a time course with respect to the ankle joint when the stride has fallen below a threshold value, a person height analysis module (5655) for analyzing the height of the person, a hand distance analysis module (5660) for analyzing the distance between the hand joint and other objects in the surrounding environment of the person, and also comprising rules for threshold comparison of the determined distances with distance threshold values saved in a memory.

14. A computer-implemented method for determining a score describing the fall risk of a person, comprising: • detecting the gait course of a person; • extracting features of the detected gait course; • classifying the extracted gait course features; • comparing at least two of the classified features of the gait course with gait course classifications saved in a memory; and • determining a fall risk score.

15. The computer-implemented method of claim 14, wherein, In the gait classification, the velocity, step length, pace and / or acceleration of the person in the horizontal and / or vertical plane are jointly analyzed.

16. The computer-implemented method according to claim 14, further comprising: • the person logs in on a detection and analysis unit, which detects and analyzes the gait course of the person; • the identity of the person is recognized by an optical sensor; • the identity recognition features of the person are saved; and • the person is tracked over time.

17. A system for determining a score describing a fall risk of a person, comprising: a processing unit (9), a memory (10), a sensor for detecting the movement of a person over time, a movement course extraction module (121) and a movement course evaluation module (122), which contains a fall risk determination module (5430) for determining a fall risk score.

18. The system of claim 17, wherein, The movement course evaluation module (122) comprises a gait course classification module (5615) for gait course classification, which comprises: step length, length of double steps, walking speed, relationship of step length in double steps, bending and / or stretching, standing duration, stride and / or distribution (position) of joint nodes from each other and / or distance and / or acceleration of joint nodes.

19. The system of claim 17, comprising: a person identification module (111) and a person tracking module (112 or 113) and a component (2, 186) that logs the person on the system, and / or wherein The movement process evaluation module (122) comprises a person speed module (5625) for determining the speed of the person.

20. A computer-implemented method for predicting a movement process, comprising: • merging person data, exercise plan configurations, movement process corrections and classified movement processes of different persons over time in at least one memory (10); • predicting a movement process on the basis of the exercise plan configurations, the person data and / or the movement process corrections; • determining exercise plan configurations and / or movement process corrections that will lead to the occurrence of a defined movement process; • transmitting the exercise plan configurations and / or movement process corrections to a system for detecting movement processes.

21. The computer-implemented method of claim 14, further comprising: detecting a person, creating a skeleton model of the person, extracting joints of the person within a movement process, classifying the extracted joints for movement processes in order to evaluate the movement process and determine movement process corrections.

22. The computer-implemented method according to claim 14, further comprising outputting instructions on the basis of an exercise plan.

23. A computer-implemented method for detecting a fall of a person, comprising: • detecting and tracking movements of a person; • detecting a fall event by feature extraction and classification of the orientation of the limbs and / or torso of the person and / or its height; • detecting and classifying movements of the person after a fall; and • evaluating the severity of the fall event.

24. The computer-implemented method according to claim 23, further comprising detecting at least one vital sign parameter of the person.

25. A system for detecting a fall by a person, comprising: a memory (10), at least one sensor for detecting movements of a person over time, a person identification module (111), a person tracking module (112 or 113), a fall detection module (5405) for extracting features from sensor data and classifying the extracted features as a fall event, and a fall event evaluation module (5410) for classifying the severity of the fall event.

26. The system of claim 25, comprising: a vital sign parameter detection unit (5415) for detecting vital sign parameters of the person and a vital sign parameter analysis module (5420) for analyzing the detected vital sign parameters of the person.

27. A computer-implemented method for detecting vital sign parameters of a person, comprising: • detecting and tracking a person; • detecting and tracking a body region of the person on which or through which vital sign parameters are acquired; • detecting vital sign parameters; • comparing the detected vital sign parameters with at least one threshold value saved; and • triggering an event if the threshold value is exceeded or undershot.

28. The computer-implemented method of 27, wherein, The event comprises reducing the speed or maneuvering to a target position.

29. The computer-implemented method according to 27, further comprising determining a fall risk on the basis of the detected vital sign parameters.

30. The computer-implemented method of 27, wherein, The vital sign parameters are detected during the execution of a test and / or during accompanying the person during an exercise.

31. A system for detecting a vital sign parameter of a person, comprising: The processing unit (9), the memory (10), at least one sensor for contactlessly detecting a motion of a person in a time course, a body region detection module (4810), a body region tracking module (4815) for tracking a detection area for vital parameter, a vital parameter detection unit (5415) for detecting vital parameters of the person, and a vital parameter analysis module (5420).

32. The system of claim 31, wherein, The vital parameter analysis module (5420) issues a system notification via an interface (188), outputs via an output unit (2 or 192), changes a speed of the system and / or initiates a maneuver to a target position of the system.

33. The system of claim 31, comprising: An application module (125) with rules for performing at least one exercise.

34. A computer-implemented method for determining cardiovascular parameters, comprising: • detecting and tracking a face of a person; • selecting a candidate region within the face; • detecting and analyzing a motion within the candidate region of the face caused by cardiovascular activity.

35. The computer-implemented method of claim 34, wherein, The motion comprises a blood flow in an artery, or a motion of a face surface and / or a head.

36. The computer-implemented method according to claim 34, further comprising: • illuminating the face; and • detecting the face in a frontal view.

37. The computer-implemented method of claim 34, wherein, Analyzing the motion comprises classifying the motion in order to determine a systolic and diastolic blood pressure, a pulse rate, a pulse rate variability, a pulse wave propagation time, or a pulse wave form.

38. The computer-implemented method according to claim 34, further comprising: • determining an orientation of the face in space; and • minimizing a detection angle of the face according to an axis perpendicular to an axis of a sensor used for detecting the face and an axis perpendicular to a sagittal plane of the face.

39. A system for determining a cardiovascular parameter of a person, comprising: The processing unit (9), the memory (10), and the camera (185) further comprise a body region detection module (4810), a body region tracking module (4815), and a cardiovascular activity module (5110) for detecting a motion caused by cardiovascular activity.

40. The system of claim 39, further comprising: A face recognition module (5005), a face candidate region module (5010), a light (5120) for illuminating the face during recording by the camera (185), and / or a blood pressure determination module (5125) for determining a systolic and diastolic blood pressure.

41. A computer-implemented method for predicting postoperative dementia / delirium in an elderly person, comprising: • detecting a person in a time course; • determining health status data of the person based on detecting the person in a time course; • acquiring preoperative data of the person; • acquiring intervention data of the person; • determining an influence of the preoperative data and the intervention data on the health status data of the person by calculating a weight estimate of parameters of the preoperative data and the intervention data.

42. The computer-implemented method of claim 34, further comprising: Predicting a health status of the detected person based on the weight estimate and newly acquired preoperative data and intervention data of the person.

43. The computer-implemented method of claim 34, wherein, The detecting a person is automatic.

44. A computer-implemented method for detecting and analyzing a folding process, comprising: • detecting, recognizing, and tracking at least one hand of a person; • detecting, recognizing and tracking paper; • jointly classifying the size, shape and / or motion of the detected paper and hand elements as a folding process.

45. The computer-implemented method of claim 1, wherein, The classification of the folding process comprises detecting a change in shape of the paper.

46. The computer-implemented method of claim 1, comprising: Recognizing and tracking at least one corner and / or edge of the paper.

47. The computer-implemented method of claim 1, wherein, The classification of the folding process comprises detecting a curvature of the paper that exceeds a threshold and / or that has a lowest modal similarity.

48. The computer-implemented method of claim 1, wherein, The classification of the folding process comprises detecting a magnitude of size reduction of the paper in a time course.

49. A system comprising: A processing unit (9), a memory (10) and a sensor for contactlessly detecting a motion of a person, in the memory (10) comprising a paper detection module (4705) for detecting paper and a folding motion determination module (4710) for determining a folding motion of the paper.

50. The system of claim 49, wherein, The folding motion determination module (4710) comprises a paper distance-corner edge module (4720) for detecting a distance of an edge and / or corner of the paper, a paper shape change module (4725), a paper bending module (4730), a paper size module (4740) and / or a paper edge orientation module (4745).

51. A computer-implemented method for comparing a detected signal of a tactile sensor (4905) with a sequence of output sound signals, comprising: • outputting a sequence of pulsed sound signals; • detecting a signal by a tactile sensor (4905); • comparing the sequence of output sound signals with the signal detected by the tactile sensor (4905).

52. The computer-implemented method according to claim 51, comprising: • assigning a value to each output sound signal; • adjusting the value after the detection of the signal according to a defined value.

53. The computer-implemented method according to claim 52, creating a diagnosis based on the adjusted value.

54. A system comprising: A processing unit (9), a memory (10), a sound signal output unit (192), a tactile sensor (4905) and a tactile sensor analysis unit (4910) for analyzing the signal of the tactile sensor (4905) and a tactile sensor output comparison module (4915) for comparing whether the detected signal occurs after the output sound signal.

55. The system of claim 54, further comprising: A cognitive ability assessment module (4845) for assessing a cognitive ability of a person.

56. An apparatus for performing the method of any one of claims 1, 8, 14, 20, 23, 27, 34, 41, 44 or 51.

57. A service robot (17) in conjunction with the method of any one of claims 1, 51, 34, 34, 42, 27, 28, 19, 1 or 14 or the system of any one of claims 6, 12, 17, 25, 32, 39, 49 or 54.

Images