Information processing device, information processing system, program, and recording medium

Abstract

The present disclosure relates to derivation of an index for a subject to whom a care necessity degree or a support necessity degree is certified. According to the present invention, a control device (101) of a sensor box (100) acquires, from sensors, detection outputs indicating movements of the subject. The control device (101) extracts a portion corresponding to a subject's unconscious movement from the acquired detection outputs. The control device (101) derives an index for the certification of the care necessity degree by using the extracted portion. The control device (101) also outputs the derived index.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, and more particularly to an information processing device that outputs ADL (activities of daily living) indices for nursing care level certification of a subject of a sensor that acquires detection output. [Background technology]

[0002] Conventionally, information processing technology has also been used in nursing care sites. For example, JP 2005-237479 A (Patent Document 1) discloses a technology for detecting a patient's turning over in bed using a biological signal such as a respiratory signal. JP 2018-507008 A (Patent Document 2) discloses a technology for monitoring the ADL of elderly people, such as medication, using a PIR (passive infrared) sensor. JP 2019-204419 A (Patent Document 3) discloses a technology for deriving a user's level of care requirement based on the analysis results of information from a group of sensors that detect the user's behavior and the analysis results of an electronic medical record. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2005-237479 A [Patent Document 2] Special Publication No. 2018-507008 [Patent Document 3] JP 2019-204419 A Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, ADL-related indices (hereinafter also referred to as "ADL indices") have been used for the certification of the level of care required. The level of care required should be certified based on the actual condition of the care recipient. However, the care recipient may behave in a manner that makes the condition of the care recipient appear worse than the actual condition when deriving the ADL indices, with the intention of being certified as having a higher level of care required. In addition, the care recipient may behave in a manner that makes the condition of the care recipient appear better than the actual condition when deriving the ADL indices, with the intention of being certified as having a lower level of care required. In consideration of these cases, a technology is needed for deriving an ADL indices based on the actual condition of the care recipient, regardless of the intention of the care recipient.

[0005] The present disclosure has been devised in consideration of the above-mentioned situation, and its purpose is to provide technology for deriving and outputting ADL indices for individuals who are to be certified as requiring nursing care or assistance based on the actual condition of the individual being cared for. [Means for solving the problem]

[0006] According to one aspect of the present disclosure, an information processing device is provided that includes an interface for acquiring detection output from a sensor representing the movements of a subject for nursing care level or assistance level certification, and an analysis unit that outputs information related to the nursing care level or assistance level certification by analyzing at least a portion of the detection output that corresponds to the subject's unconscious movements.

[0007] According to another aspect of the present disclosure, there is provided an information processing system comprising a sensor that acquires the motion state of a subject, the information processing device, and an output device that displays information relating to the level of care required or the level of assistance required output from the information processing device.

[0008] According to another aspect of the present disclosure, a program is provided that, when executed by a processor of a computer, causes the computer to perform the steps of acquiring detection output from a sensor representing the movement of a subject for nursing care level or assistance level certification, and outputting information relating to the nursing care level or assistance level certification by analyzing at least a portion of the detection output that corresponds to the subject's unconscious movement.

[0009] According to another aspect of the present disclosure, there is provided a computer-readable recording medium non-temporarily storing a program, the program being executed by a processor of a computer to cause the computer to execute the steps of acquiring detection output from a sensor, the detection output representing a movement of a subject of certification of a level of nursing care or a level of support required, and outputting information relating to the certification of a level of nursing care or a level of support required by analyzing at least a portion of the detection output corresponding to an unconscious movement of the subject. Effect of the Invention

[0010] According to the present disclosure, an information processing device outputs information related to the certification of the level of care required or the level of support required by analyzing a portion of the detection output from a sensor that corresponds to an unconscious movement of the subject of the certification of the level of care required or the level of support required. In this way, information related to the certification of the level of care required or the level of support required is derived and output based on the actual state of the subject of the certification of the level of care required or the level of support required. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a monitoring system. [Diagram 2] 13 is a diagram for explaining an example of the detection range of a sensor in a living room 900. FIG. [Diagram 3] 1 is a diagram showing the hardware configuration of a sensor box 100, a management server 200, and a mobile terminal 300 of the monitoring system. [Figure 4] FIG. 4 is a diagram illustrating a hardware configuration of a cloud server 400. [Diagram 5] FIG. 13 is a diagram showing an example of an installation mode of pressure sensors in each room 900. [Figure 6] 13 is a flowchart of an example of processing in the control device 101 for deriving an index for the item "turning over motion." [Figure 7] FIG. 11 is a diagram showing an example of skeletal information used to detect a turning motion. [Figure 8] 11 is a flowchart of an example of a process executed by the control device 101 for deriving an ADL index. [Figure 9] FIG. 1 is a diagram showing the procedure for transferring from a bed to a wheelchair. [Figure 10] FIG. 10 shows the trunk rotation angles identified in each of the four images in FIG. [Figure 11] 11A and 11B are diagrams illustrating an example of a manner in which the rotation angle of the trunk changes. [Figure 12] 13 is a flowchart of another example of the process executed by the control device 101 for deriving an ADL index. [Figure 13] 13 is a diagram showing an example of changes in the detection values ​​of a measurement sheet 560A and a measurement sheet 560B. FIG. [Figure 14] 11 is a diagram showing an example of skeletal information used to detect a standing up motion; FIG. [Figure 15] 11 is a diagram showing an example of skeletal information used to detect a walking motion; FIG. [Figure 16] FIG. 1 illustrates an example of a machine learning model. [Figure 17] FIG. 15 is a diagram for explaining machine learning by a learning device 1502. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of an information processing device will be described with reference to the drawings. In the following description, the same parts and components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, the description thereof will not be repeated.

[0013] [Configuration of monitoring system] FIG. 1 is a diagram showing an example of the configuration of a monitoring system. As an example of a monitored subject, the residents in each room provided in a room area RM of a facility are adopted. In the monitoring system of FIG. 1, rooms 900A and 900B are provided in the room area RM. Room 900A is assigned to resident 800A. Room 900B is assigned to resident 800B. In the example of FIG. 1, the number of rooms included in the monitoring system is two, but the number is not limited to this.

[0014] In the monitoring system, a sensor box 100A installed in a room 900A, a sensor box 100B installed in a room 900B, a management server 200 installed in a management center ST, and an access point AP are connected via a network NT.

[0015] In the monitoring system, a mobile terminal 300A carried by a staff member NA and a mobile terminal 300B carried by a staff member NB are connected to a network NT via an access point AP. Furthermore, the sensor boxes 100A and 100B, the management server 200, and the access point AP can communicate with a cloud server 400 via the network NT.

[0016] The living room 900A includes, as equipment, a bed 901A, a toilet 902A, and furniture 903A. A door sensor 510A is installed on the door of the living room 900A to detect whether the door is open or closed. A toilet sensor 520A is installed on the door of the toilet 902A to detect whether the toilet 902A is open or closed. An odor sensor 530A is installed on the bed 901A to obtain excretion information of the resident 800A. The resident 800A wears a vital sensor 540A that detects vital information of the resident 800A. An example of the detected vital information is the resident's body temperature. Another example is the resident's breathing. Yet another example is the resident's heart rate. Yet another example is information that includes two or more types of these types of information. In the living room 900A, the resident 800A can operate the care call handset 500A.

[0017] In this specification, when referring to matters common to the multiple sensor boxes 100A and 100B, the term "sensor box 100" is used to collectively refer to the sensor boxes 100A and 100B. Similarly, the terms "resident 800," "room 900," "toilet 902," "furniture 903," "care call handset 500," "door sensor 510," "toilet sensor 520," "odor sensor 530," and "vital sensor 540" are used for the residents 800A and 800B, the rooms 900A and 900B, the beds 901A and 901B, the toilets 902A and 902B, the furniture 903A and 903B, the care call handset 500A and 500B, the door sensors 510A and 510B, the toilet sensors 520A and 520B, the odor sensors 530A and 530B, and the vital sensors 540A and 540B, respectively.

[0018] Furthermore, a microphone (microphone 550 in FIG. 3) for outputting audio information of each room to an external device such as management server 200 may be installed in each room.

[0019] The sensor box 100A has a built-in sensor for detecting the behavior of an object in the living room 900A. One example of a sensor is a Doppler sensor for detecting the motion of an object. Another example is a camera. Further examples are a care call handset 500, a door sensor 510, a toilet sensor 520, an odor sensor 530, and a vital sensor 540. The sensor box 100A includes at least one of these sensors as a sensor.

[0020] 2 is a diagram for explaining an example of a detection range of a sensor in a living room 900. In the example of FIG.

[0021] The range AR generally represents the detection range of the sensor. If the sensor is a Doppler sensor, it detects activities that occur within the range AR. If the sensor is a camera, it takes images within the range AR.

[0022] 1, a management server 200 installed in a management center ST is connected to a display 206 and an input device 209. The input device 209 is, for example, a keyboard.

[0023] 1, the number of mobile terminals connected to the network NT via the access point AP is two (mobile terminals 300A and 300B), but the number is not limited to this. In this specification, when referring to matters common to the mobile terminals 300A and 300B, the term "mobile terminal 300" is used to collectively refer to the mobile terminals 300A and 300B.

[0024] The communication between the elements in the monitoring system of FIG. 1 may be wired or wireless.

[0025] The sensor box 100 includes a camera 105 and an interface 105A for inputting data from the camera 105 to the control device 101. The sensor box 100 also includes a Doppler sensor 106 and an interface 106A for inputting data from the Doppler sensor 106 to the control device 101. Each of the interfaces 105A and 106A is formed, for example, by a circuit for controlling input and output of data.

[0026] It should be noted that the sensor box 100 does not necessarily include the camera 105 and / or the Doppler sensor 106. The camera 105 and / or the Doppler sensor 106 may be provided outside the sensor box 100, similar to the door gate sensor 510 and the like. The control device 101 may acquire the detection output of the camera 105 and / or the Doppler sensor 106 provided outside the sensor box 100 via a given interface.

[0027] The vital sensor 540 does not necessarily have to be attached to the resident. In one example, the vital sensor 540 is realized by an infrared sensor or the like installed at a distance from the resident, and detects the temperature of the resident to output vital information (body temperature). In another example, the vital sensor 540 is realized by a Doppler sensor 106. The Doppler sensor 106 irradiates microwaves toward the resident. The microwaves cause a Doppler effect due to slight displacement of the resident's chest caused by the beating of the heart, causing the frequency to fluctuate. The Doppler sensor 106 detects the heartbeat of the resident based on the frequency fluctuation. In yet another example, the Doppler sensor 106 detects the breathing of the resident.

[0028] [Configuration of each element of the monitoring system] Fig. 3 is a diagram showing the hardware configuration of the sensor box 100, management server 200, and mobile terminal 300 of the monitoring system. Fig. 4 is a diagram showing the hardware configuration of the cloud server 400. Hereinafter, an example of the configuration of each device in the monitoring system will be described with reference to Figs. 3 and 4.

[0029] (Sensor Box 100) The sensor box 100 includes a control device 101 , a ROM (Read Only Memory) 102 , a RAM (Random Access Memory) 103 , a communication interface 104 , a camera 105 , a Doppler sensor 106 , a wireless communication device 107 , and a storage device 120 .

[0030] The control device 101 controls the sensor box 100. The control device 101 is configured, for example, by at least one integrated circuit. The integrated circuit is configured, for example, by at least one central processing unit (CPU), at least one application specific integrated circuit (ASIC), at least one field programmable gate array (FPGA), or a combination of these.

[0031] An antenna (not shown) and other elements are connected to the communication interface 104. The sensor box 100 exchanges data with external communication devices via the antenna. The external communication devices include, for example, a management server 200, a mobile terminal 300, an access point AP, a cloud server 400, and other communication terminals.

[0032] In one implementation, the camera 105 is a near-infrared camera. The near-infrared camera includes an IR (Infrared) projector that projects near-infrared light. By using the near-infrared camera, an image showing the inside of the room 900 can be captured even at night. In another implementation, the camera 105 is a surveillance camera that receives only visible light. In still another implementation, a 3D sensor or a thermography camera may be used as the camera 105. The sensor box 100 and the camera 105 may be configured integrally or separately.

[0033] The Doppler sensor 106 is, for example, a microwave Doppler sensor that emits and receives radio waves to detect the behavior (movement) of an object in the room 900. This allows the detection of biological information of the resident 800 in the room 900. In one example, the Doppler sensor 106 emits microwaves in the 24 GHz band toward the bed 901 in each room 900, and receives the reflected waves reflected by the resident 800. The reflected waves are Doppler shifted due to the movements of the resident 800. The Doppler sensor 106 can detect the breathing condition and heart rate of the resident 800 from the reflected waves.

[0034] The wireless communication device 107 receives signals from the care call handset 500, the door sensor 510, the toilet sensor 520, the odor sensor 530, the vital sensor 540, and the microphone 550, and transmits the signals to the control device 101. For example, the care call handset 500 includes a care call button 501, and when the care call button 501 is operated, it transmits a signal indicating that the operation has been performed to the wireless communication device 107. The door sensor 510, the toilet sensor 520, the odor sensor 530, and the vital sensor 540 each transmit their detection output to the wireless communication device 107.

[0035] The storage device 120 is, for example, a storage medium such as a hard disk or an external storage device. The storage device 120 stores the programs executed by the control device 101 and various data used to execute the programs. The various data may include behavioral information of the resident 800.

[0036] At least one of the above programs and data may be stored in a storage device other than the storage device 120, so long as the storage device is accessible by the control device 101. The storage device other than the storage device 120 is, for example, a storage area (e.g., cache memory, etc.) of the control device 101, the ROM 102, the RAM 103, and / or an external device (e.g., the management server 200, the mobile terminal 300, etc.).

[0037] In the monitoring system of this embodiment, a pressure sensor 560 is provided in each living room 900. The pressure sensor 560 is connected to the sensor box 100, and the control device 101 can acquire the detection output of the pressure sensor 560. The installation mode of the pressure sensor 560 in the living room 900 will be described later with reference to FIG.

[0038] (Behavioral Information) The above behavioral information will be explained. The behavioral information is, for example, information that indicates that the resident 800 has performed a predetermined behavior. In one example, the predetermined behavior includes four behaviors: "getting up" that indicates that the resident 800 has woken up, "getting out of bed" that indicates that the resident 800 has left the bed, "falling down" that indicates that the resident 800 has fallen down, and "falling down" that indicates that the resident 800 has collapsed.

[0039] In one embodiment, the control device 101 generates behavioral information of the resident 800 associated with each room 900. The control device 101 generates the behavioral information based on an image captured by a camera 105 installed in each room 900. For example, the control device 101 detects the head of the resident 800 from the image, and detects the resident 800's "getting up," "getting out of bed," "falling," and "falling down" based on a change over time in the size of the detected head of the resident 800. A specific example of generating behavioral information will be described in more detail below.

[0040] First, the storage device 120 stores the area in which the bed 901 is located in the living room 900, a first threshold value Th1, a second threshold value Th2, and a third threshold value Th3. The first threshold value Th1 distinguishes the size of the head of the resident between when the resident is in a lying position and when the resident is in a sitting position in the area in which the bed 901 is located. The second threshold value Th2 distinguishes whether the resident is in a standing position or not based on the size of the head in the living room 900 excluding the area in which the bed 901 is located. The third threshold value Th3 distinguishes whether the resident is in a lying position or not based on the size of the head in the living room RM excluding the area in which the bed 901 is located.

[0041] The control device 101 extracts a moving object region from the target image as a region of the resident 800. The extraction of the moving object region is based on, for example, a background subtraction method or a frame subtraction method. The control device 101 further extracts the head region of the resident 800 from the extracted moving object region by pattern matching using a head model prepared in advance. For example, a circular or elliptical Hough transform technique is used to extract the head region. A threshold derived by a neural network trained for head detection may be used for pattern matching. The control device 101 detects "getting up," "getting out of bed," "falling over," and "falling down" from the position and size of the extracted head.

[0042] The control device 101 may determine that the behavior "getting up" has occurred when detecting that the position of the head extracted as described above is located within the area of ​​the bed 901 and that the size of the head extracted as described above has changed from the size of a lying position to the size of a sitting position. A first threshold value Th1 is used to detect the change in the size of the head.

[0043] The control device 101 may determine that the behavior "getting out of bed" has occurred when the position of the head extracted as described above moves from within the area where the bed 901 is located to outside the area where the bed 901 is located and when it detects that the size of the head extracted as described above has changed from a certain size to that of a standing position. A second threshold value Th2 is used to detect that the size of the head has changed to that of a standing position.

[0044] The control device 101 may determine that the behavior "fall" has occurred when the head position extracted as described above moves from within the area where the bed 901 is located to outside the area where the bed 901 is located and the head size extracted as described above changes from a certain size to the size of the lying position. A third threshold value Th3 is used to detect the change in head size to the size of the lying position.

[0045] The control device 101 may determine that the behavior "fall" has occurred when the position of the head extracted as described above is located within the room 900 excluding the area where the bed 901 is located and when it detects that the size of the extracted head has changed from a certain size to the size of the lying position. A third threshold value Th3 is used to detect the change in the size of the head to the size of the lying position.

[0046] As described above, in one specific example, the control device 101 of the sensor box 100 generates behavioral information of the resident 800. Note that in the monitoring system, other elements may generate behavioral information of the resident 800 using images of the inside of the room 900.

[0047] (Management Server 200) The management server 200 includes a control device 201 , a ROM 202 , a RAM 203 , a communication interface 204 , a display interface 205 , an operation interface 207 , and a storage device 220 .

[0048] The control device 201 controls the management server 200. The control device 201 is configured, for example, by at least one integrated circuit. The integrated circuit is configured, for example, by at least one CPU, at least one ASIC, at least one FPGA, or a combination thereof.

[0049] An antenna (not shown) and other elements are connected to the communication interface 204. The management server 200 exchanges data with an external communication device via the antenna. The external communication device includes, for example, the sensor box 100.

[0050] The display interface 205 is connected to a display 206, and sends image signals for displaying images to the display 206 in accordance with commands from the control device 201 or the like.

[0051] The operation interface 207 is, for example, a Universal Serial Bus (USB) terminal, and is connected to an input device 209. The operation interface 207 receives a signal indicating a user operation from the input device 209. The input device 209 is, for example, a mouse, a keyboard, a touch panel, or any other device capable of receiving an input operation from a user.

[0052] The storage device 220 is, for example, a storage medium such as a hard disk or an external storage device. In one implementation example, the storage device 220 stores a program executed by the control device 201, but the program may be stored in another storage device accessible by the control device 201.

[0053] (Mobile terminal 300) The mobile terminal 300 includes a control unit 301 , a RAM 303 , a communication interface 304 , a display 305 , an input device 306 , and an internal memory 320 .

[0054] The control device 301 controls the mobile terminal 300. The control device 301 is configured, for example, by at least one integrated circuit. The integrated circuit is configured, for example, by at least one CPU, at least one ASIC, at least one FPGA, or a combination thereof.

[0055] An antenna (not shown) and other elements are connected to the communication interface 304. The mobile terminal 300 exchanges data with external communication devices via the antenna and an access point AP (FIG. 1). The external communication devices include, for example, the sensor box 100 and the management server 200.

[0056] The display 305 is realized by, for example, an organic EL (Electro Luminescence) display. The input device 306 is realized by, for example, a touch sensor overlaid on the display 305. The touch sensor accepts various operations on the mobile terminal 300 by touch operations, and outputs the contents of the operations to the control device 301.

[0057] The built-in memory 320 is a storage medium such as an eMMC (Embedded MultiMediaCard). In one implementation example, the built-in memory 320 stores a program executed by the control device 301, but the program may be stored in a storage device other than the built-in memory 320 as long as the storage device is accessible by the control device 301.

[0058] (Cloud Server 400) The cloud server 400 includes a control device 401 , a ROM 402 , a RAM 403 , a communication interface 404 , and a storage device 420 .

[0059] The control device 401 controls the cloud server 400. In one implementation, the control device 401 is configured with at least one integrated circuit. In one implementation, the integrated circuit is configured with at least one CPU, at least one ASIC, at least one FPGA, or a combination thereof.

[0060] An antenna (not shown) and other elements are connected to the communication interface 404. The cloud server 400 exchanges data with external communication devices via the antenna. In one implementation example, the external communication devices include the sensor box 100 and the management server 200.

[0061] The storage device 420 is a storage medium such as a hard disk, an external storage device, etc. In one implementation example, the storage device 420 stores a program executed by the control device 401, but the program may be stored in another storage device accessible by the control device 401.

[0062] (The meaning of the term "program" in this specification) In the monitoring system, various "programs" may be provided not as standalone programs but as part of any program. Various "programs" may be realized by cooperation of any multiple programs. In one example, when the control device 101 of the sensor box 100 executes a first program to realize the function described in this specification, the first program may use some modules of a second program. Even if the first program does not include some modules for realizing the function, if it mainly contributes to the realization of the function, it does not deviate from the purpose of the program for realizing the function. In addition, the functions described in this specification may be realized in part or in whole by dedicated hardware. Furthermore, the device providing the functions described in this specification may provide some or all of its functions from an external device as a so-called cloud service.

[0063] (Pressure sensor installation) 5 is a diagram showing an example of the installation mode of pressure sensors in each living room 900. The pressure sensor 560 includes measurement sheets 560A, 560B, and 560C. In one example, each of the measurement sheets 560A, 560B, and 560C is a capacitor-type pressure sensor arranged in a grid pattern. The pressure sensor 560 can output the distribution and total amount of pressure on each of the measurement sheets 560A, 560B, and 560C as a detection output.

[0064] The measuring sheet 560A is placed on the bed 901. In this way, the measuring sheet 560A can detect the pressure applied to the bed 901 (and its distribution).

[0065] The measurement sheet 560B is placed on the seat (seating position) of the wheelchair 905. This allows the measurement sheet 560B to detect the pressure (and its distribution) applied to the seat of the wheelchair 905. In the example of FIG. 5, the wheelchair 905 is positioned beside the bed 901. The measurement sheet 560B may include a communication interface for wireless communication. This allows the measurement sheet 560B to communicate with the sensor box 100 regardless of whether the wheelchair 905 is placed inside or outside the room 900.

[0066] The measurement sheet 560C is placed on the floor surface of the living room 900. In this way, the measurement sheet 560C can detect the pressure applied to the floor surface of the living room 900 (and its distribution).

[0067] [Derivation of indicators for nursing care level certification] The control device 101 of the sensor box 100 acquires detection outputs from various sensors, derives an index for the certification of the nursing care level of the resident 800, and outputs the derived index. In this sense, the sensor outputs a detection output representing the movement of the resident 800 to the control device 101. Such sensors are, for example, the Doppler sensor 106, the camera 105, the care call handset 500, the door sensor 510, the toilet sensor 520, the odor sensor 530, the vital sensor 540, the microphone 550, and / or the pressure sensor 560. In this specification, "derive" is used as a general term for identifying, acquiring, calculating, and the like.

[0068] The control device 101 acquires data from the Doppler sensor 106 via the interface 106A. The control device 101 acquires data from the camera 105 via the interface 105A. The control device 101 acquires data from the care call handset 500, the door sensor 510, the toilet sensor 520, the odor sensor 530, the vital sensor 540, the microphone 550, and the pressure sensor 560 via the wireless communication device 107. In this sense, the interface 106A, the interface 105A, and the wireless communication device 107 are each an example of an interface in the sensor box 100 from which the control device 101 acquires detection output.

[0069] The control device 101 derives information for certification of the level of care required or the level of support required by using the portion of the detection output of the sensor that corresponds to the unconscious movements of the resident 800 (hereinafter, certification of the level of care required and certification of the level of support required are collectively referred to as certification of the level of care required). Examples of unconscious movements include movements during sleep (more specifically, movements of turning over in bed) and reflexive behaviors (movements to avoid danger, etc.).

[0070] It is highly likely that residents do not make intentional movements while sleeping, and for this reason, turning over in bed is considered an example of an unconscious movement.

[0071] The types of unconscious actions are not limited to those mentioned above. An example of information related to certification of level of care required (information related to level of care required or end of form) is an ADL index.

[0072] In one example, the control device 101 specifies the score for the item "movement of turning over in sleep" as the ADL index. More specifically, the control device 101 specifies the number of times the resident 800 turns over in sleep during a given period (e.g., during sleep, from 8 p.m. to 5 a.m. the next morning) from an image of the inside of the living room 900, and specifies the score associated with the number of times as the score for the item "movement of turning over in sleep". Note that a conversion table may be stored in the storage device 120 as an example of information that associates the number of times of turning over with the score. The control device 101 may obtain the score associated with the number of times by referring to the conversion table. In this embodiment, the control device 101 may obtain the score by applying the number of times of turning over to a predetermined formula instead of the conversion table.

[0073] The control device 101 may specify, as an index related to a reflexive behavior, a time (reaction time) from the occurrence of a given event (for example, a call) to a reaction to the given event (for example, the generation of a voice in response, a look-back movement, etc.), convert the specified reaction time into a score, and specify an index from the score. As another example, the control device 101 may specify a time from the start to the end of a look-back movement (reflexive behavior time), convert the specified reflexive behavior time into a score, and specify an index from the score. The control device 101 may specify that the resident has reacted to a given event by specifying a behavior that matches a predetermined pattern from a detection output such as an image from the camera 105, or may specify by applying a detection output such as an image from the camera 105 to a trained machine learning model and using the output of the machine learning model. Note that a conversion table may be stored in the storage device 120 as an example of information that associates a reaction time with a score and / or information that associates a reflexive behavior time with a score. The control device 101 may acquire a score associated with the reaction time by referring to the conversion table.

[0074] [Derivation of indicators for the item "turning over"] The control device 101 can determine the number of turning movements in a given period of time and derive an index for the item "turning movements" based on the number.

[0075] FIG. 6 is a flowchart of an example of processing in the control device 101 for deriving an index for the item "turning over motion." In one implementation example, the control device 101 performs the processing of FIG. 6 by having the processor execute a given program. The given program may be stored in the storage device 120. In this sense, the storage device 120 is an example of a recording medium that non-temporarily stores a program that causes the control device 101 to perform the processing of FIG. 6. The given program may be stored in a storage device other than the storage device 120 that is accessible by the control device 101. In this case, the storage device other than the storage device 120 is an example of a recording medium that non-temporarily stores a program that causes the control device 101 to perform the processing of FIG. 6.

[0076] 6, in step S600, control device 101 acquires images of the inside of room 900 for a given period for the indicator of the item "turning over motion" from camera 105. Note that in step S600, control device 101 may receive images from camera 105 in real time, or may receive images from camera 105 that have been stored in a given storage device from the storage device.

[0077] In step S602, the control device 101 identifies the number of times the person turns over in their sleep in the image acquired by the camera 105. How to identify the number of times the person turns over in their sleep will be described later with reference to FIG.

[0078] In step S604, the control device 101 acquires an index corresponding to the number of times of turning over identified in step S602.

[0079] In one implementation example, the control device 101 identifies the number of times the motion of turning over occurs in a given period, and identifies the score associated with the number of times as the score for the item "motion of turning over." Note that the storage device 120 may store a conversion table as an example of information that associates the number of times the motion of turning over occurs in a given period with the score. The control device 101 may obtain the score associated with the number of times by referring to the conversion table. The conversion table is stored in the storage device 120, for example.

[0080] In one implementation, the higher the number of tossing movements in a given period of time, the higher the score, but the relationship between number and identification is not so limited.

[0081] In step S606, the control device 101 outputs the index acquired in step S604, and ends the processing in Fig. 6. The output destination may be an output device such as a display connected to the control device 101, or may be another information device (such as the management server 200) with which the control device 101 can communicate.

[0082] Fig. 7 is a diagram showing an example of skeletal information used to detect the motion of turning over. Fig. 7 shows skeletal information of a person identified by the control device 101 for a certain frame of an image from the camera 105. An outline OL1 represents the outline of the person, a point P10 represents the position of the center of gravity of the person, and points P11 and P12 represent the positions of both shoulders of the person.

[0083] When a person lying down turns to the right or left and is photographed from the front, it is expected that the distance between one shoulder and the other shoulder of the person will be shorter in the photographed image, and when a person who has turned to the right or left changes their body orientation to face the front, it is expected that the distance between one shoulder and the other shoulder of the person will be longer in the photographed image of the person from the front.

[0084] The control device 101 may count the number of times the resident turns over in his / her sleep, using the image from the camera 105. In counting the number of times the resident turns over in his / her sleep, the control device 101 may update the count value by adding 1 when a change is detected in the distance between both shoulders of a person (resident) in the image from the distance DF indicating that the person is facing forward to the distance DS indicating that the person is facing to the side (right or left).

[0085] That is, the control device 101 extracts from the image from the camera 105 a portion corresponding to the period during which the distance between the shoulders changes from distance DF to distance DS, and identifies the turning motion using the images (frames) of that period.

[0086] As described above with reference to Figs. 6 and 7, the control device 101 derives and outputs an index of the resident's turning motion.

[0087] In the process of FIG. 6, the control device 101 may specify a period during which the resident is in a sleeping state based on an image from the camera 105, instead of a predetermined given period, as a target period for derivation of an index. This allows the movement of turning over in bed to be more reliably specified as an unconscious movement of the resident. The control device 101 may specify a period during which the resident is in a sleeping state, for example, based on the detection output from the Doppler sensor 106 and / or the detection output from the vital sensor 540. For example, the control device 101 may determine whether the resident is in a sleeping state depending on whether the mode of body movement specified based on the detection output from the Doppler sensor 106 is a mode of a sleeping state. The control device 101 may determine whether the resident is in a sleeping state depending on whether the heart rate specified based on the detection output from the vital sensor 540 is a heart rate in a sleeping state.

[0088] When the distance between the shoulders of the person in the image changes from distance DS to distance DF, the control device 101 may increment and update the count value of the turning motion by 1. In this case, the control device 101 extracts a portion from the image from the camera 105 that corresponds to the period during which the distance between the shoulders changes from distance DS to distance DF, and identifies one turning motion by using the images (frames) of that period.

[0089] The control device 101 may identify the motion of turning over in bed using detection output from a measurement sheet 560A placed on the bed, instead of or in addition to the image from the camera 105. In this case, the control device 101 acquires the detection output from the measurement sheet 560A, instead of or in addition to the control of step S600. Then, in step S602, the control device 101 identifies the number of times the resident has turned over in bed using the detection output from the measurement sheet 560A, instead of or in addition to the image from the camera 105.

[0090] An example of determining the number of times of turning over using the detection output from the measurement sheet 560A will be described. The storage device 120 may store a pattern (hereinafter also referred to as a "front pattern") corresponding to a state in which a person on the bed faces forward and a pattern (hereinafter also referred to as a "sideways pattern") corresponding to a state in which a person on the bed faces sideways, for the detection output from the measurement sheet 560A. In one implementation example, the front pattern has a larger area on the measurement sheet 560A where pressure equal to or greater than a given value is detected than the sideways pattern.

[0091] The control device 101 may update the count value of one turning motion by incrementing it by 1 each time the detection output of the measurement sheet 560A changes from the front pattern to the sideways pattern. Alternatively, the control device 101 may update the count value of one turning motion by incrementing it by 1 each time the detection output of the measurement sheet 560A changes from the sideways pattern to the front pattern.

[0092] [Calculation of indicators related to actions that are naturally performed without awareness] The control device 101 may use the detection output from the sensor to derive an index related to a behavior (habitual behavior) that is naturally performed without being conscious of the user. The index related to the habitual behavior may be an index derived based on, for example, a behavior immediately after waking up from sleep (movement of getting up from bed).

[0093] The index for habitual behavior may also be the item "movement of transferring to a wheelchair" or the item "movement of standing up". "Motion of transferring to a wheelchair" is an action on the way to a purpose such as sitting on a wheelchair. In this embodiment, "movement of transferring" may refer to the action of transferring from a given place such as a bed to a wheelchair, or conversely, the action of moving from a wheelchair to a given place such as a bed. "Motion of standing up" is an action on the way to walking. A subject who does not need much care or support may be so conscious of the previous action that he or she does not pay attention to the action on the way. It is considered that it is possible to estimate whether the action includes acting by analyzing these actions. The derivation of each type of index will be described below.

[0094] [Processing flow (deriving indicators for the item "transfer to wheelchair" based on images)] Fig. 8 is a flowchart of an example of a process executed by the control device 101 to derive an ADL index. In the process of Fig. 8, a score for the item "movement of transferring to a wheelchair" is derived as an example of an ADL index. In one implementation example, the control device 101 performs the process of Fig. 8 by causing a processor to execute a given program.

[0095] Referring to FIG. 8, in step S800, the control device 101 acquires from the camera 105 an image of the room 900 corresponding to a certain time. The timing and length of time from which images corresponding to the certain time are acquired can be appropriately set in the monitoring system. In one example, the control device 101 may acquire images of a period including a time period when the resident 800 is assumed to have transferred to a wheelchair in the room 900, which is designated as an image corresponding to the certain time by an administrator of the monitoring system. In another example, the control device 101 may acquire images of each day as an image corresponding to the certain time. Note that the control device 101 may acquire images from the camera 105 sequentially.

[0096] In step S802, the control device 101 calculates the rotation angle of the person's trunk in each frame of the image acquired in step S800.

[0097] In calculating the rotation angle of the trunk of a person, the control device 101 first acquires skeletal information of the person from an image. In one example, the acquisition of skeletal information of the person from an image is performed using a known service (for example, "User Local Posture Estimation AI" by User Local Co., Ltd.)<https: / / humanpose-ai.userlocal.jp / > The control device 101 then identifies a line from the center of the head to the center of the chest from the acquired skeletal information, and calculates the rotation angle of the trunk as the angle between the line and a predetermined reference line.

[0098] The calculation of the rotation angle of a person's trunk will be described in more detail with reference to Fig. 9 and Fig. 10. Fig. 9 is a diagram showing the flow of transferring from a bed to a wheelchair. Flow 700 shown in Fig. 9 includes images 701, 702, 703, and 704 at four timings, and shows the flow of the movement of a resident transferring from a bed to a wheelchair. Image 702 shows the state at a timing after image 701, image 703 shows the state at a timing after image 702, and image 704 shows the state at a timing after image 703.

[0099] 9, heads H1, H2, H3, and H4 represent the heads of people identified from skeletal information in each of images 701, 702, 703, and 704. Chests C1, C2, C3, and C4 represent the chests of people identified from skeletal information in each of images 701, 702, 703, and 704.

[0100] FIG. 10 is a diagram showing the trunk rotation angles identified in each of the four images in FIG.

[0101] A flow 800 in Fig. 10 corresponds to the flow 700 in Fig. 9. In Fig. 10, frames 801, 802, 803, and 804 correspond to images 701, 702, 703, and 704 in Fig. 9, respectively. In frame 801, a line L1 from head H1 to chest C1 and a reference line LS are shown. Also, "85°" is shown as the angle between line L1 and reference line LS.

[0102] In each of frames 802, 803, and 804, the lines from the head H2, H3, and H4 to the chest C2, C3, and C4 are shown as lines L2, L3, and L4, respectively, and the angles between the lines L2, L3, and L4 and the reference line LS are shown as "75°", "40°", and "5°", respectively.

[0103] The reference line LS is used commonly for the information within the frames 801 to 804, that is, for the images 701 to 704. The reference line LS may be specified based on the contour of the bed in each frame (for example, parallel to a straight line constituting the contour).

[0104] 8, in step S804, the control device 101 determines whether or not the rotation angle of the person's trunk has changed from angle A1 to angle A2 in an example frame constituting the image acquired in step S800. The control device 101 repeats the control of step S804 until a change from angle A1 to angle A2 is observed in the rotation angle of the person's trunk (NO in step S804), and when a change from angle A1 to angle A2 is observed (YES in step S804), the control proceeds to step S806.

[0105] Here, a specific example of change in the rotation angle of the trunk will be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of a manner in which the rotation angle of the trunk changes.

[0106] In the graph of Fig. 11, the vertical axis represents the rotation angle of the trunk, and the horizontal axis represents time. Line LM represents an example of the change over time in the rotation angle of the trunk identified in each frame of the image from camera 105. The rotation angle of the trunk of line LM changes from angle A1 to angle A2 between time T1 and time T2. When such a change in the rotation angle of the person's trunk is detected, the control device 101 advances the control from step S804 to step S806.

[0107] Returning to Fig. 8, in step S806, the control device 101 identifies the time (transfer time) required for the person in the image to transfer from the bed to the wheelchair. In the example of Fig. 11, the control device 101 identifies the time required for the trunk rotation angle to change from angle A1 to angle A2 in the image from the camera 105 (the time from time T1 to time T2) as the time required for the transfer.

[0108] In step S808, the control device 101 acquires, as an index, a score corresponding to the transfer time identified in step S806. The score means the score for the item "movement of transferring to a wheelchair." The control device 101 may acquire the score by referring to the conversion table that associates the length of time required for the transfer movement with the score, as described above. The conversion table is stored in the storage device 120, for example.

[0109] In step S810, the control device 101 outputs the index acquired in step S808, and ends the processing in Fig. 8. The output destination may be an output device such as a display connected to the control device 101, or may be another information device (such as the management server 200) with which the control device 101 can communicate.

[0110] When the output destination is another information device, the control device 101 outputs the index via the communication interface 104. In this sense, the communication interface 104 is an example of an output unit that outputs the index. In this embodiment, the output unit may be included in an analysis unit that derives the index. When the output destination is a display, the control device 101 outputs the index via a hardware element that serves as an interface between the display and the sensor box. In this sense, the hardware element is an example of an output unit that outputs the index.

[0111] In the process described above with reference to Figure 8, the control device 101 extracts from the image from the camera 105 the portion where the rotation angle of the person's trunk in each frame changes from angle A1 to angle A2 (for example, the portion corresponding to the period from time T1 to time T2 in Figure 11), identifies the time to which this portion corresponds (transfer time: for example, the time from time T1 to time T2 in Figure 11), and obtains a score corresponding to the transfer time, thereby deriving an index for the transfer action to a wheelchair.

[0112] In one implementation, it is envisioned that a shorter transfer time corresponds to a higher score, although the relationship between transfer time and score is not so limited.

[0113] The control device 101 may impose additional conditions on the extraction of the above-mentioned part. For example, the control device 101 may extract the above-mentioned part and derive the index under the condition that a change from angle A1 to angle A2 is observed in the rotation angle of the trunk, and further, that the person is located on the bed in the frame corresponding to angle A1 and the person is located outside the bed in the frame corresponding to angle A2. That is, the control device 101 does not derive the index when, for example, the person is located on the bed in both the frame corresponding to angle A1 and the frame corresponding to angle A2, or when the person is located outside the bed in both the frame corresponding to angle A1 and the frame corresponding to angle A2. This makes it possible to avoid deriving the index when the resident is performing an action other than a transfer action to a wheelchair, such as exercising on the bed in the room or moving in a wheelchair.

[0114] The control device 101 may derive an index for the transfer operation to the wheelchair by specifying the speed of the transfer operation instead of the transfer time and acquiring a score corresponding to the speed. The speed is specified, for example, as the reciprocal of the time required for the rotation angle of the trunk of the person in each frame to change from a first angle to a second angle. In this case, the conversion table associates the speed with the score. In one implementation example, it is assumed that the higher the speed, the higher the corresponding score, but the relationship between the speed and the score is not limited thereto. The conversion table is stored, for example, in the storage device 120.

[0115] The control device 101 may derive the index based on the transfer time (or the transfer speed) of one transfer motion. The control device 101 may derive the index based on a statistical value (for example, an average value) of the transfer time (or the transfer speed) of multiple transfer motions.

[0116] In step S810, the control device 101 may further output statistical values ​​of the measurement values ​​(indexes, transfer times, etc.) in each of the multiple transfer movements as auxiliary numerical values ​​for the indexes. The output statistical values ​​are, for example, the minimum, maximum, average, standard deviation, and / or variance of the measurement values.

[0117] In one implementation example, in step S810, the control device 101 may output a standard deviation of the transfer time for multiple transfer operations together with the indicator. By checking the value of the standard deviation, the administrator of the monitoring system can infer whether the resident's behavior includes acting.

[0118] In another implementation example, the control device 101 may output the transfer time for each of multiple transfer actions. The administrator of the monitoring system can infer that the transfer action for deriving the index included the resident's performance by the existence of an extremely short and / or long transfer time.

[0119] [Process flow (deriving an index for the item "transfer to wheelchair" based on pressure)] Fig. 12 is a flowchart of another example of the process executed by the control device 101 for deriving the ADL index. In the process of Fig. 12, a score for the item "movement of transferring to a wheelchair" is derived as an example of the ADL index. In one implementation example, the control device 101 performs the process of Fig. 8 by causing the processor to execute a given program.

[0120] 12, in step S1000, the control device 101 acquires detection values ​​corresponding to a certain period from the measurement sheet 560A (placed on a bed) and the measurement sheet 560B (placed on a wheelchair seat). "Corresponding to a certain period" in step S1000 may be interpreted as having the same meaning as in step S800. The control device 101 may also acquire detection values ​​sequentially from the measurement sheets 560A and 560B.

[0121] In step S1002, the control device 101 checks the change in the detection value of the measurement sheet 560A placed on the bed at specific time intervals (e.g., every second) and judges whether the detection value has dropped to a value Pa at the specific time. In one implementation example, the control device 101 judges that the detection value of the measurement sheet 560A has dropped to a value Pa based on the fact that the detection value has dropped from a value higher than the value Pa to the value Pa. If the control device 101 judges that the detection value has dropped to the value Pa (YES in step S1002), it proceeds to control step S1004, and if not (NO in step S1002), it continues the control in step S1002 for the specific time that has not yet been checked.

[0122] In step S1004, the control device 101 judges whether the detection value of the measurement sheet 560B installed on the wheelchair seat has risen to a value Pb after it is judged in step S1002 that the detection value of the measurement sheet 560A has fallen to a value Pa. In one implementation example, the control device 101 judges that the detection value of the measurement sheet 560B has risen to a value Pb based on the fact that the detection value of the measurement sheet 560B has risen to a value Pb from a value lower than the value Pb. When the control device 101 judges that the detection value of the measurement sheet 560B has risen to a value Pb after the detection value of the measurement sheet 560A has fallen to a value Pa (YES in step S1004), the control device 101 advances control to step S1006. On the other hand, when the control device 101 cannot judge that the detection value of the measurement sheet 560B has risen to a value Pb within a predetermined time (for example, within 30 seconds) after the detection value of the measurement sheet 560A has fallen to a value Pa (NO in step S1004), the control device 101 returns control to step S1002. As a result, the detection values ​​of the measurement sheet 560A that have not yet been subjected to step S1002 are subjected to step S1002.

[0123] In step S1006, the control device 101 determines the time (transfer time) required for the transfer to the wheelchair that occurred in the room in which the measurement sheets 560A and 560B are installed, from the detection values ​​of the measurement sheets 560A and 560B. In one implementation example, the control device 101 determines the transfer time as the length of time from the time when the detection value of the measurement sheet 560A drops to a value Pa to the time when the detection value of the measurement sheet 560B reaches a value Pb.

[0124] An example of determining the transfer time will be described more specifically with reference to Fig. 13. Fig. 13 is a diagram showing an example of changes in the detection values ​​of the measurement sheet 560A and the measurement sheet 560B.

[0125] 13, line La represents the change over time in the detection value of measurement sheet 560A, and line Lb represents the change over time in the detection value of measurement sheet 560B. In the graph including each of lines La and Lb, the vertical axis represents the detected pressure value, and the horizontal axis represents time.

[0126] 13, the line La drops to a value Pa at time Ta, and the line Lb reaches a value Pb at time Tb. In this case, the control device 101 specifies the time from time Ta to time Tb (time Tx) as the transfer time.

[0127] 12, in step S1008, the control device 101 acquires a score corresponding to the transfer time identified in step S1006 as an index for the item "motion of transferring to a wheelchair." Acquisition of the score corresponding to the transfer time can be achieved, for example, in a manner similar to that described for step S808.

[0128] In step S1010, the control device 101 outputs the index acquired in step S1008, and ends the processing in FIG.

[0129] In the processing described above with reference to FIG. 12, the control device 101 extracts the detection output from the detection outputs of the measurement sheets 560A, 560B, during the period from when the detection value of the measurement sheet 560A drops to value Pa until the detection value of the measurement sheet 560B rises to value Pb, identifies the transfer time based on the detection output, derives an index (score) based on the identified transfer time, and outputs the derived index.

[0130] In the processing of FIG. 12, the detection output of the measurement sheet 560A may be only that of the area covered by the measurement sheet 560A where the resident is expected to be located immediately before transferring to the wheelchair.

[0131] [Derivation of indicators for the item "standing up"] The control device 101 can identify the speed of the motion of standing up, and derive an index for the item "motion of standing up" based on the speed.

[0132] Fig. 14 is a diagram showing an example of skeletal information used to detect a standing up motion. Fig. 14 shows skeletal information of a person identified by the control device 101 for a certain frame of an image from the camera 105. An outline OL1 represents the outline of the person, a point P10 represents the position of the center of gravity of the person, and an area H10 represents the area identified as the person's head.

[0133] 2, the sensor box 100 including the camera 105 is installed on the ceiling CL of the room 900. Therefore, the camera 105 captures an image of the resident 800 in the room 900 from above. Therefore, as the standing-up motion progresses, the distance between the head of the resident 800 and the camera 105 becomes shorter, and as a result, the area of ​​the head in the image captured by the camera 105 becomes larger.

[0134] The control device 101 may determine the time required for the area of ​​the area identified as the person's head in the image captured by the camera 105 to change from a first area to a second area, and determine the speed of the person's standing up motion based on the reciprocal of the time. That is, the control device 101 may extract a portion from the image from the camera 105 where the area of ​​the area identified as the person's head changes from the first area to the second area, and determine the speed of the person's standing up motion based on the extracted portion.

[0135] The storage device 120 may store a conversion table as an example of information relating the speed of the standing up movement to the score. The control device 101 may obtain the score associated with the specified speed by referring to the conversion table, and output the obtained score as an index of the item "standing up movement." The conversion table is stored in the storage device 120, for example.

[0136] In one implementation, the faster the speed of the rising motion, the higher the score, but the relationship between speed and score is not so limited.

[0137] The control device 101 may output statistical values ​​of the measured values ​​(the index, the speed of the standing-up movement, etc.) in each of the multiple standing-up movements as auxiliary numerical values ​​for the index. The output statistical values ​​are, for example, the minimum value, maximum value, average value, standard deviation, and / or variance of the measured values.

[0138] In one implementation example, the control device 101 may output the standard deviation of the speed together with the index in step S810. The administrator of the monitoring system can infer whether the resident's behavior includes acting by checking the value of the standard deviation.

[0139] In another implementation example, the control device 101 may output the speed of each standing up motion for multiple times. The administrator of the monitoring system can infer that the standing up motion for deriving the index included the resident's performance due to the presence of an extremely slow and / or fast standing up motion.

[0140] [Derivation of indicators for the item "walking movement"] "Walking" is an important item in evaluating ADL. Therefore, in addition to deriving an index for "naturally performed actions without consciousness", it is more preferable to derive an index for "walking". The control device 101 can identify the speed of walking and derive an index for the item "walking" based on the speed.

[0141] Fig. 15 is a diagram showing an example of skeletal information used to detect a walking movement. Fig. 15 shows skeletal information of a person identified by the control device 101 for a certain frame of an image from the camera 105. An outline OL1 represents the outline of the person, and a point P10 represents the position of the center of gravity of the person.

[0142] The control device 101 may specify the moving speed of the center of gravity position of the person in the image captured by the camera 105 as the walking speed of the person. The storage device 120 may store a conversion table as an example of information that associates the walking speed with a score. The control device 101 may obtain the score associated with the specified speed by referring to the conversion table, and output the obtained score as an index of the item "walking". The conversion table is stored in the storage device 120, for example.

[0143] The control device 101 may specify the walking speed using the detection output of the measurement sheet 560C.

[0144] That is, the control device 101 may specify the trajectory of the position where the measurement sheet 560C detects the pressure as the trajectory of the movement of the resident in the room 900, specify the movement distance of the resident from the trajectory of the movement of the resident, and specify the walking speed based on the specified movement distance and the time required for the movement. In this case, the control device 101 may extract, from the detection output of the measurement sheet 560C, a portion corresponding to a period during which the position where the measurement sheet 560C detects the pressure has changed by more than a predetermined distance as a portion for specifying the walking speed, and then specify the walking speed.

[0145] In one implementation, the faster the walking motion, the higher the score, but the relationship between speed and score is not so limited.

[0146] The control device 101 may output statistical values ​​of the measured values ​​(indices, walking speeds, etc.) in each of the multiple walking movements as auxiliary values ​​for the indices. The output statistical values ​​are, for example, the minimum, maximum, average, standard deviation, and / or variance of the measured values.

[0147] In one implementation example, the control device 101 may output the standard deviation of the walking speed together with the indicator in step S810. The manager of the monitoring system can infer whether the resident's behavior includes acting by checking the value of the standard deviation.

[0148] In another implementation example, the control device 101 may output the walking speed for each of multiple walking movements. The administrator of the monitoring system can infer that the walking movements for deriving the index included the resident's performance due to the presence of extremely slow and / or fast walking speeds.

[0149] [Identifying behavior types using machine learning models] In deriving the index, the control device 101 may use a machine learning model to identify the type of movement of the resident from an image of the inside of the room 900. The processor of the control device 101 executes a program that realizes the algorithm of the machine learning model, and the control device 101 functions as a classifier.

[0150] Fig. 16 is a diagram showing an example of a machine learning model. In the example of Fig. 16, a machine learning model 1500 includes a learning device 1501 based on a LightGBM (Gradient Boosting) framework and a learning device 1502 based on a LSTM (Long Short Term Memory) framework. Note that the machine learning model shown in Fig. 16 is merely an example, and any type of model may be used to identify the type of a resident's movement.

[0151] The learning device 1501 is subjected to a learning process using a plurality of frame images labeled with the type of human movement. More specifically, in one example, the learning device 1501 is configured to output a prediction result of the type of human movement in an input frame image by performing a learning process using frame images labeled with any of the following nine types:

[0152] Type (1): Bedridden Type (2): Lying in bed or sitting on the edge of the bed Type (3): Medium end seat Type (4): Standing from edge of seat Type (5): Transfer from edge seat to wheelchair Type (6): Wheelchair mobility Type (7): Standing to walking Type (8): Walking Type (9): Other than types (1) to (8) above "Lying in bed" means that the frame image includes a person lying on the bed 901. "From lying to sitting on edge of bed" means that the frame image includes a person changing their state on the bed 901 from lying to sitting on the edge of the bed. "Sitting on edge of bed" means that the frame image includes a person sitting on the edge of the bed 901. "From sitting on edge of bed to standing" means that the frame image includes a person transferring from the bed 901 to a wheelchair. "Moving in wheelchair" means that the frame image includes a person moving in a wheelchair. "From standing to walking" means that the frame image includes a person who is in the process of changing their state from a standing state to a walking state. "Walking" means that the frame image includes a person who is walking.

[0153] The learning device 1502 performs learning processing using a plurality of sets of frame images labeled with the types of human actions. FIG. 17 is a diagram for explaining the machine learning of the learning device 1502.

[0154] In FIG. 17, a set 1600 represents an example of a set of frame images used for machine learning by the learning device 1502. The set 1600 includes seven consecutive frame images 1601 to 1607. An example of the time difference between the corresponding consecutive frame images is 0.4 seconds. In the learning process of the learning device 1502, the set 1600 is labeled with the type of action labeled in the temporally central frame image 1604, and input to the learning device 1502. That is, the label attached to the frame image 1604 among the frame images 1601 to 1607 is used as the correct answer label. The types of labels include, for example, types corresponding to each of the nine types (type (1) to type (9)) described above. Note that the number of frame images included in one set, the label attached to which frame image among the frame images included in one set is to be used as the label to be attached to the set, and the time difference between the corresponding consecutive frame images are all merely examples.

[0155] The learning device 1502 is configured to perform the above-described learning process, and when consecutive frame images are input, to output a prediction result of the type of human movement in a specific frame image among the consecutive frame images.

[0156] The machine learning model 1500 outputs a prediction result in two stages for each frame of an image from the camera 105 using the learning device 1501 and the learning device 1502. More specifically, the machine learning model 1500 outputs a prediction result of the type of movement of each frame using the learning device 1501. Then, the machine learning model 1500 inputs a set consisting of a plurality of consecutive frames to the learning device 1502 in the manner shown in FIG. 17. The set of frames is input to the learning device 1502 together with the prediction result of the learning device 1501 for the plurality of frames constituting the set. More specifically, the set of frames is input to the learning device 1502 together with the prediction result of a specific frame in the set (the frame at the center in time in the set in the example of FIG. 17). The learning device 1502 outputs a prediction result of the type of movement of a person for a specific frame in the set by inputting the set of frames.

[0157] The machine learning model 1500 outputs the prediction result output from the learning device 1502 as the final prediction result.

[0158] The machine learning model 1500 may correct the prediction result output from the learning device 1502 and output the corrected prediction result as the final prediction result. More specifically, when the learning device 1502 outputs the same prediction result for a given number of frames or more in succession, the machine learning model 1500 outputs the prediction results output for each of the consecutive frames as the final prediction result as is. On the other hand, when the prediction result output by the learning device 1502 for a certain frame is not consecutive for a given number of frames including the frame, the machine learning model 1500 changes the prediction result of the frame to another prediction result output by the learning device 1502 for the consecutive frames before or after the frame, and outputs the changed prediction result as the final prediction result.

[0159] For example, assume that the machine learning model 1500 is configured to output the prediction result output by the learner 1502 as the final prediction result without modification, on the condition that the learner 1502 outputs the same prediction result for 10 or more consecutive frames. In such a case, when the learner 1502 outputs prediction results of type (8) for 12 consecutive frames, then outputs a prediction result of type (6) for one frame, and then outputs prediction results of type (8) for 10 or more consecutive frames, the machine learning model 1500 changes the prediction result of the frame to which the prediction result of type (8) was assigned to type (6), and outputs the changed prediction result as the final prediction result.

[0160] The control device 101 may extract a portion of the image from the camera 105 where frames with the prediction result of type (5) are consecutive, and use this portion to derive an ADL index for the item "movement of transferring to a wheelchair."

[0161] The control device 101 may extract a portion of the image from the camera 105 in which frames with a prediction result of type (4) are consecutive, and use this portion to derive an ADL index for the item "standing up".

[0162] The control device 101 may extract a portion of the image from the camera 105 where frames having a prediction result of type (8) are consecutive, and use the portion to derive the ADL index for the item "walking movement." The control device 101 may extract a portion of the image from the camera 105 where frames having a prediction result of type (7) or type (8) are consecutive, and use the portion to derive the ADL index for the item "walking movement."

[0163] Each embodiment disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is indicated by the claims, not by the above description, and is intended to include all modifications within the meaning and scope of the claims. In addition, the inventions described in the embodiments and each modified example are intended to be implemented, as far as possible, either alone or in combination. [Explanation of symbols]

[0164] 100,100A,100B Sensor box, 200 Management server, 560 Pressure sensor, 560A,560B,560C Measurement sheet, 701,702,703,704 Images, 800,800A,800B Resident, 801,802,803,804 Frame, 900,900A,900B Room, 901,901A,901B Bed, 905 Wheelchair, 1500 Machine learning model, 1501,1502 Learner, 1600 Set, 1601,1604,1607 Frame image.

Claims

1. An interface for acquiring detection output from a sensor that represents the movement of a person who is to be certified as needing a level of nursing care or a level of assistance; an analysis unit that outputs information regarding a level of care required or a level of support required by analyzing at least a portion of the detection output corresponding to an unconscious movement of the subject; The subject's unconscious movement is a movement of turning over in bed, The analysis unit is Identifying the number of times the detection output changes from a first pattern related to a turning motion to a second pattern as the number of times the subject turns over in their sleep; deriving an index of the subject's turning over movement, which is information regarding a level of care or a level of support required, based on the number of times the subject turns over in bed during a given period of time; An information processing device that outputs the index.

2. The interface acquires an image of the subject's room as the detection output, 2. The information processing device according to claim 1, wherein the analysis unit extracts from the image a portion corresponding to the turning over motion, the image being of a period during which the distance between one shoulder and the other shoulder of a person on a bed in the room changes from a first distance to a second distance different from the first distance.

3. An interface for acquiring detection output from a sensor that represents the movement of a person who is to be certified as needing a level of nursing care or a level of assistance; an analysis unit that outputs information regarding a level of care required or a level of support required by analyzing at least a portion of the detection output corresponding to an unconscious movement of the subject; The subject's unconscious movement is a transfer movement, The analysis unit is determining at least one of a transfer time and a transfer speed based on a time required for the subject to change from a first state to a second state in the detection output; deriving an index regarding a transfer motion of the subject, which is information regarding a level of care need or a level of support need, based on at least one of the transfer time and the transfer speed; An information processing device that outputs the index.

4. The interface acquires an image of the subject's room as the detection output, The information processing device according to claim 3, wherein the analysis unit extracts from the image an image of a period during which a rotation angle of the person's trunk in the image changes from a first angle to a second angle different from the first angle, as a portion corresponding to the transfer movement of the subject.

5. In the portion corresponding to the transfer motion of the subject, When the rotation angle of the trunk of the person is the first angle, the person is positioned on a bed; The information processing device according to claim 4 , wherein when a rotation angle of the trunk of the person is the second angle, the person is located outside the bed.

6. The information processing device according to claim 4 , wherein the analysis unit derives the index based on a time taken for a rotation angle of the person's trunk to change from the first angle to the second angle.

7. The information processing device according to claim 3 , wherein the interface acquires, as the detection output, a detection output of a first pressure sensor that detects pressure at a seating position before the transfer related to the transfer operation, and a detection output of a second pressure sensor that detects pressure at the seating position.

8. the analysis unit extracts, from the detection output of the first pressure sensor and the detection output of the second pressure sensor, a detection output for a period from a first timing to a second timing as a portion corresponding to the transfer motion; At the first timing, the detection output of the first pressure sensor changes so as to decrease to a first threshold value, The information processing device according to claim 7 , wherein at the second timing, the detection output of the second pressure sensor changes so as to rise to a second threshold value.

9. An interface for acquiring detection output from a sensor that represents the movement of a person who is to be certified as needing a level of nursing care or a level of assistance; an analysis unit that outputs information regarding a level of care required or a level of support required by analyzing at least a portion of the detection output corresponding to an unconscious movement of the subject; The subject's involuntary movement is a movement of standing up, The analysis unit is A speed of the standing-up movement is identified by analyzing a portion of the detection output corresponding to the subject's standing-up movement, and an index for the standing-up movement is derived based on the speed of the standing-up movement, which is information related to a level of care required or a level of support required certification; An information processing device that outputs the index.

10. The interface acquires an image of the subject's room as the detection output, 10. The information processing device according to claim 9, wherein the analysis unit extracts from the image an image of a period during which an area of ​​a region corresponding to a person's head changes from a first area to a second area different from the first area as a portion corresponding to the standing up motion.

11. An interface for acquiring detection output from a sensor that represents the movement of a person who is to be certified as needing a level of nursing care or a level of assistance; an analysis unit that outputs information regarding a level of care required or a level of support required by analyzing at least a portion of the detection output corresponding to an unconscious movement of the subject; The subject's involuntary movement is a walking movement, The interface acquires an image of the subject's room as the detection output, The analysis unit is extracting, from the images, an image of a period during which a person moved a certain distance or more in the room as a portion corresponding to the walking movement of the subject; deriving an index of walking movement, which is information related to a certification of a level of nursing care or a level of support required, based on the extracted image; An information processing device that outputs the index.

12. The information processing device according to any one of claims 1 to 11, wherein the analysis unit further derives and outputs a statistical value of the index when the index of the same type is derived a plurality of times.

13. The analysis unit is inputting data of a plurality of consecutive frames constituting the image into a trained classifier, and obtaining an estimation result of a movement in the room for each of the plurality of frames; deriving the indicator by utilizing a frame from among the plurality of frames in which the estimation result is the unconscious movement; The information processing device according to any one of claims 2, 4 to 6, 10, and 11, wherein the classifier is subjected to a learning process using, as training data, images consisting of a series of frames including frames labeled with a type of action.

14. The information processing device according to claim 1 , wherein the interface acquires, as the detection output, a detection output of a pressure sensor installed in a room of the subject.

15. A sensor for acquiring a motion state of a subject; An information processing device according to any one of claims 1 to 14, An output device that displays information relating to the level of care required or the level of support required output from the information processing device.

16. When executed by a processor of a computer, the program causes the computer to: obtaining a detection output from the sensor that represents a movement of a person who is to be certified as needing a level of care or assistance; and outputting information related to a certification of a level of care required or a level of support required by analyzing at least a portion of the detection output corresponding to an unconscious movement of the subject. The subject's unconscious movement is a movement of turning over in bed, The analyzing step includes: Identifying the number of times the detection output changes from a first pattern related to a turning motion to a second pattern as the number of times the subject turns over in their sleep; deriving an index for the subject's turning over movement, which is information relating to the level of care or assistance required, based on the number of times the subject turns over in bed in a given period of time.

17. When executed by a processor of a computer, the program causes the computer to: obtaining a detection output from the sensor that represents a movement of a person who is to be certified as needing a level of care or assistance; and outputting information related to a certification of a level of care required or a level of support required by analyzing at least a portion of the detection output corresponding to an unconscious movement of the subject. The subject's unconscious movement is a transfer movement, The analyzing step includes: determining at least one of a transfer time and a transfer speed based on a time required for the subject to change from a first state to a second state in the detection output; deriving an index for the transfer movement of the subject, which is information regarding the level of care or assistance required, based on at least one of the transfer time and the transfer speed.

18. When executed by a processor of a computer, the program causes the computer to: obtaining a detection output from the sensor that represents a movement of a person who is to be certified as needing a level of care or assistance; and outputting information related to a certification of a level of care required or a level of support required by analyzing at least a portion of the detection output corresponding to an unconscious movement of the subject. The subject's involuntary movement is a movement of standing up, The analyzing step includes: determining a speed of the subject's standing up motion by analyzing a portion of the detection output corresponding to the subject's standing up motion; and deriving an index for the standing up movement, which is information relating to a certification of a level of nursing care or a level of assistance required, based on the speed of the standing up movement.

19. When executed by a processor of a computer, the program causes the computer to: obtaining a detection output from the sensor that represents a movement of a person who is to be certified as needing a level of care or assistance; and outputting information related to a certification of a level of care required or a level of support required by analyzing at least a portion of the detection output corresponding to an unconscious movement of the subject. the detection output includes an image of the subject's room; The subject's involuntary movement is a walking movement, The analyzing step includes: extracting, from the images, an image of a period during which a person moves a certain distance or more in the living room as a portion corresponding to the walking movement of the subject; and deriving an index for walking movement, which is information relating to the level of nursing care or level of assistance required certification, based on the extracted image.

20. A computer-readable recording medium non-temporarily storing a program according to any one of claims 16 to 19.

Images