Information processing device and information processing method

Abstract

To provide an information processing apparatus and an information processing method which appropriately support caregiving of a caregiver for a care receiver.SOLUTION: The information processing apparatus includes: an acquisition unit which acquires information obtained by associating sensor information output from a wearable module with arrangement information identifying a place where a communication device having received the sensor information is arranged; and a processing unit which performs fall determination processing of performing determination about a risk of fall of a care receiver wearing the wearable module on the basis of the arrangement information and the sensor information. The processing unit performs the fall determination processing in accordance with the place where the communication device is arranged, on the basis of the arrangement information and on the basis of the fall determination processing, at least notifies a caregiver terminal of a caregiver for assisting the care receiver or controls a peripheral device placed in the periphery of the care receiver.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and the like. [Background technology]

[0002] A system for use when a caregiver provides care to a care recipient has been known. Patent Document 1 discloses a method for generating information to be provided about the condition of a resident in a living space based on time-varying changes in detected information acquired by a sensor placed in the living space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-18760 Summary of the Invention [Problem to be solved by the invention]

[0004] An information processing device and an information processing method are provided that appropriately support a caregiver in providing care to a person being assisted. [Means for solving the problem]

[0005] One aspect of the present disclosure includes an acquisition unit that acquires information that correlates sensor information output by a wearable module with placement information that identifies the location where the communication device that received the sensor information is placed, and a processing unit that performs a fall determination process, which is a determination of the risk of a person being assisted wearing the wearable module, based on the placement information and the sensor information.The processing unit performs the fall determination process according to the location where the communication device is placed based on the placement information, and is related to an information processing device that performs at least one of the following: a notification on the assistant terminal of the assistant assisting the person being assisted, and control of peripheral devices located around the person being assisted, based on the fall determination process.

[0006] Another aspect of the present disclosure relates to an information processing method including the steps of: acquiring information correlating sensor information output by a wearable module with placement information that identifies the location where the communication device that received the sensor information is placed; performing a fall detection process that determines the risk of a person being assisted wearing the wearable module falling based on the placement information and the sensor information; and performing at least one of a notification on the assistant terminal of the assistant assisting the person being assisted and control of peripheral devices located in the vicinity of the person being assisted based on the fall detection process, wherein in the step of performing the fall detection process, the fall detection process is performed based on the placement information and is appropriate for the location where the communication device is placed. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an example of the configuration of an information processing device. [Figure 2] 10A and 10B are diagrams illustrating an example of the arrangement of a wearable module and a communication device. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of a wearable module. [Figure 4] FIG. 1 illustrates an example of the configuration of a communication device. [Figure 5] 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment of the present invention. [Figure 6] FIG. 1 illustrates an example of the configuration of a server system. [Figure 7A] 10A and 10B are diagrams illustrating examples of display screens used in the registration process of a communication device. [Figure 7B] FIG. 10 is a diagram illustrating an example of a display screen used for pairing. [Figure 7C] 10 is a diagram illustrating an example of a display screen used to associate a wearable module with a person being assisted. FIG. [Figure 8A] FIG. 10 is a diagram illustrating an example of a data structure of access point information. [Figure 8B] FIG. 10 is a diagram illustrating an example of a data structure of module information. [Figure 8C]FIG. 10 is a diagram illustrating an example of a data structure of notification management information. [Figure 9] FIG. 2 is a sequence diagram illustrating processing in the information processing system. [Figure 10A] 10 is an example of a screen that presents the results of the fall determination process. [Figure 10B] 10 is an example of a screen that presents the results of the fall determination process. [Figure 11] 10 is an example of a screen that presents the results of the fall determination process. [Figure 12] FIG. 10 is a diagram illustrating an example of sensor information corresponding to a fall in bed. [Figure 13] FIG. 10 is a diagram illustrating an example of sensor information corresponding to a fall in a wheelchair. [Figure 14] FIG. 10 is a diagram illustrating an example of sensor information corresponding to a fall in a toilet. [Figure 15] FIG. 10 is a diagram illustrating an example of sensor information corresponding to a fall while walking. [Figure 16] FIG. 1 is a diagram illustrating an example of the configuration of a neural network. [Figure 17] FIG. 1 is a diagram illustrating input data and output data in machine learning. [Figure 18A] FIG. 10 is a diagram illustrating a pressure sensor disposed on a wheelchair. [Figure 18B] 1 is a diagram illustrating a cross-sectional structure of a cushion placed on a wheelchair. FIG. [Figure 18C] 3A and 3B are diagrams illustrating an operation unit and a notification unit provided in the control box. [Figure 19A] FIG. 10 is a diagram illustrating a table of peripheral devices. [Figure 19B] FIG. 2 is a diagram illustrating a driving mechanism of the table. [Figure 19C] FIG. 10 is a diagram illustrating a walking device as a peripheral device. [Figure 19D] FIG. 2 is a diagram illustrating a drive mechanism of a walker. [Figure 19E] FIG. 10 is a diagram illustrating a bed as a peripheral device. [Figure 20] FIG. 2 is a diagram illustrating an example of the configuration of a peripheral device. [Figure 21]1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment of the present invention. [Figure 22] FIG. 2 is a sequence diagram illustrating processing in the information processing system. [Figure 23] FIG. 1 is a diagram illustrating tacit knowledge about meals. [Figure 24] FIG. 10 shows a device arranged in a dining situation. [Figure 25] FIG. 1 is a diagram illustrating the relationship between devices, acquired data, and situations. [Figure 26] FIG. 10 is a diagram illustrating a device placed around a bed. [Figure 27] FIG. 10 is a diagram illustrating an example of training data registered by an expert. [Figure 28] FIG. 10 is a diagram illustrating training data that has undergone transparency processing and is displayed during caregiving. [Figure 29] 10 is an example of a display screen of the results of skeleton tracking. [Figure 30] FIG. 10 is a diagram illustrating devices arranged around a wheelchair. [Figure 31A] 10 is an example of a display screen including a person being assisted in an appropriate lateral position and the results of skeletal tracking. [Figure 31B] 10 is an example of a display screen including a person being assisted who is not in an appropriate lateral position and the results of skeletal tracking. [Figure 31C] 10 is an example of a display screen including a person being assisted in a supine position and the results of skeletal tracking. [Figure 32A] 10 is an example of a display screen used for selecting input data in end-of-life care. [Figure 32B] 10 is an example of a display screen for analysis results in end-of-life care. [Figure 32C] 10 is an example of a display screen for analysis results in end-of-life care. [Figure 32D] This is an example of a screen displaying detailed analysis results for end-of-life care. [Figure 33] FIG. 10 is a diagram illustrating an example of a device linked to the determination result of end-of-life care. [Figure 34A]10A and 10B are diagrams illustrating devices and situations in which recommendation display is performed. [Figure 34B] 10 is an example of a screen displaying recommendations. [Figure 34C] 10 is an example of a screen displaying recommendations. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, the present embodiment will be described with reference to the drawings. In the drawings, identical or equivalent elements are designated by the same reference numerals, and duplicate explanations will be omitted. Note that the present embodiment described below does not unduly limit the content described in the claims. Furthermore, not all of the configurations described in the present embodiment are necessarily essential components of the present disclosure.

[0009] The method according to this embodiment provides instructions to a caregiver so that appropriate care can be provided regardless of the caregiver's level of skill by digitizing the "intuition" or "tacit knowledge" of the caregiver, for example, in a task that is performed by the caregiver's "intuition" or "tacit knowledge." Furthermore, the method according to this embodiment is not limited to providing instructions to a caregiver, and does not prevent direct control of tools for care, etc. Specific methods will be described below.

[0010] In the following, an example will be mainly described in which the caregiver is a care worker at a nursing facility and the person being assisted is a user of the nursing facility. For example, each device, such as the communication device 200 described below, may be a device located at the nursing facility. However, the method of this embodiment is not limited to this, and the caregiver may be a nurse or licensed practical nurse at a hospital, or a family member who provides care for the person requiring care at home. Furthermore, assistance in this embodiment may include assistance with actions such as eating and toileting, and personal care in daily life. For example, "assistance" in the following description may be replaced with "care."

[0011] 1. System configuration example FIG. 1 is a diagram showing an example of the configuration of an information processing device 20 according to this embodiment. The information processing device 20 includes an acquisition unit 21 and a processing unit 23. However, the configuration of the information processing device 20 is not limited to that shown in FIG. 1, and modifications such as omitting some components or adding other components are possible. For example, the information processing device 20 may include a storage unit, a display unit, an operation unit, etc., which are not shown. The same applies to FIG. 2 and subsequent figures, in that components can be omitted or added.

[0012] The acquisition unit 21 acquires information in which the sensor information output by the wearable module 100 is associated with location information that identifies the location of the communication device 200 that received the sensor information. The wearable module 100 is a device worn by a person receiving care, and the communication device 200 is a device that is located in a specific location. Note that the wearable module 100 in this embodiment may be expanded to a sensor module that moves with the movement of the person being assisted. For example, if the target is a person being assisted who moves using a cane, walker, wheelchair, or the like, the sensor module may be attached to the cane, walker, wheelchair, or the like. Also, while the present embodiment will be described using an example in which the wearable module 100 includes the acceleration sensor 120, this is not limiting, and the wearable module 100 may include, for example, a gyro sensor, a depth sensor, or the like. In other words, although an example in which the sensor information output by the wearable module 100 represents acceleration will be described below, the sensor information may also represent other information such as angular velocity or depth (distance). The wearable module 100 and the communication device 200 will be described later with reference to Figures 2 to 4. The sensor information and the placement information will also be described in detail later.

[0013] The processing unit 23 performs an intervention determination process, which is a process for determining whether or not intervention is necessary for the person being assisted wearing the wearable module 100, based on the placement information and the sensor information. The intervention here may be intervention by a caregiver, intervention using a care device, or both. If the processing unit 23 determines that intervention is necessary based on the intervention determination process, it causes each device to perform intervention control, which is a control for causing the device to perform intervention. The intervention control may be control for causing the caregiver terminal 400 to issue a notification urging the caregiver to intervene. The caregiver terminal 400 is a device used by a caregiver who assists the person being assisted. Details of the caregiver terminal 400 will be described later using FIG. 5. Alternatively, the intervention control may be control for operating a peripheral device 700 arranged around the person being assisted and the communication device 200. Control of the peripheral device 700 will be described later using FIGS. 19A to 22.

[0014] For example, the processing unit 23 may execute, as the intervention determination process, a fall determination process according to the location where the communication device 200 is placed, based on the placement information. Then, based on the fall determination process, the processing unit 23 performs intervention control including at least one of a notification regarding the fall risk in the caregiver terminal 400 and control of the peripheral device 700. For example, the processing unit 23 may cause the caregiver terminal 400 and the peripheral device 700 to perform intervention control when a fall risk is detected as a trigger.

[0015] According to the method of this embodiment, when multiple communication devices 200 are installed in a care facility or the like, the location of the person being assisted can be estimated depending on which communication device 200 received the sensor information. As a result, the intervention determination process can be performed taking the location into consideration, which improves the processing accuracy. In this case, since the location is identified automatically, for example, the caregiver does not need to set the location, which improves convenience. The method of this embodiment will be described in detail below.

[0016] Note that the sensor information used in the intervention determination process according to this embodiment is not limited to information output by the wearable module 100. For example, the acquisition unit 21 may acquire, as sensor information, information sensed using at least one of a sensor included in the communication device 200 and a sensor included in a device disposed in the vicinity of the communication device 200. This increases the degree of freedom in selecting a device that outputs sensor information, making it possible to acquire a variety of sensor information and perform a variety of intervention determination processes. For example, as will be described later with reference to FIGS. 18A to 18C, the content of the fall determination process may be changed. Furthermore, the intervention determination process is not limited to the fall determination process, and may include a meal determination, which will be described later with reference to FIGS. 23 to 25, a position adjustment determination, which will be described later with reference to FIGS. 26 to 29, and the like.

[0017] For example, communication device 200 may include a camera, and the sensor information may be an image captured by the camera. Furthermore, a device that outputs the sensor information may be pressure sensors Se1 to Se4, which will be described later with reference to Fig. 18A, a throat microphone TM, which will be described later with reference to Fig. 24, or a detection device 810, which will be described later with reference to Fig. 33. That is, the sensor information may be information representing pressure, audio information, or information related to heartbeat or respiration.

[0018] For example, as will be described later, the sensor information used in the intervention determination process may be switched depending on the location, such that the fall determination process is performed using acceleration information from the wearable module 100 in the toilet 600 and while walking, and the fall determination process is performed using pressure information from the pressure sensors Se1 to Se4 while moving in the wheelchair 520. As can be seen from the above explanation, the sensor information output by the wearable module 100 of this embodiment does not necessarily need to be used in all locations and in all intervention determination processes. In other words, some of the intervention determination processes may not use sensor information from the wearable module 100.

[0019] FIG. 2 is a diagram showing an example of the configuration of the information processing system 10 according to this embodiment, and specifically illustrates the arrangement of the wearable module 100 and the communication device 200. As shown in FIG.

[0020] The wearable module 100 is a device worn by a person receiving care from a caregiver. The wearable module 100 is, for example, a plate-shaped device, and may be fixed to the back or chest of the person receiving care. The wearable module 100 may be attached using tape or the like over the clothing of the person receiving care. Alternatively, the wearable module 100 may be attached directly to the skin of the person receiving care. However, the wearable module 100 may be any device that is worn by the person receiving care, and the location where it is attached is not limited to the back or chest. An example configuration of the wearable module 100 will be described later using FIG. 3.

[0021] Communication device 200 is a device that communicates with wearable module 100. Communication device 200 may be a communication device such as a wireless LAN (Local Area Network) access point or router, or may be a general-purpose terminal such as a smartphone. An example of the configuration of communication device 200 will be described later with reference to FIG. 4.

[0022] In the present embodiment, there may be a plurality of communication devices 200. In FIG. 2, six communication devices 200, communication device 200-1 to communication device 200-6, are illustrated as an example of communication devices 200, but the number of communication devices 200 is not limited to this. The plurality of communication devices 200 may be placed in different locations. Places where communication devices 200 are placed include, for example, a bed 510, a wheelchair 520, a walker 540, a toilet 600, a dining room, a living room, and the like. Communication devices 200-1 to 200-6 are each connected to a network NW. The network NW may be a public communication network such as the Internet, or an internal network such as an intranet in a care facility.

[0023] In the example of FIG. 2, communication device 200-1 is placed on bed 510, which is used by the person receiving care for sleeping or the like. For example, a holder of any shape (for example, a holder with a rectangular notch on the inside of the footboard) is attached to a part of bed 510, and communication device 200-1 is held by the holder. Here, bed 510 is, for example, an adjustable bed that can automatically change the angle and height of the bottom, but a bed without such a function may also be used. Note that the bottom is a surface on which a mattress or the like is placed, and the shape is not important, and may be a plate or mesh. Furthermore, communication device 200-1 may be placed on, for example, a wall or floor surface of a room where bed 510 is placed, or on furniture other than bed 510, as long as it can be associated with bed 510. Furthermore, as will be described later with reference to FIG. 26, other devices may be placed around bed 510.

[0024] Communication device 200-2 and communication device 200-3 are arranged in equipment used to assist the person being assisted in moving. Communication device 200-2 is arranged in wheelchair 520. For example, a pocket is provided on the back of wheelchair 520, and communication device 200-2 is placed in this pocket. Pressure sensors Se1 to Se4 may also be provided on cushion 521 arranged on wheelchair 520. Pressure sensors Se1 to Se4 will be described later with reference to FIG. 18A. Communication device 200-3 is arranged in walker 540 used by the person being assisted for movement. Communication device 200-3 is arranged, for example, on a support pole of walker 540.

[0025] The communication device 200-4 is placed in the toilet 600 used by the person being assisted. The communication device 200-4 may be placed in the tank of the toilet 600, or on the floor or wall.

[0026] Communication device 200-5 and communication device 200-6 are placed in a place where the person being assisted leaves their room and engages in activities. Communication device 200-5 is placed in a dining room. For example, as shown in FIG. 2, communication device 200-5 may be placed on a dining room table in a position facing the person being assisted while eating. A throat microphone™ that detects swallowing or choking may also be used during meals. Details of devices used during meals will be described later using FIG. 24. Communication device 200-6 is placed in a place where multiple people can engage in activities, such as a living room or a hall. For example, as shown in FIG. 2, communication device 200-6 may be fixed to a television or the like placed in the living room.

[0027] 2 may be placed in another location, such as a nursing facility. For example, the communication device 200 may be placed in a location in the nursing facility where the person being assisted walks. The location where the communication device 200 is placed can be variously changed, such as a hallway or a staircase. Such a communication device 200 is used, for example, when assisting the walking of a person being assisted who is able to walk independently.

[0028] Furthermore, as will be described later, the intervention determination process according to this embodiment may include processing related to end-of-life care. Based on the results of the processing related to end-of-life care, screens to be described later using Figures 32A to 32D are displayed, and the processing mode is changed based on the output of the detection device 810. The processing related to end-of-life care may be linked to the intervention determination process at each location shown in Figure 2. For example, algorithms and parameters (e.g., thresholds) used in the processing related to end-of-life care may be changed based on the intervention determination process at each location. Alternatively, algorithms and parameters used in the processing at each location may be changed based on the processing related to end-of-life care. Details of the processing related to end-of-life care will be described later.

[0029] The communication between the communication device 200 and the wearable module 100 may be communication using Bluetooth (registered trademark), communication using a wireless LAN defined in IEEE802.11, or communication using another method.

[0030] The communication device 200 may be a device that accepts a communication connection from the wearable module 100 as an access point. Here, the access point refers to a device that directly accepts sensor information from the wearable module 100. Note that directly accepting sensor information specifically means receiving the sensor information without going through another communication device 200. For example, consider a case where the wearable module 100 establishes a connection with the communication device 200-1 using Bluetooth or the like, transmits sensor information to the communication device 200-1 using the connection, and then the communication device 200-1 transfers the sensor information to the communication device 200-2. In this example, the communication device 200-1 is an access point for the wearable module 100, but the communication device 200-2 is not an access point.

[0031] For example, communication device 200 may be a Bluetooth central and wearable module 100 may be a Bluetooth peripheral. Alternatively, communication device 200 may be an AP (access point) in a wireless LAN and wearable module 100 may be an STA (station) in the wireless LAN. As can be seen from the above examples, the access point in this embodiment is not limited to a wireless LAN AP, but broadly includes devices that communicate directly with wearable module 100 using other communication methods.

[0032] The wearable module 100 may change the communication device 200 with which it communicates depending on its location. For example, the location of the wearable module 100 changes as the care recipient wearing the wearable module 100 moves. When the communication device 200 is within a predetermined distance, the wearable module 100 attempts to connect to the communication device 200. The predetermined distance may be a distance at which Bluetooth advertisement packets can be transmitted and received, a distance at which a wireless LAN SSID (Service Set Identifier) ​​broadcast can be transmitted and received, or a distance defined by another communication method. Furthermore, being sufficiently close to the communication device 200 may be used as a condition for connection. For example, the connection between the wearable module 100 and the communication device 200 may be established on the condition that the received radio wave strength in transmitting and receiving advertisement packets or SSID broadcasts is equal to or greater than a predetermined threshold. Furthermore, when a plurality of communication devices 200 are detected within a predetermined distance range from the wearable module 100, the wearable module 100 may select a communication device 200 to connect to based on the received radio wave intensity.

[0033] 3 is a diagram showing an example of the configuration of wearable module 100. Wearable module 100 includes a control unit 110, an acceleration sensor 120, a communication module 130, and a storage unit 140. Wearable module 100 may further include components (not shown), such as a temperature sensor.

[0034] The control unit 110 controls each unit of the wearable module 100, such as the acceleration sensor 120 and the communication module 130. The control unit 110 may be a processor. The processor here may be any of various types, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a DSP (Digital Signal Processor).

[0035] The acceleration sensor 120 is a sensor that detects acceleration and outputs sensor information representing the detection results. For example, the acceleration sensor 120 may be a three-axis acceleration sensor that detects translational acceleration along three axes. In this case, the sensor information is a collection of acceleration values ​​along the x-, y-, and z-axes. For example, when the wearable module 100 is worn on the chest, the x-axis may correspond to the front-to-back direction of the person being assisted, the y-axis may correspond to the left-to-right direction, and the z-axis may correspond to the vertical direction. However, the acceleration sensor 120 may also be a six-axis acceleration sensor that detects translational acceleration along three axes and angular acceleration around each axis, and various modifications are possible.

[0036] The communication module 130 is an interface for performing communication via a network, and includes, for example, an antenna, an RF (radio frequency) circuit, and a baseband circuit. The communication module 130 may operate under the control of the control unit 110, or may include a processor for communication control that is different from the control unit 110. As described above, the communication module 130 may perform communication using a wireless LAN, may perform communication using Bluetooth, or may perform communication using another method.

[0037] The communication module 130 transmits the sensor information output by the acceleration sensor 120 to the communication device 200. As described above, when the communication device 200 is present within a predetermined distance, for example, the communication module 130 establishes a connection with the communication device 200 and then transmits the sensor information to the communication device 200 as the connection destination.

[0038] The storage unit 140 is a work area for the control unit 110, and is realized by various types of memory such as SRAM, DRAM, and ROM (Read Only Memory). The storage unit 140 may store sensor information acquired by the acceleration sensor 120. For example, if the communication module 130 is unable to transmit sensor information to the communication device 200, the storage unit 140 may store the untransmitted sensor information. In this case, the storage unit 140 may also store the reason why the sensor information could not be transmitted to the communication device 200 and the details of the error. When communication with the communication device 200 becomes possible, the communication module 130 transmits the sensor information accumulated in the storage unit 140 to the communication device 200. The communication module 130 may also transmit the reason why the transmission could not be transmitted and the details of the error in association with the sensor information.

[0039] 4 is a diagram showing an example of the configuration of the communication device 200. The communication device 200 includes a processing unit 210, a storage unit 220, a communication unit 230, a display unit 240, and an operation unit 250, for example.

[0040] The processing unit 210 is configured by the following hardware. The hardware can include at least one of a circuit for processing digital signals and a circuit for processing analog signals. For example, the hardware can be configured by one or more circuit devices or one or more circuit elements mounted on a circuit board. The one or more circuit devices are, for example, an integrated circuit (IC), a field-programmable gate array (FPGA), etc. The one or more circuit elements are, for example, a resistor, a capacitor, etc.

[0041] The processing unit 210 may also be implemented by the following processor. The communication device 200 of this embodiment includes a memory that stores information and a processor that operates based on the information stored in the memory. The information may be, for example, a program and various data. The processor includes hardware. Various types of processors, such as a CPU, a GPU, or a DSP, may be used as the processor. The memory may be a semiconductor memory such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory, or may be a register, a magnetic storage device such as a hard disk drive (HDD), or an optical storage device such as an optical disk drive. For example, the memory stores computer-readable instructions, and the processor executes the instructions to realize the functions of the processing unit 210 as processing. The instructions may be instructions from an instruction set that constitutes a program, or instructions that instruct the hardware circuitry of the processor to operate.

[0042] The storage unit 220 is a work area for the processing unit 210, and is realized by various types of memory such as SRAM, DRAM, and ROM.

[0043] The communication unit 230 is an interface for communicating via a network, and includes, for example, an antenna, an RF circuit, and a baseband circuit. The communication unit 230 performs, for example, a first communication with the wearable module 100 and a second communication with the server system 300, which will be described later with reference to FIG. 5.

[0044] The communication methods of the first communication and the second communication may be the same or different. For example, when the first communication and the second communication are the same communication method, the communication unit 230 may include one wireless communication chip and use the one wireless communication chip in a time-division manner, or may include two wireless communication chips of the same communication method. When the first communication and the second communication are different communication methods, the communication unit 230 may include two wireless communication chips of different communication methods. The first communication may be communication using Bluetooth as described above, or may be communication using a wireless LAN. The second communication may be communication using a wireless LAN, or may be communication using a mobile communication network such as LTE (Long Term Evolution) or 5G.

[0045] The first communication using Bluetooth may be a beacon method or a connection method. The beacon method is a method of transmitting data at predetermined time intervals (e.g., one minute), while the connection method is a method of transmitting data triggered by a user operation. The user operation may be, for example, pressing an update button. The update button may be provided on the wearable module 100 or may be displayed on the display unit 240 of the communication device 200. Alternatively, the update button may be displayed on a display unit other than the communication device 200, such as the caregiver terminal 400 described below, and when the operation is performed, a message to that effect may be transmitted to the wearable module 100 or the communication device 200. In this way, multiple methods with different data transmission and reception timings may be used in the first communication. The same applies when a method other than Bluetooth is used for the first communication. Furthermore, multiple methods with different data transmission and reception timings may be used for the second communication as well.

[0046] The display unit 240 is an interface that displays various information, and may be a liquid crystal display, an organic EL display, or any other type of display.

[0047] The operation unit 250 is an interface that accepts user operations. The operation unit 250 may be a button or the like provided on the communication device 200. The display unit 240 and the operation unit 250 may be a touch panel that is integrally configured.

[0048] Communication device 200 may also include components not shown in FIG. 4, such as a light-emitting unit, a vibration unit, and a sound output unit. The light-emitting unit is, for example, an LED (light emitting diode) and provides notification by emitting light. The vibration unit is, for example, a motor and provides notification by vibration. The sound output unit is, for example, a speaker and provides notification by sound. Communication device 200 may also include various sensors, such as motion sensors such as an acceleration sensor or a gyro sensor, an imaging sensor, a GPS (Global Positioning System) sensor, etc.

[0049] By using the information processing system 10 shown in Figure 2, it is possible to estimate the condition of the person being assisted depending on which communication device 200 the wearable module 100 is connected to, or more specifically, which communication device 200 the sensor information of the wearable module 100 is transmitted to.

[0050] For example, when sensor information is transmitted to communication device 200-1, it is estimated that the person being assisted is lying or sitting on bed 510. When sensor information is transmitted to communication device 200-2 or communication device 200-3, it is estimated that the person being assisted is moving using a wheelchair 520 or a walker 540. When sensor information is transmitted to communication device 200-4, it is estimated that the person being assisted is in the toilet. When sensor information is transmitted to communication device 200-5 or communication device 200-6, it is estimated that the person being assisted is performing an activity in the corresponding place, such as the dining room or living room.

[0051] Once the situation is estimated, the assistance to be provided can also be estimated. For example, if the patient is near the bed 510, assistance such as patrolling while sleeping, changing diapers, adjusting position to prevent bedsores, and assistance transferring to a wheelchair 520 are provided. If the patient is in the wheelchair 520, assistance with moving using the wheelchair 520 and assistance with eating are provided. If the patient is in the toilet, assistance with excretion in the toilet is provided. If the patient is walking, assistance such as preventing falls is provided.

[0052] As a result, it becomes possible to identify tacit knowledge for appropriately performing the estimated assistance and to notify the caregiver of specific actions to use the tacit knowledge. For example, when performing a fall detection process using the acceleration sensor 120 of the wearable module 100, the detection can be performed using different criteria depending on the location. Furthermore, when performing an intervention detection process using sensor information from the communication device 200 or another device, control may be performed to activate sensors included in the communication device 200 or the device. This allows appropriate sensor information to be obtained depending on the location, thereby improving the accuracy of the detection. However, the control of sensor activation / deactivation is not limited to being performed automatically based on the connection status between the wearable module 100 and the communication device 200; some or all of the sensors may be manually activated. As such, the method of this embodiment can automatically estimate the location of the person being assisted, making it possible to support assistance according to the situation without manually setting the specific situation.

[0053] For example, caregivers at nursing homes must provide various types of care, such as those described above, to numerous care recipients, and operate on a very tight schedule. Furthermore, when irregular events, such as fecal incontinence or falls, occur, it is difficult to complete the original schedule. Caregivers may be forced to postpone some care tasks according to priorities. Therefore, even if a system supporting caregivers is provided and the system is configured to customize the support content according to the situation, caregivers do not have the time to customize each and every one of them. For example, as will be described later with reference to Figures 12 to 15, in a process that supports caregivers by indicating the risk of falls, the processing accuracy can be improved by setting individual thresholds for falls in bed 510, falls in wheelchair 520, falls in toilet 600, and falls while walking. However, requiring caregivers to perform such individual settings is undesirable from the perspective of convenience.

[0054] In this regard, the method of this embodiment allows for automated settings based on the communication status between the wearable module 100 and the communication device 200, thereby appropriately supporting the caregiver's assistance while minimizing the burden on the caregiver.

[0055] Fig. 5 is a diagram showing a detailed configuration example of the information processing system 10 according to this embodiment. The information processing system 10 may include a server system 300 and a caregiver terminal 400 in addition to the wearable module 100 and the communication device 200 shown in Fig. 2. Note that this example shows a case where a notification is made at the caregiver terminal 400 as a result of the intervention determination process.

[0056] The server system 300 communicates with the communication device 200 via, for example, the network NW shown in FIG. 2. The network NW here may be a public communication network such as the Internet. In this case, information collected by the communication device 200 from the wearable module 100 is processed using the cloud. Alternatively, the network NW may be an internal network such as an intranet of a nursing facility. In this case, the server system 300 is, for example, a management server installed in the nursing facility.

[0057] The server system 300 may be a single server or may include multiple servers. For example, the server system 300 may include a database server and an application server. The database server stores various data such as data transmitted from the communication device 200 and processing algorithms. The application server corresponds to the processing unit 310 described below and performs processes such as steps S106 to S108 in FIG. 9. The multiple servers here may be physical servers or virtual servers. If a virtual server is used, the virtual server may be provided on a single physical server or may be distributed across multiple physical servers. As described above, the specific configuration of the server system 300 in this embodiment can be modified in various ways.

[0058] 6 is a diagram showing an example of the configuration of the server system 300. The server system 300 includes a processing unit 310, a storage unit 320, and a communication unit 330, for example.

[0059] The processing unit 310 is configured by hardware including at least one of a circuit for processing digital signals and a circuit for processing analog signals. For example, the hardware can be configured by one or more circuit devices mounted on a circuit board or one or more circuit elements.

[0060] Furthermore, the processing unit 310 may be realized by a processor including hardware. The server system 300 includes a processor and a memory. The processor may be of various types, such as a CPU, a GPU, or a DSP. The memory may be a semiconductor memory, a register, a magnetic storage device, or an optical storage device. For example, the memory stores computer-readable instructions, and the processor executes the instructions to realize the functions of the processing unit 310 as processing.

[0061] The storage unit 320 is a work area for the processing unit 310, and is realized by various types of memory such as SRAM, DRAM, and ROM.

[0062] The communication unit 330 is an interface for communicating via a network, and includes, for example, an antenna, an RF circuit, and a baseband circuit. The communication unit 330 communicates with, for example, the communication device 200 and the caregiver terminal 400. The communication unit 330 may also communicate with the peripheral device 700, as will be described later with reference to FIG. 21 . Note that communication with at least one of the caregiver terminal 400 and the peripheral device 700 may be performed via the communication device 200, and various modifications of the specific connection mode are possible.

[0063] The caregiver terminal 400 is a device used by a caregiver in a care facility or the like, and is a device used to present information to the caregiver or for the caregiver to input information. For example, the caregiver terminal 400 may be a device carried by or worn by the caregiver.

[0064] For example, as shown in FIG. 5, the caregiver terminal 400 may include a mobile terminal device 410 and a headset 420. The mobile terminal device 410 is, for example, a smartphone, but may also be another portable device. The headset 420 is a device that can be worn by the caregiver and includes, for example, earphones or headphones and a microphone. The headset 420 may also be changed to another wearable device, such as a glasses-type device or a wristwatch-type device. Note that glasses-type devices may include AR (Augmented Reality) glasses and MR (Mixed Reality) glasses. The caregiver terminal 400 may also be another device, such as a PC (Personal Computer).

[0065] 5 illustrates two sets of caregiver terminals 400, each including a mobile terminal device 410 (mobile terminal devices 410-1 and 410-2) and a headset 420 (headsets 420-1 and 420-2). However, the number of caregiver terminals 400 is not limited to two. Furthermore, the types and number of devices constituting each caregiver terminal 400 are not limited to the example in FIG. 5, and various modifications are possible.

[0066] An example of the operation of the information processing system 10 will be described. As described above, the wearable module 100 transmits sensor information to one of the communication devices 200. The communication device 200 associates the received sensor information with placement information that identifies the location where the communication device 200 is placed. The placement information here is information that can identify the location where the communication device 200 is placed, and may be flag information or identification information of the communication device 200.

[0067] The flag information is, for example, 4-bit data representing toilet 600, bed 510, wheelchair 520, and walking, with one bit having a value of 1 and the remaining three bits having a value of 0. However, the data format of the flag information is not limited to this, and may be 2-bit data that distinguishes between toilet 600, bed 510, wheelchair 520, and walking using four values ​​of 00, 01, 10, and 11, or may be data in another format. Furthermore, because the locations where communication device 200 is placed are not limited to four, the flag information may be data including more bits.

[0068] For example, when communication device 200 is a smartphone, the identification information of communication device 200 is information related to a SIM (Subscriber Identity Module). However, the identification information may be any information that uniquely identifies communication device 200, and other information such as a MAC address or a serial number may also be used.

[0069] As will be described later with reference to FIG. 8A , in the method of the present embodiment, access point information in which identification information of communication device 200 is associated with flag information that identifies a location may be stored. In this case, it is possible to identify the flag information based on the identification information of communication device 200. That is, both the flag information and the identification information of communication device 200 are information that can identify the location of communication device 200, and are included in the location information of the present embodiment. For example, flag information may be associated with sensor information in communication device 200. Alternatively, identification information of communication device 200 may be associated with sensor information in communication device 200, and processing to identify flag information from the identification information may be performed in server system 300. The latter case will be described below as an example.

[0070] The processing unit 310 of the server system 300 obtains information to support assistance to the care recipient wearing the wearable module 100 based on the identification information and sensor information. Specifically, the processing unit 310 performs a determination based on the tacit knowledge of an expert and outputs information that enables a less skilled caregiver to provide care similar to that of an expert. As an example, the sensor information may be acceleration information, and the processing unit 310 may perform a fall determination process to determine the risk of the care recipient falling. Note that the fall determination process here may include a process for determining the presence or absence and the level of the risk of falling, and is not limited to a process for detecting the fall itself. The fall determination process of this embodiment may also include a process for determining a state prior to the occurrence of a fall, such as a posture determination process for determining whether the care recipient is in a poorly balanced position that makes them more susceptible to falling. In this case, in this embodiment, the fall determination process can be performed based on the identification information of the communication device 200, depending on the location where the communication device 200 is installed, thereby improving accuracy. 2, the location-dependent fall detection process includes detection of a fall in bed 510, detection of a fall in wheelchair 520, detection of a fall in toilet 600, and detection of a fall while walking. Details of the process will be described later.

[0071] When a risk of falling is detected based on the fall determination process, the processing unit 310 notifies the caregiver terminal 400 of the risk of falling via the communication unit 330. The specific notification will be described later.

[0072] 1 corresponds to, for example, the server system 300. That is, the acquisition unit 21 that acquires the sensor information and the placement information in association with each other may be an interface that acquires the data (for example, the communication unit 330). Furthermore, the processing unit 23 of the information processing device 20 may be the processing unit 310 shown in FIG.

[0073] For example, if the server system 300 is a device installed on an external network of a nursing care facility, it becomes possible to manage information associating sensor information with location information using the cloud. For example, by integrating information from multiple nursing care facilities, it is easy to improve processing accuracy. Furthermore, since the communication device 200 does not need to execute the fall detection process, it is possible to reduce the processing load on the communication device 200. Similarly, even if the server system 300 is an administration server or the like installed on an internal network of a nursing care facility, it is possible to centralize processing in the administration server, thereby reducing the processing load on the communication device 200.

[0074] However, the information processing device 20 of this embodiment is not limited to the server system 300. For example, the information processing device 20 may be a communication device 200. The processing unit 210 of the communication device 200 may include an association processing unit that associates the sensor information acquired from the wearable module 100 with identification information or flag information of the communication device 200 itself, and a fall determination processing unit that performs fall determination processing based on the associated information. In this case, the acquisition unit 21 of the information processing device 20 may be the association processing unit, and the processing unit 23 of the information processing device 20 may be the fall determination processing unit.

[0075] In this way, it becomes possible to execute processing using sensor information in the communication device 200. In this case, it is possible to omit the server system 300. As a result, for example, a closed information processing system 10 can be constructed within a nursing care facility without using an external cloud, which makes system construction easier and reduces security risks such as data leakage. Furthermore, since there is no need to set up a dedicated management server within the nursing care facility, system construction is easier.

[0076] For example, the communication device 200 according to this embodiment may be a smartphone. In this case, both the communication device 200 and the caregiver terminal 400 can be realized by a smartphone. That is, there is less need to introduce dedicated equipment when constructing the information processing system 10 according to this embodiment. For example, even if a nursing care facility does not have a Wi-Fi (registered trademark) environment, the method according to this embodiment can be easily applied.

[0077] Note that when communication device 200 is information processing device 20, the information processing device 20 is not limited to the communication device 200 that directly acquired the sensor information. For example, communication device 200-1 may receive sensor information from wearable module 100, associate the sensor information with placement information, and then transmit the associated information to another communication device 200 such as communication device 200-2. Then, processing unit 210 of communication device 200-2 may perform fall determination processing based on the information that associates the placement information with the sensor information.

[0078] In this case, the information processing device 20 may be the communication device 200-2, and the acquisition unit 21 of the information processing device 20 may be an interface (e.g., the communication unit 230 of the communication device 200-2) that transmits and receives data to and from the communication device 200-1. Furthermore, the processing unit 23 of the information processing device 20 may be the processing unit 210 of the communication device 200-2. Alternatively, the information processing device 20 may be realized by distributed processing of the association processing unit of the communication device 200-1 and the fall determination processing unit of the communication device 200-2. For example, the multiple communication devices 200 shown in FIG. 2 and FIG. 5 may each be a device that functions as the information processing device 20. For example, a program that executes the fall determination process or the like may be provided to each communication device 200 as application software for a smartphone.

[0079] Furthermore, the information processing device 20 is not limited to either the server system 300 or the communication device 200, and may be realized by distributed processing of the server system 300 and the communication device 200. The configurations described above are examples of the information processing system 10 and the information processing device 20, and various modifications of the specific configurations are possible.

[0080] The technique of this embodiment can also be applied to an information processing method that executes the following steps. The information processing method includes the steps of acquiring information in which sensor information output by the wearable module 100 is associated with placement information that identifies the location where the communication device 200 that received the sensor information is placed, performing a fall determination process that determines the risk of a person being assisted wearing the wearable module 100 falling based on the placement information and the sensor information, and causing at least one of a notification on the caregiver terminal 400 of the caregiver assisting the person being assisted and control of peripheral devices 700 located around the person being assisted to be performed based on the fall determination process. Furthermore, in the step of performing the fall determination process, the information processing method performs the fall determination process according to the location where the communication device 200 is placed based on the placement information.

[0081] In addition, part or all of the processing performed by the information processing device 20 of this embodiment may be realized by a program. The processing performed by the information processing device 20 is processing performed by the processing unit 210 or the processing unit 310, for example.

[0082] The program according to this embodiment can be stored in, for example, a non-transitory information storage device (information storage medium), which is a medium readable by a computer. The information storage device can be realized, for example, by an optical disc, a memory card, a HDD, or a semiconductor memory. The semiconductor memory is, for example, a ROM. The processing unit 210 and the like perform various processes of this embodiment based on the programs stored in the information storage device. That is, the information storage device stores programs for causing a computer to function as the processing unit 210 and the like. A computer is a device equipped with an input device, a processing unit, a storage unit, and an output unit. Specifically, the program according to this embodiment is a program for causing a computer to execute each step described below using FIG. 9 and the like.

[0083] 2. Fall detection process Next, a fall determination process will be described in detail as an example of the intervention determination process. Note that, although the fall determination process will be mainly described below as being based on sensor information output by acceleration sensor 120 of wearable module 100, as will be described later with reference to FIG. 18A, the fall determination process may also be performed based on sensor information output from other devices, such as pressure sensors Se1 to Se4.

[0084] 2.1 Processing flow The processing of this embodiment will be described below. In this embodiment, a registration phase for registering information required for processing may be executed first, followed by a usage phase corresponding to the actual assistance situation. The processing of the registration phase will be described below with reference to FIGS. 7A to 8C, and the processing of the usage phase will be described with reference to FIG. 9. Note that, although an example will be described below in which the information processing device 20 is the server system 300, as described above, some or all of the processing may be executed in the communication device 200.

[0085] In the method of this embodiment, a first registration process is performed to associate the communication device 200 with its location, and a second registration process is performed to connect the communication device 200 with the wearable module 100. The second registration process may be Bluetooth pairing, or a process of storing the SSID and password of a wireless LAN in the wearable module 100. The second registration process may also include a process of registering the wearable module 100 that has already been paired with the communication device 200 on the system.

[0086] 7A and 7B are examples of UIs (User Interfaces) used for registration, and are examples of registration screens displayed on display unit 240 by, for example, processing unit 210 of communication device 200 operating in accordance with application software. Fig. 7A is the screen used in the first registration process, and Fig. 7B is the screen used in the second registration process.

[0087] 7A and 7B may include an object OB1 that is an access point registration button and an object OB2 that is a sensor pairing setting button at the bottom of the screen. When the user selects object OB1, the screen of FIG. 7A is displayed, and when the user selects object OB2, the screen of FIG. 7B is displayed. However, the screen configuration used in the registration phase is not limited to that of FIG. 7A or 7B, and various modifications are possible.

[0088] When performing the first registration process, for example, the user installs the application software on a device to be used as the communication device 200 according to this embodiment, and then launches the application software to display the screen shown in FIG. 7A on the display unit 240. Then, with object OB1 selected, the user selects the location where the communication device 200 will be used. For example, the screen shown in FIG. 7A may include text prompting the user to select a location, such as "Please select the location where this terminal will be installed," along with four radio buttons corresponding to toilet, wheelchair, bed, and other options. Any one of the four radio buttons can be selected.

[0089] For example, when the user selects a radio button corresponding to a toilet, a process is executed to register the communication device 200 being operated by the user as the communication device 200 located in the toilet. The same applies when a radio button other than the toilet is selected, and a process is executed to register the communication device 200 being operated by the user as the communication device 200 located in the selected location.

[0090] 7A, when selecting "Other," the user may enter additional information about the location using a text box. "Other" specifically refers to the location where the user walks. For example, the user may enter text that indicates the specific location where the target communication device 200 is located, such as a hallway, stairs, or dining room.

[0091] For example, when an operation to select a location is performed, the communication device 200 associates the identification information of the communication device 200 with information specifying the location selected by the user and transmits the associated information to the server system 300. The server system 300 stores the received information in the storage unit 320 as access point information. FIG. 8A is a diagram illustrating access point information managed by the server system 300. As shown in FIG. 8A, the access point information is information in which identification information identifying the communication device 200 according to the present embodiment is associated with the location where the communication device 200 is located. More specifically, the access point information may be information in which the identification information of the communication device 200 is associated with flag information. The access point information may include other information such as information specifying the facility where the communication device 200 is located, information specifying the registered user, and the date and time of registration. The access point information may also include additional information entered using the text box described above. In this manner, it is possible to associate and manage the device used as the communication device 200 according to the present embodiment with the location where the device is located. The flag information indicating the location input using FIG. 7A may be stored in the storage unit 220 of the communication device 200.

[0092] When performing the second registration process, for example, the user starts the application software on the device used as the communication device 200 according to this embodiment, causing the screen shown in Fig. 7B to be displayed on the display unit 240. Then, with the object OB2 selected, the user selects the wearable module 100 to be paired.

[0093] For example, the second registration process may be executed when a wearable module 100 is newly installed in a care facility or the like. The user turns on the power of the wearable module 100 to make it ready for pairing with the communication device 200. For example, when Bluetooth is used, the user puts the wearable module 100 into a state waiting for pairing.

[0094] 7B includes an object OB3 representing a switch that controls on / off of Bluetooth of communication device 200, and an area RE1 that displays a list of sensors that are candidates for pairing. For example, a user uses object OB3 to turn on Bluetooth of communication device 200. This causes communication unit 230 of communication device 200 to search for nearby devices that can be paired, and displays the search results in area RE1.

[0095] As shown in Fig. 7B, area RE1 may include the name (sensor name) of a connectable wearable module 100 and information indicating the connection status between the wearable module 100 and communication device 200. In the example of Fig. 7B, a search has found a wearable module 100 that includes at least two sensors, sensorXXX and sensorYYY. The communication device 200 is already connected to sensorXXX and is not yet connected to sensorYYY.

[0096] In this case, the communication device 200 displays "Connected" in the status field for sensorXXX. Furthermore, the communication device 200 displays "Not Connected" in the status field for sensorYYY. Furthermore, as shown in FIG. 7B , the area RE1 may include objects for changing the connection status of each wearable module 100. For example, the display unit 240 of the communication device 200 displays an object OB4 representing a disconnect button for disconnecting sensorXXX that is "Connected." Furthermore, the display unit 240 displays an object OB5 representing a connect button for establishing a connection for sensorYYY that is "Not Connected." The communication unit 230 of the communication device 200 controls communication with each wearable module 100 based on the results of operations on these objects. For example, when an operation to select object OB4 is performed, the communication unit 230 disconnects the connection with sensorXXX. When an operation to select object OB5 is performed, the communication unit 230 executes a pairing sequence with sensorYYY.

[0097] This makes it possible to control the communication state between the communication device 200 and the wearable module 100. For example, when a new wearable module 100 (e.g., sensorYYY) is added, the communication device 200 and the wearable module 100 become capable of communicating with each other, and the communication device 200 becomes able to receive sensor information from the target wearable module 100 as an access point. However, the timing at which the second registration process is performed is not limited to when the wearable module 100 is installed, and can be performed at any timing.

[0098] Furthermore, in relation to the second registration process, the processing unit 310 of the server system 300 may perform a process of storing the wearable module 100 that has been paired with the communication device 200. For example, when the connected / disconnected state changes as a result of a selection operation of the object OB4 or the object OB5, the communication device 200 may transmit, to the server system 300, identification information of the communication device 200 and identification information for identifying the wearable module 100 that has been paired with the communication device 200.

[0099] The server system 300 stores the identification information of the communication device 200 in association with the identification information of the wearable module 100. In this way, it becomes possible to manage a wearable module 100 that has been newly added to the information processing system 10, and to manage communication devices 200 that the wearable module 100 can access.

[0100] 2, when determining whether the person being assisted corresponds to bed 510, wheelchair 520, walker 540, toilet 600, dining room, living room, or other (e.g., walking), it is desirable that wearable module 100 be able to communicate with all of communication devices 200-1 to 200-6. For example, when a new wearable module 100 is introduced into a nursing facility, each of communication devices 200-1 to 200-6 may execute the second registration process for pairing with the wearable module 100. However, the second registration process for all communication devices 200 is not essential, and the second registration process for some of communication devices 200 may be omitted. For example, in the case of a person being assisted who does not require much assistance in toilet 600, the second registration process for communication device 200-4 may be omitted.

[0101] Alternatively, the second registration process may be executed by transmitting connection information used for connecting with communication device 200 to wearable module 100. For example, a device such as server system 300 may collectively manage the SSIDs and passwords of communication devices 200, and transmit the SSIDs and passwords to a newly registered wearable module 100. For example, when wearable module 100 is registered on the system by pairing the wearable module 100 with communication device 200-1, server system 300 may transmit the SSIDs and passwords of communication devices 200-2 to 200-6 to the wearable module 100. In this way, it is possible to reduce the burden on the user in registration.

[0102] In this embodiment, a third registration process may be performed to register the identification information of the wearable module 100 in association with the person being assisted who wears the wearable module 100. The third registration process may be performed, for example, by the caregiver using the caregiver terminal 400 (portable terminal device 410). Here, the application software installed in the communication device 200 and the application software installed in the portable terminal device 410 may be the same or different.

[0103] 7C is an example of a screen used in the third registration process, and is displayed on the display unit of mobile terminal device 410 as described above. The screen shown in FIG. 7C may include areas RE2 to RE4. Area RE2 has buttons arranged therein for selecting the location where communication device 200 will be placed. In the example of FIG. 7C, four buttons corresponding to toilet 600, wheelchair 520, bed 510, and others are displayed.

[0104] When a location is selected using area RE2, area RE3 displays a list of wearable modules 100 that are paired with the communication devices 200 located at the selected location. For example, if information about the paired communication devices 200 and wearable modules 100 is stored in the server system 300 in connection with the second registration process, the list of wearable modules 100 displayed in area RE3 is determined based on that information. Area RE3 also displays information about the person being assisted that is associated with the wearable module 100 based on the module information.

[0105] FIG. 8B is an example of module information. The module information includes identification information for identifying the wearable module 100 and information for specifying the person being assisted who uses the wearable module 100. The identification information for the wearable module 100 is, for example, the MAC address of the communication module 130, but other information may be used. The information for specifying the person being assisted may be the name of the person being assisted or other information such as an ID. The module information may also include other information such as identification information for the paired communication device 200, information for specifying the facility where the wearable module 100 is installed, information for specifying the registered user, and the date and time of registration.

[0106] As shown in Fig. 7C, for wearable modules 100 that are already associated with a person being assisted (sensorXXX, sensorYYY in the example of Fig. 7), information about the corresponding person being assisted (e.g., user names AAA, BBB) is displayed based on the module information. On the other hand, for wearable modules 100 that are not associated with a person being assisted, "unregistered" is displayed.

[0107] The user selects one of the wearable modules 100 in area RE3, and then uses area RE4 to change or newly register the person being assisted associated with that wearable module 100. For example, area RE4 displays information about the assisted person who is a facility user. As an example, area RE4 displays a list of assisted persons. When one of the assisted persons is selected, detailed information about the assisted person shown in FIG. 7C is displayed. For example, area RE4 includes a registration button. When the user selects the registration button, a process is performed to associate the wearable module 100 selected in area RE3 with the assisted person displayed in area RE4. Specifically, the mobile terminal device 410 associates the identification information of the wearable module 100 with the information about the assisted person and transmits the associated information to the server system 300. The processing unit 310 of the server system 300 updates the module information in FIG. 8B based on the information transmitted from the mobile terminal device 410.

[0108] In this embodiment, information about a single wearable module 100 may be managed as different data depending on the location. For example, there may be sensorZZZ data registered in association with a toilet and sensorZZZ data registered in association with a wheelchair. However, because these sensorZZZ represent the same wearable module 100, they are considered to be associated with the same person being assisted. Therefore, even if the communication devices 200 to be paired are different, the third registration process may be performed collectively for the same wearable module 100. For example, FIG. 7C illustrates an example in which the communication device 200 located in the toilet is paired with sensorZZZ, and sensorZZZ is associated with CCC. In this case, as long as the same sensorZZZ is used, the process of associating sensorZZZ with CCC is performed collectively, even if the communication device 200 to be paired is located outside the toilet.

[0109] Furthermore, considering the notification to the caregiver terminal 400 described above using Figure 5, the method of this embodiment may perform a fourth registration process in which the wearable module 100 and the caregiver terminal 400 to which information based on the wearable module 100 is to be notified are associated and registered.

[0110] For example, the user uses the mobile terminal device 410 to transmit information associating a person being assisted with a caregiver terminal 400 to which information about the person being assisted will be notified. For example, the application software may communicate with the server system 300 to display a list of wearable modules 100 registered in the second registration process or people being assisted associated with the wearable modules 100. For example, a given caregiver may select one or more wearable modules 100 from the list while logging in to the system using their own ID and password. The mobile terminal device 410 transmits information identifying the logged-in caregiver and information identifying the selected wearable modules 100 to the server system 300. Furthermore, if a caregiver uses multiple caregiver terminals 400, an input may be made to specify the caregiver terminal 400 to be notified.

[0111] The processing unit 310 of the server system 300 performs processing to update the notification management information shown in FIG. 8C based on the received information. As shown in FIG. 8C, the notification management information includes identification information of the wearable module 100 and identification information of the caregiver terminal 400 to which information based on the wearable module 100 is notified. The identification information of the caregiver terminal 400 may be SIM information, a MAC address, or other information. The identification information of the wearable module 100 may be replaced with information about the corresponding person being assisted. The identification information of the caregiver terminal 400 may be replaced with information about the corresponding caregiver. The notification management information may also include additional information that indicates more detailed notification conditions.

[0112] As described above, the storage unit 320 of the server system 300 may store access point information, module information, notification management information, etc. based on the first to fourth registration processes. By using this information, the processing unit 310 can appropriately manage the devices included in the information processing system 10 shown in Fig. 2 and Fig. 5.

[0113] 9 is a sequence diagram illustrating the processing of the information processing system 10 in the usage phase after the completion of the above registration phase. First, in step S101, the wearable module 100 determines whether there is a connectable communication device 200 in the vicinity. The processing of step S101 may be, for example, processing to execute a sequence including sending and receiving advertising packets in Bluetooth, or processing to execute a sequence including SSID scanning in wireless LAN.

[0114] If a connectable communication device 200 is present, in step S102, a connection is established between the wearable module 100 and the communication device 200. The information necessary for establishing the connection has already been acquired, for example, in the second registration process described above. Therefore, if a registered communication device 200 is present within a predetermined distance, the wearable module 100 establishes a connection with that communication device 200.

[0115] In step S103, the wearable module 100 transmits sensor information detected by the acceleration sensor 120 to the communication device 200 using the communication module 130. The process of step S103 is executed periodically, for example, at predetermined intervals. Note that the sensor information may include identification information that identifies the wearable module 100 that is the sender of the sensor information.

[0116] In step S104, communication device 200 performs a process of associating the sensor information received in step S103 with the identification information of communication device 200. Then, in step S105, the associated information is transmitted to server system 300. For example, communication device 200 transmits the identification information of wearable module 100, the sensor information, and the identification information of communication device 200 in association with each other.

[0117] In step S106, the server system 300 executes a determination process according to the location of the person being assisted based on the received information. For example, the processing unit 310 identifies flag information based on the acquired identification information of the communication device 200 and the access point information shown in FIG. 8A. Then, the processing unit 310 executes a determination process according to the location of the person being assisted based on the flag information and the sensor information. Note that the determination process may be a fall determination process. Details of the fall determination process will be described later with reference to FIGS. 16 and 17. The output of the fall determination process is, for example, information indicating the likelihood of the presence or absence of a fall risk, as will be described later.

[0118] If it is determined that there is a risk of falling, in step S107, the processing unit 310 identifies the caregiver terminal 400 to be notified of the risk of falling. Specifically, the processing unit 310 identifies the caregiver terminal 400 to be notified based on the identification information of the wearable module 100 acquired in step S105 and the notification management information of Fig. 8C. For example, the processing unit 310 may acquire the IP (Internet Protocol) address of the caregiver terminal 400 to be notified.

[0119] In step S108, the server system 300 notifies the identified caregiver terminal 400 of information related to the risk of falling. The information notified here may include information indicating the risk of falling, the name of the person being assisted who is at risk of falling, and the location of the person being assisted, such as "Mr. / Ms. AAA is likely to fall in the toilet." The notification may be given in a variety of ways, such as by displaying text on the display unit of the mobile terminal device 410 or by outputting audio to the headset 420. Notification may also be given by illuminating an LED or by vibration using a motor.

[0120] 9 also shows an example in which, when a fall risk is detected, a push notification is sent from the server system 300 to the caregiver terminal 400. However, the method by which the caregiver obtains the determination result based on the sensor information, etc. is not limited to this, and the caregiver may actively obtain the determination result by operating the caregiver terminal 400.

[0121] 10A and 10B are examples of screens displayed on the display unit of mobile terminal device 410. These screens may be displayed by application software that performs the first to fourth registration processes described above, or may be displayed by other application software.

[0122] The mobile terminal device 410 may display information for each communication device 200 and for each person being assisted. For example, the screens of Fig. 10A and Fig. 10B may include, at the bottom of the screen, an object OB6 that is a button for displaying information for each access point and an object OB7 that is a button for displaying information for each person being assisted. When the user selects object OB6, the screen of Fig. 10A is displayed, and when the user selects object OB7, the screen of Fig. 10B is displayed. However, the screen configuration used when viewing the results of the fall determination process is not limited to Fig. 10A or 10B, and various modifications are possible.

[0123] As shown in FIG. 10A , the screen displaying information for each communication device 200 may include areas RE5 to RE7. Area RE5 is an area for displaying a notification. For example, if there is a person being assisted who has been determined to have a high risk of falling, area RE5 displays the location where the communication device 200 is located and the name of the person being assisted. Area RE5 may display an object OB8 representing a confirmation button and an object OB9 representing a corresponding button. When an operation for selecting object OB8 is performed, the display unit of the mobile terminal device 410 transitions to a screen displaying detailed information about the fall risk. The screen displaying detailed information is, for example, a screen described later with reference to FIG. 11 . On the other hand, when an operation for selecting object OB9 is performed, the mobile terminal device 410 transmits information to the processing unit 310 indicating that the caregiver associated with the mobile terminal device 410 will be responsible for dealing with the fall risk. When the corresponding caregiver has been determined, the processing unit 310 may exclude information about the fall risk from the display targets. In other words, the information displayed in area RE5 may be information about the fall risk for which a corresponding caregiver has not yet been determined. However, information regarding the risk of falling for which a corresponding caregiver has already been determined may also be displayed together with that fact and the name of the caregiver in charge, and the specific display format can be modified in various ways.

[0124] Area RE6 also contains buttons for selecting the location of communication device 200. In the example of Fig. 7C, four buttons are displayed: toilet 600, wheelchair 520, bed 510, and others.

[0125] When a location is selected using area RE6, area RE7 displays a list of information about communication devices 200 located at the selected location. In the example of Fig. 10A, toilets 600 are located in four locations in the nursing facility: the first floor of building A, the second floor of building A, the first floor of building B, and the second floor of building B, and a different communication device 200 is located in each location. Note that it is not necessary to distinguish between these toilets 600 in the fall detection process, but from the perspective of facilitating confirmation and intervention by a caregiver, information including the location of each toilet 600 may be stored in storage unit 320 or the like.

[0126] In this case, area RE7 displays information about the four communication devices 200. For example, area RE7 displays the name of the person being assisted who is located near the communication device 200, along with information specifying the building and floor on which the communication device 200 is located. The processing unit 310 controls the display of the screen shown in FIG. 10A by checking whether or not there is a wearable module 100 transmitting sensor information to the communication device 200, and, if there is a wearable module 100, by referring to the information about the person being assisted (module information in FIG. 8B) associated with that module. In the example of FIG. 10A, the caregiver is informed that BBB is in the toilet 600 on the second floor of building A, CCC is in the toilet 600 on the second floor of building B, and that the remaining two toilets 600 are empty.

[0127] If there is a person being assisted who is determined to be at risk of falling, this may be displayed in area RE7. For example, in the example of Fig. 10A, as shown in area RE5, a fall risk has been detected for CCC in the restroom on the second floor of building B. Therefore, an object OB10 indicating a warning may be displayed in area RE7 in association with the data corresponding to CCC.

[0128] As shown in Fig. 10B, the screen displaying information for each person being assisted may include areas RE5 and RE8. Area RE5 is the same as in Fig. 10A, and is an area for displaying notifications.

[0129] In the example of FIG. 10B, area RE8 displays information on at least three or more care recipients, including AAA, BBB, and CCC. The care recipients to be displayed here may be all care recipients using the nursing care facility, or may be those assisted by the caregiver using the mobile terminal device 410. In the example of FIG. 10B, area RE8 includes an ID uniquely identifying each care recipient, the name of each care recipient, and information on the location of each care recipient. For example, the processing unit 310 can identify the communication device 200 to which each wearable module 100 is connected based on information from the communication device 200. Therefore, the processing unit 310 can identify the location of each care recipient based on information identifying the communication device 200 to which the wearable module 100 is connected, the access point information in FIG. 8A, and the module information in FIG. 8B. In the example of FIG. 10B, area RE8 displays that AAA is in a wheelchair 520, BBB is in a toilet on the second floor of building A, and CCC is in a toilet on the second floor of building B. In addition, in the area RE8, an object OB11 representing a warning may be displayed in association with the data corresponding to CCC, similar to the area RE7 in FIG. 10A.

[0130] 11 is an example of a presentation screen that is displayed when object OB8 is selected in FIGS. 10A and 10B. For example, the presentation screen may include information that identifies the wearable module 100, the name of the person being assisted, information that identifies the location, and information about the risk of falling. In the example of FIG. 11, sensorXXX is the identification information for the wearable module 100 and is the name of the person being assisted, Mr. AAA, who is wearing the module. In other words, FIG. 11 is an example of a presentation screen that presents the situation in the toilet of the person being assisted who is wearing sensorXXX to the caregiver.

[0131] For example, if the fall detection process determines the deviation from the reference posture in terms of angle, the display screen may display the angle using a pictogram, as shown in FIG. 11. Using the display screen shown in FIG. 11 makes it possible to clearly display to the caregiver, for example, that the forward tilt relative to the upright posture is approximately 20°. The display screen may also include text indicating the presence or absence of a fall risk, such as "You are likely to fall" or "You are not likely to fall." The displayed text is not limited to this. For example, if the characteristics of movements in the toilet are changing, text may be displayed that predicts a future fall, specifically indicating the location and circumstances of the fall, such as "Your risk of falling is increasing. There is a high possibility of falling in the toilet." Similarly, if the user is walking, a display such as "Your risk of falling is increasing. There is a high possibility of falling on uneven surfaces" may be displayed based on changes in the characteristics of walking movements. Furthermore, while consideration must be given to privacy, if a camera is installed in the target location, images captured by the camera may also be displayed.

[0132] The information displayed using FIG. 11 may be real-time information or information showing the location and past history of the person being assisted. By displaying the information shown in FIGS. 10A to 11, the caregiver can easily understand the location and situation of the person being assisted. The screen of FIG. 11 may also display simulation information predicting a future fall scenario. For example, the processing unit 310 may predict a change in the posture of the person being assisted based on sensor information obtained when a fall risk is determined, and display the prediction result on the display unit of the mobile terminal device 410. For example, the processing unit 310 may estimate the posture of the person being assisted at a later time based on sensor information obtained over a predetermined period of time (e.g., three seconds). This makes it possible to display how the person being assisted will fall in the event of a fall that may occur in the future. For example, if a fall that will result in a hard hit to the head is predicted, the caregiver can be urged to take early action. Alternatively, if a fall that will result in a slow buttock fall poses a relatively low risk of injury, the caregiver can determine whether to prioritize other assistance that requires more urgent attention.

[0133] The method of this embodiment may also output information regarding whether the assisted person will suffer a serious injury due to a fall, such as "possibility of hitting the head" or "possibility of femoral fracture." For example, as described above, this embodiment can estimate the location where a fall occurred or is likely to occur, and can also estimate the assisted person's posture at the time of the fall and the direction of the fall based on the output of the acceleration sensor 120, etc. Therefore, the processing unit 310 may perform processing to estimate, as information regarding the risk of a fall, whether a serious injury will occur in a specific part of the body, or a probability value indicating the likelihood of such an injury. The obtained information may also be displayed using a screen such as FIG. 11. For example, if no staff member, such as an assistant, is present at the scene of a fall, the assistant would conventionally determine whether a detailed examination is necessary by asking the assisted person about the part of the body that was hit, confirming the presence and location of the injury, and observing the assisted person's behavior. The accuracy of this determination is also related to the assistant's level of expertise, so an assistant with low experience may make an incorrect determination and, for example, fail to perform an appropriate detailed examination. In this respect, in this embodiment, the possibility of serious injury can be presented to caregivers who are not present at the scene, making it possible to encourage appropriate action regardless of the caregiver's level of experience. Furthermore, in this embodiment, if it is determined that there is a "possibility of a head hit" or a "possibility of a femur fracture," the caregiver or the like may be prompted to undergo a detailed examination, or a process may be performed to automatically arrange for a detailed examination or the like.

[0134] At least one of the algorithms and parameters used in the process of determining the "possibility of hitting one's head" or the "possibility of a femur fracture" may be changed depending on the location. For example, different determinations may be used to determine the "possibility of hitting one's head" or the "possibility of a femur fracture" when the person being assisted is in the toilet 600 and when the person being assisted is walking. Furthermore, machine learning such as neural networks may be applied to the process of determining these. For example, training data may be generated by adding correct answer data with a probability value of 1 representing the "possibility of hitting one's head" to sensor information and positioning information when the person actually hits their head due to a fall. By performing machine learning based on such training data, a trained model that outputs a probability value representing the "possibility of hitting one's head" can be generated. Furthermore, whether to display or hide such a display regarding the possibility of serious injury may be determined for each caregiver or each care facility. For example, the display items in FIG. 11 and the like can be configured, and the person in charge at the care facility may change whether to display or hide the possibility of serious injury.

[0135] 2.2 Fall detection based on location Next, a specific example of the fall determination process will be described.

[0136] 2.2.1 Examples of specific situations in which falls occur Falls can occur in a variety of locations in nursing care facilities, and the circumstances, causes, and manner of falls vary depending on the location. Below, we will explain falls in bed 510, wheelchair 520, toilet 600, and while walking. Note that falls in the following description are not limited to falls in the narrow sense of the word, where the body falls onto the floor, etc., but may also include a state in which balance is lost compared to a normal state.

[0137] First, we will explain falls on the bed 510. Falls can occur when trying to stand up from a seated position on the edge of the bed 510. Normally, a person first lowers their head and places their hands on their knees, the bed surface, or a handrail, shifting their weight onto their feet, and then raising their head to stand up. However, if they try to stand up without putting enough force into their feet and their center of gravity not moving forward, they will lose balance backward and end up sitting back down on the bed surface. Also, if their center of gravity shifts too far forward when they put their weight on their feet, they will fall forward (forward fall). Also, if their hands slip off when they try to put their hands down, they may fall onto the floor on the side of the hand that has come off.

[0138] On the other hand, falls can also occur when sitting on the bed surface from a standing position. Normally, a person first places one hand on the bed surface while facing the bed 510, then turns their body halfway to face away from the bed 510, and then places their buttocks on the bed surface. However, if the hand that is on the bed surface comes off, the person will not be able to support their weight and will fall. Also, if they sit down without carefully checking the bed surface after turning their body halfway on their hands, their buttocks may not even come on the bed surface in the first place, or they may get on the bed but slide off if they sit too shallowly.

[0139] Falls may also occur when transferring from the bed 510 to the wheelchair 520 or the like. Here, falls during transfer are considered to be falls on the bed 510. For example, a fall may occur if the brakes on the wheelchair 520 to which the person is transferring are not applied, or if the person being assisted accidentally releases the brakes. For example, when the person being assisted places one hand on the armrest or the like of the wheelchair 520 and puts their weight on it, the wheelchair 520 may move, causing the person being assisted to fall. Or, when the person being assisted tries to sit on the seat of the wheelchair 520, the wheelchair 520 may move backward, causing the person being assisted to fall.

[0140] Next, we will explain falling in the wheelchair 520. When riding in the wheelchair 520, the person being assisted may feel pain in their buttocks due to maintaining the same posture, and may gradually shift the position of their buttocks forward on the seat. In this case, as the amount of shift increases, the person sitting shallower will eventually lose their buttocks and fall, causing them to fall. Furthermore, while traveling in the wheelchair 520, the person being assisted places their feet on the foot supports. However, if the person tries to stand without removing their feet from the foot supports, the wheelchair 520 itself may tilt forward, causing the person being assisted to fall forward. Alternatively, if the brakes on the wheelchair 520 are released and the person accidentally tries to stand up, the wheelchair 520 may move backward during the standing-up motion, causing the person being assisted to fall backward. Furthermore, although not a direct fall, the person being assisted may also be impacted by the wheelchair 520 colliding with a wall or furniture due to an operating error or the like.

[0141] Next, we will explain falls on the toilet 600. When using the toilet 600, the person being assisted first faces the toilet, lifts the lid, turns his / her body halfway, then pulls down his / her pants slightly and sits on the toilet. For example, when turning his / her body halfway, if the person's legs do not move properly, a fall may occur. Also, when sitting on the toilet, the seat is narrow, so if the person does not sit properly, a fall may occur.

[0142] After sitting on the toilet, you pull down your pants further, defecate, and then remove the toilet paper to wipe. Because you need to lean your body when pulling down your pants further and when wiping with toilet paper, you lose your balance and fall. In addition, abdominal pressure is applied when defecating, which can cause your blood pressure to rise and lead to fainting. When this happens, you may fall forward, or if there is no backrest, you may fall backward. There is also the possibility of falling sideways.

[0143] After this, the person being assisted leaves the toilet 600 by reversing the previous steps. Specifically, the person being assisted pulls up their pants a little, grabs the handrail to stand up, and pulls up their pants all the way. After that, the person being assisted turns their body halfway to face the toilet, flushes, closes the toilet lid, turns their body halfway again to turn around and exit the toilet 600. For example, there is a risk of falling if the person is unable to move their legs properly when turning their body.

[0144] Next, we will explain falls while walking. When walking, for example, if the foot does not move forward or if the person being assisted trips over something, they may fall forward. Also, if something happens and the person being assisted shifts their center of gravity to the rear, they may fall backward. Also, if the feet cannot fully support the weight, for example, one foot may bend and the person may fall on that side.

[0145] 2.2.2 Examples of location-dependent sensor information As described above, the circumstances, causes, and manner of falls differ for bed 510, wheelchair 520, toilet 600, and walking. As a result, even if the event to be detected is a common fall, the tendency of the sensor information of acceleration sensor 120 differs depending on the location of the fall.

[0146] 12 to 15 show examples of time-series sensor information including the time when a fall occurs. The sensor information here is the output of the acceleration sensor 120 attached to the chest or back of the person being assisted as shown in FIG. 2, and is acceleration on the x-axis, y-axis, and z-axis, as well as the root mean square of the acceleration on the three axes. The horizontal axis of FIGS. 12 to 15 represents time, and the vertical axis represents acceleration (G). In each graph, points marked with arrows correspond to falls.

[0147] FIG. 12 is an example of sensor information including a fall in bed 510, FIG. 13 is an example of sensor information including a fall in wheelchair 520, FIG. 14 is an example of sensor information including a fall in toilet 600, and FIG. 15 is an example of sensor information including a fall while walking.

[0148] 12 to 15, the magnitude of acceleration, which indicates the impact of a fall, varies depending on the location. For example, in the example of bed 510 shown in Fig. 12, three falls are included, but the root mean square value is sometimes as large as 2.8 G, and sometimes as relatively small as 1.5 G. Therefore, when performing processing to detect falls in bed based on whether acceleration equal to or greater than a threshold is detected, it is necessary to appropriately determine whether a fall has occurred based on values ​​with large variations.

[0149] In the example of wheelchair 520 shown in Figure 13, the root mean square value at the time of tipping is about 2.0 G. Therefore, when determining whether wheelchair 520 has tipped over based on threshold judgment, it is sufficient to set a threshold that can distinguish this value from the normal value.

[0150] In the example of toilet 600 shown in Figure 14, the root mean square value at the time of a fall is about 1.5 G or less. Therefore, when determining whether a fall has occurred in toilet 600 based on threshold determination, a large threshold cannot be set as in other locations, and it is necessary to set a threshold that can detect such a relatively small impact as a fall.

[0151] In the example of walking shown in Fig. 15, the root mean square value at the time of a fall is approximately 2.0 to 2.4 G. Therefore, when determining whether a fall has occurred while walking based on threshold determination, a threshold value that can distinguish between this value and normal values ​​should be set. Furthermore, because normal movement while walking is greater than that of the bed 510, wheelchair 520, or toilet 600, it is not desirable to set an excessively small threshold value (for example, a threshold value smaller than 1.5 G) in order to prevent impacts due to normal walking from being erroneously determined to be a fall.

[0152] Furthermore, the above explanation only focuses on differences in the magnitude of the root mean square. However, as can be seen from the above explanation, the conditions of the posture (position of the center of gravity, tilt of the body, etc.) before the fall occurs differ depending on the location. Therefore, not only the root mean square value at one timing, but also the time-series change trend is thought to differ depending on the location. Furthermore, since the accelerations on the x-axis, y-axis, and z-axis correspond to the front-to-back, left-to-right, and up-to-down directions of the person being assisted, each of these represents information that represents the posture of the person being assisted. Therefore, even when focusing on one of the x-axis, y-axis, and z-axis, or a combination of two or more, the tendency of the sensor information will differ depending on the location.

[0153] As described above, since the sensor information at the time of a fall differs depending on the location, the accuracy of the process can be improved by adapting the fall determination process based on the sensor information to the location.

[0154] 2.2.3 Example of fall detection process As described above, the fall determination process according to this embodiment may be a process of comparing the acceleration value with a threshold value. In this case, the storage unit 320 stores a threshold value for each location, and in step S107 of FIG. 9, the storage unit 320 identifies the corresponding threshold value based on the location identification result and performs a fall determination process of comparing the threshold value with the sensor information. However, the fall determination process is not limited to this.

[0155] For example, the processing unit 310 may perform a fall detection process using machine learning. An example in which a neural network is used as machine learning will be described below. Hereinafter, neural network will be abbreviated as NN. However, machine learning is not limited to NN, and other methods such as SVM (support vector machine) and k-means may also be used, or methods developed from these may be used. Furthermore, although supervised learning will be exemplified below, other machine learning methods such as unsupervised learning may also be used.

[0156] FIG. 16 shows an example of the basic structure of a neural network (NN). Each circle in FIG. 16 is called a node or neuron. In the example of FIG. 16, the NN has an input layer, two or more hidden layers, and an output layer. The input layer is I, the hidden layers are H1 and Hn, and the output layer is O. In the example of FIG. 16, the number of nodes in the input layer is 2, the number of nodes in each hidden layer is 5, and the number of nodes in each layer can be modified in various ways. In addition, FIG. 16 shows an example in which each node in a given layer is connected to all nodes in the next layer, but this configuration can also be modified in various ways.

[0157] The input layer receives input values ​​and outputs them to the intermediate layer H1. In the example of Fig. 16, the input layer I receives two types of input values. Each node in the input layer may perform some processing on the input value and output the processed value.

[0158] In a NN, a weight is set between two connected nodes. W1 in Figure 16 is the weight between the input layer I and the first hidden layer H1. W1 represents the set of weights between a given node included in the input layer and a given node included in the first hidden layer. For example, W1 in Figure 16 is information containing 10 weights.

[0159] Each node in the first hidden layer H1 performs a weighted sum of the outputs of the nodes in the input layer I connected to that node using a weight W1, and then adds a bias. Each node then applies a nonlinear activation function to the sum to determine the output of that node. The activation function may be a ReLU function, a sigmoid function, or another function.

[0160] The same is true for subsequent layers. That is, in a given layer, the output of the previous layer is weighted using the weight W, and then a bias is added and an activation function is applied to determine the output for the next layer. The output of the output layer is the output of the NN.

[0161] As can be seen from the above explanation, in order to obtain desired output data from input data using a NN, it is necessary to set appropriate weights and biases. In learning, training data is prepared in which given input data is associated with ground truth data that represents correct output data for that input data. The NN learning process is a process of determining the most likely weights based on the training data. Note that various learning methods, such as backpropagation, are known for the NN learning process. In this embodiment, these learning methods can be widely applied, and therefore detailed description will be omitted.

[0162] Furthermore, the NN is not limited to the configuration shown in Fig. 16. For example, a network with another configuration, such as an RNN (Recurrent Neural Network), may be used as the NN. The RNN may be, for example, an LSTM (Long Short Term Memory). Furthermore, a convolutional neural network (CNN) may be used as the NN.

[0163] FIG. 17 is a diagram illustrating the relationship between input data and output data in this embodiment. For example, the input data includes sensor information from the acceleration sensor 120 and flag information that identifies a location. As described above, the sensor information includes acceleration values ​​for the x-axis, y-axis, and z-axis, and the root mean square of the acceleration along the three axes. However, it is not necessary to use all four values, and some may be omitted. The flag information may be, for example, 4-bit data that respectively represent the toilet 600, bed 510, wheelchair 520, and walking. For example, the 4-bit data may have a value of "1000" that represents the toilet 600, "0100" that represents the bed 510, "0010" that represents the wheelchair 520, and "0001" that represents walking.

[0164] 17, in the method of this embodiment, not only the sensor information of the acceleration sensor 120 but also information specifying the location is included in the input data. This makes it possible to perform processing that takes into account the influence of location on a fall. Specifically, since it is possible to perform a fall determination process according to the location, it is possible to improve the processing accuracy.

[0165] 17, the input data may be time-series data. For example, if acceleration sensor 120 is configured to perform one measurement at a predetermined time interval and obtain four acceleration values, x, y, z, and root mean square, as the result of one measurement, the input data is a set of N×4 acceleration values ​​obtained by N measurements, where N is an integer equal to or greater than 2. The root mean square may be calculated by acceleration sensor 120, or by processing unit 210 of communication device 200 or processing unit 310 of server system 300.

[0166] As described above, the flag information indicating the location is information identified based on the identification information of the communication device 200, and the identification information of the communication device 200 may be assigned each time the communication device 200 receives sensor information.

[0167] In this way, by treating the input data as time-series data, it becomes possible to process the input data while taking into account changes in the input data over time. For example, as mentioned above, depending on the location, the time-series behavior, such as the circumstances leading up to the fall and the manner of the fall, will differ. In this regard, by processing the time-series input data using LSTM or the like, it becomes possible to reflect time-series differences depending on the location in the fall detection process.

[0168] In addition, the output data in machine learning is information that indicates the likelihood of whether or not the person being assisted is at risk of falling. For example, the output layer of a neural network may output a probability value between 0 and 1 as output data. The larger this value, the higher the probability that there is a risk of falling, i.e., the higher the risk of falling.

[0169] For example, the processing unit 310 of the server system 300 may acquire a trained model generated by machine learning based on training data including training sensor information output by the wearable module 100 and training placement information identifying the location where the training sensor information was acquired, and the trained model outputs information indicating the likelihood of a fall risk. For example, in the learning stage, the processing unit 310 acquires training data in which the input data shown in FIG. 17 is associated with correct answer data indicating the presence or absence of a fall risk. The input data is data in which flag information indicating a location is assigned to the sensor information of the acceleration sensor 120 acquired via the communication device 200 as described above. Note that the communication device 200 may be omitted in the learning stage, and the user may associate the sensor information with flag information identifying a location. The correct answer data may be information assigned by, for example, an expert. For example, the expert may input information identifying the timing when the risk of a fall is high and a caregiver should intervene. In this case, correct answer data indicating a high risk of a fall is associated with input data for a period corresponding to the timing (e.g., a predetermined period before the timing). Alternatively, correct answer data may be assigned based on a history of whether or not the person being assisted has actually fallen. For example, if the person being assisted has actually fallen, correct answer data indicating a high risk of falling is associated with the input data for the corresponding period.

[0170] Next, processing unit 310 performs processing to update the weights of the NN. Specifically, processing unit 310 inputs input data to the NN and obtains output data by performing forward calculations using the weights at that stage. Processing unit 310 calculates an objective function based on the output data and correct answer data. The objective function here is, for example, an error function based on the difference between the output data and correct answer data, or a cross-entropy function based on the distribution of the output data and the distribution of the correct answer data. Processing unit 310 updates the weights, for example, so that the error function decreases. The above-mentioned backpropagation method and the like are known weight update methods, and these methods can be widely applied to this embodiment as well.

[0171] The processing unit 310 ends the learning process when a given condition is satisfied. For example, the training data may be divided into learning data and validation data. The processing unit 310 may end the learning process when the process of updating the weights using all the learning data has been performed, or may end the learning process when the correct answer rate based on the validation data exceeds a given threshold. After the learning process ends, the NN including the weights at that stage is stored in the storage unit 320 as a learned model. Note that the learning process is not limited to being executed in the server system 300 and may be executed in an external device. The server system 300 may acquire the learned model from the external device.

[0172] Also, in the inference stage, the processing unit 310 reads out the learned model from the storage unit 320. Then, the processing unit 310 acquires input data with flag information for the sensor information of the acceleration sensor 120 acquired via the communication device 200, and inputs the input data into the learned model. The processing unit 310 obtains output data by performing a forward calculation based on the weights acquired by the learning process. The output data is numerical data representing the level of the fall risk, as described above.

[0173] For example, when a threshold Th where 0 < Th < 1 is set in advance, the processing unit 310 may determine that there is a fall risk when the value of the output data is equal to or greater than Th. When it is determined that there is a fall risk, the processes of steps S107 and S108 in FIG. 9 are executed.

[0174] 2.2.4 Variations in the fall determination process <Other examples of information used in the fall determination process (assessment, fall history)> In FIG. 17, an example where the input data is sensor information and flag information for specifying a location was described. However, the method of the present embodiment is not limited to this, and the input data may include other information.

[0175] For example, the input data may include information representing a classification result in which users are classified into several classes. The classification may be performed using a device other than the wearable module 100.

[0176] For example, the following URL (Uniform Resource Locator) discloses Waltzin, a device that includes an insole-type pressure sensor. By using such a device, it is possible to divide the sole of the foot into multiple parts and measure the plantar pressure at each part in real time. https: / / media.paramount.co.jp / service / rehabilitation / waltwin /

[0177] The following URL also discloses SR AIR, a device that measures pressure in each area subdivided into 15 x 15, as well as the subject's center of gravity in real time. By using such a device, it becomes possible to perform a detailed analysis of pressure changes in each situation of the person receiving care, such as standing, sitting, and walking. http: / / www.fukoku-jp.net / srsoftvision / common / img / download / download_pdf_007.pdf

[0178] For example, the processing unit 310 classifies the person receiving care into one of several classes based on the length of time they can maintain a standing or sitting position, the direction they tend to lean when their posture is disrupted, and the areas of the body that are prone to pressure. By using a device that outputs such highly accurate and detailed information to assess the person receiving care, it is possible to classify the person receiving care according to the circumstances in which they are likely to fall, how they lose balance, and the direction in which they fall. By including the classification results in the input data for the fall detection process, it becomes possible to process the person receiving care in a way that takes into account their tendency to fall, thereby further improving the accuracy of the processing. Furthermore, because these devices are used to classify the person receiving care and do not need to be used continuously, system construction is easy.

[0179] The method of this embodiment is not limited to machine learning. For example, the processing unit 310 may perform the fall determination process using different algorithms depending on the classification results. Alternatively, the processing unit 310 may perform the fall determination process using different parameters (e.g., threshold values) depending on the classification results. In addition, various modifications of the method of using the classification results in the fall determination process are possible.

[0180] Furthermore, the information used in the fall detection process is not limited to the output of the assessment device. For example, a user such as a caregiver may be able to input fall history information that indicates the fall history of the person being assisted. For example, the user may use the caregiver terminal 400 or the like to input information identifying the person being assisted, the timing of the fall, the location of the fall, the circumstances of the fall, etc. The caregiver terminal 400 or the like transmits the input fall history information to the server system 300. The processing unit 310 determines the fall risk by performing a fall detection process based on the sensor information, flag information, and fall history information. The fall history information, like the classification results described above, may be used as input data for machine learning, or may be used for processes other than machine learning. Furthermore, both the fall history information and the classification results may be used in the fall detection process.

[0181] <Forward slippage of wheelchairs, etc.> The above describes the fall determination process based on sensor information from the wearable module 100 worn on the chest, etc. However, other sensors may also be used in the fall determination process.

[0182] Fig. 18A shows an example of pressure sensors arranged on a wheelchair 520. In the example of Fig. 18A, four pressure sensors Se1 to Se4 are arranged on the back side of a cushion 521 that is placed on the seat of the wheelchair 520. Pressure sensor Se1 is a sensor arranged in the front, pressure sensor Se2 is a sensor arranged in the rear, pressure sensor Se3 is a sensor arranged on the right, and pressure sensor Se4 is a sensor arranged on the left. Note that front, back, left, and right here refer to directions when a person being assisted is sitting in the wheelchair 520.

[0183] As shown in FIG. 18A, the pressure sensors Se1 to Se4 are connected to a control box 523. The control box 523 includes a processor that controls the pressure sensors Se1 to Se4 and a memory that serves as a work area for the processor. The processor here is, for example, an MCU (Micro Controller Unit), but other processors may also be used. The memory is, for example, an SRAM, a DRAM, a ROM, etc. An external memory such as a USB memory may also be connected to the control box 523. The control box 523 is stored in a pocket provided on the back of the wheelchair 520, for example, similar to the communication device 200-2.

[0184] The processor detects pressure values ​​by operating the pressure sensors Se1 to Se4, and performs a storage process to accumulate the detected pressure values ​​in memory (ROM). The storage process may be performed periodically, for example, or the start / end of the process may be controlled based on an operation by a caregiver. The control box 523 may also include a communication module (not shown), and the processor may transmit the accumulated pressure values ​​to a device such as the communication device 200-2 via the communication module. For example, the pressure values ​​may be transmitted to the server system 300 via the communication device 200-2, and the processing unit 310 may perform a fall determination process (a process for detecting forward or sideways slippage, which will be described later) based on the pressure values.

[0185] The processor may also perform a fall determination process based on pressure values. For example, the processor may start up the pressure sensors Se1 to Se4, initialize various parameters, detect pressure values, perform a fall determination process, and perform a recording process. After performing initialization, the processor may repeatedly perform the processes of detecting pressure values, performing a fall determination process, and performing a recording process at predetermined intervals. In this way, it becomes possible to perform a fall determination process using the pressure sensors Se1 to Se4 without going through the server system 300.

[0186] A person being assisted sitting in wheelchair 520 may feel pain in the buttocks and may shift the position of the buttocks. For example, forward slippage occurs when the buttocks are shifted further forward than normal, and lateral slippage occurs when the buttocks are shifted to the left or right. In addition, forward slippage and lateral slippage may occur simultaneously, causing the center of gravity to shift diagonally.

[0187] Although forward slippage and side slippage do not constitute a fall, they are situations in which a fall is likely to occur, and therefore can be a risk of falling. In this regard, by using a pressure sensor arranged on cushion 521 as shown in Fig. 18A, it is possible to appropriately detect changes in the position of the buttocks, and therefore it is possible to accurately detect forward slippage and side slippage.

[0188] For example, the initial state is the time when the person being assisted first transfers to the wheelchair 520 and assumes a normal posture. In the initial state, the person being assisted sits deep in the seat of the wheelchair 520, so it is expected that the value of the rear pressure sensor Se2 will be relatively large. On the other hand, if forward slippage occurs, the position of the buttocks will move forward, and the value of the front pressure sensor Se1 will increase. For example, the processing unit 310 may determine that forward slippage has occurred when the value of the pressure sensor Se1 increases by a predetermined amount compared to the initial state. Furthermore, instead of using the value of the pressure sensor Se1 alone, processing may be performed using the relationship between the values ​​of the pressure sensors Se2 and Se1. For example, the difference in voltage values ​​output by the pressure sensors Se2 and Se1 may be used, or the ratio of the voltage values ​​may be used, or the rate of change of the difference or ratio from the initial state may be used.

[0189] Similarly, when a lateral shift occurs, the position of the buttocks moves to either the left or the right, and if the shift is to the left, the value of pressure sensor Se4 increases, and if the shift is to the right, the value of pressure sensor Se3 increases. Therefore, processing unit 310 may determine that a left shift has occurred when the value of pressure sensor Se4 increases by a predetermined amount compared to the initial state, and may determine that a right shift has occurred when the value of pressure sensor Se3 increases by a predetermined amount compared to the initial state. Alternatively, processing unit 310 may determine a right shift or a left shift using the relationship between the values ​​of pressure sensors Se4 and Se3. As with the example of a forward shift, the difference in voltage values ​​output by pressure sensors Se4 and Se3 may be used, or the ratio of the voltage values ​​may be used, or the rate of change of the difference or ratio from the initial state may be used.

[0190] As shown in FIG. 18A , pressure sensor Se1 may be positioned offset to either the left or right of the center of the seat, and pressure sensor Se2 may be positioned offset to the other side of the center of the seat. Many wheelchairs 520 are foldable, and the seat may be made of a soft material that can be folded left and right. For example, as shown in FIG. 18A , cushion 521 placed on the seat has a notch N on the backside, allowing it to be folded left and right at the notch N. In this case, pressure sensor Se1 is positioned, for example, to the right of notch N, and pressure sensor Se2 is positioned to the left of notch N. By shifting the positions of pressure sensors Se1 and Se2 from the center in this way, overlap between the positions of pressure sensors Se1 and Se2 and the position of notch N can be avoided. As a result, the weight of the person being assisted sitting on the cushion is accurately transmitted to pressure sensors Se1 and Se2, allowing for accurate detection of forward / backward displacement.

[0191] 18A, the pressure sensors Se3 and Se4 may be positioned further back than the center of the cushion 521 in the front-to-back direction. From the perspective of preventing the wheelchair 520 from tipping over, it is desirable for the person being assisted to sit deep in the wheelchair 520. In other words, in a standard posture, the buttocks of the person being assisted are positioned slightly back from the center of the seat. Furthermore, if a positional shift occurs, the buttocks are expected to move forward of the seat. Therefore, by positioning the pressure sensors Se3 and Se4 further back than the standard buttocks position (or, more specifically, the initial position of the buttocks), it is possible to prevent the buttocks and the pressure sensors Se3 and Se4 from being too close to each other. As a result, it is possible to prevent the detection value from saturating even if the upper limit of the detectable pressure value is low. For example, small, thin sensors can be used as the pressure sensors Se3 and Se4, which facilitates system construction.

[0192] FIG. 18B is a diagram illustrating the cross-sectional structure of cushion 521. As shown in FIG. 18B, cushion 521 may have a structure in which a first layer 522a, a second layer 522b, and a third layer 522c are stacked in this order in a vertically downward direction when in use. For example, first layer 522a is a cushion provided independently of the other layers, while second layer 522b and third layer 522c are a cushion provided integrally with a cut N formed on the underside. Here, first layer 522a and third layer 522c may be softer than second layer 522b. Softness may be expressed by, for example, Young's modulus or the magnitude of the load when an object is pressed down to a predetermined strain. By providing relatively stiff second layer 522b, the weight of the person being assisted can be distributed, improving comfort and preventing bedsores. On the other hand, since the first layer 522a and the third layer 522c, which are relatively soft, are provided as layers that come into direct contact with the buttocks and the pressure sensors Se1 to Se4, it becomes possible to accurately detect pressure fluctuations according to position. Specifically, when the position of the buttocks of the person being assisted changes, the pressure value is likely to change significantly, which makes it possible to improve processing accuracy.

[0193] 18C shows an example of operation unit 524 and notification unit 525 provided in control box 523. As shown in Fig. 18C, operation unit 524 includes a power switch 524a, a recording start / stop button 524b, and a judgment button 524c. Notification unit 525 includes a measurement in progress lamp 525a, a recording in progress lamp 525b, a forward deviation lamp 525c, and a lateral deviation lamp 525d. In this embodiment, forward deviation and lateral deviation are described separately, but it is also possible to recognize them together as "deviation" and turn on lamp 525c when the processor does not detect any deviation (normal), and turn on lamp 525d when it detects a deviation.

[0194] The power switch 524a is a switch that starts the supply of power to, for example, the processor, memory, pressure sensors Se1 to Se4, etc. When the power switch 524a is turned on, the above-mentioned components transition to an operable state, and the pressure sensors Se1 to Se4 start measuring pressure values. While the pressure values ​​are being measured, the measurement in progress lamp 525a lights up. When the pressure value measurement starts, the processor performs a fall determination process based on the pressure values, and if it is determined that a forward deviation has occurred, it turns on the forward deviation lamp 525c, and if it is determined that a lateral deviation has occurred, it turns on the lateral deviation lamp 525d.

[0195] The recording start / stop button 524b is a button that controls the start / stop of a recording process that stores pressure values ​​detected by the pressure sensors Se1 to Se4 in memory. For example, if the recording start / stop button 524b is pressed during measurement and the recording process has not yet started, the processor starts the process of storing the pressure values ​​in memory. While the recording process is being executed, the recording lamp 525b is lit. On the other hand, if the recording start / stop button 524b is pressed while the recording process is being executed, the processor stops recording the pressure values. When the recording process ends, the recording lamp 525b is turned off.

[0196] The judgment button 524c is a button that adds a flag to the pressure value being measured. For example, when the result of the fall judgment process does not match the caregiver's tacit knowledge (when the result of the fall judgment process indicates a discrepancy but the caregiver determines that the discrepancy is not present, or vice versa), the caregiver presses this judgment button 524c. The processor flags the pressure values ​​of the pressure sensors Se1 to Se4 when the judgment button 524c is pressed and stores the flagged values ​​in memory. When the processor finishes recording, it transmits the data to the server system 300 via the communication device 200 or the like. The server system 300 updates the trained model based on the original training data and the flagged pressure value data and sends the updated trained model to the control box 523. When the control box 523 is turned on again, it communicates with the server system 300 and downloads the updated trained model. The control box 523 uses the updated trained model to determine whether the person being assisted has slipped forward or to the side, and then performs the process of detecting a fall. Therefore, the caregiver can perform the process of detecting a fall that is optimal for each person being assisted.

[0197] For example, in the method of this embodiment, the server system 300 may perform processing using a trained model that acquires input data including pressure values ​​corresponding to the pressure sensors Se1 to Se4 and outputs output data including the likelihood of the presence or absence of a deviation. The input data may be time-series data, which is a collection of four pressure values ​​at multiple timings. The output data may be numerical data representing the probability of deviation. The output data may also include both a probability value representing the possibility of a forward deviation and a probability value representing the possibility of a lateral deviation. As described above, by using the judgment button 524c, the caregiver can point out errors in the estimation results using the trained model. For example, if the trained model determines that a deviation exists and a flag is assigned while the forward deviation lamp 535c or the lateral deviation lamp 535d is illuminated, the flag indicates that "no deviation" is the correct answer. In this case, data to which correct answer data indicating that the probability of deviation is 0 (or sufficiently small) for the corresponding pressure value is added is transmitted to the server system 300 as training data for update. The same applies in the reverse case. For example, if a flag is assigned when the trained model has determined that there is no deviation, data to which correct answer data indicating that the probability of deviation is 1 (or sufficiently large) for the corresponding pressure value is added is transmitted to the server system 300 as training data for updating. In this way, it becomes possible to appropriately update the trained model. At this time, as described above, the trained model may be updated for each person being assisted. For example, a different trained model may be used for each person being assisted.

[0198] As described above, the fall detection process may be executed by a processor. Alternatively, a mobile information terminal capable of communicating with the control box 523 may be used to input responses to a questionnaire regarding the attributes of the person being assisted. The responses to the questionnaire are transmitted to the server system 300 and used to update the trained model. The server system 300 classifies the people being assisted based on the results of the questionnaire and updates the trained model for each class. As a result, the control box can perform optimal fall detection processing for people being assisted with similar attributes. For example, a different trained model may be used for each of the above-mentioned attributes, or data representing the attributes of the person being assisted may be added to the input data. Here, the attributes of the person being assisted may include information such as gender, age, physique, physical condition, movement, and communication ability. For example, the physique may be information indicating whether the person is thin, normal, or obese, or may be a numerical value such as BMI. The physical condition is information identifying the presence and location of hemiplegia, the presence and location of pain, and spinal deformity (such as the presence and severity of kyphosis or scoliosis). The movement information indicates whether the person being assisted can adjust their sitting position by themselves. The communication information indicates whether the person being assisted can communicate with others.

[0199] Although the present embodiment uses one determination button 524c, this is not limiting. For example, multiple buttons may be set to distinguish between forward and lateral deviations and record them separately. For example, if the trained model is configured to separately output the possibility of forward deviation and the possibility of lateral deviation, it is desirable to be able to input whether the currently occurring deviation is a forward deviation, a lateral deviation, or both, in order to assign correct answer data. In this case, providing a button for assigning a forward deviation flag and a button for assigning a lateral deviation flag enables appropriate flagging. Furthermore, because the flag assigned by a user such as a caregiver is not limited to information identifying the type of deviation (no deviation, forward deviation, lateral deviation, etc.), the number of determination buttons 524c may be expanded to three or more. For example, the caregiver may use the determination button 524c to input a flag for identifying data for a partial period of a series of pressure values ​​recorded by the recording process. For example, the caregiver may determine that data for a partial period of the period from the start to the end of the recording process using the recording start / end button 524b is characteristic data (e.g., corresponding to a period in which a deviation occurred). In this case, the caregiver can use the determination button 524c to input the start / end of the corresponding period, thereby assigning a period flag to some of the data. For example, from among the pressure values ​​included in one file recorded by the recording process, only the pressure values ​​to which a period flag has been assigned can be extracted and used in the update process of the trained model. Furthermore, the flags that the user can assign are not limited to the type of deviation or period, and various modifications are possible. Note that the expansion of the determination button 524c is not limited to increasing the number of buttons. For example, an interface other than a button may be used, or different inputs may be possible depending on the number of times a button is pressed or the duration of the button press. In other words, the determination button 524c may be an interface that allows inputs according to the type of flag to be used, and various modifications are possible.

[0200] Furthermore, a mobile terminal device such as a smartphone may be used as a sensor to detect forward or lateral slippage. For example, smartphones that include an acceleration sensor are widely used. For example, the fall detection process may be performed by fastening the smartphone to the underside of the seat using a fastener such as a band.

[0201] As described above, the seat of the wheelchair 520 may be made of a foldable, soft material. If the seat is soft, the area where the person being assisted places their buttocks sinks significantly, while the other areas of the person being assisted float relatively. This means that the angle of the seat is likely to change significantly depending on the posture of the person being assisted. For example, if forward slippage occurs, the front of the seat will sink more than in the normal state. In this case, the posture of the smartphone attached to the back of the wheelchair will also change in accordance with the change in the seat, so forward slippage can be properly detected by using an acceleration sensor.

[0202] Similarly, if a lateral shift occurs, one side of the seat sinks significantly, while the other side rises. In this case, the smartphone's posture also changes depending on the direction of the shift, so the lateral shift can be detected appropriately by using the smartphone's acceleration sensor.

[0203] <Variations regarding walking> The above describes an example in which the target sensor information is determined to be data during walking when, for example, a communication device 200 corresponding to walking is provided in addition to communication devices 200-1 to 200-6 in Fig. 2 and is associated with identification information of that communication device 200. However, because the sensor information during walking has different characteristics from those in bed 510, wheelchair 520, and toilet 600, processing unit 310 may determine whether the sensor information corresponds to walking based on those characteristics.

[0204] Specifically, unlike other locations, when walking, the person being assisted must repeatedly alternate between putting out their right and left feet. As a result, the upper body of the person being assisted sways from side to side, with two steps forming one cycle. Therefore, the acceleration value of the axis corresponding to the left and right direction of the person being assisted becomes periodic data. In the example above, the axis corresponding to the left and right direction is the y-axis.

[0205] For example, the processing unit 310 determines whether the person is walking by determining the periodicity of the acceleration values ​​on the y-axis. As an example, the processing unit 310 detects upper or lower peaks in the acceleration values ​​on the y-axis and calculates the peak interval. An upper peak is a point where the value changes from increasing to decreasing, and a lower peak is a point where the value changes from decreasing to increasing. The peak interval is the time difference between a given peak and the next peak. For example, the processing unit 310 calculates the variation in the peak interval over a predetermined period, and if the variation is equal to or less than a predetermined value, it determines that the periodicity is high and that the person being assisted is walking.

[0206] The process for determining periodicity is not limited to this. For example, the interval between zero-crossing points may be used instead of peaks. A zero-crossing point is a point where a value changes from positive to negative, or from negative to positive. Alternatively, the processing unit 310 may perform a frequency transformation such as FFT (fast Fourier transform) and determine periodicity based on the distribution after the transformation. For example, the processing unit 310 determines that the person being assisted is walking when it determines that the frequency variation is equal to or less than a predetermined value based on the half-width of the peaks in the waveform on the frequency axis.

[0207] Although the fall detection process using machine learning such as NN has been described above, the fall detection process for walking may also be performed based on the periodic signals described above. For example, the processing unit 310 may determine that there is a high risk of falling when the periodicity becomes lower than in a normal state. This is because when the periodicity becomes lower, the rhythm of putting out the right and left feet is disrupted, which may indicate, for example, that the feet are not putting out properly or that the person has tripped over something.

[0208] Furthermore, the applicant's experiments have revealed that even if there is periodicity in the y-axis acceleration values ​​before and after a fall, there are differences from the normal state, such as fluctuations in amplitude and period, or fluctuations in the lower and upper peak values ​​of the acceleration values. Therefore, when walking, the processing unit 310 may calculate the parameters such as the amplitude and period described above and determine the risk of falling based on changes in these parameters. For example, the processing unit 310 may classify patterns in which the periodicity is disrupted into several categories and determine whether a fall has occurred based on which of these categories the pattern falls into. Note that pattern classification during walking will be explained in the walking ability estimation process described below.

[0209] 2.3 Interaction with peripheral devices The above has described an example in which a notification is given to a caregiver when a fall risk is detected, as described above with reference to Figure 5 etc. However, the processing unit 23 of the information processing device 20 may execute other processes based on the fall determination process. Note that, similar to the above example, an example in which the information processing device 20 is the server system 300 will be described below.

[0210] For example, the processing unit 310 may control the peripheral device 700 based on the fall determination process. The trigger for this control may be, for example, detection of a risk of the assisted person falling based on the fall determination process. Here, the peripheral device 700 refers to a device used by the assisted person and placed near the assisted person in the assisted person's daily life. In this way, cooperation with the peripheral device 700 makes it possible to prevent the assisted person from falling, or, even if the fall itself cannot be prevented, to mitigate the impact of the fall.

[0211] 19A and 19B are diagrams illustrating a table 530, which is an example of a peripheral device 700. For example, a table having a compact operating mechanism is described in Japanese Patent Application No. 2015 / 229220, entitled "Operating Mechanism and Mobile Table Having the Same," filed on November 24, 2015. This patent application is incorporated herein by reference in its entirety.

[0212] 19C and 19D are diagrams illustrating a walker 540, which is an example of peripheral device 700. For example, a walker that is lightweight, stable, and easy to maintain is described in Japanese Patent Application No. 2005 / 192860, entitled "Walking Aid," filed on June 30, 2005. This patent application is incorporated herein by reference in its entirety.

[0213] The table 530 is, for example, a movable table including casters Ca11 to Ca14. The walker 540 is, for example, a device including casters Ca21 to Ca24 and assisting the walking of a person receiving care. The table 530 has a function of restricting movement by locking at least a portion of the casters Ca11 to Ca14. For example, Japanese Patent Application No. 2015 / 229220 discloses a brake mechanism and an operating wire that transmits operation to the brake mechanism. Similarly, the walker 540 has a function of restricting movement by locking at least a portion of the casters Ca21 to Ca24. For example, a known walker has a brake lever near a grip portion that is held by a person receiving care, and the brake lever is activated using a wire when the brake lever is gripped. The walker 540 may have a function of fixing the brake mechanism in the brake state, or may have a locking mechanism that uses a wire different from the wire that is linked to the brake lever.

[0214] However, this does not necessarily mean that the peripheral device 700 is locked. For example, the movable table disclosed in Japanese Patent Application No. 2015 / 229220 has an off-lock function in which the brake is normally applied and the brake is released when the operating lever 531 is operated. For example, in FIG. 19A , the off-lock is released by moving two operating levers 531 upward. However, even in a table with an off-lock, there are cases in which the off-lock is released. For example, when a caregiver moves the table 530, they may fix the operating lever 531 in an operating state using a lock lever rather than continuing to manually operate the operating lever 531. In this case, the off-lock is released. There is no problem because the off-lock functions when the caregiver returns the lock lever to its original position after moving the table 530. For example, the table 530 may be configured so that the lock lever returns to its original position by further operating (e.g., moving further upward) the operating lever 531 that is fixed in the operating state. However, human error can cause the lock lever to remain in the operating position, thereby maintaining the off-lock release.

[0215] Furthermore, in the case of the walker 540, for example, when the person being assisted is walking using the walker 540, there is a risk of the person falling if they lose their balance. In this case, as described above, if the person being assisted can operate the brake lever, the casters Ca21 to Ca24 can be locked. However, it is not easy for a person being assisted who is about to fall to pull the brake lever appropriately, and the locking mechanism may not function. Furthermore, depending on the configuration of the walker 540, there may be cases where the locking mechanism is provided only near the caster, such as brake 547 described below with reference to FIG. 19D, and the walker 540 does not have a brake lever itself. In this case, it is difficult for a person being assisted who is about to fall to quickly lock the walker using their foot or other means.

[0216] Therefore, when a risk of falling is detected, the processing unit 310 may perform control to lock the casters of the peripheral device 700 that can move on casters. The peripheral device 700 that can move on casters is, for example, the table 530 or the walker 540, but other peripheral devices 700 may also be used. For example, a bed 510 (including a child's bed) with casters that has a function to electrically lock the casters is known. The peripheral device 700 of this embodiment may include such a bed 510. When a person being assisted is about to fall, the person usually grabs onto the peripheral device 700. According to the method of this embodiment, the peripheral device 700 that the person being assisted tries to grab can be reliably locked. In the locked state, the peripheral device 700 is restricted from moving relative to the floor, so that the peripheral device 700 can appropriately support the body of the person being assisted and prevent the person being assisted from falling.

[0217] Alternatively, the peripheral device 700 may be a device having a height adjustment function. The peripheral device having a height adjustment function may be, for example, a bed 510 shown in FIG. 19E. The bed 510 here is an adjustable bed whose bottom height can be changed. However, other devices may also be used as the peripheral device 700 having a height adjustment function.

[0218] When a risk of falling is detected, the processing unit 310 may perform control to lower the height of the peripheral device 700. The bed 510 adjusts the angle and height of its bottom depending on the situation, such as when sitting down to stand up or transfer to the wheelchair 520, when eating in the bed 510, or when changing a diaper. However, when the bottom is positioned high, the height of the mattress on the bed and the side rails attached to the sides are also high. Therefore, it may be difficult for a care recipient who is about to fall to grab onto the mattress or handrail or to fall safely onto the mattress. In this regard, according to the method of the present embodiment, the height of the bed 510 is lowered when there is a risk of falling, thereby appropriately preventing the care recipient from falling and being injured.

[0219] 20 is a diagram showing the configuration of the peripheral device 700. The peripheral device 700 includes a control unit 710, a storage unit 720, a communication unit 730, and a drive mechanism 740.

[0220] The control unit 710 controls each unit of the peripheral device 700. The control unit 710 may be a processor. Various types of processors such as a CPU, a GPU, or a DSP can be used as the processor here. The control unit 710 in this embodiment corresponds to, for example, a processor included in a board box 533 (described later) or a processor included in the housing 542 or the second housing.

[0221] The storage unit 720 is a work area of ​​the control unit 710, and is realized by various types of memory such as SRAM, DRAM, ROM, etc. The storage unit 720 in this embodiment corresponds to, for example, a memory included in the board box 533 described later, or a memory included in the housing 542 or the second housing.

[0222] The communication unit 730 is an interface for communicating via a network and includes, for example, an antenna, an RF circuit, and a baseband circuit. The communication unit 730 may operate under the control of the control unit 710, or may include a processor for communication control that is different from the control unit 710. The communication unit 730 may communicate with the server system 300, for example, by communication using a wireless LAN. Alternatively, similar to the example of the bed 510 in FIG. 2, the communication device 200 may be fixed to the peripheral device 700 using a holder or the like. In this case, the communication device 200 may communicate with the server system 300. The communication unit 730 acquires information from the processing unit 310 by communicating with the communication device 200 using any method, such as Bluetooth.

[0223] The driving mechanism 740 includes a mechanical component for operating the peripheral device 700. For example, the driving mechanism 740 may be a solenoid 534. As shown in FIGS. 19A and 19B, the table 530 includes a pair of operating levers 531 and a fixing member 532 for fixing the driving mechanism 740 to the table 530. The fixing member 532 has a relatively large main surface 532a, a surface 532b that intersects with the main surface 532a and is parallel to the table surface, and a surface 532c that intersects with the main surface 532a and is parallel to one surface of the support portion. The fixing member 532 is fixed to the table 530 using these surfaces. Note that "parallel" here includes "approximately parallel," and includes, for example, a surface that forms an angle of a predetermined value or less with a symmetry plane (such as the table surface in the above example). Various fixing methods, such as screwing or adhesive bonding, can be used. As shown in FIG. 19B, the fixing member is provided with a solenoid 534 and a board box 533 that houses a board that drives the solenoid 534. The substrate here is, for example, a substrate on which a processor and memory that control the solenoid 534 are mounted.

[0224] 19A, in a state in which the fixing member 532 is fixed to the table 530, the solenoid 534 is disposed below one of the pair of operating levers 531. More specifically, the solenoid 534 is disposed at a position where the movable iron core will collide with the operating lever 531 when the movable iron core is moved by the drive of the processor included in the circuit board box 533. For example, when the processing unit 310 outputs a control signal instructing the table 530 to be locked, the control signal is transmitted to the circuit board via the communication device 200 provided in the table 530, and the circuit board drives the solenoid 534 based on the control signal. In this way, an operation to move the operating lever 531 upward is performed based on the control signal from the processing unit 310, and the fixing of the operating lever 531 is released, and as a result, the table 530 transitions to a state in which the off-lock function is activated.

[0225] The drive mechanism 740 may also include a wire 546 for operating the brake mechanism and a motor 545 for winding up the wire. As shown in FIGS. 19C and 19D , the walker 540 includes a base frame, a support column attached to the base frame, an adjustable support column that can be extended or retracted, and a reclining section attached to the top of the adjustable support column for supporting the user's upper body. The base frame includes a straight horizontal leg pipe 541a, a pair of vertical leg pipes 541b integrally connected at one end near both ends of the horizontal leg pipe 541a and expanding at the other end farther apart than the distance between the two ends, and a base frame member 541c integrally connected between the pair of vertical leg pipes 541b for attaching the support column. In this case, the drive mechanism 740 may be housed in a housing 542. Housing 542 includes hook portions 543 and 544, and is held in a suspended state from one of a pair of vertical leg pipes 541b by hook portions 543 and 544. As shown in Fig. 19D, a motor 545 is provided inside the housing, and motor 545 winds and releases wire 546. Although not shown in Fig. 19D, a processor that controls the drive of motor 545 and a memory that serves as a work area for the processor may be installed inside housing 542.

[0226] As shown in FIG. 19A, the caster Ca23 is provided with a brake 547. The brake 547 includes, for example, a plate-shaped member, and the caster Ca23 is locked when the plate-shaped member is pulled up. The above-mentioned wire 546 is connected to the plate-shaped member of the brake 547. Therefore, when the motor 545 winds up the wire 546, the plate-shaped member moves upward, and the caster Ca23 is locked. On the other hand, when the motor 545 returns the wire 546 to its original position, the plate-shaped member moves downward, and the caster Ca23 is unlocked.

[0227] Note that the configuration of the drive mechanism 740 in the walker 540 is not limited to this. For example, a second housing that houses a processor and memory may be provided in addition to the housing 542. The second housing may be fixed to, for example, the base frame member 541c. The motor 545 of the housing 542 and the processor of the second housing are electrically connected using a signal line. Also, while FIG. 19C describes a mechanism in which the caster Ca23 is locked, other casters may be locked. Furthermore, two or more casters among the casters Ca21 to Ca24 may be locked. For example, housings 542 may be provided near the casters Ca23 and Ca24, and both of these may be connected to the second housing provided on the base frame member 541c.

[0228] The driving mechanism 740 may also include various mechanisms for changing the height of the bottom of the bed 510. For example, the driving mechanism 740 may be a mechanism that lowers the height of the bottom by driving the legs of the bed 510 while maintaining the angle of the bottom.

[0229] Fig. 21 shows an example of the configuration of an information processing system 10 including a peripheral device 700. The wearable module 100 and the communication device 200 are the same as those in the example of Fig. 5. The server system 300 also acquires information from the communication device 200 that associates sensor information with identification information of the communication device 200, and performs a location-based fall determination process, similar to the above-described example.

[0230] In the example of FIG. 21, the peripheral devices 700 are exemplified by the table 530, the walker 540, and the bed 510 shown in FIGS. 19A to 19E, but the peripheral devices 700 may include other devices that can be moved by casters or other devices that can be adjusted in height.

[0231] Fig. 22 is a sequence diagram illustrating the processing in the system shown in Fig. 21. First, in step S201, the wearable module 100 determines whether there is a nearby connectable communication device 200. In step S202, a connection between the wearable module 100 and the communication device 200 is established.

[0232] In step S203, the wearable module 100 transmits sensor information detected by the acceleration sensor 120 to the communication device 200 using the communication module 130. In step S204, the communication device 200 performs a process of associating the sensor information received in step S203 with the identification information of the communication device 200. In step S205, the associated information is transmitted to the server system 300. In step S206, the server system 300 performs a fall determination process according to the location of the person being assisted based on the received information. The processes shown in steps S201 to S206 are the same as steps S101 to S106 in FIG. 9.

[0233] If it is determined that there is a risk of falling, in step S207, a process is performed to identify peripheral devices 700 located in the vicinity of the person being assisted based on at least one of the location where the communication device 200 is located, as identified by the placement information, and information identifying the person being assisted associated with the wearable module 100.

[0234] As described above, in this embodiment, control is performed to transition a device that a care recipient who is about to fall tries to grab in an instant to a state suitable for preventing the device from falling. Therefore, controlling a peripheral device 700 that is in a position that the care recipient cannot easily grab is not useful in preventing injuries due to falls. Furthermore, if a peripheral device 700 that is being used by another care recipient or caregiver is activated, it is dangerous and reduces convenience. Therefore, it is expected that multiple peripheral devices 700 will be used in a nursing care facility or the like, and it is necessary to appropriately determine which of the peripheral devices 700 will be controlled.

[0235] For example, as described above in the fall determination process, the processing unit 310 identifies the location where the communication device 200 is located based on the identification information of the communication device 200 associated with the sensor information. The processing unit 310 may identify the peripheral device 700 located in the identified location as the device to be controlled. For example, the server system 300 may store peripheral device information that associates the peripheral device 700 with the location where the peripheral device is located. The processing unit 310 identifies the peripheral device 700 located near the person being assisted who is at risk of falling as the device to be controlled based on the location identified based on the identification information of the communication device 200 and the peripheral device information. Note that the location information included in the peripheral device information may be information registered by a user such as a caregiver, or may be information that is dynamically changed by tracking processing using a sensor.

[0236] Alternatively, as shown in FIG. 8B , the processing unit 310 can identify the person being assisted associated with the wearable module 100 based on the module information. The server system 300 may also store peripheral device information that associates the peripheral device 700 with the person being assisted who uses the peripheral device 700. For example, in a nursing care facility, a schedule is established for what type of assistance is provided to which person being assisted at what time. This may allow the user to identify when and by which person being assisted a peripheral device 700, such as a walker 540, is used. Furthermore, because a bed 510 is likely to be occupied by a single person being assisted, it is easy to associate the peripheral device 700 with the person being assisted who uses it. Therefore, by identifying a person being assisted who is at risk of falling based on the identification information of the wearable module 100, which is the source of the sensor information, it is possible to identify a peripheral device 700 that is likely to be used by that person being assisted.

[0237] In step S208, the processing unit 310 performs processing to transmit a control signal to the identified peripheral device 700. In step S209, the control unit 710 of the peripheral device 700 operates the drive mechanism 740 in accordance with the control signal. Note that the control signal transmitted in step S208 may be a signal instructing locking or a signal instructing lowering of the bottom height. Alternatively, the control signal may be a signal indicating that there is a risk of tipping over, and the specific control content may be determined by the control unit 710 of the peripheral device 700.

[0238] Although the above description has been given of an example in which the casters of the peripheral device 700 that can be moved by casters are locked based on the risk of tipping over, the method of this embodiment is not limited to this.

[0239] As described above, the method of this embodiment prevents injuries due to a fall by having a person being assisted who is about to fall grab onto a stable peripheral device 700. Therefore, it is important that the distance between the person being assisted and the peripheral device 700 is close enough that the person being assisted can grab onto it instantly. Therefore, when a risk of falling is detected, the processing unit 310 may perform control to move the peripheral device 700 closer to the person being assisted by driving the casters of the peripheral device 700. In this way, the distance between the peripheral device 700 and the person being assisted is reduced, making it easier for the person being assisted to grab onto the peripheral device 700, thereby further reducing the impact of a fall.

[0240] For example, since the wearable module 100 of this embodiment has the acceleration sensor 120, it is possible to perform autonomous positioning based on sensor information from the acceleration sensor 120. Note that the positioning of the wearable module 100 may be performed by the communication device 200 or the server system 300. In particular, since the position of the communication device 200 is known, the position of the person being assisted can be estimated by correcting the autonomous positioning result using the presence or absence of communication with the communication device 200, the strength of the received radio wave during communication, and the like.

[0241] Furthermore, peripheral device 700 may include a device such as a smartphone compatible with communication device 200 as described above. These devices include an acceleration sensor, and therefore are capable of autonomous positioning, similar to wearable module 100. Furthermore, the position of the person being assisted can be estimated by correcting the results of autonomous positioning using the status of communication with other communication devices 200 and usage management information for equipment in a care facility or the like. The usage management information may include, for example, information on when and where a specific walker 540 is used, and where it is stored when not in use.

[0242] In this way, it is possible to estimate the position of the person being assisted and the position of the peripheral device 700. Although an example of autonomous positioning based on an acceleration sensor has been described above, position estimation may be performed by other methods, such as image processing using images captured by a camera installed in the care facility, or three-point positioning using a BLE beacon.

[0243] The processing unit 310 of the server system 300 identifies the positional relationship between the person being assisted who is likely to fall and the peripheral device 700 located near the person being assisted, based on the estimated position information. For example, the processing unit 310 estimates the direction and amount of movement required to move the peripheral device 700 closer to the person being assisted, and determines the amount of drive of the casters of the peripheral device 700 based on the estimation result. More specifically, the processing unit 310 may perform a process to determine the amount of rotation of a motor that drives the casters. The processing unit 310 notifies the peripheral device 700 of the determined amount of rotation, and the control unit 710 of the peripheral device 700 controls the motor to drive the determined amount of rotation. Furthermore, part of the processing in the server system 300 may be executed by the peripheral device 700 or the communication device 200 arranged in the peripheral device 700.

[0244] Furthermore, the processing unit 310 may perform control to move the peripheral device 700 to a range within a predetermined distance from the person being assisted, and then perform control to lock the peripheral device 700. In this way, the peripheral device 700 after control is in a position that is easy for the person being assisted to grab and is in a locked state, so that it is possible to appropriately suppress the effects of the person being assisted falling.

[0245] Furthermore, the peripheral device 700 is not limited to the bed 510, the table 530, and the walker 540, and may be other devices. For example, the peripheral device 700 may include an airbag worn by the person being assisted. The airbag is a device that is worn around the waist or the like of the person being assisted in a deflated state, and automatically inflates when it receives a control signal. For example, the airbag includes a communication module that communicates with the communication device 200 and a processor such as a microcomputer.

[0246] When the processing unit 310 detects a risk of a fall for an assisted person, it outputs a control signal to an airbag worn by the assisted person, instructing the airbag to inflate. The control signal is transmitted to an airbag processor, for example, via the communication device 200. The airbag processor controls the airbag inflation based on the control signal. In this way, it is possible to identify an assisted person at risk of falling and activate the airbag for that assisted person, thereby preventing injuries from occurring due to a fall.

[0247] Furthermore, the peripheral device 700 of this embodiment may be an airbag placed on a wall or floor of the toilet 600. When the processing unit 310 detects a risk of a care recipient falling in the toilet 600, the processing unit 310 may output a control signal to the airbag placed in the toilet 600 to instruct the airbag to inflate. This makes it possible to reduce the occurrence of injuries due to falls. In particular, the toilet 600 has a smaller area than a living room or dining room, making it easier to narrow down the locations of walls and floors that a care recipient may hit hard if they fall. Therefore, by placing an airbag in advance and inflating it according to the risk of a fall, it is possible to appropriately reduce the occurrence of injuries. However, this does not prevent airbags from being placed in locations other than the toilet 600.

[0248] In addition, in this embodiment, when a fall risk is detected, a notification may be sent to the caregiver terminal 400 as described above with reference to Fig. 5, or the peripheral device 700 may be controlled as described above with reference to Fig. 21, or both may be performed. In addition, which of these is performed may be switched depending on the result of the fall determination process.

[0249] For example, the output data of the fall determination process may output information specifying the time until a fall. As one example, as will be described later in relation to the explanation of walking ability, the process of determining whether a person will fall while walking (including the process of estimating walking ability) may perform pattern classification of sensor information from acceleration sensor 120. For example, storage unit 320 may hold a table that associates patterns with the time until a fall, and processing unit 310 may determine the time until a fall based on the result of pattern classification and the table.

[0250] The processing unit 310 may control the peripheral device 700 when the time until the fall is equal to or less than a predetermined threshold, and may notify the caregiver terminal 400 when the time until the fall is greater than the threshold. The control of the peripheral device 700 may, for example, activate an airbag. If the time until the fall is short, even if the caregiver terminal 400 is notified, the caregiver may not be able to intervene appropriately. For example, the caregiver may not be able to immediately support the person being assisted because they are not near the person being assisted or are assisting another person being assisted. In this regard, because airbag activation can be performed in a short time, it is possible to appropriately prevent injuries. Furthermore, if there is sufficient time until the fall, prioritizing the intervention of the caregiver can reduce the cost of replacing the airbag.

[0251] 3. Examples of location-specific processing The above describes the fall detection process as an example of location-dependent processing. However, the processing executed in each location is not limited to this. Below, we will explain techniques for appropriately utilizing tacit knowledge in specific situations, such as eating, adjusting position in bed 510 or wheelchair 520, and changing diapers.

[0252] As described above, in each of the following processes, the result of identifying the location of the person being assisted based on the location information may be used as at least one trigger. For example, the processing unit 310 identifies the location of the person being assisted based on the location information and executes control to activate sensors arranged at the location. Then, based on information from the activated sensors, the processing unit 310 executes each of the processes described below. Specifically, when the person being assisted is in a wheelchair 520, the processing unit 310 activates the seat sensors (pressure sensors Se1 to Se4) shown in FIG. 18. When the person being assisted is in bed 510, the processing unit 310 activates a detection device 810 that detects heart rate, breathing, body movement, etc., which will be described later with reference to FIG. 33, thereby starting processing related to getting out of bed and sleeping. When the person being assisted is in the toilet 600, the processing unit 310 may activate a pressure sensor arranged on the floor of the toilet 600. The processing unit 310 is not limited to activating all sensors arranged in the target location. For example, the processing unit 310 may select sensors to activate depending on the attributes of the target person being assisted. In this way, it becomes possible to appropriately activate the necessary sensors based on the location information.

[0253] However, the method of this embodiment is not limited to this, and the location and situation may be identified by other methods, and the following processes may be started based on the identification results. In other words, in each of the following processes, the process of identifying the location based on the location information is not essential.

[0254] 3.1 Meals For example, a person being assisted who uses a wheelchair 520 may transfer from a bed 510 in a room or the like to the wheelchair 520, then move to a dining room in the wheelchair 520 and sit at the table to begin eating. Therefore, in this embodiment, the processes described below may be executed when the wearable module 100 transmits sensor information to the communication device 200-5 located in the dining room. Alternatively, if the communication device 200-5 is omitted, the processes described below may be executed when the wearable module 100 transmits sensor information to the communication device 200-2 corresponding to the wheelchair 520 and it is determined that the wheelchair 520 is located in a place where meals are eaten, such as a dining room. The location of the wheelchair 520 may be determined by autonomous positioning using an acceleration sensor. Alternatively, a camera or other sensor may be placed in the dining room or the like, and the sensor may recognize the person being assisted, thereby determining whether the person being assisted is in a place where meals are eaten. The following processes may also be triggered by other conditions, such as pressing a start button displayed on the caregiver's mobile terminal device 410.

[0255] Figure 23 is a diagram illustrating tacit knowledge in eating. Figure 23 as a whole is tacit knowledge in eating, and the tacit knowledge is categorized into food form, thickness, and meal assistance. Food form corresponds to tacit knowledge for adjusting the form of the meal, such as the size at which ingredients are cut. Thickness corresponds to tacit knowledge for adjusting the degree of thickness of the meal. Meal assistance corresponds to tacit knowledge for supporting the person being assisted in the act of eating.

[0256] The "situation" in Figure 23 represents the situation of the person being assisted, and the action represents the action that the caregiver should take when that situation occurs. For example, an expert may determine, based on their own experience, whether the person being assisted is "becoming unable to chew" their food, and if that situation occurs, take action such as "breaking the food into small pieces on the spot," "stopping," or "seeking dietary advice from a dentist." In other words, the expert's tacit knowledge may be information that associates the situation of the person being assisted with the action that should be taken in that situation.

[0257] When multiple actions are associated with one situation, as in FIG. 23 , a priority may be assigned to each action. For example, in the case of the tacit knowledge corresponding to the example described above, the priority may be to "break up the food immediately and provide it to the patient" when the problem becomes difficult to chew, and if this does not resolve the issue, to "stop the action." Furthermore, at a different time after the meal is finished, the patient may attempt to restore eating function by "receiving dietary guidance from a dentist." These series of actions according to the situation are desirable responses by an experienced caregiver, and the method of this embodiment supports the caregiver so that similar actions can be performed regardless of the caregiver's level of expertise. Note that each action shown in FIG. 23 is an example of an action to be performed according to the situation, and other actions may be added. For example, for the situation "It's becoming difficult to chew," actions such as "reviewing the meal content" or "adjusting the amount of food" may be added. In other words, the actions in this embodiment may include an action to improve the situation when "it's becoming difficult to chew" occurs, and an action to make the situation "it's becoming difficult to chew" less likely to occur at a later time. This also applies to other situations.

[0258] An experienced caregiver can observe the state of the person being assisted and determine whether the situation corresponds to one of the situations shown in FIG. 23, such as whether the person is "finding it difficult to bite." However, in order to enable even beginners to provide assistance appropriate to the situation, it is necessary to automatically detect the situation of the person being assisted using a device including a sensor. As shown in FIG. 23, the tacit knowledge may include user attributes. This indicates the attributes to which the target tacit knowledge can be applied to the person being assisted. Therefore, in the method of this embodiment, the attributes of the person being assisted are determined, and whether or not each situation is subject to automatic detection may be switched based on the attributes.

[0259] FIG. 24 is a diagram illustrating devices used in a mealtime. As shown in FIG. 24, the devices used include a throat microphone™ worn around the neck of the person being assisted, and a communication device 200-5 equipped with a camera. Note that another terminal device equipped with a camera may be used instead of the communication device 200-5. The throat microphone™ outputs audio data resulting from the person being assisted swallowing, coughing, etc. The camera of the communication device 200-5 outputs captured images of the person being assisted eating. The communication device 200-5 is, for example, a smartphone placed on a table where the person being assisted eats. As described above with reference to FIG. 2, a wearable module 100 is worn on the chest or the like of the person being assisted.

[0260] The audio data from the throat microphone™ and the captured image by communication device 200-5 are transmitted to server system 300. For example, communication device 200-5 acquires audio data from the throat microphone™ using Bluetooth or the like, and transmits the audio data and the captured image captured using a camera to server system 300. Note that the audio data and the captured image may also be transmitted to server system 300 via communication device 200-2 placed in wheelchair 520. Various other variations are possible for the method of transmitting the output of each device to server system 300.

[0261] Figure 25 is a diagram illustrating the association between the above-mentioned devices and the situation shown in Figure 23. As shown on the left side of Figure 25, devices used in the implicit knowledge of eating include, for example, the throat microphone™, the camera of communication device 200-5, and the acceleration sensor 120 of wearable module 100. In Figure 25, the items written on the lines extending from each device represent information that can be determined based on the device. The area surrounded by a dashed line in Figure 25 represents the situation shown in Figure 23.

[0262] The ThroatMic™ determines whether the person receiving care choked or swallowed. A device that detects swallowing using a microphone worn around the neck is described, for example, in U.S. Patent Application No. 16 / 276,768, filed February 15, 2019, entitled "Swallowing action measurement device and swallowing action support system." This patent application is incorporated by reference in its entirety. By using the ThroatMic™, the processing unit 310 can detect the number of choking incidents, the duration of the choking (time of occurrence, duration, etc.), and whether or not the person swallowed, as shown in FIG. 25.

[0263] The camera of the communication device 200-5 can detect the mouth, eyes, chopsticks, spoon, etc. of the person being assisted by capturing an image of the person being assisted from the front, as shown in Fig. 24. There are various known methods for detecting these facial features and objects based on image processing, and a wide range of known methods can be applied in this embodiment.

[0264] For example, the processing unit 310 can determine, based on the image captured by the camera, whether the person being assisted has their mouth open, whether food is coming out of their mouth, and whether they are chewing their food. The processing unit 310 can also determine, based on the image captured by the camera, whether the person being assisted has their eyes open. The processing unit 310 can also determine, based on the image captured by the camera, whether chopsticks, a spoon, or the like are near tableware, whether the person being assisted can hold them, and whether they are spilling food.

[0265] In the method of this embodiment, the condition of the person being assisted is estimated based on information that can be identified from these devices. For example, the processing unit 310 may perform processing to identify an action that the caregiver should take based on the detection results of choking and swallowing and the determination results of whether the person being assisted is opening or closing their mouth.

[0266] For example, as shown in Fig. 25, it is possible to determine whether a situation corresponds to "frequent choking" based on the number of times and duration of choking. For example, the processing unit 310 may determine that choking has occurred frequently when the number of times per unit time exceeds a threshold. In this way, choking situations can be automatically determined, making it possible to present appropriate actions to the caregiver.

[0267] 25, the processing unit 310 may calculate the swallowing time from when the person being assisted opens their mouth until they swallow, based on the swallowing detection result and the result of determining whether the person being assisted has opened or closed their mouth, and may perform processing to identify an action to be taken by the caregiver based on the calculated swallowing time. The detection of swallowing itself is described in U.S. Patent Application No. 16 / 276,768. However, even if it is determined that the number of swallows has decreased, it is not easy to determine the specific situation, such as whether the person is not even putting food in their mouth, or whether they have put food in their mouth but are not swallowing.

[0268] In this regard, by determining the swallowing time from when the mouth is opened to when the patient swallows, it is possible to calculate the time required for chewing and swallowing. For example, the processing unit 310 may start counting up a timer when the mouth transitions from a closed state to an open state based on the image captured by the communication device 200-5, and stop counting the timer when swallowing is detected by the throat microphone™. The time when the timer stops represents the swallowing time. In this way, it is possible to accurately determine whether a situation requires the caregiver to take some kind of action during mealtime, making it possible to appropriately utilize the tacit knowledge of experts.

[0269] For example, if the swallowing time is short, it can be determined that the situation is "if the pace is fast." Furthermore, if the swallowing time is long, the processing unit 310 may determine whether there are other circumstances that need to be taken into consideration based on other situation determination results using a device. The processing unit 310 may also determine whether the swallowing time is long based on changes in the swallowing time during a single meal (for example, the increase or ratio compared to the swallowing time at the beginning). Alternatively, the processing unit 310 may calculate the average swallowing time for each of multiple meals for the same person receiving care, and determine whether the swallowing time has become longer based on the change in the average swallowing time.

[0270] For example, by using the results of determining whether the mouth is open or closed based on the captured image of the communication device 200-5, it can be determined whether the person "no longer opens his / her mouth" even when the caregiver approaches a spoon or other object. In this way, if the person being assisted is reluctant to open his / her mouth and the swallowing time becomes longer, it can be inferred that "stopping" has occurred. Furthermore, by using the captured image to determine whether food is coming out of the mouth and whether the person is chewing, it can be determined whether the person "cannot chew" has occurred. For example, if the number of chews is normal but the swallowing time is long, it can be inferred that the person "cannot chew" has occurred. Furthermore, if the captured image determines that the eyes are closed, it can be determined that the person "seems sleepy." Note that the above is an example of situation determination, and the processing content is not limited thereto. For example, the processing unit 310 may infer that "stopping" has occurred when the person spits out food based on the captured image. For example, in the case of an assisted person with advanced dementia, the person may forget that they are eating and open their mouth, causing stopping. For example, the processing unit 310 may switch the content of the situation determination process based on device data based on attributes of the person being assisted, such as the degree of progression of dementia.

[0271] 25, it may be determined whether the person being assisted is dozing off based on the fall determination process described above. For example, the processing unit 310 determines that the person being assisted is dozing off when the person's posture becomes worse than normal or when periodic body swaying is detected. In this case, the processing unit 310 determines that the person being assisted is "looking sleepy," just as if the person had their eyes closed.

[0272] On the other hand, when other situation determination results are taken into consideration and there are no circumstances that would increase the swallowing time, the processing unit 310 determines that the situation is "the time until swallowing has increased." As an example, this corresponds to a case where the person being assisted has become full, but here, a device that senses the degree of fullness is not assumed, and it is not a direct determination of whether the person being assisted has become full.

[0273] 25, by performing a recognition process for chopsticks, spoons, etc. using the captured image, it may be determined whether the situation is "playing with food," "unable to hold a bowl," "spilling food," etc. Furthermore, based on the above-mentioned fall determination process, a situation determination such as "poor posture" may be made.

[0274] As described above, by appropriately using the output of each device, it is possible to determine the situation of the person being assisted. Furthermore, by storing information associating situations with actions as the tacit knowledge of an expert, as shown in FIG. 23, it is possible to present the caregiver with an appropriate action according to the situation. The information may be presented to the caregiver, for example, as an audio output to the headset 420, as a display on the display unit of the mobile terminal device 410, or as a presentation using another method. For example, since the person being assisted is sitting in a wheelchair 520, a notification may be made by illuminating a light-emitting unit provided on the wheelchair 520.

[0275] In particular, the method of this embodiment uses the swallowing time from opening the mouth to swallowing as the primary condition, as described above, to appropriately determine whether the basic actions of eating, such as putting food in the mouth, chewing, and swallowing, are impaired. Furthermore, by combining other situation determination results as additional conditions, the specific factors that cause the swallowing time to be long can be narrowed down, making it possible to estimate the situation in more detail and present appropriate actions. As a result, instructions appropriate to the situation can be given to the caregiver during mealtimes, making it possible to appropriately utilize the tacit knowledge of experts.

[0276] Furthermore, in the method of this embodiment, when an action to stop eating is presented, the processing unit 310 may perform control to increase the number of activated sensors included in the wearable module 100. For example, the wearable module 100 may include a temperature sensor in addition to the acceleration sensor 120. For example, when the wearable module 100 is fixed to the skin of the person being assisted, the temperature sensor can measure the temperature of the body surface, and therefore the body temperature of the person being assisted can be estimated based on the measured value.

[0277] This allows appropriate monitoring of vital signs of the person being assisted, for example, when there is a possibility of aspiration pneumonia. The temperature sensor may be activated for several hours, several days, or some other period after the event of stopping eating is detected. Furthermore, if the wearable module 100 includes sensors capable of detecting heart rate, respiration, SpO2, etc., these sensors may be activated when an action to stop eating is presented.

[0278] 3.2 Position adjustment The position of the person being assisted needs to be adjusted in bed 510 or wheelchair 520. For example, adjusting the position in bed 510 is useful for preventing bedsores. Adjusting the position in wheelchair 520 is also useful for preventing slippage and preventing bedsores. Therefore, if it is determined based on the communication results between wearable module 100 and communication device 200 that the person being assisted is in bed 510, processing may be executed to support assistance in adjusting the bed position. Similarly, if it is determined that the person being assisted is in wheelchair 520, processing may be executed to support assistance in adjusting the wheelchair position. Specific examples are described below.

[0279] 3.2.1 Bed position adjustment FIG. 26 is a diagram illustrating devices arranged around a bed 510. As shown in FIG. 26, the devices here include a communication device 200-1 fixed to the footboard side of the bed 510, a second terminal device CP2 fixed to a side rail of the bed 510, and a display DP fixed on the opposite side of the second terminal device CP2. The second terminal device CP2 may be the communication device 200 according to this embodiment, or may be a device that does not function as the communication device 200. The communication device 200 corresponding to the bed 510 may be provided at another position, such as a wall of the room, and another terminal device that does not function as the communication device 200 may be used instead of the communication device 200-1. The display DP is not limited to being fixed to the bed 510, and may be arranged at another position where it can be viewed naturally by a caregiver adjusting the bed position. For example, the display DP may be fixed to a wall surface or a stand that stands on the floor. Either the communication device 200-1 or the second terminal device CP2 may be omitted. For example, an example in which the bed position is adjusted using the communication device 200-1 will be described below. The second terminal device CP2 is used, for example, for changing a diaper, which will be described later. The communication device 200-1 may also be used for changing a diaper.

[0280] The communication device 200-1 and the second terminal device CP2 are devices such as smartphones equipped with cameras. The communication device 200-1 transmits captured images directly to the server system 300. The second terminal device CP2 transmits captured images from the camera to the server system 300 directly or via the communication device 200-1. The display DP receives images transmitted from the server system 300 directly or via another device such as the communication device 200-1, and displays the received images. Note that the communication device 200-1 and the second terminal device CP2 may have a depth sensor instead of or in addition to the camera. That is, these devices may output depth images.

[0281] For example, in adjusting a bed position, a process of registering teacher data and a process of adjusting a position using the teacher data may be performed. The teacher data is, for example, information registered by an experienced caregiver. When adjusting the bed position, an unskilled caregiver selects the teacher data and adjusts the bed position so that the state of the actual person being assisted matches the teacher data. For example, the communication device 200-1 acquires a captured image of the person being assisted lying on the bed (including the state of the cushion, etc.), and the display DP displays an image showing the comparison result between the captured image and the teacher data. In this way, it becomes possible for a caregiver to perform position adjustment similar to that of an experienced person, regardless of the caregiver's level of skill.

[0282] Fig. 27 is an example of a registration screen for teacher data. Fig. 27 shows an image including a captured image captured by, for example, communication device 200-1, and is a screen displayed on, for example, a display unit of mobile terminal device 410 of an expert. Note that the captured image for teacher data may be captured using mobile terminal device 410. Furthermore, the teacher data may be registered using a device other than mobile terminal device 410.

[0283] The skilled person lies the person being assisted on the bed 510, positions the person in a position suitable for preventing bedsores, etc., and then uses the communication device 200-1 to capture an image of the person being assisted. The display unit of the portable terminal device 410 may display the images captured by the communication device 200-1 as moving images in real time, or may display still images captured by the communication device 200-1. After confirming that the bed position is appropriate, the skilled person selects the registration button. The portable terminal device 410 transmits the still image displayed when the registration button was pressed to the server system 300 as training data. In this way, it becomes possible to register the position that the skilled person considers to be preferable as training data.

[0284] At this time, the mobile terminal device 410 may accept an input operation of additional information by an expert. For example, the expert performs an operation to select a portion that is considered to be particularly important using an operation unit such as a touch panel of the mobile terminal device 410. For example, the expert user performs an operation to acquire a captured image of the person being assisted in an appropriate bed position and an operation to add additional information, and then selects the registration button shown in FIG.

[0285] In the example of FIG. 27, the vicinity of the left shoulder and the vicinity of the right knee of the person being assisted are selected. The mobile terminal device 410 may also be capable of inputting specific text, etc., in addition to specifying a position. For example, an experienced person may not only specify the left shoulder, but also input text about important points for achieving an appropriate bed position, such as the angle with other parts and the positional relationship with a pillow or cushion. The same applies to the vicinity of the right knee. The mobile terminal device 410 may also accept input of the priority of each position when there is an input specifying multiple positions. For example, if the smaller the value, the higher the priority, and the mobile terminal device 410 accepts user input indicating that the priority of the vicinity of the left shoulder is relatively high, the mobile terminal device 410 sets the priority of the vicinity of the left shoulder to 1 and the priority of the vicinity of the right knee to 2.

[0286] Furthermore, when the caregiver actually adjusts the bed position, the communication device 200-1 is first activated and begins capturing images. For example, the caregiver activates the communication device 200-1 by voice, and the display DP displays the moving image captured by the communication device 200-1. The processing unit 310 of the server system 300 may also accept a selection process of teacher data by the caregiver. For example, the processing unit 310 may display a list of teacher data on the display unit of the mobile terminal device 410. The processing unit 310 determines teacher data based on a selection operation on the mobile terminal device 410, and controls the display DP to display the teacher data.

[0287] Alternatively, the processing unit 310 may automatically select the training data based on a similarity determination between the attributes of the person being assisted whose bed position is to be adjusted and the attributes of the person being assisted captured in the training data. The attributes here include information such as the age, sex, height, weight, medical history, and medication history of the person being assisted.

[0288] Alternatively, the processing unit 310 may automatically select training data based on a comparison between the attributes of the person being assisted whose bed position is to be adjusted and additional information contained in the training data. For example, the additional information in the training data may include text such as "For a person being assisted who shows a tendency of XX, it is recommended to adjust the left shoulder so that it is YY." In this case, if the person being adjusted corresponds to XX, the training data is likely to be selected. For example, the caregiver adjusting the bed position may send information identifying the person being assisted to the server system 300 via a mobile terminal device 410 or the like, and the processing unit 310 may identify the attributes of the person being assisted based on the information.

[0289] The processing unit 310 may also classify the person being assisted into several classes using the results of the fall detection process or assessment equipment such as the Waltz in or SR AIR described above. The processing unit 310 may then perform processing to automatically select training data based on a comparison process between the class of the person being assisted to be adjusted and the class of the person being assisted captured in the training data.

[0290] The processing unit 310 may perform processing to superimpose, for example, the teacher data that has been subjected to transparency processing on the real-time captured image captured by the communication device 200-1. Fig. 28 shows an example of an image in which the teacher data shown in Fig. 27 is superimposed. In this way, even a less skilled caregiver can easily adjust the bed position by adjusting the image so that the actual person being assisted and the person being assisted in the teacher data overlap.

[0291] Furthermore, as shown in FIG. 28, additional information on the training data may be displayed in a recognizable manner. For example, in FIG. 28, objects that are numbers surrounded by circles are displayed at positions near the left shoulder and near the right knee specified by the expert. The caregiver adjusting the bed position can grasp important points by looking at the objects. Furthermore, the processing unit 310 may display text added by the expert on the display DP when an object selection operation is performed. Furthermore, when it is detected that the caregiver has uttered "Tell me the key points" using the microphone of the headset 420, the processing unit 310 may output the text as voice from the headset 420.

[0292] The processing unit 310 determines whether the image captured during position adjustment is OK or NG based on the degree of similarity between the image and the training data, for example, and displays the determination result on the display DP. Alternatively, the processing unit 310 may output the determination result as audio from the headset 420. The processing unit 310 may also perform processing to display specific points determined to be NG. For example, the processing unit 310 may compare the image captured by the communication device 200-1 with the training data, and perform processing to highlight areas determined to have a large difference.

[0293] In this way, by providing the display DP at a position different from the communication device 200-1 that captures images, for example, on the side frame side, the caregiver can view the display DP in a natural posture while adjusting the position of the person being assisted. For example, since there is no need to view the captured image of the communication device 200-1 using the display unit of the communication device 200-1, convenience can be improved.

[0294] In this case, as shown in Figures 27 and 28, it is possible to register points that the expert considers important as additional information and present this additional information to the caregiver. If an assistant with a low level of experience only looks at the images in the training data, they may be able to imitate the position, but they will not understand the particularly important points and will therefore be unable to prioritize when adjusting the position. In this regard, the method of this embodiment clearly conveys the expert's intentions, allowing even an assistant with a low level of experience to appropriately utilize tacit knowledge.

[0295] In addition, when superimposing training data in the form of a photograph as shown in Figures 27 and 28, information about objects in the background, such as cushions, is also stored in the training data, which has the advantage of allowing for appropriate adjustment of the positional relationship between the person being assisted and the cushion.

[0296] 29 is a diagram for explaining another method for adjusting the bed position, illustrating the results of skeletal tracking. Note that various image-based skeletal tracking methods are known, such as OpenPose, disclosed in Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields by Zhe Cao et al. (https: / / arxiv.org / pdf / 1611.08050.pdf), and these methods can be widely applied in this embodiment.

[0297] For example, when registering training data, the skilled worker, as in the example described above, has the person being assisted lie down on the bed 510, position the person in a position suitable for preventing bedsores, and then captures an image of the person being assisted using the communication device 200-1. The processing unit 310 performs skeletal tracking on the captured image and displays a predetermined number of resulting positions on the captured image. The number of tracked locations is, for example, 17, but is not limited to this.

[0298] The processing unit 310 may include all of the results of skeletal tracking in the training data. Alternatively, the processing unit 310 may accept an operation to select some of the points detected by skeletal tracking. For example, an expert may specify three points that are considered important for adjusting the bed position. As an example, an expert may specify three points: shoulders, hips, and knees. However, the combination of specified parts is not limited to this, and the number of specified parts is not limited to three.

[0299] In adjusting the bed position using training data, as in the example described above, a captured image of the person being assisted is acquired by the camera of the communication device 200-1. The server system 300 performs skeletal tracking on the captured image and displays the processing result on the display DP. The processing result is, for example, as in FIG. 29, an image in which the results of skeletal tracking of the captured image registered as training data, the captured image currently being captured by the camera of the communication device 200-1, and the results of skeletal tracking for the currently captured image are superimposed. In this case, the captured image registered as training data itself is not displayed. Note that in this case, all points detected by skeletal tracking may be displayed, or only some of the points specified by the expert may be displayed.

[0300] Adjusting the bed position using the results of skeletal tracking is different from the previously described method of overlaying captured images (images from training data and the image currently being captured), and is advantageous in that it can be applied even when the equipment used by the person being assisted, such as cushions, is different, making it highly versatile.

[0301] The processing unit 310 performs a process of comparing the three points of the shoulders, waist, and knees in the training data with the three points of the shoulders, waist, and knees in the captured image. For example, the processing unit 310 may determine whether the three points of the shoulders, waist, and knees are at a desired angle, or whether the three points are within a certain range of a straight line. The processing unit 310 may determine, for example, whether the result is OK or NG, and display the determination result on the display DP. Alternatively, the processing unit 310 may output the determination result as audio from the headset 420. The processing unit 310 may also perform a process of displaying the specific points determined to be NG.

[0302] The above describes bed position adjustment when the person being assisted lies on a mattress that is parallel (including approximately parallel) to the floor surface, but this is not limited to this, and bed position adjustment may also be made according to the situation (scene) of the person being assisted.

[0303] The bed position adjustment may also be performed by controlling the bed 510. For example, if choking occurs due to posture when eating, the meal can be eaten smoothly by controlling to change the angle of the bed bottom of the bed 510. Control to change the angle of the bed bottom includes controls such as raising the backboard, raising the waistboard, and tilting.

[0304] For example, the expert may register training data in association with information identifying the target situation. In the above example, training data is acquired in which the situations are "eating" and "frequent choking" and tags indicating that the cause of choking is "posture" are associated with a captured image of the person being assisted after the bottom adjustment. The caregiver who actually provides care adjusts the bed position by changing the bottom angle based on the training data. Alternatively, the control to change the bottom angle may be performed automatically, and the caregiver may provide care by adjusting the details of the bed position based on the training data. In other words, in addition to or instead of a notification on the caregiver terminal 400, the bed 510, which is the peripheral device 700, may be controlled.

[0305] Furthermore, the processing unit 310 may determine the situation based on the device and automatically select teacher data based on the determination result. The situation determination can be performed using a method similar to the processing described above using Figures 23 and 25. For example, if the throat microphone™ determines that choking is occurring frequently and the acceleration sensor 120 determines that the cause of the choking is posture, the processing unit 310 makes it easier to select the above-mentioned teacher data.

[0306] When snoring is detected while the user is sleeping, the bed position may be adjusted by controlling the pillow rather than the bed 510. For example, the following URL discloses Motion Pillow, which has a built-in airbag and inflates the built-in airbag when snoring is detected, thereby encouraging the user to turn over in bed. For example, when the processing unit 310 detects a situation in which the user is "sleeping" or "snoring" occurs, the processing unit 310 may encourage the user to move to a lateral position by controlling the airbag of the pillow. In other words, the peripheral device 700 that is the target of intervention control may include a pillow. http: / / www.motionpillow.com /

[0307] 3.2.2 Wheelchair position Fig. 30 is a diagram illustrating a system configuration when adjusting the wheelchair position. As shown in Fig. 30, adjusting the wheelchair position may use a third terminal device CP3 that has a camera and is fixed at a height that allows the camera to capture at least the upper body of the person being assisted sitting in the wheelchair 520. The third terminal device CP3 may be capable of capturing a wider range of the person being assisted, for example, up to the knees, or the entire body. The third terminal device CP3 is placed in a predetermined position in, for example, a nursing facility, and the caregiver transfers the person being assisted into the wheelchair 520, moves the person to the front of the third terminal device CP3, and then adjusts the wheelchair position.

[0308] The third terminal device CP3 includes a display unit and displays the comparison results between the image captured by the camera and the training data. The training data is registered in the same manner as for the bed position, and may be data in which additional information is added to the captured image as shown in Fig. 27, or data in which the results of skeletal tracking are added as shown in Fig. 29. The processing unit 310 may superimpose the training data that has been subjected to transparency processing on the display unit of the third terminal device CP3, or may display the difference in the results of skeletal tracking.

[0309] When the system shown in Fig. 30 is used, the camera of the third terminal device CP3 can capture an image of the person being assisted from the front, and therefore the face of the person being assisted can be captured more clearly than when a device fixed to the bed 510 such as the communication device 200-1 in Fig. 26 is used. Therefore, when automatically selecting training data according to the person being assisted, the processing unit 310 may automatically identify the person being assisted to be adjusted based on the results of face recognition processing.

[0310] Note that instead of the third terminal device CP3, a communication device 200-5 shown in Fig. 24 that is placed at a table where a meal is eaten may be used. In this case, since it is difficult for the camera of the communication device 200-5 to capture an image of the lower half of the body of the person being assisted, the processing unit 310 performs a determination process based on the upper half of the body of the person being assisted. Note that when the communication device 200-5 shown in Fig. 24 is used, the processing unit 310 may detect forward or lateral deviation based on the captured image. For example, the processing unit 310 determines that a forward deviation has occurred if the head and shoulder positions are lower than when the meal started, and determines that a lateral deviation has occurred if the head and shoulder positions are laterally shifted.

[0311] The same applies to wheelchair position in that position adjustment may include control of devices, etc. For example, if choking or the like occurs due to posture while eating, the processing unit 310 may automatically perform controls such as raising the backrest, tightening the ceiling, or pulling the seat back, or may perform a presentation process to prompt the caregiver to perform such controls. For example, if posture is detected using the pressure sensor shown in Figure 18, the processing unit 310 may continue the above-mentioned controls until it is determined based on the pressure sensor that the position of the center of gravity has returned to a normal state.

[0312] Furthermore, in the case of a person being assisted using a wheelchair 520, since the person is at least able to remain in a seated position, there is a possibility that the person can correct his / her posture by himself / herself. In this case, a device with a relatively large display size may be used as the third terminal device CP3. In this case, the person being assisted can use the third terminal device CP3 as if it were a full-length mirror, since an image of the person being assisted is displayed on the third terminal device CP3 in front of the person being assisted. For example, as described above, by displaying the parts that need to be corrected on the third terminal device CP3, it is possible to encourage the person being assisted to correct their posture.

[0313] 3.3 Diaper changing It was found that the following points are important to experts in diaper changing tacit knowledge: A.Is the patient in a lateral position? B. Is the diaper in the correct position? C. Check if the padding is coming out of the diaper D. Was the diaper properly fitted?

[0314] Therefore, in this embodiment, it is determined whether the above points A to D are met and the determination result is displayed. This makes it possible for the caregiver to change the diaper appropriately, regardless of their skill level.

[0315] The system for changing diapers is similar to that shown in Fig. 26. For example, the second terminal device CP2 captures video of the person being assisted using a camera and transmits the video to the server system 300 directly or via the communication device 200 placed on the bed 510. The processing unit 310 of the server system 300 performs skeletal tracking processing on each image constituting the video, and displays an image on the display DP in which the skeletal tracking results are superimposed on the original image. In this way, the caregiver can check the display DP in a natural posture while changing the diaper of the person being assisted.

[0316] In consideration of cases where diaper changes are performed at night, the second terminal device CP2 may include a lighting unit. Furthermore, in consideration of the privacy of the person being assisted, a depth sensor or the like may be used instead of a camera. The depth sensor may be a sensor using a ToF (Time of Flight) method, a sensor using structured illumination, or a sensor using another method.

[0317] 31A and 31B are examples of images displayed on the display DP when changing a diaper. As described above, each image includes the person being assisted and the results of skeletal tracking of the person being assisted.

[0318] In the state shown in Figure 31A, the person being assisted is stable in a lateral position, and the camera of the second terminal device CP2 captures an image of the person being assisted directly from behind. For example, in Figure 31A, there is a small difference between the front-to-back direction of the person being assisted's body and the direction of the camera's optical axis. As a result, as shown in Figure 31A, many points that can be detected by skeletal tracking are detected.

[0319] On the other hand, in Figure 31B, the posture is less stable than in Figure 31A, and the person being assisted appears to be on the verge of falling onto his back. Because the camera of the second terminal device CP2 captures the image of the person being assisted from diagonally behind, the number of points detected by skeletal tracking decreases. For example, the points corresponding to the waist are not detected because they are hidden by a diaper or other object.

[0320] Therefore, the processing unit 310 may determine whether or not the subject is in the lateral position shown in A above based on the results of skeletal tracking. For example, the processing unit 310 may determine that the subject is in the lateral position when a point corresponding to a specific part such as the waist is detected by skeletal tracking. However, the determination of the lateral position may also be based on whether or not a point other than the waist is detected, or the relationship between multiple points, and the specific method is not limited to this.

[0321] The processing unit 310 also performs object tracking processing based on the video image from the second terminal device CP2 to continuously detect the diaper area in the image. Object tracking is well known, so a detailed description will be omitted. For example, in Figures 31A and 31B, the diaper area ReD is detected.

[0322] The processing unit 310 may determine whether the position of the diaper shown in B above is appropriate based on, for example, the relationship between the results of skeleton tracking and the diaper region ReD detected by object tracking. For example, taking into account the position where the diaper is worn, it determines whether the waist position detected by skeleton tracking and the diaper region ReD have a predetermined positional relationship. For example, the processing unit 310 may determine that the diaper position is appropriate if a line including two points corresponding to the pelvis passes through the diaper region ReD. Alternatively, the processing unit 310 may extract the results of skeleton tracking and the detection results of the diaper region ReD as features from training data by an expert, and perform machine learning using these features as input data. The trained model is a model that, for example, outputs the likelihood that the diaper position is appropriate when it receives the results of skeleton tracking and the detection results of the diaper region ReD.

[0323] The processing unit 310 may also determine whether the pads are protruding from the diaper (see C above) based on the horizontal length of the diaper area ReD. Because pads typically fit inside the diaper, the length of the diaper area ReD in the image corresponds to the length of the diaper itself. The expected size of the diaper area ReD can be estimated based on the type and size of the diaper and the optical characteristics of the camera of the second terminal device CP2. On the other hand, if the pads are protruding, the length of the diaper area ReD in the image will be correspondingly longer. Therefore, if the length of the diaper area ReD detected from the image is greater than the expected length by a predetermined threshold or more, the processing unit 310 determines that the diaper is protruding from the pads and is inappropriate.

[0324] The processing unit 310 may also determine whether the diaper described above in D is properly put on by detecting the tape used to secure the diaper when it is worn. Typically, the tape is a different color from the diaper itself. For example, the diaper itself is white and the tape is blue. The way the tape should be secured to properly put on the diaper is known from the diaper's structure. Therefore, the processing unit 310 can detect the tape area in the image based on the color and determine whether the diaper is properly put on based on the relationship between the tape area and the diaper area ReD or the relationship between the tape area and the position of the waist or other parts detected by skeletal tracking. When multiple diapers of different manufacturers or types are used, the processing unit 310 may acquire information identifying the diapers and determine whether the diapers are properly put on based on the identified type of diaper, etc.

[0325] In this way, it is possible to appropriately utilize tacit knowledge in diaper changing and have the caregiver appropriately change the diaper. For example, the processing unit 310 judges each of the above A to D as OK or NG and displays the judgment result on the display DP. Furthermore, if the judgment is NG, the processing unit 310 may highlight the part that is significantly different from the correct data.

[0326] The processes described above are merely examples of automating the determinations A to D using a device, and other methods may be used. For example, a pressure sensor may be used instead of skeletal tracking when determining the lateral position. For example, the pressure sensor may be placed closer to the side frame than the center of the bed or mattress (offset to the left or right of the center). A caregiver can transition a person being assisted, who is lying face-up near the center of the bed, into the lateral position by rotating the person 90° to either the left or right. In other words, when the lateral position is achieved, the person's body moves toward the side frame by the amount of rotation, increasing the load on the pressure sensor. The processing unit 310 may determine that the person has transitioned to the lateral position when the output value of the pressure sensor is equal to or greater than a predetermined value.

[0327] Furthermore, when the person being assisted is made to sit up to change a diaper, it is expected that the person being assisted will hold on to the side rails. Therefore, a pressure sensor may be provided on the side rails, and the processing unit 310 may determine that the person has transitioned to the lateral position when the output value of the pressure sensor is equal to or greater than a predetermined value.

[0328] When determining the lateral position of the person being assisted in step A using a pressure sensor or the depth sensor described above, if the determination determines that the person being assisted is in a lateral position, image capture by the camera of the second terminal device CP2 and the determination of step B and beyond may be initiated. For example, when changing a diaper at night, the processing unit 310 may turn on the lights of the second terminal device CP2 when it determines that the person being assisted is in a lateral position. Alternatively, the processing unit 310 may turn on the lights in the person being assisted's room when it determines that the person being assisted is in a lateral position. In this way, it becomes possible to appropriately control the lights at the timing when processing using the image captured by the camera is required.

[0329] The processing unit 310 may also execute processing related to diaper changing using the communication device 200-1 shown in FIG. 26. FIG. 31C is an example of an image displayed on the display DP when a diaper change is performed based on an image captured by the communication device 200-1. The output of the communication device 200-1 is an image of a person being assisted in a supine position captured from the footboard side. As shown in FIG. 31C, the displayed image includes the person being assisted, the skeletal tracking results of the person being assisted, and the diaper area ReD. Note that FIG. 31C illustrates waist detection results Det1 and Det2 as skeletal tracking results, but as described above with reference to FIG. 29, detection results for other parts may also be displayed.

[0330] For example, instead of the determination in A described above, the processing unit 310 may determine whether the person being assisted is in a posture suitable for diaper changing based on a comparison process between teacher data that indicates a posture suitable for diaper changing in the supine position and an actual captured image. For example, the processing unit 310 may superimpose the teacher data and the captured image on the display DP, as in the case of adjusting the bed position, or may compare the results of skeletal tracking.

[0331] The processing unit 310 may also determine whether the diaper is OK or NG from the viewpoints of B to D described above. For example, with regard to B above, the processing unit 310 may designate a trapezoidal region for the waist positions (Det1 and Det2) detected by skeletal tracking and determine whether the diaper is set along the trapezoidal region. For example, the processing unit 310 may determine that the diaper is OK if the center of the diaper is located on the perpendicular line of a line segment connecting two waist points or within a predetermined distance from the perpendicular line, and the trapezoidal region and the diaper region ReD have a predetermined positional relationship (for example, the trapezoidal region is contained within the diaper region ReD). The trapezoidal region here is a region set based on the diaper region ReD in the training data. For example, the trapezoidal region is a region in which the perpendicular bisectors of the upper and lower bases respectively coincide (including approximately coincide) with the perpendicular bisector of the line segment connecting the waist detection results Det1 and Det2, and has a predetermined height. For example, the trapezoidal region is a region that includes the waist detection results Det1 and Det2, with the distance from Det1 to the upper base being H1 and the distance from Det1 to the lower base being H2, and H1 and H2 may be stored as parameters in the storage unit 320 or the like. However, the relationship between the waist detection results Det1 and Det2 and the trapezoidal region is not limited to this and various modifications are possible. Furthermore, the position and size of the trapezoidal region may be fixed values ​​or may be dynamically changed according to the positions of the waist detection results Det1 and Det2.

[0332] The above-mentioned determinations C and D are the same as those when the second terminal device CP2 is used, and therefore detailed explanations thereof will be omitted.

[0333] 3.4 Capacity estimation and use of the estimation results In this embodiment, the abilities of the person being assisted may be estimated using the process described above. The abilities here include the ability to maintain a sitting position, the ability to walk, and the ability to swallow. Each of these will be described below.

[0334] 3.4.1 Ability to maintain sitting position As shown in the fall determination process in wheelchair 520 and the meal-related process described above using FIG. 25 and the like, the method of this embodiment makes it possible to detect forward or lateral slippage in wheelchair 520. This may be detected by wearable module 100, by the pressure sensor shown in FIG. 18A, or by using a device such as Waltzin or SR AIR. Forward or lateral slippage may also be detected when eating in bed 510. Similarly, in the case of bed 510, wearable module 100 may be used, a pressure sensor arranged in bed 510 may be used, or Waltzin or SR AIR may be used.

[0335] For example, the processing unit 310 measures the time from when the person starts eating on the wheelchair 520 until their posture collapses. The level of the person's ability to maintain a sitting position is evaluated based on the length of this time. The processing unit 310 may also evaluate whether the person's posture collapses due to a forward or lateral shift (right or left shift) and the degree of the shift. Furthermore, the processing unit 310 may classify the person being assisted into multiple classes based on the presence or absence of the ability to maintain a sitting position, the degree of the ability, and the degree of the lateral or forward shift. When Waltzin or SR AIR is used, more detailed classification may be performed using time-series changes in pressure distribution.

[0336] Note that evaluation of sitting ability, such as JSSC misalignment measurement, is a conventionally known method. However, in this embodiment, it is possible to use the results of processing performed during daily assistance to the person being assisted, such as mealtimes and fall detection processing, making it easier to estimate sitting ability compared to conventional methods.

[0337] In this way, the processing unit 310 of this embodiment may estimate the sitting ability, which indicates the ability of the person being assisted to maintain a sitting position, based on sensor information corresponding to the bed 510 or sensor information corresponding to the wheelchair 520. Note that the sensor information here corresponds to the output of the acceleration sensor 120 of the wearable module 100, but as described above, the output of other sensors may also be used to estimate the sitting ability. Then, the processing unit 310 may perform a determination process regarding assistance in other places, including at least the toilet 600, based on the estimated sitting ability.

[0338] For example, the processing unit 310 uses the estimated results of the sitting ability to determine whether to provide assistance when using the toilet, to change parameters in the process of determining whether to fall when using the toilet (for example, the threshold for forward falls), to change parameters in the process of determining whether to fall when walking, etc. For example, if the sitting ability is high, it is more likely to be determined that no assistance is necessary, or that the risk of falling is low even if balance is lost to some extent. Furthermore, depending on the tendency for forward or lateral slippage, changes may be made such that the risk of falling in one direction is more likely to be evaluated as high, but the risk of falling in another direction is more likely to be evaluated as low.

[0339] In this way, the results of ability estimation based on sensor information at one location may affect processing at another location. In other words, when information that can be applied regardless of location, such as the ability of a person being assisted, is required, the information can be shared with other locations, thereby improving the accuracy of processing at each location.

[0340] 3.4.2 Walking ability As shown in the process of determining whether a person falls while walking, the method of this embodiment can detect the risk of a person falling while walking. For example, as described above, the processing unit 310 may determine the risk of a person falling based on whether the rhythm of periodic lateral swaying has been disrupted.

[0341] The processing unit 310 evaluates walking ability based on the length of time from when walking begins until the risk of falling increases. The processing unit 310 may also evaluate the manner of fall, such as forward fall or backward fall, and the severity thereof. Walking ability may also be evaluated based on the evaluation of the sitting ability described above.

[0342] However, determining all possible fall cases that may occur while walking in real time may place a heavy load on the server. Therefore, in this embodiment, gait assessment may be performed using the Waltz in. For example, based on the output of the Waltz in, the processing unit 310 may determine the center of gravity position (front or rear), the time the feet are planted, the order in which pressure is released, the speed at which pressure is applied over time, and so on. The processing unit 310 may then narrow down patterns of rhythm disruption based on this information and execute the fall detection process using those patterns as detection targets.

[0343] For example, an assisted person whose center of gravity tends to shift backward is more likely to fall backward. In the case of a backward fall, the signal value on the y-axis often gradually increases. For example, in the case of a backward fall, the lower and upper peak values ​​of the periodic signal increase over time. Therefore, if an assessment shows that the assisted person is prone to backward falls, the processing unit 310 can reduce the processing load in the fall determination process by focusing its determination on whether or not the acceleration value is gradually increasing. Since the walking ability estimation process can reuse the results of the fall determination process as described above, it is also possible to reduce the processing load of the walking ability estimation process.

[0344] As another example, for a person receiving care whose feet spend a long time on the ground, a breakdown in rhythm can be detected as a change in the time it takes for the feet to be on the ground. Therefore, the processing unit 310 may perform a fall detection process based on the change in the time it takes for the feet to be on the ground. Alternatively, for a person who applies pressure slowly, a breakdown in rhythm appears as a change in the slope or period of the pressure value. Therefore, the processing unit 310 may determine the slope or period of the acceleration value and perform a fall detection process based on these changes. Even with these methods, processing can be limited to patterns that are likely to occur in people receiving care, thereby reducing the processing load.

[0345] In this way, the processing unit 310 of this embodiment may estimate the walking ability, which indicates the ability of the person being assisted to walk stably, based on sensor information corresponding to walking. Then, the processing unit 310 may perform a determination process regarding assistance in other places, including at least the toilet 600, based on the estimated walking ability.

[0346] For example, the processing unit 310 uses the walking ability estimation result to determine whether to provide assistance when using the toilet, change parameters in the process of determining whether to fall when using the toilet (for example, the threshold for forward falls), etc. In this way, the result of ability estimation based on sensor information in one place may affect processing in another place, and this is also true for the ability to maintain a sitting position.

[0347] Furthermore, as described above, different patterns appear in the sensor information of the acceleration sensor 120 depending on the manner in which a fall occurs while walking. For example, if the walking ability of a person receiving care is estimated based on one pattern, parameters (thresholds, etc.) used when performing fall determination processing on that person based on other patterns may be changed based on the estimated walking ability. For example, if the server system 300 has ample processing capacity, or if the number of ways in which people receiving care fall has increased, a fall determination processing that combines multiple patterns may be performed. In this case, the processing accuracy can be improved by reflecting the walking ability that has already been estimated in the other patterns.

[0348] In this embodiment, if there are multiple ways in which the rhythm is disrupted or if the disruption has increased, the caregiver may be recommended to use the foot pressure sensor at all times. This makes it possible to obtain detailed information about the gait of the person being assisted and to appropriately identify patterns that should be detected.

[0349] As described above, the wearable module 100 may include a temperature sensor to detect body surface temperature. For example, when the processing unit 310 determines that the person being assisted has fallen, it activates the temperature sensor in the wearable module 100 corresponding to the person being assisted and acquires the temperature change. This makes it possible to appropriately monitor the vital signs of the person being assisted if there is a possibility of an injury, such as a fracture. Note that by deactivating the temperature sensor in cases other than a fall, it is possible to reduce the power consumption of the wearable module 100.

[0350] Furthermore, in the fall determination process, the processing unit 310 may estimate whether or not there is a possibility that the person has hit their head by simulating the manner of falling. If the processing unit 310 determines that there is a possibility that the person has hit their head, it may present information regarding the need for a detailed examination using the mobile terminal device 410 or headset 420 of the caregiver.

[0351] 3.4.3 Swallowing ability As described above with reference to FIG. 25, in this embodiment, the swallowing time, which is the time from opening the mouth to swallowing, is measured using the throat microphone™ and the camera of the communication device 200-5. The processing unit 310 may estimate the swallowing ability of the person being assisted based on long-term changes in the swallowing time. For example, the processing unit 310 continuously measures the swallowing time for breakfast, lunch, dinner, snacks, etc. throughout the day and calculates the swallowing time for that day based on the average value, etc. Then, when the daily swallowing time has been accumulated for 30 days, the processing unit 310 determines whether the value has changed. For example, the processing unit 310 may determine the swallowing time on a monthly basis and determine that the swallowing ability is declining if the swallowing time is getting longer over time.

[0352] Furthermore, the processing unit 310 may classify swallowing ability into a plurality of classes based on the swallowing sound, for example, the amplitude and period of the output signal of a throat microphone™, in addition to the swallowing time.

[0353] 3.5 Applications of Skeleton Tracking In the above, examples have been described in which skeletal tracking is used in bed position, wheelchair position, diaper change, etc. However, skeletal tracking may also be used in other situations.

[0354] For example, a camera may be placed in a place where many people gather and engage in activities, such as a living room or hall of a nursing home, and skeletal tracking may be performed based on images captured by the camera. As described above with reference to FIG. 2, communication device 200-6 may be placed on a television or the like in a living room, and images may be captured using the camera of communication device 200-6. In the example of FIG. 2, communication device 200-6 outputs a captured image including three people being assisted. For example, the above-mentioned OpenPose discloses a method of performing skeletal tracking on each of multiple people captured in an image and displaying the results.

[0355] For example, the processing unit 310 may perform skeletal tracking of each person using a similar method on the captured image from the communication device 200-6, and may also perform processing to identify the target person being assisted by face recognition processing.The processing unit 310 then performs a fall determination process for each person being assisted based on the results of the skeletal tracking.For example, as described above, the processing unit 310 may classify people being assisted according to their walking ability, sitting ability, etc., and perform a fall determination process according to the class.

[0356] For example, a care recipient with poor walking ability may fall even when attempting to stand up. Therefore, the processing unit 310 may use skeletal tracking to determine whether the person is assuming a standing-up posture. For example, if the processing unit 310 determines that the person has leaned forward from a sitting position, placing their hands on their knees or the seat of a chair, the processing unit 310 determines that the person is assuming a standing-up posture and notifies the caregiver of the risk of falling. Alternatively, the processing unit 310 may divide the processing target data into windows of several seconds and determine that a posture change, such as standing up, has occurred if a specific position, such as the head or neck, moves within each window by more than a predetermined threshold. Note that the body part targeted for movement detection may be other than the head or neck. The movement direction may be vertical, horizontal, or diagonal. The threshold used for detection may be changed depending on the body part targeted for detection. These conditions may also be changed depending on the attributes of the person being assisted. Various modifications of the state of the person being assisted and the risk of falling to be detected are possible.

[0357] In this way, even if the device is placed in a location where multiple people being assisted are active, it is possible to appropriately perform fall detection processing according to the person being assisted.

[0358] 4. End-of-life care The tacit knowledge provided in this embodiment may also include information suggesting whether end-of-life care should be initiated for each care recipient after a predetermined period of time. For example, the processing unit 310 acquires five types of information as input data: the amount or proportion of each meal type (e.g., main dish, side dish, or individual ingredient such as meat or fish), the amount and timing of water intake, disease information, and body weight (or BMI). Based on the input data, the processing unit 310 then outputs output data indicating whether end-of-life care should be initiated after a predetermined period of time or whether it is time to change the content of care after the start of end-of-life care. For example, machine learning may be performed on the input data based on training data to which expert data has been assigned. In this case, the processing unit 310 obtains output data by inputting the input data into a trained model. Other machine learning techniques, such as SVM, may also be used, or techniques other than machine learning may also be used.

[0359] End-of-life care here refers to assistance for individuals who are considered likely to die in the near future. End-of-life care differs from standard care in that it emphasizes the alleviation of physical and mental pain and the support of the individual living a dignified life. Furthermore, as the individual's condition changes over time, the appropriate care may change accordingly. In other words, by indicating the timing for initiating end-of-life care and the timing for changes in the care content during end-of-life care, it becomes possible to provide appropriate care to the individual until the end. For example, experienced caregivers possess tacit knowledge to estimate the timing and content of end-of-life care needed based on various factors such as the amount of food consumed. By digitizing this tacit knowledge, other caregivers can also provide appropriate end-of-life care.

[0360] 32A to 32D are examples of screens that display the determination results of end-of-life care. The screens shown in Fig. 32A to 32D may be displayed on the display unit of mobile terminal device 410, or may be displayed on the display unit of a PC or the like used in a care facility. An example in which mobile terminal device 410 is used will be described below.

[0361] FIG. 32A shows an example of a screen for uploading input data and issuing instructions for executing analysis processing related to end-of-life care. For example, log data serving as input data for end-of-life care for each person receiving care is stored in the storage unit of a management server of a nursing facility or a mobile terminal device 410. As described above, the log data is time-series data such as the amount or ratio of each type of food intake, the amount and timing of fluid intake, disease information, weight or BMI, etc. A user such as a caregiver presses object OB12, which is a reference button, to specify a file containing log data for the person receiving care to be analyzed. For example, the box in FIG. 32A displays the name of a selected file, which is the selected file. When a user selects object OB13, which is an analysis start button, with the selected file specified, the mobile terminal device 410 or the like uploads the selected file to the server system 300. The processing unit 310 of the server system 300 inputs the uploaded selected file as input data into a trained model to obtain output data. The processing unit 310, for example, determines the probability of starting end-of-life care after 30 days. The processing unit 310 may also output the predicted results of the transition of intake amount for each type of food.

[0362] FIG. 32B is an example of a screen displaying the analysis results. FIG. 32B is an example of a display screen when it is determined based on the output data that there is no need to start end-of-life care after 30 days. Note that FIG. 32B shows an example of using a file with the extension "xlsx" as the uploaded file, but the data format is not limited to this. The same applies to FIG. 32C.

[0363] For example, the processing unit 310 of the server system 300 determines that end-of-life care does not need to be started when the probability value, which is the output data, is equal to or less than a given threshold. In this case, as shown in Fig. 32B, for example, the display unit of the mobile terminal device 410 displays the text "There is no possibility that end-of-life care will be started in 30 days" and an object including a check mark.

[0364] FIG. 32C is an example of a screen displaying the analysis results, showing a case where it is determined that end-of-life care may be initiated in 30 days. For example, the processing unit 310 of the server system 300 determines that end-of-life care may be initiated when the output data probability value is greater than the given threshold. For example, the display unit of the mobile terminal device 410 displays the text "End-of-life care may be initiated." As shown in FIG. 32C, the text may include the date of the input data and the date on which end-of-life care may be initiated. Furthermore, as shown in FIG. 32, the display unit of the mobile terminal device 410 may display an object representing a warning. Furthermore, the display unit of the mobile terminal device 410 may display an object OB14 corresponding to a details button for displaying detailed analysis results.

[0365] Fig. 32D is an example of an analysis result screen displayed on the display unit of the mobile terminal device 410 when a selection operation for object OB14 is performed. The analysis result screen may be displayed as a pop-up screen different from the screen of Fig. 32C, for example. However, the specific display mode can be modified in various ways.

[0366] As shown in FIG. 32D, the analysis result screen may include time-series changes in feature values ​​calculated based on the input data and a determination result of whether end-of-life care should be initiated after a predetermined period of time. The feature values ​​may be important input information, such as a moving average of food intake, or information calculated based on the five pieces of input information. For example, when a neural network (NN) is used, the feature values ​​may be the output of a given intermediate layer or output layer. For example, the input data may include actual measurements of main dish intake, water content, and BMI up to February 13, 2020. The processing unit 310 may estimate trends in main dish intake, water content, and BMI from February 14, 2020 onward based on the trained model. The analysis screen may also include graphs showing time-series changes in the actual and estimated values ​​for these three items. Note that FIG. 32D illustrates a graph of the moving average of these values ​​over a seven-day period. This allows caregivers to easily understand the trends in important items in end-of-life care. As mentioned above, the input data may include other items, and the information displayed on the analysis result screen is not limited to the example of FIG. 32D.

[0367] The analysis result screen may also display a period during which end-of-life care may be provided. In the example of Figure 32D, the text "End-of-life care may be provided from 2020-03-14" is displayed, and the corresponding period in the graph is displayed with a different background color from the other periods for easy identification. This clearly indicates the timing and period during which end-of-life care is estimated to be necessary, making it possible to appropriately present information about end-of-life care to the user.

[0368] In the method of this embodiment, the processing unit 310 may estimate information 30 days later as described above. For example, the processing unit 310 determines whether end-of-life care should be started 30 days later based on input data. At this time, the input data may be set in multiple ways. For example, the processing unit 310 may be able to switch between a process in which the amount of food intake, etc. for the past 15 days is used as input data and end-of-life care 30 days later is determined based on the input data, and a process in which the amount of food intake, etc. for the past 30 days is used as input data and end-of-life care 30 days later is determined based on the input data.

[0369] Because end-of-life care is care provided immediately before a care recipient dies, it may not be easy to collect a large amount of data for use in making a judgment. In this regard, by enabling judgments based on a relatively small amount of data, such as 15 days' worth, as described above, it becomes possible to make judgments regarding end-of-life care even when data collection is not yet advanced. Furthermore, when data collection progresses, the accuracy of judgments can be improved by inputting data from a relatively longer period, such as 30 days' worth. While two types of input data, 15 days' worth and 30 days' worth, are illustrated here, the input data may cover three or more periods. Furthermore, the timing for determining whether end-of-life care needs to be initiated is not limited to 30 days. For example, the input data period and the timing for determining whether end-of-life care needs to be initiated may be set by the user. For example, because different facilities may have different approaches to end-of-life care, these values ​​may be changed depending on the facility.

[0370] In this embodiment, control may be performed to switch processing modes based on the device output, based on the end-of-life care assessment result. FIG. 33 illustrates an example of a device involved in the control of switching processing modes. As shown in FIG. 33, the device may be a sheet-like detector 810 placed between the bottom of a bed 510 and a mattress 820. The detector 810 detects body vibrations as biosignals of a care recipient residing on the mattress 820. The detector 810 then calculates bioinformation of the care recipient based on the detected vibrations. For example, the bioinformation may include a respiratory rate, a heart rate, and an activity level. Note that the process of determining bioinformation based on vibrations is not limited to being executed by the detector 810, but may also be executed by a processing unit 310 of a server system 300, etc. Note that such a detector 810 is described in Japanese Patent Application No. 2017-231224, entitled "Abnormality Determination Apparatus and Program," filed on November 30, 2017. This patent application is incorporated herein by reference in its entirety.

[0371] Japanese Patent Application No. 2017-231224 discloses a method for determining whether death is imminent based on biological information, such as whether there are characteristics such as almost no body movement for a long period of time and no getting out of bed after the respiratory rate and heart rate no longer show abnormal values.

[0372] When combined with end-of-life care as in the present embodiment, if it is determined that end-of-life care is not required, processing in a normal mode may be performed based on the biological information output by the detection device 810. If it is determined that end-of-life care is required, processing in an abnormality determination mode may be performed based on the biological information output by the detection device 810. The normal mode is a processing mode that does not involve determining the time of death and may be a mode that determines the sleep state, etc., based on the respiratory rate and heart rate, for example. The abnormality determination mode is a mode that determines whether death is approaching as described above. Note that processing based on the biological information output by the detection device 810 may be performed by the server system 300, the detection device 810, or another device such as the communication device 200. In other words, the processing mode here may represent the operating mode of the server system 300, the detection device 810, or another device.

[0373] In this way, it becomes possible to link the determination result of end-of-life care based on tacit knowledge with the processing mode based on the biological information detected by the detection device 810. Specifically, since it is possible to roughly estimate the time of death in end-of-life care, it becomes possible to execute processing in the abnormality determination mode when it is highly necessary. In other words, if it is determined that death is not imminent, processing is executed in the normal mode, which makes it possible to reduce the processing load, etc.

[0374] 5. Recommendations In addition, in this embodiment, a process may be performed to recommend tools and equipment necessary for the person being assisted based on the results of each of the determination processes described above.

[0375] For example, the processing unit 310 may recommend the type and size of cushions to be used in the bed 510, wheelchair 520, etc., based on information about the bed position or wheelchair position, information representing the attributes of the person being assisted, etc. In this case, the processing unit 310 may make the recommendation using information collected in a facility other than the facility where the person being assisted resides. The processing unit 310 may also recommend the type of diaper or pad based on information collected when using tacit knowledge about diaper changing. The processing unit 310 may also recommend a change in the type of utensils used for eating, such as a spoon or self-help device, based on information collected when using tacit knowledge about eating.

[0376] The processing unit 310 may also recommend a tilt-type wheelchair or a reclining wheelchair based on the estimated sitting ability and walking ability. More specifically, the processing unit 310 may estimate the timing for replacing a wheelchair or the required rental period by performing machine learning using time-series data on sitting ability as input data. This makes it possible to create an efficient usage plan for expensive equipment. The processing unit 310 may also perform machine learning using time-series data on sitting ability and walking ability as input data to predict the degree of deterioration in the need for care and recommend care products based on the prediction results. For example, in a case where deterioration progresses in the order of walking independently → walking with a cane → walker → wheelchair, the processing unit 310 may recommend the timing for purchasing a cane or walker and the type of cane or walker that is recommended for use.

[0377] Furthermore, when the processing unit 310 receives input of several items, such as the living environment / equipment environment / area of ​​the home or facility, and the facility or family's opinions, it may comprehensively recommend equipment that is considered necessary for the person receiving care, such as walking aids, wheelchairs, or beds. The facility or family's opinions are information that represents the opinions of the family or facility personnel regarding the desired lifestyle for the person receiving care, such as ensuring safety while making use of remaining abilities. In this way, it is possible to propose all the equipment necessary for the home or facility at once, thereby improving convenience for the caregiver.

[0378] FIG. 34A illustrates an example of a system used for recommendations. For example, the caregiver wears a glasses-type device 430, such as AR glasses or MR glasses, as the caregiver terminal 400. The glasses-type device 430 has, for example, a camera that captures an image of the area corresponding to the user's field of view. The glasses-type device 430 has a display in part or all of its lens, allowing the user to visually recognize the external situation by transmitting light from the outside world or by displaying an image corresponding to the user's field of view captured by the camera. Furthermore, the glasses-type device 430 uses a display to display additional information in the user's field of view. For example, as shown in FIG. 34A, when the caregiver views a person being assisted while wearing the glasses-type device 430, recommendations suitable for the person being assisted are displayed on the display of the glasses-type device 430. For example, a processing unit of the glasses-type device 430 or a processing unit 310 of the server system 300 may perform face recognition processing of the person being assisted, and when the person being assisted is detected, the display of the recommendation screen may be controlled.

[0379] FIG. 34B is an example of a recommendation display screen. As shown in FIG. 34B, the image captured by the glasses-type device 430 includes, for example, the target person being assisted and a wheelchair 520. In this case, recommended tools and the like for the target person being assisted to move around in the wheelchair 520 may be recommended. For example, in addition to facial recognition processing of the person being assisted, the processing unit 310 recognizes tools and the like for assistance located around the person being assisted, and displays recommendation information based on the results. Note that information indicating which communication device 200 the wearable module 100 is connected to may be used to identify the tools located around the person being assisted.

[0380] In the example of FIG. 34B, the display of the glasses-type device 430 displays an object OB15 representing recommendation information about a new wheelchair and an object OB16 representing recommendation information about a cushion on a captured image of the person being assisted. As shown in FIG. 34B, the object OB15 displays the text "Why not try a different wheelchair?". A speech bubble frame is also used to clearly indicate that the object OB15 corresponds to the wheelchair 520 in the captured image. In this way, it is possible to clearly communicate to the caregiver or the like that it is recommended to replace the currently used wheelchair 520 with a new wheelchair.

[0381] The object OB15 includes, for example, an image of the proposed appliance, text describing its features, a price, and a rating given by the user. The object OB15 may also include a bookmark button, a video button, and a reason display button. The bookmark button allows the caregiver to easily access information about the displayed appliance. For example, when an operation to select the bookmark button is performed on the screen shown in FIG. 34B, information about the displayed wheelchair is stored as a bookmark in association with the caregiver. For example, when the caregiver selects the bookmark using the caregiver terminal 400, the same information as the object OB15 or corresponding information is presented on the caregiver terminal 400. For example, the image, price, etc. included in the object OB15 may be information quoted from a manufacturer's website or a shopping site, and the bookmark may be information representing the URL of such information.

[0382] The video button is a button for displaying a video about the target appliance. The video may be a promotional video created by the appliance manufacturer or a review video posted by the user. Pressing the video button may also launch other application software, such as a video posting / viewing application. For example, pressing the video button may display a search result screen showing a video search by product name.

[0383] The reason display button is a screen that displays the reason why the target device has been recommended. As described above, in this embodiment, tacit knowledge is used to make judgments in various situations, such as the fall detection process and the assessment of the ability to maintain a sitting position, and the device to be recommended is determined as a result. By presenting the reason for the decision based on the reason display button, it is possible to provide information to help the caregiver, the person being assisted, or the caregiver's family member, etc., decide whether or not to introduce the target device.

[0384] Object OB16 is an example of recommendation information recommending a cushion. The displayed information is the same as that of object OB15, and therefore a detailed description thereof will be omitted. Note that object OB17, which indicates the location where the target cushion should be placed, may be displayed in conjunction with object OB16. In the example of FIG. 34B, object OB17 is displayed to the right of the person being assisted. This makes it possible to make recommendations that include not only the product name and type, but also the placement and usage of the cushion. For example, if it is determined that the person being assisted has hemiplegia based on the recognition results of the captured image or information such as the attributes of the person being assisted, the cushion shown in FIG. 34B may be recommended. This makes it possible to present to the caregiver a cushion that is useful for preventing contracture and how to use it.

[0385] Although the above describes an example in which recommendation information is displayed using the glasses-type device 430, the present invention is not limited to this. For example, a similar display may be performed using an AR application on a smartphone or the like. FIG. 34C is an example of a screen displayed on the display unit of a smartphone. In area RE9 of FIG. 34C, an image in which numbers and an object OB20 are superimposed on an image captured by the smartphone camera is displayed. In area RE10, objects OB18 and OB19 representing recommendation information are displayed in association with the same numbers as in area RE9. Note that objects OB18 to OB20 are similar to objects OB15 to OB17 in FIG. 34B, and therefore detailed description thereof will be omitted.

[0386] In this way, it becomes possible to view the recommendation information using a widely used device such as a smartphone, etc. For example, a family member of the person being assisted can view the screen of Fig. 34C by taking an image of the person being assisted using their own smartphone, making it easy to obtain the recommendation information.

[0387] Although the present embodiment has been described in detail above, it will be readily apparent to those skilled in the art that many modifications are possible without substantially departing from the novel features and advantages of the present embodiment. Therefore, all such modifications are intended to be included within the scope of the present disclosure. For example, a term described at least once in the specification or drawings together with a different term having a broader or equivalent meaning may be replaced with that different term anywhere in the specification or drawings. Furthermore, all combinations of the present embodiment and modifications are also included within the scope of the present disclosure. Furthermore, the configurations and operations of the wearable module, communication device, server system, and the like are not limited to those described in the present embodiment, and various modifications are possible. [Explanation of symbols]

[0388] 10...information processing system, 20...information processing device, 21...acquisition unit, 23...processing unit, 100...wearable module, 110...control unit, 120...acceleration sensor, 130...communication module, 140...storage unit, 200, 200-1 to 200-6...communication devices, 210...processing unit, 220...storage unit, 230...communication unit, 240...display unit, 250...operation unit, 300...server system, 310...processing unit, 320...storage unit, 330...communication unit, 400...caregiver terminal , 410, 410-1, 410-2...portable terminal device, 420, 420-1, 420-2...headset, 430...glasses-type device, 510...bed, 520...wheelchair, 521...cushion, 522a...first layer, 522b...second layer, 522c...third layer, 523...control box, 524...operation unit, 524a...power switch, 524b...end button, 524c...determination button, 525...alarm unit, 525a...measurement in progress lamp, 525b...recording in progress lamp , 525c...forward shift lamp, 525d...lateral shift lamp, 530...table, 531...operating lever, 532...fixing member, 532a to 532c...surface, 533...board box, 534...solenoid, 540...walker, 541a...horizontal leg pipe, 541b...vertical leg pipe, 541c...base frame member, 542...casing, 543, 544...hook portion, 545...motor, 546...wire, 547...brake, 600...toilet, 700...peripheral equipment, 710...control unit, 720...storage unit, 730...communication unit, 740...drive mechanism, 810...detection device, 820...mattress, Ca11 to Ca14, Ca21-Ca24...casters, CP2...second terminal device, CP3...third terminal device, Det1, Det2...waist detection results, DP...display, N...notch, NW...network, OB1 to OB20...object, RE1 to RE10...area, ReD...diaper area, Se1 to Se4...pressure sensor, TM...throat microphone

Claims

1. an acquisition unit that acquires information in which the sensor information output by the wearable module is associated with location information that identifies a location where the communication device that received the sensor information is located; a processing unit that performs a fall determination process to determine a risk of a person being assisted wearing the wearable module falling based on the placement information and the sensor information; Including, the fall determination process includes at least a first determination condition and a second determination condition, The processing unit performing the fall determination process based on the first determination condition according to the location information and the sensor information acquired from the communication device disposed at a first location; performing the fall determination process based on the second determination condition according to the location information and the sensor information acquired from the communication device that is located at a second location different from the first location; Based on the fall determination process, at least one of a notification on an assistant terminal of an assistant assisting the person being assisted and control of peripheral devices located around the person being assisted is performed; The processing unit performing a process of identifying peripheral devices located around the person being assisted based on at least one of the location information and information identifying the person being assisted associated with the wearable module; Controlling the peripheral device identified based on the fall determination process; the peripheral device is a device that can be moved using casters, The processing unit When the risk of falling is detected, the information processing device controls the peripheral device to move closer to the person being assisted by driving the casters of the peripheral device.

2. In claim 1, The location where the communication device is placed includes a bed, a wheelchair, and a toilet; The processing unit An information processing device that performs the fall detection process, which includes the fall detection in bed, the fall detection in the wheelchair, the fall detection in the toilet, and the fall detection while walking.

3. In claim 1, the peripheral device is a device having a height adjustment function, The processing unit When the risk of tipping over is detected, the information processing device controls to lower the height of the peripheral device.

4. In claim 3, The peripheral device having the height adjustment function is an information processing device including an adjustable bed.

5. In any one of claims 1 to 4, The processing unit acquiring a trained model for determining the likelihood of the fall risk, the trained model being generated by machine learning based on training data including training sensor information output by the wearable module and training placement information that identifies a location where the training sensor information was acquired; An information processing device that performs the fall determination process based on the sensor information, the placement information, and the trained model.

6. A computer comprising: acquiring information in which the sensor information output by the wearable module is associated with location information identifying a location where the communication device that received the sensor information is located; performing a fall determination process for determining a risk of a person being assisted wearing the wearable module falling based on the placement information and the sensor information; a step of performing at least one of a notification on an assistant terminal of an assistant assisting the person being assisted and control of peripheral devices located around the person being assisted based on the fall determination process; Perform a process including the fall determination process includes at least a first determination condition and a second determination condition, The computer In the step of performing the fall determination process, the fall determination process is performed using the first determination condition according to the placement information and the sensor information acquired from the communication device that is placed at a first location, and the fall determination process is performed using the second determination condition according to the placement information and the sensor information acquired from the communication device that is placed at a second location that is different from the first location. performing a process of identifying peripheral devices located around the person being assisted based on at least one of the location information and information identifying the person being assisted associated with the wearable module; Controlling the peripheral device identified based on the fall determination process; the peripheral device is a device that can be moved using casters, The computer When the risk of falling is detected, the peripheral device is controlled to move closer to the person being assisted by driving the casters of the peripheral device. Information processing methods.

Images