Information processing system and information processing method

Abstract

To provide an information processing system capable of properly supporting the assistance of a carereceiver by a caregiver, and an information processing method and the like.SOLUTION: An information processing system includes a first processing unit which operates according to a positioning application for performing processing on a position of at least one of a person and an object in assistance, and a second processing unit which operates according to a meal application for determining an amount of food ingested by a carereceiver who receives meal assistance. When it determines that at least the amount of food ingested by the carereceiver or the nourishment of the carereceiver is insufficient, the positioning application changes an operation mode of the positioning application from a first mode to a second mode.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and the like. [Background technology]

[0002] A system for use when a caregiver provides care to a care recipient has been known. Patent Document 1 discloses a method for generating information to be provided about the condition of a resident in a living space based on time-varying changes in detected information acquired by a sensor placed in the living space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-18760 Summary of the Invention [Problem to be solved by the invention]

[0004] An information processing system and an information processing method are provided that appropriately support a caregiver in providing assistance to a person being assisted. [Means for solving the problem]

[0005] One aspect of the present disclosure relates to an information processing system that includes a first processing unit that operates according to a first application that can detect agitated behavior of a person being assisted, and a second processing unit that operates according to a second application that performs processing related to the position of at least one of a person and an object during assistance, and when the first processing unit detects agitated behavior of the person being assisted using the first application, the second processing unit performs processing using the second application. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 illustrates an example of the configuration of an information processing system. [Figure 2] FIG. 1 illustrates an example of the configuration of a server system. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of a terminal device. [Figure 4A] This is an example of training data for a positioning application. [Figure 4B] This is an example of training data that is superimposed and displayed in a positioning application. [Figure 5] FIG. 10 is a diagram illustrating a device placed around a bed. [Figure 6] FIG. 10 is a diagram illustrating an example of the relationship between an application and a device in adjusting a bed position. [Figure 7] 10 is an example of a screen used to acquire teacher data in the setting mode. [Figure 8] 10 is an example of a screen used to obtain additional information in a setting mode. [Figure 9] FIG. 10 is a diagram illustrating a device placed on a bed. [Figure 10] FIG. 10 is a diagram illustrating devices arranged around a wheelchair. [Figure 11] FIG. 10 is a diagram illustrating an example of the relationship between an application and a device in adjusting a wheelchair position. [Figure 12] 10 is an example of a screen used to acquire teacher data in the setting mode. [Figure 13] FIG. 10 is a diagram illustrating a pressure sensor disposed on a wheelchair. [Figure 14] This is an example of collaboration between positioning applications in terms of pressure ulcers. [Figure 15] 1 is an example of a choking detection device. [Figure 16A] 10 is an example of a display screen of a meal application. [Figure 16B] 10 is an example of a display screen of a meal application. [Figure 17A] FIG. 10 is a diagram illustrating the processing of a meal application. [Figure 17B] FIG. 10 is a diagram illustrating the processing of a meal application. [Figure 18]This is an example of collaboration between positioning applications in terms of falls. [Figure 19A] FIG. 10 is a diagram illustrating a table of peripheral devices. [Figure 19B] FIG. 2 is a diagram illustrating a driving mechanism of the table. [Figure 19C] FIG. 10 is a diagram illustrating a walking device as a peripheral device. [Figure 19D] FIG. 2 is a diagram illustrating a drive mechanism of a walker. [Figure 20A] 1 is a configuration example of a communication tag. [Figure 20B] 1 is a configuration example of a communication tag. [Figure 20C] 1 is a configuration example of a communication tag. [Figure 21A] This is an example of attaching a communication tag to clothing. [Figure 21B] This is an example of attaching a communication tag to clothing. [Figure 21C] This is an example of attaching a communication tag to clothing. [Figure 22] 10 is a display example of information acquired by a communication tag or the like. [Figure 23] This is an example of collaboration between positioning applications in terms of suspicious behavior. [Figure 24] FIG. 2 is a sequence diagram illustrating a processing flow of the information processing system according to the present embodiment. [Figure 25] FIG. 2 is a sequence diagram illustrating a processing flow of the information processing system according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, the present embodiment will be described with reference to the drawings. In the drawings, identical or equivalent elements are designated by the same reference numerals, and duplicate explanations will be omitted. Note that the present embodiment described below does not unduly limit the content described in the claims. Furthermore, not all of the configurations described in the present embodiment are necessarily essential components of the present disclosure.

[0008] 1. System configuration example 1 shows an example of the configuration of an information processing system 10 according to this embodiment. The information processing system 10 according to this embodiment is for providing instructions to caregivers so that they can provide appropriate care regardless of their level of skill by digitizing the "intuition" and "tacit knowledge" of caregivers who perform tasks that are normally performed by the "intuition" and "tacit knowledge" of caregivers in, for example, medical facilities or nursing care facilities.

[0009] The caregiver here may be a caregiver at a nursing facility, or a nurse or licensed practical nurse at a medical facility such as a hospital. That is, assistance in this embodiment includes various actions to support the person being assisted, and may include nursing care or medical actions such as giving an injection. The person being assisted here is a person who receives assistance from the caregiver, and may be a resident of a nursing facility or a patient who is hospitalized or visits a hospital. The person being assisted may also be, for example, a person who may be suffering from dementia.

[0010] In addition, the assistance in this embodiment may be provided at home. For example, the person being assisted in this embodiment may be a person requiring care who receives home care, or a patient receiving home medical care. The caregiver may be a family member of the person requiring care or the patient, or a visiting helper, etc.

[0011] As shown in FIG. 1, the information processing system 10 includes a server system 100, a terminal device 200, a management terminal device 300, and a sensing device 400. The information processing system 10 of this embodiment includes at least a device on which a positioning application AP1 runs. The positioning application AP1 here is software that performs processing related to the position of at least one of a person and an object during care. The positioning application AP1 runs on, for example, the terminal device 200. However, the positioning application AP1 may also run on other devices, such as the server system 100 and the sensing device 400.

[0012] The configuration of the information processing system 10 is not limited to that shown in Fig. 1, and modifications such as omitting some components or adding other components are possible. For example, Fig. 1 illustrates, as sensing devices 400, a bedside sensor 420, a detection device 430, and a choking detection device 460, which will be described later with reference to Fig. 9. However, the sensing device 400 is not limited to these, and may be a seat sensor 440, which will be described later with reference to Fig. 13, any of the motion determination devices 410, which will be described later with reference to Fig. 14, or a communication tag 470, which will be described later with reference to Fig. 18, etc. In the following, when there is no need to distinguish between multiple sensing devices 400, they will be simply referred to as sensing device 400.

[0013] The server system 100 is connected to a terminal device 200, a management terminal device 300, and a sensing device 400 via, for example, a network. The network here is, for example, a public communication network such as the Internet. However, the network is not limited to a public communication network and may be a LAN (Local Area Network) or the like. For example, the server system 100 may perform communication in accordance with the IEEE802.11 standard. However, various modifications are possible regarding the communication method between the devices.

[0014] The server system 100 may include one server or multiple servers. For example, the server system 100 may include a database server and an application server. The database server may store at least one of the processing results of the terminal device 200 and the processing results of the sensing device 400. The application server performs various processes. For example, the information processing system 10 of this embodiment may be realized by distributed processing, and at least a portion of the processing performed by the terminal device 200 or the sensing device 400 in the following description may be performed by an application server. Note that the multiple servers here may be physical servers or virtual servers. Furthermore, if a virtual server is used, the virtual server may be provided on one physical server or may be distributed across multiple physical servers. As described above, the specific configuration of the server system 100 of this embodiment can be modified in various ways.

[0015] The terminal device 200 is a device used by, for example, a caregiver who provides care to a person being assisted. The terminal device 200 here is, for example, a mobile terminal device such as a smartphone or a tablet terminal. However, the terminal device 200 may be other devices, such as a personal computer (PC), a headset, or a wearable device such as augmented reality (AR) glasses or mixed reality (MR) glasses. One caregiver may use multiple terminal devices 200. For example, the caregiver may use both a smartphone and a headset. The terminal device 200 of this embodiment may be a device carried by the caregiver, or may be a device installed in a predetermined location in a care facility.

[0016] The management terminal device 300 is a device used to manage information about residents who require care in, for example, a nursing facility. The management terminal device 300 is, for example, a PC, but other devices may also be used. The management terminal device 300 has, for example, nursing software installed, and manages residents who require care and the schedules of caregivers (staff at the nursing facility). For example, the management terminal device 300 stores information about the attributes of residents who require care. The attributes here include age, sex, height, weight, medical history, medication history, etc.

[0017] The sensing device 400 is a device used to assist a person being assisted. For example, the sensing device 400 has various sensors and acquires sensing data based on the sensors. The sensing data here may be the sensor output itself or information obtained by arithmetic processing based on the sensor output. The sensing device 400 may be a device that outputs input data used when setting up or using the positioning application AP1. The sensing device 400 may also be a device whose operation mode changes based on the processing results of the positioning application AP1. The sensing device 400 may also be a device that activates / deactivates the positioning application AP1 or changes the functions to be used based on the processing results of the sensing device 400. Details will be described later.

[0018] 2 is a block diagram showing a detailed configuration example of the server system 100. The server system 100 includes, for example, a processing unit 110, a storage unit 120, and a communication unit .

[0019] The processing unit 110 of this embodiment is configured by the following hardware. The hardware can include at least one of a circuit for processing digital signals and a circuit for processing analog signals. For example, the hardware can be configured by one or more circuit devices or one or more circuit elements mounted on a circuit board. The one or more circuit devices are, for example, an integrated circuit (IC), a field-programmable gate array (FPGA), etc. The one or more circuit elements are, for example, a resistor, a capacitor, etc.

[0020] The processing unit 110 may also be implemented by the following processor. The server system 100 of this embodiment includes a memory that stores information and a processor that operates based on the information stored in the memory. The information may be, for example, a program and various data. The memory may be the storage unit 120 or another memory. The processor includes hardware. Various processors, such as a central processing unit (CPU), a graphics processing unit (GPU), or a digital signal processor (DSP), may be used. The memory may be a semiconductor memory such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory, or may be a register, a magnetic storage device such as a hard disk drive (HDD), or an optical storage device such as an optical disk drive. For example, the memory stores computer-readable instructions, and the processor executes the instructions to realize the functions of the processing unit 110. The instructions may be instructions from an instruction set that constitutes a program, or instructions that instruct the hardware circuitry of the processor to operate.

[0021] The storage unit 120 is a work area for the processing unit 110 and stores various information. The storage unit 120 can be realized by various types of memory, and the memory may be a semiconductor memory such as an SRAM, a DRAM, a ROM (Read Only Memory), or a flash memory, or may be a register, a magnetic storage device, or an optical storage device.

[0022] The communication unit 130 is an interface for communicating via a network, and when the server system 100 performs wireless communication, it includes, for example, an antenna, an RF (radio frequency) circuit, and a baseband circuit. However, the server system 100 may also perform wired communication, in which case the communication unit 130 may include a communication interface such as an Ethernet connector and a control circuit for the communication interface. The communication unit 130 may operate under control of the processing unit 110, and may include a processor for communication control different from the processing unit 110. The communication unit 130 may perform communication according to a method specified in, for example, IEEE802.11 or IEEE802.3. However, the specific communication method can be modified in various ways.

[0023] 3 is a block diagram showing a detailed configuration example of the terminal device 200. The terminal device 200 includes, for example, a processing unit 210, a storage unit 220, a communication unit 230, a display unit 240, an operation unit 250, and an imaging unit 260. However, the configuration of the terminal device 200 is not limited to that shown in FIG. 3, and modifications such as omitting some components or adding other components are possible. For example, the terminal device 200 may have various sensors, such as motion sensors such as an acceleration sensor or a gyro sensor, a pressure sensor, a GPS (Global Positioning System) sensor, etc., in addition to the image sensor included in the imaging unit 260.

[0024] The processing unit 210 is configured by hardware including at least one of a circuit for processing digital signals and a circuit for processing analog signals. The processing unit 210 may also be realized by a processor. Various types of processors, such as a CPU, a GPU, or a DSP, can be used as the processor. The processor executes instructions stored in the memory of the terminal device 200, thereby realizing the functions of the processing unit 210 as processing.

[0025] The storage unit 220 is a work area for the processing unit 210, and is realized by various types of memory such as SRAM, DRAM, and ROM. The storage unit 220 stores, for example, a positioning application AP1. The processing executed by the positioning application AP1 will be described in detail later.

[0026] The communication unit 230 is an interface for communication via a network, and includes, for example, an antenna, an RF circuit, and a baseband circuit. The communication unit 230 communicates with the server system 100 via, for example, the network. The communication unit 230 may perform wireless communication with the server system 100 in accordance with, for example, the IEEE 802.11 standard.

[0027] The display unit 240 is an interface that displays various information and may be a liquid crystal display, an organic EL display, or another type of display. The operation unit 250 is an interface that accepts user operations. The operation unit 250 may be buttons or the like provided on the terminal device 200. The display unit 240 and the operation unit 250 may also be a touch panel that is integrally configured.

[0028] The imaging unit 260 includes an image sensor that captures an image of a predetermined imaging range and outputs image information. The image information here may be a still image or a moving image. The image information may be color or monochrome. The imaging unit 260 may also include a depth sensor that detects the distance to the subject, or a sensor (e.g., an infrared sensor) that detects the heat of the subject.

[0029] The terminal device 200 may also include components not shown in FIG. 3, such as a light-emitting unit, a vibration unit, a sound input unit, and a sound output unit. The light-emitting unit is, for example, an LED (light emitting diode) and provides notification by emitting light. The vibration unit is, for example, a motor and provides notification by vibration. The sound input unit is, for example, a microphone. The sound output unit is, for example, a speaker and provides notification by sound.

[0030] The information processing system 10 according to this embodiment includes a first processing unit that operates according to a positioning application AP1 that performs processing related to the position of at least one of a person and an object during assistance. For example, the positioning application AP1 operates in a terminal device 200, and the first processing unit is included in a processing unit 210 of the terminal device 200. The positioning application AP1 may be used, for example, to adjust the posture of a person being assisted in a bed 610, as described later with reference to FIGS. 5 to 9, or may be used to adjust the posture of a person being assisted in a wheelchair 630, as described later with reference to FIGS. 10 to 13. The positioning application AP1 may be an application that acquires image information representing a desired posture as training data (described later with reference to FIG. 4A), performs transparency processing on the training data, and then superimposes the image data on a captured image acquired during position adjustment (described later with reference to FIG. 4B).

[0031] By using the positioning application AP1, position adjustments can be easily made in the bed 610 or wheelchair 630, which can prevent bedsores and falls in the person being assisted and also reduce the burden on the caregiver in terms of position adjustments.

[0032] The information processing system 10 also includes a second processing unit that operates in accordance with a meal application AP2 that detects the amount of food ingested by the person being assisted when the person being assisted receives meal assistance. For example, the meal application AP2 may operate on a terminal device 462 of a choking stool detection device 460, which will be described later with reference to FIG. 15. In this case, the second processing unit is included in the processing unit (processor) of the terminal device 462. However, the meal application AP2 may also be included in the terminal device 200. In this case, both the first processing unit and the second processing unit are realized by the processing unit of the terminal device 200. The meal application AP2 may also perform processing to detect the type of food ingested by the person being assisted. The meal application AP2 may also detect an insufficient amount of food by comparing the amount of food ingested by the person being assisted with a desired amount of food intake. The meal application AP2 may also detect the calorie intake and nutrient intake when the person being assisted ingests food based on the type of food (type of cuisine) ingested by the person being assisted. Furthermore, the meal application AP2 may detect a nutrient deficiency in the person being assisted by comparing the amount of nutrients the person being assisted has taken with the amount of nutrients the person needs. The detailed process will be described later.

[0033] By using the meal application AP2, it is possible to automate the management of the meal contents (type of food and intake amount) for each person being assisted, making it possible to provide appropriate assistance based on the meal contents and reduce the burden on caregivers and others in meal management.

[0034] As described above, the positioning application AP1 and the meal application AP2 are each useful for providing assistance. However, in the information processing system 10 of this embodiment, the positioning application AP1 and the meal application AP2 may operate in cooperation with each other. Specifically, when the meal application AP2 detects at least one of an insufficient amount of food ingested by the person being assisted and a nutritional deficiency, the positioning application AP1 changes the operation mode of the positioning application AP1 from a first mode to a second mode, which imposes a relatively higher processing load on the positioning application AP1. Note that the change from the first mode to the second mode may be a change from inactive to active. Alternatively, the trigger for activating the positioning application AP1 may be, for example, the detection of a person in the wheelchair 630, the detection of a person in the bed 610, or a diaper change. The change from the first mode to the second mode may be the addition of a function. Specific examples will be described later.

[0035] According to the method of the present embodiment, by switching the position adjustment process in response to changes in the eating situation, it is possible to appropriately support the caregiver in providing assistance to the person being assisted. For example, it is known that the risk of bedsores increases when the person being assisted is insufficient in terms of the amount of food consumed or nutrition. Details will be described later using the concept of SSKIN and other methods. With the method of the present embodiment, when such a person is being assisted, the positioning application AP1 can be actively operated, thereby reducing the risk of bedsores. For example, when a high-risk person is being assisted, the number of functions executed by the positioning application AP1 may be increased so that the caregiver can provide more thorough assistance compared to when a low-risk person is being assisted. Furthermore, because the posture preferred for the person being assisted changes depending on the level of the risk of bedsores, the training data used by the positioning application AP1 may be changed when an insufficient amount of food is detected.

[0036] The information processing system 10 of this embodiment may further include a third processing unit that operates in accordance with an agitation application AP3 that determines agitation in the person receiving care. Agitation refers to excessive and restless behavior. Examples of agitation include various symptoms not seen in normal conditions, such as insomnia, excitement, wandering, hallucinations, anxiety, misidentification, hyperactivity, unclean behavior, verbal abuse, and violence. For example, the agitation application AP3 may operate on the management terminal device 300, and the third processing unit may be included in the processing unit (processor) of the management terminal device 300. Alternatively, the agitation application AP3 may be included in the terminal device 200. In this case, both the first processing unit and the third processing unit are implemented by the processing unit 210 of the terminal device 200. The agitation application AP3 may also operate on another device, such as the server system 100.

[0037] When the agitation application AP3 detects agitated behavior of the person being assisted, the positioning application AP1 may execute a process for supporting the placement of objects located around the person being assisted.

[0038] According to the method of this embodiment, it is possible to have the positioning application AP1 and the agitation application AP3 operate in cooperation with each other. This makes it possible to support the execution of appropriate assistance in accordance with changes in the situation. Specifically, as will be described later, an abnormal arrangement of objects around the person being assisted may be a cause of agitated behavior, but by presenting an appropriate arrangement of objects using the positioning application AP1, it is possible to suppress the influence of this cause. Specific cooperation and processing contents will be described later using FIG. 23 etc.

[0039] Furthermore, the objects with which the positioning application AP1 cooperates are not limited to the eating application AP2 and the agitation application AP3, but may also include a motion determination device 410, which will be described later with reference to Fig. 14, and a peripheral device 500, which will be described later with reference to Fig. 18. That is, the method of this embodiment improves the quality of care by coordinating the positioning application AP1 with various applications and devices used in care. Specific cooperation will be described later.

[0040] Furthermore, some or all of the processing performed by the information processing system 10 of this embodiment may be realized by a program. The processing performed by the information processing system 10 may be processing executed by the processing unit 210 of the terminal device 200, processing executed by the processing unit 110 of the server system 100, or processing executed by a processor included in the management terminal device 300 or the sensing device 400. The processing performed by the information processing system 10 may also be processing executed by two or more devices among the server system 100, the terminal device 200, and the sensing device 400.

[0041] The program according to this embodiment can be stored in, for example, a non-transitory information storage medium (information storage device), which is a medium readable by a computer. The information storage medium can be realized by, for example, an optical disc, a memory card, a HDD, or a semiconductor memory. The semiconductor memory is, for example, a ROM. The processing unit 110 and the like perform various processes of this embodiment based on the program stored in the information storage medium. That is, the information storage medium stores a program for causing a computer to function as the processing unit 110 and the like. A computer is a device equipped with an input device, a processing unit, a storage unit, and an output unit. Specifically, the program according to this embodiment is a program for causing a computer to execute each step described below using Figures 24, 25, etc.

[0042] The technique of this embodiment can also be applied to an information processing method including the following steps: performing processing related to the position of at least one of a person and an object during assistance according to a positioning application AP1, performing processing related to the position of at least one of a person and an object during meal assistance according to a meal application AP2, and, when the meal application AP2 determines that the amount of food ingested by the person being assisted is insufficient or that there is a nutrient deficiency, transitioning the operation mode of the positioning application AP1 from a first mode to a second mode in which the processing load of the positioning application AP1 is higher than that of the first mode.

[0043] 2. Positioning Applications Next, a specific example of the positioning application AP1 will be described. The positioning application AP1 is application software that presents the desired position of a person or object on a bed 610, a wheelchair 630, or the like. Here, the person is, for example, a person being assisted or an assistant. The object is, for example, a cushion or a diaper. As will be described later, the positioning application AP1 may also present the desired position of an object in the room of the person being assisted.

[0044] For example, the positioning application AP1 may operate in a setting mode for setting the position and a usage mode for supporting actual position adjustment according to the setting. In the setting mode, the positioning application AP1 acquires training data by capturing an image of a person or object at a desired position. In the operation mode, the positioning application AP1 superimposes the training data, which has been subjected to transparency processing, on the captured image of the person or object to be adjusted.

[0045] FIG. 4A is an example of training data acquired in the setting mode. In the example of FIG. 4A, image information representing a desirable posture for a care recipient named "AAA" lying on the bed 610 is acquired as training data. FIG. 4B is an example of training data that has undergone transparency processing and is superimposed on a captured image in the usage mode. For example, the terminal device 200 superimposes the image of FIG. 4B on a captured image of a care recipient whose position is to be adjusted. The caregiver assists the care recipient so that the care recipient in the captured image approaches the training data. In this way, it is possible to appropriately support the caregiver in adjusting the position. Note that while an example of superimposing training data, which is image information, has been described here, the positioning application AP1 may also output a determination result (OK / NG) as to whether the care recipient's posture is appropriate. Details will be described later.

[0046] Below, the setting mode and usage mode will be described in detail for each of the position of a person or object in bed 610 (hereinafter referred to as bed position) and the position of a person or object in wheelchair 630 (hereinafter referred to as wheelchair position). Examples of application to position adjustment of objects not limited to bed 610 and wheelchair 630 will also be described.

[0047] 2.1 Bed position FIG. 5 is a diagram showing an example of device arrangement around a bed 610. As shown in FIG. 5, the terminal device 200 in this embodiment may be a terminal device 200-1 fixed to the footboard side of the bed 610, a terminal device 200-2 fixed to a side rail of the bed 610, or both. However, the location where the terminal device 200 is fixed is not limited thereto, and the terminal device 200 may be fixed to another position on the bed 610 or to another location (for example, a wall or furniture in a room) from which an image of the person being assisted on the bed 610 can be captured. Alternatively, the terminal device 200 may be carried by a caregiver, and the process described below may be performed by the caregiver using the terminal device 200 while positioned on the footboard side or the side rail side of the bed 610.

[0048] A display DP may be disposed on the opposite side of the bed 610 from the terminal device 200-2. The display DP may be fixed to the bed 610, or may be disposed in another position where it can be viewed naturally by a caregiver who adjusts the bed position using the positioning application AP1. For example, the display DP may be fixed to a wall surface, or may be fixed to a stand or the like that stands on the floor. The display DP may also be omitted.

[0049] 3, the terminal device 200 is a device such as a smartphone having an imaging unit 260 (camera). The terminal device 200 transmits a captured image to the server system 100. The display DP receives the image transmitted from the server system 100 directly or via another device such as the terminal device 200, and displays the received image.

[0050] The positioning application AP1 may be an application that supports adjusting the bed position. The adjustment of the bed position may be, for example, controlling the posture of the person being assisted to prevent bedsores or controlling the placement of a cushion. The adjustment of the bed position may also be controlling the posture of the person being assisted during diaper changing or controlling the placement of a diaper.

[0051] <Setting mode> The positioning application AP1 first operates in a setting mode in which data used for position adjustment is set. The data used for position adjustment is, for example, ground truth data representing a desired position of a person or object.

[0052] 6 is a diagram illustrating the relationship between devices used in the setting mode. For example, the positioning application AP1 may acquire image information output from the imaging unit 260 included in the terminal device 200 and information from at least one of the bed 610 and the mattress 620. The bed 610 and the mattress 620 here are devices capable of visualizing the pressure distribution when the person being assisted is using them. For example, known methods disclosed in Japanese Patent Application Laid-Open No. 2019-039788 and the like can be widely applied as a method for visualizing the pressure distribution.

[0053] Fig. 7 is an example of a screen displayed in the setting mode. The screen shown in Fig. 7 may be displayed on the display unit 240 of the terminal device 200, or on the display DP. The screen shown in Fig. 7 may also be displayed on the display unit of another device connected via a network.

[0054] The screen shown in Fig. 7 includes an area RE1 for displaying image information and an area RE2 for displaying pressure distribution. In the example of Fig. 7, an image of a person being assisted lying on a bed 610, captured from the foot side of the person being assisted, is displayed in area RE1. The image is captured, for example, using the imaging unit 260 of the terminal device 200-1 of Fig. 5. For example, in the setting mode, a moving image captured using the imaging unit 260 of the terminal device 200-1 may be displayed in real time in area RE1.

[0055] The pressure distribution is information output by the bed 610 and mattress 620, and is, for example, information that expresses the pressure value at each position on the surface on which the person being assisted lies using shades of color, etc. For example, in the setting mode, the pressure distribution obtained in chronological order may be displayed in real time.

[0056] The screen shown in FIG. 7 may also include an object OB1 corresponding to a checkbox for turning skeletal tracking on or off, an object OB2 corresponding to a radio button for selecting either a bed 610 or a wheelchair 630, an object OB3 corresponding to a rotation button, and an object OB4 corresponding to a photo taking button.

[0057] Based on the user's operation on the object OB1, the positioning application AP1 switches whether or not to perform skeletal tracking of the person being assisted. For example, when the check box is checked, the positioning application AP1 performs skeletal tracking based on captured images, and when the check box is unchecked, the skeletal tracking is omitted. Note that various image-based skeletal tracking methods are known, such as "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields" (https: / / arxiv.org / pdf / 1611.08050.pdf) and OpenPose disclosed by Zhe Cao et al., and these methods can be widely applied in this embodiment.

[0058] Object OB2 is an object that indicates whether to adjust the bed position or the wheelchair position. In the example of FIG. 7, "mattress" is displayed as the text corresponding to the bed position adjustment. Also, "seat sensor" is displayed as the text corresponding to the wheelchair position adjustment. "Seat sensor" is a sensor that detects pressure on the seat of the wheelchair 630, and corresponds to, for example, the seat sensor 440 in FIG. 13.

[0059] For example, the user may use the object OB2 to instruct the positioning application AP1 whether to perform processing for the bed position or the wheelchair position, and the positioning application AP1 may execute processing for connecting to the target device based on the instruction. For example, as shown in FIG. 7, when a user input is made to select the bed position, the communication unit 230 of the terminal device 200 executes processing for connecting to the bed 610 or the mattress 620. Alternatively, the user may manually establish a connection with a device used for position adjustment, such as the bed 610 or the mattress 620, and the positioning application AP1 may determine whether to perform processing for the bed position or the wheelchair position based on the connection result. In this case, the positioning application AP1 automatically changes the display mode of the object OB2 based on the result of the determination.

[0060] Object OB3 is an operation interface for rotating at least one of the image information displayed in area RE1 and the pressure distribution displayed in area RE2. In the example of FIG. 7, the left side of the screen is the head of the person being assisted in both the image information and the pressure distribution, so a user viewing the screen of FIG. 7 can easily associate the image information with the pressure distribution. However, in an environment in which the positioning application AP1 is used, the direction in which the terminal device 200 captures an image of the person being assisted is not necessarily fixed. Therefore, the direction of the person being assisted may differ between the image information and the pressure distribution. In this regard, when the positioning application AP1 receives a selection operation on the rotation button, it performs a process of rotating the image information or the pressure distribution, thereby making it possible to align the direction of the person being assisted.

[0061] However, the orientation of the image information and the orientation of the pressure distribution are not limited to being manually changed by the user, and may be automatically adjusted by the positioning application AP1. For example, a marker such as a two-dimensional barcode (e.g., a QR code; QR code is a registered trademark) may be attached in advance to a predetermined position within the range to be imaged by the imaging unit 260. For example, a sticker or the like with a marker printed on it may be affixed to the pillowside of the bed. The positioning application AP1 performs processing to detect the position and orientation of the marker from an image captured using the imaging unit 260 of the terminal device 200. In this way, the direction of the person's head in the image information can be appropriately detected, making it possible to automatically adjust the orientation of the image information and the orientation of the pressure distribution.

[0062] Furthermore, object OB3 corresponding to the rotation button may be used to rotate both the image information and the pressure distribution by the same angle. For example, areas RE1 and RE2 are arranged side by side, but by rotating the image information and pressure distribution by 90 degrees clockwise or counterclockwise, areas RE1 and RE2 may be arranged side by side. In this way, it becomes possible to use the rotation button to arrange various pieces of information in the positional relationship desired by the user.

[0063] As described above, by using the screen shown in Fig. 7, the user can refer to the captured image of the person being assisted and the pressure distribution, and can also view the results of the skeletal tracking process as needed. The results of the skeletal tracking process may be displayed superimposed on the image information in area RE1, for example, as shown in Fig. 7.

[0064] For example, the user here is an experienced caregiver, and the user determines whether the person being assisted is in an appropriate posture by referring to various information displayed on the screen of Fig. 7. The appropriate posture here may be a posture suitable for preventing bedsores, a posture suitable for changing a diaper, or some other posture. For example, the user, who is an experienced caregiver, determines whether the posture is appropriate for the patient based on his or her own tacit knowledge, taking into consideration various information such as the patient's attributes, medical history, and medication history.

[0065] For example, when the user determines that the person being assisted has assumed an appropriate posture, the user selects the photo-taking button indicated by object OB4. The terminal device 200 stores the image displayed when the photo-taking button was operated in the storage unit 220 as training data. The training data may also be transmitted to the server system 100. In this manner, positions that an expert considers preferable can be registered as training data. The training data is not limited to image information, but may also include pressure distribution or the results of skeletal tracking. Although the above describes an example in which the terminal device 200-1 is used, the present invention is not limited to this, and the terminal device 200-2 may also be used to acquire image information. In other words, the image information may be an image of the person being assisted taken from the side.

[0066] Further, some additional information may be added to training data including image information. FIG. 8 shows an example of a screen used when adding additional information to image information. For example, when a user inputs to select image information, the image information is displayed on the screen of FIG. 8 together with its file name. For example, as described above, image information obtained when the photo-taking button is selected is stored in the storage unit 220 of the terminal device 200 or the storage unit 120 of the server system 100, and the positioning application AP1 accepts a user input to select one of the image information. In the example of FIG. 8, the selected image information is displayed in the area RE3, and an object OB5 representing the file name of the image information is also displayed.

[0067] As shown in FIG. 8 , the additional information input screen may also display an object OB6, which is a button for adding a specific mark to the image information. For example, the specific mark may be a mark attached to a cushion used to promote proper posture. Here, a triangle corresponds to a cushion of a predetermined size or smaller, and a square corresponds to a cushion of a larger size. For example, the above correspondence may be presented to the user in advance using a manual for the positioning application AP1, and when the user places a cushion to promote proper posture, the user uses the object OB6 to attach a triangular or square mark to the cushion in the image. This allows information identifying the position and size of the cushion to be included in the training data.

[0068] The additional information may also be a comment added by a skilled caregiver. For example, the additional information input screen may display an object OB7 that is a text box for inputting a comment. The comment here may be some kind of text input using the object OB7 that is a text box. For example, the skilled caregiver inputs in text important points for achieving an appropriate bed position, such as the angle between a specific body part and other body parts, the positional relationship with pillows and cushions, and details of the cushions used (size, shape, manufacturer, product number, etc.).

[0069] The additional information input screen shown in FIG. 8 may also accept input of information specifying a part of the person being assisted that an experienced caregiver considers to be a key point. For example, if the position of the shoulders of the person being assisted is important for ensuring that the person is in an appropriate posture, the experienced caregiver may enter an input specifying the shoulders of the person being assisted. For example, the positioning application AP1 may accept input to place a triangular or square mark shown in FIG. 8 on a specific part. The experienced caregiver may also consider multiple parts to be key points. In this case, the positioning application AP1 may accept input of the priority of each position. The additional information may also be audio information. For example, the additional information input screen may display an object OB8 that is a checkbox for setting whether or not audio information can be input. For example, if the checkbox is checked, a process of recording the user's voice may be performed.

[0070] The screen shown in Fig. 8 may also allow for a review that anticipates use. For example, the input screen for additional information may display an object OB9, which is a button for previewing. Here, previewing refers to displaying a screen that corresponds to the display screen in the use mode. The preview display may be performed on the input screen for additional information shown in Fig. 8. Alternatively, a pop-up screen different from the input screen in Fig. 8 may be displayed based on a selection operation of object OB9, and the preview display may be performed on that pop-up screen.

[0071] For example, as described above with reference to FIGS. 4A and 4B, when using the positioning application AP1, it is conceivable to superimpose transparently processed teacher data and a real-time image of the person being assisted. Therefore, when a selection operation is performed on object OB9 in FIG. 8, the positioning application AP1 acquires a captured image using the imaging unit 260 and superimposes an image of the teacher data being set, which has been subjected to transparency processing, on the captured image. This allows the user to check in advance what the display screen will look like when the teacher data being set is actually used. In this case, the user may be able to set the transparency of the teacher data using the screen shown in FIG. 8, or, if the imaging unit 260 includes an internal camera and an external camera, select which camera to use. For example, the additional information input screen may display object OB10, which is a text box for inputting transparency, and object OB11, which is a radio button for selecting a camera, as shown in FIG. 8.

[0072] The training data used by the positioning application AP1 is not limited to this. For example, video images of an experienced caregiver taking action to place the care recipient in an appropriate posture may be used as training data. The training data may also include the results of tracking the caregiver's skeleton detected based on the video images.

[0073] <Usage mode> In the usage mode in which the bed position of the person being assisted is actually adjusted, the positioning application AP1 starts acquiring captured images using the imaging unit 260 of the terminal device 200. The positioning application AP1 also selects training data to be used for adjusting the bed position. For example, in the usage mode, the positioning application AP1 may display multiple tabs on a screen that displays transparently as shown in FIG. 4B and switch training data based on a selection operation of the tab. For example, each tab may correspond to one training data, and training data may be selected based on manual switching of the tab by the caregiver who adjusts the position. Note that the bed position adjustment may be performed by the caregiver, or the person being assisted may change their own posture while viewing the screen displayed by the positioning application AP1.

[0074] The positioning application AP1 may select training data based on the selection process of training data by the caregiver. Alternatively, the positioning application AP1 may automatically select training data based on a similarity determination between the attributes of the person being assisted and the attributes of the person being assisted captured in the training data. The attributes here include information such as the age, sex, height, weight, medical history, and medication history of the person being assisted.

[0075] Alternatively, the positioning application AP1 may automatically select training data based on a comparison between the attributes of the person being assisted whose bed position is to be adjusted and additional information contained in the training data. For example, the additional information in the training data may include text such as "For a person being assisted who shows a tendency of XX, it is recommended to adjust the left shoulder so that it is YY." In this case, if the person being adjusted corresponds to XX, the training data is likely to be selected. For example, a caregiver who adjusts the bed position may input information identifying the person being assisted into the terminal device 200, and the terminal device 200 may identify the attributes of the person being assisted based on the information.

[0076] The positioning application AP1 may output an image in which, for example, transparently processed training data is superimposed on a real-time image captured by the terminal device 200. At this time, additional information about the training data may be presented. For example, text data may be superimposed, or audio data may be output from a speaker of the terminal device 200 or the like.

[0077] The positioning application AP1 may also determine whether the posture is OK or NG based on the degree of similarity between an image captured during position adjustment and the training data, and output the determination result. For example, if the numerical value representing the degree of similarity is equal to or greater than a predetermined threshold, the positioning application AP1 outputs OK, indicating that the posture is appropriate. If the numerical value representing the degree of similarity is less than the predetermined threshold, the positioning application AP1 outputs NG, indicating that the posture is inappropriate. The determination result may be displayed on the display unit 240 of the terminal device 200, or may be displayed on the display DP directly or via the server system 100. The positioning application AP1 may also perform processing to display specific points determined to be NG. For example, the positioning application AP1 may compare the captured image with the training data and highlight areas determined to have a large difference.

[0078] Furthermore, when the positioning application AP1 is used, the results of pressure distribution and skeletal tracking may be acquired in the same manner as during setup. In this case, the positioning application AP1 may determine whether the application is OK or NG based on a comparison process between the pressure distribution acquired during use and the pressure distribution included in the training data, or a comparison process between the skeletal tracking results acquired during use and the skeletal tracking results included in the training data.

[0079] In this way, it becomes possible to present a comparison between the bed position of the person being assisted and the ideal bed position, and to present information for realizing the ideal bed position.

[0080] As described above, the training data used by the positioning application AP1 may include video images of the skilled caregiver's movements to position the person being assisted in an appropriate posture, and skeletal tracking results of the caregiver detected based on the video images. In this case, the positioning application AP1 may capture an image of the caregiver in the usage mode, perform skeletal tracking of the caregiver, and compare the results with the training data to determine whether the results are OK or NG. In this way, it is possible to have the caregiver perform movements similar to those of a highly skilled caregiver, regardless of the caregiver's level of skill.

[0081] The positioning application AP1 may also be used to assist in diaper changing. For example, the positioning application AP1 may determine whether the person being assisted is in a lateral position, whether the diaper is properly positioned, whether the pads are protruding from the diaper, and whether the diaper is properly worn. These functions may be realized by superimposing training data captured in an appropriate state, or may be performed based on skeletal tracking, object detection processing, or the like.

[0082] This not only allows the posture (sleeping posture) of the person being assisted to be appropriately adjusted when lying continuously on the bed 610, but also allows the caregiver to appropriately perform the diaper change by appropriately utilizing tacit knowledge in diaper changing. Note that the body of the person being assisted needs to be moved during the diaper change process. For example, the caregiver may first place the person in a lateral position to make it easier to place the diaper, or may raise the person's legs to put on the diaper. Therefore, when the diaper change is complete, the person being assisted may not be in a posture appropriate for lying down on the bed. Therefore, the positioning application AP1 may automatically execute a process to adjust the sleeping posture when it detects that the diaper change is complete.

[0083] As described above, the positioning application AP1 may have multiple functions, such as a function for performing superimposed display and a function for making an OK / NG judgment, and each function may be switchable between active and inactive. For example, the positioning application AP1 may include an operation mode for performing only superimposed display and an operation mode for performing both superimposed display and OK / NG judgment. The active / inactive status of each function may be set by a user, such as a caregiver, but is not limited to this, and may be determined automatically by the positioning application AP1.

[0084] The functions of the positioning application AP1 of this embodiment are not limited to this, and a function for controlling the bed 610 used by the person being assisted may also be used. For example, the storage unit 220 of the terminal device 200 stores setting information for the bed 610 used by the person being assisted. The setting information here may be information in which multiple setting items are associated with respective setting values. The multiple setting items include the height and bottom angle of the bed 610. The bottom refers to the surface on which the mattress 620 is placed. If the bed 610 has multiple bottoms, the setting items may include the angle of each bottom. When the positioning application AP1 determines that the difference between the height and bottom angle of the bed 610 in the training data and the height and bottom angle of the bed 610 when in use is equal to or greater than a predetermined value, it outputs information instructing the bed 610 to control the height and bottom angle so that they approach those of the training data.

[0085] For example, the positioning application AP1 may communicate with the bed 610 to acquire setting information for the bed 610. For example, in the setting mode, the positioning application AP1 may acquire setting information from the bed 610 when the photo capture button is selected and include the setting information in the training data. In the use mode, the positioning application AP1 may acquire real-time setting information from the bed 610 and control the bed 610 by comparing the acquired setting information with the training data. Alternatively, the positioning application AP1 may determine differences in the height or bottom angle of the bed 610 by comparing training data, which is image information, with captured images captured in real time. The positioning application AP1 controls the bed 610 to reduce the differences. For example, the positioning application AP1 may ask a question such as "May I configure the bed 610?" using voice or text, along with the process of superimposing and displaying the training data shown in FIG. 4B. The positioning application AP1 starts controlling the bed 610 when the user gives permission in response to the question by voice or button selection input.

[0086] Although the above describes an example of controlling the bed 610 when adjusting the position, the controlled device is not limited to this. For example, if a mattress 620 is used that encourages the person being assisted to change their posture by adjusting the amount of air, the positioning application AP1 may control the mattress 620 when adjusting the position. For example, if a mattress 620 is used that includes multiple air cells and the air pressure (hardness) of each air cell can be adjusted, the positioning application AP1 may perform processing to adjust the hardness of each air cell. Furthermore, when adjusting the wheelchair position (described later), the positioning application AP1 may control the seat height and backrest angle of the wheelchair 630 when adjusting the position.

[0087] In addition, the function of controlling the bed 610 etc. when adjusting the position can also be changed to active / inactive based on user settings or automatic determination.

[0088] <Examples of devices used in positioning applications> In the above, the bed 610 and mattress 620 capable of detecting pressure have been exemplified as devices used to set and use the positioning application AP1, but the present invention is not limited to this.

[0089] 9 is a diagram illustrating an example of another device that can be used by the positioning application AP1, namely, a bedside sensor 420 and a detection device 430 that are placed at the bottom of a bed 610. The bedside sensor 420 and the detection device 430 are, for example, sheet-like or plate-like devices that are provided between the bottom of the bed 610 and the mattress 620, as shown in FIG.

[0090] The bedside sensor 420 and the detection device 430 may be devices used to set up and use the positioning application AP1, or may be a motion determination device 410 whose behavior changes based on the processing results of the positioning application AP1, as described below.

[0091] The bedside sensor 420 includes a pressure sensor that outputs pressure values ​​as sensing data, and is arranged on the side of the bottom that the caregiver uses to get in and out of the bed 610. In the example of FIG. 9, the caregiver gets in and out of the bed 610 using the front side of the bed 610. In this case, as shown in FIG. 9, a railing to prevent falls may be arranged on the front side of the bed 610, and the bedside sensor 420 may be arranged in a position where the railing is not provided. In this way, the user getting in and out of the bed 610 first sits on the bedside sensor 420. The bedside sensor 420 may output time-series pressure data as sensing data to the server system 100. Alternatively, the bedside sensor 420 may determine whether or not the user has started to move by executing the process described below, and output the determination result as sensing data to the server system 100.

[0092] The bedside sensor 420 operates, for example, according to an application installed on the bedside sensor 420, to acquire pressure values ​​as input data and execute a process of determining the movement of the person being assisted on the bed 610 from the pressure values.

[0093] For example, when a person being assisted stands up from bed 610, it is assumed that the person being assisted transitions from a lying position on the bed to a sitting position at the side of the bed (hereinafter referred to as "edge-sitting position"), and then performs a standing-up motion by placing their hands on their knees or the bedside to apply force. The pressure values ​​detected by the bedside sensor 420 increase in the order of lying position, edge-sitting position, and standing-up motion. For example, the bedside sensor 420 may determine that the start of movement has been detected when it detects a change from an edge-sitting position to a standing-up motion based on a comparison process between the pressure value and a given threshold value. Alternatively, from the perspective of detecting a standing-up motion at an earlier stage, the bedside sensor 420 may determine that the start of movement has been detected when it detects a change from a lying position to an edge-sitting position based on a comparison process between the pressure value and a given threshold value.

[0094] Alternatively, as the standing-up motion continues, the buttocks of the person being assisted rise from the bottom surface, causing a significant decrease in the pressure value output from the pressure sensor. Therefore, the bedside sensor 420 may determine that the standing-up motion has been performed when the pressure value increases above a first threshold and then decreases below a second threshold that is smaller than the first threshold, based on the time-series change in the pressure value. In addition, various modifications can be made to the specific processing content of the movement start determination.

[0095] 9 is a device that senses information related to the sleep of the person being assisted. The detection device 430 includes a pressure sensor that outputs a pressure value.

[0096] When the user gets into bed, the detection device 430 detects the user's body vibrations (body movements, vibrations) through the mattress 620. Based on the body vibrations detected by the detection device 430, information on the breathing rate, heart rate, activity level, posture, wakefulness / asleep, and whether the user is out of bed or in bed can be obtained. The detection device 430 may also determine whether the user is in non-REM sleep or REM sleep, and the depth of sleep. For example, the periodicity of body movements may be analyzed, and the breathing rate and heart rate may be calculated from the peak frequency. The periodicity may be analyzed using, for example, a Fourier transform. The breathing rate is the number of breaths per unit time. The heart rate is the number of heartbeats per unit time. The unit time is, for example, one minute. Body vibrations may also be detected per sampling unit time, and the number of detected body vibrations may be calculated as the amount of activity. When the user gets out of bed, the detected pressure value decreases compared to when the user is in bed, so it is possible to determine whether the user is out of bed or in bed based on the pressure value and its time-series changes.

[0097] For example, the detection device 430 may output the output of a pressure sensor as sensing data to the server system 100. Alternatively, the detection device 430 may output the above-mentioned information on the respiratory rate, heart rate, activity level, posture, awake / asleep, and getting out / being in bed as sensing data to the server system 100.

[0098] The detection device 430 may also determine whether the person being assisted has started to move. For example, the detection device 430 may determine that the person being assisted has started to move when the person being assisted transitions from a bed-in state to an out-of-bed state. From the perspective of detecting signs of movement at an earlier stage, the detection device 430 may also determine that the person being assisted has started to move when the person being assisted transitions from a sleeping state to an awake state. The detection device 430 may output the detection result of the start of movement to the server system 100 as sensing data.

[0099] 2.2 Wheelchair position Fig. 10 is a diagram illustrating a system configuration when adjusting the wheelchair position. As shown in Fig. 10, in adjusting the wheelchair position, the terminal device 200 is fixed at a height that allows capturing an image of at least the upper body of the person being assisted who is sitting in the wheelchair 630. The terminal device 200 is placed in a predetermined position in, for example, a nursing facility, and the caregiver transfers the person being assisted into the wheelchair 630, moves the person being assisted to the front of the terminal device 200, and then adjusts the wheelchair position. However, the process described below may also be performed by the caregiver standing in front of the wheelchair 630 and holding the terminal device 200 in his or her hand.

[0100] As described above with reference to FIG. 3, the terminal device 200 is, for example, a device such as a smartphone having a camera as the imaging unit 260. The positioning application AP1 may be an application that supports adjustment of the wheelchair position. The adjustment of the wheelchair position may be control of the posture of the person being assisted to prevent bedsores or control of cushion placement. The adjustment of the wheelchair position may also be control of the posture of the person being assisted when eating a meal while sitting in a wheelchair.

[0101] <Setting mode> As in the case of bed position, the positioning application AP1 first operates in a setting mode in which data used for position adjustment is set.

[0102] 11 is a diagram illustrating an example of the relationship between devices used in the setting mode. For example, the positioning application AP1 may acquire image information output from the imaging unit 260 included in the terminal device 200, and information from at least one of the wheelchair 630 and the seat sensor 440. The wheelchair 630 here is a device that can visualize the pressure distribution when the wheelchair 630 is being used by a person being assisted. The seat sensor 440 will be described later with reference to FIG. 13.

[0103] Fig. 12 is an example of a screen displayed in the setting mode. The screen shown in Fig. 12 may be displayed on the display unit 240 of the terminal device 200, or on the display unit of another device connected via a network.

[0104] The configuration of the screen shown in Fig. 12 is the same as that of Fig. 7, except that the image information displayed in area RE1 is an image of the person being assisted sitting in the wheelchair 630, and the pressure distribution displayed in area RE2 is information representing the pressure on the seat of the wheelchair 630. Therefore, further details of the information displayed on the screen of Fig. 12 will not be described.

[0105] By using the screen shown in Fig. 12, the user can refer to the captured image of the person being assisted and the pressure distribution, and can also view the results of the skeletal tracking process as needed. Therefore, for example, a user who is an experienced caregiver can determine whether the person being assisted is in an appropriate posture by referring to the various information displayed on the screen of Fig. 12. The appropriate posture here may be a posture suitable for preventing bedsores, a posture suitable for eating, or another posture.

[0106] Similar to the example of bed position, the position considered preferable by the expert can be registered as training data based on the selection of the photo taking button shown on object OB4. The training data is not limited to image information, but may also include pressure distribution and the results of skeletal tracking.

[0107] Furthermore, as in the example of bed position, some additional information may be added to the training data including image information. To add the additional information, for example, the screen described above with reference to FIG. 8 is used.

[0108] <Usage mode> In the usage mode in which the bed position of the person being assisted is actually adjusted, the positioning application AP1 starts acquiring captured images using the imaging unit 260 of the terminal device 200. The positioning application AP1 also selects training data to be used for adjusting the wheelchair position.

[0109] The process of selecting teacher data by the positioning application AP1 is similar to the example of bed position, and the teacher data may be selected based on user input or may be automatically selected based on the attributes of the person being assisted, etc.

[0110] The positioning application AP1 superimposes the training data, which has been subjected to transparency processing, on the captured image as described above with reference to FIG. 4B. The positioning application AP1 may also determine whether the image captured during position adjustment is OK or NG based on the degree of similarity between the image and the training data, and output the determination result. Similarly, when the positioning application AP1 is used, the results of pressure distribution and skeletal tracking may be acquired, just as during setup.

[0111] In this way, it becomes possible to present a comparison between the wheelchair position of the person being assisted and the ideal wheelchair position, and to present information for realizing the ideal wheelchair position.

[0112] Note that when using the arrangement shown in FIG. 10 , the imaging unit 260 of the terminal device 200 can capture an image of the person being assisted from the front, thereby enabling a clearer image of the face of the person being assisted compared to when using a device fixed to the bed 610, such as the terminal device 200-1 in FIG. 5 . Therefore, when automatically selecting training data according to the person being assisted, the positioning application AP1 may automatically identify the person being assisted to be adjusted based on the results of facial recognition processing. Furthermore, the selection of training data may be performed based on the results of facial recognition processing. For example, if multiple positioning applications AP1, each corresponding to different training data, are installed in the terminal device 200, the positioning application AP1 to be launched may be determined based on the results of facial recognition processing. Alternatively, if multiple training data are associated with one positioning application AP1, the training data to be used may be selected from the multiple training data based on the results of facial recognition processing.

[0113] <Examples of devices used in positioning applications> In the above, the wheelchair 630 capable of detecting pressure has been exemplified as a device used to set up and use the positioning application AP1, but the device is not limited to this.

[0114] 13 is a diagram showing an example of another device that can be used by the positioning application AP1, such as a seat sensor 440 that is placed on the seat of a wheelchair 630. The seat sensor 440 includes a pressure sensor that outputs a pressure value, and outputs the pressure value. For example, the output of the seat sensor 440 may be used as information corresponding to the pressure distribution on the screen of FIG. 12.

[0115] The seat sensor 440 may also be capable of processing other than outputting pressure values ​​to the positioning application AP1. For example, the seat sensor 440 may determine, based on the pressure value of the pressure sensor, whether the posture of the person being assisted when sitting in the wheelchair 630 (hereinafter also referred to as the sitting posture) is one of a plurality of postures including normal, forward shift, side shift, etc. Forward shift refers to a state in which the user's center of gravity is shifted further forward than normal, and side shift refers to a state in which the user's center of gravity is shifted to either the left or right than normal. Both forward shift and side shift correspond to states in which the risk of falling from the seat is relatively high. The seat sensor 440 may also perform a fall possibility determination to determine whether or not the person being assisted is likely to fall from the seat.

[0116] 13, four pressure sensors Se1 to Se4 are arranged on the back side of a cushion 441 that is placed on the seat of a wheelchair 630. Pressure sensor Se1 is a sensor that is arranged in the front, pressure sensor Se2 is a sensor that is arranged in the rear, pressure sensor Se3 is a sensor that is arranged on the right, and pressure sensor Se4 is a sensor that is arranged on the left. Note that front, back, left, and right here refer to directions as seen from the perspective of the person being assisted when the person is sitting in the wheelchair 630.

[0117] 13, the pressure sensors Se1 to Se4 are connected to a control box 442. The control box 442 includes a processor that controls the pressure sensors Se1 to Se4 and a memory that serves as a work area for the processor. The processor detects pressure values ​​by operating the pressure sensors Se1 to Se4.

[0118] A person being assisted sitting in a wheelchair 630 may feel pain in the buttocks and shift the position of the buttocks. For example, forward slippage occurs when the buttocks are shifted further forward than usual, and lateral slippage occurs when the buttocks are shifted to the left or right. Forward slippage and lateral slippage may occur simultaneously, causing the center of gravity to shift diagonally. By using a pressure sensor disposed on the cushion 441 as shown in FIG. 13, changes in the position of the buttocks can be appropriately detected, making it possible to accurately detect forward slippage or lateral slippage. For example, the seat sensor 440 determines that forward slippage has occurred when the value of pressure sensor Se1 has increased by a predetermined amount compared to the initial state, and determines that lateral slippage has occurred when the value of pressure sensor Se3 or Se4 has increased by a predetermined amount compared to the initial state.

[0119] The seat sensor 440 may output pressure values ​​output from the pressure sensors Se1 to Se4 as sensing data to the server system 100, or may output the results of the determination of forward or lateral slippage, the determination of the possibility of a fall, etc. as sensing data to the server system 100. The control box 442 may also include a light emitting unit or the like, which may be used to notify the caregiver. In this way, the caregiver can be notified of changes in the sitting posture of the wheelchair 630 or the like in an easily understandable manner, making it possible to prevent the person being assisted from falling.

[0120] 2.3 Position of objects in the living environment of the person receiving care The above describes an example in which the position of an object such as a cushion is adjusted as additional information when adjusting the position of the person being assisted. However, in the method of this embodiment, adjusting the position of the person being assisted is not essential, and only the positions of objects placed around the person being assisted may be adjusted.

[0121] The vicinity of the person being assisted refers to a place such as a room used by the person being assisted in daily life. In this case, the device on which the positioning application AP1 runs may be the terminal device 200-1 or the terminal device 200-2 shown in Fig. 5, the terminal device 200 shown in Fig. 10, another terminal device 200 placed in the room or the like, or the terminal device 200 carried by the caregiver.

[0122] The objects placed around the person being assisted may be furniture, electrical appliances, decorative items, or other small items. The objects placed around the person being assisted may also include equipment used in medical care or nursing care, such as bed 610 with an adjustable base height and angle, wheelchair 630 with an adjustable backrest angle, and peripheral device 500, which will be described later with reference to Figures 19A to 19D.

[0123] The teacher data acquired in the setting mode is similar to the examples of the bed position and wheelchair position, and is, for example, image information captured when an object is placed in a reference position. In the use mode, the positioning application AP1 may perform a process of superimposing the teacher data and the captured image in real time, or may perform a process of determining that an object whose deviation from the reference position is equal to or greater than a predetermined threshold is NG.

[0124] In this way, it is possible to support the alignment of the objects around the person being assisted with the reference position. This process is useful, for example, in the coordination process related to agitated behavior. Details of the coordination process related to agitated behavior will be described later using Figure 23 etc.

[0125] However, if there is implicit knowledge regarding the placement of objects to ensure the safety of the caregiver, the person being assisted, the person being assisted's family, etc. (for example, to prevent falls or collisions, or to prevent objects from falling from high places), the positioning application AP1 may be configured in accordance with that implicit knowledge. This makes it possible to create a safer living environment. For example, the positioning application AP1 that adjusts the position of objects may be used to ensure safety in hospitals or nursing homes. In addition, in situations such as home care, training data acquired in hospitals or nursing homes can be used at home as well, from the perspective of utilizing the placement of objects in hospitals or nursing homes. Furthermore, training data indicating safe object placement at home may be acquired by an experienced person such as a visiting caregiver.

[0126] 3. Integration including positioning applications Next, specific cooperative processing between the positioning application AP1 and other applications and other devices will be described using several examples.

[0127] 3.1 Collaboration based on the perspective of pressure ulcers 14 is a diagram illustrating the process of linking the positioning application AP1 with other applications and devices from the viewpoint of preventing bedsores in the person being assisted. As shown in FIG. 14, in order to prevent bedsores in the person being assisted, the meal application AP2, the positioning application AP1, the motion determination device 410, and the bed 610, mattress 620, wheelchair 630, etc. may be linked together.

[0128] 3.1.1 Meal Application <Meal application operation> The meal application AP2 detects at least one of the type and amount of food ingested by the person being assisted during meal assistance for the person being assisted. That is, the information processing system 10 of this embodiment may include a device on which the meal application AP2 runs.

[0129] For example, SSKIN is known as a fundamental concept for preventing bedsores. SSKIN stands for Skin Inspection, Support Surface, Keep Moving, Incontinence Management, and Nutrition & Hydration. In other words, to prevent bedsores, it is effective to comprehensively use items that can be managed using the positioning application AP1, such as Support Surface and Keep Moving, and items that can be managed using the diet application AP2, called Nutrition & Hydration. Therefore, by linking the operation of the positioning application AP1 and the diet application AP2, it is possible to effectively prevent bedsores.

[0130] Fig. 15 is a diagram illustrating an example of a device on which meal application AP2 runs, namely, choking detection device 460 used during mealtimes. As shown in Fig. 15, choking detection device 460 includes a throat microphone 461 worn around the neck of the person being assisted, and a terminal device 462 equipped with a camera. However, meal application AP2 may run on a device other than choking detection device 460, such as terminal device 200 or server system 100.

[0131] The throat microphone 461 outputs audio data resulting from swallowing, coughing, etc. of the person being assisted. The camera of the terminal device 462 outputs captured images of the person being assisted eating. The terminal device 462 is, for example, a smartphone or tablet PC placed on a table where the person being assisted eats. The throat microphone 461 is connected to the terminal device 462 using Bluetooth (registered trademark) or the like, and the terminal device 462 is connected to the server system 100 via a network. However, both the throat microphone 461 and the terminal device 462 may be directly connectable to the server system 100, and various modifications can be made to the specific connection mode.

[0132] For example, the terminal device 462 may include a memory that stores the meal application AP2 and a processor that operates in accordance with the meal application AP2. The terminal device 462 operates in accordance with the meal application AP2 to perform a process of determining at least the type and amount of food to be ingested by the person being assisted.

[0133] 16A and 16B are diagrams illustrating examples of operation screens of the meal application AP2. These operation screens are displayed on the display unit of the terminal device 462, for example.

[0134] FIG. 16A shows a screen for inputting information identifying the person receiving care and information identifying the time before or after a meal. For example, in a nursing care facility, in order to perform dietary management for each person receiving care, a tag or the like is attached to each dish served to indicate which person the dish is prepared for. Note that a dish refers to food served in a dish. The tag here may include, for example, a barcode. Information identifying the target person receiving care is obtained by reading the barcode using a barcode reader such as a camera. The screen shown in FIG. 16A displays text instructing the user to read the barcode, as well as an image captured by the camera of the terminal device 462. The caregiver adjusts the positional relationship between the camera and the barcode so that the barcode is clearly captured, and the meal application AP2 of the terminal device 462 obtains information identifying the person receiving care. When the barcode is read, the meal application AP2 may display the name of the person receiving care, which is the read result, in the "Name" field of the screen shown in FIG. 16A.

[0135] 16A may also include radio buttons for selecting either "before" or "after." "Before" indicates before a meal, and "after" indicates after a meal. By selecting either option, the meal application AP2 can determine whether the captured image acquired using the screen of FIG. 16B is a before-meal image or an after-meal image.

[0136] When the next button is selected in FIG. 16A, meal application AP2 displays the screen in FIG. 16B. The screen in FIG. 16B displays the name of the person being assisted and before / after information acquired using the screen in FIG. 16A. The screen in FIG. 16B also displays images captured by the camera of terminal device 462 as moving images. While FIG. 16B shows an example in which the food is captured from directly above, the food may also be captured from an oblique angle. When the picture button is selected on the screen in FIG. 16B, the image displayed at that time is stored as a still image. This allows images of the food before and after the meal to be acquired. While FIG. 16B shows an example in which images of food served in three dishes are captured, the specific number of dishes in a single menu can be varied in various ways.

[0137] When the analysis button is selected in Fig. 16B, meal application AP2 determines the type and amount of food to be ingested by the person being assisted based on the stored still image. Figs. 17A and 17B are diagrams illustrating the processing executed by meal application AP2. First, as shown in Fig. 17A, meal application AP2 detects an area corresponding to a dish from a still image. For example, meal application AP2 performs processing to detect rectangular areas that contain dishes on which food is served. Here, the detection processing detects areas that contain each of the three dishes, and therefore rectangular areas R1 to R3 are detected. A wide range of well-known object detection techniques can be applied to this processing, and therefore a detailed description will be omitted.

[0138] The meal application AP2 performs processing to determine the type of food based on the rectangular regions R1 to R3 detected by object detection. For example, Yanai et al.'s "FOOD IMAGE RECOGNITION USING DEEP CONVOLUTIONAL NETWORK WITH PRE-TRAINING AND FINE-TUNING" (http: / / img.cs.uec.ac.jp / pub / conf15 / 150703yanai_0.pdf) discloses a method for recognizing food from an image based on a deep convolutional neural network (DCNN). The meal application AP2 of this embodiment may determine the type of food based on the results of image processing, as with these methods. For example, the meal application AP2 identifies the type of food by inputting images corresponding to each of the rectangular regions R1 to R3 into the DCNN. The example in FIG. 17A shows an example in which the three dishes are identified as "rice," "miso soup with tofu and wakame seaweed," and "sauteed mushrooms and lettuce," respectively. Furthermore, the calories and nutrients ingested by each food are identified based on the results of the food type identification. As shown in FIG. 17A, the identified calories and nutrients may be displayed in association with the type of food.

[0139] Furthermore, as shown in FIG. 17B, meal application AP2 may perform a process of trimming a circular area from an image of food served in a dish. While the following description uses rectangular area R1 as an example, similar processing is performed for rectangular areas R2 and R3. The circular area is the area inside a circular shape C1. For example, meal application AP2 may transform rectangular area R1 detected by the process shown in FIG. 17A into a square, and then perform trimming using a circular shape C1 inscribed in the square. This increases the proportion of food and the dish on which the food is served in the trimmed result, thereby reducing the influence of other objects such as a table. Note that the circular shape C1 here is not limited to a perfect circle, but may be an ellipse, a shape with a concave or convex portion, or a wide range of shapes that differ from a circle by a predetermined degree or less.

[0140] Furthermore, meal application AP2 calculates a first pixel value, which is a pixel value of a first region corresponding to the center of the trimmed result, and a second pixel value, which is a pixel value of a second region corresponding to the peripheral portion of the trimmed result. Hereinafter, for convenience of explanation, the first region will be referred to as the central region, and the second region will be referred to as the peripheral region. The first pixel value will be referred to as the central pixel value, and the second pixel value will be referred to as the peripheral pixel value. The pixel value here may be a pixel value in a grayscale image, or one or more of the pixel values ​​of each of RGB in a color image. Given that the central region and peripheral region contain multiple pixels, the central pixel value and peripheral pixel value may not be a single value, but may be a collection (e.g., a distribution) of multiple pixel values.

[0141] For example, the meal application AP2 may set two circular shapes C1 and C2 as the trimming result of the rectangular region R1, thereby setting a central region and a peripheral region. The central region is the region inside the circular shape C2. The peripheral region is the region outside the circular shape C2 and inside the circular shape C1. The circular shape C2 is a region smaller than the circular shape C1, and the size ratio between the two circular regions may be fixed or may be changeable based on user input, etc. Note that the circular shapes C1 and C2 are, for example, concentric circles with a common center, but may also be set with offset centers.

[0142] As shown in the example of Figure 17B, there is a high probability that food will be placed in the center of the dish. Therefore, in the central region of the image before eating, the area where food is captured is relatively wide. As a result, the distribution of central pixel values ​​before eating has a high frequency of pixel values ​​corresponding to food and a low frequency of pixel values ​​corresponding to the dish. In contrast, there is a low probability that food will be placed all the way to the peripheral region of the dish. Therefore, even in the image before eating, in the peripheral region of the image, the area where food is captured is narrow and the area where the dish is captured is wide. As a result, the distribution of peripheral pixel values ​​before eating has a low frequency of pixel values ​​corresponding to food and a high frequency of pixel values ​​corresponding to the dish.

[0143] For example, the meal application AP2 may compare the distribution of central pixel values ​​in the image before eating with the distribution of peripheral pixel values ​​in the image before eating to determine the range of pixel values ​​corresponding to food and the range of pixel values ​​corresponding to bowls. Specifically, as described above, the range with high frequency in both central pixel values ​​and peripheral pixel values ​​corresponds to bowls, and the range with high frequency in central pixel values ​​but low frequency in peripheral pixel values ​​corresponds to food. For example, the meal application AP2 may set a threshold value that defines the boundary between pixel values ​​corresponding to food and pixel values ​​corresponding to bowls within the range of possible pixel values ​​(e.g., 0 to 255). Since Otsu's binarization method or other well-known methods can be widely applied to set the threshold value, a detailed description will be omitted.

[0144] While the above example illustrates the use of DCNN as an example of the process by which meal application AP2 identifies food types, this is not limiting. For example, meal application AP2 may determine the type of food (dish) based on the range of pixel values ​​corresponding to the identified food. For example, table data correlating pixel value ranges with food types is stored in advance, and meal application AP2 identifies the type of food based on the table data. Furthermore, meal application AP2 may acquire user input indicating whether the target dish is a main dish or a side dish, and use the user input in the process of identifying the type of food.

[0145] Furthermore, when a person receiving care eats food from a bowl, the amount of food left in the bowl changes depending on the amount of food consumed. In other words, the area in which food is captured in the post-meal image becomes narrower depending on the amount of food consumed compared to before eating, so when looking at the distribution of central pixel values ​​after eating, the frequency of pixel values ​​corresponding to food decreases compared to before eating. The degree of decrease in the frequency of pixel values ​​corresponding to food is related to the amount of food consumed by the person receiving care.

[0146] Therefore, the meal application AP2 may calculate an index value representing the amount of intake by calculating the decrease in the frequency of pixel values ​​corresponding to food in the central pixel value after a meal compared to the frequency of pixel values ​​corresponding to food in the central pixel value before a meal. The meal application AP2 performs a process of matching pre-meal images and post-meal images of the same food based on information identifying the person receiving care and the timing at which the still images were acquired. For example, the meal application AP2 uses the capture timing of the pre-meal image of the target person receiving care as a reference and identifies a post-meal image of the same person receiving care that was captured after the capture timing but closest to the capture timing. The meal application AP2 then calculates the index value representing the amount of intake by treating the two images as a pair of images of the same food served to the same person receiving care.

[0147] For example, the meal application AP2 stores in advance table data that associates actual intake amounts with index values. Here, the intake amounts are evaluated, for example, on a scale of 1 to 10. The meal application AP2 then calculates an index value, which is the difference or ratio between the number of pixels corresponding to food before eating and the number of pixels corresponding to food after eating, and compares this index value with the table data to calculate the intake amount on a 10-point scale. While the above describes an example using table data, a function that calculates the intake amount from the index value may also be used. This function may be a linear function or a nonlinear function. Furthermore, the number of stages for determining the intake amount may be more or less than 10, and various modifications of the specific processing are possible.

[0148] <Integration of food application and positioning application> As mentioned above, diet is important for preventing pressure ulcers, and it is believed that assisted individuals with dietary problems are at a higher risk of developing pressure ulcers. Therefore, when the dietary application AP2 detects at least one of an insufficient amount of food consumed by the assisted individual and a nutritional deficiency, the positioning application AP1 changes the operating mode of the positioning application from mode 1 to mode 2, which imposes a relatively higher processing load. The amount of food consumed is information that represents the total amount of food consumed by the assisted individual over a specified period, regardless of the type of food. For ease of explanation, the total amount of food consumed by the assisted individual over a specified period will also be referred to as the dietary amount. Nutritional deficiency refers to an assisted individual's inadequate intake of each nutrient contained in food, such as carbohydrates, lipids, proteins, and vitamins, without reaching the required amount.

[0149] In this way, the positioning application AP1 can be actively used in situations where there is a high risk of bedsores, such as when the amount of food eaten is insufficient to begin with, or when the amount of food eaten is sufficient but a specific nutrient is lacking, etc. As a result, it is possible to reduce the risk of bedsores.

[0150] For example, the meal application AP2 determines whether the amount of food consumed is insufficient by comparing the actual amount of food consumed with a reference amount of food determined based on the attributes of the target person receiving care. The meal application AP2 may also determine the amount of nutrients contained per unit amount in the target food (dish) by identifying the type of food. The meal application AP2 then determines the amount of nutrients ingested through the target meal by multiplying the amount of nutrients per unit amount by the amount of food consumed by the person receiving care. For example, the meal application AP2 determines whether the target person receiving care is nutritionally insufficient by comparing the actual amount of nutrients ingested with a reference amount of nutrients determined based on the attributes of the target person receiving care.

[0151] Among the operation modes of the positioning application AP1, the first mode is an operation mode corresponding to an off state or a standby state, and may be, for example, a state in which the positioning application AP1 is not running. The terminal device 200 can execute applications other than the positioning application AP1, regardless of whether the terminal device 200 itself is on or off. The second mode is an operation mode in which the positioning application AP1 is running, and outputs, for example, superimposed display of training data or OK / NG based on skeletal tracking. This makes it possible to appropriately launch the positioning application AP1 when a high risk of bedsore is determined based on the meal application AP2. For example, when the positioning application AP1 is in an inactive state, the meal application AP2 may be running first, and the positioning application AP1 may be launched when the meal application AP2 detects a food intake or nutritional deficiency.

[0152] However, examples of the first mode and the second mode are not limited to these. For example, the first mode may be a state in which the positioning application AP1 is already running. The positioning application AP1 may be started using triggers such as detection of a person in bed 610, detection of a diaper change, or detection of a person getting into wheelchair 630. Then, the positioning application AP1 transitions to the second mode when the meal application AP2 detects a lack of food intake or nutrition while the positioning application AP1 is running. For example, the second mode may be a state in which the positioning application AP1 uses more functions than the first mode. The functions of the positioning application AP1 include the various functions described above, such as the function of superimposing teacher data, the function of determining OK / NG, the function of automatically selecting teacher data, and the function of automatically controlling the bed 610, etc.

[0153] For example, the first mode may be an operation mode that executes only one of superimposing the training data and making an OK / NG judgment, and the second mode may be an operation mode that executes both of these processes. In this way, it becomes possible to increase the number of functions used by the positioning application AP1 when it is determined that the risk of bedsores is high based on the diet application AP2.

[0154] Furthermore, the positioning application AP1 can operate in various situations, such as when the person being assisted is in bed, when a diaper is being changed, when the person being assisted is moving in the wheelchair 630, and when the person being assisted is eating in the wheelchair 630. For example, the second mode may operate in more situations than the first mode. For example, the first mode may be a mode that operates in one of the above-mentioned multiple situations, and the second mode may be a mode that operates in two or more of the above-mentioned multiple situations.

[0155] In addition, the second mode may be any operation mode that imposes a higher processing load than the first mode, and various modifications of the specific aspects are possible.

[0156] <Examples of other processes performed by the choking detection device> The above describes an example in which meal application AP2 operates on terminal device 462 of choking hazard detection device 460. However, choking hazard detection device 460 may execute a process different from the process for determining the type and amount of food. For example, choking hazard detection device 460 may execute the process described below based on audio data and captured images to determine various pieces of information related to the meal. Note that the process described below may be implemented by an application different from meal application AP2, or may be implemented as part of meal application AP2.

[0157] For example, the choking detection device 460 determines whether the person being assisted has choked or swallowed based on audio data from a throat microphone 461. A device that detects swallowing using a microphone worn around the neck is described, for example, in U.S. Patent Application No. 16 / 276,768, filed February 15, 2019, entitled "Swallowing action measurement device and swallowing action support system." This patent application is incorporated by reference in its entirety. Based on the audio data, the processor can detect the number of choking incidents, the duration of the choking (time of occurrence, duration, etc.), and whether or not the person swallowed.

[0158] The camera of the terminal device 462 can detect the mouth, eyes, and chopsticks and spoons used by the person being assisted by capturing an image of the person being assisted from the front, as shown in Fig. 15. There are various known methods for detecting these facial features and objects based on image processing, and a wide range of known methods can be applied in this embodiment.

[0159] For example, the choking detection device 460 can determine, based on the image captured by the camera, whether the person being assisted has their mouth open, whether food is coming out of their mouth, and whether they are chewing food. The choking detection device 460 can also determine, based on the image captured by the camera, whether the person being assisted has their eyes open. The choking detection device 460 can also determine, based on the image captured by the camera, whether chopsticks, a spoon, etc. are near tableware, whether the person being assisted can hold them, and whether food is being spilled.

[0160] In the method of this embodiment, the swallowing and choking status of the person being assisted is estimated based on this information. For example, the choking detection device 460 may obtain information about meals based on the detection results of choking and swallowing and the determination result of whether the person being assisted is opening or closing their mouth.

[0161] For example, the swallowing choking detection device 460 may determine whether choking is occurring frequently based on the number of times or duration of choking, and output the determination result. For example, the swallowing choking detection device 460 may determine that choking is occurring frequently when the number of times choking per unit time exceeds a threshold. In this way, the situation regarding choking can be automatically determined.

[0162] The choking detection device 460 may also calculate the swallowing time from when the person being assisted opens their mouth to when they swallow, based on the swallowing detection result and the result of determining whether the person being assisted has opened or closed their mouth. This makes it possible to determine specific situations, such as whether the person is not putting food into their mouth, or whether the person has put food in their mouth but is not swallowing it, when the number of swallows has decreased. For example, the choking detection device 460 may start counting up a timer when the mouth transitions from a closed state to an open state based on the captured image of the terminal device 462, and stop counting the timer when swallowing is detected by the throat microphone 461. The time when the timer stops counting represents the swallowing time. This makes it possible to accurately determine whether the risk of aspiration is high during meals and whether the caregiver should take some kind of action.

[0163] Choking detection device 460 may also determine the pace of eating based on the swallowing time. Choking detection device 460 may also determine whether the swallowing time is long based on a change in the swallowing time during one meal (for example, the increase or ratio compared to the swallowing time at the beginning). Alternatively, the processor may calculate the average swallowing time for each of multiple meals for the same person receiving care, and determine whether the swallowing time has become long based on the change in the average swallowing time.

[0164] Furthermore, by using the results of determining whether the mouth is open or closed based on the captured image of the terminal device 462, it can be determined whether the mouth no longer opens even when the caregiver approaches a spoon or the like. In this way, if the swallowing time becomes longer when the person being assisted is reluctant to open their mouth, it can be estimated that food is remaining in the mouth and becoming stagnant. Furthermore, by using the captured image to recognize whether food is coming out of the mouth and whether the food is being chewed, it can be determined whether the person being assisted is no longer able to chew the food. For example, if the number of chews is normal but the swallowing time is long, it can be estimated that the person being assisted is no longer able to chew the food. Furthermore, if it is determined that the eyes are closed based on the captured image, it can be determined whether the person being assisted is becoming sleepy.

[0165] Furthermore, by performing a recognition process on chopsticks, spoons, etc. using the captured image, it may be determined whether the person is playing with food, unable to hold a dish, doing nothing, etc. For example, if an object such as a spoon is overlapping the hand of the person being assisted, but the time it takes for the person to bring the object to their mouth is equal to or greater than a predetermined threshold, it is determined that the person is unable to hold a dish or is playing with food. Furthermore, if an object such as a spoon is not overlapping the hand of the person being assisted, and the time the person's gaze is directed toward food (food) is equal to or greater than a predetermined threshold, it is determined that the person is looking at the food without doing anything.

[0166] Furthermore, in the method of this embodiment, the operation of the positioning application AP1 may be changed based on the processing results of the choking stool detection device 460. Note that the control of the positioning application AP1 may be triggered by the eating application AP2, by other processing results of the choking stool detection device 460, or by both.

[0167] For example, the choking detection device 460 may determine the swallowing ability of the person being assisted based on the number and frequency of choking, the severity of choking, changes in swallowing time, etc., determined by the above-mentioned techniques. When the choking detection device 460 determines that the swallowing ability has decreased to a predetermined level or below, the positioning application AP1 may transition from the first mode to the second mode. For example, the positioning application AP1 may be started up in response to a decrease in swallowing ability.

[0168] Furthermore, when the person being assisted's ability to eat declines, the choking hazard detection device 460 may start monitoring the meal. Therefore, the positioning application AP1 may be launched when the operation of the choking hazard detection device 460 starts. As described above, the choking hazard detection device 460 has various functions, each corresponding to a different process, and these functions may be activated or deactivated. For example, when the swallowing ability of the person being assisted declines, the number of functions set to be activated in the choking hazard detection device 460 increases. In this case, the positioning application AP1 may be launched when the number of functions set to be activated in the choking hazard detection device 460 increases or when a specific function becomes activated. Furthermore, the operation mode of the positioning application AP1 may be controlled based on the ADL of the person being assisted, in addition to the processing results of the choking hazard detection device 460 and the functions set to be activated.

[0169] Furthermore, the choking detection device 460 determines the timing of swallowing and the amount of swallowing based on the audio data detected by the throat microphone 461, and it is known that the feature amount of the audio data changes depending on various circumstances. The feature amount here may be the amplitude of the waveform of the audio data, the frequency, or the number of waves included in one swallow. For example, the sound of swallowing changes depending on factors such as the hardness or thickness of the food, the pace of eating, etc. It has also been found that the sound of swallowing changes depending on the physical condition of the person being assisted, even when the hardness, etc., are the same.

[0170] Therefore, the choking detection device 460 may obtain features representing the swallowing sound based on the output of the throat microphone 461, and estimate the physical condition of the person being assisted based on the features. For example, the choking detection device 460 stores data that associates the features of the swallowing sound with the physical condition of the person being assisted based on past history. The choking detection device 460 then estimates the physical condition of the person being assisted by comparing the data with the actually acquired features. Note that the data that associates the features of the swallowing sound with the physical condition of the person being assisted may be a trained model created by machine learning.

[0171] The positioning application AP1 switches processing when the choking detection device 460 determines that the physical condition of the person being assisted is worse than a reference value. For example, if the positioning application AP1 is capable of executing multiple functions as described above, when the physical condition of the person being assisted is determined to be poor, the number of functions that are set to be active may be greater than when the physical condition of the person being assisted is determined to be good.

[0172] In addition, in the method of this embodiment, when a meal event is detected, a medication management system may operate in cooperation with the medication management system. For example, a medication support robot that notifies users of medication times, dispenses medication based on user operations, and manages medication histories is known, and the medication support robot may be used as a medication management system. Also, medication support applications that allow users to understand medication status and cloud systems that analyze and display medication information are known, and a medication support system that includes these may be used.

[0173] For example, by using the above-described meal application AP2, events such as the start and end of a meal can be detected. The medication support system operates using these meal events as triggers, enabling accurate medication management for each meal (or, in a narrower sense, after each meal). For example, the positioning application AP1 may detect, as a meal start event, the person being assisted getting into a position to eat in the wheelchair 630 or bed 610. Furthermore, the meal application AP2 may detect a meal end event when it receives an image of a person after eating, and the choking detection device 460 may detect a meal end event when no swallowing occurs for a predetermined period of time or longer. However, various variations on the method for detecting meal events are possible, such as detecting a meal start event when the meal application AP2 receives an image of a person before eating.

[0174] Although the above describes an example in which a meal event is detected using both the positioning application AP1 and the meal application AP2 (or the choking detection device 460), the present invention is not limited to this. For example, when the meal application AP2 is inactive, a meal event may be detected based on the positioning application AP1, and collaboration with the medication support system may be performed based on the detection result.

[0175] Furthermore, rather than providing a separate medication assistance system, medication assistance may be performed by the meal application AP2 and the choking detection device 460. For example, the meal application AP2 may receive an image of the medication to be taken and, based on the image, perform processing to determine the type and amount of medication the person being assisted is about to take. The choking detection device 460 may also perform processing to determine whether the medication has been properly swallowed based on an image of the area around the mouth captured using a camera and the sound of swallowing, etc. In this way, the meal application AP2 and the choking detection device 460 can provide medication assistance at the appropriate time.

[0176] Furthermore, although the above describes an example in which the positioning application AP1 and the like work together with the medication support system based on a meal event, this does not prevent other systems that operate in response to a meal from working together with the positioning application AP1 and the like.

[0177] 3.1.2 Motion detection device <Collaboration based on positioning application> 14, the information processing system 10 of this embodiment may also include a motion determination device 410 that determines motion. Note that, although the motion determination here refers to the determination of motion related to bedsores in a narrow sense, the motion determination device 410 may also perform other motion determinations.

[0178] The motion determination device 410 may change the operation mode of the motion determination device 410 when the operation mode of the positioning application AP1 transitions to the second mode. This allows the motion determination device 410 to operate in conjunction with the activation of the positioning application AP1 or the addition of new functions. The motion determination device 410 may also change the operation mode of the motion determination device 410 when an abnormality in the person being assisted is detected based on the processing results of the positioning application AP1. An abnormality here includes, for example, when the position adjustment is not completed even after a predetermined time has elapsed since the start of the position adjustment, or when the OK / NG determination result is NG. This allows the motion determination device 410 to perform an operation to address the detected abnormality. Details are described below.

[0179] 14, the motion determination device 410 may include an acceleration sensor 411, a hand switch 413, a detection device 430, and a seat sensor 440. However, other devices such as a bedside sensor 420, a bed 610 or mattress 620 capable of detecting pressure, etc. may also be used as the motion determination device 410.

[0180] The acceleration sensor 411 is a sensor that is attached to, for example, the skin or clothing of the person being assisted. The acceleration sensor 411 may be a three-axis acceleration sensor, a six-axis gyro sensor, or other configurations.

[0181] The hand switch 413 is a user interface provided on the bed 610 or the wheelchair 630, and includes, for example, buttons for changing the angle of the bed bottom of the bed 610 or the angle of the backrest of the wheelchair 630. The hand switch 413 may also include a button used to call a caregiver when the person being assisted senses something is wrong. For example, the hand switch 413 is connected to a control device provided on the bed 610 or the wheelchair 630, and the angle of the bed bottom or the backrest is controlled based on the operation result of the hand switch 413. The detection device 430, the seat sensor 440, and the mattress 620 are as described above.

[0182] In this embodiment, the change in the operation mode of the motion determination device 410 may be a transition from inactive to active, or a transition from a state in which fewer functions are used to a state in which more functions are used when the motion determination device 410 has multiple functions.

[0183] The motion determination device 410 may perform various processes. For example, the acceleration sensor 411 may determine the posture of the person being assisted in the bed 610 (sitting, right lateral position, left lateral position, etc.). Furthermore, if the hand switch 413 is frequently operated, it is considered that the person being assisted intends to change to a different posture, and one possible reason for this is that the person is experiencing pain in the current posture. Therefore, based on the output of the hand switch 413, information such as the frequency and time period when the person being assisted feels abnormal, and the angle of the seat bottom or backrest when the abnormality is felt, is obtained. The detection device 430, the seat surface sensor 440, and the mattress 620 obtain information indicating the tendency of pressure distribution. Furthermore, the detection device 430 may obtain information such as sleep / wake and whether the person is in bed / out of bed, as described above. Furthermore, the seat surface sensor 440 may output the determination results of the possibility of forward or sideways slippage, or of tipping over or falling (hereinafter referred to as "possibility of tipping over") as attribute information of the person being assisted.

[0184] In this way, not only can the risk of bedsores be reduced by adjusting the position using the positioning application AP1, but it can also be used to monitor the movements of the person being assisted even when no position adjustment is being performed.

[0185] For example, the risk of the person being assisted may be determined based on the above-mentioned various information obtained by the motion determination device 410. For example, the risk of the person being assisted is determined to be high when conditions such as a higher than normal heart rate, insufficient sleep time, frequent awakenings during the night, getting out of bed in the middle of the night, no decrease in body movement even when asleep, frequent occurrence of forward slippage, high possibility of falling, etc. are met.

[0186] Furthermore, if the following detection results are obtained by the motion determination device 410, it may be determined that the risk of bedsore is high. The acceleration sensor 411 detects that the person being assisted is facing the same direction for a predetermined period of time. The detection device 430 detects that the person being assisted has been facing the same direction for a predetermined period of time. The seat sensor 440 detects that pressure has been applied to a certain point for a certain period of time. The hand switch 413 detects a behavior that suggests the person being assisted is in pain (for example, the hand switch 413 is operated more frequently than a predetermined frequency due to frequent changes in posture).

[0187] When such a risk is detected, the motion determination device 410 may notify the caregiver or the like of information about the risk. For example, the motion determination device 410 may present the caregiver with the name of the high-risk care recipient, the details of the risk, the degree of the risk, etc., by outputting audio or displaying text. The motion determination device 410 may also output information urging the caregiver or the like to take specific interventions, such as reviewing bedding or applying medicine to the skin.

[0188] <Collaboration based on motion detection devices> As described above, by operating the motion determination device 410 in cooperation with the positioning application AP1, it is possible to monitor the movements of a person being assisted who is estimated to be at high risk of developing a bedsore based on their dietary habits. For example, as described above, it is possible to evaluate the risk of developing a bedsore based on the specific movements of the person being assisted even when position adjustment is not being performed.

[0189] The risk detected by the motion determination device 410 may be output to the positioning application AP1, for example, and used to change the operating mode of the positioning application AP1. For example, when the motion determination device 410 determines that the risk of the person being assisted is equal to or greater than a predetermined threshold, the positioning application AP1 may transition from a motion suitable for a person being assisted with a first attribute to a motion suitable for a person being assisted with a second attribute different from the first attribute. The risk here refers to the risk of bedsores in a narrow sense. However, the risk of the person being assisted may also be other risks. The attributes here include information such as age, gender, height, weight, medical history, and medication history, but may also include information representing risk. The information representing risk may be a numerical value indicating the degree of risk or an output of the motion determination device 410 used to calculate the risk. For example, the acceleration value of the acceleration sensor 411, the pressure value of the detection device 430 or the seat sensor 440, the operation history of the hand switch 413, etc. may be used as attribute information related to risk.

[0190] According to the method of this embodiment, it is possible to select training data suited to the risk of a person being assisted who has been determined to be at high risk by the motion determination device 410. For example, the training data of this embodiment is associated with attribute information that identifies attributes of the person being assisted that are suitable for the training data, with the first attribute being an attribute corresponding to a person being assisted at low risk, and the second attribute being an attribute corresponding to a person being assisted at high risk. Because training data suited to the person being assisted can be selected, it is possible to adjust the position to reduce the risk to the person being assisted.

[0191] Furthermore, even among individuals receiving care who are at high risk of bedsores, the desired teacher data varies depending on various factors, such as the degree of dementia, the presence or absence of contractures, and sleeping posture preferences (whether they prefer a supine or lateral position). Furthermore, individuals receiving care who are at high risk of bedsores are more likely to need position adjustments, making it easier to create teacher data for them. Therefore, it has not been easy for caregivers to select appropriate teacher data for individuals receiving care who are at high risk of bedsores. In this regard, the method of this embodiment reviews attributes when a high risk of bedsores is detected, thereby automatically selecting teacher data tailored to the individual receiving care. As a result, convenience is also improved. Below, several specific examples are described.

[0192] For example, adjusting the bed position allows the person receiving care to adopt an appropriate posture, which is expected to lead to higher quality sleep. However, as described above, if the detection device 430 detects conditions such as a higher-than-normal heart rate, insufficient sleep time, or frequent awakenings during the night, the current posture may not be suitable for the person receiving care. In such cases, changing the training data can allow the person receiving care to adopt a more suitable posture. For example, as described above, information representing the attributes of the person receiving care is obtained in advance, and the training data is also associated with attribute information representing the attributes of the person receiving care that the training data is suitable for. Then, by switching the training data based on the attributes, it is possible to use training data that is appropriate for the person receiving care. In addition to changing the training data, the positioning application AP1 may also notify the person receiving care of risks. For example, the positioning application AP1 may present the content and degree of risk to the caregiver by outputting audio or displaying text. The positioning application AP1 may also output information prompting the caregiver to take specific interventions, such as reviewing bedding.

[0193] The same applies to wheelchair position. For example, if the seat sensor 440 detects a possibility of forward slippage or a fall, the current posture may not be suitable for the person being assisted in the wheelchair 630. Therefore, by changing the training data, it becomes possible to allow the person being assisted to adopt a posture that is more suitable for the person being assisted. As described above, the positioning application AP1 may also notify the person being assisted of any risks.

[0194] When a risk is detected by the motion determination device 410, it is possible that the attributes of the person being assisted have changed from their previous attributes. For example, the attributes (ability to maintain a sitting position or medical history) may change due to factors such as a decline in the person being assisted's ability to maintain a sitting position or the progression of dementia in the person being assisted. Therefore, when a risk is detected by the motion determination device 410, the positioning application AP1 may update the information representing the attributes of the person being assisted and then update the training data.

[0195] As described above, the risk of bedsores can be detected based on the hand switch 413. For example, if the hand switch 413 is frequently operated, it is likely that the person being assisted intends to change to a different position. One possible reason for this is that the person is experiencing pain in their current position. Therefore, attribute information, such as the frequency and time period when the person being assisted feels unwell, and the angle of the chair bottom or backrest when the person feels unwell, may be obtained based on the output of the hand switch 413. If a risk is detected based on the hand switch 413 in this way (if the attributes of the person being assisted include information indicating that the hand switch 413 is operated frequently), the situation may not improve despite the person being assisted attempting to change their position. Therefore, the positioning application AP1 may suggest updating the training data. This makes it possible to create new training data suitable for the target person being assisted. Alternatively, the use of tacit knowledge from a perspective other than position adjustment (for example, changing bedding) may be suggested.

[0196] Furthermore, the output of the motion determination device 410 is not limited to that used for automatic selection of training data. For example, the output of the motion determination device 410 may be used when setting (creating) training data. For example, when the acceleration sensor 411 is changed from inactive to active, the output of the acceleration sensor 411 may be displayed on the screens of FIG. 7 and FIG. 12. This allows for more information to be presented when an experienced caregiver is setting training data. Furthermore, the output of the motion determination device 410 may be used when the positioning application AP1 determines whether the training data is OK or NG. For example, the positioning application AP1 may determine whether the current posture of the person being assisted is OK or NG by comparing the acceleration value from the acceleration sensor 411 with the acceleration value included in the training data.

[0197] Although the risk of the person being assisted has been determined in the above, the determination by the motion determination device 410 is not limited to this. For example, the motion determination device 410, such as the detection device 430, may determine whether the person being assisted is asleep or awake. The positioning application AP1 may switch its operation mode depending on whether the person being assisted is asleep or awake.

[0198] For example, if the person being assisted is asleep, the positioning application AP1 transitions to an operating mode in which the function for displaying the superimposed image is active and the function for determining whether the position is OK / NG is inactive. When the person being assisted is asleep, the person being assisted will not voluntarily adjust their position as a user, so it is likely that the caregiver will move the person being assisted or the cushion. Therefore, since the user in this case is more of an expert than the person being assisted, there is a high probability that they will be able to determine whether the position is OK / NG by looking at the superimposed image.

[0199] Alternatively, when the person being assisted is asleep, the positioning application AP1 may be inactive, and tracking may be performed using the acceleration sensor 411, the detection device 430, the mattress 620 having a pressure detection function, and the like. Then, when an alert is detected based on the tracking results, the positioning application AP1 may be activated, or a notification may be sent to the caregiver. The alert may be output, for example, when it is determined based on the acceleration sensor 411 that the person being assisted has been in a lateral position with the same side of the body facing down for a predetermined period of time or when it is determined based on the detection device 430 or the mattress 620 that the person being assisted is located at the edge of the bed 610. When an alert is detected, an automatic position change may be performed using the mattress 620 or the like. If the alert still persists or if multiple types of alerts are output, the positioning application AP1 may be activated or a notification may be sent to the caregiver. Various modifications of the specific control may be implemented.

[0200] On the other hand, when the person being assisted is awake, the positioning application AP1 switches to an operation mode in which the function for superimposing display and the function for determining OK / NG are active. This is because when the person being assisted is awake, there is a high probability that the person being assisted will act as a user and voluntarily adjust the position, and it is difficult for a non-professional person being assisted to determine OK / NG.

[0201] Furthermore, the activation / deactivation of each function may be controlled by combining the sleep / wake determination result with other information. For example, the positioning application AP1 may perform facial recognition of the person being assisted and switch the activation / deactivation of functions for each person being assisted. Alternatively, the positioning application AP1 may control the activation / deactivation of functions based on an index value of the person being assisted's ADL (Activities of Daily Living). For example, the positioning application AP1 may add a function to be used when the ADL of the person being assisted declines. Note that the detection device 430 or the seat sensor 440 may be used to determine the ADL of the person being assisted. Alternatively, the positioning application AP1 may add a function to be used when the startup time of the positioning application is equal to or longer than a predetermined threshold. A startup time equal to or longer than a predetermined threshold indicates that it takes a long time for the person being assisted to assume an appropriate posture even when the positioning application AP1 is being used. Therefore, it is useful to support position adjustment by adding a function.

[0202] The positioning application AP1 may also switch the training data depending on which motion determination device 410 it is communicating with. For example, the training data used by the positioning application AP1 includes training data for adjusting the bed position and training data for adjusting the wheelchair position. When communicating with the bed 610, mattress 620, detection device 430, etc., the positioning application AP1 selects the training data for adjusting the bed position. When communicating with the wheelchair 630 or seat sensor 440, the positioning application AP1 selects the training data for the wheelchair position. This enables automatic selection of training data according to the situation.

[0203] 3.1.3 Other Equipment If the motion determination device 410 detects a risk to the person being assisted, it is considered that the posture of the person being assisted is not appropriate, as described above. In this case, it would be effective to change the training data of the positioning application AP1, but it is also possible that the conditions of the bed 610, mattress 620, and wheelchair 630 are not suitable for the person being assisted in the first place.

[0204] Therefore, as shown in FIG. 14 , the motion determination device 410 may cooperate with a bed 610, a mattress 620, a wheelchair 630, and the like. For example, if the motion determination device 410 determines that the person being assisted is at high risk, the bed 610 executes control to change the height or angle of the bottom. Furthermore, mattresses capable of automatic position changes are known as the mattress 620. Therefore, if the motion determination device 410 determines that the person being assisted is at high risk, the mattress 620 executes control to prompt the person being assisted to automatically change position. For example, the mattress 620 may repeatedly execute control to increase some air and decrease other air multiple times. Furthermore, if the motion determination device 410 determines that the person being assisted is at high risk, the wheelchair 630 executes control to change the height of the seat, the angle of the backrest, and the like.

[0205] As described above, when the positioning application AP1 holds setting information for the bed 610 and the wheelchair 630, the control of the bed 610, etc. may be executed by the positioning application AP1. In other words, when a risk is detected by the motion determination device 410, the control of the bed 610, etc. may be executed independently of the positioning application AP1, or the positioning application AP1 may be the main controller of the bed 610, etc., and various modifications of the specific aspects are possible.

[0206] The positioning application AP1 may also cooperate with a positioning pillow that includes a pressure sensor. For example, when the positioning pillow is used while being sandwiched between the feet of the person being assisted, the pressure value detected by the pressure sensor is higher than when the positioning pillow is not sandwiched between the feet. For example, the positioning application AP1 may start a process of determining whether the positioning pillow has been detached upon communication with the positioning pillow.

[0207] For example, the positioning application AP1 communicates with a positioning pillow to acquire a pressure value output from a pressure sensor included in the positioning pillow. The positioning application AP1 then sets the pressure value at the time of communication with the positioning pillow as an initial value, and determines that the positioning pillow has become detached when the pressure value output from the pressure sensor of the positioning pillow becomes smaller than the initial value. This makes it possible to appropriately support position adjustment using the positioning pillow. Note that if the frequency of the positioning pillow becoming detached is equal to or greater than a predetermined threshold, the positioning application AP1 may execute a process to recommend a different positioning pillow.

[0208] 3.2 Collaboration based on the perspective of falls Fig. 18 is a diagram illustrating a collaboration process based on the viewpoint of preventing a fall of a person being assisted. As shown in Fig. 18, in order to prevent a fall of a person being assisted, a positioning application AP1, a motion determination device 410, and a peripheral device 500 collaborate with each other. In addition, the operation history and the like of the peripheral device 500 are transmitted to the server system 100, and at that time, the operation results and the like of the communication tag 470, the reader 481, the trash can with a weighing scale 483, etc. may be stored in association with each other.

[0209] In FIG. 18 , the positioning application AP1 and the motion determination device 410 work together, similar to the example of bedsore prevention described above with reference to FIG. 14 and other figures. For example, as described above, the meal application AP2 and the positioning application AP1 work together, and as a result, the positioning application AP1 works together with the motion determination device 410. In this case, since a care recipient who is malnourished is at high risk of falling, the risk of falling can be reduced by working with the motion determination device 410. However, in the method of this embodiment, the meal application AP2 may be omitted. For example, the positioning application AP1 and the motion determination device 410 may work together to prevent falls by a care recipient who is at high risk for reasons other than malnutrition, or to prevent sudden falls by a care recipient who is not determined to be at high risk. Furthermore, although FIG. 18 illustrates an acceleration sensor 411 as the motion determination device 410, as described above, the motion determination device 410 may include other devices.

[0210] <Peripheral devices> 18, the information processing system 10 of this embodiment may further include a peripheral device 500 that is located near the person being assisted and has a movable part. The motion determination device 410 determines whether the person being assisted will fall. When the motion determination device 410 detects a risk of the person being assisted falling (hereinafter referred to as a fall risk), the peripheral device 500 executes at least one of locking the movable part and moving it to a predetermined position.

[0211] The peripheral device 500 here refers to a device used by the person being assisted and placed near the person being assisted in their daily life. In this way, by linking with the peripheral device 500, it becomes possible to prevent the person being assisted from falling, or even if the fall itself cannot be prevented, to mitigate the impact of the fall. Details of the processing will be described below.

[0212] For example, the acceleration sensor 411 may determine the risk of falling by performing the following process. The acceleration sensor 411 calculates the acceleration on the x-axis, y-axis, and z-axis, and the root mean square of the acceleration on the three axes. When a person being assisted wearing the acceleration sensor 411 falls, the magnitude of the acceleration increases to a degree that can be distinguished from that in a normal state due to the impact of the fall. Therefore, the acceleration value in a normal state and the acceleration value when a fall occurs may be calculated in advance, and a threshold value for distinguishing between the two states may be set. The acceleration sensor 411 determines the risk of falling by comparing at least one of the x-axis, y-axis, z-axis, and root mean square with a threshold value.

[0213] Furthermore, before a fall actually occurs, signs of a fall appear, such as tangling of the feet or loss of balance. Therefore, it is possible to obtain in advance the acceleration value under normal conditions and the acceleration value when signs of a fall appear, and set a threshold value to distinguish between these two states. In this way, it is possible to detect signs of a fall, and therefore to detect the risk of a fall before it actually occurs.

[0214] Furthermore, the fall detection process is not limited to this, and for example, a fall detection process using machine learning may be performed. Here, machine learning is, for example, learning using a neural network. Hereinafter, neural network will be abbreviated as NN. However, machine learning is not limited to NN, and other methods such as SVM (support vector machine) and k-means may be used, or methods developed from these may be used. Furthermore, although supervised learning is exemplified below, other machine learning methods such as unsupervised learning may also be used.

[0215] The NN here is, for example, a recurrent neural network (RNN). The RNN may be, for example, a long short-term memory (LSTM). Alternatively, a convolutional neural network (CNN) may be used as the NN.

[0216] For example, the input data of NN includes the sensor information of the acceleration sensor 411. As described above, the sensor information includes the acceleration values of the x-axis, y-axis, and z-axis, and the root mean square of the accelerations of the three axes. However, it is not essential to use all of these four values, and some of them may be omitted. Also, the input information may include information about the location where the care recipient is, such as the bed 610, wheelchair 630, toilet, etc. Further, the input data may be time-series data. For example, when the acceleration sensor 411 makes a measurement once every predetermined time and four acceleration values of x, y, z, and the root mean square are obtained as the measurement result for one time, the input data is a set of N×4 acceleration values acquired by N measurements. N is an integer of 2 or more.

[0217] In this way, by using the input data as time-series data, it becomes possible to perform processing that takes into account the time-series changes in the input data. For example, if the location where a fall occurs is different, the circumstances leading up to the fall and the manner of the fall, etc., may have different time-series behaviors. In that regard, by using an LSTM or the like to process the time-series input data, it becomes possible to reflect the time-series differences in the fall determination process.

[0218] Also, the output data in machine learning is information indicating the likelihood of the presence or absence of the care recipient's fall risk. For example, the output layer of the NN may output a probability value between 0 and 1 as the output data. The larger this value, the higher the probability of a fall risk, that is, it indicates a high fall risk.

[0219] For example, when a threshold Th where 0 < Th < 1 is set in advance, the acceleration sensor 411 may determine that there is a fall risk when the value of the output data is Th or more. Note that the fall determination based on the acceleration value is not limited to being performed by the acceleration sensor 411. For example, the acceleration sensor 411 may be connected to the server system 100 directly or via another device such as the terminal device 200, and the server system 100 may perform the fall determination based on the acceleration value.

[0220] In the above, an example has been described in which a fall of the person being assisted is detected using the acceleration sensor 411. However, the fall detection is not limited to this, and a fall detection may be performed using other motion detection devices 410. For example, a fall in bed 610 (e.g., a fall from the mattress 620) may be detected based on the output of the bedside sensor 420 or the detection device 430. A fall in wheelchair 630 (e.g., a fall from the seat) may also be detected based on the output of the seat sensor 440.

[0221] 19A and 19B are diagrams illustrating a table 530, which is an example of a peripheral device 500. For example, a table having a compact operating mechanism is described in Japanese Patent Application No. 2015 / 229220, filed on November 24, 2015, entitled "Operating Mechanism and Mobile Table Including the Same." This patent application is incorporated herein by reference in its entirety. For example, the table 530 is a mobile table including casters Ca11 to Ca14, which are movable parts. The table 530 is normally in a braked state, and has an off-lock function that releases the brake when an operating lever 531 is operated. The operating lever 531 can be locked with the off-lock function unlocked, and further operation can unlock the lock (turn the off-lock function on).

[0222] 19C and 19D are diagrams illustrating a walker 540, which is an example of peripheral device 500. For example, a walker that is lightweight, stable, and easy to maintain is described in Japanese Patent Application No. 2005 / 192860, filed June 30, 2005, entitled "Walking Aid." This patent application is incorporated herein by reference in its entirety. Walker 540 includes casters Ca21-Ca24, which are movable parts, and is a device that assists the walking of a person receiving assistance. Table 530 has a function of restricting movement by locking at least some of casters Ca11-Ca14.

[0223] Therefore, in this embodiment, when a risk of falling is detected, control may be performed to lock the casters of the peripheral device 500 that is capable of moving on casters. A signal instructing this control may be output by the motion determination device 410 or another device such as the server system 100. The peripheral device 500 that is capable of moving on casters is, for example, a table 530 or a walker 540, but other peripheral devices 500 may also be used.

[0224] For example, as shown in FIGS. 19A and 19B, table 530 includes a pair of operating levers 531 and a fixing member 532 that fixes a drive mechanism to table 530. The drive mechanism here refers to a mechanism that operates to lock a movable part of peripheral device 500. Fixed member 532 has a relatively large main surface 532a, a surface 532b that intersects with main surface 532a and is parallel to the table surface, and a surface 532c that intersects with main surface 532a and is parallel to one surface of the support column. These surfaces are used for fixing fixed member 532 to table 530. As shown in FIG. 19B, fixed member 532 is provided with a solenoid 534 and a board box 533 that houses a board that drives solenoid 534. The board here is, for example, a board on which a processor and memory that control solenoid 534 are mounted. As shown in FIG. 19A, when fixed member 532 is fixed to table 530, solenoid 534 is located below one of the pair of operating levers 531. For example, when the motion determination device 410 outputs a control signal instructing to lock the table 530, the circuit board drives the solenoid 534 based on the control signal. In this way, the operation lever 531 is operated based on the control signal from the motion determination device 410. Therefore, even if the operation lever 531 is fixed in a state that releases the off-lock function, the fixation is released and the table 530 transitions to a state in which the off-lock function is active.

[0225] As shown in Figures 19C and 19D, the walker 540 includes a base frame, a support column attached to the base frame, an adjustable support column attached to the support column for adjustable extension and retraction, and a reclining section attached to the top of the adjustable support column for supporting the user's upper body. The base frame includes a straight horizontal leg pipe 541a, a pair of vertical leg pipes 541b integrally connected at one end near each end of the horizontal leg pipe 541a and expanding at the other end greater than the distance between the two ends, and a base frame member 541c integrally connected between the pair of vertical leg pipes 541b for attaching the support column. The housing 542 includes hook portions 543 and 544, and is held suspended from one of the pair of vertical leg pipes 541b by the hook portions 543 and 544. As shown in Figure 19D, a motor 545 is provided inside the housing, and the motor 545 winds and releases a wire 546. 19C, the wire 546 is connected to a plate-like member of the brake 547. Therefore, when the motor 545 winds up the wire 546, the plate-like member moves upward, and the caster Ca23 is locked.

[0226] This allows the peripheral device 500 to be locked when there is a risk of falling, thereby preventing the occurrence of injury or the like when the person being assisted grabs onto the peripheral device 500 in an instant.

[0227] Furthermore, the peripheral device 500 may execute control to lock the movable part after moving to a predetermined position. The predetermined position here is a position where a person being assisted can easily grab hold of if they are about to fall, for example, a position a predetermined distance away from the current position of the person being assisted. In this way, the peripheral device 500 can be locked at a position where a person being assisted can easily grab hold of, thereby further reducing the risk of injury due to a fall.

[0228] Alternatively, the peripheral device 500 may be a device having a height adjustment function. The peripheral device having a height adjustment function may be, for example, a bed 610. Here, the bed 610 is an adjustable bed with an adjustable bottom height. By adjusting the height, the mattress 620 and the side rails can be adjusted to a height that is easy to grasp, thereby preventing injury when the person being assisted falls onto the mattress 620 or grabs onto the side rails. However, other devices may be used as the peripheral device 500 having a height adjustment function.

[0229] Furthermore, when the above-described control is performed, the motion determination device 410 and the peripheral device 500 according to this embodiment may transmit information relating to the control to the server system 100. This makes it possible to appropriately store information relating to the fall of the person being assisted.

[0230] For example, when the motion determination device 410 detects a risk of falling, it transmits to the server system 100 information such as an index indicating the degree of risk of falling, the expected manner of falling, and the type of peripheral device 500 that instructed to lock. The peripheral device 500 also transmits to the server system 100 information such as identification information and control details of the peripheral device 500. The motion determination device 410 and the peripheral device 500 may also transmit information such as the target person being assisted and the caregiver in charge of the person being assisted. If the position of the peripheral device 500 can be identified, the position of the peripheral device 500 may be used as information regarding a location with a high risk of falling. For example, if the peripheral device 500 has a function to communicate with an access point within a facility, the position of the peripheral device 500 can be estimated based on the communication history of the access point.

[0231] <Communication tags, etc.> As described above, information about the fall situation can be collected through cooperation between the positioning application AP1, the motion determination device 410, and the peripheral device 500. The information processing system 10 of this embodiment may collect other information about the fall by using other devices. For example, as shown in FIG. 18 , the information processing system 10 may include a communication tag 470, and more detailed information may be collected using the communication tag 470.

[0232] Here, the communication tag 470 is, for example, an RFID (radio frequency identifier) ​​tag. For example, an RFID reader 481 is placed at a predetermined location within the facility, and the time-series position of the communication tag 470 is determined based on the reading results of the reader 481. The reader 481 can be placed in various locations, for example, a room, a toilet, a dining room, etc. Since an ID is assigned to each communication tag 470, the position of the person being assisted can be tracked by associating the ID with the person being assisted. The information processing system 10 associates the reading results of the communication tag 470 by the reader 481 with the fall determination result by the motion determination device 410 and stores them in the memory unit 120 or the like.

[0233] The communication tag 470 may also be a device that can be attached to a diaper. For example, as shown in FIG. 18 , the information processing system 10 according to this embodiment may include a trash can with a scale 483 that has a function of measuring the weight of discarded trash. By installing a reader 481 in the trash can with a scale 483, the weight of the diaper discarded in the trash can with a scale 483 can be measured, and the diaper can be easily identified as belonging to the person receiving care. Therefore, it is easy to record whether the person receiving care has excreted or not and the weight of the excretion. In other words, the communication tag 470 may be used to record information related to the excretion of the person receiving care, in addition to tracking the location. In this case, the reader 481 may be provided separately from the trash can with a scale 483, or may be built into the trash can with a scale 483.

[0234] Furthermore, although the above description has been given assuming an example in which the communication tag 470 is always capable of communication, the communication tag 470 of the present embodiment is not limited to this. For example, the communication tag 470 of the present embodiment may be attached to the clothing of the person being assisted, be in a communication-disabled state when the clothing is properly worn, and be in a communication-enabled state when the person being assisted moves the clothing. In this case as well, the information processing system 10 has a storage unit, and stores information in the storage unit that associates the result of reading the communication tag 470 by the reader 481 with the result of the fall determination by the motion determination device 410. The storage unit here is, for example, the storage unit 120 of the server system 100, but the storage unit 220 of the terminal device 200, etc. may also be used. Furthermore, as described above, the result of reading the communication tag 470 by the reader 481 may be stored in association with information representing the control result of the peripheral device 500.

[0235] 20A to 20C are diagrams showing an example of the configuration of a communication tag 470. For example, the communication tag 470 includes a tag main body and two clip portions CL1 and CL2 provided on both ends of the tag main body. The tag main body includes a first tag portion PA1 having a coil provided therein, a second tag portion PA2 having a communication antenna provided therein, and a third tag portion PA3 having a coil provided therein. The clip portions CL1 and CL2 are provided, for example, on the ends of the first tag portion PA1 and the third tag portion PA3. Furthermore, a metal member MT is provided on the surface of the third tag portion PA3.

[0236] 20A, the first tag portion PA1, the second tag portion PA2, and the third tag portion PA3 are each part of the tag body and are arranged in this order along a predetermined direction. Bendable folding portions are provided between the first tag portion PA1 and the second tag portion PA2, and between the second tag portion PA2 and the third tag portion PA3. By folding the two folding portions in different directions, the communication tag 470 is folded so that the second tag portion PA2 is sandwiched between the first tag portion PA1 and the third tag portion PA3.

[0237] 20B is a view of the communication tag 470 in a folded state observed from the front, and FIG. 20C is a view of the communication tag 470 in a folded state observed from the side (for example, from above in FIG. 20B). As shown in FIGS. 20B and 20C, in the folded state, the metal member MT provided on the surface of the third tag portion PA3 is arranged so as to overlap the antenna provided on the second tag portion PA2. In this state, the antenna is shielded by the metal member MT, so that the communication tag 470 cannot be read by the reader 481 even if it enters the communication range of the reader 481. On the other hand, when the communication tag 470 is in an unfolded state as shown in FIG. 20A, the antenna is not shielded by the metal member MT, so the communication tag 470 can communicate with the reader 481.

[0238] 21A to 21C are diagrams illustrating communication tag 470 attached to clothing. FIG. 21A is a diagram of communication tag 470 attached to clothing observed from the front side of the clothing, and FIG. 21B is a diagram of communication tag 470 attached to clothing observed from the back side of the clothing. As shown in FIGS. 21A and 21B, communication tag 470 is attached to clothing by clip portions CL1 and CL2 so that the tag main body (first tag portion PA1 to third tag portion PA3) is inside the clothing. FIG. 21C is a diagram of attached communication tag 470 observed from above.

[0239] As shown in FIGS. 21B and 21C, when no force is applied to the communication tag 470, the communication tag 470 is configured to open naturally. For example, in the example of FIG. 21C, the antenna inside the second tag portion PA2 and the metal member MT on the surface of the third tag portion PA3 do not overlap, so communication is possible. On the other hand, when the garment is worn properly by the person being assisted, the wearing portion (for example, the waist portion of the garment) is in close contact with the skin or underwear of the person being assisted. In this case, the communication tag 470 is pressed against the garment by the body of the person being assisted, so it is in a folded state as described above with reference to FIGS. 20B and 20C.

[0240] As described above, the communication tag 470 of this embodiment is incapable of communication when the person being assisted is wearing the clothing properly, but is capable of communication when the clothing is being moved relative to the normal state. The state of moving the clothing includes a state in which the person being assisted puts their hand inside the clothing and stretches the elastic cord to a certain extent.

[0241] For example, if a person being assisted feels the need to defecate, they may reach into their clothing. Also, if the person has already defecate in a diaper or the like, they may reach into their clothing to reach for the feces. Therefore, by using the communication tag 470 of this embodiment, it is possible to determine whether or not the person being assisted is taking any action related to the need to defecate, and if so, the location where such action occurred. In other words, rather than simply tracking the location, it is possible to detect the location where a specific situation occurred.

[0242] In particular, a person receiving care who suffers from dementia may defecate in a place other than the toilet. In this regard, by using the communication tag 470 shown in FIGS. 20A to 21C, it is possible to obtain information on whether or not the person receiving care defecates inappropriately and the location where the inappropriate defecation occurs. If the location where the person receiving care defecates is identified to some extent, a reader 481 may be provided in that location. In the method of this embodiment, a normal communication tag (for example, a communication tag capable of continuous communication) and the communication tag 470 shown in FIGS. 20A to 21C may be used in combination. In this way, it is possible to continuously track the location using the normal tag, while detecting the timing and location of defecation, etc., using the communication tag 470.

[0243] Furthermore, by accumulating the reading results of the reader 481, it is possible to infer the reason why the person being assisted touches their clothes. For example, if the communication tag 470 is read many times by the reader 481 near the bed 610, it is considered to be a sudden skin discomfort. In this case, a moisturizer may be prescribed or a purchase recommendation may be made. Also, if the communication tag 470 is read many times by the reader 481 placed in an unexpected place (a place other than the toilet, which may be a place specific to the person being assisted), it is possible that the person is excreting inappropriately, and it can be inferred that this is restless behavior due to dementia, for example.

[0244] 20A to 21C illustrate an example of communication tag 470 fixed by clip portions CL1 and CL2, but the embodiment of communication tag 470 is not limited to this. For example, a stretch belt with flexibility may be used instead of clip portions CL1 and CL2. In this case, the person being assisted wears communication tag 470 by wearing the stretch belt around their waist. In this case, too, if the stretch belt is properly fastened, communication tag 470 is folded and becomes incommunicable, but if the stretch belt is stretched by putting a hand inside the clothing, communication tag 470 opens and becomes communicable.

[0245] For example, the server system 100 may acquire and present information about places that require attention when assisting a person being assisted, based on information from the motion determination device 410 and the peripheral device 500, and information from the reader 481.

[0246] FIG. 22 is an example of a screen presenting a place requiring attention. The image here may be, for example, an image in which an object OB12 representing the place requiring attention is superimposed on a map MP of a care facility or the like. The object OB12 is displayed in a manner that makes it more visible than other areas of the map. As described above, the place requiring attention may be a place where the motion determination device 410 detects a risk of falling, a location where the peripheral device 500 is locked, or a destination of the peripheral device 500. The place requiring attention may also be a place where the person being assisted excretes, such as a location where the communication tag 470 corresponding to FIGS. 20A to 21C is read. By presenting the screen shown in FIG. 22 to the caregiver, the caregiver can estimate the level of risk depending on whether the person being assisted is present in that location. As a result, the caregiver can change how they respond to the person being assisted depending on the location, thereby enabling them to smoothly assist the person being assisted. Note that, although one screen is shown in FIG. 22, the object OB12 representing the place requiring attention may be changed depending on the time of day. The time periods here may be classified as morning, afternoon, evening, night, late night, etc., or may be more specific time periods. In this way, it is possible to present information such as, for example, that a certain location is at high risk in the evening. As a result, it is possible to contribute to optimizing the allocation of personnel in care facilities, etc. For example, in the above example, it is possible to encourage measures such as increasing the number of personnel allocated around high-risk locations in the evening.

[0247] As shown in FIGS. 15 to 17B, the meal application AP2 can detect the type and amount of food ingested by the person being assisted, as well as the timing of the meal. Furthermore, by using the communication tag 470, the timing, location, and weight of excretion, etc., can be detected, as described above. In other words, the information processing system 10 of this embodiment can monitor the entire process from eating to excretion, allowing the caregiver to provide more appropriate assistance. For example, the server system 100 may predict whether or not an excretion will occur and the amount of excretion by comparing the amount of food eaten with the amount of excretion. The server system 100 may also estimate the next excretion timing based on the history of meal timing and excretion timing. Furthermore, if no excretion occurs even though the excretion timing has passed, the server system 100 may prompt the caregiver to take action, such as requesting a laxative prescription.

[0248] 3.3 Cooperation based on the perspective of unrest 23 is a diagram illustrating a process of cooperation between the positioning application AP1 and other applications and devices based on the viewpoint of agitated behavior of the person being assisted. As shown in FIG. 23, in order to reduce the risk of agitated behavior of the person being assisted, the agitation application AP3, the positioning application AP1, and the peripheral device 500 may cooperate with each other. As described above, the agitation application AP3 is software that detects agitated behavior of the person being assisted, and may run on the server system 100 or on other devices such as the terminal device 200 or the management terminal device 300. When the agitation application AP3 detects agitated behavior of the person being assisted, the positioning application AP1 may execute a process of supporting the placement of objects located around the person being assisted.

[0249] In this context, agitated behavior may be included in the peripheral symptoms of dementia in a narrow sense. For example, peripheral symptoms include agitated behavior and delirium. Delirium is a disorder of mental function and is accompanied by a decline in attention and thinking ability. Peripheral symptoms include mental symptoms and behavioral symptoms, and are also called BPSD (Behavioral and Psychological Symptoms of Dementia). However, agitated behavior in this embodiment broadly includes behavior that differs from the normal state and is not limited to behavior caused by dementia.

[0250] One of the possible causes of agitated behavior is environmental factors resulting from the surrounding environment of the person receiving care. For example, the surrounding environment of the person receiving care includes the state of objects placed around the person receiving care. More specifically, the location of furniture, home appliances, decorations, small items, etc. in the person receiving care's room is a factor that determines the surrounding environment. For example, for a person receiving care who suffers from dementia, changes in the placement of surrounding objects can be a major source of stress, and such changes can be a cause of agitated behavior.

[0251] However, since the type of object placement that is suitable for suppressing agitated behavior differs depending on the person being assisted, it is a heavy burden for the caregiver to memorize the placement of objects for each person being assisted. In this regard, by executing a process in the positioning application AP1 that supports the placement of objects when agitated behavior is detected, it becomes possible for the caregiver to appropriately place objects that are suitable for the person being assisted.

[0252] A system for assessing agitation risk based on biometric information acquired from a sensor worn by a user is described, for example, in PCT / JP2018 / 037384 (International Publication No. 2019 / 073927) entitled "Biometric Information Processing System, Biometric Information Processing Method, and Biometric Information Processing Program Storage Medium," filed on October 5, 2018. This patent application is incorporated herein by reference in its entirety.

[0253] For example, in this embodiment, the agitation application AP3 is connected to the acceleration sensor 411 and the detection device 430, and detects agitated behavior based on the acceleration value output by the acceleration sensor 411 and the biological information output by the detection device 430. Furthermore, other sensors included in the sensing device 400, such as the bedside sensor 420 and the seat sensor 440, may be used in the agitated behavior detection process.

[0254] When agitated behavior is detected, the positioning application AP1 may perform a process of comparing the current position of the peripheral device 500 with a reference position of the peripheral device 500 as a process of supporting the placement of an object. The reference position here represents the position of the peripheral device 500 in the training data of the positioning application AP1. When the positioning application AP1 determines that the current position of the peripheral device 500 is different from the reference position, the peripheral device 500 of this embodiment may move (self-propel) to the reference position.

[0255] In this way, the peripheral device 500 can be automatically adjusted to a position suitable for the person being assisted, thereby reducing the burden on the caregiver who suppresses agitated behavior. The positioning application AP1 may retain information specifying the specific position of the peripheral device 500 as training data and transmit the information to the peripheral device 500. For example, the training data may be coordinate values ​​in a predetermined coordinate space or numerical values ​​specifying an angle or height. Alternatively, the training data may be image information, and the positioning application AP1 may perform control to move the peripheral device 500 to a reference position based on the position of the peripheral device 500 on an image in the training data and the position of the peripheral device 500 on an image that has actually been captured. In this case, feedback control may be performed, such as the positioning application AP1 determining the position of the peripheral device 500 again after the peripheral device 500 has changed its position and transmitting a control signal for readjustment.

[0256] Furthermore, when agitated behavior is detected by the agitation application AP3, the assisted person is in a state different from their normal state, which may make them more susceptible to falling or colliding with objects. Therefore, when agitated behavior is detected, the peripheral device 500 disposed near the assisted person may perform control to lock the moving parts. In this way, it is possible to ensure the safety of the assisted person who is exhibiting agitated behavior. For example, when agitated behavior is detected, priority may be given to locking the peripheral device 500, and when it is determined that the agitated behavior has subsided, the peripheral device 500 may be moved to the reference position described above.

[0257] As described above, the positioning application AP1 may store setting information for the bed 610 and the wheelchair 630, and may have a function for adjusting the height and bottom angle of the bed 610 and the angle of the backrest of the wheelchair 630 based on the setting information. Therefore, the positioning application AP1 may control the bed 610 and the wheelchair 630 to a state suitable for the person being assisted by turning on this function in cooperation with the agitation application AP3.

[0258] 23, the processing result of the agitation application AP3 may be transmitted to the server system 100. In this way, it becomes possible to accumulate log data of agitation behavior of the person being assisted in the server system 100. For example, the server system 100 may determine the degree of progression of dementia by chronologically storing the history of occurrence of agitation behavior, or may determine whether the frequency of agitation behavior has decreased by controlling the peripheral device 500 described above, or the like.

[0259] 3.4 Specific operation flow example Next, a specific processing flow of the information processing system 10 will be described using Figures 24 and 25. Note that Figures 24 and 25 are diagrams illustrating a part of the cooperative processing, and processing not shown may also be executed. Furthermore, in Figures 24 and 25, when a given application or device communicates with another application or device, the communication may be performed directly or via another device such as the server system 100, and various modifications of the communication mode are possible.

[0260] 24 is a diagram illustrating the coordination process based on the above-described viewpoints of bedsores and falls. When this process starts, first, in step S101, the meal application AP2 accepts a pre-meal image based on the screen described above with reference to FIGS. 16A and 16B. Similarly, in step S102, the meal application AP2 accepts a post-meal image based on the screen described above with reference to FIGS. 16A and 16B.

[0261] In step S103, the meal application AP2 determines whether the amount of food eaten is insufficient and whether there is nutrient deficiency based on a comparison process between the pre-meal image and the post-meal image. Specifically, as described above with reference to Figures 17A and 17B, the meal application AP2 performs processes such as object detection, circular area setting, pixel value distribution calculation, and comparison of the pre-meal and post-meal distributions.

[0262] When meal application AP2 detects an insufficient amount of food or insufficient nutrition, the following processing of steps S104 to S112 is executed. First, in step S104, meal application AP2 outputs information to cause positioning application AP1 to execute cooperative processing. This information may be, for example, a control signal that activates inactive positioning application AP1, or a control signal that instructs a change in the operation mode of positioning application AP1. A change in operation mode may be, for example, the addition of a function to be used. Note that when the operation mode is changed (such as the addition of a function), a separate trigger for operating positioning application AP1 may be provided. For example, the trigger here may be the detection of a person being present in bed 610 or a person sitting in wheelchair 630. In this case, when the trigger is received, the processing of steps S105 and subsequent steps is executed.

[0263] The positioning application AP1 starts the collaboration process starting from the process of step S104. For example, in step S105, the positioning application AP1 transitions to an active state and performs a process of selecting teacher data. As described above, the teacher data may be selected based on the selection operation of the caregiver, or may be automatically selected based on the attributes of the person being assisted.

[0264] In step S106, the positioning application AP1 executes processing based on the training data. For example, the positioning application AP1 performs transparency processing on the image information, which is the training data, and then superimposes the image on the captured image. However, in step S106, processing may be executed to determine whether the position is OK or NG.

[0265] In step S107, the positioning application AP1 outputs information for causing the motion determination device 410 to execute a cooperative process. This information may be, for example, a control signal for activating an inactive motion determination device 410. If the motion determination device 410 is capable of executing multiple processes, a control signal for activating an inactive process may be output.

[0266] In step S108, the motion determination device 410 performs a process of detecting the movement of the person being assisted based on the output of the sensor included in the motion determination device 410.

[0267] If a specific movement is detected in the process of step S108, the motion determination device 410 outputs information instructing the positioning application AP1 to change its movement in step S109. The specific movement here is, for example, a decrease in the frequency of turning over in bed 610 or a concentration of pressure on a specific part.

[0268] In step S110, the positioning application AP1 executes a cooperative process based on the processing result of the motion determination device 410. Specifically, the positioning application AP1 operates in an operation mode different from that before receiving the information of step S109. For example, in step S110, the positioning application AP1 performs a process to change the teacher data. Alternatively, the positioning application AP1 may change its operation mode from not performing an OK / NG determination before step S109 to performing an OK / NG determination after step S110.

[0269] Furthermore, if a risk of the person being assisted falling is detected in the process of step S108, in step S111, the motion determination device 410 may output an operation instruction to the peripheral device 500. The operation instruction here is an instruction to lock or move (self-propel).

[0270] In step S112, the peripheral device 500 locks the movable part or moves it to a predetermined position based on the processing result of the motion determination device 410.

[0271] By performing the above process, if the risk of bedsores increases due to eating, it becomes possible to reduce the risk of bedsores using the positioning application AP1. Furthermore, by linking the positioning application AP1 with the motion determination device 410 and the peripheral device 500, it becomes possible to appropriately respond to the risk of falls.

[0272] 25 is a diagram illustrating the collaboration process based on the viewpoint of the above-mentioned agitated behavior. When this process starts, first, in step S201, the agitation application AP3 acquires sensor information obtained by sensing the person being assisted. For example, as shown in FIG. 23, the agitation application AP3 may acquire output from the acceleration sensor 411 or the detection device 430.

[0273] In step S202, the agitation application AP3 determines whether the person being assisted is exhibiting agitated behavior. As described above, a wide range of known methods can be applied to the specific processing.

[0274] When it is determined that the instability application AP3 has detected instability behavior, the following steps S203 to S207 are executed. First, in step S203, the instability application AP3 outputs information for causing the positioning application AP1 to execute a cooperative process. This information may be, for example, a control signal for activating the inactive positioning application AP1, or a control signal for instructing a change in the operation mode of the positioning application AP1.

[0275] The positioning application AP1 starts the collaboration process starting from the process of step S203. Specifically, in step S204, the positioning application AP1 selects training data. In step S205, the positioning application AP1 executes processing based on the training data. As described above, this processing may include a process of determining whether the position of an object placed in the living environment of the person being assisted is close to a reference position.

[0276] If the position of the object differs from the reference position by a predetermined amount or more in the process of step S205, the positioning application AP1 outputs information instructing an operation to the peripheral device 500 in step S206. The operation instruction here may be an instruction to move to the reference position.

[0277] In step S207, the peripheral device 500 executes a cooperative process for moving based on the processing result of the positioning application AP1. Specifically, the peripheral device 500 moves (self-propels) to a reference position determined based on the training data of the positioning application AP1.

[0278] By the above processing, when agitated behavior is detected, it becomes possible to automatically execute control to eliminate the environmental factors.

[0279] Although the present embodiment has been described in detail above, those skilled in the art will readily understand that many modifications are possible without substantially departing from the novel features and advantages of the present embodiment. Therefore, all such modifications are intended to be included within the scope of the present disclosure. For example, a term described at least once in the specification or drawings with a different term having a broader or equivalent meaning may be replaced with that different term anywhere in the specification or drawings. Furthermore, all combinations of the present embodiment and modifications are also intended to be included within the scope of the present disclosure. Furthermore, the configurations and operations of the information processing system, terminal device, server system, sensing device, etc. are not limited to those described in the present embodiment, and various modifications are possible.

[0280] [Additional Notes] One aspect of the present disclosure is an information processing system that includes a first processing unit that operates according to a positioning application that performs processing regarding the position of at least one of a person and an object during assistance, and a second processing unit that operates according to a meal application that determines the amount of food ingested by a person being assisted during meal assistance for the person being assisted, wherein the positioning application changes the operating mode of the positioning application from a first mode to a second mode in which the processing load of the positioning application is higher than that of the first mode when the meal application determines that the amount of food ingested by the person being assisted and / or nutrition is insufficient. Related to information processing systems.

[0281] Another aspect of the present disclosure relates to an information processing method including the steps of: performing processing related to the position of at least one of a person and an object during assistance according to a positioning application; performing processing to determine the amount of food ingested by a person receiving assistance during meal assistance according to a meal application; and, when the meal application determines that the amount of food ingested by the person receiving assistance and / or nutrition is insufficient, transitioning the operating mode of the positioning application from a first mode to a second mode in which the processing load of the positioning application is higher than that of the first mode. [Explanation of symbols]

[0282] 10...information processing system, 100...server system, 110...processing unit, 120...storage unit, 130...communication unit, 200, 200-1, 200-2...terminal device, 210...processing unit, 220...storage unit, 230...communication unit, 240...display unit, 250...operation unit, 260...imaging unit, 300...management terminal device, 400...sensing device, 410...motion determination device, 411...acceleration sensor, 413...Hand switch, 420...Bedside sensor, 430...Detection device, 440...Seat sensor, 441...Cushion, 442...Control box, 460...Choking detection device, 461...Throat microphone, 462...Terminal device, 470...Communication tag, 481...Reader, 483...Trash can with weighing scale, 500...Peripheral equipment, 530...Table, 531...Operation lever, 532...Fixing member, 5 32a to 532c...surface, 533...circuit board box, 534...solenoid, 540...walker, 541a...horizontal leg pipe, 541b...vertical leg pipe, 541c...base frame member, 542...housing, 543, 544...hook portion, 545...motor, 546...wire, 547...brake, 610...bed, 620...mattress, 630...wheelchair, AP1...positioning application, AP2...eating application, AP3...restlessness application, Ca11 to Ca14, Ca21 to Ca24...casters, CL1, CL2...clip portion, DP...display, OB1 to OB12...object, PA1...first tag portion, PA2...second tag portion, PA3...third tag portion, R1 to R3...rectangular area, C1, C2...circular shape, RE1 to RE3...area, Se1 to Se4...pressure sensor

Claims

1. a first processing unit that operates according to a first application that can detect agitated behavior of the person being assisted; a second processing unit that operates according to a second application that performs processing related to the position of at least one of a person and an object during assistance; Including, When the first processing unit detects agitated behavior of the person being assisted using the first application, the second processing unit performs processing using the second application. Information processing system.

2. 2. The information processing system according to claim 1, wherein when the second application determines that the current position of the peripheral device is different from a reference position, the second application controls the peripheral device to move the position of the peripheral device to the reference position.

3. The information processing system according to claim 2 , wherein the first application acquires information from a communication tag that can be attached to the person being assisted and detects agitated behavior of the person being assisted.

Images