Information processing system, information processing device, and information processing method

Abstract

To provide an information processing system, an information processing apparatus, and an information processing method for appropriately supporting a caregiver's assistance for a person to be cared.SOLUTION: An information processing system includes: a device that operates in any of a plurality of operation modes and is used for assistance for a person to be cared; and a server system that is connected with the device via a network. The server system determines ability information representing the activity ability of the person to be cared based on sensing data transmitted from the device, and transmits the determined ability information to the device. The device determines in which of the plurality of operation modes it operates based on the ability information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing device, an information processing method, and the like. [Background technology]

[0002] A system for use when a caregiver provides care to a care recipient has been known. Patent Document 1 discloses a method for generating information to be provided about the condition of a resident in a living space based on time-varying changes in detected information acquired by a sensor placed in the living space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-18760 Summary of the Invention [Problem to be solved by the invention]

[0004] An information processing system, an information processing device, an information processing method, etc. are provided that appropriately support a caregiver in providing assistance to a person being assisted. [Means for solving the problem]

[0005] One aspect of the present disclosure includes a device that operates in one of a plurality of operating modes and is used to assist a person being assisted, and a server system connected to the device via a network, wherein the server system determines ability information representing the activity ability of the person being assisted based on sensing data transmitted from the device, transmits the determined ability information to the device, and the device is related to an information processing system that determines in which of the plurality of operating modes the device will operate based on the ability information.

[0006] Another aspect of the present disclosure relates to an information processing device that includes a communication unit that operates in one of a plurality of operating modes and communicates with a device used to assist a person being assisted, and a processing unit that performs processing to determine ability information representing the activity ability of the person being assisted based on sensing data transmitted from the device, wherein the processing unit transmits the ability information to the device via the communication unit as information that determines in which of the plurality of operating modes the device will operate.

[0007] Yet another aspect of the present disclosure relates to an information processing method in an information processing system including a device that operates in one of a plurality of operating modes and is used to assist a person being assisted, and a server system connected to the device via a network, the information processing method determining ability information that represents the activity ability of the person being assisted based on sensing data acquired by the device, and determining in which of the plurality of operating modes the device will operate based on the determined ability information. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example of the configuration of an information processing system. [Figure 2] FIG. 10 is a diagram illustrating an example of a relationship between devices and tacit knowledge. [Figure 3] FIG. 1 illustrates an example of the configuration of a server system. [Figure 4] FIG. 2 is a diagram illustrating an example of the configuration of a device. [Figure 5] FIG. 10 is a diagram showing an example of the relationship between the ability of a person being assisted and expected risks. [Figure 6] FIG. 2 is a sequence diagram illustrating processing of the information processing system. [Figure 7] FIG. 2 is a sequence diagram illustrating processing of the information processing system. [Figure 8] FIG. 10 shows a specific example of a device related to fall risk. [Figure 9] FIG. 10 shows a specific example of a device related to fall risk. [Figure 10]FIG. 10 shows a specific example of a device related to fall risk. [Figure 11] FIG. 10 shows a specific example of a device related to fall risk. [Figure 12] FIG. 10 shows a specific example of a device related to aspiration risk. [Figure 13] FIG. 1 shows a specific example of a device related to pressure ulcer risk. [Figure 14] FIG. 1 shows a specific example of a device related to pressure ulcer risk. [Figure 15A] 10 is an example of a screen used for adjusting the bed position. [Figure 15B] 10 is an example of a screen used for adjusting the bed position. [Figure 16] This is an example of a screen used in end-of-life care. [Figure 17] FIG. 10 is a diagram illustrating an example of operation modes of a device according to its capabilities. [Figure 18] FIG. 2 is a sequence diagram illustrating processing of the information processing system. [Figure 19] 10 is a flowchart illustrating a process for determining an operation mode in a device. [Figure 20] 10 is a flowchart illustrating a process for determining an operation mode in a device. [Figure 21] FIG. 10 is a diagram illustrating an example of cooperation between the choking hazard detection device and another device. [Figure 22] 10 is a flowchart illustrating a process for determining an operation mode in a device. [Figure 23] FIG. 2 is a diagram illustrating a specific example of a device to be controlled. [Figure 24] FIG. 2 is a diagram illustrating a specific example of a device to be controlled. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, the present embodiment will be described with reference to the drawings. In the drawings, identical or equivalent elements are designated by the same reference numerals, and duplicate explanations will be omitted. Note that the present embodiment described below does not unduly limit the content described in the claims. Furthermore, not all of the configurations described in the present embodiment are necessarily essential components of the present disclosure.

[0010] 1. System configuration example 1 shows an example of the configuration of an information processing system 10 according to this embodiment. The information processing system 10 according to this embodiment is for providing instructions to caregivers so that they can provide appropriate care regardless of their level of skill by digitizing the "intuition" and "tacit knowledge" of caregivers who perform tasks that are normally performed by the "intuition" and "tacit knowledge" of caregivers in, for example, medical facilities or nursing care facilities.

[0011] The caregiver here may be a caregiver at a nursing facility, or a nurse or licensed practical nurse at a medical facility such as a hospital. That is, assistance in this embodiment includes various actions to support the person being assisted, and may include nursing care or medical actions such as injections. The person being assisted here is a person receiving assistance from the caregiver, and may be a resident of a nursing facility or a patient who is hospitalized or visiting a hospital.

[0012] In addition, the assistance in this embodiment may be provided at home. For example, the person being assisted in this embodiment may be a person requiring care who receives home care, or a patient receiving home medical care. The caregiver may be a family member of the person requiring care or the patient, or a visiting helper, etc.

[0013] The information processing system 10 shown in FIG. 1 includes a server system 100, devices 200, and a gateway 300. However, the configuration of the information processing system 10 is not limited to that shown in FIG. 1 and various modifications are possible, such as omitting some components or adding other components. For example, FIG. 1 illustrates devices 200 as examples of a tablet-type terminal device such as a smartphone, a seat sensor 440 (described later using FIG. 10) placed on a wheelchair 630, and a detection device 430 (described later using FIG. 9) placed on a bed 610, but the number and types of devices 200 are not limited thereto. For example, the information processing system 10 may include various devices 200, which will be described later using FIG. 8 to FIG. 14. Furthermore, the information processing system 10 may include devices other than the devices 200 shown in FIG. 8 to FIG. 14. Note that, hereinafter, when it is not necessary to distinguish between multiple devices 200, they will be simply referred to as devices 200. Furthermore, modifications such as omissions and additions of components are possible, as in FIG. 3, FIG. 4, and so forth, which will be described later.

[0014] The information processing device of this embodiment corresponds to, for example, the server system 100. However, the method of this embodiment is not limited to this, and the processing of the information processing device may be executed by distributed processing using the server system 100 and other devices. For example, the information processing device of this embodiment may include the server system 100 and the device 200. An example in which the information processing device is the server system 100 will be described below.

[0015] The server system 100 is connected to the device 200 via a network, for example. For example, the server system 100 is connected to the gateway 300 via a public communication network such as the Internet, and the gateway 300 is connected to the device 200 using a LAN (Local Area Network) or the like. For example, the gateway 300 may be an access point (AP) that performs communication in accordance with the IEEE802.11 standard, and the device 200 may be a station (STA) that performs communication in accordance with the IEEE802.11 standard. However, various modifications can be made to the communication method between the devices.

[0016] The server system 100 may be one server, or may include multiple servers. For example, the server system 100 may include a database server and an application server. The database server stores various data, which will be described later with reference to FIG. 3. The application server performs processing, which will be described later with reference to FIGS. 6 to 7, etc. The multiple servers here may be physical servers or virtual servers. If a virtual server is used, the virtual server may be provided on one physical server, or may be distributed across multiple physical servers. As described above, the specific configuration of the server system 100 in this embodiment can be modified in various ways.

[0017] The device 200 may have, for example, various sensors and perform processing based on data sensed by the sensors (hereinafter referred to as sensing data). The digitization of the expert's tacit knowledge described above may be performed, for example, by the vendor of the device 200. For example, suppose an expert has the tacit knowledge to determine whether a person being assisted is moving forward or to the side based on the posture of the person being assisted in a wheelchair. In this case, the tacit knowledge can be digitized by collecting sensing data corresponding to the posture of the person being assisted detected by the seat sensor 440 and creating an application that determines whether a person is moving forward or to the side based on the sensing data. For example, the vendor provides the seat sensor 440 and the application. As a result, an unskilled person (e.g., a new employee) can utilize the same level of ability as an expert to determine whether a person is moving forward or to the side.

[0018] Furthermore, the amount of tacit knowledge digitized in one device 200 is not limited to one. For example, the tacit knowledge digitized using the seat sensor 440 is not limited to the determination of forward or lateral slippage, and may include a determination of the possibility of a fall or a combination of both the determination of forward or lateral slippage and the possibility of a fall. The determination of forward or lateral slippage corresponds to determining whether the assisted person's posture is good or bad, and the determination of the possibility of a fall corresponds to determining whether the assisted person has slipped off the seat. Furthermore, in the determination of forward or lateral slippage, multiple tacit knowledge may exist that differ in the determination criteria for determining the degree of slippage as a forward or lateral slippage, the determination process contents, etc. Therefore, each device 200 may be capable of executing processes corresponding to one or more pieces of tacit knowledge and may switch between executing and not executing processes corresponding to each piece of tacit knowledge. For example, a vendor may implement tacit knowledge as application software (hereinafter simply referred to as an application) and register it in the server system 100. Each device 200 executes processes corresponding to the tacit knowledge by downloading and installing registered applications that are executable.

[0019] FIG. 2 is a diagram showing an example of the relationship between devices 200 and tacit knowledge (applications). FIG. 2 illustrates five devices 200a to 200e as devices 200 connected to the server system 100. In the example of FIG. 2, two tacit knowledges, tacit knowledge 1 and tacit knowledge 2, are associated with device 200a. For example, the applications of tacit knowledge 1 and tacit knowledge 2 are already installed in device 200a. Device 200a may be, for example, a device related to the risk of falling, which will be described later. Specifically, device 200a may be a seat sensor 440, and tacit knowledge 1 may be tacit knowledge related to determining whether a device is slipping forward or sideways, and tacit knowledge 2 may be tacit knowledge related to determining the possibility of a fall. Processing results corresponding to tacit knowledge 1 and tacit knowledge 2 are transmitted to, for example, the server system 100.

[0020] Devices 200b and 200c are, for example, devices related to aspiration risk, which will be described later, and each is associated with multiple pieces of tacit knowledge for dealing with aspiration risk, etc. Devices 200d and 200e are, for example, devices related to pressure ulcer risk, which will be described later, and each is associated with multiple pieces of tacit knowledge for dealing with pressure ulcer risk, etc. In this way, it is possible to digitize a variety of tacit knowledge using various devices. Note that, although two pieces of tacit knowledge are illustrated for one device 200 here, the number of tacit knowledge associated with one device 200 is not limited to this.

[0021] The method of this embodiment makes it possible to switch the tacit knowledge to be used depending on the situation. For example, it is not necessary to use all of tacit knowledge 1 to tacit knowledge 10 shown in FIG. 2 , and use / non-use can be switched as needed. Switching of tacit knowledge may be achieved by switching the device 200 to be used. For example, when the target is a care recipient with a high risk of falling but a low risk of aspiration and bedsores, device 200a is used, and devices 200b-200e are not used. In this way, at least one of tacit knowledge 1 and tacit knowledge 2 is used, and other tacit knowledge is not used, making it possible to appropriately use tacit knowledge that is highly necessary for the care recipient. Furthermore, when the risk of aspiration of the care recipient increases, the tacit knowledge to be used can be switched by using device 200b or device 200c. Furthermore, when addressing the risk of aspiration, switching may be performed between using only device 200b, only device 200c, or both devices 200b and 200c. In this way, for example, when the risk of aspiration is high, it is possible to use the necessary tacit knowledge from tacit knowledge 3 to tacit knowledge 6 in Fig. 2. The same applies when the risk of bedsores increases; by using device 200d or device 200e, it is possible to switch the tacit knowledge to be used. Also, when the risk of falling decreases, it is possible to switch, such as stopping the use of device 200a.

[0022] Furthermore, switching of tacit knowledge may be realized by switching the tacit knowledge used within the device 200 while maintaining the device 200 in use. For example, in the device 200a, switching may be made between a case where only tacit knowledge 1 is used, a case where only tacit knowledge 2 is used, and a case where both tacit knowledge 1 and tacit knowledge 2 are used. This processing can be realized, for example, by controlling the activation / inactivation of applications corresponding to each tacit knowledge.

[0023] The switching of tacit knowledge may be performed using ability information of the person being assisted. The ability information here is information that indicates the activity ability of the person being assisted, and is, for example, information obtained as a result of the device 200 performing some kind of processing using tacit knowledge. The ability information may be, for example, information related to the level of risk of falling, risk of aspiration, or risk of bedsore. Details of the ability information will be described later.

[0024] For example, when the processing result of tacit knowledge is obtained in a certain device 200, a process of switching the tacit knowledge to be used may be performed in the same device 200. For example, based on the processing result of tacit knowledge 1 in device 200a in Fig. 2, the activation / inactivation of an application corresponding to tacit knowledge 1 and an application corresponding to tacit knowledge 2 may be switched.

[0025] Furthermore, when a processing result of tacit knowledge is obtained in one device 200, the processing result may affect another device 200. For example, based on the processing result of tacit knowledge 1 in device 200a, the tacit knowledge used in device 200b may be switched. For example, a switch may be made from a state in which device 200b is not in use to a state in which device 200b is in use. Alternatively, tacit knowledge 3 and tacit knowledge 4 associated with device 200b may be individually switched between active and inactive.

[0026] Furthermore, the switching of tacit knowledge is not limited to being based on the ability information, but may be performed based on the assistance situation of the person being assisted, the combined use of multiple devices 200, etc. The information used for switching tacit knowledge and the process of switching tacit knowledge will be described in detail later.

[0027] The correspondence between the devices 200 and the tacit knowledge can be flexibly changed. For example, processing corresponding to one piece of tacit knowledge may be executed based on sensing data acquired using multiple devices 200. For example, processing corresponding to tacit knowledge 1 may be executed based on sensing data acquired by device 200a and sensing data acquired by device 200b. This increases the types of sensing data used in processing, thereby improving processing accuracy, etc. In the above example, the processing corresponding to tacit knowledge 1 may be executed by device 200a or device 200b, or may be realized by distributed processing between device 200a and device 200b. Furthermore, the processing corresponding to tacit knowledge is not limited to being executed by device 200 that acquires sensing data. For example, processing corresponding to tacit knowledge 1 may be executed by a device 200 other than device 200a or device 200b based on sensing data acquired by device 200a and sensing data acquired by device 200b. For example, in FIG. 1, the device 200, which is a smartphone, may execute processing corresponding to tacit knowledge based on sensing data acquired by the seat sensor 440 and sensing data acquired by the detection device 430.

[0028] Although the above describes an example in which tacit knowledge is digitized by the vendor of the device 200, the present invention is not limited to this. For example, a caregiver who uses the device 200 may also digitize his or her own tacit knowledge. For example, the caregiver may create an application corresponding to the tacit knowledge and register the application in the server system 100. In this way, it is possible to promote the digitization and use of tacit knowledge.

[0029] 3 is a block diagram showing a detailed configuration example of the server system 100. The server system 100 includes, for example, a processing unit 110, a storage unit 120, and a communication unit .

[0030] The processing unit 110 of this embodiment is configured by the following hardware. The hardware can include at least one of a circuit for processing digital signals and a circuit for processing analog signals. For example, the hardware can be configured by one or more circuit devices or one or more circuit elements mounted on a circuit board. The one or more circuit devices are, for example, an integrated circuit (IC), a field-programmable gate array (FPGA), etc. The one or more circuit elements are, for example, a resistor, a capacitor, etc.

[0031] The processing unit 110 may also be implemented by the following processor. The server system 100 of this embodiment includes a memory that stores information and a processor that operates based on the information stored in the memory. The information may be, for example, a program and various data. The memory may be the storage unit 120 or another memory. The processor includes hardware. Various processors, such as a central processing unit (CPU), a graphics processing unit (GPU), or a digital signal processor (DSP), may be used. The memory may be a semiconductor memory such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory, or may be a register, a magnetic storage device such as a hard disk drive (HDD), or an optical storage device such as an optical disk drive. For example, the memory stores computer-readable instructions, and the processor executes the instructions to realize the functions of the processing unit 110. The instructions may be instructions from an instruction set that constitutes a program, or instructions that instruct the hardware circuitry of the processor to operate.

[0032] The processing unit 110 includes, for example, a capability information acquisition unit 111 , a scene information acquisition unit 112 , a device type information acquisition unit 113 , and a communication processing unit 114 .

[0033] The ability information acquisition unit 111 performs a process of acquiring ability information representing the activity ability of the person being assisted. For example, the ability information acquisition unit 111 acquires sensing data from the device 200. The sensing data here may be, for example, log data for a predetermined period. The ability information acquisition unit 111 determines time-series changes in the state of the person being assisted based on the log data and estimates the ability information based on the changes. The ability information here may be index information related to activities of daily living (ADL). For example, various methods for evaluating ADL, such as the Barthel Index, are known, and these can be widely applied in this embodiment. Furthermore, the ability information may be based on the nine-level index disclosed as the Clinical Frailty Scale in "Frailty and the potential kidney transplant recipient: time for a more holistic assessment?" by Henry H.L. Wu et al. For example, the ability information acquisition unit 111 in this embodiment determines which of the nine levels the person being assisted belongs to based on the sensing data.

[0034] The capability information may be obtained in the device 200 or the like. The capability information acquisition unit 111 may execute a process of acquiring the capability information from the device 200 or the like via the communication unit 130.

[0035] The scene information acquisition unit 112 determines a scene in which assistance is provided to the person being assisted. The scene information here may be information specifying the type of assistance to be provided, such as meal assistance, excretion assistance, or movement / transfer assistance. The scene information may also be information about the assistants providing assistance to the person being assisted, such as the number and skill level of the assistants. The scene information may also be information about the person being assisted, such as the attributes of the person being assisted. For example, the scene information acquisition unit 112 performs processing to obtain scene information based on user input, the schedules of the assistants, etc.

[0036] The device type information acquisition unit 113 acquires information that identifies the type of device 200 that operates together with the target device 200. The device type here represents a rough classification such as a wheelchair or a bed, and may be information that does not distinguish between vendors. For example, a wheelchair from a first vendor and a wheelchair from a second vendor different from the first vendor may have the same device type. For example, the device type information acquisition unit 113 may identify other devices 200 used by the person being assisted who uses the target device 200 (or an assistant assisting the person being assisted), and may perform processing to acquire information indicating the type of the other devices 200 as device type information.

[0037] The communication processing unit 114 controls communication using the communication unit 130. For example, the communication processing unit 114 executes processing to create data to be transmitted, such as a MAC frame in the data link layer. The communication processing unit 114 may also perform processing such as interpreting the frame structure of data received by the communication unit 130, extracting necessary data, and outputting it to a higher layer such as an application.

[0038] The storage unit 120 is a work area for the processing unit 110 and stores various types of information. The storage unit 120 can be realized by various types of memory, and the memory may be a semiconductor memory such as an SRAM, a DRAM, a ROM, or a flash memory, or may be a register, a magnetic storage device, or an optical storage device.

[0039] The storage unit 120 may store user information 121 , device information 122 , and application information 123 .

[0040] In the method of this embodiment, for example, tacit knowledge is registered as an application in the server system 100. Each user who uses the tacit knowledge may register a device 200 in the system and then download the necessary application to the device 200 for use.

[0041] The user information 121 includes information such as a user ID and a user name that uniquely identify a user of the information processing system 10, and a device ID that is information that uniquely identifies a device 200 used by the user.

[0042] The device information 122 is information relating to the device 200, and includes a device ID, a device type ID indicating the type of the device 200, a vendor, an application ID of an installed application, etc. The application ID is information that uniquely identifies an application.

[0043] The application information 123 is information about an application, and includes an application ID, an application name, a creator, and the like. The application information may also include information that identifies the specific processing content of the application. The information that identifies the processing content may be the source code of a program or an executable file. If the application corresponds to a trained model, the application information may also include information about the structure of the trained model. For example, if the trained model is a neural network (hereinafter referred to as NN), the structure of the trained model includes the number of layers of the NN, the number of nodes included in each layer, the connection relationships between nodes, weights, activation functions, and the like.

[0044] By using the user information 121, it becomes possible to appropriately manage users who use the information processing system 10. Furthermore, by referring to the device information 122, it becomes possible to check details of the device 200 used by each user. Furthermore, by referring to the application information 123, it becomes possible to check details of each application registered in the server system 100.

[0045] The communication unit 130 is an interface for communicating via a network, and when the server system 100 performs wireless communication, it includes, for example, an antenna, an RF (radio frequency) circuit, and a baseband circuit. However, the server system 100 may also perform wired communication, in which case the communication unit 130 may include a communication interface such as an Ethernet connector and a control circuit for the communication interface. The communication unit 130 operates under the control of the communication processing unit 114. However, the communication unit 130 may also include a processor for communication control different from the communication processing unit 114. The communication unit 130 may perform communication according to a method specified in, for example, IEEE802.11 or IEEE802.3. However, the specific communication method can be modified in various ways.

[0046] FIG. 4 is a block diagram showing a detailed configuration example of the device 200. The device 200 includes, for example, a processing unit 210, a storage unit 220, a communication unit 230, a display unit 240, and an operation unit 250. As will be described later with reference to FIGS. 8 to 14, various aspects of the device 200 can be used in the method of this embodiment. The configuration of each device 200 is not limited to that shown in FIG. 4, and modifications such as omitting some components or adding other components are possible. For example, the device 200 may have various sensors appropriate for the device 200, such as a motion sensor such as an acceleration sensor or a gyro sensor, an imaging sensor, a pressure sensor, a GPS (Global Positioning System) sensor, etc.

[0047] The processing unit 210 is configured by hardware including at least one of a circuit for processing digital signals and a circuit for processing analog signals. The processing unit 210 may also be realized by a processor. Various types of processors, such as a CPU, a GPU, or a DSP, can be used as the processor. The processor executes instructions stored in the memory of the device 200, thereby realizing the functions of the processing unit 210 as processing.

[0048] The storage unit 220 is a work area for the processing unit 210, and is realized by various types of memory such as SRAM, DRAM, and ROM.

[0049] The communication unit 230 is an interface for communicating via a network, and includes, for example, an antenna, an RF circuit, and a baseband circuit. The communication unit 230 communicates with the server system 100, for example, via the network. The communication unit 230 may perform wireless communication with the gateway 300 in accordance with, for example, the IEEE 802.11 standard, and communicate with the server system 100 via the gateway 300.

[0050] The display unit 240 is an interface that displays various information and may be a liquid crystal display, an organic EL display, or another type of display. The operation unit 250 is an interface that accepts user operations. The operation unit 250 may be a button or the like provided on the device 200. The display unit 240 and the operation unit 250 may also be a touch panel that is integrally configured.

[0051] Device 200 may also include components not shown in FIG. 4, such as a light-emitting unit, a vibration unit, a sound input unit, and a sound output unit. The light-emitting unit is, for example, an LED (light emitting diode) and provides notification by emitting light. The vibration unit is, for example, a motor and provides notification by vibration. The sound input unit is, for example, a microphone. The sound output unit is, for example, a speaker and provides notification by sound.

[0052] 2. Device control based on capability information As shown in FIG. 1, the information processing system 10 of this embodiment includes a server system 100 and a device 200. The device 200 may operate in any of a plurality of operation modes. The operation mode in this embodiment may be determined by the combination of tacit knowledge (applications) to be used. For example, as described above, the device 200 can install multiple applications corresponding to different tacit knowledge, and each of the multiple applications can be switched between active and inactive. In this way, it becomes possible to appropriately switch the tacit knowledge to be used depending on the situation.

[0053] In particular, the server system 100 (capability information acquisition unit 111) may obtain capability information representing the activity capability of the person being assisted based on sensing data transmitted from the device. The server system 100 then transmits the obtained capability information to the device 200. The device 200 determines in which of a plurality of operation modes to operate based on the received capability information. Since the assistance to be performed changes in accordance with changes in the capability information of the person being assisted, the desired operation of the device 200 may change. However, according to the method of this embodiment, it is possible to appropriately switch the operation mode in accordance with the capability information. For example, the device 200 can appropriately switch between active and inactive states of an application (use / non-use of tacit knowledge) in accordance with the capability information. Processing based on the capability information will be described below.

[0054] 2.1 Overview Figure 5 shows an example of the relationship between the ability information of a person receiving care and the expected risks. For example, if the ability of a person receiving care is sufficiently high, they can perform daily activities without the assistance of others, so the risk in daily life is not high. However, as their abilities begin to decline, the person receiving care will first have difficulty, for example, performing daily activities. Daily activities refer to the actions of standing up and sitting down. In this case, it becomes difficult to maintain balance when starting to move, including getting up and down, increasing the risk of the person receiving care falling. Starting to move refers to starting to move from a state of small movement (or, in the narrow sense, a stationary state). On the other hand, at this stage, it is expected that there will be little difficulty in daily activities other than starting to move. For example, the person receiving care can maintain a sitting position for long periods of time, walk using a walker after standing up, and eat with some freedom.

[0055] As their abilities decline further, for example, the person receiving care may find it difficult to walk and may need assistance moving around using a wheelchair or other device. In this case, the risk of falling when starting to move is high, just as in the example above, but the person's ability to maintain a sitting position also declines, so the risk of falling must also be considered. For example, even when sitting in a bed or wheelchair, the person receiving care at this stage may lose their balance and fall off the bed mattress or wheelchair seat.

[0056] If the person's abilities decline further, for example, they may find it difficult to eat properly. For example, their swallowing ability may decline, increasing the risk of aspiration. The risk of aspiration means that there is a higher likelihood of developing aspiration pneumonia, for example. At this stage, the person receiving care is considered to be able to move around, so the high risk of falling and tripping described above remains, and the risk of aspiration pneumonia must also be considered.

[0057] As abilities decline further, for example, the person receiving care may become bedridden, requiring assistance with most daily activities. In this case, the risk of falls is still high, as diaper changes may be required in bed and the person may need to move around in a wheelchair. Furthermore, the risk of aspiration pneumonia is also high, as oral feeding will continue unless there are special circumstances such as a gastrostomy. Furthermore, the person will spend a lot of time in bed, and turning over on their own will become difficult, increasing the risk of bedsores. On the other hand, since the person receiving care is not expected to move on their own when they are bedridden, the risk of falling when they do so is low.

[0058] As shown in FIG. 5, the expected risks vary depending on the ability of the person being assisted, and therefore the assistance that the caregiver should provide also changes. Therefore, when using digitized tacit knowledge using device 200, the tacit knowledge to be used also changes depending on the ability. In this regard, in this embodiment, the operation mode of device 200 can be set according to the ability information, so processing can be performed according to changes in ability. For example, when providing assistance to approximately 10 people being assisted as a unit in a nursing home, it is conceivable that the people being assisted have different abilities. However, according to the method of this embodiment, the caregiver does not need to manually set the necessary assistance for each person being assisted. In other words, even when caring for multiple people being assisted, it is possible to set an appropriate operation mode without increasing the burden on the caregiver.

[0059] Although Figure 5 illustrates an example in which ability declines from top to bottom, the direction of ability change is not limited to this. For example, ability may recover due to the cure or remission of a disease, rehabilitation, etc. The method of this embodiment sets the operation mode according to ability, so it can flexibly respond to cases in which ability recovers. Furthermore, Figure 5 illustrates four stages: "unable to sit up and down," "unable to walk," "unable to eat properly," and "bedridden." However, the ability stages represented by the ability information are not limited to these; some stages may be omitted, or other stages may be added. Furthermore, a state in which the risk of falling is low because the patient can walk, but the risk of aspiration is high due to a decline in swallowing ability may also be considered. In other words, the above four stages are not limited to those that change in the order described above, and more complex combinations may be considered.

[0060] FIG. 6 is a sequence diagram illustrating the operations of the server system 100 and the device 200, and is a diagram illustrating pre-processing that is executed before the device 200 executes processing corresponding to tacit knowledge.

[0061] First, in step S101, the server system 100 performs a process of accepting application registration in advance. For example, as described above, each application corresponds to tacit knowledge and is created by the vendor of the device 200 or the like. The creator of the application logs in to the information processing system 10 of this embodiment using, for example, any terminal device (such as a PC or a smartphone), and then performs a process of registering the application in the server system 100 using a vendor screen (not shown) displayed on the display unit of the terminal device. The processing unit 110 of the server system 100 stores information about the registered application in the storage unit 120 as application information 123. Here, an example is shown in which vendor applications 1 to 3, which are applications created by vendors, are registered.

[0062] In step S102, a registration request for the device 200 is transmitted to the server system 100 based on an operation by a user who uses the device 200. The user here may be a caregiver who uses tacit knowledge, or may be an administrator of a nursing facility, for example. For example, the user executes the process of step S102 when introducing a new device 200 into his or her environment. For example, the user logs in to the information processing system 10 using the operation unit of the device 200 or the operation unit of a terminal device connected to the device 200, and then performs a process of registering the device 200 in the server system 100 using a user screen (not shown). The registration request includes, for example, a user ID that identifies the user, and information such as the vendor and model number of the device 200.

[0063] In step S103, the processing unit 110 of the server system 100 executes processing based on the registration request. For example, the processing unit 110 assigns a device ID that uniquely identifies the device 200 to the target device 200 and transmits the device ID to the device 200. The processing unit 110 may also execute processing to associate the logged-in user with the device 200 for which the registration request has been made. For example, the processing unit 110 may perform processing to add the device ID of the device 200 for which the registration request has been made to the user information 121 of the logged-in user. The processing unit 110 may also store the device ID of the device 200 for which the registration request has been made, in association with the device type ID of the device 200, etc., in the device information 122. The device type ID can be identified based on information such as the vendor and model number included in the registration request. Through the above processing, the newly introduced device 200 is registered in the information processing system 10.

[0064] Next, in step S104, an application to be used by the device 200 is selected based on an operation by the user of the device 200. For example, when a registered device 200 accesses the server system 100, the server system 100 may return a screen listing applications that can be used by the device 200. The user performs a user operation to select an application to be used from the listed applications. Here, consider an example in which a list of applications including vendor app 1 to vendor app 3 that were registered in the processing shown in step S101 is displayed, and the user performs an operation to select vendor app 1 to vendor app 3.

[0065] In step S105, the server system 100 permits the download of the selected application, and the device 200 downloads the selected application. The server system 100 may also perform a process of associating the device 200 with the application downloaded to the device 200. For example, the processing unit 110 performs a process of adding application IDs corresponding to vendor applications 1 to 3 to the device information 122 related to the device 200.

[0066] In step S106, the device 200 executes a process of installing the downloaded vendor applications 1 to 3. This enables the device 200 to operate in one of a plurality of operation modes. For example, the device 200 may switch between active and inactive for each of the vendor applications 1 to 3. In this case, the device 200 3= 8 possible operation modes can be selected. In this embodiment, a state in which all of vendor applications 1 to 3 are inactive is also considered to be one operation mode. The relationship between applications and operation modes is not limited to this. For example, multiple applications may operate exclusively. In the above example, device 200 may be able to set four operation modes: a mode in which all vendor applications are inactive, a mode in which only vendor application 1 is active, a mode in which only vendor application 2 is active, and a mode in which only vendor application 3 is active.

[0067] FIG. 7 is a sequence diagram illustrating the operations of the server system 100 and the device 200, and illustrates an example in which the operation mode of the device 200 changes based on the ability information of the person being assisted.

[0068] First, in step S201, the server system 100 performs processing to transmit data including capability information to the device 200. Fig. 7 shows an example in which data with an ADL index value of 2 is transmitted.

[0069] In step S202, the device 200 controls the activation / deactivation of installed vendor applications based on the acquired capability information. For example, the storage unit 220 of the device 200 may store information associating capability information with operation modes. The processing unit 210 of the device 200 performs processing to determine the operation mode based on the information and the capability information acquired from the server system 100. For example, the storage unit 220 may store table data associating ADL index values ​​with the activation / deactivation of each application. The processing unit 210 determines the activation / deactivation of each application by extracting records from the table data that match the received ADL index value. Here, an operation mode is set in which, of vendor applications 1 to 3, vendor applications 1 and 2 are activated and vendor application 3 is deactivated. However, the processing to determine the operation mode based on capability information is not limited to the above example, and various modifications are possible.

[0070] After the process of step S202, the device 200 executes a process according to the vendor application 1 and a process according to the vendor application 2. Specifically, the processing unit 210 of the device 200 acquires sensing data using a sensor, and obtains a processing result by executing a process defined in the application using the sensing data as input. In step S203, the device 200 transmits the processing result to the server system 100. The processing result here corresponds to the result of a judgment made using the tacit knowledge of an expert. The information transmitted to the server system 100 here is not limited to the processing result, and may include information such as a log of the sensing data.

[0071] In step S204, the server system 100 executes control based on the processing result received from the device 200. For example, the processing unit 110 may perform processing to identify a device to be controlled and transmit a control signal to operate the device to be controlled. The device to be controlled here may be a reclining wheelchair 510, which will be described later using FIG. 23, or a nursing care bed 520, which will be described later using FIG. 24. In this case, the control signal may be a signal that instructs the device to be controlled to change the angle of the back of the reclining wheelchair 510 or the angle of the bottom of the nursing care bed 520. The control signal may also be a signal that instructs the device to be controlled to issue an alert. For example, the device to be controlled is a device that includes an alert unit such as a display unit or a light-emitting unit, and the control signal is a signal that instructs the device to be controlled to issue an alert using image display, light emission, etc.

[0072] Furthermore, the process of determining the control target device and the control content based on the processing result of the device 200 may be performed by the processing unit 110 of the server system 100 or by the processing unit 210 of the device 200. In the latter case, in step S203, information identifying the control target device and the control content may be transmitted in addition to the processing result. Alternatively, the device 200 or the server system 100 may only identify the control target device, and the specific control content may be determined in the control target device. In this case, in steps S203 and S204, processes of transmitting the processing results are respectively performed. In addition, various modifications are possible regarding the control target device, the control signal, and the like. For example, the control target device may be the device 200. Furthermore, in step S204, the server system 100 may perform a process of storing a log of the sensing data transmitted from the device 200 in the storage unit 120.

[0073] Furthermore, in step S205, the server system 100 executes a process to update the ability information of the person being assisted. For example, the ability information acquisition unit 111 may obtain the ability information based on the log of the sensing data. For example, the storage unit 120 of the server system 100 may store information associating the sensing data with the ability information. The processing unit 110 executes a process to obtain the ability information based on the stored information and the sensing data transmitted from the device 200. Here, the information associating the sensing data with the ability information may be a trained model. The training data for generating the trained model here is, for example, data in which the ability information of the person being assisted, determined by a person with specialized knowledge (e.g., a doctor or an experienced caregiver), is assigned as correct answer data to the sensing data related to the person being assisted. As described above, the correct answer data may be an index value representing the ability, or may be a collection of information representing the presence or absence or level of individual abilities (such as the ability to maintain a sitting position and the ability to swallow, which will be described later). The processing unit 110 obtains the ability information by inputting the sensing data into the trained model. Alternatively, the information stored in the memory unit 120 may be reference data, which is sensing data whose correspondence with the ability information is known. The processing unit 110 may determine the similarity between the acquired sensing data and the reference data and calculate the ability information based on the similarity. The reference data may be, for example, sensing data acquired from a highly capable person receiving care. In this case, if the similarity with the reference data is high, the ability value is determined to be high, and if the similarity is low, the ability value is determined to be low. The reference data may also be other information, such as sensing data acquired from a less capable person receiving care. The processing unit 110 may also use current ability information for processing. For example, the processing unit 110 may calculate a change in the ability information based on the sensing data and calculate updated ability information based on the change and the current ability information. A specific example of the process for calculating ability information will be described later. In addition, in the process of step S205, information other than sensing data, such as a report entered by a caregiver or examination results by a doctor, may be used.

[0074] In step S206, the server system 100 performs processing to transmit data including the updated capability information to the device 200. In Fig. 7, an example is shown in which data with an ADL index value of 3 is transmitted.

[0075] In step S207, the device 200 controls the activation / inactivation of installed vendor applications based on the acquired capability information. For example, as described above, the device 200 determines the activation / inactivation of each vendor application based on table data. In the example of FIG. 7, vendor applications 1 and 2 are maintained in an active state, and vendor application 3 is changed from inactive to active. As a result, from step S207 onwards, the device 200 transitions to a state in which it operates in an operation mode in which all of vendor applications 1 to 3 are active. The operations from step S207 onwards are similar to, for example, steps S203 to S206.

[0076] The method of the present embodiment is not limited to application to the information processing system 10 including the server system 100 and the device 200, but may also be applied to an information processing device. The information processing device here refers to the server system 100 in a narrow sense. The information processing device includes a communication unit (corresponding to the communication unit 130 in FIG. 3 ) that operates in one of a plurality of operation modes and communicates with the device 200 used to assist the person being assisted, and a processing unit (corresponding to the processing unit 110 in FIG. 3 , or more specifically, the ability information acquisition unit 111) that performs processing to obtain ability information representing the activity ability of the person being assisted based on sensing data transmitted from the device 200. The processing unit of the information processing device then performs processing to transmit the ability information to the device 200 via the communication unit as information for determining in which of a plurality of operation modes the device 200 will operate. In this way, it is possible to estimate the transition of the ability information of the person being assisted based on information collected from the device 200, and to operate the device 200 in accordance with the ability information.

[0077] Furthermore, part or all of the processing performed by the information processing system of this embodiment may be realized by a program. The processing performed by the information processing system includes, for example, at least one of processing performed by the processing unit 110 of the server system 100 and processing performed by the processing unit 210 of the device 200. Similarly, part or all of the processing performed by the information processing device of this embodiment may be realized by a program.

[0078] The program according to this embodiment can be stored in, for example, a non-transitory information storage device (information storage medium), which is a medium readable by a computer. The information storage device can be realized, for example, by an optical disc, a memory card, a HDD, or a semiconductor memory. The semiconductor memory is, for example, a ROM. The processing unit 110 and the like perform various processes of this embodiment based on the programs stored in the information storage device. In other words, the information storage device stores programs for causing a computer to function as the processing unit 110 and the like. A computer is a device equipped with an input device, a processing unit, a storage unit, and an output unit. Specifically, the program according to this embodiment is a program for causing a computer to execute each of the steps described above using Figures 6, 7, etc.

[0079] The technique of this embodiment can also be applied to an information processing method in an information processing system 10 including a device 200 that operates in any of a plurality of operation modes and is used to assist a person being assisted, and a server system 100 that is connected to the device 200 via a network. The information processing method includes the steps of: determining ability information that represents the activity ability of the person being assisted based on sensing data acquired by the device 200; and determining in which of a plurality of operation modes the device 200 will operate based on the determined ability information.

[0080] In the following, a specific device 200 and the processing executed in the device 200 will be described using the steps shown in FIG. 5 as examples.

[0081] 2.2 Unable to move around <Example of device and operation: Fall risk assessment> First, a description will be given of device 200 for dealing with the risk of falling when starting to move in a state where getting up and down becomes difficult. Figures 8 and 9 show an example of device 200 used to determine the risk of falling when starting to move.

[0082] Fig. 8 is a diagram showing an example of an imaging device 410 that captures an image of a person being assisted, and an example of an output image IM1 of the imaging device 410. In addition to the components shown in Fig. 4, the imaging device 410 has an image sensor that outputs the captured image as sensing data. The imaging device 410 may be placed in a place where many people gather together to engage in activities, such as a living room or hall in a care facility. In the example of Fig. 8, the imaging device 410 is placed on top of a television set.

[0083] The processing unit 210 of the imaging device 410 may perform processing to detect whether a person has started to move based on the captured image. The processing unit 210 operates, for example, in accordance with an application installed in the imaging device 410, to acquire the captured image as input data, and executes processing to detect a person from the captured image and processing to determine whether the detected person has started to move.

[0084] For example, the image capture device 410 performs face recognition processing to recognize a person's face based on a captured image. For example, the storage unit 220 of the image capture device 410 may store a facial image of the person to be detected, and the processing unit 210 may perform face recognition processing based on a matching process using the facial image as a template. Various face recognition processing techniques are known, and a wide variety of such techniques can be applied in this embodiment. For example, when the movement of a detected facial area continues to be below a given threshold for a certain period of time, the image capture device 410 sets the position of the facial area in that state as a reference position. The image capture device 410 may then set a detection area at a position a predetermined distance from the reference position and determine that movement has occurred when the facial area reaches the detection area. For example, when a person stands up, the position of the face is expected to move relatively upward, so the detection area may be set at a position a predetermined distance above the reference position. In this case, movement is detected when the position of the facial area on the image moves upward by more than a predetermined distance from the reference position. Note that the detection area here is, for example, a linear area, but other shapes may also be set.

[0085] Furthermore, the image capture device 410 can identify the target person being assisted by face recognition processing. Therefore, the image capture device 410 may perform movement detection for a person being assisted whose ability represented by the ability information is equal to or lower than a predetermined threshold (corresponding to being unable to perform daily activities), and may omit movement detection for a person being assisted whose ability is higher than the threshold.

[0086] The process of detecting the start of movement is not limited to the above method. For example, the image capture device 410 may perform skeletal tracking based on captured images. Note that various image-based skeletal tracking methods are known, such as "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields" (https: / / arxiv.org / pdf / 1611.08050.pdf) and OpenPose disclosed by Zhe Cao et al., and these methods can be widely applied in this embodiment.

[0087] OpenPose also discloses a method for performing skeletal tracking on each of multiple people captured in an image and displaying the results. In the example of Fig. 8, the image sensor outputs a captured image including three people being assisted, and the image capture device 410 determines whether each of the three people being assisted has started to move.

[0088] For example, a person receiving care whose mobility has declined and who finds it difficult to get up and down may fall even when attempting to stand up. Therefore, the image capture device 410 may use skeletal tracking to determine whether the person is assuming a standing up posture. For example, if the image capture device 410 determines that the person has leaned forward from a sitting position, placing their hands on their knees or the seat of a chair, the image capture device 410 determines that the person is assuming a standing up posture and notifies the caregiver of the risk of falling. For example, the image capture device 410 may determine that the person receiving care is assuming a standing up posture when it detects from the skeletal tracking results that the distance between the positions of the hands and knees is less than a predetermined value, or that the position of the shoulders has moved downward by more than a predetermined value, etc.

[0089] Alternatively, the image capture device 410 may divide the data to be processed into windows of several seconds each, and determine that a posture change, such as standing up, has occurred when a specific position, such as the head or neck, moves within each window by more than a predetermined threshold. The target body part for movement detection may be other than the head or neck. The direction of movement may be vertical, horizontal, or diagonal. The threshold used for detection may be changed depending on the target body part. Alternatively, the image capture device 410 may determine an area containing feature points detected by skeletal tracking of a stationary person receiving care, and determine that a movement, such as standing up, has occurred when a predetermined number of feature points or more fall outside the area. Various other variations are possible for the method of movement detection using the image capture device 410.

[0090] IM1 in FIG. 8 is an example of an output image from the imaging device 410. The imaging device 410 may superimpose some kind of display object on the captured image. In the example of FIG. 8, an object including an exclamation mark is displayed in association with the person being assisted whose movement has been detected. This makes it possible to clearly notify the caregiver of the person being assisted whose movement has been detected. For example, in step S203 of FIG. 7, the imaging device 410 transmits the output image IM1 to the server system 100. In step S204, the server system 100 outputs the output image IM1 to a smartphone or the like used by the caregiver. However, the output of the imaging device 410 may be information identifying the person being assisted whose movement has been detected (e.g., the ID of the person being assisted), and various modifications are possible. For example, although an example of notifying the person being assisted whose movement has been detected is shown here, information for stopping the movement of the person being assisted may also be output. For example, the imaging device 410 may identify the person being assisted who has started moving and output audio data, video data, etc. of the person being assisted's family, etc. In particular, dementia patients tend to respond poorly to calls, but they often remember the faces and voices of family members, making stopping their movements an effective way to stop them. By stopping the movements of the person being assisted in this way, it is possible to buy time for the caregiver to intervene. An example of outputting family voice and video data will be described later in connection with device control based on scene information.

[0091] FIG. 9 is a diagram illustrating an example of a bedside sensor 420 and a detection device 430 disposed at the bottom of a bed 610. The bedside sensor 420 and the detection device 430 are, for example, as shown in FIG. 9, sheet- or plate-shaped devices 200 provided between the bottom of the bed 610 and the mattress 620. Although FIG. 9 illustrates both the bedside sensor 420 and the detection device 430, only one of them may be used. Furthermore, as will be described below, the bedside sensor 420 and the detection device 430 have in common the fact that they include a pressure sensor, so the bedside sensor 420 may also function as the detection device 430, or the detection device 430 may also function as the bedside sensor 420. Various other modifications are possible with regard to the specific aspects.

[0092] The bedside sensor 420 includes a pressure sensor that outputs a pressure value as sensing data, and is placed on the side of the bottom that the caregiver uses to get in and out of the bed. In the example of FIG. 9, the caregiver gets in and out of the bed using the front side of the bed 610. In this case, as shown in FIG. 9, a railing to prevent falls may be placed on the front side of the bed 610, and the bedside sensor 420 may be placed in a position where the railing is not provided. In this way, the user getting in and out of the bed 610 first sits on the bedside sensor 420.

[0093] The processing unit 210 of the bedside sensor 420 operates, for example, according to an application installed on the bedside sensor 420, to acquire pressure values ​​as input data and perform processing to determine the movement of the person being assisted on the bed 610 from the pressure values.

[0094] For example, when a person being assisted stands up from bed 610, it is assumed that the person being assisted transitions from a lying position on the bed to a sitting position at the side of the bed (hereinafter referred to as "edge-sitting position"), and then performs a standing-up motion by placing their hands on their knees or the bedside to apply force. The pressure values ​​detected by the bedside sensor 420 increase in the order of lying position, edge-sitting position, and standing-up motion. For example, the bedside sensor 420 may determine that the start of movement has been detected when it detects a change from an edge-sitting position to a standing-up motion based on a comparison process between the pressure value and a given threshold value. Alternatively, from the perspective of detecting a standing-up motion at an earlier stage, the bedside sensor 420 may determine that the start of movement has been detected when it detects a change from a lying position to an edge-sitting position based on a comparison process between the pressure value and a given threshold value.

[0095] Alternatively, as the standing-up motion continues, the buttocks of the person being assisted rise from the bottom surface, causing a large decrease in the pressure value output from the pressure sensor. Therefore, the processing unit 210 may determine that the standing-up motion has been performed when the pressure value increases above a first threshold and then decreases below a second threshold that is smaller than the first threshold, based on the time-series change in the pressure value. In addition, various modifications can be made to the specific processing content of the movement start determination.

[0096] When the bedside sensor 420 detects that the person being assisted has started to move, it transmits information indicating this to the server system 100. The server system 100 then transmits the information to, for example, a smartphone used by the caregiver, and a notification process is executed on the smartphone. In this way, it becomes possible to clearly notify the caregiver that the person being assisted has started to move.

[0097] 9 is the device 200 that senses information related to the sleep of the person being assisted. The detection device 430 includes a pressure sensor that outputs a pressure value.

[0098] When the user gets into bed, the detection device 430 detects the user's body vibrations (body movements, vibrations) through the mattress 620. Based on the body vibrations detected by the detection device 430, information on the breathing rate, heart rate, activity level, posture, wakefulness / asleep, and whether the user is out of bed or in bed can be obtained. The detection device 430 may also determine whether the user is in non-REM sleep or REM sleep, and the depth of sleep. For example, the periodicity of body movements may be analyzed, and the breathing rate and heart rate may be calculated from the peak frequency. The periodicity may be analyzed using, for example, a Fourier transform. The breathing rate is the number of breaths per unit time. The heart rate is the number of heartbeats per unit time. The unit time is, for example, one minute. Body vibrations may also be detected per sampling unit time, and the number of detected body vibrations may be calculated as the amount of activity. When the user gets out of bed, the detected pressure value decreases compared to when the user is in bed, so it is possible to determine whether the user is out of bed or in bed based on the pressure value and its time-series changes.

[0099] For example, the processing unit 210 of the detection device 430 may determine that the start of movement has been detected when the person being assisted transitions from being in bed to being out of bed, based on the result of the bed-out / bed-in determination.

[0100] When it is detected that the person being assisted has started to move, the detection device 430 transmits information indicating this to the server system 100. The server system 100 transmits the information to, for example, a smartphone used by the caregiver, and a notification process is executed on the smartphone. In this way, it is possible to notify the caregiver in an easy-to-understand manner that the person being assisted has started to move.

[0101] For example, each device 200 shown in FIGS. 8 and 9 is set to inactive operation mode 0 when the ability of the person being assisted, as represented by the ability information, is equal to or greater than a predetermined level, and is set to active operation mode 1 when the ability is less than the predetermined level. In this case, less than the predetermined level of ability means that the person is unable to get up and down. In this way, the above-mentioned movement start determination operation can be performed in an appropriate situation. As a result, when there is a person being assisted who is at high risk of falling, the risk of the person being assisted falling can be appropriately reduced.

[0102] <Ability information update> 7, the capability information acquisition unit 111 of the server system 100 may perform processing to update the capability information based on the sensing data. For example, the capability information acquisition unit 111 determines a change in the status related to living and / or drinking based on the sensing data.

[0103] For example, the ability information acquisition unit 111 may determine the manner of standing up based on sensing data. When standing up, as described above, the person places their hands on the platform or the like from a sitting position on the edge of the chair, then puts their weight on their feet, straightens their knees and straightens their spine. At this time, if the weight is not transferred sufficiently to the feet, the center of gravity will shift relatively backward, causing the person to fall onto the platform. If the weight is transferred excessively to the feet, the center of gravity will shift forward, causing the person to fall forward. Furthermore, if the person is sitting too shallowly on the edge of the chair, the buttocks may fall off the platform.

[0104] Therefore, the ability information acquisition unit 111 may determine whether the body movement of the person being assisted when standing up is appropriate based on the log of skeletal tracking by the imaging device 410 and the log of pressure values ​​from the bedside sensor 420 and the detection device 430. For example, the ability is determined to be higher when the movement when standing up is closer to a normal state, and the ability is determined to be lower when the deviation of the center of gravity or the position of the buttocks when sitting on the edge of the bed deviates from the normal state.

[0105] Alternatively, the ability information acquiring unit 111 may obtain the ability information based on the number of times the person stands up within a predetermined period. For example, the more times the person stands up, the higher the ability is determined to be, and the fewer times the person stands up, the lower the ability is determined to be.

[0106] The ability information acquisition unit 111 may also obtain ability information based on the elapsed time from sitting on the edge of the bed to standing up. Whether the person is lying down, sitting on the edge of the bed, or standing can be determined based on the relative positions of feature points in skeletal tracking and time-series changes in pressure values. For example, the shorter the time that elapsed from sitting on the edge of the bed to standing up, the higher the ability is determined to be, and the longer the time that elapsed, the lower the ability is determined to be.

[0107] Furthermore, the ability information acquiring unit 111 may obtain the ability information based on the amount of activity in bed 610. The amount of activity is detected by, for example, the detection device 430 as described above. The amount of activity may also be obtained based on the results of skeletal tracking or the output of the bedside sensor 420. For example, the greater the amount of activity, the higher the ability is determined to be, and the smaller the amount of activity, the lower the ability is determined to be.

[0108] In the method of this embodiment, the ability information may be determined using any one of the above methods, or a combination of two or more of them. Furthermore, as described above, the ability information may be determined based on a combination of sensing data and data other than sensing data. The method of this embodiment reduces the risk of a care recipient who is unable to sit up or down, and also makes it possible to appropriately determine changes in the ability information of the care recipient.

[0109] 2.3 Unable to walk <Example of device and action: Fall risk> Next, a description will be given of device 200 that operates to deal with the risk of falling from wheelchair 630 or the like when walking becomes difficult. Figures 10 and 11 show an example of device 200 used to determine the risk of falling.

[0110] FIG. 10 is a diagram showing a seat sensor 440, which is a device 200 placed on the seat of a wheelchair 630, for example. The seat sensor 440 includes a pressure sensor that outputs a pressure value, and determines, based on the pressure value, whether the posture of the person being assisted when sitting on the wheelchair 630 (hereinafter also referred to as the sitting posture) is one of a plurality of postures, including normal, forward shift, side shift, etc. Forward shift refers to a state in which the user's center of gravity is shifted further forward than normal, and side shift refers to a state in which the user's center of gravity is shifted to either the left or right than normal. Both forward shift and side shift correspond to states in which the risk of slipping is relatively high. The seat sensor 440 may be a sensor device placed on an ordinary chair, or may be a sensor device that determines the posture of a user sitting on a bed, etc. The seat sensor 440 may also perform a fall possibility determination, which determines whether or not the person being assisted is likely to fall from the seat.

[0111] 10, four pressure sensors Se1 to Se4 are arranged on the rear side of a cushion 441 that is placed on the seat of a wheelchair 630. Pressure sensor Se1 is a sensor that is arranged in the front, pressure sensor Se2 is a sensor that is arranged in the rear, pressure sensor Se3 is a sensor that is arranged on the right, and pressure sensor Se4 is a sensor that is arranged on the left. Note that front, back, left, and right here refer to directions as seen from the perspective of the person being assisted when the person is sitting in the wheelchair 630.

[0112] 10, the pressure sensors Se1 to Se4 are connected to a control box 442. The control box 442 includes a processor that controls the pressure sensors Se1 to Se4 and a memory that serves as a work area for the processor. For example, the processor corresponds to the processing unit 210, and the memory corresponds to the storage unit 220. The processor detects pressure values ​​by operating the pressure sensors Se1 to Se4.

[0113] A person being assisted sitting in wheelchair 630 may feel pain in the buttocks and may shift the position of the buttocks. For example, forward slippage occurs when the buttocks are shifted further forward than usual, and lateral slippage occurs when the buttocks are shifted to the left or right. In addition, forward slippage and lateral slippage may occur simultaneously, causing the center of gravity to shift diagonally. As shown in FIG. 10, by using a pressure sensor arranged on cushion 441, changes in the position of the buttocks can be appropriately detected, making it possible to accurately detect forward slippage or lateral slippage.

[0114] For example, the initial state is the timing when the person being assisted transfers to the wheelchair 630 and assumes a normal posture. In the initial state, the person being assisted sits deep in the seat of the wheelchair 630, so it is expected that the value of the rear pressure sensor Se2 will be relatively large. On the other hand, when forward slippage occurs, the position of the buttocks moves forward, and the value of the front pressure sensor Se1 will increase. For example, the processor of the control box 442 may determine that forward slippage has occurred when the value of the pressure sensor Se1 increases by a predetermined amount compared to the initial state. When the value of the pressure sensor Se1 exceeds a certain threshold, it may be determined that the person being assisted is in the wheelchair 630, and it may be determined that forward slippage has occurred based only on the change in the value of the pressure sensor Se2 without comparing it with the pressure sensor Se1. Furthermore, instead of using the value of the pressure sensor Se1 alone, processing may be performed using the relationship between the values ​​of the pressure sensors Se2 and Se1. For example, the difference in voltage values ​​output by the pressure sensors Se2 and Se1 may be used, or the ratio of the voltage values ​​may be used, or the rate of change of the difference or ratio relative to the initial state may be used.

[0115] Similarly, when a lateral slippage occurs, the position of the buttocks moves to either the left or right, and if the slippage is to the left, the value of pressure sensor Se4 increases, and if the slippage is to the right, the value of pressure sensor Se3 increases. Therefore, the processor may determine that a left slippage has occurred when the value of pressure sensor Se4 increases by a predetermined amount compared to the initial state, and may determine that a right slippage has occurred when the value of pressure sensor Se3 increases by a predetermined amount compared to the initial state. Alternatively, the processor may determine a right slippage or a left slippage using the relationship between the values ​​of pressure sensor Se4 and pressure sensor Se3. As with the example of a forward slippage, the difference in voltage values ​​output by pressure sensor Se4 and pressure sensor Se3 may be used, or the ratio of the voltage values ​​may be used, or the rate of change of the difference or ratio from the initial state may be used.

[0116] If a forward or lateral slippage is detected, the seat sensor 440 transmits information indicating this to the server system 100. The server system 100 then transmits the information to, for example, a smartphone used by the caregiver, and a notification process is executed on the smartphone. The control box 442 may also include a light-emitting unit, which may be used to notify the caregiver. In this way, the caregiver can be notified of changes in the seating posture of the wheelchair 630 or the like in an easy-to-understand manner, making it possible to prevent the person being assisted from falling.

[0117] FIG. 11 is a diagram showing a terminal device 450, which is a device 200 used to assist in adjusting a wheelchair position. The wheelchair position is information related to the position and posture of the person being assisted in the wheelchair 630. The wheelchair position may represent the seated posture described above, or may represent information including the placement of cushions, etc. As shown in FIG. 11 , in adjusting the wheelchair position, a terminal device 450 having a camera and fixed at a height that allows the camera to capture an image of at least the upper body of the person being assisted sitting in the wheelchair 630 may be used. Note that the terminal device 450 may be capable of capturing an image of a wider range of the person being assisted, for example, up to the knees, or the entire body. The terminal device 450 is placed in a predetermined position in, for example, a nursing facility, and the caregiver transfers the person being assisted into the wheelchair 630, moves the person to the front of the terminal device 450, and then adjusts the wheelchair position.

[0118] The terminal device 450 includes a display unit and displays the comparison result between the image captured by the camera and the training data. The training data here is, for example, image data of the person being assisted, captured by a highly skilled caregiver while the person is seated in the wheelchair 630 in an appropriate posture. For example, the training data is registered in advance using the terminal device 450 or the like. The training data may also be registered in the storage unit 120 of the server system 100. The training data is not limited to image data itself that represents an appropriate posture, but may also be data to which additional information has been added. The additional information here may be information indicating points that a highly skilled caregiver considers important. The training data may also be the results of skeletal tracking.

[0119] The terminal device 450 (processing unit 210) may output an image in which the teacher data that has been subjected to transparency processing is superimposed on the actual captured image. Furthermore, the terminal device 450 may determine whether the position of the person being assisted in the wheelchair 630 is appropriate based on a comparison process between the teacher data and the actual captured image, and output the determination result. Furthermore, if the position of the person being assisted is not appropriate, the terminal device 450 may present points that need to be corrected. Points that need to be corrected are, for example, areas where the difference between the teacher data and the actual captured image is greater than a predetermined value.

[0120] For example, each device 200 shown in FIGS. 10 and 11 is set to inactive operation mode 0 when the ability of the person being assisted, as represented by the ability information, is equal to or greater than a predetermined level, and is set to active operation mode 1 when the ability is less than the predetermined level. In this case, less than the predetermined level of ability means that the person is unable to walk. In this way, the position determination for the wheelchair 630 or the like described above can be performed in an appropriate situation. As a result, when there is a person being assisted who is at high risk of falling from the wheelchair 630 or bed 610, the risk of the person being assisted falling can be appropriately reduced.

[0121] <Device behavior regarding fall risk> 5, even if a care recipient who cannot walk is not bedridden, there is a possibility that the care recipient may start to move, such as standing up, and therefore the risk of falling is high. Even when a care recipient who cannot walk is the target, it is desirable to continue detecting the start of movement by each device 200 shown in FIGS. 8 and 9.

[0122] In this case, between a state where a person cannot sit up but can walk and a state where a person cannot walk, the latter may have lower ability and a higher risk of falling. Therefore, each device 200 shown in FIGS. 8 and 9 may operate in operation mode 1 when a person cannot sit up but can walk, and in operation mode 2, which is different from operation mode 1, when a person cannot walk. For example, when a detection device 430 is used as device 200, the processing unit 210 of detection device 430 may detect the start of movement based on the result of determining whether the person is out of bed or in bed in operation mode 1, and may detect the start of movement based on the result of determining whether the person is awake or asleep in operation mode 2. For example, in operation mode 2, the processing unit 210 may determine that there is a possibility of the start of movement when the person transitions from a sleeping state to an awake state. In this way, operation mode 2 can detect the start of movement at an earlier stage than operation mode 1, thereby further reducing the risk of falling.

[0123] <Ability information update> 7, the ability information acquisition unit 111 of the server system 100 may perform processing to update the ability information based on the sensing data. For example, the ability information acquisition unit 111 determines a change in sitting ability, which is the ability to maintain a sitting position, based on the sensing data.

[0124] For example, the ability information acquisition unit 111 may determine the number of forward or lateral shifts based on the sensing data. For example, the fewer the number of forward or lateral shifts, the higher the ability to maintain sitting posture is determined to be, and vice versa.

[0125] The ability information acquisition unit 111 may also calculate the sitting ability based on the time the person can maintain a certain posture (hereinafter referred to as the posture maintenance time). The posture maintenance time is, for example, the length of a period of time beginning when the person assumes a given reference posture and ending when the person's posture changes by a predetermined amount from the reference posture. The posture maintenance time may be, for example, the time from when the caregiver corrects the person's posture using the terminal device 450 until the seat sensor 440 determines that the person has shifted forward or to the side. For example, the longer the posture maintenance time, the higher the ability to maintain a certain posture is determined to be, and the shorter the posture maintenance time, the lower the ability to maintain a certain posture is determined to be. Furthermore, if the person's ability to maintain a certain posture is high, the same amount of pressure will be applied to each pressure sensor continuously. On the other hand, if the person's ability to maintain a certain posture decreases, the person will frequently tilt or correct their posture, even if it is not determined that the person has shifted forward or to the side. As a result, the pressure value is likely to decrease (pressure loss). Therefore, the ability information acquiring unit 111 may estimate the sitting ability based on the number of times such pressure is released, the frequency, the degree of decrease in value, the direction of the release (which of the pressure sensors Se1 to Se4 has decreased in value), etc.

[0126] In the method of this embodiment, the ability information may be determined using any one of the above methods, or a combination of two or more of them. Furthermore, as described above, the ability information may be determined based on a combination of sensing data and data other than sensing data. The method of this embodiment reduces the risk of a fall for a care recipient who cannot walk, and makes it possible to appropriately determine changes in the ability information of the care recipient.

[0127] 2.4 Difficulty eating <Example of device and action: Aspiration risk> Next, a device 200 that operates to address the risk of aspiration (or, more narrowly, the risk of aspiration pneumonia) when a person is unable to eat properly will be described. Fig. 12 shows an example of device 200 used to determine the risk of aspiration during eating.

[0128] FIG. 12 is a diagram illustrating a choking detection device 460, which is a device 200 used in a mealtime. As shown in FIG. 12, the choking detection device 460 includes a throat microphone 461 attached around the neck of the person being assisted and a terminal device 462 equipped with a camera. Note that another device equipped with a camera may be used instead of the terminal device 462. The throat microphone 461 outputs audio data resulting from the person being assisted swallowing, coughing, etc. The camera of the terminal device 462 outputs captured images of the person being assisted eating. The terminal device 462 is, for example, a smartphone placed on a table where the person being assisted eats. The throat microphone 461 is connected to the terminal device 462 using Bluetooth (registered trademark) or the like, and the terminal device 462 is connected to the server system 100 via the gateway 300. However, both the throat microphone 461 and the terminal device 462 may be connectable to the gateway 300, and various modifications of the specific connection configuration are possible.

[0129] For example, a processor included in the choking detection device 460 acquires audio data from the throat microphone 461 and an image captured using a camera. The processor here corresponds to the processing unit 210 and may be a processor included in the terminal device 462, for example.

[0130] The processor determines whether the person being assisted choked or swallowed based on the audio data from the throat microphone 461. A device for detecting swallowing using a microphone worn around the neck is described, for example, in U.S. Patent Application No. 16 / 276,768, filed February 15, 2019, entitled "Swallowing action measurement device and swallowing action support system." This patent application is incorporated by reference in its entirety. Based on the audio data, the processor can detect the number of choking episodes, the duration of the choking (time of occurrence, duration, etc.), and whether or not the person swallowed.

[0131] The camera of the terminal device 462 can detect the mouth, eyes, and chopsticks and spoons used by the person being assisted by capturing an image of the person being assisted from the front, as shown in Fig. 12. There are various known methods for detecting these facial features and objects based on image processing, and a wide range of known methods can be applied in this embodiment.

[0132] For example, the processor can determine, based on the image captured by the camera, whether the person being assisted has their mouth open, food coming out of their mouth, and whether they are chewing. The processor can also determine, based on the image captured by the camera, whether the person being assisted has their eyes open. The processor can also determine, based on the image captured by the camera, whether chopsticks, a spoon, or the like are near tableware, whether the person being assisted can hold them, and whether they are spilling food.

[0133] In the method of the present embodiment, the swallowing or choking status of the person being assisted is estimated based on this information. For example, the processor may perform processing based on the detection results of choking and swallowing and the determination result of whether the person being assisted is opening or closing their mouth.

[0134] For example, the processor may determine whether choking is occurring frequently based on the number of times or duration of choking and output the determination result. For example, the processor may determine that choking is occurring frequently when the number of times choking per unit time exceeds a threshold. In this way, choking situations can be automatically determined.

[0135] The processor may also calculate the swallowing time, measured from when the person being assisted opens their mouth to when they swallow, based on the swallowing detection results and the results of determining whether the person being assisted has opened or closed their mouth. In this way, even if it is determined that the number of swallows has decreased, it is possible to determine specific circumstances, such as whether the person has not even put food in their mouth, or whether the person has put food in their mouth but has not swallowed it. For example, the processor may start counting up a timer when the mouth transitions from a closed state to an open state based on the captured image of the terminal device 462, and stop counting the timer when swallowing is detected by the throat microphone 461. The time when the timer stops represents the swallowing time. This allows for accurate determination of whether the risk of aspiration is high during meals and whether the caregiver should take some kind of action, thereby making it possible to appropriately utilize the tacit knowledge of experts.

[0136] For example, the choking detection device 460 may output the swallowing time as a processing result to the server system 100. The processor may also determine the pace of eating based on the swallowing time. The processor may also determine whether the swallowing time is long based on changes in the swallowing time during one meal (for example, the increase or ratio compared to the swallowing time at the beginning). Alternatively, the processor may calculate the average swallowing time for each of multiple meals for the same person receiving care, and determine whether the swallowing time has become long based on changes in the average swallowing time.

[0137] Furthermore, by using the results of determining whether the mouth is open or closed based on the captured image of the terminal device 462, it can be determined whether the person no longer opens their mouth even when the caregiver approaches them with a spoon or the like. In this way, if the swallowing time becomes longer when the person being assisted is reluctant to open their mouth, it can be estimated that food is remaining in the mouth and becoming stagnant. Furthermore, by using the captured image to determine whether food is coming out of the mouth and whether the person being assisted is chewing the food, it can be determined whether the person being assisted is no longer able to chew the food. For example, if the number of chews is normal but the swallowing time is long, it can be estimated that the person being assisted is no longer able to chew the food. Furthermore, if it is determined that the eyes are closed based on the captured image, it can be determined whether the person being assisted is becoming sleepy.

[0138] Furthermore, by performing recognition processing of chopsticks, spoons, etc. using the captured image, it may be determined whether the child is playing with food, unable to hold a bowl, spilling food, etc.

[0139] As described above, various mealtime situations can be determined by using the choking detection device 460. In the choking detection device 460 for input data according to this embodiment, each of these determinations may be implemented as an application corresponding to tacit knowledge. For example, the choking detection device 460 may include an application for detecting choking and swallowing and calculating swallowing time, an application for detecting frequent choking, an application for detecting dangerous choking, an application for determining whether the patient is sleepy, and an application for determining whether the patient is playing with food. In this embodiment, the activation / inactivation of each application is controlled based on the ability information, etc. In other words, the execution / inactivation of the various processes described above may be flexibly settable. If the choking detection device 460 determines that a predetermined situation exists, it transmits information indicating this to the server system 100. The server system 100 may, for example, cause a terminal device used by the caregiver to execute a notification process based on the information. The server system 100 may also output information indicating an appropriate action according to the eating situation detected by the choking detection device 460 to a device used by the caregiver, and the device may then present an action based on the information. Presentation to the caregiver may be, for example, audio output to a headset, a display on a display unit of a smartphone or the like, or presentation using other methods.

[0140] <Device behavior regarding fall risk> Furthermore, as shown in Fig. 5, even if a person receiving care has difficulty eating, there is a possibility that the person may start to move, such as standing up, and therefore the risk of falling is high. Therefore, even when a person receiving care has difficulty eating, it is desirable to continue detecting the start of movement by each device 200 shown in Figs. 8 and 9. Each device 200 may operate, for example, in the above-mentioned operation mode 2. Furthermore, each device 200 may operate in operation mode 3, which is capable of detecting the start of movement at an earlier stage than operation mode 2.

[0141] <Device behavior regarding fall risk> Furthermore, even if a person receiving care who has difficulty eating is likely to use a wheelchair 630 or the like, the risk of falling is high. Therefore, even when a person receiving care who has difficulty eating is being treated as the target, it is desirable to continue the processing related to the wheelchair position by each device 200 shown in FIGS. 10 to 11. For example, each device 200 operates in operation mode 1, just as when a person receiving care who cannot walk is being treated as the target. Furthermore, each device 200 may operate in a mode different from when a person receiving care who cannot walk is being treated as the target. For example, when a person receiving care who cannot walk is being treated as the target, the device may operate in an operation mode that only performs a forward or lateral slippage determination, whereas when a person receiving care who has difficulty eating is being treated as the target, the device may operate in an operation mode that can perform a fall possibility determination in addition to a forward or lateral slippage determination.

[0142] <Ability information update> 7, the ability information acquiring unit 111 of the server system 100 may perform processing to update the ability information based on the sensing data. For example, the ability information acquiring unit 111 determines a change in swallowing ability based on the sensing data.

[0143] For example, the ability information acquisition unit 111 may determine that the shorter the swallowing time from when the person being assisted opens their mouth to when they swallow, the higher their swallowing ability is, and that the longer the swallowing time, the lower their swallowing ability is. For example, the ability information acquisition unit 111 may determine that the swallowing ability has declined when the most recent average swallowing time of the person being assisted becomes longer than the average swallowing time of past people being assisted by a threshold or more, determine that the swallowing ability has not declined (is maintained) when the average swallowing time does not exceed the threshold, and determine that the swallowing ability has recovered when the average swallowing time has shortened. Furthermore, if the change in the average swallowing time before and after changing the food texture is equal to or less than a threshold, the swallowing ability may be determined to have recovered. Furthermore, the ability information acquisition unit 111 may obtain ability information based on a combination of swallowing ability and ability to maintain a sitting position. For example, the ability information acquisition unit 111 may obtain ability information based on the duration or frequency of use of the wheelchair 630. For example, a person receiving care whose ability to maintain a sitting position has declined may transition to eating in bed, and therefore, the ability to use a wheelchair 630 indicates a high level of ability to maintain a sitting position. For example, the more frequently the wheelchair is used, the higher the ability is determined to be, and the less frequently the wheelchair is used, the lower the ability is determined to be. As described above, the choking detection device 460 may execute various applications that perform different processes. For example, the choking detection device 460 may first activate an application that detects choking and swallowing and calculates swallowing time, and the ability information acquisition unit 111 may obtain ability information based on the output of that application. When the ability information changes, other applications, such as an application that detects frequent choking, are changed from inactive to active. This allows the processing content executed by the choking detection device 460 to be appropriately controlled based on the ability information. Furthermore, when the number of active applications increases, the ability information acquisition unit 111 may combine the output of each application to obtain swallowing ability based on more sensing data.

[0144] 2.5 Bedridden <Examples of devices and actions: Pressure ulcer risk> Next, a device 200 that operates to address the risk of bedsores when the care recipient's abilities have further declined and he or she has become bedridden will be described. Figures 13 and 14 show an example of device 200 used to determine the risk of bedsores.

[0145] FIG. 13 is a diagram illustrating a bed position detection device 470, which is a device 200 arranged around a bed 610. As shown in FIG. 13, the bed position detection device 470 includes a first terminal device 471 fixed to the footboard side of the bed 610, a second terminal device 472 fixed to a side rail of the bed 610, and a display 473 fixed on the opposite side of the second terminal device 472. The display 473 is not limited to being fixed to the bed 610, and may be arranged in another position where it can be viewed naturally by a caregiver adjusting the bed position. For example, the display 473 may be fixed to a wall surface or a stand that stands on the floor. The display 473 can be omitted. Alternatively, either the first terminal device 471 or the second terminal device 472 may be omitted. For example, the following describes an example in which the first terminal device 471 is used to adjust the bed position and the second terminal device 472 is used to change diapers, but the present invention is not limited to this example.

[0146] The first terminal device 471 and the second terminal device 472 are devices such as smartphones equipped with cameras. The first terminal device 471 transmits captured images directly to the server system 100. The second terminal device 472 transmits captured images of the camera to the server system 100 directly or via the first terminal device 471. The display 473 receives images transmitted from the server system 100 directly or via another device such as the first terminal device 471, and displays the received images. Note that the first terminal device 471 and the second terminal device 472 may have a depth sensor instead of or in addition to a camera. That is, these devices may output depth images.

[0147] For example, in adjusting a bed position, a process of registering training data and a process of adjusting a position using the training data may be performed. The training data is information registered, for example, by an experienced caregiver. When adjusting a bed position, an inexperienced caregiver selects training data and adjusts the bed position so that the actual state of the person being assisted matches the training data. For example, the first terminal device 471 acquires a captured image of the person being assisted lying on the bed (including the state of cushions, etc.), and the display 473 displays an image representing the comparison result between the captured image and the training data. This enables caregivers of all skill levels to perform position adjustments similar to those performed by an experienced caregiver. For example, the experienced caregiver lies the person being assisted on the bed 610, positions the person in a position suitable for preventing bedsores, etc., and then uses the first terminal device 471 to capture an image of the person being assisted. After confirming that the bed position is appropriate, the experienced caregiver selects the registration button. As a result, the still image displayed when the registration button was pressed is transmitted to the server system 100 as training data. In this way, it becomes possible to register positions that an expert considers to be preferable as training data. At this time, additional information may be added, as in the case of the wheelchair position example described above.

[0148] Furthermore, when the caregiver actually adjusts the bed position, first, the first terminal device 471 is activated and image capturing begins. For example, the caregiver activates the first terminal device 471 by voice, and the display 473 displays the moving image captured by the first terminal device 471. The bed position detection device 470 may also accept a selection process of teacher data by the caregiver. The processing unit 110 determines the teacher data based on a user operation and performs control to display the teacher data on the display 473.

[0149] Alternatively, the processing unit 110 may automatically select the training data based on a similarity determination between the attributes of the person being assisted whose bed position is to be adjusted and the attributes of the person being assisted captured in the training data. The attributes here include information such as the age, sex, height, weight, medical history, and medication history of the person being assisted.

[0150] Alternatively, the bed position detection device 470 may automatically select training data based on a comparison between the attributes of the person being assisted whose bed position is to be adjusted and additional information contained in the training data. For example, the additional information in the training data may include text such as "For a person being assisted who shows a tendency of XX, it is recommended to adjust the left shoulder so that it is YY." In this case, if the person being adjusted corresponds to XX, the training data is likely to be selected. For example, a caregiver who adjusts the bed position may input information identifying the person being assisted into the first terminal device 471, and the bed position detection device 470 may identify the attributes of the person being assisted based on the information.

[0151] The bed position detection device 470 may output an image in which the teacher data that has been subjected to transparency processing is superimposed on a real-time image captured by the first terminal device 471, for example. At this time, additional information about the teacher data may be displayed in a recognizable manner. For example, when it is detected that the caregiver has uttered "Tell me the key points" using a microphone or the like of a headset, the bed position detection device 470 may output the text as voice from the headset via the server system 100, for example.

[0152] The bed position detection device 470 may determine whether the image captured during position adjustment is OK or NG based on the degree of similarity between the image and the training data, and output the determination result. The determination result is displayed on the display 473 directly or via the server system 100. The bed position detection device 470 may also perform processing to display specific points determined to be NG. For example, the server system 100 or the bed position detection device 470 may perform processing to compare the image captured by the first terminal device 471 with the training data and highlight areas determined to have a large difference.

[0153] In this way, it becomes possible to present a comparison between the bed position of the person being assisted and the ideal bed position, and to present information for realizing the ideal bed position.

[0154] The bed position detection device 470 may also be used to assist in changing diapers. It has been found that experts place importance on the following points as tacit knowledge in changing diapers. A.Is the patient in a lateral position? B. Is the diaper in the correct position? C. Check if the padding is coming out of the diaper D. Was the diaper properly fitted?

[0155] Therefore, the processing unit 210 of the bed position detection device 470 determines whether the above points A to D are satisfied, and transmits the determination result to the server system 100. The processing unit 210 here corresponds to, for example, the processor of the second terminal device 472. This makes it possible to change the diaper appropriately regardless of the skill level of the caregiver.

[0156] For example, the second terminal device 472 performs skeletal tracking processing on each image constituting a moving image of the person being assisted captured using a camera, and outputs an image in which the skeletal tracking results are superimposed on the original image as the processing result. The processing result may be displayed on the display 473, for example. In this way, the caregiver can check the display 473 in a natural posture while changing the diaper of the person being assisted.

[0157] In consideration of the case where diaper changes are performed at night, the second terminal device 472 may include a lighting unit. Also, in consideration of the privacy of the person being assisted, a depth sensor or the like may be used instead of a camera. The depth sensor may be a sensor using a ToF (Time of Flight) method, a sensor using structured illumination, or a sensor using another method.

[0158] 15A and 15B are examples of images displayed on the display 473 when a diaper is being changed. As described above, each image includes the person being assisted and the results of skeletal tracking of the person being assisted.

[0159] In the state shown in Fig. 15A, the person being assisted is stable in a lateral position, and the camera of the second terminal device 472 captures an image of the person being assisted directly from behind. For example, in Fig. 15A, there is a small difference between the front-to-back direction of the person being assisted's body and the direction of the camera's optical axis. As a result, as shown in Fig. 15A, many points that can be detected by skeletal tracking are detected.

[0160] 15B, the person being assisted is less stable than in FIG. 15A, and appears to be on the verge of falling onto his back. The camera of the second terminal device 472 captures the image of the person being assisted from diagonally behind, so the number of points detected by skeletal tracking is reduced. For example, the point corresponding to the waist is not detected because it is hidden by a diaper or the like.

[0161] Therefore, the second terminal device 472 may determine whether or not the subject is in the lateral position shown in A above based on the results of skeletal tracking. For example, the second terminal device 472 may determine that the subject is in the lateral position when a point corresponding to a specific part such as the waist is detected by skeletal tracking. However, the determination of the lateral position may also be based on whether or not a point other than the waist is detected, or the relationship between multiple points, and the specific method is not limited to this.

[0162] The bed position detection device 470 also performs object tracking processing based on moving images captured by the camera of the second terminal device 472 to continuously detect the diaper area in the image. Object tracking is well known, so a detailed description will be omitted. For example, in Figures 15A and 15B, the diaper area ReD is detected.

[0163] The bed position detection device 470 may determine whether the diaper position shown in B above is appropriate based on, for example, the relationship between the results of skeletal tracking and the diaper area ReD detected by object tracking. For example, taking into account the position where the diaper is worn, it determines whether the waist position detected by skeletal tracking and the diaper area ReD have a predetermined positional relationship. For example, the processing unit 210 may determine that the diaper position is appropriate if a line including two points corresponding to the pelvis passes through the diaper area ReD. Alternatively, the results of skeletal tracking and the detection results of the diaper area ReD may be extracted as features from training data by an expert, and machine learning may be performed using the features as input data. The trained model is a model that, for example, outputs the likelihood that the diaper position is appropriate when it receives the results of skeletal tracking and the detection results of the diaper area ReD.

[0164] The bed position detection device 470 may also determine whether the pad is protruding from the diaper shown in C above based on the horizontal length of the diaper area ReD. Since the pad is usually contained within the diaper, the length of the diaper area ReD on the image corresponds to the length of the diaper itself. The expected size of the diaper area ReD can be estimated based on the type and size of the diaper and the optical characteristics of the camera of the second terminal device 472. On the other hand, if the pad is protruding, the length of the diaper area ReD on the image will be longer by that amount. Therefore, if the length of the diaper area ReD detected from the image is greater than the expected length by a predetermined threshold or more, the bed position detection device 470 determines that the diaper is protruding from the pad and is inappropriate.

[0165] The bed position detection device 470 may also determine whether the diaper described above in D is properly put on by detecting the tape used to secure the diaper in place. Typically, the tape is a different color from the diaper itself. For example, the diaper itself is white and the tape is blue. The way the tape should be secured to properly put on the diaper is known from the diaper's structure. Therefore, the processing unit 210 can detect the tape area in the image based on the color and determine whether the diaper is properly put on based on the relationship between the tape area and the diaper area ReD or the relationship between the tape area and the position of the waist or other parts detected by skeletal tracking. When multiple diapers from different manufacturers or types are used, the bed position detection device 470 may acquire information identifying the diapers and determine whether the diapers are properly put on based on the identified type of diaper, etc.

[0166] For example, the bed position detection device 470 judges whether each of the above A to D is OK or NG, and transmits the judgment results to the server system 100. The server system 100 transmits the judgment results to the display 473 or the like. Furthermore, if the result is NG, the bed position detection device 470 may highlight the part that is significantly different from the correct data.

[0167] This approach not only ensures that the patient assumes a position that reduces the risk of bedsores, but also allows the caregiver to appropriately perform diaper changes by appropriately utilizing tacit knowledge in diaper changing. Note that the diaper change requires the patient to move. For example, the caregiver may first place the patient in a lateral position to facilitate diaper placement, or may raise the patient's legs to put on the diaper. Therefore, when the diaper change is complete, the patient may not be in a position suitable for lying down in bed. Therefore, the bed position detection device 470 may automatically execute the above-described process to adjust the bed position when it detects that the diaper change is complete. For example, the first terminal device 471 begins capturing an image of the patient, and the display 473 displays the training data, which has been subjected to transparency processing, superimposed on the real-time image captured by the first terminal device 471.

[0168] In the above, an example has been described in which the first terminal device 471 and the second terminal device 472 such as a smartphone are used for bed position adjustment support and diaper change support, but the specific device 200 is not limited to this.

[0169] FIG. 14 is a diagram illustrating a glasses-type device 480 such as AR (Augmented Reality) glasses or MR (Mixed Reality) glasses, which is another example of the device 200 used for bed position adjustment assistance or diaper change assistance. The glasses-type device 480 has, for example, a camera that captures an image of an area corresponding to the user's field of view. The glasses-type device 480 has a display in part or all of its lens portion, and is capable of allowing the user to visually recognize the situation in the outside world by transmitting light from the outside world or by displaying an image corresponding to the user's field of view captured by the camera. Furthermore, the glasses-type device 480 uses a display to additionally display some information in the user's field of view.

[0170] Even when the glasses-type device 480 is used, it is possible to acquire an image of the user on the bed 610. Therefore, the glasses-type device 480 may perform processing to assist in bed position adjustment by superimposing the captured image and the teacher data as described above. The glasses-type device 480 may also perform processing to assist in diaper changing by making a determination regarding the diaper area ReD. Similarly, when the glasses-type device 480 is used, bed position adjustment may be started when diaper changing is completed. For example, when completion of diaper adjustment is detected, correct answer data related to bed position adjustment may be displayed transparently on the display unit of the glasses-type device 480.

[0171] The glasses-type device 480 may also perform processing to automatically detect the presence and extent of a bedsore based on captured images of the skin of the person being assisted. This makes it possible to determine whether a bedsore actually exists and, if so, what state the bedsore is in. For example, a trained model is generated by machine learning based on training data that associates images of the person being assisted with correct answer data that identifies the bedsore area provided by a doctor or other expert. The glasses-type device 480 performs processing based on the trained model to determine whether or not a bedsore exists.

[0172] Mattresses and pillows capable of pressure detection and automatic position change are also known. Pressure detection is performed using a pressure sensor, similar to the bedside sensor 420 and the detection device 430. The mattress and pillow may encourage the care recipient to turn over by adjusting the height (thickness) of each part using air or other means. The device 200 that detects pressure and the device 200 that encourages automatic position change may be different. For example, if the detection device 430 determines that the same posture continues based on the pressure value, it transmits information indicating this to the server system 100. The server system 100 may control the mattress or pillow based on this information to encourage the care recipient to change position. This can also reduce the risk of bedsores in bedridden care recipients. The control method for these mattresses and pillows is not limited to one. For example, even for the same bedridden person, detailed ability information (such as ADLs) may be determined, and the encouragement to change position may be changed depending on the level of the ability information. For example, if a care recipient has extremely low abilities, it may be difficult to maintain posture, and the posture may change excessively when prompted to change position. Therefore, mattresses and pillows may have multiple operation modes that differ in the frequency of prompting for position changes, the amount of air used when prompting for position changes, etc., and the operation mode may be controlled according to the level of ability information. Furthermore, mattresses and pillows may operate in a mode that improves sleep conditions (for example, suppresses snoring by encouraging the care recipient to lie on their side) while the ability is high, and in a mode that reduces the risk of bedsores if the care recipient becomes bedridden. Operation mode control according to ability information may also be performed for conditions other than being bedridden.

[0173] Furthermore, when targeting a bedridden person receiving care, tacit knowledge regarding end-of-life care may be used. The device 200 on which an application corresponding to the tacit knowledge runs may be a terminal device such as a smartphone. For example, the processing unit 210 of the device 200 acquires five types of information as input data: the amount or intake ratio of each type of meal (for example, main dish, side dish, or each ingredient such as meat or fish), the amount of water intake, the timing of intake, information regarding illness, and weight (or BMI). These input data may be entered by a caregiver. Alternatively, some of the input data may be acquired automatically by the device 200 by using an automatic recording device for food intake.

[0174] Based on the input data, the processing unit 210 outputs output data indicating whether end-of-life care should be started after a predetermined period of time, or whether it is time to change the content of care after end-of-life care has started. For example, machine learning may be performed on the input data based on training data to which expert-provided correct answer data has been added. In this case, the processing unit 210 obtains output data by inputting the input data into a trained model. Other machine learning methods, such as SVM, may also be used, or methods other than machine learning may also be used.

[0175] End-of-life care here refers to assistance for individuals who are considered likely to die in the near future. End-of-life care differs from standard care in that it emphasizes the alleviation of physical and mental pain and the support of the individual living a dignified life. Furthermore, as the individual's condition changes over time, the appropriate care may change accordingly. In other words, by indicating the timing for initiating end-of-life care and the timing for changes in the care content during end-of-life care, it becomes possible to provide appropriate care to the individual until the end. For example, experienced caregivers possess tacit knowledge to estimate the timing and content of end-of-life care needed based on various factors such as the amount of food consumed. By digitizing this tacit knowledge, other caregivers can also provide appropriate end-of-life care.

[0176] When the ability represented by the ability information has declined to a state indicating bedriddenness, the processing unit 210 of the device 200 starts a judgment based on the input data and outputs an analysis result screen showing the judgment result to the server system 100. The analysis result screen may be displayed on the display unit 240 of the device 200.

[0177] FIG. 16 illustrates an example of an analysis result screen. The analysis result screen may include time-series changes in feature values ​​calculated based on input data and a determination result of whether end-of-life care should be initiated after a predetermined period of time. The feature values ​​may be important input information, such as a moving average of food intake, or information calculated based on the five pieces of input information. For example, when a neural network (NN) is used, the feature values ​​may be the output of a given intermediate layer or output layer. For example, the input data may include actual measurements of main dish intake, water content, and BMI up to February 13, 2020. The processing unit 210 may estimate trends in main dish intake, water content, and BMI from February 14, 2020 onward based on the trained model. The analysis screen may also include graphs showing time-series changes in the actual and estimated values ​​for these three items. Note that FIG. 16 illustrates a graph of the moving average of these values ​​over a seven-day period. This allows caregivers to easily understand the trends in important items in end-of-life care. As mentioned above, the input data may include other items, and the information displayed on the analysis result screen is not limited to the example of FIG.

[0178] The analysis result screen may also display a period during which end-of-life care may be provided. In the example of Figure 16, the text "End-of-life care may be provided from 2020-03-14" is displayed, and the corresponding period in the graph is displayed in a distinctive background color different from the other periods. This clearly indicates the timing and period during which end-of-life care is estimated to be necessary, making it possible to appropriately present information about end-of-life care to the user.

[0179] <Device behavior regarding fall risk> 5, a bedridden caregiver is unlikely to spontaneously start moving, such as standing up, and therefore the risk of falling is assumedly low. Therefore, when a bedridden care recipient is the target, each device 200 shown in FIGS. 8 and 9 may be set to operation mode 0, which is inactive.

[0180] <Device behavior regarding fall risk> On the other hand, even for bedridden care recipients, the risk of falling is high because they may use a wheelchair 630 or the like. Therefore, even when targeting a bedridden care recipient, it is desirable to continue the fall risk processing by each device 200 shown in FIGS. 10 and 11. In this case, the care recipient has difficulty changing their posture voluntarily, increasing the risk of bedsores. Therefore, the seat sensor 440 shown in FIG. 10 may operate in operation mode 3, which determines the level of the risk of bedsores in the wheelchair 630, in addition to determining forward and lateral slippage and falls. For example, the seat sensor 440 may determine that there is a risk of bedsores when a state in which there is little change in pressure value has continued for a predetermined period of time or more. Similarly, the terminal device 450 shown in FIG. 11 may operate in an operation mode that presents the caregiver with a position to prevent bedsores.

[0181] <Device behavior regarding aspiration risk> Even bedridden individuals receiving care may continue to eat orally. Therefore, the choking detection device 460 shown in FIG. 12 also operates when the target is a bedridden individual receiving care. The choking detection device 460 may operate in the same operating mode when the target is a bedridden individual receiving care who is not bedridden but has difficulty eating, and when the target is a bedridden individual receiving care. Furthermore, the choking detection device 460 may operate in an operating mode that can further reduce the risk of aspiration when the target is a bedridden individual receiving care, compared to when the target is a bedridden individual receiving care who has difficulty eating. For example, the choking detection device 460 may more easily detect the risk of aspiration by lowering the threshold used for comparison with the swallowing time. Alternatively, when the target is a bedridden individual receiving care, the choking detection device 460 may operate to reduce the risk of aspiration by adding a food type determination. Alternatively, when targeting a bedridden person receiving care, the choking detection device 460 may perform an operation to reduce the risk of aspiration by adding a determination as to whether or not there is a dangerous choking, in addition to the usual processes of detecting choking and calculating swallowing time, etc. Furthermore, the choking detection device 460 may add a process to determine whether choking occurs frequently, or a process to determine whether or not the person receiving care appears sleepy.

[0182] Furthermore, as described above in the end-of-life care, when the target is a bedridden person receiving care, the amount of food eaten may be automatically recorded. For example, the terminal device 462 of the choking detection device 460 may capture images of the meal before and after the meal and automatically record the amount of food eaten based on the difference between the images. That is, when the target is a bedridden person receiving care, the choking detection device 460 may operate in an operation mode that includes automatic recording of the amount of food eaten. Furthermore, a device other than the choking detection device 460 may be used as the device 200 that automatically records the amount of food eaten.

[0183] <Ability information update> 7, the capability information acquisition unit 111 of the server system 100 may perform processing to update the capability information based on the sensing data. For example, the capability information acquisition unit 111 performs an evaluation related to a bedsore based on the sensing data, and determines a change in capability based on the evaluation result.

[0184] For example, the capability information acquisition unit 111 determines the degree of dispersion of pressure values ​​using a detection device 430 or the like, and if pressure is applied to a specific location for a predetermined period of time or longer, it determines that the posture change required to prevent bedsores has not been achieved and that the capability is low.

[0185] Furthermore, the ability information acquiring unit 111 may determine the number of times that the positioning adjustment was judged as NG or the number of times that the diaper position was inappropriate based on information from the bed position detecting device 470, and obtain the ability information based on these numbers. For example, the fewer the number of times that the positioning adjustment was judged as NG or the fewer the number of times that the diaper position was inappropriate, the higher the ability is judged to be, and the more the number of times, the lower the ability is judged to be.

[0186] Furthermore, the ability information acquiring unit 111 may update the ability information based on the presence or absence of a bedsore output from the eyeglass-type device 480 such as MR glasses. For example, if there is no bedsore, the ability is determined to be higher than if there is a bedsore.

[0187] 2.6 Summary 8 to 14, etc., specific examples and operation examples of device 200 have been described. As described above, each device 200 switches its operation mode between at least an inactive operation mode 0 and an active operation mode 1 according to capability information. Also, as described above, the operation mode when active is not limited to one, and multiple operation modes with different processing contents may be set.

[0188] Fig. 17 shows an example of the operation mode of each device 200. As shown in Fig. 17, the imaging device 410 for detecting the risk of falling operates in operation mode 1 for a person being assisted who cannot sit up, in operation mode 2 for a person being assisted who cannot walk or who cannot eat well, and in inactive operation mode 0 for a person being assisted who is bedridden.

[0189] For example, operation mode 1 may be a mode in which it is determined that movement has occurred when edge-sitting is detected, and operation mode 2 may be a mode in which it is determined that movement has occurred when awakening is detected. Operation mode 2 detects movement at an earlier stage than operation mode 1, and therefore can further reduce the risk of falling.

[0190] In addition, the seat sensor 440 that detects the risk of falling operates in an inactive operation mode 0 for an assisted person who cannot stand up, in operation mode 1 for an assisted person who cannot walk, in operation mode 2 for an assisted person who cannot eat properly, and in operation mode 3 for an assisted person who is bedridden.

[0191] For example, operation mode 1 may be a mode that only performs forward and lateral slippage, operation mode 2 may be a mode that adds a fall risk assessment, and operation mode 3 may be a mode that adds a pressure ulcer assessment in a wheelchair. In this example, additional assessment targets are added as ability declines, making it possible to appropriately respond to increased risk associated with changes in ability.

[0192] In addition, the choking detection device 460, which detects the risk of aspiration, operates in an inactive operating mode 0 for assisted persons who cannot sit up or walk, operates in operating mode 1 for assisted persons who cannot eat properly, and operates in operating mode 2 for assisted persons who are bedridden.

[0193] For example, operation mode 1 may be a mode that makes a judgment based on swallowing time, and operation mode 2 may be a mode that adds options such as food type to operation mode 1. In this example, additional judgment targets are added as ability declines, making it possible to appropriately respond to increased risks associated with changes in ability.

[0194] In addition, the bed position detection device 470, which detects the risk of bedsores, operates in an inactive operating mode 0 for assisted persons who are unable to sit up, walk, or eat properly, and operates in operating mode 1 for assisted persons who are bedridden.

[0195] As described above, the method of this embodiment makes it possible to take into consideration the relationship between capabilities and various risks, and to appropriately determine the presence or absence of risks in situations where it is highly necessary, and to perform processing to reduce the risks. Note that Fig. 17 shows an example of the relationship between the device 200, capability information, and operation mode, and the method of this embodiment is not limited to this.

[0196] When setting the operating mode of device 200 based on the ability information, it is expected that the functions of device 200 will be added to compensate for the decline in the ability of the person being assisted. However, a first device among devices 200 may be set to an operating mode in which more functions are used as the ability value represented by the ability information declines within a range above a predetermined threshold, and may be set to an operating mode in which fewer functions are used within a range below the predetermined threshold compared to a range above the predetermined threshold. That is, some of the devices in this embodiment may operate by increasing functions to compensate for a certain degree of decline in ability, but may shift to reducing functions when a certain level of decline in ability is observed. In this way, it is possible to appropriately consider the necessity of each tacit knowledge according to ability and reduce the processing load by deactivating applications corresponding to less necessary tacit knowledge.

[0197] The first device here may be a device 200 used to determine the risk of a fall when the care recipient begins to move. For example, the first device may be at least one of the imaging device 410 shown in FIG. 8, the bedside sensor 420 shown in FIG. 9, and the detection device 430 shown in FIG. 9. In the example of FIG. 17, within the ranges of "unable to sit up," "unable to walk," and "unable to eat properly," the operation modes of these devices 200 are changed to add processing content, while for the lower ability level of "bedridden," an operation mode corresponding to inactive is set. In this way, the continuation of the operation of a device whose necessity has decreased can be suppressed. Note that the detection device 430 may be used to determine end-of-life care for "bedridden," and it is not necessary for all of the devices 200 used to determine the risk of a fall when the care recipient begins to move to reduce their functions when the ability level is low.

[0198] 3. Device control based on scene information and device type information Furthermore, in the method of this embodiment, information other than capability information may be used to set the operation mode of the device 200. Scene information and device type information will be described below.

[0199] 3.1 Overview FIG. 18 is a sequence diagram illustrating the operation of the server system 100 and the device 200, and illustrates an example in which the operation mode of the device 200 changes based on the ability information of the person being assisted, the scene information, and the device type information.

[0200] First, in step S301, the server system 100 performs processing to transmit data including capability information, scene information, and device type information to the device 200. Note that either the scene information or the device type information may be omitted. Fig. 18 shows an example in which data is transmitted in which the ADL index value is 2, the scene ID that identifies the scene is 0, and the device type ID that identifies the device type is xx.

[0201] In step S302, the device 200 controls the activation / inactivation of installed vendor applications based on the acquired capability information, scene information, and device type information. For example, the storage unit 220 of the device 200 may store table data in which ADL index values, scene IDs, device type IDs, and the activation / inactivation of each application are associated with each other. The processing unit 210 of the device 200 determines the activation / inactivation of each application by extracting records from the table data that match the received data. Furthermore, as will be described later with reference to FIGS. 19, 20, and 22, each device 200 may store an algorithm for determining an operation mode based on the capability information, scene information, and device type information. FIG. 18 illustrates an example in which an operation mode is set in which, of vendor applications 1 to 3, vendor applications 1 and 2 are activated and vendor application 3 is deactivated.

[0202] After the process of step S302, the device 200 executes a process according to the vendor application 1 and a process according to the vendor application 2. In step S303, the device 200 transmits a process result to the server system 100. The process result here corresponds to a result of a judgment made using the tacit knowledge of an expert. The process result here may also include a log of sensing data detected by the device 200.

[0203] In step S304, the server system 100 executes control based on the processing result received from the device 200. For example, the processing unit 110 may identify a device to be controlled and transmit a control signal to operate the device to be controlled.

[0204] In step S305, the server system 100 executes a process of updating the ability information, scene information, and device type information of the person being assisted. The ability information updating process is as described above. The scene information acquisition unit 112 obtains scene information based on at least one of the log of sensing data acquired from the device 200, information about the person being assisted such as attributes, and information about the caregiver such as schedule. The device type information acquisition unit 113 acquires, as device type information, information indicating the types of other devices 200 used together with the target device 200. A specific example of the process will be described later.

[0205] In step S306, the server system 100 performs processing to transmit data including the updated capability information, scene information, and device type information to the device 200. In Fig. 18, an example is shown in which the scene ID has been changed from 0 to 1.

[0206] In step S307, the device 200 controls the activation / inactivation of installed vendor applications based on the acquired data. For example, as described above, the device 200 determines the activation / inactivation of each vendor application based on table data. In the example of FIG. 18, vendor applications 1 and 2 are maintained in an active state, and vendor application 3 is changed from inactive to active. As a result, from step S307 onwards, the device 200 transitions to a state in which it operates in an operation mode in which all of vendor applications 1 to 3 are active. The operations from step S307 onwards are similar to, for example, steps S303 to S306.

[0207] An example of setting an operation mode based on scene information and an example of setting an operation mode using device type information will be described below.

[0208] 3.2 Example of processing based on scene information The server system 100 may obtain scene information identifying an assistance scene for the person being assisted and transmit the obtained scene information to the device 200. The device 200 then determines which of a plurality of operation modes to operate in based on the capability information and the scene information. For example, the storage unit 220 of the device 200 stores information associating the capability information, the scene information, and the operation mode. The processing unit 210 determines the operation mode based on the stored information and the capability information and scene information acquired from the server system 100. The information associating the capability information, the scene information, and the operation mode may be, for example, table data associating the capability information value, the scene information value, and the active / inactive status of each application. Alternatively, the information associating the capability information, the scene information, and the operation mode may be an algorithm that determines the active / inactive status of each application based on the capability information and the scene information. This makes it possible to operate the device 200 appropriately according to the assistance scene, in other words, to perform assistance using appropriate tacit knowledge.

[0209] For example, the scene information acquisition unit 112 may obtain, as scene information, information about the caregivers who will be assisting the person being assisted. Specifically, the scene information may be information about the number of caregivers who can assist the person being assisted, or information about years of service, skill level, qualifications, etc. An example using the number of caregivers will be described below. In this case, the device 200 determines the operation mode depending on the number of caregivers.

[0210] The device 200 here may be, for example, the imaging device 410 described above with reference to FIG. 8. For example, when there are a certain number of caregivers in a living room or other area where care recipients tend to be active together, the caregivers can follow each other, allowing the caregivers to visually determine when the care recipient begins to move, or to intervene appropriately if they determine that the care recipient is at risk of falling. Intervention here refers to, for example, moving close to the care recipient who is about to start moving and supporting the body as needed to support the start of movement. In this case, there is a relatively low need for the imaging device 410 to be active.

[0211] On the other hand, when the number of caregivers in the target space is small, each caregiver has to perform many tasks, making it difficult to observe the movements of the person being assisted in detail and intervening at the appropriate time. In this case, the need for the imaging device 410 to be active becomes relatively high.

[0212] In view of the above, by determining the operation mode of the image capture device 410 using the scene information in addition to the capability information, it becomes possible to operate the image capture device 410 appropriately.

[0213] 19 is a flowchart illustrating the process of determining the operation mode in the image capturing device 410. First, in step S401, the processing unit 210 of the image capturing device 410 performs a process of identifying the target person being assisted. For example, the processing unit 210 identifies the person being assisted by performing a face recognition process based on the captured image.

[0214] In step S402, the processing unit 210 acquires the ability information of the person being assisted. For example, the processing unit 210 identifies the ability information by receiving the data shown in steps S301 and S306 in Fig. 18 from the server system 100.

[0215] In step S403, the processing unit 210 determines, based on the ability information, whether the ability of the person being assisted has declined to the point where the person is unable to sit up or walk. If the ability has not declined (step S403: No), the risk of falling is low. Therefore, in step S404, the processing unit 210 sets the operation mode of the imaging device 410 to mode 0, which corresponds to inactivity. Note that the ability information in this embodiment is not limited to the ADL index value but may be more detailed information, such as information indicating the above-mentioned standing-up method, sitting ability, swallowing ability, walking ability, etc. Furthermore, various sensing data can be used to determine sitting ability, etc., and various modifications can be made to the algorithm used to determine sitting ability, etc. from the sensing data. In determining whether the ability of the person being assisted has declined to the point where the person is unable to sit up or walk, as described above, processing using detailed ability information, which has a larger amount of data than the ADL index value, may be performed. In this case, the processing load of step S403 is large, so this processing may be performed in the server system 100.

[0216] If the ability of the person being assisted has deteriorated to the point where the person is unable to move around (step S403: Yes), in step S405, the processing unit 210 identifies the number of caregivers based on the scene information. For example, the scene information acquisition unit 112 of the server system 100 may determine the number of caregivers based on a captured image, or may determine the number of caregivers based on an assistance schedule (e.g., a roster) in a nursing care facility or the like. Alternatively, information indicating the number of caregivers in the target space may be obtained by attaching RFID (radio frequency identifier) ​​tags or the like to the caregivers and installing a reader at an entrance or exit of the target space. The processing unit 210 of the device 200 executes the process shown in step S405 by acquiring this information from the server system 100.

[0217] In step S406, the processing unit 210 makes a determination based on the scene information. Specifically, the processing unit 210 determines whether the number of caregivers is equal to or less than a predetermined threshold. If the number of caregivers is greater than the predetermined threshold (step S406: No), there are enough people available to deal with the risk of falling, so the processing proceeds to step S404, and the processing unit 210 sets the operation mode of the imaging device 410 to mode 0, which corresponds to inactivity.

[0218] If the number of people is equal to or less than the predetermined threshold (step S406: Yes), there are insufficient caregivers, and support from the device 200 is important to reduce the risk of falling. Therefore, in step S407, the processing unit 210 sets the operation mode of the imaging device 410 to mode 1, which corresponds to active. Note that, while steps S405 and S406 have been described in FIG. 19 as being performed by the processing unit 210, the present invention is not limited to this. For example, the server system 100 may perform steps S405 and S406, and the processing unit 210 may receive only the results and determine whether to transition to step S404 or step S407.

[0219] Furthermore, the scene information is not limited to information about the caregiver. For example, the scene information acquisition unit 112 of the server system 100 may obtain information about the person being assisted as the scene information. Specifically, the scene information may be attribute information that indicates the attributes of the person being assisted. The attribute information includes the age, height, weight, sex, medical history, medication history, etc. of the person being assisted. For example, the scene information acquisition unit 112 obtains information that indicates whether the person being assisted has dementia as the scene information.

[0220] To reduce the risk of falling when starting to move, it is important for the caregiver to call out to the person to stop them temporarily, but dementia patients may not respond when the caregiver calls out to them. However, it is known that even dementia patients often remember the voices of close family members and other people, and are more likely to respond when called by a family member.

[0221] Therefore, the imaging device 410 may have an operation mode in which audio data of family members' voices or video data of family members calling attention to something is stored and the audio data or video data is output using a speaker (not shown). For example, the processing unit 210 of the imaging device 410 determines whether the imaging device 410 is active or inactive based on the ability information and scene information, which is the number of caregivers, as shown in FIG. 19 . When the imaging device 410 is set to active, the processing unit 210 then determines whether the person being assisted identified in step S401 is a dementia patient. If the person being assisted is a dementia patient, the imaging device 410 operates in an operation mode that outputs audio data of family members, etc., and if the person being assisted is not a dementia patient, the imaging device 410 operates in an operation mode that does not output audio data, etc. This makes it possible to set an operation mode that matches the attributes of the person being assisted. Note that multiple audio data of family members, etc. may be prepared for one person being assisted. For example, if a family member, etc., only calls in one way, the person being assisted may memorize the call and become unresponsive. Therefore, the image capturing device 410 may perform a process of outputting one randomly selected piece of voice data from the plurality of voice data stored in association with the target person being assisted. This increases the variety of calls that family members and others can make, thereby effectively stopping the person being assisted from moving.

[0222] Furthermore, the scene information in this embodiment may be information indicating the type of assistance, such as meal assistance, excretion assistance, or mobility / transfer assistance. For example, the scene information acquisition unit 112 may acquire scene information indicating the type of assistance based on a user input by a caregiver. The scene information acquisition unit 112 may also determine the type of assistance being provided based on the relationship between an assistance schedule at a nursing facility or the like and the current time. Alternatively, the scene information acquisition unit 112 may determine the type of assistance by estimating the location of the person being assisted at the nursing facility or the like. For example, the scene information acquisition unit 112 determines that meal assistance is being provided if the person being assisted is in the dining room, and that excretion assistance or assistance to prevent falls is being provided if the person being assisted is in the toilet. The location determination may be performed using a motion sensor or the like installed at various locations in the facility. Alternatively, access points (APs) may be installed at various locations in the facility, and the location determination may be performed depending on which AP a station device (STA) carried by the caregiver connects to.

[0223] For example, the seat sensor 440 shown in Fig. 10 can determine whether the person is slipping forward or to the side, and whether there is a possibility of falling, as described above. However, when the person is eating while sitting in a wheelchair 630, a table on which food is served is placed in front of the person being assisted, as shown in Fig. 12, and the table provides support, making it less likely that the person will fall. Therefore, by deactivating the seat sensor 440's function for determining whether there is a possibility of falling while the person is eating, it is possible to omit less-essential processing.

[0224] 20 is a flowchart illustrating the process of determining the operation mode in the seat sensor 440. First, in step S501, the processing unit 210 of the seat sensor 440 performs a process of identifying the target person being assisted. For example, the processing unit 210 may identify the person being assisted based on an assistance schedule or the like, or may identify the person being assisted based on user input.

[0225] In step S502, the processing unit 210 acquires the ability information of the person being assisted. For example, the processing unit 210 identifies the ability information by receiving the data shown in steps S301 and S306 in Fig. 18 from the server system 100.

[0226] In step S503, the processing unit 210 determines, based on the ability information, whether the ability of the person being assisted has declined to the point where the person is unable to walk. If the ability has not declined (step S503: No), there is little need for a determination using the seat sensor 440. Therefore, in step S504, the processing unit 210 sets the operation mode of the seat sensor 440 to mode 0, which corresponds to inactive. Also, as in the example of step S403 described above, in step S503, processing may be performed that takes into account more detailed information about the ability information. Therefore, the processing of step S503 may be performed in the server system 100.

[0227] If the ability of the person being assisted has deteriorated to the point where the person is unable to walk (Yes in step S503), in step S505 the processing unit 210 acquires information indicating the type of assistance as scene information. As described above, the processing in step S505 may be performed based on a user input or on some kind of sensing data.

[0228] In step S506, the processing unit 210 makes a determination based on the scene information. Specifically, the processing unit 210 determines whether the assistance type is meal assistance. If it is meal assistance (step S506: Yes), the forward slippage / lateral slippage determination is useful, but the need for determining the possibility of a fall is low. Therefore, in step S507, the processing unit 210 sets the operation mode of the seat sensor 440 to a mode that performs the forward slippage / lateral slippage determination but does not perform the determination of the possibility of a fall.

[0229] If the assistance type is not meal assistance (step S506: No), there is not necessarily a table in front of the person being assisted, and there is a high risk of the person falling from the wheelchair 630. Therefore, in step S508, the processing unit 210 sets the operation mode of the seat sensor 440 to a mode that performs both the forward / lateral slippage determination and the determination of the possibility of a fall. Also, in step S508, the processing unit 210 may set the operation mode of the seat sensor 440 to a mode that does not perform the forward / lateral slippage determination, but performs the determination of the possibility of a fall.

[0230] The above describes an example in which processing based on scene information is performed on the imaging device 410 and the seat sensor 440. However, it goes without saying that the operation mode of the other devices 200 may also be determined based on scene information.

[0231] 3.3 Example of processing based on device type information The server system 100 (the device type information acquisition unit 113) may also obtain device type information that identifies the type of a concurrent device used to assist the same person being assisted as the device 200. The server system 100 transmits the device type information to the device 200. The device 200 determines in which of a plurality of operation modes the device 200 will operate based on the capability information and the device type information. For example, the storage unit 220 of the device 200 stores information that associates the capability information, device type information, and operation modes. The processing unit 210 determines the operation mode based on the stored information and the capability information and device type information acquired from the server system 100. The information that associates the capability information, device type information, and operation modes may be, for example, table data that associates the value of the capability information, the value of the device type information, and the active / inactive status of each application. Alternatively, the information that associates the capability information, device type information, and operation modes may be an algorithm that determines the active / inactive status of each application based on the capability information and device type information.

[0232] Fig. 21 is a diagram illustrating an example of the combined device and the choking hazard detection device 460 described above with reference to Fig. 12. The combined device here may specifically be the device 200 according to this embodiment. The choking hazard detection device 460 is used during meals, but meals may be taken using a wheelchair 630 or a bed 610.

[0233] For example, the device used in combination with the choking detection device 460 may be the seat sensor 440 shown in Fig. 10. The device type information acquiring unit 113 may determine that the choking detection device 460 and the seat sensor 440 are used in combination when the choking detection device 460 is active and a log of the detection results and sensing data from the seat sensor 440 has been acquired. Alternatively, the device type information acquiring unit 113 may determine that the choking detection device 460 and the seat sensor 440 are used in combination when it is determined that the person being assisted is sitting in a wheelchair 630 based on the detection results and sensing data log from the seat sensor 440. In this case, it is considered that the person being assisted is eating using the wheelchair 630.

[0234] Similarly, the device used in combination with the choking hazard detection device 460 may be the detection device 430 shown in Fig. 9. The device type information acquiring unit 113 may determine that the choking hazard detection device 460 and the detection device 430 are being used in combination when the choking hazard detection device 460 is active and a detection result or a log of sensing data from the detection device 430 has been acquired. Alternatively, the device type information acquiring unit 113 may determine that the choking hazard detection device 460 and the detection device 430 are being used in combination when it is determined that the person being assisted is in bed based on the detection result or the log of sensing data from the detection device 430. In this case, it is considered that the person being assisted is eating in bed 610.

[0235] 22 is a flowchart illustrating the process of determining the operation mode in the choking stool detection device 460. First, in step S601, the processing unit 210 of the choking stool detection device 460 performs a process of identifying the target person being assisted. For example, the processing unit 210 may identify the person being assisted based on face recognition processing of an image captured by the terminal device 462.

[0236] In step S602, the choking hazard detection device 460 acquires ability information of the person being assisted. For example, the processing unit 210 identifies the ability information by receiving the data shown in steps S301 and S306 in FIG. 18 from the server system 100.

[0237] In step S603, the choking hazard detection device 460 determines, based on the ability information, whether the ability of the person being assisted has declined to the point where the person is unable to eat properly. If the ability has not declined (step S603: No), there is little need for determination using the choking hazard detection device 460, and therefore in step S604 the processing unit 210 sets the operation mode of the choking hazard detection device 460 to mode 0, which corresponds to inactive.

[0238] If the ability of the person being assisted has deteriorated to the point where they are unable to eat properly (Yes in step S603), in step S605 the choking hazard detection device 460 acquires information identifying the combined device from the device type information acquisition unit 113. As can be seen from the above example, it is sufficient to determine whether the person is eating in bed 610 or in wheelchair 630, so while the type of combined device is important, there is little need to identify the vendor or model number. Therefore, in step S605 the choking hazard detection device 460 acquires the device type ID of the combined device.

[0239] In step S606, the choking hazard detection device 460 makes a determination based on the device type information. Specifically, the choking hazard detection device 460 determines whether the meal is being eaten using the wheelchair 630. Specifically, as described above, the choking hazard detection device 460 may determine whether the combined device represented by the device type information is the seat sensor 440 or the detection device 430. If the meal is being eaten using the wheelchair 630 (step S606: Yes), the condition of the person being assisted is relatively good because it means that the person can move to a restaurant or the like. Therefore, in step S607, the processing unit 210 sets the operation mode of the choking hazard detection device 460 to a mode that makes a normal determination using swallowing time, etc.

[0240] On the other hand, if the meal is being eaten using the bed 610 (step S606: No), it is estimated that the person being assisted has difficulty moving around or would have difficulty eating without a device that allows for flexible adjustment of the back angle, such as the bed 610. Therefore, in step S608, the processing unit 210 sets the operation mode of the choking detection device 460 to a mode that can further reduce the risk of aspiration. For example, the choking detection device 460 may not only detect the presence or absence of a choking, but also perform processing to determine the nature of the choking. For example, the choking detection device 460 may determine whether the choking is a dangerous choking that is likely to lead to aspiration. The choking detection device 460 may also determine whether the person being assisted looks sleepy. The determination of whether the person looks sleepy may be made based on, for example, the opening and closing of the eyes, the frequency of mouth and hand movements, etc., based on images captured by the terminal device 462. The determination of whether the person looks sleepy may also be performed using the detection device 430.

[0241] Although the above examples show two types of combined devices, the wheelchair 630 and the bed 610, the present invention is not limited to this. For example, a normal wheelchair and a reclining wheelchair may be used as the wheelchair 630. In this case, the flexibility of back angle adjustment increases in the order of the normal wheelchair, the reclining wheelchair, and the bed 610. Therefore, the operation mode of the choking stool detection device 460 may be controlled so that more thorough care can be provided in the order of the normal wheelchair, the reclining wheelchair, and the bed 610. For example, the choking stool detection device 460 may increase the number of applications set to active in the above order.

[0242] In the above, an example has been described in which the choking stool detection device 460 is the device 200 for which the operation mode is to be set, and the seat sensor 440 and the detection device 430 are the combined devices. However, the seat sensor 440 and the detection device 430 may be the device 200 for which the operation mode is to be set, and the choking stool detection device 460 may be the combined device. In other words, in the method of this embodiment, when multiple devices 200 are used in combination, the operation mode may be changed by the multiple devices 200 working together. For example, using either the seat sensor 440 shown in FIG. 10 or the terminal device 450 used to adjust the wheelchair position shown in FIG. 11 is useful for reducing the risk of falling. However, both the seat sensor 440 and the terminal device 450 may be used in combination. In this case, the control of one device 200 may be linked to the control of the other device 200.

[0243] For example, when it is determined based on the ability information that the person being assisted has transitioned to a state where he or she is unable to walk, the seat sensor 440 may first transition to an operation mode corresponding to active, while the terminal device 450 may maintain an operation mode corresponding to inactive. In this way, first, a determination is made as to whether the person is shifting forward or sideways, and the ability to maintain a sitting position is calculated using the log data.

[0244] If it is determined that the ability to maintain a seated position has dropped below a predetermined level, the terminal device 450 transitions to an active operation mode. This causes the device type information acquisition unit 113 to transmit data indicating that the terminal device 450 is a combined device to the seat sensor 440. The seat sensor 440 may be set to a different operation mode when the terminal device 450 is combined with the seat sensor 440 than when it is not combined with the seat sensor 440. For example, the processing unit 210 of the seat sensor 440 may change the threshold value used for the determination. Alternatively, the seat sensor 440 may switch from an operation mode in which a criterion (initial value) for determining forward or lateral shift, etc., is internally determined to an operation mode in which the criterion is determined based on, for example, the timing of a position adjustment using the terminal device 450. Similarly, in other cases, the control of the seat sensor 440 or the terminal device 450 may be changed based on information from the other device.

[0245] Alternatively, the operation mode of the seat sensor 440 may be controlled based on information acquired by the terminal device 450. For example, the terminal device 450 can input points that a caregiver should pay attention to when adjusting the posture of the person being assisted. For example, points such as paying attention to the position of the right shoulder or placing a cushion under the right arm may be input, and the terminal device 450 can estimate the attributes of the person being assisted based on the input points. In the above example, the attribute that the person being assisted has a tendency toward right shoulder paralysis is determined. The seat sensor 440 may determine the operation mode based on this information. For example, the seat sensor 440 may store multiple applications that perform forward / lateral slippage detection, which differ depending on the presence or absence of paralysis and the location of the paralysis. The seat sensor 440 then controls the activation / deactivation of each application based on the attributes acquired from the terminal device 450. In the above example, the seat sensor 440 activates an application that performs forward / lateral slippage detection suitable for a person being assisted with right shoulder paralysis, and deactivates other applications that perform forward / lateral slippage detection. In this way, in the seat sensor 440 that can use a plurality of implicit knowledge for determining whether the seat is shifting forward or sideways, it becomes possible to switch the implicit knowledge according to the attributes of the person being assisted.

[0246] Furthermore, the server system 100 of this embodiment may acquire mode information that specifies the operation mode of the concurrent device, and transmit the device type information and the mode information to the device 200. The device 200 determines in which of a plurality of operation modes to operate based on the capability information, device type information, and mode information.

[0247] For example, when the seat sensor 440 transitions to an operation mode in which it determines the possibility of falling in addition to determining whether the seat sensor 440 is shifting forward or sideways, the terminal device 450 may transition to an operation mode in which it performs an additional process of recommending a cushion or the like in addition to the normal process of determining whether the wheelchair position is appropriate. Alternatively, when the operation mode of the seat sensor 440 changes, the terminal device 450 may execute a process of switching the teacher data that is determined to be correct. Various other variations are possible regarding the specific method of linking operation modes.

[0248] In this way, it becomes possible to link multiple devices 200 using more detailed operation modes, not limited to simple active / inactive. In the above example, the device 200 related to detecting the wheelchair position and the device 200 related to assisting in adjusting the wheelchair position can be appropriately linked, which makes it possible to further reduce the risk of the person being assisted falling. Note that although the linkage between the seat sensor 440 and the terminal device 450 has been described here, this does not prevent other devices 200 from operating in linkage.

[0249] Furthermore, although the combination of capability information and scene information, and the combination of capability information and device type information have been described above, a combination of capability information, scene information, and device type information may also be used.

[0250] Furthermore, in the method of this embodiment, the type of assistance may be associated with the type of device 200. For example, the choking spurt detection device 460 shown in Fig. 12 is a device 200 specific to meal assistance, and when the choking spurt detection device 460 is operating, there is a high probability that the person being assisted is receiving meal assistance. In this way, among the devices 200, there are devices for which device type information can be associated with the type of assistance. For example, when the choking spurt detection device 460 is active, it communicates with the server system 100 via the gateway 300. Therefore, the scene information acquisition unit 112 can determine the operating state of the choking spurt detection device 460 based on whether or not communication with the choking spurt detection device 460 is occurring.

[0251] For example, the scene information acquisition unit 112 of the server system 100 may obtain scene information indicating the type of assistance based on the device type information acquired by the device type information acquisition unit 113, and transmit the scene information to the device 200. For example, the scene information acquisition unit 112 may determine that a person is eating when the choking spurt detection device 460 is active, and that a person is not eating when the choking spurt detection device 460 is inactive. Alternatively, the server system 100 may transmit the device type information acquired by the device type information acquisition unit 113 to the device 200, and the device 200 may perform processing to determine an operation mode according to the type of assistance associated with the device 200. For example, when the choking spurt detection device 460 is a combined device, the device 200 may perform processing to select an operation mode suitable for meal assistance. As described above, setting the operation mode according to the type of assistance may be performed as processing related to the scene information or as processing related to the device type information, and various modifications are possible in specific embodiments.

[0252] 4. Other examples of devices Furthermore, the device 200 in this embodiment is not limited to the one described above. For example, the device 200 in this embodiment may be an MCI assessment device that assesses mild cognitive impairment (MCI). For example, the MCI assessment device performs a process of asking questions to the person being assisted using voice or images and receiving a response to the questions. The questions here may be based on the Mini-Mental State Examination (MMSE) or other methods. The MCI assessment device assesses MCI based on the response of the person being assisted. The MCI assessment device may also assess MCI based on information about the sleep of the person being assisted.

[0253] Furthermore, the device 200 in this embodiment may be an assistance recording device that automatically records the history of assistance. For example, the assistance recording device may be device 200 that detects the position information of at least one of the caregiver and the person being assisted. The assistance recording device determines, for example, the time the person in question spends in bed 610, the toilet, the bath, the dining room, etc., and stores, as an assistance record, the type of assistance provided, how often, and for how long, based on the determination results. The assistance recording device is useful, for example, in home care where assistance schedules are not strictly set.

[0254] Furthermore, device 200 of this embodiment may include a reclining wheelchair 510 with an adjustable backrest angle shown in Fig. 23, or a nursing bed 520 with an adjustable bottom angle shown in Fig. 24. Reclining wheelchair 510 and nursing bed 520 are, for example, the controllable devices shown in Fig. 7 and Fig. 18. For example, when choking is detected by choking detection device 460, the angles of the backrest and bottom are controlled to angles suitable for the person being assisted to swallow.

[0255] Furthermore, the reclining wheelchair 510 may be controlled based on the processing results of the seat sensor 440 and the terminal device 450. For example, the wheelchair 630 shown in Figures 10 and 11 may be the reclining wheelchair 510, which is a device to be controlled. Similarly, the nursing bed 520 may be controlled based on the processing results of the bedside sensor 420, the detection device 430, the bed position detection device 470, the glasses-type device 480, etc. For example, the bed 610 shown in Figures 9, 13, and 14 may be the nursing bed 520, which is a device to be controlled.

[0256] Additionally, the device 200 used in this embodiment can be modified in various ways with respect to the shape, the number and type of sensors, the processing contents, and the like.

[0257] Although the present embodiment has been described in detail above, it will be readily apparent to those skilled in the art that many modifications are possible without substantially departing from the novel features and advantages of the present embodiment. Therefore, all such modifications are intended to be included within the scope of the present disclosure. For example, a term described at least once in the specification or drawings together with a different term having a broader or equivalent meaning may be replaced with that different term anywhere in the specification or drawings. Furthermore, all combinations of the present embodiment and modifications are also included within the scope of the present disclosure. Furthermore, the configurations and operations of server systems, devices, information processing systems, etc. are not limited to those described in the present embodiment, and various modifications are possible. [Explanation of symbols]

[0258] 10...information processing system, 100...server system, 110...processing unit, 111...capability information acquisition unit, 112...scene information acquisition unit, 113...device type information acquisition unit, 114...communication processing unit, 120...storage unit, 121...user information, 122...device information, 123...application information, 130...communication unit, 200...device, 210...processing unit, 220...storage unit, 230...communication unit, 240...display unit, 250...operation unit, 300...gateway, 410...imaging device, 420...bedside sensor, 430...detection output device, 440...seat sensor, 441...cushion, 442...control box, 450...terminal device, 460...choking detection device, 461...throat microphone, 462...terminal device, 470...bed position detection device, 471...first terminal device, 472...second terminal device, 473...display, 480...glasses-type device, 510...reclining wheelchair, 520...nursing bed, 610...bed, 620...mattress, 630...wheelchair, IM1...output image, ReD...diaper area, Se1 to Se4...pressure sensors

Claims

1. a device that operates in any one of a plurality of operation modes and is used to assist a person being assisted; a server system connected to the device via a network; Including, The server system includes: determining ability information representing the activity ability of the person being assisted based on the sensing data transmitted from the device, and transmitting the determined ability information to the device; The device comprises: determining, based on the ability information, to operate in an operation mode among the plurality of operation modes that reduces the risk to the person being assisted; An information processing system in which the ability information includes information regarding the risk of the person being assisted.

2. In claim 1, The server system includes: transmitting scene information identifying a scene in which the person being assisted is being assisted to the device; The device comprises: An information processing system that determines in which of the plurality of operation modes to operate based on the capability information and the scene information.

3. In claim 1 or 2, The server system includes: transmitting, to the device, device type information that identifies the type of a concurrent device that is used to assist the same person as the device; The device comprises: An information processing system that determines in which of the plurality of operation modes to operate based on the capability information and the device type information.

4. In claim 3, The server system includes: acquiring mode information that identifies an operation mode of the combined device that operates in one of a plurality of operation modes, and transmitting the device type information and the mode information to the combined device; The device comprises: an information processing system that determines in which of the plurality of operation modes to operate based on the capability information, the device type information, and the mode information;

5. In claim 1 or 2, an information processing system in which a first device among the devices is set to an operating mode in which, when the capability value represented by the capability information is in a range above a predetermined threshold, the number of functions used increases as the capability value decreases, and when the capability value is in a range below the predetermined threshold, the first device is set to an operating mode in which fewer functions are used compared to a range above the predetermined threshold.

6. In claim 1 or 2, An information processing system in which the risk to the person being assisted is one of the risk of falling, the risk of falling off a floor, the risk of pneumonia, and the risk of bedsores.

7. a communication unit that operates in one of a plurality of operation modes and communicates with a device used to assist the person being assisted; a processing unit that performs processing to obtain ability information representing the activity ability of the person being assisted based on the sensing data transmitted from the device; Including, The processing unit transmitting the capability information to the device via the communication unit as information for determining whether the device should operate in an operation mode that reduces a risk to the person being assisted, among the plurality of operation modes; An information processing device in which the ability information includes information regarding the risk of the person being assisted.

8. An information processing method in an information processing system including a device that operates in any one of a plurality of operation modes and is used to assist a person being assisted, and a server system that is connected to the device via a network, The server system, determining ability information representing the activity ability of the person being assisted based on the sensing data acquired by the device; determining whether the device will operate in an operation mode among the plurality of operation modes that reduces risk to the person being cared for based on the obtained capability information; The capacity information includes information regarding the risk of the person being assisted. Information processing methods.

Images