Information processing system and control method

Abstract

This information processing system includes: a first device used for assistance of a person to be assisted; a second device used in coordination with the first device; and a server system. The server system stores association information in which a plurality of devices used in coordination are associated with each other. The first device, upon acquiring trigger information indicating start of assistance, transmits a coordination request to the second device that has been determined as a coordination target device on the basis of the association information. The first device and the second device start processing related to assistance after executing preprocessing concerning the association information in accordance with a connection state with the server system.

Description

Information processing system and control method

[0001] The present invention relates to an information processing system, a control method, etc. This application claims priority to Japanese Patent Application No. 2023-129301, filed in Japan on August 8, 2023, the contents of which are incorporated herein by reference.

[0002] BACKGROUND ART Conventionally, systems for managing a user's health condition, etc. are known. For example, Patent Document 1 discloses a system for sharing real-time measurement data of a user (patient).

[0003] Special table 2019-503017 publication

[0004] The present invention provides an information processing system and a control method suitable for fields such as caregiving.

[0005] In one aspect of the present disclosure, a system includes a first device used to assist a person being assisted, a second device used in conjunction with the first device in the assistance of the person being assisted, and a server system connected to the first device and the second device, wherein the server system stores correspondence information that associates multiple devices used in conjunction with the first device in the assistance of the person being assisted, and when the first device acquires trigger information indicating the start of the assistance, the first device determines a device to be linked based on the correspondence information and sends a link request to the second device determined to be the device to be linked, and the first device and the second device are related to an information processing system that performs preprocessing related to the correspondence information depending on the connection status with the server system and then starts processing related to the assistance.

[0006] Another aspect of the present disclosure relates to a control method for an information processing system including a first device used to assist a person being assisted, a second device used in conjunction with the first device in the assistance of the person being assisted, and a server system connected to the first device and the second device, wherein correspondence information correlating multiple devices used in conjunction with the assistance of the person being assisted is stored in the server system, and when trigger information indicating the start of the assistance is acquired in the first device, a device to be linked based on the correspondence information is determined, a link request is sent to the second device determined to be the device to be linked, preprocessing related to the correspondence information is performed depending on the connection status of the first device and the second device with the server system, and then processing related to the assistance is started.

[0007] 1 is a diagram illustrating an example of the configuration of an information processing system. FIG. 1 is a diagram illustrating an example of the configuration of a server system. FIG. 2 is a diagram illustrating an example of the configuration of association information (table data). FIG. 3 is a diagram illustrating an example of the configuration of a first device. FIG. 4 is an example of teacher data for a positioning application. FIG. 5 is a specific example of a usage mode of a positioning application. FIG. 6 is an example of a seat sensor disposed on a wheelchair. FIG. 7 is an example of a choking choke detection device. FIG. 8 is an example of a screen displaying the processing results of the choking choke detection device. FIG. 9 is a sequence diagram illustrating periodic processing of an information processing system. FIG. 10 is a sequence diagram illustrating cooperative processing of an information processing system. FIG. 11 is an example of a screen displayed on the display unit of a first device. FIG. 12 is an example of a screen displayed on the display unit of a first device. FIG. 13 is a sequence diagram illustrating periodic processing of an information processing system. FIG. 14 is a sequence diagram illustrating cooperative processing of an information processing system. FIG. 15 is an example of a screen displayed on the display unit of a first device. FIG. 16 is an example of a display screen for a positioning application using a smartphone. FIG. 17 is an example of a display screen for a positioning application using a smartphone. FIG. 18 is an example of a display screen in the setting mode of the positioning application. FIG. 19 is an example of a display screen in the setting mode of the positioning application. 1 is an example of a dashboard screen of a positioning application. FIG. 2 is an example of a dashboard screen of a positioning application. FIG. 3 is an example of a dashboard screen of a positioning application. FIG. 4 is an example of a dashboard screen of a positioning application. FIG. 5 is an example of a dashboard screen of a positioning application. FIG. 6 is an example of a dashboard screen of a positioning application. FIG. 7 is a diagram showing an example of the configuration of a MAC frame. FIG. 8 is a diagram showing an example of the configuration of a frame body. FIG. 9 is a diagram showing an example of the relationship between data type ID and contents. FIG. 10 is a diagram explaining an information processing system of a prior application.

[0008] Hereinafter, the present embodiment will be described with reference to the drawings. In the drawings, identical or equivalent elements are designated by the same reference numerals, and duplicate explanations will be omitted. Note that the present embodiment described below does not unduly limit the content described in the claims. Furthermore, not all of the configurations described in the present embodiment are necessarily essential components of the present disclosure.

[0009] The process leading to this embodiment will be outlined below. The information processing system of the prior application will be described with reference to FIG. 26. FIG. 26 is a diagram showing an example of the relationship between devices 40 and tacit knowledge (referring to a care application, described later). Five devices 40a to 40e are shown as examples of devices 40 connected to the server system 30. In the example of FIG. 2, two tacit knowledge items, tacit knowledge 1 and tacit knowledge 2, for addressing, for example, the risk of tripping or falling, are associated with the device 40a. For example, the applications for tacit knowledge 1 and tacit knowledge 2 are already installed on the device 40a. The processing results corresponding to tacit knowledge 1 and tacit knowledge 2 are transmitted to, for example, the server system 30.

[0010] Devices 40b and 40c are, for example, devices related to aspiration risk, and each is associated with multiple pieces of tacit knowledge for dealing with aspiration risk, etc. Devices 40d and 40e are, for example, devices related to bedsore risk, and each is associated with multiple pieces of tacit knowledge for dealing with bedsore risk, etc.

[0011] In the information processing system of the prior application, consideration is given to switching the tacit knowledge to be used depending on the situation of the person being assisted, etc. In the information processing system of the prior application, initially only device 40a is used for the person being assisted based on the situation of the person being assisted, but there are cases where the situation of the person being assisted changes and the device 40a and the device 40b start to be used in conjunction with each other.

[0012] In this case, even if the information processing system of the prior application itself detects a change in the status of the person being assisted, devices 40a and 40b may not be able to recognize the change in the status of the person being assisted because devices 40a and 40b are not always in communication with server system 30. In other words, neither the status of the person being assisted recorded in, for example, the memory of device 40a nor the status of the person being assisted recorded in device 40b necessarily holds the latest status of the person being assisted, and furthermore, the status of the person being assisted recorded in device 40a may differ from the status of the person being assisted recorded in device 40b.

[0013] Therefore, taking into consideration the circumstances of the information processing systems of the above-mentioned prior applications, the inventor has arrived at the information processing system 1 of this embodiment in order to achieve cooperation (or synchronization) so that each device operates according to the latest status of the person being assisted, whether or not it is communicating with the server system.

[0014] 1. System Configuration Example Fig. 1 shows a configuration example of an information processing system 1 according to this embodiment. The information processing system 1 includes a server system 10, a first device 100, and a second device 200. The server system 10 is connected to the first device 100 and the second device 200 via a network. The first device 100 and the second device 200 may also be connected directly without going through the server system 10 (without going through a network used for communication with the server system 10). However, the configuration of the information processing system 1 is not limited to that shown in Fig. 1, and various modifications are possible, such as omitting some components or adding other components.

[0015] The information processing system 1 according to this embodiment digitizes the tacit knowledge of experienced caregivers, such as those providing care in medical facilities or nursing homes, which is typically performed based on their "intuition" or "tacit knowledge" (hereinafter simply referred to as tacit knowledge), thereby supporting other caregivers in providing appropriate care regardless of their level of expertise. For example, the tacit knowledge of an experienced caregiver is realized as application software. Applications using tacit knowledge include applications that include decision criteria or algorithms used in providing care in medical facilities or nursing homes. These algorithms may include decisions based on trained models using machine learning or deep learning.

[0016] Hereafter, application software will be referred to simply as application. Applications that use tacit knowledge will be referred to as care applications.

[0017] For example, in the information processing system 1 of this embodiment, the server system 10 may store and manage care applications and transmit the care applications to the first device 100 and the second device 200 as needed. In this way, management of care applications can be centralized using the server system 10, which has a relatively high processing capacity. Furthermore, by transmitting the care applications to the first device 100 and the second device 200, a caregiver using each device can provide care similar to that of a skilled caregiver.

[0018] However, the application running on at least one of the first device 100 and the second device 200 is not limited to digitized tacit knowledge, and may be software that supports assistance by a caregiver without using tacit knowledge. Below, an example will be described in which a care application runs on both the first device 100 and the second device 200, but the application on at least one of the devices can be replaced with an application that does not use tacit knowledge.

[0019] In this embodiment, the caregiver is a person who provides care to the person being assisted, and includes, for example, a care manager, a caregiver, a visiting helper, etc. In addition, in home care, the family of the person being assisted may also provide care to the person being assisted, so the caregiver in this embodiment may also include the family of the person being assisted. In addition, the person being assisted in this embodiment may be a patient suffering from some kind of illness, and the caregiver may include medical professionals such as doctors, nurses, and pharmacists.

[0020] The information processing system 1 and each device included in the information processing system 1 will be described in detail below. The server system 10 is connected to the first device 100 and the second device 200 via, for example, a network. The network here may be a public communication network such as the Internet, or a network such as a LAN. For example, the server system is connected to the first device 100 and the second device 200 using a communication method conforming to the IEEE 802.11 standard. However, various modifications can be made to the communication method between the devices.

[0021] The server system 10 may be a single server, or may include multiple servers. For example, the server system 10 may include a database server and an application server. The database server stores various data such as association information, which will be described later using FIG. 3. The application server performs processing, which will be described later using FIG. 9, etc. Note that the multiple servers here may be physical servers or virtual servers. Furthermore, if a virtual server is used, the virtual server may be provided on a single physical server, or may be distributed across multiple physical servers. As described above, the specific configuration of the server system 10 in this embodiment can be modified in various ways.

[0022] The first device 100 and the second device 200 are devices used to assist a person being assisted (a patient) in, for example, a medical facility or a nursing care facility. In particular, the first device 100 and the second device 200 may be apparatuses used in cooperation with each other to provide specific assistance to a person being assisted.

[0023] 1 illustrates smartphones as the first device 100 and the second device 200. For example, at least one of the first device 100 and the second device 200 may be a device on which a positioning application, which will be described later with reference to FIGS. 5A and 5B , runs. The positioning application is an example of a care application.

[0024] However, the first device 100 and the second device 200 in this embodiment may be any device used in care and are not limited to smartphones. For example, the first device 100 and the second device 200 may be a seat sensor 440 described later with reference to Fig. 6 or a choking scare detection device 460 described later with reference to Fig. 7. The first device 100 and the second device 200 may be a detection device that detects information such as sleep, heart rate, and breathing based on the body movements of the person being assisted in bed, or may be other devices.

[0025] 2 is a block diagram showing a detailed configuration example of the server system 10. The server system 10 includes, for example, a processing unit 11, a storage unit 12, and a communication unit 13.

[0026] The processing unit 11 of this embodiment is configured by the following hardware. The hardware can include at least one of a circuit for processing digital signals and a circuit for processing analog signals. For example, the hardware can be configured by one or more circuit devices or one or more circuit elements mounted on a circuit board. The one or more circuit devices are, for example, an integrated circuit (IC), a field-programmable gate array (FPGA), etc. The one or more circuit elements are, for example, a resistor, a capacitor, etc.

[0027] The processing unit 11 may also be implemented by the following processor. The server system 10 of this embodiment includes a memory that stores information and a processor that operates based on the information stored in the memory. The information may be, for example, a program and various data. The memory may be the storage unit 12 or another memory. The processor includes hardware. Various processors, such as a central processing unit (CPU), a graphics processing unit (GPU), or a digital signal processor (DSP), may be used as the processor. The memory may be a semiconductor memory such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory, or may be a register, a magnetic storage device such as a hard disk drive (HDD), or an optical storage device such as an optical disk drive. For example, the memory stores computer-readable instructions, and the processor executes the instructions to realize the functions of the processing unit 11 as processing. The instructions may be instructions from an instruction set that constitutes a program, or instructions that instruct the processor's hardware circuitry to operate.

[0028] The processing unit 11 includes, for example, a capability information acquisition unit 14 , a scene information acquisition unit 15 , and a correspondence information processing unit 16 .

[0029] The ability information acquisition unit 14 performs a process of acquiring ability information representing the activity ability of the person being assisted. For example, the ability information acquisition unit 14 acquires sensing data from at least one of the first device 100 and the second device 200. The sensing data here may be output values ​​of sensors included in each device, or may be processing results based on the output values. For example, the sensing data may include log data of a nursing care application for a predetermined period of time.

[0030] For example, the ability information acquisition unit 14 determines time-series changes in the condition of the person being assisted based on the sensing data and estimates the ability information based on the changes. The ability information here may be index information related to activities of daily living (ADL). Various methods for assessing ADL, such as the Barthel Index, are known, and these methods can be widely applied in this embodiment. Furthermore, the ability information may be a nine-level index disclosed as the Clinical Frailty Scale in "Frailty and the potential kidney transplant recipient: time for a more holistic assessment?" by Henry H.L. Wu et al. For example, the ability information acquisition unit 14 in this embodiment determines which of the nine levels the person being assisted belongs to based on the sensing data.

[0031] The capability information may be obtained in the first device 100 or the like. The capability information acquisition unit 14 may execute a process of acquiring the capability information from the first device 100 or the like via the communication unit 13.

[0032] The scene information acquisition unit 15 determines a scene in which assistance is to be provided to the person being assisted. The scene information here may be information specifying the type of assistance to be provided, such as meal assistance, toilet assistance, or movement / transfer assistance. The scene information may also be information about the assistants providing assistance to the person being assisted, such as the number and skill level of the assistants. The scene information may also be information about the person being assisted, such as the attributes of the person being assisted. For example, the scene information acquisition unit 15 performs processing to obtain scene information based on user input, the schedule of the assistant, or the like. The scene information may also be information about the location where assistance is to be provided, as shown in FIG. 3.

[0033] The association information processing unit 16 performs processes related to the creation, updating, and transmission of association information. Fig. 3 shows an example of association information. The association information in this embodiment may include information identifying the person being assisted, ability information of the person being assisted, information identifying the assistance scene, information identifying the type of assistance to be performed, information identifying the device used for assistance, information identifying tacit knowledge used for assistance, and information on the update date and time of the association information. The association information may also include information on the expiration date of the association information (not shown in Fig. 3). The association information is, for example, table data (or, in a narrow sense, a relational database) in which item values ​​are associated with each item.

[0034] However, the configuration of the correspondence information is not limited to the example in Figure 3, and some items may be omitted or other items may be added. Furthermore, the correspondence information is not limited to table data, and other formats of data may be used. Furthermore, while Figure 3 illustrates four rows of table data, the item values ​​in each row are merely an example, and various modifications of the specific item values ​​are possible, as described below.

[0035] The information identifying the person being assisted is, for example, an assisted person ID, which is an ID uniquely assigned to the person being assisted. The assisted person ID is associated with information such as the person's name, address, and facility where they reside. The assisted person ID may also be associated with attribute information of the person being assisted, such as age, sex, height, weight, medical history, and medication history. Hereinafter, the information associated with the assisted person ID is also referred to as user information.

[0036] The ability information of the person being assisted corresponds to the ability information acquired by the ability information acquisition unit 14. The assistance scene corresponds to the scene information acquired by the scene information acquisition unit 15.

[0037] The information specifying the type of assistance to be performed is information specifying specific assistance content, such as, for example, meal assistance, excretion assistance, mobility assistance, etc. Note that meal assistance and excretion assistance may be further subdivided.

[0038] The information identifying the device used for assistance may be information (type ID) that identifies the type of device, such as a "smartphone," "choking detection device," or "seat sensor." Alternatively, the information identifying the device may be information (device ID) that identifies one of multiple devices belonging to a specific type. The device ID is information that uniquely identifies a device, such as a serial number or MAC address.

[0039] The information for identifying the tacit knowledge used for the care is, for example, an application ID, which is an ID uniquely assigned to each of a plurality of care applications.

[0040] The update date and time is information indicating when the target association information was updated, and is, for example, information specifying the year, month, and day. The update date and time may also include hour and minute information. For example, if the association information is table data with the configuration shown in FIG. 3, the data for each row includes, as an update date and time field value, information indicating when the row was newly added or when at least one of the field values ​​for the row was updated. Note that, although an example in which an update date and time is associated with each row of data has been shown here, this is not limiting. For example, the association information may include one update date and time information for the entire association information. In this case, the update date and time indicates when the last update process was performed on any row of the association information.

[0041] As described above, the association information may also include expiration date information (not shown in FIG. 3). The expiration date is information that indicates the period during which the target data is valid (or, more narrowly, the timing at which the data becomes invalid). For example, the expiration date is information that is used by a device (the first device 100 or the second device 200) that received the association information to determine whether or not the association information may be used. Here, the expiration date may be, for example, a value of expiration data, which will be described later with reference to FIG. 24. Details of the expiration date will be described later. The expiration date may also be data that is set for each row of the association information, or may be data that is set only once for all of the association information received by the first device 100 or the like.

[0042] As described above, the update date and time is information indicating the timing of the update and is used, for example, for version management (described later using steps S203-S207 in FIG. 10). On the other hand, the expiration date is used to determine whether or not the received association information may be used, as described above (described later using FIGS. 23-25). Therefore, the update date and time and the expiration date may be different pieces of information. However, since the expiration date may be set based on the update deadline, one piece of information may also serve as the other piece of information, thereby omitting the other piece of information. Furthermore, the expiration date is information acquired as part of a packet, as described later using FIG. 24, and may be treated as information different from the association information (metadata of the association information).

[0043] For example, even when providing the same meal assistance, an experienced caregiver is likely to change the tacit knowledge they use depending on the person being assisted and the situation (ability, scene, etc.) of the person being assisted. In other words, tacit knowledge is considered not only about the content of the assistance, but also about the condition judgment for performing the assistance. In this regard, the association information processing unit 16 of this embodiment associates a device ID and an app ID with the ID of the person being assisted, the ADL, and the scene, respectively. This makes it possible not only to digitize the assistance itself performed by the experienced caregiver (create a care application), but also to appropriately digitize the usage conditions of the care application.

[0044] The correspondence information may also include information associating multiple devices used in conjunction with each other in assisting a person receiving assistance. For example, consider table data that includes a row in which the assisted person ID, ADL, scene, and assistance type are common, and a row in which the device ID is device ID1 and another row in which the device ID is device ID2. This indicates that, for one type of assistance identified by the assisted person ID, ADL, scene, and assistance type, two devices, one corresponding to device ID1 and the other corresponding to device ID2, are used in conjunction with each other. However, one row may include information on multiple device IDs used in conjunction with each other, and various modifications are possible for the data structure for associating multiple devices used in conjunction with each other.

[0045] The association information processing unit 16 performs a process of updating the association information when, for example, sensing data related to the person being assisted is acquired from the first device 100 and the second device 200. For example, when the ADL of the target person being assisted decreases, the association information processing unit 16 performs a process of adding data to the association information so that more thorough assistance can be provided than before. Specifically, the association information processing unit 16 may add to the table data a row including item values ​​of the assistance type, device ID, and app ID that were not associated with the target person being assisted.

[0046] Furthermore, when the association information processing unit 16 is connected to the first device 100 via a network, the association information processing unit 16 may perform a process of transmitting association information to the first device 100. Specific processing will be described later using FIG. 10 and other figures. In this case, by including information on the update date and time in the association information, it is possible to prevent the first device 100 from using excessively old data for processing. Similarly, when the association information processing unit 16 is connected to the second device 200 via a network, the association information processing unit 16 may perform a process of transmitting association information to the second device 200.

[0047] The storage unit 12 is a work area for the processing unit 11 and stores various information. The storage unit 12 can be realized by various types of memory, and the memory may be a semiconductor memory such as an SRAM, a DRAM, a ROM, or a flash memory, or may be a register, a magnetic storage device, or an optical storage device.

[0048] The storage unit 12 stores, for example, nursing care applications created in the server system 10 or other devices, and the association information shown in FIG. 3 . The storage unit 12 may also store user information, device information, application information, etc. As described above, user information is various information related to the person being assisted that is associated with the person being assisted ID. Device information is information such as a type ID, vendor, and app ID of an installed application that is associated with the device ID. Application information is information such as an application name and creator that is associated with the app ID.

[0049] The communication unit 13 is an interface for communication via a network, and includes, for example, an antenna, an RF (radio frequency) circuit, and a baseband circuit when the server system 10 performs wireless communication. However, the server system 10 may also perform wired communication, in which case the communication unit 13 may include a communication interface such as an Ethernet connector and a control circuit for the communication interface. The communication unit 13 may operate under the control of the processing unit 11, or may include a processor for communication control that is different from the processing unit 11.

[0050] FIG. 4 is a block diagram showing a detailed configuration example of the first device 100. The first device 100 includes, for example, a processing unit 110, a storage unit 120, a communication unit 130, a display unit 140, and an operation unit 150. As will be described later with reference to FIGS. 6 and 7 , the method of this embodiment allows devices of various configurations to be used as the first device 100. Therefore, the configuration of the first device 100 is not limited to that shown in FIG. 4 , and modifications such as omitting some components or adding other components are possible. For example, the first device 100 may have various sensors depending on the type of device, such as a motion sensor such as an acceleration sensor or a gyro sensor, an image sensor, a pressure sensor, a GPS (Global Positioning System) sensor, or the like.

[0051] The processing unit 110 is configured by hardware including at least one of a circuit for processing digital signals and a circuit for processing analog signals. The processing unit 110 may also be realized by a processor. Various types of processors, such as a CPU, a GPU, or a DSP, can be used as the processor. The processor executes instructions stored in the memory of the first device 100, thereby realizing the functions of the processing unit 110 as processing.

[0052] The storage unit 120 is a work area of ​​the processing unit 110 and is realized by various types of memory such as SRAM, DRAM, ROM, etc. For example, the storage unit 120 stores the association information and the care application acquired from the server system 10.

[0053] The communication unit 130 is an interface for communicating via a network, and includes, for example, an antenna, an RF circuit, and a baseband circuit. The communication unit 130 communicates with the server system 10 via, for example, the network. The communication unit 130 may perform wireless communication conforming to the IEEE 802.11 standard with a gateway (relay device) such as a router, and communicate with the server system 10 via the gateway.

[0054] The communication unit 130 of the first device 100 may include a first communication unit that communicates with the server system 10 and a second communication unit that communicates with the second device 200. The first communication unit and the second communication unit may use different communication methods. For example, the first communication unit connects to the server system 10 via a network such as a LAN or the Internet using communication in accordance with the IEEE 802.11 standard. The second communication unit connects to the second device 200 using short-range wireless communication such as Bluetooth (registered trademark) or ZigBee.

[0055] For example, when a care application is used on the first device 100, the first device 100 communicates with the server system 10 via a network using the first communication unit. The server system 10 acquires information used for authentication from the first device 100 and performs authentication. As a result, when login authentication is successful, the first device 100 accesses the care application.

[0056] When login authentication is performed through communication with the server system 10 , the first device 100 temporarily stores information used for authentication in the storage unit 120 .

[0057] For example, the first device 100 independently controls the operation of the first communication unit and the operation of the second communication unit. For example, the first device 100 may turn on the second communication unit when the first communication unit is off or in standby mode. In this case, the first device 100 is not connected to the server system 10 but is connected to the second device 200. By reducing the frequency of communication with the server system 10, the processing load on the server system 10 and the communication load on the network can be reduced. Furthermore, even when the first device 10 is not connected to the server system 10, the first device 100 can connect to the second device 200, thereby enabling collaborative processing related to assistance. While the above description has been given of an example in which the first communication unit and the second communication unit use different communication methods, this is not limiting. For example, the first communication unit and the second communication unit may use a common communication method, and the connection / disconnection with the server system 10 and the connection / disconnection with the second device 200 can be independently controlled.

[0058] For example, when the first device 100 is offline or in standby mode and uses a care application, the first device 100 is not connected to the network. In this case, the first device 100 performs authentication based on the information used for authentication stored in the storage unit 120, and the first device 100 accesses the care application. When the first device 100 is logged in to the care application, the first device 100 is connected to the second device 200 via the second communication unit.

[0059] The display unit 140 is an interface that displays various information and may be a liquid crystal display, an organic EL display, or another type of display. The operation unit 150 is an interface that accepts user operations. The operation unit 150 may be a button or the like provided on the first device 100. The display unit 140 and the operation unit 150 may also be a touch panel that is integrally configured.

[0060] The first device 100 may also include components not shown in FIG. 4 . For example, the first device 100 has an imaging unit (image sensor) that captures an image of a predetermined imaging range and outputs image information. The image information here may be a still image or a moving image. The image information may also be color or monochrome. The imaging unit may also include a depth sensor that detects the distance to the subject, or a sensor (e.g., an infrared sensor) that detects the heat of the subject. The first device 100 may also include a light-emitting unit, a vibration unit, a sound input unit, a sound output unit, etc. The light-emitting unit is, for example, an LED (light emitting diode) and provides notification by light emission. The vibration unit is, for example, a motor and provides notification by vibration. The sound input unit is, for example, a microphone. The sound output unit is, for example, a speaker and provides notification by sound.

[0061] The same applies to the second device 200, which includes, for example, a processing unit, a storage unit, a communication unit, a display unit, and an operation unit. The communication unit of the second device 200 may also be capable of independently controlling communication with the server system 10 and communication with the first device 100. Also, like the first device 100, various types of devices, which will be described later with reference to Figures 6 and 7, can be used as the second device 200, and the configuration of the second device 200 can be modified in various ways.

[0062] As described above, the information processing system 1 according to this embodiment includes the first device 100 used to assist the person being assisted, the second device 200 used in cooperation with the first device 100 when assisting the person being assisted, and the server system 10 connected to the first device 100 and the second device 200. As described above with reference to Fig. 3 , the server system 10 stores association information that associates a plurality of devices used in cooperation when assisting the person being assisted.

[0063] In this embodiment, when the first device 100 acquires trigger information indicating the start of assistance, it may determine a target device for collaboration based on the association information and transmit a collaboration request to the second device determined to be a target device for collaboration. The collaboration request corresponds to, for example, step S209 in FIG. 10 (described later). The first device 100 and the second device 200 then perform preprocessing related to the association information depending on the connection status with the server system 10, and then start the assistance-related processing. The preprocessing may be an update or determination process of the association information used in the collaboration processing, and the specific processing content varies depending on the connection status. For example, in the example shown in steps S204-S217 in FIG. 10 (described later) or steps S404-S418 in FIG. 14 (described later), the preprocessing involves updating the association information used in the collaboration processing. Note that this update process may include a process of inquiring whether an update is necessary and, if an update is not necessary, using the existing association information as is. Furthermore, in the example shown in steps S504-S506 in FIG. 15 (described later), the preprocessing involves determining the association information used in the collaboration processing.

[0064] For example, in a nursing care facility, the first device 100 and the second device 200 (e.g., a smartphone used by a caregiver or a device described later with reference to Figures 6 and 7) may not always be connected to the server system 10 via a network. For example, these devices may connect to the server system 10 for a limited period, such as when transmitting log data of operation results, and may be disconnected from the server system 10 at other times. Even in this case, if the first device 100 and the second device 200 acquire association information in advance, they can identify the associated device and the tacit knowledge to be used based on the association information. Furthermore, if the first device 100 and the second device 200 are directly connected as described above, multiple devices can operate in cooperation with each other. By shortening the connection period with the server system 10, communication load and power consumption can be reduced.

[0065] For example, in a nursing home or the like, various medical devices are connected to electronic medical records and nursing software via a network. Repeaters for connecting to the network are installed in various locations, such as in the nursing home. Therefore, when connecting the first device 100 and the second device 200 to the network, these devices must also be connected to the repeater. As the number of medical devices and the like connected to the repeater increases, the communication load and processing load become significant. Therefore, the first device 100 and the second device 200 may automatically disconnect from the server system 10 upon completion of a predetermined process, such as login authentication or transmitting log data of operation results.

[0066] However, because the assistance to be performed and the tacit knowledge used in the assistance change depending on the ability information of the person being assisted, the association information may be updated as described above. Therefore, if the first device 100 is not connected to the server system 10, it may not be able to refer to the latest association information depending on the timing. The same applies to the second device 200.

[0067] In this regard, in this embodiment, pre-processing is performed according to the connection state with the server system 10, and then the first device 100 and the second device 200 start cooperative processing. Therefore, it is possible to perform appropriate cooperative processing according to the situation in each of the following cases: when both devices are connected to the server system 10 (for example, FIG. 10 described later), when only one of the devices is connected to the server system 10 (for example, FIG. 14 described later), and when neither device is connected to the server system 10 (for example, FIG. 15 described later). Details of the processing will be described later.

[0068] In the above description, an example has been described in which the information processing system 1 includes the first device 100 and the second device 200, and these two devices perform cooperative processing. However, the information processing system 1 may include three or more devices. Furthermore, cooperative processing related to a specific care may involve three or more devices working in cooperation with each other.

[0069] Furthermore, some or all of the processes performed by the information processing system 1 of this embodiment may be realized by a program. In a narrow sense, the processes performed by the information processing system 1 include the processes performed by the processing unit 11 of the server system 10, the processing unit 110 of the first device 100, and the processing unit of the second device 200, but one or two of these processes may be omitted.

[0070] The program according to this embodiment can be stored in, for example, a non-transitory information storage medium (information storage device), which is a medium readable by a computer. The information storage medium can be realized, for example, by an optical disc, a memory card, a HDD, or a semiconductor memory. The semiconductor memory is, for example, a ROM. The processing unit 110 and the like perform various processes of this embodiment based on the program stored in the information storage medium. In other words, the information storage medium stores a program for causing a computer to function as the processing unit 110 and the like. A computer is a device equipped with an input device, a processing unit, a storage unit, and an output unit. Specifically, the program according to this embodiment is a program for causing a computer to execute each step described below using Figures 9-10, 13-15, etc.

[0071] The technique of this embodiment can also be applied to a control method including the following steps. The control method according to this embodiment is a control method for an information processing system including a first device used to assist a person being assisted, a second device used in cooperation with the first device when assisting the person being assisted, and a server system connected to the first and second devices. The control method includes the steps of: storing, in the server system, correspondence information that associates multiple devices used in cooperation when assisting the person being assisted; determining, when trigger information indicating the start of assistance is acquired in the first device, a device to be linked based on the correspondence information and transmitting a link request to the second device determined to be the device to be linked; and performing preprocessing according to the connection states of the first and second devices and the server system, and then starting processing related to assistance.

[0072] 2. Specific Examples of First Device and Second Device Next, specific examples of the first device 100 and the second device 200 will be described. In this embodiment, the master device that detects a trigger for the coordination process is referred to as the first device 100, and the slave device that performs the coordination process based on a coordination request from the first device 100 is referred to as the second device 200. Each device described below may function as either the first device 100 or the second device 200. Furthermore, the first device 100 or the second device 200 does not necessarily have to be fixed. That is, each device described below may function as the first device 100 for one type of assistance and as the second device 200 for another type of assistance. Furthermore, the first device 100 and the second device 200 are not limited to the devices described below and may include a device that detects food intake, a device that detects whether the assisted person is inserting their hand into their clothing (pants or underwear) and trying to touch the skin or the inside of the clothing (trying to tamper with the clothing), or the like. Furthermore, the devices described below may perform different processes by operating according to different applications. For example, the choking detection device 460 described below may function as a medication management device that manages medication administration by a person receiving care by detecting the swallowing of medicine.

[0073] 2.1 Smartphone or PC (Positioning Application) For example, at least one of the first device 100 and the second device 200 may be a smartphone or a PC. A positioning application may run on the smartphone or PC as a care application.

[0074] A positioning application is an application that processes the position or posture of at least one of a person and an object during care. The positioning application may be used to adjust the posture of a person being assisted in a bed, or may be used to adjust the posture of a person being assisted in a wheelchair. For example, the positioning application may operate in a setting mode for setting the position and a usage mode for supporting actual position adjustment according to the setting. For example, in the setting mode, the positioning application acquires training data by capturing an image of a person or object at a desired position based on the operation of an expert. Then, in the operation mode, the positioning application superimposes the training data, which has been subjected to transparency processing, on the captured image of the person or object to be adjusted.

[0075] FIG. 5A shows an example of training data acquired in the setting mode. In the example of FIG. 5A, image information representing the desired posture of the person being assisted when lying on a bed is acquired as training data. Note that in the example shown in FIG. 5A, the training data may be an image of a cushion, with a triangle or square displayed at a position corresponding to the cushion as additional information. The shape may be input, for example, by a user (such as an administrator, as described below with reference to FIGS. 18-19B) who sets the training data. FIG. 5B shows the usage status of the positioning application in the usage mode and a specific example of a display screen. For example, a device such as a smartphone running the positioning application may superimpose the training data shown in FIG. 5A on a captured image of a person being assisted whose position is to be adjusted. For example, the positioning application may apply transparency processing to the training data and then superimpose the training data after the transparency processing (corresponding to the dashed line in FIG. 5B) on the captured image. The caregiver assists the person being assisted so that the person in the captured image approaches the training data. This makes it possible to appropriately support the caregiver's position adjustment.

[0076] While the above example illustrates the superimposition of image information, i.e., training data, on the display, the positioning application may output a judgment result (OK / NG) indicating whether the posture of the person being assisted is appropriate. For example, the positioning application may determine whether the posture of the person being assisted or the caregiver is appropriate based on the degree of similarity between the image captured during position adjustment and the training data, and output the judgment result. Specifically, the positioning application may determine whether the posture of the person being assisted or the caregiver is appropriate, or whether the position of a cushion or other equipment is appropriate. The judgment of whether the posture is appropriate or not may be based on skeletal information of the person being assisted contained in the training data and skeletal information of the person being assisted contained in the image captured during position adjustment. For example, as will be described later with reference to FIG. 19B, the skeletal information includes information on several feature points, making it possible to identify the coordinate values ​​of the feature points. The coordinate values ​​here are x and y coordinates representing the vertical and horizontal positions of the image. Furthermore, if depth information is acquired, the coordinate values ​​may include a z coordinate representing the distance from the camera. The positioning application determines whether the degree of difference between the coordinate values ​​of the training data and each of the feature points included in the skeletal information during position adjustment is equal to or greater than a predetermined value, and determines the adjustment as NG if the number of points with large differences exceeds a threshold.

[0077] The positioning application may also perform processing to display specific points determined to be NG. For example, the positioning application may compare the captured image with training data and highlight areas determined to have large differences. Alternatively, the positioning application may output specific instructions, such as how to change the position of the cushion or how to change the posture. The areas determined to have large differences may be based on skeletal information of the person being assisted contained in the training data and skeletal information of the person being assisted contained in images captured during position adjustment. For example, the positioning application may determine the coordinate values ​​of the above-mentioned feature points and highlight points with a difference greater than a predetermined value as points with large differences. For example, the black frame shown in FIG. 5B and the feature points corresponding to the ankles correspond to the highlighting results. The rightward arrow in FIG. 5B indicates information indicating the direction of posture change.

[0078] The positioning application may also accept additional information, such as points considered important by an expert, in the setting mode and present the additional information in the use mode. The additional information may include information such as the position and angle of specific body parts, the presence, size, and softness of a cushion, etc. The positioning application may also recommend appropriate welfare equipment for the user based on the results of posture assessment of the user. Even if the user is already using a cushion, the application may recommend a new cushion if the cushion's position is not properly adjusted or if the cushion's size differs from the size of the cushion included in the training data. The recommendation may specify the size, hardness, etc. of the cushion, or a specific product (e.g., model number). When a specific product is specified, the positioning application may output a link to an e-commerce site or other website selling the product.

[0079] The positioning application may also display a questionnaire about the set teacher data when setting the teacher data in the setting mode. This questionnaire about the teacher data is conducted to obtain information necessary for interpreting the teacher data, such as the purpose of setting the teacher data (e.g., for pressure ulcer treatment, pressure relief) and the target area for the pressure ulcer treatment.

[0080] The positioning application may accumulate training data by prompting the caregiver to manually enter information into a questionnaire at the start of service. When a predetermined number of pieces of training data have been accumulated, the positioning application performs a learning process based on the training data. The learning process is performed based on training data that includes both training data and the results of the questionnaire. As a result, the trained model generated by the learning process can output appropriate training data for the purpose of setting the training data and for each target body part. For example, in the case of bed positioning targeting a certain body part, the learning model can determine the size of a cushion to be placed where and the posture of the person being assisted. In other words, after creating a trained model, it is possible to confirm optimal positioning in the position app by simply selecting the target body part without setting training data. Furthermore, when starting service, training data may be accumulated by prompting the caregiver to manually enter desired welfare equipment. When a predetermined number of pieces of training data have been accumulated, the positioning application may perform a learning process based on the training data to create a trained model that recommends welfare equipment.

[0081] Specific examples of the setting mode of the positioning application will be described later with reference to Figures 17A to 19B. Specific examples of the dashboard screen when presenting the processing results of the positioning application to the user will be described later with reference to Figures 20 to 22C.

[0082] Furthermore, various modifications are possible regarding how application A used to adjust the position of person being assisted AAA and application B used to adjust the position of person being assisted BBB are managed and used. For example, application A and application B may be treated as different applications in the first place. For example, on a smartphone, each application is launched by selecting a different icon.

[0083] Alternatively, application A and application B may share a common interface application and be switchable within the interface application. For example, on a smartphone, an icon of a positioning application may be selected first, and then application A and application B may be switched by selecting a tab within the positioning application.

[0084] Note that application A and application B may share the same processing (algorithm) but use different parameters for the processing, or both the processing and the parameters may be different. Furthermore, even if they are the same application A, application A1, which cooperates with other applications (devices), and application A2, which does not cooperate with other applications, may be treated as different applications. Furthermore, the management and usage of applications described above is not limited to positioning applications, and the same applies to other applications, such as the choking hazard application used in choking hazard detection device 460.

[0085] 6 is an example of at least one of the first device 100 and the second device 200, and is a diagram showing a seat sensor 440 arranged on the seat of a wheelchair 630. The seat sensor 440 includes a pressure sensor that outputs a pressure value, and outputs the pressure value.

[0086] In the example of FIG. 6, four pressure sensors Se1 to Se4 are disposed on the underside of a cushion 441 placed on the seat of a wheelchair 630. Pressure sensor Se1 is disposed in the front, pressure sensor Se2 is disposed in the rear, pressure sensor Se3 is disposed on the right, and pressure sensor Se4 is disposed on the left. Note that front, back, left, and right here refer to the directions as seen from the perspective of the person being assisted when sitting in the wheelchair 630. Pressure sensors Se1 to Se4 are connected to a control box 442. The control box 442 contains a processor that controls pressure sensors Se1 to Se4 and memory that serves as the processor's work area. The processor detects pressure values ​​by operating pressure sensors Se1 to Se4.

[0087] The seat sensor 440 may determine whether the posture of the person being assisted when sitting in the wheelchair 630 (hereinafter also referred to as "sitting posture") is normal, shifted forward, shifted to the side, or any of a number of other postures, based on the pressure values ​​from the pressure sensors Se1 to Se4. A "forward shift" refers to a state in which the user's center of gravity is shifted further forward than normal, while a "lateral shift" refers to a state in which the user's center of gravity is shifted to either the left or right. For example, the seat sensor 440 determines a "forward shift" when the value of the pressure sensor Se1 increases by a predetermined amount compared to the initial state, and determines a "lateral shift" when the value of the pressure sensor Se3 or Se4 increases by a predetermined amount compared to the initial state. The seat sensor 440 may also perform a fall possibility determination, which determines whether or not the person being assisted is likely to fall from the seat.

[0088] The criteria for determining whether the seat is normal, forward slippage, or sideways slippage, or the criteria for determining whether the seat is likely to fall, may vary depending on the person being assisted and the situation. For example, the seat sensor 440 may perform the above determination based on a care application. The seat sensor 440 may also add or update the care application based on communication with the server system 10.

[0089] In addition, at least one of the first device 100 and the second device 200 of this embodiment may be a mattress capable of detecting pressure, or may be a device that communicates with the mattress. These devices determine the posture of the person being assisted in bed.

[0090] 7 is a diagram illustrating a swallowing choking detection device 460 used during a meal, which is an example of at least one of the first device 100 and the second device 200. As shown in Fig. 7, the swallowing choking detection device 460 includes a throat microphone 461 worn around the neck of the person being assisted, and a terminal device 462 with a camera.

[0091] The throat microphone 461 outputs audio data resulting from swallowing, coughing, etc. of the person being assisted. The camera of the terminal device 462 outputs captured images of the person being assisted eating. The terminal device 462 is, for example, a smartphone or tablet PC placed on a table where the person being assisted eats. The throat microphone 461 is connected to the terminal device 462 using Bluetooth or the like. However, the specific connection method can be modified in various ways.

[0092] The choking detection device 460 determines whether the person being assisted has choked or swallowed based on audio data from the throat microphone 461. A device that detects swallowing using a microphone worn around the neck is described, for example, in U.S. Patent Application No. 16 / 276,768, filed February 15, 2019, entitled "Swallowing action measurement device and swallowing action support system." This patent application is incorporated by reference in its entirety. The choking detection application can detect the number of choking incidents, the duration of the choking (time of occurrence, duration, etc.), and whether or not the person swallowed, based on the audio data.

[0093] The camera of the terminal device 462 captures an image of the person being assisted from the front, as shown in Fig. 7, for example. Therefore, the choking detection application may detect the mouth and eyes of the person being assisted, as well as chopsticks and spoons used by the person being assisted, based on the captured image. Note that various methods for detecting these facial features and objects based on image processing are known, and a wide range of known methods can be applied in this embodiment.

[0094] For example, the choking detection device 460 may calculate the swallowing time from when the person being assisted opens their mouth to when they swallow, based on the swallowing detection result and the result of determining whether the person being assisted is opening or closing their mouth. In this way, if the number of swallows has decreased, for example, it is possible to determine specific circumstances, such as whether the person is not even putting food in their mouth, or whether the person has put food in their mouth but is not swallowing it. As a result, it is possible to accurately determine the risk of aspiration during meals. For example, the choking detection device 460 may make these determinations based on a care application. The choking detection device 460 may also add or update care applications based on communication with the server system 10.

[0095] 8 is an example of a display screen that displays the processing results of the choking scare detection device 460. The screen shown in FIG. 8 may be displayed on the display unit of the terminal device 462, or may be displayed on another device connected to the choking scare detection device 460.

[0096] The screen displaying the processing results of the choking hazard detection device 460 may include areas RE1 to RE6 as shown in Fig. 8. Area RE1 is an area where an image of the person being assisted captured by an imaging unit of the terminal device 462 is displayed, for example. When the choking hazard detection device 460 and a positioning application work together, area RE1 may display the processing results of the positioning application (for example, an image on which training data is superimposed as shown in Figs. 5A and 5B).

[0097] The area RE2 displays, for example, the swallowing time described above. Note that the display in the area RE2 may be reset every time the person being assisted opens their mouth, or the area RE2 may display the average time for one meal.

[0098] Area RE3 is an area used to input parameters used in the assessment process of the choking stool detection device 460. For example, the sounds detected by the throat microphone 461 change depending on the type of food the person being assisted is eating (e.g., solid food, jelly, chopped food, etc.). Therefore, the accuracy of the assessment can be improved by switching the parameters of the assessment process. The parameters of the assessment process may be, for example, parameters of a trained model used in a nursing care application. For example, area RE3 may be pre-displayed to indicate the type of food (e.g., solid food, jelly, chopped food, etc.). Selecting one of these options changes the display of the checkbox on the left. This makes it possible to easily switch parameters using the screen shown in FIG. 8 and clearly indicate the selected parameter to the user. However, the information displayed in area RE3 is not limited to this, and the assessment results of the choking stool detection device 460 may also be displayed. For example, the choking stool detection device 460 may assess the condition of the person being assisted as normal, caution, or abnormal based on the occurrence of choking, the risk level, the swallowing condition, etc., and display the result in area RE3 using color, text, etc. The result of the determination process may be displayed in a separate pop-up screen instead of in area RE3.

[0099] Region RE4 is an area where the detection results of the throat microphone are displayed. For example, the horizontal axis of region RE4 may represent time, and the vertical axis may represent the intensity (loudness) of the audio signal. For example, choking detection device 460 may display the detection results of the throat microphone in region RE4 in real time, or may display an audio waveform corresponding to each swallowing event in region RE4 each time a swallowing event is detected.

[0100] Area RE5 is an area where a camera switching button is displayed. For example, if the terminal device 462 includes multiple imaging units (an in-camera, an out-camera, an external webcam, etc.), the choking hazard detection device 460 determines which imaging unit's image to display in area RE1 based on the operation result of the camera switching button.

[0101] An execution button is displayed in the area RE6. When the execution button is operated, the choking choke detection device 460 starts the various processes described above and the screen update process shown in FIG.

[0102] 2.4 Other Detection Devices The devices of the information processing system 1 of this embodiment may include other detection devices, without being limited to the above-described smartphone, PC, seat sensor, and choking detection device. For example, the devices may include a device including an application for detecting standing up, a device including an application for estimating the amount of food consumed by a person receiving care, a device including a medication application, a device for detecting tampering by a person receiving care using a communication tag, and a feces detection device. These devices are described, for example, in U.S. Patent Application No. 18 / 381,809, filed October 19, 2023. This patent application is incorporated herein by reference in its entirety.

[0103] Other detection devices may include a sheet-type device including a pressure sensor or a mattress including a pressure sensor that can acquire vital information such as blood pressure, heart rate, breathing, body movement, and activity level.

[0104] 3. Details of Processing Next, the processing of the information processing system 1 according to this embodiment will be described for each connection state between the server system 10 and the first device 100 and the second device 200.

[0105] 3.1 When Both Devices are Connected to the Server System For example, the processing of this embodiment may include two processes: a periodic process that is periodically executed, and a coordination process in which the first device 100 and the second device 200 coordinate with each other using the association information to perform assistance-related processing. Note that the periodic process here is not necessarily limited to a process executed at a fixed interval, and various variations in the execution timing are possible. Furthermore, while the periodic process is expected to be executed, for example, when the coordination process is not being performed, it is not prevented from being executed in parallel when the coordination process is being performed. As will be described later with reference to FIG. 9 , the periodic process includes a process of transmitting data from a device such as the first device 100 to the server system 10, a process of updating the association information in the server system 10, and a process of transmitting association information and a care application from the server system 10 to the first device 100, etc.

[0106] 9 is a sequence diagram illustrating periodic processing when both the first device 100 and the second device 200 are connected to the server system 10. The information processing system 1, for example, periodically executes the processing shown in FIG. 9. For example, the periodic processing may be performed during a time when the processing load and communication load are relatively low, such as late at night.

[0107] First, in step S101, the first device 100 acquires sensing data using a sensor mounted on the first device 100. Note that the acquisition of sensing data here is not limited to a process of acquiring sensor output values ​​by executing a care application in real time, but may also be a process of reading sensing data accumulated by past executions of a care application from the storage unit 120. In step S102, the communication unit 130 of the first device 100 transmits the acquired sensing data to the server system 10. For example, the first device 100 accumulates sensing data detected after the last transmission of sensing data to the server system 10 as log data in the storage unit 120 (step S101), and transmits the log data to the server system 10 (step S102).

[0108] In step S103, the second device 200 acquires sensing data using a sensor mounted on the second device 200. In step S104, the communication unit of the second device 200 transmits the acquired sensing data to the server system 10. Similarly, in the process of step S103, the second device 200 may be executing a care application, or may read from the storage unit 120 the results of a care application that was executed in the past.

[0109] In step S105 , the processing unit 11 (capability information acquisition unit 14 ) of the server system 10 calculates the ADL of the person being assisted based on the sensing data acquired from the first device 100 and the second device 200 .

[0110] In step S106, the processing unit 11 (correspondence information processing unit 16) of the server system 10 updates the correspondence information as necessary based on the ADL calculated in step S105. If the correspondence information has been updated, the processes of steps S107 to S117 described below are executed. If the correspondence information has not been updated, the processes of steps S107 to S117 may be omitted.

[0111] In step S107, the communication unit 13 of the server system 10 transmits the updated association information to the first device 100. The information transmitted here may be the entire association information or a part of it. For example, if the association information is table data, the server system 10 may transmit all rows of the table data to the first device 100, or may transmit only some rows of the table data that are related to the first device 100 to the first device 100.

[0112] In step S108, the first device 100 stores the association information transmitted from the server system 10 in the storage unit 120. Note that, hereinafter, the association information stored in the first device 100 will also be referred to as first association information.

[0113] In step S109, the processing unit 110 of the first device 100 determines, based on the newly acquired association information, an application that may be executed on the first device 100, and determines whether the application has been installed.

[0114] If there is an application that has not been installed, in step S110, the communication unit 130 of the first device 100 requests the necessary application from the server system 10. In step S111, the server system 10 transmits the requested application to the first device 100. In step S112, the processing unit 110 of the first device 100 installs the acquired application.

[0115] The processing of steps S109-S112 installs applications that the first device 100 may execute while assisting the person being assisted. Whether or not an application is likely to be executed is determined based on the correspondence information updated in step S106. That is, the processing of steps S109-S112 enables the first device 100 to execute processing that corresponds to the state of the person being assisted at that time.

[0116] 9, since the second device 200 is also connected to the server system 10, steps S113-S117 are similar to steps S107-S112 for the second device 200. Hereinafter, the association information stored in the second device 200 will also be referred to as "second association information." The first association information and the second association information may be the same or different. For example, if the server system 10 transmits only a portion of the association information, the first association information and the second association information may be different even if the latest association information (hereinafter also referred to as "master table") in the server system is the same. Furthermore, depending on the timing of updating the association information (S106) and the timing of transmitting the association information (S107, S113), the master tables may be different in the first place.

[0117] 9 illustrates an example in which the processing unit 11 of the server system 10 updates the association information based on sensing data from the first device 100 and the second device 200, but this is not limiting. For example, a sensing device that only performs sensing without executing processing according to an application may be placed around the person being assisted. The server system 10 may then update the association information based on sensing data from the sensing device. The sensing device here may include devices that include various sensors, such as a thermometer, a hygrometer, and a light meter.

[0118] As described with reference to FIG. 9 , the server system 10 may transmit the association information to the first device and the second device (steps S107 and S113). The first device 100 may then store the association information as first association information (step S108), and the second device 200 may store the association information as second association information (step S114). In this manner, the association information is distributed to each device used for assistance, enabling each device to perform appropriate processing according to the ability, situation, etc. of the person being assisted. Furthermore, as described above, the association information may include information about devices used in conjunction with each other, thereby enabling the first device 100 and the second device 200 to be appropriately linked.

[0119] FIG. 10 is a sequence diagram illustrating the cooperation process between the first device 100 and the second device 200. First, in step S201, the first device 100 acquires trigger information indicating the trigger of the cooperation process. Here, the trigger information is, for example, information indicating that a specific application has been launched in the first device 100. For example, when the application (choking ap- plication) described above using FIG. 8 is launched in the choking choke detection device 460, it is considered that this is the timing for the person being assisted to start eating. Therefore, by launching a positioning application in the second device 200 (e.g., a smartphone) when the application of the choking choke detection device 460 is launched, it is possible to ensure that the person is in a position that is less likely to choke or aspiration while eating. In this case, the choking choke detection device 460 is the first device 100, and the smartphone is the second device 200. The device that cooperates based on the trigger of the choking choke detection device 460 may be the seat sensor 440. As will be described later, the trigger for starting the cooperation process is not limited to the launch of an application, but may also be the satisfaction of certain conditions in the sensing data.

[0120] Although the first device 100 and the second device 200 operate independently after startup, when the first device 100 and the second device 200 perform collaborative processing, data is exchanged between the first device 100 and the second device 200.

[0121] For example, the processing unit 110 of the first device 100 makes a desired judgment based on information acquired by the first device 100 and stored in the memory unit 120 and information acquired from the second device 200, and displays the judgment results, etc. on the display unit 140 of the first device 100.

[0122] Without being limited to this, the second device 200 may display the judgment results, etc. on the display unit, or the processing unit of the second device 200 may make the desired judgment based on the information acquired by the second device 200 and stored in the memory unit and the information acquired from the first device 100, and display the judgment results, etc. on the display unit of the second device 200 or the display unit 140 of the first device 100.

[0123] Hereinafter, a specific description will be given using as an example the cooperation process between the swallowing choking detection device 460 (first device 100) and the positioning application (second device 200).

[0124] When a caregiver assists a care recipient with low ADL with eating, the care recipient is considered to be at high risk of aspiration. There are various causes of aspiration, such as the care recipient's posture, the type of food, and the care recipient's swallowing ability. By linking with a positioning application, the swallowing choking detection device 460 can not only determine the care recipient's aspiration risk based on the sounds detected by the swallowing choking detection device 460, but also determine whether posture is a factor in increasing the aspiration risk.

[0125] The swallowing choking detection device 460 detects swallowing sounds of the person being assisted, etc. When the swallowing choking detection device 460 determines that the person being assisted has a high risk of aspiration based on the swallowing sounds of the person being assisted (for example, when the swallowing choking detection device 460 detects that the person being assisted is choking), it determines whether the cause of the high aspiration risk is the posture of the person being assisted based on information about the posture of the person being assisted obtained from the positioning application.

[0126] The swallowing choking detection device 460 may constantly acquire information about the posture from the positioning application through cooperative processing. Note that, without being limited to this, for example, only when the swallowing choking detection device 460 determines that the assisted person has a high risk of aspiration, the swallowing choking detection device 460 may inquire of the positioning application and acquire information about the posture of the assisted person or information indicating whether the assisted person's posture is correct.

[0127] In step S202, the processing unit 110 of the first device 100 identifies the person being assisted. For example, if the first device 100 is a choking stool detection device 460, the choking stool detection device 460 may identify the person being assisted based on image recognition processing using the imaging unit of the terminal device 462. For example, the terminal device 462 may perform facial recognition processing to recognize the face of the person being assisted. Alternatively, the person being assisted or the caregiver may carry a marker (such as a barcode or QR code; QR code is a registered trademark) corresponding to the person being assisted, and the terminal device 462 may perform recognition processing of the marker. Alternatively, the application executed by the first device 100 may be an application that is associated one-to-one with the person being assisted, and the processing unit 110 may identify the person being assisted based on the launched application. Alternatively, the application may be an application that allows target users to be switched by tab selection, etc., and the processing unit 110 may identify the person being assisted based on the selected tab.

[0128] When the trigger information is acquired, the first device 100 may determine whether it is connected to the server system 10, and if it is connected, determine whether the first association information is up to date by communicating with the server system 10. In this way, appropriate preprocessing can be realized when a connection with the server system 10 is established.

[0129] Specifically, when a connection between the first device 100 and the server system 10 is established, in step S203, the first device 100 performs a process of inquiring of the server system 10 as to whether the association information (first association information) stored in the storage unit 120 is the latest data. For example, the processing unit 110 may perform a process of transmitting the first association information itself to the server system 10, or may transmit information included in the first association information, such as the update timing (corresponding to the update date and time in FIG. 3 ) and the expiration date and time (not shown in FIG. 3 ).

[0130] Based on the request of step S203, the server system 10 determines whether the first association information is up to date. For example, the processing unit 11 may determine whether the content of a row in the master table corresponding to the first association information matches the first association information. Alternatively, the processing unit 11 may determine whether the update timing of the master table matches the update timing of the first association information. The processing unit 11 may also make the determination based on the expiration date instead of the update timing.

[0131] If the first association information is the latest, in step S204, the processing unit 11 performs a process of returning information indicating that the first association information is the latest. If the first association information is not the latest, in step S205, the processing unit 11 identifies the latest association information (master table).

[0132] In step S205, the processing unit 11 may determine whether or not there is an application that needs to be newly installed in the first device 100. For example, the processing unit 11 identifies a first list, which is a list of applications required for the first device 100, based on the master table. The processing unit 11 also acquires a second list, which is a list of applications already installed in the first device 100, based on the device information stored in the storage unit 12. Note that the second list may be acquired from the first device 100 in step S203. The processing unit 11 determines that, among the applications included in the first list, applications that are not included in the second list are applications that need to be installed in the first device. In step S206, the communication unit 13 transmits the latest association information and the applications that need to be installed to the first device 100. Note that, as shown in steps S108 to S112 of FIG. 9 , the first device 100 may determine whether or not there is an application that needs to be newly installed.

[0133] In step S207, the processing unit 110 of the first device 100 stores the association information transmitted from the server system 10 as first association information in the storage unit 120. The processing unit 110 also installs the application transmitted from the server system 10. This updates the association information and application of the first device 100 to the latest state. For example, even if the association information is updated after the processing of FIG. 9 and before the processing of FIG. 10 starts, the first association information of the first device 100 is updated to the latest state by the processing shown in FIG. 10. That is, by the processing of steps S203 to S207, the preprocessing for executing processing in accordance with the latest association information is completed in the first device 100.

[0134] In step S208, the processing unit 110 performs a process of identifying devices that operate in coordination with each other during assistance, based on the first association information. For example, here, the person being assisted and the application used by the first device 100 are identified (steps S201-S202). Based on this information and the association information, the processing unit 110 may perform a process of searching for rows that include the same person being assisted, have the same ADL, scene, and type of assistance, and have different device IDs. If a row that satisfies the conditions is found, the processing unit 110 determines the device identified by the device ID in that row as the collaboration destination.

[0135] Although the example described here is one in which the first association information includes information identifying the link destination, the present invention is not limited to this. For example, the first association information may include only information related to the first device 100 and not include information about the link destination. In such a case, in step S203, the first device 100 may request information identifying the link destination from the server system 10 in addition to checking the association information. The storage unit 12 of the server system 10 also stores information about devices other than the first device 100 as association information (master table), so the link destination of the first device 100 can be appropriately determined.

[0136] 10 , the second device 200 is identified as the collaboration destination. Therefore, in step S209, the first device 100 transmits a collaboration request to the second device 200 to request that collaboration processing be started. For example, the communication unit (second communication unit) of the first device 100 establishes a connection with the second device 200 by performing Bluetooth communication pairing, and then transmits the collaboration request shown in step S209 using the Bluetooth communication.

[0137] When the second device 200 receives the link request, it determines whether it is connected to the server system 10, and if it is connected, it determines whether the second association information is up to date by communicating with the server system 10. In this way, when a connection with the server system 10 is established, appropriate preprocessing can be realized.

[0138] Specifically, in steps S210-S214, the second device 200 and the server system 10 execute the same processes as those in steps S203-S207 described above. This completes the preprocessing in the second device 200 for executing processing in accordance with the latest association information.

[0139] 10 illustrates an example in which the second device 200 directly communicates with the server system 10, but the present invention is not limited to this. For example, the second device 200 may communicate with the server system 10 via the first device 100, which is the sender of the link request. For example, in step S210, the second device 200 may first transmit a confirmation request to the first device 100, and the first device 100 may then transmit the confirmation request to the server system 10. Similarly, in steps S211 and S214, the server system 10 may first transmit data to the first device 100, and the first device 100 may then transfer the data to the second device 200.

[0140] In step S215, the second device 200 identifies an application to be used in the collaboration process based on the second association information. This puts the second device 200 in a state where it can launch an application that collaborates with the application launched in step S201. Therefore, in step S216, the second device 200 transmits an acceptance response to the collaboration request of step S209 to the first device 100.

[0141] In step S217, the first device 100 displays a message indicating that preparation is complete on the screen of the application started in step S201.

[0142] FIG. 11 is an example of an application screen displayed in step S217. For example, when using an application executed by the choking stool detection device 460, text stating "Application settings for AAA have been completed" may be superimposed on the display screen described above with reference to FIG. 8 , where "AAA" is the name of the person being assisted. Also, as shown in FIG. 11 , the display unit 140 of the first device 100 (choking stool detection device 460) may display the second device 200 that performs processing in cooperation with the first device 100. In the example of FIG. 11 , a message is displayed indicating that the smartphone XXX running the positioning application is the second device 200. Note that "XXX" here may be an ID (device ID) that uniquely identifies the smartphone, or information indicating the smartphone's manufacturer name or model number. If multiple second devices 200 cooperate with the first device 100, information about the multiple second devices 200 may be displayed on the screen shown in FIG. 11 . 11 may also display information about applications running on the second device 200 (such as app IDs and app names). This allows the user to easily understand that the application of the choking choke detection device 460 and the application of the second device 200 linked thereto (e.g., a smartphone positioning application) are appropriately configured. It is assumed that the screen shown in FIG. 11 displays, as the second device 200, a device that has sent a link acceptance response (step S216) in response to the link request (step S209). However, the method of the present embodiment is not limited to this. For example, the first device 100 may display information about devices that are candidates for the second device 200. For example, the first device 100 may display, on the screen of FIG. 11, a list of devices whose Bluetooth radio wave intensity is equal to or greater than a threshold, and display devices connected as the second device 200 among the devices in a manner that allows them to be distinguished from unconnected devices. In this way, it is possible to present to the user not only information about the second device 200 that will start the linking process, but also information about devices that may link as the second device 200 in the future.

[0143] 11 may also display an OK button used to allow the start of the linked process. When the OK button is selected, the processing unit 110 of the first device 100 transitions to a state in which the linked process can be started, and when the execution button in area RE6 is selected in this state, the linked process starts. Alternatively, the linked process may start when the OK button in FIG. 11 is selected, and the execution button selection operation may be omitted. Alternatively, the OK button and the execution button in area RE6 may have different functions. For example, the OK button in FIG. 11 may be used to start the linked process, and the execution button in area RE6 may be used to start processing in which the first device 100 does not link with other devices (processing by the first device 100 alone).

[0144] In this state, both the first device 100 and the second device 200 are capable of processing based on the latest association information, and therefore, in step S218, specific cooperation processing is started. Note that a detailed description of the cooperation processing will be omitted.

[0145] Although not described above, the first device 100 may not accept operation inputs for the application launched in step S201 until the display in step S217 is performed. For example, the processing unit 110 of the first device 100 does not start processing even if the user presses the execute button in area RE6 during the period from the launch of the application until the display in step S217. This prevents processing from being executed before the application is updated. For example, it prevents inappropriate processing from being started when at least one of the first device 100 and the second device 200 does not correspond to the latest association information.

[0146] At this time, the processing unit 110 of the first device 100 may cause the display unit 140 to display the screen shown in Fig. 12. For example, the processing unit 110 may display the screen shown in Fig. 12 during the period from after the application is launched until before step S217, and then perform display processing to change to the screen shown in Fig. 11 in step S217. In this way, it is possible to clearly notify the user that necessary settings have not been made during execution of preprocessing.

[0147] A details button may also be displayed on the screen shown in FIG. 12 . When the processing unit 110 receives a selection operation of the details button, it performs processing to transition to a details setting screen. The details setting screen may display, for example, a list of care applications required for the first device 100, and may receive a user operation to select which of the applications to install. For example, before the processing of steps S206-S207 in FIG. 10 , the server system 10 may first transmit a list of care applications required for the first device 100, and the display unit 140 may display the list on the details setting screen. Then, when a user performs a user operation to select one or more applications from the list, the first device 100 may request the selected applications from the server system 10. In step S206, the server system 10 transmits the care applications selected by the user to the first device 100. This allows the user to manually set the care applications to be installed. The details setting screen may also be used to perform various settings, such as setting parameters used in the care applications, in addition to selecting the care applications to be installed.

[0148] The association information may include information on tacit knowledge data that is a digitalized version of the tacit knowledge of an experienced caregiver used in care. Although the tacit knowledge data is, in a narrow sense, a nursing application, the tacit knowledge data of this embodiment may include data in a form other than an application. For example, the tacit knowledge data may not be the application itself, but may be setting data (parameters, etc.) used in the application.

[0149] Then, at least one of the first device 100 and the second device 200 acquires tacit knowledge data from the server system based on the association information and executes a process related to assistance based on the tacit knowledge data. For example, the application transmitted in steps S206 and S214 may be a care application that utilizes tacit knowledge. In this manner, when tacit knowledge digitized in each device is used, data related to the tacit knowledge can be appropriately provided to each device. Note that, as described above, the use of tacit knowledge is not required in the method of this embodiment, and the application in one or both of the first device 100 and the second device 200 may not utilize tacit knowledge.

[0150] In addition, this embodiment assumes that various devices are placed around the person being assisted, and that these devices flexibly cooperate according to the situation. Therefore, the information processing system 1 of this embodiment may include a variety of devices of different types and vendors. Therefore, to accommodate the diversity of devices, a common data structure may be used for communication between the server system 10 and each device, or for communication between devices. Details of the data structure will be described later using Figures 23 to 25, etc.

[0151] 3.2 When one device is not connected to the server system <When the second device is not connected> When the second device 200, which is a slave device, is not connected to the server system 10 and the first device 100, which is a master device, is connected to the server system 10, for example, the second device 200 may communicate with the server system 10 via the first device 100.

[0152] 10 described above, the second device 200 may determine whether it is connected to the server system 10 when it receives a link request, and if it is not connected, determine whether the first device 100 is connected to the server system 10. If the first device 100 is connected to the server system 10, the second device 200 may send a confirmation request to the server system 10 to the first device 100.

[0153] Specifically, as described above, in step S210, the second device 200 may first transmit a confirmation request to the first device 100, and the first device 100 may then transmit the confirmation request to the server system 10. Alternatively, in steps S211 and S214, the server system 10 may first transmit data to the first device 100, and the first device 100 may then transfer the data to the second device 200. In this manner, even when the second device 200 is not connected to the server system 10, it is possible to achieve collaboration processing in the same manner as when both devices are connected to the server system 10.

[0154] 13 is a sequence diagram illustrating periodic processing when the first device 100 is not connected to the server system 10 and the second device 200 is connected to the server system 10. As in the example of FIG. 9 , the information processing system 1 periodically executes the processing shown in FIG. 13 , for example.

[0155] First, in step S301, the first device 100 acquires sensing data using a sensor mounted on the first device 100. However, in this case, since the first device 100 is not connected to the server system 10, the sensing data is not transmitted to the server system 10, but is instead accumulated as log data in the storage unit 120 of the first device 100. For example, the first device 100 transmits the log data to the server system 10 the next time it is connected to the server system 10.

[0156] Meanwhile, in step S302, the second device 200 acquires sensing data using a sensor mounted on the second device 200. In step S303, the communication unit of the second device 200 transmits the acquired sensing data to the server system 10. That is, since the second device 200 has already been connected to the server system 10, the same processing as in FIG. 9 is executed.

[0157] Steps S304-S305, which are the processing of the server system 10, are the same as steps S105-S106 in Fig. 9, except that the sensing data of the first device 100 acquired in step S301 is not used. The processing of steps S306-S310 thereafter is also the same as steps S113-S117, and therefore detailed description thereof will be omitted.

[0158] As described above, the periodic processing is executed for devices connected to the server system 10, and the processing may be omitted for devices not connected to the server system 10. For example, although an example in which the first device 100 is not connected is shown in Fig. 13, if the second device 200 is not connected, the processing for the second device 200 is similarly omitted.

[0159] 14 is a sequence diagram illustrating the cooperation process between the first device 100 and the second device 200. First, in step S401, the first device 100 acquires trigger information indicating a trigger for the cooperation process. In step S402, the processing unit 110 of the first device 100 identifies the person being assisted who is the target of assistance. The processes in steps S401-S402 are the same as steps S201-S202 in FIG. 10.

[0160] If the first device 100 is not connected to the server system 10, in step S403, the first device 100 transmits a collaboration request including information related to the first association information to the second device 200. Note that in step S403, information about the person being assisted identified in step S402 may also be transmitted. For example, the first device 100 identifies the second device 200 as the collaboration destination based on the first association information, the launched application, and information about the identified person being assisted that are stored in the storage unit 120 at that time. Then, the first device 100 performs, for example, Bluetooth pairing with the identified second device 200.

[0161] If the second device 200 is connected to the server system 10 when it receives the cooperation request, it determines whether the first association information and the second association information are up to date by communicating with the server system 10. In this way, even if the first device 100, which is the master device, is not connected to the server system 10, it is possible to realize appropriate preprocessing using the second device 200, which is the slave device.

[0162] Specifically, in step S404, the second device 200 performs processing to inquire of the server system 10 whether the first association information of the first device 100 and the second association information of the second device 200 are the latest data. As in the example described above with reference to Fig. 10 , the second device 200 may perform processing to transmit the association information itself to the server system 10, or may perform processing to transmit information such as update timing.

[0163] Based on the request in step S403, the server system 10 determines whether each of the first association information and the second association information is up to date.

[0164] If both the first association information and the second association information are the latest, in step S405, the processing unit 11 performs processing to return information indicating that both the first association information and the second association information are the latest to the second device 200. As a result, the second device 200 recognizes that the second association information is the latest. Also, in step S406, the second device 200 performs processing to transmit information indicating that the first association information is the latest to the first device 100. As a result, the first device 100 recognizes that the first association information is the latest.

[0165] If the second association information is the latest but the first association information is not the latest, in step S407, the processing unit 11 of the server system 10 identifies the latest association information (master table). Also in step S407, the processing unit 11 may determine whether or not there is an application that needs to be newly installed on the first device 100. Also in step S407, the processing unit 11 may determine whether or not the second device 200 with which the first device 100 should be linked has changed. For example, assume that the choking spurt detection device 460 is the first device 100, and the smartphone CP is determined to be the second device 200 based on existing information, and the above-described processing is performed. However, as a result of the server system 10 making a determination based on the latest association information, it is possible that the second device 200 with which the choking spurt detection device 460 should be linked has been changed to the seat sensor 440 instead of the smartphone CP. Alternatively, it is also possible that the smartphone CP remains the second device, but the seat sensor 440 is also added to the second device 200 (three or more devices are linked together). Therefore, the server system 10 may transmit information indicating these determination results to the first device 100 via the smartphone CP (the second device 200 before the change).

[0166] In step S408, the communication unit 13 transmits the latest association information and the applications that need to be installed in the first device 100 to the second device 200. Also in step S408, the server system 10 may transmit information indicating that the second association information is the latest. In step S409, the second device 200 transmits the latest association information and the applications that need to be installed in the first device 100 to the first device 100.

[0167] In step S410, the processing unit 110 of the first device 100 stores the association information transmitted from the server system 10 via the second device 200 as first association information in the storage unit 120. The processing unit 110 also installs the application transmitted from the server system 10 via the second device 200. As a result, the association information and application of the first device 100 are updated to the latest state.

[0168] If both the first association information and the second association information are not up to date, the processing unit 11 of the server system 10 identifies the latest association information (master table) in step S411. Also, in step S411, the processing unit 11 may determine whether or not there is an application (referred to as a first application) that needs to be newly installed in the first device 100, and whether or not there is an application (referred to as a second application) that needs to be newly installed in the second device 200.

[0169] In step S412 , the communication unit 13 transmits the latest association information, the first application, and the second application to the second device 200 .

[0170] In step S413, the processing unit of the second device 200 stores the association information transmitted from the server system 10 in the storage unit as second association information. The second device 200 also installs the second application transmitted from the server system 10. As a result, the association information and application of the second device 200 are updated to the latest state.

[0171] In step S414, the second device 200 transmits the latest association information and the first application to the first device 100. In step S415, the processing unit 110 of the first device 100 stores the association information transmitted from the server system 10 via the second device 200 as first association information in the storage unit 120. The processing unit 110 also installs the application transmitted from the server system 10 via the second device 200. This updates the association information and application of the first device 100 to the latest state.

[0172] As described above, when the second device 200 determines that the first association information is not the latest, the second device 200 may receive the latest first association information and the tacit knowledge data for the first device identified by the latest first association information from the server system (steps S408 and S412). The first device 100 acquires the latest first association information and the tacit knowledge data for the first device from the second device 200 (steps S409 and S414). In this way, even if the first device 100, which is the master device, is not connected to the server system 10, it is possible to perform appropriate preprocessing using the second device 200, which is the slave device.

[0173] 14 omits a case where the first association information is the latest and the second association information is not the latest. However, such a case may occur, and the server system 10 may transmit to the second device 200 a message that the first association information is the latest, as well as the latest association information and the second application. The second device 200 updates the second association information, installs the second application, and transmits to the first device a message that the first association information is the latest.

[0174] In step S416, the first device 100 performs a process of transmitting a request to start a collaboration process to the second device 200. As described above in step S407, the second device 200 with which the first device 100 is to collaborate may have been changed in the determination process in the server system 10. Alternatively, although not shown in FIG. 14 , before step S416, the processing unit 110 may identify a device to collaborate with in the assistance process based on the latest first association information. If a change in the second device 200 is necessary, the start request of step S416 is transmitted to the identified device. For example, if the collaboration destination has been changed based on an update process of at least one of the first association information and the second association information, the start request of step S416 may be transmitted to a second device 200 different from the destination of the collaboration request of step S403.

[0175] In step S417, the second device 200 identifies the application to be used in the collaboration process, and then returns a start acceptance response (collaboration acceptance response) to the first device 100. In step S418, the first device 100 displays a message indicating that preparation is complete on the screen of the application launched in step S401. The screen displayed in step S418 is, for example, the same as the screen described above with reference to FIG. 11. Alternatively, the screen described above with reference to FIG. 12 may be displayed during a period prior to step S418.

[0176] In this state, both the first device 100 and the second device 200 are capable of processing based on the latest association information, and therefore, in step S419, specific cooperation processing is started. Note that detailed explanation of the cooperation processing will be omitted.

[0177] The above describes an example in which the connection state is the same for the periodic processing and the cooperative processing. For example, the description above using Figures 9 and 10 describes an example in which both the first device 100 and the second device 200 are connected to the server system 10 in both the periodic processing and the cooperative processing. Also, the description above using Figures 13 and 14 describes an example in which the first device 100 is disconnected and the second device 200 is connected to the server system 10 in both the periodic processing and the cooperative processing.

[0178] However, the processing of this embodiment is not limited to a case in which the connection status during the periodic processing and the connection status during the coordination processing are the same. For example, the coordination processing and the periodic processing may be executed at different times, such as when the coordination processing is executed during daytime meal assistance and the periodic processing is executed late at night. Therefore, the connection status of each device with the server system 100 may change between the coordination processing and the periodic processing. For example, if the first device 100 is not connected to the server system 10 during the periodic processing but is connected when a trigger is detected, the coordination processing shown in FIG. 10 may be executed after the periodic processing shown in FIG. 13 is executed. Similarly, the coordination processing shown in FIG. 14 may be executed after the periodic processing shown in FIG. 9 is executed. In other words, any combination of the connection status during the periodic processing and the connection status during the coordination processing is permitted.

[0179] 3.3 When Both Devices are Not Connected to the Server System When neither the first device 100 nor the second device 200 is connected to the server system 10, the periodic processing is a processing in which, for example, steps S102 and S104-S117 in Fig. 9 are omitted. That is, the first device 100 and the second device 200 accumulate the sensing data as log data, but do not transmit the data to the server system 10 or receive association information. Note that the processing described above using Fig. 9 or Fig. 13 may be performed as the periodic processing.

[0180] If both the first device 100 and the second device 200 are not connected to the server system 10, the server system 10 cannot be requested to confirm the first and second association information during the collaboration process. Therefore, if the first device 100 is not connected to the server system 10 when trigger information is acquired and the second device 200 is not connected to the server system when a collaboration request is received, at least one of the first device 100 and the second device 200 may select one of the association information based on a comparison process between the first and second association information. The first device 100 and the second device 200 then collaborate to perform assistance-related processing based on the selected association information. In this way, even if the latest association information cannot be referenced, the collaboration process can be performed using previously acquired association information. Since the first and second association information are compared, the collaboration process can be optimized to the greatest extent possible. A specific example of the process is described below.

[0181] 15 is a sequence diagram illustrating the cooperation process between the first device 100 and the second device 200. First, in step S501, the first device 100 acquires trigger information indicating a trigger for the cooperation process. In step S502, the processing unit 110 of the first device 100 identifies the person being assisted who is the target of assistance. The processes in steps S501-S502 are the same as steps S201-S202 in FIG. 10.

[0182] Here, the first device 100 is not connected to the server system 10. Therefore, in step S503, the first device 100 transmits a link request including information related to the first association information to the second device 200, similar to step S403 in FIG.

[0183] In step S504, the processing unit of the second device 200 that has received the link request checks the connection status between the second device 200 and the server system 10 and determines that they are not connected. In other words, unlike the processing shown in Fig. 14, communication with the server system 10 via the second device 200 cannot be performed.

[0184] Therefore, in step S504, the second device 200 performs a process of comparing the first association information acquired in step S503 with the second association information stored in the storage unit of the second device 200. For example, the second device 200 may select the association information with the most recent update date and time from the first association information and the second association information. In step S505, the second device 200 transmits data including the selected association information to the first device 100 as a collaboration acceptance response. Note that the data included in the collaboration acceptance response in step S505 may be the association information itself or information identifying the association information (such as an ID of the association information). In this way, newer association information is more likely to be selected, so that even if the selected association information is not the latest, it is possible to use data that is close to the latest.

[0185] In step S506 (before executing the assistance-related process), the first device 100 performs a process of presenting the updated date and time of the association information to the user. Fig. 16 is an example of an application screen displayed in step S506. For example, when using an application executed by the choking choke detection device 460, text stating "Operation will be performed based on data with MM / DD" may be superimposed on the display screen described above with reference to Fig. 8. "MM / DD" is information indicating the updated date and time of the selected association information.

[0186] According to the method of this embodiment, it is possible to present to the user in an easy-to-understand manner the association information at what point in time the subsequent link processing will be based on.

[0187] Furthermore, as shown in Fig. 16, the screen displayed in step S506 may display information for accepting user input regarding whether or not to perform the operation. In the example of Fig. 16, in addition to the text "Operation will be performed based on the data with MM / DD," a confirmation text "Are you sure?" and a selection option of "Yes / No" are displayed. Then, when the user inputs "Yes," the first device 100 and the second device 200 execute the cooperation process in step S507.

[0188] This makes it possible to suppress the execution of linkage processes that the user deems undesirable. For example, if the ADL of the person being assisted has significantly decreased after the displayed date, it is possible that the content of the assistance to be performed for that person has changed. If the user, such as a caregiver, is aware of the timing of the change in ADL, they can consider that the association information before that time is old and does not reflect the current state of the person being assisted, and can select "No."

[0189] 15 illustrates an example in which the second device 200 performs the comparison process and selection process of the association information, but the present invention is not limited to this. For example, the first device 100 may perform the comparison process and selection process of the association information after acquiring information related to the second association information from the second device 200.

[0190] In this embodiment, the determination may be made not only from the perspective of the update timing of the association information but also from the perspective of whether the target association information can be executed. For example, consider an example in which the association information includes information identifying the first tacit knowledge data used in the first device 100 and the second tacit knowledge data used in the second device 200 as information regarding tacit knowledge data that is digitized tacit knowledge of an experienced caregiver used for care. In this case, at least one of the first device 100 and the second device 200 may select, from the first association information and the second association information, association information in which the first tacit knowledge data has been installed in the first device and the second tacit knowledge data has been installed in the second device. Below, an example in which the tacit knowledge data is a nursing care application will be described.

[0191] For example, it is assumed that the first association information is information indicating that a care application 1A is executed on the first device 100 and a care application 2A is executed on the second device 200. It is also assumed that the second association information is information indicating that a care application 1B is executed on the first device 100 and a care application 2B is executed on the second device 200. In this example, it is assumed that the care application 1A is installed on the first device 100 but the care application 1B is not installed.

[0192] For example, simply considering the update timing, the second association information may be newer than the first association information. However, according to the second association information, the first device 100 needs to operate according to the care application 1B, but the care application 1B is not installed on the first device 100. Furthermore, since both devices are not connected to the server system 10, it is not easy to install a new care application 1B. In other words, in this example, there is a possibility that the linking process based on the second association information cannot be executed.

[0193] In this regard, according to the above process, the selection priority of the second association information requesting the care application 1B is lowered, and the selection priority of the first association information requesting the installed care application 1A is higher. In this way, by taking into consideration the installation status of the care application, it is possible to select association information from the perspective of whether or not the linking process according to the association information is actually possible.

[0194] 3.4 Modifications Although specific examples of processing have been described above using FIGS. 9 to 16, the processing of this embodiment is not limited to these, and various modifications are possible.

[0195] (Notification Process) As specific examples of the notification process (display process) to the user, the screen examples of Fig. 11, Fig. 12, and Fig. 16 have been described above. However, the information notified to the user is not limited to this.

[0196] For example, the first device 100 and the second device 200 may each check whether they are connected to the server system 10 (network) when they are started up, and if they are not connected, may display a message indicating that they are not connected.

[0197] Furthermore, when the first device 100 is not connected to a network, the first device 100 may display a list of candidate devices to be linked to the first device 100 and prompt the user to select one. For example, the first device 100 may display devices that can be paired via Bluetooth as candidate devices to be linked to the first device 100.

[0198] Furthermore, at least one of the first device 100 and the second device 200 may perform a process to display a mismatch between the first association information and the second association information. In this case, the first device 100 may notify the server system 10 via the network of a mismatch notification message, the association information of the device where the mismatch occurred, and the time when the mismatch occurred. For example, if a mismatch in the association information occurs because one of the first device 100 and the second device 200 is not connected to the server system 10, it is possible to tolerate the mismatch and execute a collaboration process while notifying the server system 10 of the mismatch, thereby enabling a future response.

[0199] Furthermore, if a mismatch occurs, a queue for scheduling an update of the association information may be automatically set in a device that is not connected to the server system 10. In this case, when the device is connected to the network, updating of the association information is automatically started. For example, if the association information of the second device 200, which is a slave device, has not been updated, the first device 100, which is a master device, may perform processing to set an update queue for the second device 200 using Bluetooth communication.

[0200] (Trigger Information) While the above describes an example in which the launch of an application is the trigger for the coordination process, this is not limiting. For example, in the coordination process for fall prevention, an application that uses a camera to determine the degree of posture stability is first activated, and when the application detects standing instability, a foot pressure sensor is activated as a slave device to detect posture in detail, or a locking mechanism provided on furniture such as a desk is activated so that the person being assisted can hold it stably. In this way, the trigger in this embodiment may be that the determination result (application processing result) of the master device satisfies a given condition.

[0201] In this example, it is assumed that the application is operating with a specific person being assisted as the target, so the process of launching the app and identifying the person being assisted shown in steps S201-S202 of Figure 9 is automatically executed when the trigger information is obtained.

[0202] (Identifying the Collaboration Device) The above describes an example in which the first device 100 identifies the collaboration device 200 based on the latest association information or the association information stored in the storage unit 120. However, depending on the device, the user may already be identified. For example, a device placed in the room of a person receiving care is likely to be used to assist the person receiving care. In this case, the collaboration device 200 may be identified based on the result of the process of identifying the person receiving care shown in step S202 of FIG. 9 . For example, if a trigger is detected in a master device placed in a room, another device placed in the room may be selected as a slave device.

[0203] The first device 100, which is the master device, may also select a nearby device as a partner device. For example, the first device 100 performs processing related to Bluetooth pairing and selects a device with a predetermined or higher radio wave intensity as a partner device. Alternatively, the position of each device may be detected using RFID (Radio Frequency Identification) or camera images, and a device near the first device 100 may be determined based on the detected position. In this way, it becomes possible to select a partner device based on the relative positions of the devices.

[0204] If there is no device to be linked nearby, the first device 100 may display that fact on the screen of the display unit 140. If inventory location management is performed for the device to be linked, the first device 100 may display the location where the device is stored on the display unit 140. The notification of the storage location may be sent to a device other than the first device 100, such as a terminal device used by a caregiver.

[0205] (ADL and Correspondence Information Update Timing) The periodic processing and the cooperation processing have been described above as different processes. Specifically, the ADL and correspondence information based on communication between each device and the server system 10 are updated by a predetermined time, but these update processes are not executed in the cooperation processing (FIGS. 9 and 10, 13 and 15).

[0206] However, this embodiment is not limited to this, and the ADL and association information may be updated each time an application is launched in each device. For example, when the first device 100 and the server system 10 are connected, log data may be transmitted from the first device 100 to the server system 10 when an application is launched (step S201 in FIG. 10 ), and the ADL and association information may be updated in the server system 10. In this case, although it may take a long time until preparation for the linking process is completed (until the "Ready" display shown in FIG. 11 is displayed), it becomes possible to update the ADL and association information (master table of the server system 10) to the latest data at the linking process stage.

[0207] For example, by updating the master table during the collaboration process, even if the condition of the person being assisted changes between the processing of Figure 9 and the processing of Figure 10, it is possible to reflect the change in the master table.

[0208] 4. Specific Example of Positioning Application Next, a specific example of a screen of the positioning application will be described.

[0209] 4.1 Example of Setting Screens Figures 17A and 17B are example screens displayed on the display unit of a device such as a smartphone when the positioning application is launched. Figure 17A is the screen used by the worker, and Figure 17B is the screen used by the administrator. The worker here is, for example, a general care worker at a nursing facility. The administrator is, for example, a functional trainer with knowledge of positioning appropriate for individuals receiving care. However, the administrator may also be a care manager or a management staff member at the nursing facility.

[0210] For example, when the positioning application is launched, the screen shown in Fig. 17A for general caregivers is first displayed. The general caregiver performs a login operation on the screen shown in Fig. 17A, and after completing the login, adjusts the position of the person being assisted by operating the positioning application in operation mode. General caregivers may be restricted from creating new applications (registering training data) in setting mode.

[0211] For example, when a worker has logged in, applications available to the worker may be displayed, as shown in Fig. 17A. In the example of Fig. 17A, the assisted person AAA is selected, and three applications used to assist the assisted person AAA are displayed. Furthermore, by selecting assisted persons BBB and CCC, applications corresponding to the selected assisted person may be displayed. Furthermore, a search process for assisted persons may be performed using a text box at the top of the screen.

[0212] Furthermore, when an object OB1 representing a right arrow is selected on the screen shown in FIG. 17A , the positioning application transitions to the screen shown in FIG. 17B . For example, an administrator such as a functional trainer selects object OB1 on the screen shown in FIG. 17A and performs a login operation on the screen shown in FIG. 17B . After completing the login, the administrator creates a new application in the setting mode or adjusts the position in the operation mode. For example, the positioning application operates in the setting mode when an object OB2 corresponding to a plus sign is selected. Note that the positioning application may also perform a process of transitioning to the screen shown in FIG. 17A when an object OB3 representing a left arrow is selected on the screen shown in FIG. 17B . Similarly to FIG. 17A , FIG. 17B may also display applications available to the administrator who logged in. While FIG. 17B shows an example in which the same person being assisted and the same applications as those in FIG. 17A are displayed, different applications may be displayed depending on the user's authority, responsibility, etc.

[0213] In this way, even when the same positioning application is shared by multiple caregivers, it is possible to switch between executable operations depending on the authority granted to each caregiver. In the above example, only highly specialized administrators are allowed to operate in the setting mode, while general caregivers are only allowed to operate in the operation mode using the set training data. In particular, in this embodiment, the worker screen is displayed when the application is launched, and the screen will not transition to the administrator screen unless a specific operation is performed on that screen (e.g., selecting object OB1 in FIG. 17A). Therefore, since the trainee screen is not displayed at launch, it is possible to prevent processes such as the creation of training data from being inadvertently performed.

[0214] Figure 18 is an example of a setting screen that is displayed when, for example, object OB2 is selected in Figure 17B. The setting screen shown in Figure 18 allows the user to enter the group, application name, name of the person being assisted, and specific registration details. The group is used to group people being assisted in nursing care facilities, etc. The application name is the name of the application to be created. The registration details include various items, including the training data shown in Figure 5A.

[0215] 19A and 19B are example screens displayed when adding training data to the registration content, for example, when the camera icon in the registration content in FIG. 18 is selected. In FIG. 19A , a guide message saying, "Please take a picture of the posture you want to register" is displayed, along with a captured image captured by the smartphone camera. For example, the administrator adjusts the position and angle of the smartphone so that the captured image shows a person in the desired posture. Furthermore, when the object OB4, which is the shooting button, is selected, the positioning application acquires the captured image at that time as a candidate for training data. Note that the screen shown in FIG. 19A may also display an object OB5 corresponding to the in-camera / out-camera switching button and an object OB6 corresponding to the back button.

[0216] FIG. 19B shows an example of a display screen displayed when the capture button is pressed. For example, the positioning application performs skeletal tracking of a person and displays the results superimposed on the captured image. Similar to the examples of FIGS. 5A and 5B , FIG. 19B illustrates an example of acquiring training data representing the position and posture of the person being assisted and the cushion considered appropriate for lying on a bed. The circles in FIG. 19B represent detected skeletal reference points (e.g., shoulders, elbows, wrists, etc.). The screen shown in FIG. 19B may display a text message such as "Please select your first point" prompting the user to select one of the reference points. For example, the administrator may select a point considered important for appropriate position adjustment. When a point is selected, the positioning application stores the selected point and then displays a text message such as "Please select your second point" prompting the user to select another point. Similarly, the positioning application may accept N selections of points. N is, for example, 3, but may be a different number.

[0217] For example, the positioning application accepts the selection of three points and stores the magnitude of the angle formed by the three points as the correct angle. Among the registration items in FIG. 18 , the correct / incorrect judgment is an item that sets whether or not the position of the person being assisted is in an appropriate posture in the operation mode. For example, when the correct / incorrect judgment is set to ON, the positioning application may perform skeletal tracking processing in the operation mode, identify the three set points, and calculate the angle formed by the three points. The positioning application then performs the correct / incorrect judgment by comparing the angle in the operation mode with the correct angle stored in the setting mode. However, the correct / incorrect judgment is not limited to this, and other judgments may be performed based on the positions of the points detected by skeletal tracking, etc.

[0218] The correct / incorrect judgment in FIG. 18 is a setting item for switching whether to perform an OK / NG judgment based on the training data or to only superimpose the training data. The transparency is an item that indicates the transparency when superimposing the training data. Although not shown in FIG. 18 , an item for switching the skeletal information acquisition process on / off may be displayed. Furthermore, when the skeletal information is used only for correct / incorrect judgment, the item for setting the correct / incorrect judgment on / off may also serve as the item for setting the skeletal information acquisition process on / off. These items are not limited to those displayed on the screen of FIG. 18 and may also be displayed on the screens shown in FIGS. 19A and 19B. For example, when an operation to change the transparency is performed on the screens shown in FIGS. 19A and 19B, the transparency of the captured image may be changed in real time according to the operation. Furthermore, when the skeletal information acquisition process is switched on / off, the display of the skeletal information in FIGS. 19A and 19B may be switched. For example, when the skeletal information is off, the display of the skeletal information shown in FIG. 19B and the reception of the selection input of feature points may be omitted. Adjusting the transparency in the setting mode is not required. For example, in the setting mode, the transmittance may be set to a given fixed value, and a display object such as a scroll bar with adjustable transmittance may be displayed on the screen in the use mode (e.g., the screen described above using FIG. 5B ). Making the transmittance adjustable in the use mode allows for flexible control when actually providing care. For example, by setting the transmittance low, it is possible to make it easier to check the details of the training data, or by setting the transmittance high, it is possible to make it easier to check the captured image in real time. Furthermore, the transmittance may be configurable in both the setting mode and the use mode.

[0219] 18 are items in which the manager inputs items that he or she considers important for position adjustment using text, etc. The point display direction is an item that sets the direction in which the text, etc. included in points 1 and 2 are displayed.

[0220] There may also be cases where it is desired to have a single person being assisted assume different postures. For example, when adjusting the sleeping posture of a person being assisted, if only one preferred posture is allowed, maintaining that posture for a long period of time could lead to the development of bedsores, etc. Therefore, the positioning application can register multiple sleeping postures (for example, right lateral position and left lateral position), and the caregiver can switch over time between position adjustment using the right lateral position as training data and position adjustment using the left lateral position as training data.

[0221] From the above perspective, one positioning application may include multiple pieces of training data. For example, the example in FIG. 18 shows an example of registering one piece of training data in an area labeled "Tab 1." However, by scrolling down this screen, a screen labeled "Tab 2" for registering other pieces of training data may be displayed. Naturally, the number of training data is not limited to two, so the number of tabs may be three or more. In the operating mode of the positioning application, for example, tabs are displayed on the screen, and switching between the tabs switches the superimposed training data, transparency, whether the correct / incorrect judgment is on or off, comments, and so on.

[0222] Furthermore, multiple training data sets set using different tabs are not limited to those used at different times. For example, the execution of specific movements may be supported by sequentially using multiple training data sets. For example, when performing rehabilitation for an assisted individual, it is considered possible to have the individual perform the desired movements at a hospital or other facility with the support of a specialist. On the other hand, at home, the individual's family may support the rehabilitation, or the individual may perform the rehabilitation alone, making it difficult to perform the movements as appropriately as at a hospital. In this case, for example, an administrator may subdivide the movements to be performed in rehabilitation and set training data representing intermediate and final postures in different tabs. For example, if a rehabilitation movement can be achieved by performing three postures (first posture, second posture, and third posture) in this order, the administrator may register the first posture in Tab 1, the second posture in Tab 2, and the third posture in Tab 3. The positioning application then automatically switches between Tabs 1 to 3 to support the execution of the rehabilitation movement. For example, the positioning application first executes processing using the training data corresponding to Tab 1. If the positioning application determines that the assisted person's posture is sufficiently close to the first posture (OK judgment), it executes processing using the training data corresponding to Tab 2. Furthermore, if the positioning application determines that the assisted person's posture is sufficiently close to the second posture, it executes processing using the training data corresponding to Tab 3. If the assisted person's posture is sufficiently close to the third posture, it determines that the target rehabilitation movement is complete. This allows the assisted person to appropriately support the movement. Furthermore, by referencing the positioning application history, caregivers and managers can confirm whether the rehabilitation is being performed appropriately at home. Furthermore, as described below with reference to Figures 20-22C, the positioning application can also display the time until the OK judgment is reached, thereby enabling caregivers and others to know which part of the series of movements is taking the longest.

[0223] By using the screens shown in FIGS. 17A to 19B, it becomes possible to determine the authority of the caregiver and then have the caregiver perform the setting process using an easy-to-understand interface.

[0224] 4.2 Example of Dashboard Screen In this embodiment, a dashboard screen may be used that displays data representing the results of using the positioning application. The data representing the results of using the positioning application may be, for example, statistical data obtained when multiple position adjustments are performed.

[0225] The dashboard screen is likely to display a large amount of information, and is therefore expected to be displayed on a device with a large display, such as a PC. For example, when a PC accesses the server system 10 using a web browser, the dashboard screen described below is displayed. However, the dashboard screen may be displayed using an application other than a web browser. Furthermore, the dashboard screen may also be displayed on a mobile terminal device, such as a smartphone.

[0226] Figure 20 is an example of a screen that is displayed when using a positioning application in a web browser. In the screen shown in Figure 20, a menu screen is displayed on the left side, and one of the following items is selected: Dashboard, Application Creation, or Device. If Dashboard is selected, the dashboard screen described below is displayed. If Application Creation is selected, a screen for creating a new application is displayed, similar to the example described above using Figures 18-19B. If Device is selected, a screen for adding and managing devices on which the positioning application runs is displayed.

[0227] The menu screen may also display the name of the currently logged-in user and the number of notifications. For example, in Fig. 20, it is displayed that a user with the name "XXXX" is currently logged in and that three notifications have been received for that user. The menu screen may also display items such as settings and logout.

[0228] Information corresponding to an item selected on the menu screen is displayed in the area on the right side of the screen shown in Fig. 20. An example in which the dashboard is selected on the menu screen will be described below.

[0229] As shown in FIG. 20 , the dashboard screen displays information such as the number of times the positioning application was used during a predetermined period, the accuracy rate of the accuracy assessment, and the time until the accuracy assessment was deemed acceptable. Consider a case in which a caregiver (worker) adjusts the positions of one or more patients for whom he or she is responsible using a positioning application installed on a smartphone. For example, the caregiver uses positioning application A to perform various position adjustments for patient A according to an assistance schedule, such as position adjustments for mealtimes (e.g., to prevent aspiration), position adjustments for diaper changes, and position adjustments for preventing bedsores (position changes). Since one caregiver may be responsible for multiple patients, the same caregiver may also use positioning application B to perform various position adjustments for patient B.

[0230] A device such as a smartphone accumulates log data indicating which positioning application was launched and when it was terminated. Furthermore, since the positioning application may output the result of a correct / incorrect judgment as described above, the log data may include data indicating whether a positioning application launched at a certain timing was judged as OK and terminated, or whether it was judged as NG. Furthermore, since the positioning application may output the timing at which it was judged as OK, this timing may be included in the log data. Furthermore, as described above, one positioning application may include multiple training data (tabs). Therefore, the log data may include data identifying which tab of which positioning application was processed.

[0231] The log data accumulated by the operation of the positioning application may be transmitted to, for example, the server system 10. The transmission timing is arbitrary, but the log data of the positioning application is transmitted, for example, at the timings shown in steps S102 and S104 in FIG.

[0232] The server system 10 then displays the number of uses, accuracy rate, and time to OK based on the acquired log data. The number of uses is, for example, the total number of times the positioning application was turned on during the display target period. The accuracy rate is the percentage of the number of times the application was determined to be OK out of the total number of uses during the display target period. The time to OK is a statistical value (average, median, etc.) of the time it takes for the positioning application to be determined to be OK after it is launched. In the example of FIG. 20 , the “Months” tab is selected to set one month as the display target period, and the results displayed are that the number of uses during that month was 47, the accuracy rate was 14 percent, and the average time to OK was 1 minute and 12 seconds.

[0233] The display period may be switchable between one month and one week as shown in Fig. 20. Alternatively, a different period may be set as the display period. For example, the server system 10 displays the number of uses and the like based on the data corresponding to the logged-in user among the log data transmitted from various devices and acquired within the display period.

[0234] If the logged-in user is a worker (general staff member), the data corresponding to the logged-in user may be data representing the results of the position adjustment performed by the user. Alternatively, the results of the position adjustment for the entire user group to which the user belongs may be displayed. If the logged-in user is an administrator (e.g., a functional trainer), the data corresponding to the logged-in user may be data representing the execution results of part or all of an application created by the functional trainer. If an application created by the functional trainer is used on the terminals of multiple workers (general staff members), data for each terminal may be displayed. If the user is the manager of a nursing care facility, the data corresponding to the logged-in user may be the position adjustment results for a specific caregiver belonging to the nursing care facility, or the position adjustment results for the entire nursing care facility.

[0235] Furthermore, the log data to be displayed is not limited to being automatically selected, but may be manually determined by the user. For example, the log data may be stored as a file in the server system 10. The file format here is, for example, CSV (comma separated values) format, but other file formats may also be used. For example, when a user selects a log data file using the file selection field in FIG. 20 , the dashboard screen displays data contained in the log data file within the display target period. Furthermore, the log data file may not be stored in the server system 10 but may be stored in a PC. In this case, the PC transmits the selected log data file to the server system 10 via a web browser, and the server system 10 executes a process to display a screen corresponding to the selected log data file on the web browser. As described above, various methods, such as automatic selection and manual selection, can be applied to the process of selecting data to be displayed. For example, the file selection display items in FIG. 20 may be omitted.

[0236] 21A and 21B are examples of dashboard screens that display detailed information about the number of uses, accuracy rate, and time to OK. For example, the screens of Fig. 21A and Fig. 21B may be displayed when the scroll bar on the screen of Fig. 20 is scrolled downward.

[0237] The detailed information may be displayed in units of one day or one hour. The display unit may also be a period of other length. FIG. 21A shows an example in which detailed information about one month's worth of log data is displayed in units of one day. As shown in FIG. 21A , the detailed display may include a display that allows the positioning application (target care recipient) to be identified. In the example of FIG. 21A , there are two care recipients, AAA and BBB, and position adjustments have been performed using the positioning application for these two care recipients.

[0238] The number of uses per day is displayed as a graph (e.g., a bar graph). The horizontal axis of the graph represents time (in daily units), and the vertical axis represents the total number of uses. In this case, the area corresponding to AAA and the area corresponding to BBB are displayed in different ways (e.g., different colors). In this way, the user can easily see the history of how many position adjustments were made, for which person being assisted, and on which date.

[0239] Furthermore, when the mouse cursor is placed over any area of ​​the bar graph (mouseover), the accuracy rate corresponding to that area may be displayed separately in a pie chart. In the example of FIG. 21A, the mouse cursor is placed over the bar graph representing the number of uses on June 29, 2023, and accompanying information is displayed indicating that the number of uses for the assisted person AAA was four and the number of uses for the assisted person BBB was 17. In this case, the accuracy rate for the assisted person AAA is displayed as a pie chart (hereinafter also referred to as the first pie chart) showing the percentage of the four uses that were judged to be OK. In this example, since one use was judged to be OK, a pie chart showing an accuracy rate of approximately 25% is displayed. Similarly, the accuracy rate for the assisted person BBB is displayed as a pie chart (hereinafter also referred to as the second pie chart) showing the percentage of the 17 uses that were judged to be OK. Although omitted in FIG. 21A, a scale representing percentages may be displayed with the pie chart, or specific accuracy rate values ​​may be displayed. 21A, the first pie chart for the assisted person AAA and the second pie chart for the assisted person BBB are arranged above and below each other, but this is not limited to this. For example, on the dashboard screen, one of the first pie chart and the second pie chart may be displayed on the inner periphery and the other on the outer periphery, so that the two pie charts are arranged concentrically. In this way, it is possible to clearly present the differences in the accuracy rates for each assisted person.

[0240] Furthermore, the average time to OK is displayed on a graph (e.g., a curved line graph) with the horizontal axis representing time (daily) and the vertical axis representing the average time to OK. As with the number of uses, the points corresponding to AAA and BBB are displayed in different formats. In this way, the user can easily see the history of the time required to get the patient into an appropriate posture for each day and for each person being assisted.

[0241] FIG. 21B shows an example screen when one hour is selected as the display unit for detailed information. For example, when an item corresponding to "Hours" is selected in FIG. 21A, the screen shown in FIG. 21B is displayed. Similarly, when an item corresponding to "Days" is selected in FIG. 21B, the screen shown in FIG. 21A is displayed. The display content in FIG. 21B is the same as that in FIG. 21A, except that the display unit has been changed to one hour. For example, the number of uses is displayed using a bar graph, the time until OK is displayed using a line graph, and the accuracy rate within a given hour is displayed as a pie chart based on mouse-over operations. Note that in FIG. 21B, because the display unit is short, the display is limited to two days' worth of data rather than displaying all information for one month (for example, from midnight on June 30th to midnight on July 2nd). For this reason, FIG. 21B does not display data for assisted person AAA, who has no usage history during the two days to be displayed. However, the fact that information about multiple assisted people can be displayed in an identifiable manner is also true in FIG. 21B.

[0242] As described above with reference to FIG. 18 , from the perspective of preventing pressure ulcers and other conditions, maintaining the same position for a long period of time is not recommended, so it is useful to change the position every few hours. However, if the display unit for detailed information is set to one day or more, it is difficult to grasp the timing of position adjustments within a day. For example, even if log data shows that position adjustments were performed a set number of times within a day, it is difficult to distinguish whether the adjustments were performed at appropriate intervals to prevent excessive maintenance of a single position, or whether the position adjustments were concentrated in a short period of time (i.e., whether the same position was maintained for a long period of time). Considering the above, it is useful to display log data in units of shorter periods than one day, as shown in FIG. 21B . In the example of FIG. 21B , the data for June 30th, which shows a sufficient usage history for the care recipient BBB, shows a peak in the number of uses every two hours. This indicates that position changes were performed at appropriate intervals to prevent the same position from being maintained for long periods of time.

[0243] 22A to 22C are example screens that display detailed information for each person being assisted (each positioning application). For example, the screens shown in Fig. 22A to 22C may be displayed when the scroll bar on the screen of Fig. 21A or 21B is scrolled downward.

[0244] As shown in Fig. 22A, the left side of this screen displays the name, number of times used, accuracy rate, and average time until OK for each person being assisted. As in Fig. 20, the display target period may be one month or one week.

[0245] When a user selects a user in the left area, details about the selected user are displayed in the right area of ​​the screen. The displayed information may include history, day of the week, and worst case failure. The history, as shown in FIG. 22A, is a log of the date and time of use of the positioning application for the user, the tab, the result of the evaluation, and the time until the OK.

[0246] The date and time of use may be, for example, the timing of starting or ending the application. In other words, when "History" in Fig. 22A is selected, the dashboard screen displays details of each operation history of the positioning application.

[0247] A tab is information indicating which of one or more pieces of teacher data registered in the positioning application was used to adjust the position. For example, in the setting mode using FIG. 18 , teacher data corresponding to the right lateral position is registered as Tab 1, and teacher data corresponding to the left lateral position is registered as Tab 2. The text "Right" is registered as the name of Tab 1, and the text "Left" is registered as the name of Tab 2. In the example shown in FIG. 22A , right and left are recorded as the usage history of the teacher data (tabs). In other words, the history of the use of Tab 1 corresponding to the right lateral position and the history of the use of Tab 2 corresponding to the left lateral position remain.

[0248] The judgment result indicates whether the result of the judgment is OK or NG. As described above, whether it is OK or NG is determined based on the angle formed by the selected skeletons (feature points of the skeleton tracking result), the respective positions of the selected multiple skeletons, etc.

[0249] The time to OK is as described above, and indicates the time elapsed from the start of the application until it is determined to be OK. If the result of the determination is NG, the time to OK will be blank because no OK determination has been made.

[0250] This allows users to know the timing of use of the target positioning application in detail, and also displays the training data used, the judgment results for each use, and the time until OK. For example, in the case of the position change for preventing bedsores described above, it is meaningless to have the user assume the same posture each time; it is important to have the user assume a different posture from the previous time. Therefore, for example, when adjusting position at night, the user can determine whether "right" (Tab 1) and "left" (Tab 2) alternate. Alternatively, the server system 10 may automatically determine whether a position change is appropriate by determining whether the tabs are being changed periodically. For example, this determination may be made by using text corresponding to each tab to determine whether the postures for each tab are different.

[0251] Furthermore, because the judgment results and the time until OK are known for each date and time of use, it is possible to determine, for example, that position adjustment during the day is easy but that position adjustment at night is difficult. An unsuccessful adjustment may mean that the position is likely to be judged as NG, or that the position is judged as OK but the time until OK is longer than a predetermined value. The type of assistance may also be identified by matching the time with the assistance schedule. In this case, it is possible to determine that position adjustment during mealtimes is successful but that position adjustment for diaper changes is not successful. Furthermore, because the judgment results and the time until OK are known for each tab, it is possible to determine, for example, that position adjustment to position A is easy but that position adjustment to position B is difficult.

[0252] Fig. 22B is an example of a screen that is displayed when "By day of the week" is selected in the area on the right side of the screen shown in Fig. 21. As shown in Fig. 22B, when By day of the week is selected, the NG rate for each day of the week may be displayed. The NG rate may be numerical data expressed as 100 - accuracy rate (%).

[0253] In nursing care facilities and the like, caregivers often work in shifts, and it is expected that the same care recipient will be assigned a different caregiver (or caregiver team) on different days of the week. In this regard, by displaying the display shown in FIG. 22B , the days of the week on which the judgment result is likely to be NG can be clearly shown to the user. This makes it possible to identify caregivers who are likely to be judged NG and provide appropriate guidance. Alternatively, it is also possible to identify caregivers with a low NG rate and share their knowledge (for example, by having them create a nursing care application).

[0254] Fig. 22C is an example of a screen that is displayed when "Worst NG" is selected in the area on the right side of the screen shown in Fig. 21. As shown in Fig. 22C, when Worst NG is selected, the reason for the NG determination is displayed.

[0255] For example, when performing an NG judgment using multiple feature points selected from the skeleton tracking results as described above, the positioning application can identify the feature point among the multiple feature points that caused the NG judgment (hereinafter referred to as the NG cause point). For example, the positioning application compares multiple feature points included in the training data with multiple feature points acquired from images captured in operation mode, and judges the result as NG if the degree of difference is equal to or greater than a threshold. In this case, the positioning application determines the feature point that deviates most from the training data as the NG cause point. Alternatively, a priority order for the skeleton may be set in advance, and the positioning application may identify the NG cause point based on the priority order when an NG judgment is made.

[0256] In the example of Figure 22C, it is displayed that, of the total number of times that a test was judged to be NG, the percentage of the left ankle that was the cause of the NG test was 14%, and the percentage of the left wrist that was the cause of the NG test was 34%. In this way, it is possible for the user to confirm the reasons for the NG test as statistical values. For example, when a functional trainer views this data, it is possible to consider whether the photographs used as training data are bad, or what kind of guidance should be given to the worker (caregiver) to improve the accuracy rate.

[0257] In the above example, "History," "By Day of the Week," and "Worst NG" are displayed, but the present invention is not limited to this example, and other items may be displayed. For example, the history items that took the longest time to be judged as OK may be displayed in a ranking format.

[0258] Furthermore, when using a device such as a PC, the display size is larger than that of a mobile terminal device such as a smartphone. Furthermore, peripheral devices such as a keyboard and a pointing device can be easily used, making user operation more convenient. Taking this into consideration, the positioning application of this embodiment may be edited using a web browser on a PC.

[0259] For example, a supervisor, such as a functional trainer, may use a smartphone to perform the registration process shown in FIGS. 18-19B and then use a web browser to perform detailed editing. The editing process here involves editing, for example, photographic data, which serves as training data. For example, when superimposing a geometric object such as a triangle or square on an image, using a PC makes it easy to adjust the shape and size of the geometric object. Furthermore, using a PC for trimming, brightness and saturation adjustment, filter processing, and other operations has fewer limitations on functionality and ease of operation. In this embodiment, multiple types of terminals of different sizes may be used as mobile terminal devices. These multiple types of terminals include, for example, tablet terminals, which are relatively large and allow for easy data viewing and operation, and smartphones, which are relatively small and highly portable. Therefore, in this embodiment, three types of screens may be used in combination: a setting screen for a PC, a setting screen for a tablet terminal, and a setting screen for a smartphone. This allows for a screen display appropriate for the device being used. For example, a supervisor may use a PC when editing an application, a tablet terminal when explaining the application to a worker, and a smartphone when providing actual assistance.

[0260] 5. Data Structure Example In the method of this embodiment, the data structure used in communication may be standardized regardless of the source device or the destination device. Here, the data used in communication is, for example, a MAC frame, and more specifically, a data frame. Standardizing the data frame structure makes it possible to unify communication control regardless of the vendor, even when various devices are used as the first device 100 and the second device 200. In a narrow sense, the data frame structure refers to the bit allocation in the frame body.

[0261] 23 shows an example of the format of a MAC frame. The MAC header includes the following fields: Frame Control, Duration ID, Address 1, Address 2, Address 3, Sequence Control, Address 4, QoS Control, and HT Control. Some of these fields may be omitted.

[0262] Frame Control includes a type field for determining whether the target MAC frame is a data frame, a management frame, or a control frame. Duration / ID stores information indicating the planned period for using radio waves. Address 1 to Address 4 store information indicating the addresses of the receiving and sending devices. Sequence Control corresponds to the sequence number of the data to be transmitted. QoS Control stores information used for QoS control. HT Control is a field used, for example, in management frames. The structure of the frame body will be described later using Figure 24. The trailer is, for example, an FCS (Frame Check Sequence). The FCS is information used to detect errors in the frame, and is, for example, a checksum code.

[0263] As shown in FIG. 23, the server system 10, the first device 100, and the second device 200 each transmit a data frame including a MAC (Media Access Control) header, a frame body, and a trailer at the data link layer of their communication with each other.

[0264] 24 is a diagram illustrating an example of bit allocation in the frame body of a data frame transmitted by the server system 10, etc. As shown in Fig. 24, the frame body may include the following fields: data type ID1, User ADL, scene flag, device type ID, data type ID2, Instruction length, contents, and expiration date.

[0265] Data type ID1 and data type ID2 are fields that store information indicating the type and purpose of the data frame. The types here may include five types, for example, ID=1 to ID=5. Figure 24 shows an example in which the data type ID is divided into two areas. For example, data type ID1 is a fixed-length field of 1 bit, and data type ID2 is a fixed-length field of 2 bits or more. However, these two areas may be combined into one.

[0266] The User ADL is a field for storing the ability information of the person receiving assistance. For example, the ability information is numerical data that indicates which level the ability of the person receiving assistance belongs to when the degree of ability is classified into a predetermined number of levels. Since the number of bits required is known depending on the definition of the ability information, the User ADL may be a fixed-length field.

[0267] The scene flag is a field for storing scene information. For example, the scene information may include a bit indicating whether the number of caregivers is greater than or equal to a predetermined number. The scene information may also include a bit specifying the type of care.

[0268] The device type ID is a field for storing device information. The device information may be a type ID that identifies the type of device, or a device ID that uniquely identifies each device.

[0269] "Instruction length" and "contents" are fields used for sending data to the destination device. "Instruction length" is a field that stores the data length. "Contents" stores information that identifies the specific data content.

[0270] FIG. 25 is a diagram illustrating specific examples of contents according to the value of data type ID. For example, data type ID = 0 indicates an "alert" to the device to which data is sent (an alert to the user of that device). In this case, contents includes an alarm contents field. The alarm contents is a variable-length field that stores information indicating the specific alert content. The alarm contents may include information specifying the alert mode (display, light emission, vibration, sound output), or may include information specifying the specific alert content (text, light emission color, vibration pattern, sound).

[0271] Data type ID = 1 indicates an instruction to "move / transport" the device to which data is being sent. In this case, the contents include the fields current location and destination. Current location is information that specifies the current location of the target device. Destination is information that specifies the target location of the target device. How the current location and target location are expressed is up to you, but they are fixed-length fields that represent, for example, two-dimensional or three-dimensional coordinate values.

[0272] Data type ID = 2 indicates "control" of the device to which data is sent. In this case, the contents includes the fields "controlled part" and "how to control." The controlled part stores information specifying the part to be controlled in the target device. The "how to control" stores information specifying the specific control method for the part to be controlled. In the case of a nursing bed as an example, the contents may store information on which of multiple bottom parts to drive and by what angle.

[0273] Data type ID = 3 represents "recommendations, etc." (recommendations, etc. for the user of the device) to the data destination device. In this case, the contents includes the recommend contents field. The recommend contents is a variable-length field that represents the contents of the recommendation or news. Alternatively, data type ID = 3 may represent the transmission of a "log" to the destination device. In this case, the contents may include the log contents field. The log contents is a variable-length field that represents the log data accumulated on the device.

[0274] Data type ID = 4 indicates a "table confirmation request" to the device to which data is to be sent. In this case, the contents include the table information and app information fields. This is used, for example, when the confirmation request in steps S203 and S210 of FIG. 10 is sent from the first device 100 or the second device 200 to the server system 10. Table information is information about the correspondence information (table data) to be confirmed, and may be the correspondence information itself or information such as the update timing of the correspondence information. App information is information indicating applications installed on the device. Note that data type ID = 4 may also be used when the first device 100 sends a cooperation request to the second device 200 as shown in step S403 of FIG. 14, step S503 of FIG. 15, etc.

[0275] Data type ID = 5 indicates "table delivery" that is executed based on table confirmation. In this case, the contents include the table data and application fields. This is used when the server system 10 transmits the latest data to the first device 100 or the second device 200, for example, as in steps S206 and S213 of FIG. 10. The table data is the latest association information. The application is the application that needs to be installed on the device.

[0276] 9 , the server system 10 also transmits the latest data to the first device 100 or the second device 200, but data type ID = 5 is not used in these cases. This is because these steps are not caused by a correspondence information confirmation request from the first device 100 or the second device 200. In these cases, the correspondence information or the care application is transmitted using, for example, recommend contents or log contents of data type ID = 3. For example, when transmitting correspondence information or an application, the processing unit 11 of the server system 10 may determine whether a data packet of data type ID = 4 has previously been received (whether a correspondence information confirmation request has been received), and if so, set the data type ID to 5, and if not, set the data type ID to 3.

[0277] As mentioned above, the length of the contents field in the frame body varies depending on the data type ID and the specific instruction content. Therefore, as shown in Figure 24, the frame body may include an instruction length field before the contents field, which stores the length of the contents field. Since the maximum length of the contents field is considered to be known, the instruction length may be, for example, a fixed-length field.

[0278] Furthermore, methods for setting communication priorities are known in various communication systems. For example, in IEEE 802.11, a method for realizing QoS has been standardized in IEEE 802.11e. Specifically, IEEE 802.11e uses a method called Enhanced Distributed Channel Access (EDCA), which prioritizes the transmission of frames with high priority. Since EDCA is a well-known method, a detailed description thereof will be omitted. Furthermore, priorities may be set using methods other than EDCA.

[0279] In the method of this embodiment, for example, "recommendations, etc." and "logs" are set to have lower priorities than the other functions, "alert," "movement / transport," "control," "table confirmation," and "table distribution." In this way, it is possible to set the priority of the correspondence information confirmation request from the first device 100 and the second device 200 according to this embodiment higher than the transmission of recommendations, logs, etc.

[0280] Therefore, even if the data to be transmitted is common to the association information and application, the communication priority is changed between when data type ID = 3 and when data type ID = 5. Specifically, when processing is based on a request to confirm association information from the first device 100 or the like (data type ID = 5), the communication priority is set relatively high. This allows the server system 10 to quickly respond to a request to confirm association information, for example. This makes it possible to shorten the waiting time after a trigger is detected, for example, the time from step S201 in FIG. 10 until step S217 is performed.

[0281] On the other hand, if the processing is not based on a request to confirm association information from the first device 100 or the like (data type ID = 3), even if the data content is association information or an application, the communication priority is set relatively low. This makes it possible to reduce the communication load when the linked processing is not to be executed immediately.

[0282] 24, the frame body may also include a field corresponding to the expiration date. The expiration date is information indicating the expiration date of each message, which is set in consideration of the timing at which the message (data frame) is issued and the valid duration of each set data (User ADL, etc.). The expiration date may be data specifying the hour and minute, data specifying the year, month, and day, or both.

[0283] For example, if the ADL in the server system 10 is updated once a day, the expiration date of a message is set to a maximum of 24 hours. The server system 10, the first device 100, and the second device 200 that receive the message compare the expiration date set in the message with the current time and discard the message if the expiration date has passed. This prevents outdated data from being referenced after being transferred between multiple terminals and server systems 10. In the above example of the ADL, when a new ADL is calculated in the server system 10, it is possible to invalidate messages whose old ADL data is stored in the User ADL.

[0284] Although the present embodiment has been described in detail above, those skilled in the art will readily understand that many modifications are possible without substantially departing from the novel features and advantages of the present embodiment. Therefore, all such modifications are intended to be included within the scope of the present disclosure. For example, a term described at least once in the specification or drawings with a different term having a broader or equivalent meaning may be replaced with that different term anywhere in the specification or drawings. Furthermore, all combinations of the present embodiment and modifications are also included within the scope of the present disclosure. Furthermore, the configurations and operations of the information processing system, server system, first device, second device, etc. are not limited to those described in the present embodiment, and various modifications are possible.

Claims

1. An information processing system comprising: a first device used to assist a person being assisted; a second device used in conjunction with the first device in the assistance of the person being assisted; and a server system connected to the first device and the second device, wherein the server system stores correspondence information that associates a plurality of devices used in conjunction with the assistance of the person being assisted, the first device, when acquiring trigger information indicating the start of the assistance, determines a device to be linked based on the correspondence information and sends a linking request to the second device determined to be the device to be linked, and the first device and the second device perform pre-processing on the correspondence information depending on the connection state with the server system, and then start processing related to the assistance.

2. An information processing system according to claim 1, wherein the server system transmits the correspondence information to the first device and the second device, the first device stores the correspondence information as first correspondence information, and the second device stores the correspondence information as second correspondence information.

3. An information processing system as described in claim 2, wherein the first device, when the trigger information is acquired, determines whether it is connected to the server system, and, if connected, determines whether the first correspondence information is up to date by communicating with the server system.

4. An information processing system as described in claim 3, wherein the second device, when receiving the collaboration request, determines whether it is connected to the server system, and if connected, determines whether the second correspondence information is up to date by communicating with the server system.

5. An information processing system according to claim 2, wherein, when the first device is not connected to the server system, the first device transmits the link request including information regarding the first correspondence information to the second device, and, when the second device is connected to the server system upon receiving the link request, the second device communicates with the server system to determine whether the first correspondence information and the second correspondence information are up to date.

6. An information processing system according to claim 2, wherein, if the first device is not connected to the server system when the trigger information is acquired and the second device is not connected to the server system when the collaboration request is received, at least one of the first device and the second device selects one of the correspondence information based on a comparison process of the first correspondence information and the second correspondence information, and the first device and the second device cooperate to execute the assistance-related processing based on the selected correspondence information.

7. An information processing system according to claim 6, wherein at least one of the first device and the second device selects the association information with a newer update date and time from the first association information and the second association information, and the first device performs a process of presenting the update date and time of the association information to a user before executing the processing related to the assistance.

8. According to any one of claims 1 to 7, the association information includes information regarding tacit knowledge data that is a digitalized version of the tacit knowledge of an experienced caregiver used in the assistance, and at least one of the first device and the second device is an information processing system that acquires the tacit knowledge data from the server system based on the association information and executes processing related to the assistance based on the tacit knowledge data.

9. An information processing system according to claim 5, wherein the association information includes information relating to tacit knowledge data that is a digitized version of the tacit knowledge of an experienced caregiver used in the care; and when it is determined that the first association information is not the latest, the second device receives from the server system the latest first association information and the tacit knowledge data for the first device identified by the latest first association information; and the first device acquires from the second device the latest first association information and the tacit knowledge data for the first device.

10. An information processing system according to claim 7, wherein the correspondence information includes information specifying first tacit knowledge data used in the first device and second tacit knowledge data used in the second device as information relating to tacit knowledge data that is a digitalized version of the tacit knowledge of an experienced caregiver used in the assistance, and at least one of the first device and the second device selects, from the first correspondence information and the second correspondence information, the correspondence information in which the first tacit knowledge data is already installed in the first device and the second tacit knowledge data is already installed in the second device.

11. A control method for an information processing system including a first device used to assist a person being assisted, a second device used in conjunction with the first device in the assistance of the person being assisted, and a server system connected to the first device and the second device, comprising: storing in the server system correspondence information that associates a plurality of devices used in conjunction with the assistance of the person being assisted; when trigger information indicating the start of the assistance is acquired in the first device, determining a device to be linked based on the correspondence information and sending a linkage request to the second device determined to be the device to be linked; performing pre-processing related to the correspondence information depending on the connection state between the first device and the second device and the server system, and then starting processing related to the assistance.

Images