Information processing system and information processing method

Abstract

To provide an information processing system and an information processing method, etc., that appropriately support the provision of assistance to persons being assisted by caregivers.SOLUTION: The information processing system includes a first terminal device that operates in accordance with a positioning application that performs processing based on teaching data that represents correct answers to positions regarding persons and / or things in the provision of assistance, and a second terminal device that is used by a second user who supports the provision of assistance regarding a first user using the first terminal device. The first terminal device transmits a sensing result to the second terminal device via a network, and the second terminal device presents the sensing result to the second user and accepts input for setting up teaching data by the second user, with the first terminal device acting in accordance with a positioning application that corresponds to the teaching data created by setup input.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and the like. [Background technology]

[0002] A system for use when a caregiver provides care to a care recipient has been known. Patent Document 1 discloses a method for generating information to be provided regarding the condition of a resident in a living space based on time-varying changes in detected information acquired by a sensor placed in the living space.

[0003] Furthermore, Patent Document 2 discloses a method for providing an action such as calling an ambulance based on biometric information and a user response acquired from a dialogue device in remote medical care. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-18760 [Patent Document 2] US Patent Application Publication No. 2018 / 0114591 Summary of the Invention [Problem to be solved by the invention]

[0005] An information processing system and an information processing method are provided that appropriately support a caregiver in providing assistance to a person being assisted. [Means for solving the problem]

[0006] One aspect of the present disclosure relates to an information processing system that includes a first terminal device that stores a positioning application that performs processing based on teacher data that represents the correct position of at least one of a person and an object during assistance and operates in accordance with the positioning application, and a second terminal device used by a second user that supports the assistance of a first user using the first terminal device, wherein the first terminal device transmits sensing results of at least one of the person and the object during the assistance to the second terminal device via a network, the second terminal device presents the sensing results to the second user and accepts setting input of the teacher data by the second user, and the first terminal device operates in accordance with the positioning application that corresponds to the teacher data created by the setting input.

[0007] Another aspect of the present disclosure relates to an information processing method in an information processing system including a first terminal device that stores a positioning application that performs processing based on teacher data that represents the correct position of at least one of a person and an object during assistance and operates in accordance with the positioning application, and a second terminal device used by a second user that supports the assistance of a first user using the first terminal device, wherein sensing results regarding at least one of the person and the object during the assistance are transmitted from the first terminal device to the second terminal device via a network, the second terminal device presents the sensing results to the second user, accepts setting input of the teacher data by the second user, and operates the first terminal device in accordance with the positioning application corresponding to the teacher data created by the setting input. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example of the configuration of an information processing system. [Figure 2] FIG. 1 illustrates an example of the configuration of a server system. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of a first terminal device. [Figure 4A] This is an example of training data for a positioning application. [Figure 4B] This is an example of training data that is superimposed and displayed in a positioning application. [Figure 5] FIG. 2 is a diagram illustrating an example of the configuration of a second terminal device. [Figure 6] FIG. 10 is a sequence diagram illustrating a process flow in a setting mode. [Figure 7] FIG. 10 is a diagram illustrating a device placed around a bed. [Figure 8] FIG. 10 is a diagram illustrating an example of the relationship between an application and a device in adjusting a bed position. [Figure 9A] 10 is an example of a screen used to acquire teacher data in the setting mode. [Figure 9B] 10 is an example of a screen used to acquire teacher data in the setting mode. [Figure 9C] 10 is an example of a screen used to acquire teacher data in the setting mode. [Figure 10] 10 is an example of a screen used to obtain additional information in a setting mode. [Figure 11] FIG. 10 is a diagram illustrating a device placed on a bed. [Figure 12] FIG. 10 is a diagram illustrating devices arranged around a wheelchair. [Figure 13] FIG. 10 is a diagram illustrating an example of the relationship between an application and a device in adjusting a wheelchair position. [Figure 14] 10 is an example of a screen used to acquire teacher data in the setting mode. [Figure 15] FIG. 10 is a diagram illustrating a pressure sensor disposed on a wheelchair. [Figure 16] 10 is an example of a screen used to acquire teacher data in the setting mode. [Figure 17] 10 is an example of a screen that presents the results of creating an algorithm. [Figure 18] 10A and 10B are diagrams illustrating devices placed during meals. [Figure 19]10 is an example of a screen used to acquire teacher data in the setting mode. [Figure 20A] 10 is an example of teacher data to be set. [Figure 20B] 10 is an example of teacher data to be set. [Figure 21] 10 is an example of a screen used to acquire teacher data in the setting mode. [Figure 22] FIG. 10 is a sequence diagram illustrating a process of selecting teacher data. [Figure 23A] FIG. 10 is a sequence diagram illustrating a process of changing an operation mode in consideration of a load. [Figure 23B] FIG. 10 is a sequence diagram illustrating a process of changing an operation mode in consideration of a load. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, the present embodiment will be described with reference to the drawings. In the drawings, identical or equivalent elements are designated by the same reference numerals, and duplicate explanations will be omitted. Note that the present embodiment described below does not unduly limit the content described in the claims. Furthermore, not all of the configurations described in the present embodiment are necessarily essential components of the present disclosure.

[0010] 1. System configuration example 1 shows an example of the configuration of an information processing system 10 according to this embodiment. The information processing system 10 according to this embodiment digitizes the "intuition" and "tacit knowledge" of caregivers in tasks that are performed based on their "intuition" or "tacit knowledge" in, for example, medical or nursing care situations, and thereby provides instructions to caregivers so that they can provide appropriate care regardless of their level of proficiency.

[0011] The caregiver here may be a caregiver at a nursing facility, or a nurse or licensed practical nurse at a medical facility such as a hospital. That is, assistance in this embodiment includes various actions to support the person being assisted, and may include nursing care or medical actions such as giving an injection. The person being assisted here is a person who receives assistance from the caregiver, and may be a resident of a nursing facility or a patient who is hospitalized or visits a hospital. The person being assisted may also be, for example, a person who may be suffering from dementia.

[0012] The assistance in this embodiment may also be provided at home. For example, the person being assisted in this embodiment may be a person requiring care receiving home care, or a patient receiving home medical care. The caregiver may be a family member of the person requiring care or the patient, or a visiting helper. For example, when a family member of the person being assisted who is not a professional in home care provides care, the method of this embodiment may support the assistance provided by the family member by utilizing the tacit knowledge of an experienced caregiver or care manager.

[0013] As shown in FIG. 1, the information processing system 10 includes a server system 100, a first terminal device 200, a second terminal device 300, and a sensing device 400. However, the configuration of the information processing system 10 is not limited to that shown in FIG. 1, and modifications such as omitting some components or adding other components are possible. For example, there may be multiple first terminal devices 200. There may also be multiple second terminal devices 300. In addition, in FIG. 1, a bedside sensor 420 and a detection device 430, which will be described later with reference to FIG. 11, are illustrated as examples of the sensing device 400. However, the sensing device 400 is not limited to these and may be a seat sensor 440, which will be described later with reference to FIG. 15, a choking detection device 460, which will be described later with reference to FIG. 18, or other devices. The sensing device 400 may also include a bed 610, a mattress 620, a wheelchair 630, or the like, which are capable of detecting pressure. In the following, when it is not necessary to distinguish between multiple sensing devices 400, they will be simply referred to as sensing device 400.

[0014] The server system 100 is connected to a first terminal device 200 and a second terminal device 300 via, for example, a network NW. The network NW here is, for example, a public communication network such as the Internet. However, the network NW is not limited to a public communication network and may be a LAN (Local Area Network) or the like. For example, the server system 100 may perform communication in accordance with the IEEE802.11 standard. However, various modifications are possible regarding the communication method between the devices.

[0015] The server system 100 may be a single server or may include multiple servers. For example, the server system 100 may include a database server and an application server. For example, the application server may perform processing to create a positioning application AP1. The positioning application AP1 here is software that performs processing related to the position of at least one of a person and an object during assistance. Details of the positioning application AP1 will be described later. The database server may also store the created positioning application AP1. The multiple servers here may be physical servers or virtual servers. If a virtual server is used, the virtual server may be provided on a single physical server, or may be distributed across multiple physical servers. As described above, the specific configuration of the server system 100 in this embodiment can be modified in various ways.

[0016] The first terminal device 200 is a device used, for example, by a person receiving care at home, or by a family member or the like who takes care of the person receiving care. However, the person receiving care here may be a person receiving care residing in a facility such as a nursing home. Hereinafter, a user who uses the first terminal device 200 will be referred to as a first user.

[0017] The first terminal device 200 is a device on which the positioning application AP1 runs. The first terminal device 200 is, for example, a mobile terminal device such as a smartphone or a tablet terminal. However, the first terminal device 200 may be other devices, such as a personal computer (PC), a headset, or a wearable device such as augmented reality (AR) glasses or mixed reality (MR) glasses. The first user may use multiple first terminal devices 200. For example, the first user may use both a smartphone and a headset. The person being assisted and the caregiver (family member) may each use different smartphones.

[0018] The second terminal device 300 is a device used by a user with specialized knowledge about care, such as a caregiver, nurse, doctor, or care manager. Hereinafter, a user who uses the second terminal device 300 will be referred to as a second user. The second terminal device 300 is similar to the first terminal device 200, and can be realized by various devices such as a smartphone, a tablet device, a PC, a headset, or a wearable device.

[0019] The sensing device 400 is a device used to assist a person being assisted. For example, the sensing device 400 has various sensors and acquires sensing data based on the sensors. The sensing data here may be the sensor output itself, or may be information obtained by arithmetic processing based on the sensor output. The sensing device 400 may also be a device that outputs input data used when setting up or using the positioning application AP1. Details will be described later.

[0020] 2 is a block diagram showing a detailed configuration example of the server system 100. The server system 100 includes, for example, a processing unit 110, a storage unit 120, and a communication unit .

[0021] The processing unit 110 of this embodiment is configured by the following hardware. The hardware can include at least one of a circuit for processing digital signals and a circuit for processing analog signals. For example, the hardware can be configured by one or more circuit devices or one or more circuit elements mounted on a circuit board. The one or more circuit devices are, for example, an integrated circuit (IC), a field-programmable gate array (FPGA), etc. The one or more circuit elements are, for example, a resistor, a capacitor, etc.

[0022] The processing unit 110 may also be implemented by the following processor. The server system 100 of this embodiment includes a memory that stores information and a processor that operates based on the information stored in the memory. The information may be, for example, a program and various data. The memory may be the storage unit 120 or another memory. The processor includes hardware. Various processors, such as a central processing unit (CPU), a graphics processing unit (GPU), or a digital signal processor (DSP), may be used. The memory may be a semiconductor memory such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory, or may be a register, a magnetic storage device such as a hard disk drive (HDD), or an optical storage device such as an optical disk drive. For example, the memory stores computer-readable instructions, and the processor executes the instructions to realize the functions of the processing unit 110. The instructions may be instructions from an instruction set that constitutes a program, or instructions that instruct the hardware circuitry of the processor to operate.

[0023] The storage unit 120 is a work area for the processing unit 110 and stores various information. The storage unit 120 can be realized by various types of memory, and the memory may be a semiconductor memory such as an SRAM, a DRAM, a ROM (Read Only Memory), or a flash memory, or may be a register, a magnetic storage device, or an optical storage device.

[0024] The communication unit 130 is an interface for communicating via a network, and when the server system 100 performs wireless communication, it includes, for example, an antenna, an RF (radio frequency) circuit, and a baseband circuit. However, the server system 100 may also perform wired communication, in which case the communication unit 130 may include a communication interface such as an Ethernet connector and a control circuit for the communication interface. The communication unit 130 may operate under control of the processing unit 110, and may include a processor for communication control different from the processing unit 110. The communication unit 130 may perform communication according to a method specified in, for example, IEEE802.11 or IEEE802.3. However, the specific communication method can be modified in various ways.

[0025] 3 is a block diagram showing a detailed configuration example of the first terminal device 200. The first terminal device 200 includes, for example, a processing unit 210, a storage unit 220, a communication unit 230, a display unit 240, an operation unit 250, and an imaging unit 260. However, the configuration of the first terminal device 200 is not limited to that shown in FIG. 3, and modifications such as omitting some components or adding other components are possible. For example, the first terminal device 200 may have various sensors, such as an acceleration sensor, a gyro sensor, or other motion sensors, a pressure sensor, a GPS (Global Positioning System) sensor, etc., in addition to the image sensor included in the imaging unit 260.

[0026] The processing unit 210 is configured by hardware including at least one of a circuit for processing digital signals and a circuit for processing analog signals. The processing unit 210 may also be realized by a processor. Various types of processors, such as a CPU, a GPU, or a DSP, can be used as the processor. The processor executes instructions stored in the memory of the first terminal device 200, thereby realizing the functions of the processing unit 210 as processing.

[0027] The storage unit 220 is a work area for the processing unit 210 and is realized by various types of memory such as SRAM, DRAM, and ROM. The storage unit 220 stores, for example, a positioning application AP1. The processing unit 210 may operate in accordance with the positioning application AP1.

[0028] The positioning application AP1 in this embodiment is application software that presents a desired position for a person or an object on a bed 610, a wheelchair 630, or the like. The person in this case is, for example, a person being assisted or an assistant, and the object is, for example, a cushion or a diaper.

[0029] For example, the positioning application AP1 may operate in a setting mode for setting the position and a usage mode for supporting actual position adjustment according to the setting. In the setting mode, the positioning application AP1 acquires training data by capturing an image of a person or object at a desired position. In the usage mode, the positioning application AP1 superimposes the training data, which has been subjected to transparency processing, on the captured image of the person or object to be adjusted.

[0030] FIG. 4A shows an example of training data acquired in the setting mode. In the example of FIG. 4A, image information representing a desirable posture for a care recipient named "AAA" lying on the bed 610 is acquired as training data. FIG. 4B shows an example of training data that has undergone transparency processing and is superimposed on a captured image in the usage mode. For example, the terminal device 200 superimposes the image of FIG. 4B on a captured image of a care recipient whose position is to be adjusted. The caregiver assists the care recipient so that the image of the care recipient in the captured image matches the training data. This makes it possible to appropriately support the caregiver in adjusting the position. Note that while an example of superimposing training data, which is image information, has been described here, the positioning application AP1 may also output a determination result (OK / NG) as to whether the posture of the care recipient is appropriate. For example, the positioning application AP1 may determine OK / NG based on the difference between the training data and the captured image. For example, the positioning application AP1 may calculate a norm (for example, the sum of absolute differences) based on each RGB pixel value, and determine that the difference is large and NG if the ratio of pixels whose norm is equal to or greater than a predetermined threshold is greater than or equal to a predetermined value. Details of the processing executed by the positioning application AP1 will be described later.

[0031] The communication unit 230 is an interface for communication via a network, and includes, for example, an antenna, an RF circuit, and a baseband circuit. The communication unit 230 communicates with the server system 100 via, for example, the network NW. The communication unit 230 may perform wireless communication with the server system 100 in accordance with, for example, the IEEE 802.11 standard.

[0032] The display unit 240 is an interface that displays various information and may be a liquid crystal display, an organic EL display, or another type of display. The operation unit 250 is an interface that accepts user operations. The operation unit 250 may be buttons or the like provided on the first terminal device 200. The display unit 240 and the operation unit 250 may also be a touch panel that is integrally configured.

[0033] The imaging unit 260 includes an image sensor that captures an image of a predetermined imaging range and outputs image information. The image information here may be a still image or a moving image. The image information may be color or monochrome. The imaging unit 260 may also include a depth sensor that detects the distance to the subject, or a sensor (e.g., an infrared sensor) that detects the heat of the subject.

[0034] The first terminal device 200 may also include components not shown in FIG. 3, such as a light emitting unit, a vibration unit, a sound input unit, and a sound output unit. The light emitting unit is, for example, an LED (light emitting diode) and provides notification by emitting light. The vibration unit is, for example, a motor and provides notification by vibration. The sound input unit is, for example, a microphone. The sound output unit is, for example, a speaker and provides notification by sound.

[0035] Fig. 5 is a block diagram showing a detailed configuration example of the second terminal device 300. The second terminal device 300 includes, for example, a processing unit 310, a storage unit 320, a communication unit 330, a display unit 340, an operation unit 350, and an imaging unit 360. However, the configuration of the second terminal device 300 is not limited to that shown in Fig. 5, and modifications such as omitting some components or adding other components are possible. The configuration of each unit is the same as that of the first terminal device 200, so detailed explanations will be omitted.

[0036] It is known that adjusting the position of the person being assisted, the position of the caregiver, and the position of objects placed around the person being assisted are important when providing care. For example, adjusting the position of the person being assisted is useful for reducing the risk of bedsores and falls. Adjusting the position of the caregiver is useful for teaching the movements required to properly perform care. Adjusting the position of objects is useful, for example, when placing cushions or the like to help the person being assisted assume an appropriate posture. By using the positioning application AP1 described above, it is possible to support position adjustment during care.

[0037] However, how to adjust the position changes depending on the situation and is tacitly known by experienced caregivers. For example, when adjusting the position of a person being assisted, the desired posture may differ depending on the attributes of the person being assisted. Attributes here include age, gender, height, weight, medical history, medication history, etc. Therefore, the desired posture will differ depending on the person being assisted, and even for the same person, the desired posture may change if there are factors that change the attributes. The same applies to the position of the caregiver providing the care.

[0038] Therefore, when using the positioning application AP1, it is difficult for anyone other than an experienced caregiver with tacit knowledge to set the training data representing the desired position. However, in recent years, home care and remote medical care have become widespread. In these cases, it is difficult for an experienced caregiver to directly adjust the posture of the person being assisted, so it is desirable to set the caregiver's tacit knowledge remotely. Conventional methods such as Patent Documents 1 and 2 do not disclose a method for efficiently setting tacit knowledge remotely.

[0039] As shown in FIG. 1 , the information processing system 10 of this embodiment includes a first terminal device 200 and a second terminal device 300. The first terminal device 200 stores a positioning application AP1 that performs processing based on training data that indicates the correct position of at least one of a person and an object during assistance, and operates in accordance with the positioning application AP1. The second terminal device 300 is used by a second user who supports assistance for the first user who uses the first terminal device 200. Note that the first user may include a person being assisted (e.g., a person receiving home care) and a caregiver (e.g., a family member of the person being assisted), and the support for assistance for the first user may be supporting the actions of the person being assisted or supporting the assistance actions of the family member.

[0040] The first terminal device 200 then transmits the sensing results relating to at least one of the person and the object being assisted to the second terminal device 300 via the network NW. The sensing results here may be captured images, which are sensing data from the image sensor (imaging unit 260) of the first terminal device 200, or various information, which are sensing data from the sensing device 400. Specific examples of the sensing data from the sensing device 400 will be described later. The second terminal device 300 presents the sensing results to the second user and accepts the second user's input of teacher data settings. The first terminal device 200 operates in accordance with the positioning application AP1 corresponding to the teacher data created based on the input settings.

[0041] According to the method of the present embodiment, it is possible for the second user to view sensing results related to position adjustment provided by the first terminal device 200. Furthermore, the setting input of the second user is reflected in the creation of training data used by the positioning application AP1. Therefore, even in a situation where the first terminal device 200 and the second terminal device 300 are connected via a network NW, it is possible to digitize the tacit knowledge of an expert as training data and use the digitized training data in the positioning application AP1.

[0042] Furthermore, part or all of the processing performed by the information processing system 10 of this embodiment may be realized by a program. The processing performed by the information processing system 10 may be processing executed by the processing unit 210 of the first terminal device 200, processing executed by the processing unit 310 of the second terminal device 300, or processing executed by the processing unit 110 of the server system 100. Furthermore, the processing performed by the information processing system 10 may be processing executed by two or more devices among the server system 100, the first terminal device 200, and the second terminal device 300.

[0043] The program according to this embodiment can be stored in, for example, a non-transitory information storage medium (information storage device), which is a medium readable by a computer. The information storage medium can be realized by, for example, an optical disc, a memory card, a HDD, or a semiconductor memory. The semiconductor memory is, for example, a ROM. The processing unit 110 and the like perform various processes of this embodiment based on the program stored in the information storage medium. That is, the information storage medium stores a program for causing a computer to function as the processing unit 110 and the like. A computer is a device equipped with an input device, a processing unit, a storage unit, and an output unit. Specifically, the program according to this embodiment is a program for causing a computer to execute each step described below using Figures 6, 22, 23A, 23B, etc.

[0044] The technique of this embodiment can also be applied to an information processing method including the following steps. The information processing method is an information processing method in an information processing system 10 including: a first terminal device 200 that stores a positioning application AP1 that performs processing based on teacher data that indicates a correct position of at least one of a person and an object during assistance and that operates in accordance with the positioning application AP1; and a second terminal device 300 that is used by a second user who supports assistance of a first user using the first terminal device 200. The information processing method includes the steps of transmitting sensing results of at least one of the person and the object during assistance from the first terminal device 200 to the second terminal device 300 via a network NW; presenting the sensing results to the second user and accepting setting input of the teacher data by the second user in the second terminal device 300; and operating the first terminal device 200 in accordance with the positioning application AP1 that corresponds to the teacher data created by the setting input.

[0045] 2. Positioning application setup Next, a specific example of the positioning application AP1 will be described. The positioning application AP1 may operate in a setting mode in which teacher data used for position adjustment is set, and in a usage mode in which processing based on the set teacher data is performed. The positioning application AP1 may include multiple application software. For example, the positioning application AP1 may include a setting application that sets the teacher data, and a usage application that executes processing based on the teacher data. In this case, the setting mode is realized by the setting application, and the usage mode, which will be described later, is realized by the usage application. In addition, various modifications are possible with respect to the specific configuration of the positioning application AP1.

[0046] Below, we will first provide an overview of the processing in the setting mode. Then, we will provide detailed explanations of examples of setting modes corresponding to the bed position, wheelchair position, and meal position. The bed position refers to the position of a person or object in bed 610. The wheelchair position refers to the position of a person or object in wheelchair 630. The meal position refers to the position of a person or object when eating, and may be included in the wheelchair position or the bed position, or may be a different position.

[0047] 2.1 Processing flow in setting mode FIG. 6 is a diagram illustrating the flow of processing when setting the positioning application AP1 in the information processing system 10 of this embodiment. As described above with reference to FIG. 1, the information processing system 10 includes a server system 100, a first terminal device 200, a second terminal device 300, and a sensing device 400. Of these, the first terminal device 200 and the sensing device 400 are disposed, for example, in a home where home care is provided. For example, the sensing device 400 may be a care device purchased by a person being assisted or their family for use in home care. The second terminal device 300 is disposed in a location different from the home, such as the workplace of a caregiver or other professional. The server system 100 may be disposed in any location.

[0048] First, in step S101, the first terminal device 200 and the second terminal device 300 establish a connection. For example, a positioning application AP1 (setting application) is installed in the first terminal device 200 and the second terminal device 300, and communication between the devices is realized by the processing units 210 and 310 operating in accordance with the positioning application AP1. For example, as will be described later with reference to FIGS. 9A to 9C, the positioning application AP1 (setting application) here may have the function of a web conferencing application that transmits images and audio to each other. Note that this communication may be via the server system 100. For example, the first terminal device 200 and the second terminal device 300 may be connected via the server system 100 by establishing communication between the first terminal device 200 and the server system 100 and communication between the second terminal device 300 and the server system 100.

[0049] In step S102, the first terminal apparatus 200 acquires information about the installed sensing device 400 and transmits the information to the second terminal apparatus 300. For example, if the family member of the person being assisted has purchased a mattress 620 that detects pressure for assistance, a bedside sensor 420 (described later with reference to FIG. 11 ), a detection device 430, a seat sensor 440 (described later with reference to FIG. 15 ), or the like, information identifying these devices is transmitted to the second terminal apparatus 300. The processing of step S102 may be implemented by a first user of the first terminal apparatus 200 inputting the type, model number, etc. of the device using the operation unit 250. Alternatively, the processing of step S102 may be processing for transmitting information about the sensing device 400 stored in the storage unit 220. For example, when the sensing device 400 is initially installed, the first terminal apparatus 200 and the sensing device 400 may be connected to each other, causing the storage unit 220 of the first terminal apparatus 200 to store information about the sensing device 400. The initial installation can be performed by a knowledgeable person, such as a manufacturer's serviceman, so even if the first user does not have knowledge about the sensing device 400, the processing of step S102 can be performed appropriately.

[0050] In step S103, the second terminal apparatus 300 accepts an input by the second user to select a connection target device. As described above, the second user is a professional (also referred to as a Caregiver) who can provide care. The connection target device here is a sensing device 400 that is to be connected to the first terminal apparatus 200. For example, the second terminal apparatus 300 may display a list of sensing devices 400 transmitted from the first terminal apparatus 200, and accept an input by the second user to select a connection target device from the list.

[0051] As described above, various sensing devices 400 such as a bedside sensor 420, a detection device 430, and a seat sensor 440 may already be installed around the first terminal device 200. These devices are capable of sensing posture in bed 610 or wheelchair 630, and are therefore useful for setting teacher data and using the set teacher data.

[0052] However, the sensing device 400 useful for setting teacher data varies depending on the target location. For example, when setting teacher data for a bed position, the bedside sensor 420 and the detection device 430 are useful, and when setting teacher data for a wheelchair position, the seat sensor 440 is useful.

[0053] Furthermore, even when the same location (for example, bed 610) is the target, the sensing device 400 useful for setting the training data may differ depending on the manufacturer and model number of the bed 610 and mattress 620, the attributes of the target person being assisted, etc. In addition, which sensing device 400 is useful for setting the training data (and using the training data, which will be described later) may be implicitly known to an experienced caregiver.

[0054] In the method of this embodiment, it is possible to accept selection input of the device to be connected from the second user, so it is possible to appropriately utilize the tacit knowledge of experts not only in the selection of the teacher data itself but also in the selection of the sensing device 400 related to the teacher data.

[0055] In step S104, the second terminal apparatus 300 outputs connection information used to connect the first terminal apparatus 200 to the sensing device 400 that outputs the sensing results via the network NW to the first terminal apparatus 200. For example, the sensing device 400 may be provided with a QR code (registered trademark) or the like for connection, and the first terminal apparatus 200 may be able to establish a connection with the sensing device 400 by reading the QR code using the first terminal apparatus 200. In this case, the connection information transmitted by the second terminal apparatus 300 may include information identifying the device to be connected and control information instructing the launch of a QR code reading application. For example, upon receiving the connection information, the first terminal apparatus 200 executes a process of displaying text such as "Please read the QR code on the seat sensor 440" and a process of launching a QR code reading application. Note that the positioning application AP1 may have a QR code reading function.

[0056] However, the connection information is not limited to this. For example, if the IP address or the like of the sensing device 400 is known, the connection information may be information instructing connection to the device with that IP address. In this case, for example, the first terminal device 200 may display an authentication screen including text such as "May I connect to the seat sensor? Yes / No?", and when the first user performs an input operation to allow connection on the authentication screen (for example, an operation to select Yes), a connection may be established with the sensing device 400 having the specified IP address.

[0057] In addition, the connection information here may be any information used to establish a connection between the sensing device 400 specified by the second user and the first terminal apparatus 200, and various specific modifications are possible.

[0058] In step S105, the first terminal apparatus 200 establishes a connection with the sensing device 400 designated by the second user based on the connection information. In step S106, the sensing device 400 acquires sensing data and transmits the sensing data to the first terminal apparatus 200. Details of the sensing data will be described later.

[0059] Note that FIG. 6 illustrates an example in which the first terminal device 200 and the sensing device 400 are successfully connected based on the connection information. However, there are cases in which the connection between the first terminal device 200 and the sensing device 400 fails due to the occurrence of some kind of error. For example, the connection fails if the sensing device 400 has already been introduced (as included in the information transmitted in step S102) but is not powered on, or if some abnormality occurs in the communication function of the sensing device 400. In this case, the first terminal device 200 may transmit information to the second terminal device 300 indicating that the connection with the sensing device 400 has failed. The second terminal device 300 may then transmit advice to the first terminal device 200 to improve the connection status. For example, the second user may advise the second user to turn on the sensing device 400 or reboot the sensing device 400. In this way, the second user can receive support from the second user not only in selecting the sensing device 400 to use, but also in improving the connection status if the connection with the sensing device 400 fails. The second user is likely to be more familiar with how to use and set up the sensing device 400 than the first user, and the sensing device 400 can be used appropriately with the support of the second user.

[0060] Although the above has described connection errors during connection attempts, the connection status between the first terminal apparatus 200 and the sensing device 400 may be periodically monitored even after the connection is successful. In this way, even if an error occurs during communication, the error can be appropriately detected. For example, although not shown in FIG. 6 , after completing the process of step S105, the first terminal apparatus 200 periodically executes a process to check whether the connection with the sensing device 400 has been established. For example, the first terminal apparatus 200 may periodically transmit a confirmation packet to the sensing device 400 and determine whether an ACK has been received. Alternatively, the first terminal apparatus 200 may detect a connection error when no sensing data is transmitted from the sensing device 400 for a predetermined period of time or longer. Then, when a connection error is detected, the first terminal apparatus 200 may transmit information indicating that the connection with the sensing device 400 has failed to the second terminal apparatus 300, as described above.

[0061] In step S107, the first terminal device 200 acquires a captured image including the person being assisted using the imaging unit 260. For example, in a scene where the bed position is being adjusted, the imaging unit 260 may acquire a moving image capturing an image of a family member who is providing a care adjustment adjusting the posture of the person being assisted lying on the bed 610.

[0062] The information processing system 10 of this embodiment may also perform emotion analysis of the first user of the first terminal device 200 based on a captured image of the first user (also referred to as the first captured image). For example, in step S108 of FIG. 6, the first terminal device 200 performs a process of estimating the emotion of the person being assisted based on the captured image acquired in step S107. Note that, as a method for estimating a person's emotion based on an image, machine learning based on training data in which information identifying the emotion of the person is assigned as ground truth data to input data, such as an image of a person's face, may be used. For example, the first terminal device 200 performs emotion analysis by inputting the captured image acquired in step S107 into a trained model acquired by the machine learning. The model here may be, for example, a convolutional neural network (CNN). Various techniques for analyzing emotions based on image processing are known, such as the Face API disclosed at "https: / / azure.microsoft.com / ja-jp / services / cognitive-services / face / ," and these techniques can be widely applied in this embodiment. Furthermore, if a family member of the person being assisted is included in the captured image, emotion analysis of the family member of the person being assisted may be performed in step S108. Note that emotion estimation is not limited to being performed in the first terminal device 200. For example, the captured image may be transmitted to the server system 100, and emotion analysis may be performed in the processing unit 110 of the server system 100.

[0063] In step S109, the first terminal device 200 transmits information including the sensing data, the captured image, and the result of emotion analysis to the second terminal device 300. In step S110, the display unit 340 of the second terminal device 300 displays a screen including the transmitted information. Specific screens will be described later with reference to Figures 9A to 9C, 14, 16, 19, etc.

[0064] 6 is not limited to being performed once, but may be executed repeatedly. For example, the first terminal device 200 may transmit a first captured image of the first user of the first terminal device 200 to the second terminal device 300 during the setting period of the positioning application AP1 (the period during which the teacher data is set). In this way, it is possible to present to the second user of the second terminal device 300 the status of the person being assisted and the family of the person being assisted while the teacher data is being set remotely in an easy-to-understand manner.

[0065] In this case, the second terminal device 300 may display the first captured image transmitted from the first terminal device 200 in association with the emotion analysis results. For example, as described below with reference to FIG. 9C , an object EM expressing an emotion may be superimposed on a portion of the captured image. This allows the position and movement of the person being assisted, etc., to be associated with the emotion of the person being assisted, etc., and presented to the second user in an easy-to-understand manner. For example, the second user can easily understand how the person being assisted feels about their current posture, and how the family member providing the assistance feels about the assistance and the person's reaction. As a result, tacit knowledge that is unfavorable to the person being assisted or their family member is prevented from being set as training data, making it possible to set training data that is appropriate for the person being assisted.

[0066] 6 are not limited to being performed once each, but may be repeatedly performed. For example, the sensing data and the emotion analysis results may be acquired at predetermined intervals and transmitted from the first terminal device 200 to the second terminal device 300 at any timing. In step S110, the second terminal device 300 may display a screen in which the display areas for the sensing data, captured images, and emotion analysis results are updated at predetermined intervals. In other words, the display process of step S110 may be a process of displaying an image that is updated over time.

[0067] Next, in step S111, the second user inputs the teacher data settings while viewing the displayed screen, and the second terminal device 300 accepts the input settings. For example, the second user determines whether the current posture is appropriate for the person being assisted by viewing the captured image and sensing data transmitted from the first terminal device 200. If the current posture is determined to be inappropriate, the second user may instruct the person's family to change the posture or add a cushion, for example, using voice or video. If the current posture is determined to be appropriate, the second user may input settings to that effect. For example, when the algorithm creation button (object OB4) described later with reference to FIG. 9C is selected, the captured image displayed at that time is set as the teacher data. As described later with reference to FIG. 16, the second user may select a portion of the sensing data, and various modifications are possible regarding the details of the input settings.

[0068] As can be seen from the above description, in the setting mode, the voice of the second user may be transmitted to the first user. For example, the second terminal device 300 may transmit a second captured image of the second user of the second terminal device 300 to the first terminal device during the setting period in which the teacher data is remotely set. In this way, the second user can give instructions regarding posture using voice or movement, making it possible to clearly present to the person being assisted and their family the actions and auxiliary goods (cushions, etc.) required to maintain an appropriate posture.

[0069] The first terminal device 200 may also store a captured image (second captured image) of the second user of the second terminal device 300 as log data. For example, the second captured image may be used as part of training data, and the first terminal device 200 may store a positioning application AP1 corresponding to the training data. Alternatively, the second captured image may be stored as information separate from the positioning application AP1.

[0070] For example, the second captured image may be an image in which the second user, a care professional, demonstrates a motion to help the person being assisted adopt an appropriate posture. Because the second user is assumed to be located at a distance from the first user, it may be difficult for the second user to actually assist the person being assisted by the first user. Therefore, the person being assisted by the second user may be another person or a dummy such as a doll. It is also possible to demonstrate the motion in a virtual space such as the Metaverse. This allows the first user to review the movements of the skilled caregiver, enabling even non-professional family members of the person being assisted to make appropriate position adjustments. For example, the first terminal device 200 displays the second captured image stored as log data during the usage period of the positioning application AP1. This allows the skilled caregiver's movements to be confirmed when actually adjusting the position.

[0071] The log data here may also include an image (first captured image) of the first user of the first terminal device 200. For example, if the second captured image is an image showing a sample, the first captured image is an image of the first user acting in accordance with the sample. In this way, the first user can check their own past (setting) actions when adjusting their position, making it easier to perform appropriate actions. For example, if the positioning application AP1 (setting application) has a web conferencing function, information (video) including the voice and image of the first user and the voice and image of the second user may be recorded in chronological order, and the recorded data may be saved as log data. In this way, the first user can check the type of interaction they had with the second user, making it easier to perform more appropriate actions.

[0072] When the setting input of the training data is performed, in step S112, the second terminal device 300 transmits information based on the setting input to the server system 100. When the training data is a captured image or sensing data, the information transmitted in step S112 may be the training data itself. Alternatively, the information transmitted in step S112 may be information for generating training data. For example, the setting input of the second user may be data representing a correct answer when performing machine learning such as classification or regression, and training data may be generated by performing machine learning based on the data. Note that when processing such as machine learning is performed, the processing may be executed by the second terminal device 300 or the server system 100. Details will be described later using Figures 16 and 17.

[0073] In step S113, the server system 100 creates an algorithm corresponding to the training data created based on the setting input of the second user. The algorithm here may be update data for adding training data to the existing positioning application AP1. Alternatively, the algorithm may be a new positioning application AP1.

[0074] In step S114, the server system 100 transmits the created algorithm to the first terminal device 200. This makes it possible for the first terminal device 200 to use the positioning application AP1 corresponding to the teacher data created by the setting input of the second user.

[0075] 2.2 Bed position FIG. 7 is a diagram showing an example of device arrangement around a bed 610. As shown in FIG. 7, the first terminal device 200 in this embodiment may be a terminal device 200-1 fixed to the footboard side of the bed 610, or a terminal device 200-2 fixed to a side rail of the bed 610, or both. However, the location where the first terminal device 200 is fixed is not limited thereto, and the first terminal device 200 may be fixed to another position on the bed 610 or to another location (for example, a wall or furniture in a room) from which an image of the person being assisted on the bed 610 can be captured. Alternatively, the first terminal device 200 may be carried by a caregiver, and the process described below may be performed by using the first terminal device 200 while the caregiver is positioned on the footboard side or the side rail side of the bed 610.

[0076] A display DP may be disposed on the opposite side of the bed 610 from the terminal device 200-2. The display DP may be fixed to the bed 610, or may be disposed in another position where it can be viewed naturally by a caregiver who adjusts the bed position using the positioning application AP1. For example, the display DP may be fixed to a wall surface, or may be fixed to a stand or the like that stands on the floor. The display DP may also be omitted.

[0077] 3, the first terminal device 200 is a device such as a smartphone having an imaging unit 260 (camera). The first terminal device 200 transmits the captured image to the server system 100. The display DP receives the image transmitted from the server system 100 directly or via another device such as the first terminal device 200, and displays the received image.

[0078] The positioning application AP1 may be an application that supports adjusting the bed position. The adjustment of the bed position may be, for example, controlling the posture of the person being assisted to prevent bedsores or controlling the placement of a cushion. The adjustment of the bed position may also be controlling the posture of the person being assisted during diaper changing or controlling the placement of a diaper.

[0079] 8 is a diagram illustrating the relationship between devices used in the setting mode. For example, the positioning application AP1 may acquire image information, which is an image captured by the imaging unit 260 included in the first terminal device 200, and information from at least one of the bed 610 and the mattress 620. The bed 610 and the mattress 620 here are devices capable of visualizing the pressure distribution when the bed 610 and the mattress 620 are being used by a person being assisted. For example, known methods disclosed in JP 2019-039788 A and the like can be widely applied as a method for visualizing the pressure distribution.

[0080] For example, in step S109 of FIG. 6, the first terminal device 200 transmits to the second terminal device 300 information including a first captured image of the first user, the result of emotion analysis based on the first captured image, and pressure distribution, which is sensing data acquired by the bed 610 and the mattress 620.

[0081] 9A to 9C are examples of screens displayed on the display unit 240 of the first terminal device 200 and the display unit 340 of the second terminal device 300 during the teacher data setting process. The positioning application AP1 (setting application) according to this embodiment may have a web conference function. The screen shown in FIG. 9A includes various buttons, such as a participant button, a chat button, a camera button, and a microphone button. When the participant button is selected, the participants of the web conference are displayed. When the chat button is selected, an area for chatting (text communication) is displayed. When the camera button is selected, the camera of the device (the first terminal device 200 or the second terminal device 300) displaying the screen of FIG. 9A is switched on and off. When the microphone button is selected, the microphone of the device displaying the screen of FIG. 9A is switched on and off. The screen shown in FIG. 9A also includes a display area for each participant. In the example of FIG. 9A, the participants are user A and user B, and the area corresponding to user A is relatively large. For example, when one of the first user and the second user sets up a Web conference and each user participates in the conference, the process of step S101 in FIG. 6 is executed and the screen shown in FIG. 9A is displayed.

[0082] FIG. 9B is a diagram that is displayed when a selection operation is performed on the link button displayed at the top of the screen shown in FIG. 9A, and corresponds to the selection process of the sensing device 400 to be connected shown in step S103 of FIG. 6.

[0083] When the operation to select the link button is performed, an object OB2 for selecting the sensing device 400 to be connected may be displayed in a part of the screen (the left side of the screen in the example of FIG. 9B). As described above with reference to FIG. 6, the process of step S103 is performed by the second user, and therefore the link button and object OB2 may be displayed on the second terminal device 300 but may not be displayed on the first terminal device 200. However, because the first user may select the sensing device 400 to be connected, the link button and object OB2 may be displayed on the first terminal device 200. An example of display on the second terminal device 300 will be described below.

[0084] As shown in step S102 of FIG. 6, the second terminal device 300 may acquire information about sensing devices 400 already installed around the first terminal device 200, and may display, as object OB2, an object for selecting connection / disconnection for each of the sensing devices 400. In the example of FIG. 9B, a mattress 630, a bed 610, a sleep scan (a detection device 430 described later using FIG. 11), a bedside sensor 420, a seat sensor 440 (described later using FIG. 15), and a choking choke app (a choking choke detection device 460 described later using FIG. 18) are connectable to the first terminal device 200, and a corresponding check box is displayed as object OB2. The second terminal device 300 may also display text prompting the user to select a device to connect, such as "Please select a device to connect." The second user selects the sensing device 400 to connect by checking a check box using object OB2. The first terminal apparatus 200 establishes a connection with the sensing device 400 based on the result of the selection operation on the object OB2 (corresponding to steps S104-S105 in FIG. 6).

[0085] Furthermore, as shown in FIG. 9B, when a selection operation of the cooperation button is performed, an object OB1 corresponding to a check box for switching skeleton tracking on / off may be displayed.

[0086] Based on the user's operation on the object OB1, the positioning application AP1 switches whether or not to perform skeletal tracking of the person being assisted. For example, when the check box is checked, the positioning application AP1 performs skeletal tracking based on captured images, and when the check box is unchecked, the skeletal tracking is omitted. Note that various image-based skeletal tracking methods are known, such as "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields" (https: / / arxiv.org / pdf / 1611.08050.pdf) and OpenPose disclosed by Zhe Cao et al., and these methods can be widely applied in this embodiment.

[0087] FIG. 9C is an example of a screen displayed on the display unit 340 of the second terminal device 300 in step S110 of FIG. 6. For example, a first terminal device 200 capturing an image of the person being assisted, a first terminal device 200 capturing an image of the family member of the person being assisted, and a second terminal device 300 capturing an image of the second user may participate in a web conference. In the example of FIG. 9C, a screen corresponding to the person being assisted (e.g., user A) is displayed relatively large, and screens corresponding to the family member (e.g., user C) and the second user (e.g., user B) are displayed relatively small. The screen corresponding to the person being assisted is a real-time captured image of the person being assisted. Note that it may be possible to switch which user's screen is displayed larger. Furthermore, the image of the person being assisted is not limited to being captured in real time, and the second user may provide advice based on a recorded video or the like. Furthermore, the person being assisted and the family member may participate in a web conference from a single first terminal device 200. Furthermore, the screen shown in FIG. 9C may be displayed on the display unit of another device connected via a network. In addition, the specific aspects of the display screen can be modified in various ways.

[0088] Among the screens shown in Fig. 9C, for example, the screen corresponding to the person being assisted includes an area RE1 for displaying image information and an area RE2 for displaying pressure distribution. In the example of Fig. 9C, an image of the person being assisted lying on the bed 610 captured from the foot side of the person being assisted is displayed in area RE1. The image is captured, for example, using the imaging unit 260 of the terminal device 200-1 of Fig. 7. For example, in the setting mode, moving images captured using the imaging unit 260 of the terminal device 200-1 may be displayed in real time in area RE1. Naturally, an image captured by the terminal device 200-2 may be displayed in area RE1, or an image captured by the first terminal device 200 located in another position may be displayed in area RE1.

[0089] As described above, the information processing system 10 of this embodiment may perform emotion analysis of the first user. In the example of FIG. 9C, an object EM representing the result of the emotion analysis of the user is displayed in part of the image information corresponding to the captured image. The object EM may be, for example, an illustration representing an emotion. In the example of FIG. 9C, an object EM is displayed that is an illustration indicating that the person being assisted is feeling positive emotions. However, the result of the emotion analysis may also be displayed using text, etc., and various modifications of the specific display mode are possible.

[0090] The pressure distribution is information output by the bed 610 and mattress 620, and is, for example, information that expresses the pressure value at each position on the surface on which the person being assisted lies using shades of color, etc. For example, in the setting mode, the pressure distribution obtained in chronological order may be displayed in real time.

[0091] The screen shown in FIG. 9C may also include an object OB3 corresponding to the rotation button, an object OB4 corresponding to the algorithm creation button, and pointers PO1 and PO2 corresponding to the first and second users, respectively.

[0092] Object OB3 is an operation interface for rotating at least one of the image information displayed in area RE1 and the pressure distribution displayed in area RE2. In the example of FIG. 9C, the left side of the screen is toward the head of the person being assisted in both the image information and the pressure distribution, making it easy for a user viewing the screen of FIG. 9C to associate the image information with the pressure distribution. However, in an environment in which the positioning application AP1 is used, the direction in which the first terminal device 200 captures an image of the person being assisted is not necessarily fixed. Therefore, the orientation of the person being assisted may differ between the image information and the pressure distribution. In this regard, when the positioning application AP1 receives a selection operation on the rotation button, it performs a process of rotating the image information or the pressure distribution, thereby making it possible to align the orientation of the person being assisted.

[0093] However, the orientation of the image information and the orientation of the pressure distribution are not limited to being manually changed by the second user, and may be automatically adjusted by the positioning application AP1. For example, a marker such as a two-dimensional barcode (QR code) may be attached in advance to a predetermined position within the range to be imaged by the imaging unit 260. For example, a sticker or the like with a marker printed on it may be affixed to the head of the bed. The positioning application AP1 performs processing to detect the position and orientation of the marker from an image captured using the imaging unit 260 of the first terminal device 200. In this way, the direction of the person's head in the image information can be appropriately detected, making it possible to automatically adjust the orientation of the image information and the orientation of the pressure distribution.

[0094] Furthermore, object OB3 corresponding to the rotation button may be used to rotate both the image information and the pressure distribution by the same angle. For example, regions RE1 and RE2 may be arranged vertically, but by rotating the image information and pressure distribution by 90 degrees clockwise or counterclockwise, regions RE1 and RE2 may be arranged horizontally. In this way, various pieces of information can be arranged in a positional relationship desired by the user using the rotation button. For example, the second user may arrange regions RE1 and RE2 in a positional relationship that is easy for the second user to see, or in a positional relationship that is suitable for explaining to the first user.

[0095] 9C based on the operations of the first and second users. For example, if the first terminal device 200 and the second terminal device 300 are PCs, the pointers PO1 and PO2 may be objects that move based on the operations of devices such as a mouse or a trackball. If the first terminal device 200 and the second terminal device 300 are smartphones, the pointers PO1 and PO2 may be objects that move based on the touch operations on a touch panel.

[0096] For example, if the second user wants to explain to the first user about a specific part of the screen displayed in FIG. 9C , the second user may use the pointer PO2 to point to the relevant part. For example, when explaining that it is recommended to position a certain part of the skeleton tracking results in this way, the first user can easily understand by pointing to the relevant part or target position with the pointer PO2. The pointer PO2 is displayed on both the first terminal device 200 and the second terminal device 300. Similarly, when the first user asks a question to the second user, the first user can easily help the second user understand the question by pointing to a part related to the question with the pointer PO1. The pointer PO1 is displayed on both the first terminal device 200 and the second terminal device 300. Note that while FIG. 9C illustrates both the pointer PO1 for the first user and the pointer PO2 for the second user, either one or both may be omitted.

[0097] 9A to 9C, the second user can view the captured image of the person being assisted, the pressure distribution, and the emotion analysis results, even if the second user is located far away from the first user, and can also view the results of the skeleton tracking process, as necessary. The results of the skeleton tracking process may be displayed superimposed on the image information in area RE1, as shown in FIG. 9C, for example.

[0098] The second user here is an experienced caregiver, and the second user determines whether the person being assisted is in an appropriate posture by referring to various information displayed on the screen of FIG. 9C. The appropriate posture here may be a posture suitable for preventing bedsores, a posture suitable for changing a diaper, or some other posture. For example, the experienced caregiver determines whether the posture is appropriate for the patient based on his or her own tacit knowledge, taking into account the patient's attributes, etc.

[0099] For example, if the second user determines that the person being assisted has assumed an appropriate posture, the second user selects the algorithm creation button indicated by object OB4. The first terminal device 200 transmits the image displayed when the algorithm creation button was operated to the server system 100 as training data. This allows the position considered preferable by the expert to be registered as training data. The training data is not limited to image information and may include pressure distribution or skeletal tracking results. While the above description uses the terminal device 200-1, this is not limiting. The terminal device 200-2 may also be used to acquire the image information. That is, the image information may be an image captured from the side of the person being assisted. As can be seen from the above description, the algorithm creation button is intended to be operated by the second user, who is an expert, and is not intended to be operated by the first user. Therefore, the second terminal device 300 may display object OB4, while the first terminal device 200 may hide object OB4. For example, the positioning application AP1 may determine whether the logged-in user is a first user or a second user, and switch the objects to be displayed according to the user attributes.

[0100] Further, some additional information may be added to training data including image information. FIG. 10 shows an example of a screen used when adding additional information to image information. For example, when a user inputs to select image information, the image information is displayed on the screen of FIG. 10 together with its file name. For example, the image information obtained when the algorithm creation button is selected is stored in the storage unit 220 of the first terminal device 200 or the storage unit 120 of the server system 100, and the positioning application AP1 accepts a user input to select one of the image information. In the example of FIG. 10, the selected image information is displayed in the area RE3, and an object OB5 representing the file name of the image information is also displayed.

[0101] As shown in FIG. 10 , the additional information input screen may also display an object OB6, which is a button for adding a specific mark to the image information. For example, the specific mark may be a mark added to a cushion used to promote proper posture. Here, a triangle corresponds to a cushion of a predetermined size or smaller, and a square corresponds to a cushion of a larger size. For example, the above correspondence is presented to the second user in advance using a manual for the positioning application AP1, etc., and when the second user places a cushion to promote proper posture, the second user uses the object OB6 to add a triangular or square mark to the cushion in the image. Note that, since this embodiment assumes remote setting, the actual cushion placement may be performed by the first user (e.g., a family member of the person being assisted). In this way, information identifying the position and size of the cushion can be included in the training data.

[0102] The additional information may also be a comment added by a skilled caregiver. For example, the additional information input screen may display an object OB7 that is a text box for inputting a comment. The comment here may be some kind of text input using the object OB7 that is a text box. For example, the skilled caregiver inputs in text important points for achieving an appropriate bed position, such as the angle between a specific body part and other body parts, the positional relationship with pillows and cushions, and details of the cushions used (size, shape, manufacturer, product number, etc.).

[0103] The input screen for additional information shown in FIG. 10 may also accept input of information specifying a part of the person being assisted that the experienced caregiver considers to be a key point. For example, if the position of the shoulders of the person being assisted is important for ensuring that the person is in an appropriate posture, the experienced caregiver may enter an input specifying the shoulders of the person being assisted. For example, the positioning application AP1 may accept input to place a triangular or square mark shown in FIG. 10 on a specific part. The experienced caregiver may also consider multiple parts to be key points. In this case, the positioning application AP1 may accept input of the priority of each position. The additional information may also be audio information. For example, the input screen for additional information may display an object OB8 that is a checkbox for setting whether or not audio information can be input. For example, if the checkbox is checked, a process of recording the voice of the second user may be performed.

[0104] Furthermore, the screen shown in Fig. 10 may allow a preview that assumes actual use. For example, an object OB9, which is a button for previewing, may be displayed on the input screen for additional information. The preview here refers to displaying a screen that corresponds to the display screen in the usage mode. The preview display may be performed on the input screen for additional information shown in Fig. 10. Alternatively, a pop-up screen different from the input screen in Fig. 10 may be displayed based on a selection operation of the object OB9, and the preview display may be performed on the pop-up screen.

[0105] For example, as described above with reference to FIGS. 4A and 4B, when using the positioning application AP1, it is conceivable to superimpose transparently processed teacher data and a real-time image of the person being assisted. Therefore, when a selection operation is performed on object OB9 in FIG. 10, the positioning application AP1 acquires a captured image using the imaging unit 260 and superimposes an image of the teacher data being set with transparency processing on the captured image. This allows the second user to check in advance what the display screen will look like when the teacher data being set is actually used. In this case, the second user may be able to set the transparency of the teacher data using the screen in FIG. 10, or, if the imaging unit 260 of the first terminal device 200 includes an internal camera and an external camera, may be able to select which camera to use. Furthermore, since it is the first user who actually uses the positioning application AP1, the transparency may be set by the first user. For example, the additional information input screen may display object OB10, which is a text box for inputting the transparency, and object OB11, which is a radio button for selecting a camera, as shown in FIG. 10.

[0106] The training data used in the positioning application AP1 is not limited to this. For example, the training data may include sensing data corresponding to the timing at which the algorithm creation button is selected. The sensing data serving as training data may be data at a single timing at which the algorithm creation button is selected, or may be time-series data for a predetermined period including the timing at which the selection operation is performed. As described above, video images capturing the actions of the second user until the second user places the person being assisted in an appropriate posture may be used as training data. The training data may also include the results of skeletal tracking of the second user detected based on the video images.

[0107] Through processing using the above screens, the server system 100 acquires training data. In step S113 of FIG. 6, the processing unit 110 of the server system 100 generates an algorithm based on the training data. Specifically, the algorithm here is a positioning application AP1 that performs processing based on the training data. For example, the processing unit 110 may create an algorithm that superimposes image information, which is the training data, on an image captured by the imaging unit 260 of the first terminal device 200 using a set transparency. Alternatively, the processing unit 110 may create an algorithm that compares the training data with data acquired during actual position adjustment, and determines that the posture is appropriate (OK) if the similarity is greater than or equal to a threshold, and determines that the posture is inappropriate (NG) if the similarity is less than the threshold. As described above, the algorithm here may be a new application or update data for an existing application.

[0108] Although the above examples have been given of a bed 610 and a mattress 620 capable of detecting pressure as examples of sensing device 400 used to set up and use positioning application AP1, the present invention is not limited to this. For example, the output of sensing device 400 described below may be displayed in area RE2 in FIG. 9C.

[0109] 11 is a diagram illustrating another example of a sensing device 400 that can be used in the positioning application AP1, which is an example of a bedside sensor 420 and a detection device 430 that are placed at the bottom of a bed 610. The bedside sensor 420 and the detection device 430 are, for example, sheet-like or plate-like devices that are provided between the bottom of the bed 610 and the mattress 620, as shown in FIG.

[0110] The bedside sensor 420 includes a pressure sensor that outputs a pressure value as sensing data, and is arranged on the side of the bottom that the caregiver uses to get in and out of the bed 610. In the example of FIG. 11, the caregiver gets in and out of the bed 610 using the front side of the bed. In this case, as shown in FIG. 11, a railing to prevent falls may be arranged on the front side of the bed 610, and the bedside sensor 420 may be arranged in a position where the railing is not provided. In this way, a user getting in and out of the bed 610 first sits on the bedside sensor 420. The bedside sensor 420 may output time-series pressure data as sensing data. Alternatively, the bedside sensor 420 may determine whether or not the user has started to move by executing the process described below, and output the determination result as sensing data.

[0111] The bedside sensor 420 operates, for example, according to an application installed on the bedside sensor 420, to acquire pressure values ​​as input data and execute a process of determining the movement of the person being assisted on the bed 610 from the pressure values.

[0112] For example, when a person being assisted stands up from bed 610, it is assumed that the person being assisted transitions from a lying position on the bed to a sitting position at the side of the bed (hereinafter referred to as "edge-sitting position"), and then performs a standing-up motion by placing their hands on their knees or the bedside to apply force. The pressure values ​​detected by the bedside sensor 420 increase in the order of lying position, edge-sitting position, and standing-up motion. For example, the bedside sensor 420 may determine that the start of movement has been detected when it detects a change from an edge-sitting position to a standing-up motion based on a comparison process between the pressure value and a given threshold value. Alternatively, from the perspective of detecting a standing-up motion at an earlier stage, the bedside sensor 420 may determine that the start of movement has been detected when it detects a change from a lying position to an edge-sitting position based on a comparison process between the pressure value and a given threshold value.

[0113] Alternatively, as the standing-up motion continues, the buttocks of the person being assisted rise from the bottom surface, causing a significant decrease in the pressure value output from the pressure sensor. Therefore, the bedside sensor 420 may determine that the standing-up motion has been performed when the pressure value increases above a first threshold and then decreases below a second threshold that is smaller than the first threshold, based on the time-series change in the pressure value. In addition, various modifications can be made to the specific processing content of the movement start determination.

[0114] 11 is a device that senses information related to the sleep of the person being assisted. The detection device 430 includes a pressure sensor that outputs a pressure value.

[0115] When the user gets into bed, the detection device 430 detects the user's body vibrations (body movements, vibrations) through the mattress 620. Based on the body vibrations detected by the detection device 430, information on the breathing rate, heart rate, activity level, posture, wakefulness / asleep, and whether the user is out of bed or in bed can be obtained. The detection device 430 may also determine whether the user is in non-REM sleep or REM sleep, and the depth of sleep. For example, the periodicity of body movements may be analyzed, and the breathing rate and heart rate may be calculated from the peak frequency. The periodicity may be analyzed using, for example, a Fourier transform. The breathing rate is the number of breaths per unit time. The heart rate is the number of heartbeats per unit time. The unit time is, for example, one minute. Body vibrations may also be detected per sampling unit time, and the number of detected body vibrations may be calculated as the amount of activity. When the user gets out of bed, the detected pressure value decreases compared to when the user is in bed, so it is possible to determine whether the user is out of bed or in bed based on the pressure value and its time-series changes.

[0116] For example, the detection device 430 may output the output of a pressure sensor as the sensing data, or may output the above-mentioned information on the respiratory rate, heart rate, activity level, posture, awake / asleep, and getting out / being in bed as the sensing data.

[0117] The detection device 430 may also determine whether the person being assisted has started to move. For example, the detection device 430 may determine that the person being assisted has started to move when the person being assisted transitions from a state of being in bed to a state of getting out of bed. From the viewpoint of detecting signs of movement at an earlier stage, the detection device 430 may also determine that the person being assisted has started to move when the person being assisted transitions from a sleeping state to a waking state. The detection device 430 may output the detection result of the start of movement as sensing data.

[0118] 2.3 Wheelchair position Fig. 12 is a diagram illustrating a system configuration when adjusting the wheelchair position. As shown in Fig. 12, in adjusting the wheelchair position, the first terminal device 200 is fixed at a height that allows it to capture an image of at least the upper body of the person being assisted sitting in the wheelchair 630. The first terminal device 200 is placed in a predetermined position in a home where home care is provided, for example, and the family member who is providing the care transfers the person being assisted into the wheelchair 630, moves the person to the front of the first terminal device 200, and then adjusts the wheelchair position. However, the family member may also perform the processing described below by standing in front of the wheelchair 630 and holding the first terminal device 200 in their hand.

[0119] As described above with reference to FIG. 3, the first terminal device 200 is, for example, a device such as a smartphone having a camera as the imaging unit 260. The positioning application AP1 may be an application that supports adjustment of the wheelchair position. The adjustment of the wheelchair position may include controlling the posture of the person being assisted to reduce the risk of bedsores or falls, or controlling the placement of cushions.

[0120] 13 is a diagram illustrating the relationship between apps and devices used in wheelchair positioning. For example, the positioning application AP1 may acquire image information output from the imaging unit 260 included in the first terminal device 200, and information from at least one of the wheelchair 630 and the seat sensor 440. The wheelchair 630 here is a device that can visualize the pressure distribution when the wheelchair 630 is being used by a person being assisted. The seat sensor 440 will be described later with reference to FIG. 15.

[0121] Fig. 14 is an example of a screen displayed on the display unit 340 of the second terminal device 300 in the setting mode. The screen shown in Fig. 14 may also be displayed on the display unit 240 of the first terminal device 200 for confirmation by the first user. For example, the screen shown in Fig. 14 may be displayed in an area of ​​the web conference screen corresponding to the person being assisted, similar to the example of Fig. 9C.

[0122] The configuration of the screen shown in Fig. 14 is the same as Fig. 9C except that the image information displayed in area RE1 is an image of the person being assisted sitting in the wheelchair 630, and the pressure distribution displayed in area RE2 is information representing the pressure on the seat of the wheelchair 630. Therefore, further details of the information displayed on the screen in Fig. 14 will not be described.

[0123] By using the screen shown in Fig. 14, the second user can refer to the captured image of the person being assisted, the pressure distribution, and the results of emotion analysis, and can also view the results of the skeletal tracking process if necessary. Therefore, for example, the second user who is an experienced caregiver can determine whether the person being assisted at a remote location is in an appropriate posture by referring to the various information displayed on the screen of Fig. 14. The appropriate posture here may be a posture suitable for preventing bedsores, or it may be another posture.

[0124] Similar to the example of bed position, the position considered preferable by the expert can be registered as training data based on the selection operation of the algorithm creation button shown in object OB4. The training data is not limited to image information, but may also include pressure distribution and the results of skeletal tracking.

[0125] Furthermore, as in the example of bed position, some additional information may be added to the training data including image information. To add the additional information, for example, the screen described above with reference to FIG. 10 is used.

[0126] In the above, the wheelchair 630 capable of detecting pressure has been exemplified as a device used to set up and use the positioning application AP1, but the device is not limited to this.

[0127] 15 is another example of a sensing device 400 that can be used by the positioning application AP1, and is a diagram showing a seat sensor 440 that is placed on the seat of, for example, a wheelchair 630. The seat sensor 440 includes a pressure sensor that outputs a pressure value, and outputs the pressure value.

[0128] The seat sensor 440 may also be capable of processing other than outputting pressure values ​​to the positioning application AP1. For example, the seat sensor 440 may determine, based on the pressure value of the pressure sensor, whether the posture of the person being assisted when sitting in the wheelchair 630 (hereinafter also referred to as the sitting posture) is one of a plurality of postures including normal, forward shift, side shift, etc. Forward shift refers to a state in which the user's center of gravity is shifted further forward than normal, and side shift refers to a state in which the user's center of gravity is shifted to either the left or right than normal. Both forward shift and side shift correspond to states in which the risk of falling from the seat is relatively high. The seat sensor 440 may also perform a fall possibility determination to determine whether or not the person being assisted is likely to fall from the seat.

[0129] 15, four pressure sensors Se1 to Se4 are arranged on the rear side of a cushion 441 that is placed on the seat of a wheelchair 630. Pressure sensor Se1 is a sensor that is arranged in the front, pressure sensor Se2 is a sensor that is arranged in the rear, pressure sensor Se3 is a sensor that is arranged on the right, and pressure sensor Se4 is a sensor that is arranged on the left. Note that front, back, left, and right here refer to directions as seen from the perspective of the person being assisted when the person is sitting in the wheelchair 630.

[0130] 15, the pressure sensors Se1 to Se4 are connected to a control box 442. The control box 442 includes a processor that controls the pressure sensors Se1 to Se4 and a memory that serves as a work area for the processor. The processor detects pressure values ​​by operating the pressure sensors Se1 to Se4.

[0131] A person being assisted sitting in a wheelchair 630 may feel pain in the buttocks and shift the position of the buttocks. For example, forward slippage occurs when the buttocks are shifted further forward than usual, and lateral slippage occurs when the buttocks are shifted to the left or right. Forward slippage and lateral slippage may occur simultaneously, causing the center of gravity to shift diagonally. By using a pressure sensor disposed on the cushion 441 as shown in FIG. 15, changes in the position of the buttocks can be detected appropriately, making it possible to accurately detect forward slippage or lateral slippage. For example, the seat sensor 440 determines that forward slippage has occurred when the value of pressure sensor Se1 has increased by a predetermined amount compared to the initial state, and determines that lateral slippage has occurred when the value of pressure sensor Se3 or Se4 has increased by a predetermined amount compared to the initial state.

[0132] The seat sensor 440 may output the pressure values ​​output from the pressure sensors Se1 to Se4 as sensing data, or may output the results of the determination of forward or lateral slippage, the determination of the possibility of falling, etc., as sensing data. The control box 442 may also include a light-emitting unit or the like, which may be used to notify the caregiver. In this way, the caregiver can be notified of changes in the sitting posture of the wheelchair 630 or the like in an easy-to-understand manner, making it possible to prevent the person being assisted from falling.

[0133] Fig. 16 is another example of a screen displayed on the display unit 340 of the second terminal device 300 in the setting mode. The screen shown in Fig. 16 may be displayed on the display unit 240 of the first terminal device 200 for confirmation by the first user.

[0134] 14, the screen shown in Fig. 16 differs in that it includes areas RE4 and RE5 that display the output of seat sensor 440, instead of area RE2 that displays the pressure distribution. For example, area RE4 displays information plotting the time-series change in the center of gravity position of the person being assisted, acquired using pressure sensors Se1 to Se4, on a two-dimensional plane with the front-to-back direction of the seat of wheelchair 630 as a first axis (e.g., horizontal axis) and the left-to-right direction as a second axis intersecting the first axis (e.g., vertical axis perpendicular to the horizontal axis). Area RE5 displays information showing the time-series change in the pressure values ​​of pressure sensors Se1 to Se4, with the horizontal axis representing time and the vertical axis representing pressure values.

[0135] 16, similarly to the example of Fig. 14, the second user may determine whether the first user is in an appropriate posture based on image information or sensing data, and select the algorithm creation button at the timing when it is determined that the posture is appropriate. However, the setting input of the second user is not limited to this.

[0136] For example, the second user may perform an input to select a range of sensing data corresponding to an appropriate posture on the screen of FIG. 16. In the example of FIG. 16, for example, the second terminal device 300 accepts input of a figure enclosing a portion of the two-dimensional plane in area RE4. FIG. 16 illustrates a figure S as an input result by the second user. Here, figure S may be drawn freehand, or a predetermined shape such as a circle or rectangle may be used. For example, if the second terminal device 300 is a smartphone with a touch panel, the second user may input figure S, which selects a portion of the two-dimensional plane, by drawing a line in part of area RE4 with a finger or a stylus. Similarly, the second terminal device 300 may accept an input to select a range in area RE5 that the second user considers to be an appropriate posture. Furthermore, the second terminal device 300 may accept an input to select a range in area RE2 that displays sensing data, which the second user considers to be an appropriate posture, on the screen described above with reference to FIG. 9C or FIG. 14.

[0137] In step S112 of FIG. 6, the second terminal device 300 transmits information identifying the range selected by the second user to the server system 100. The server system 100 performs classification with the information as the correct answer, and sets a boundary for classifying the sensing data into a class corresponding to OK and a class corresponding to NG. The processing in the server system 100 may be regression analysis. For example, in the example of FIG. 16, the second user draws a figure S to distinguish between the inside of the figure S (correct answer) and the outside of the figure S (deviation). Therefore, in this case, the boundary that is set is a boundary that divides a two-dimensional plane into two. However, the input of the second user and the number of categories are not limited to these.

[0138] FIG. 17 is an example of a screen that presents information generated by the server system 100 to the second user, and is displayed on the display unit 340 of the second terminal device 300, for example. The screen in FIG. 17 includes an area RE6 that displays the processing results of the server system 100 and an object OB12 that represents a registration button. Here, an example is shown in which boundaries are set to divide a two-dimensional plane into four regions, with the front-to-back direction of the seat of the wheelchair 630 as the first axis and the left-to-right direction as the second axis. For example, the second user may draw a figure in area RE4 or the like that divides the two-dimensional plane into four regions: a region representing normal, a region representing forward deviation, a region representing left deviation, and a region representing right deviation. In this case, the results of the four classifications are obtained as shown in FIG. 17. For example, one of the four regions represents an appropriate posture (OK), and the other three regions correspond to inappropriate posture (NG), representing a forward deviation, a left deviation, and a right deviation, respectively. If the second user determines that there is no problem with the processing results based on the screen in FIG. 17, the registration button is selected. When the registration button is selected, the server system 100 creates an algorithm that uses information representing the set boundaries as training data. For example, when sensing data is acquired from the seat sensor 440, the processing unit 110 creates an algorithm that identifies the center of gravity position and determines to which of four areas classified by boundaries the center of gravity position belongs. The processing unit 110 may also create an algorithm that determines OK / NG based on the determination result of the area to which the center of gravity position belongs.

[0139] As described above, by accepting the second user's operation on the sensing data as a setting input for the training data, it becomes possible to appropriately digitize and utilize the second user's tacit knowledge.

[0140] 2.4 Eating Position Next, adjustment of the eating position will be described. The following will exemplify a case where the user eats while sitting in a wheelchair 630. Therefore, the eating position may be set for a wheelchair position using, for example, a wheelchair 630 or seat sensor 440 that is capable of detecting pressure. However, since it is also conceivable that the user may eat while lying in bed 610, the eating position may be set for a bed position using, for example, a bed 610, mattress 620, bedside sensor 420, detection device 430, etc. that are capable of detecting pressure.

[0141] Fig. 18 is a diagram illustrating a system configuration when adjusting the wheelchair position. As shown in Fig. 18, when eating, first terminal device 200 and choking detection device 460, which is sensing device 400 that detects various information related to eating, such as swallowing and choking by the person being assisted, may be used. As shown in Fig. 18, choking detection device 460 includes a throat microphone 461 worn around the neck of the person being assisted, and terminal device 462 with a camera.

[0142] The throat microphone 461 outputs audio data resulting from swallowing, coughing, etc. of the person being assisted. The camera of the terminal device 462 outputs captured images of the person being assisted eating. The terminal device 462 is, for example, a smartphone or tablet PC placed on a table where the person being assisted eats. The throat microphone 461 is connected to the terminal device 462 using Bluetooth (registered trademark) or the like, and the terminal device 462 is connected to the server system 100 via a network. However, both the throat microphone 461 and the terminal device 462 may be directly connectable to the server system 100, and various modifications can be made to the specific connection mode.

[0143] 18, in adjusting the eating position, the first terminal device 200 may be placed on the side of the person being assisted. The imaging section 260 of the first terminal device 200 outputs a first captured image obtained by capturing an image of the side of the person being assisted.

[0144] Fig. 19 is an example of a screen displayed on the display unit 340 of the second terminal device 300 in the setting mode. The configuration of the screen shown in Fig. 19 is the same as that of Fig. 16, except that instead of area RE1 displaying an image of the person being assisted seated in the wheelchair 630 captured from the front, area RE7 is used to display an image of the person being assisted captured from the side. Therefore, further details of the information displayed on the screen of Fig. 19 will not be described. Note that area RE7 in Fig. 19 does not include a table, food, etc.

[0145] By using the screen shown in FIG. 19, the second user can refer to the captured image of the person being assisted, the emotion analysis results, and the sensing data, and can also view the results of the skeletal tracking process as needed. Therefore, for example, the second user, who is an experienced caregiver, can determine whether the person being assisted at a remote location is in an appropriate posture by referring to the various information displayed on the screen of FIG. 19. The appropriate posture here may be a posture suitable for eating, or may be another posture. An appropriate posture for eating is, for example, a posture that prevents choking or aspiration.

[0146] Similar to the examples of bed position and wheelchair position, a position that an expert considers preferable can be registered as training data based on the selection operation of the algorithm creation button shown in object OB4. Furthermore, similar to the examples of bed position and wheelchair position, some additional information may be added to the training data including image information. To add the additional information, for example, the screen described above with reference to FIG. 10 is used.

[0147] Furthermore, the training data for the eating position is not limited to the above examples. For example, to prevent choking or aspiration while eating, it is important to consider whether the spine is as straight as possible, whether the chin is tucked in, whether the person is leaning forward too much, etc. Therefore, for example, the second user may input a setting indicating whether the head position of the person being assisted is appropriate.

[0148] 20A and 20B are examples of training data acquired by setting input by the second user. For example, the training data in the eating position may be a boundary dividing an image of the person being assisted captured from the side. Here, the boundary is, for example, a straight line. In the example of FIG. 20A, two straight lines are set to divide the image into four regions. In an example using this training data, if the head of the person being assisted is located in the upper right region, it is determined to be OK, and if it is located in any other region, it is determined to be NG. However, the number of straight lines is not limited to two, and may be one, as shown in FIG. 20B. In the example of FIG. 20B, one straight line is set to divide the image into two regions. In an example using this training data, if the head of the person being assisted is located in the right region, it is determined to be OK, and if it is located in the left region, it is determined to be NG. Furthermore, three or more straight lines may be set, and curved lines may be used as the boundary dividing the image.

[0149] Furthermore, when using the training data of FIG. 20A , skeletal tracking may be used in combination. The positioning application AP1 may determine whether the training data is OK or NG depending on the region in which a certain part (e.g., neck, nose, head, etc.) of the training data is located. Alternatively, the positioning application AP1 may compare a partial image belonging to a certain region (narrowly speaking, the correct region) of the training data with a partial image belonging to the same region of the image captured during position adjustment, and determine the result as NG if the difference is large. The image difference determination may use, for example, a norm calculated from RGB pixel values, as described above. Furthermore, when using the training data of FIG. 20A , the OK / NG determination is not required. For example, the positioning application AP1 may superimpose data indicating a boundary included in the training data (two straight lines in the case of FIG. 20A ) on the captured image. In this case, the first user viewing the superimposed image can appropriately adjust their posture based on the positional relationship between a specific part and the boundary. As described above, various modifications are possible to the processing using the training data of FIG. 20A . The same applies when using the training data in FIG. 20B.

[0150] For example, the second user may set the straight line shown in FIGS. 20A and 20B by drawing a line with his / her finger or a touch pen in area RE7 of the screen in FIG. 19. Alternatively, the second user may determine whether each of a plurality of captured images is OK / NG and input the determination result as a setting input to the second terminal device 300. The server system 100 may perform processing to automatically set the straight line shown in FIG. 20A, etc., by performing classification or regression analysis that considers the setting input to be the correct answer. Alternatively, the second user may perform a setting input to select a range of the sensing data that the second user considers preferable, as in the example of FIG. 16, and processing based on the setting input may be performed to obtain the straight line shown in FIG. 20A, etc.

[0151] 21 shows another example of a screen displayed on the display unit 340 of the second terminal device 300 in the setting mode. The first terminal device 200 may include a device that captures an image of the person being assisted from the front and a device that captures an image of the person being assisted from the side. For example, the terminal device 462 of the choking detection device 460 in FIG. 18 may operate as the first terminal device 200. Alternatively, either the device that captures an image of the front of the person being assisted or the device that captures an image of the side of the person being assisted may operate as the first terminal device 200, and the other device may perform processing to transmit the captured image to the first terminal device 200.

[0152] The screen shown in Fig. 21 may include an area RE1 that displays an image of the person being assisted captured from the front, and an area RE7 that displays an image of the person being assisted captured from the side. This increases the amount of information that can be presented to the second user, making it possible to improve the accuracy of the training data, etc. Note that other points are the same as those in Figs. 16 and 19, and therefore detailed description will be omitted.

[0153] Next, the details of the swallowing choking detection device 460 will be described. For example, the swallowing choking detection device 460 determines whether the person being assisted has choked or swallowed based on audio data from a throat microphone 461. A device that detects swallowing using a microphone worn around the neck is described, for example, in U.S. Patent Application No. 16 / 276,768, filed February 15, 2019, entitled "Swallowing Action Measurement Device and Swallowing Action Support System." This patent application is incorporated herein by reference in its entirety. The processor can detect the number of choking incidents, the duration of the choking (time of occurrence, duration, etc.), and whether or not the person swallowed based on the audio data. For example, the output of the swallowing choking detection device 460 may be output as sensing data and displayed on a screen such as FIG. 19 or FIG. 21. The swallowing choking detection device 460 can also obtain various information related to meals, as described below. In this embodiment, this information may be output as sensing data. For example, any of the information described below may be displayed in area RE4 or area RE5 of FIG. 19. For example, a waveform signal representing a sound detected by the throat microphone 461, an average value of the swallowing time described below, etc. may be displayed on the screen shown in FIG.

[0154] The camera of the terminal device 462 can detect the mouth, eyes, and chopsticks and spoons used by the person being assisted by capturing an image of the person being assisted from the front, as shown in Fig. 18. There are various known methods for detecting these facial features and objects based on image processing, and a wide range of known methods can be applied in this embodiment.

[0155] For example, the choking detection device 460 can determine, based on the image captured by the camera, whether the person being assisted has their mouth open, whether food is coming out of their mouth, and whether they are chewing food. The choking detection device 460 can also determine, based on the image captured by the camera, whether the person being assisted has their eyes open. The choking detection device 460 can also determine, based on the image captured by the camera, whether chopsticks, a spoon, etc. are near tableware, whether the person being assisted can hold them, and whether food is being spilled.

[0156] In the method of this embodiment, the swallowing and choking status of the person being assisted is estimated based on this information. For example, the choking detection device 460 may obtain information about meals based on the detection results of choking and swallowing and the determination result of whether the person being assisted is opening or closing their mouth.

[0157] For example, the swallowing choking detection device 460 may determine whether choking is occurring frequently based on the number of times or duration of choking, and output the determination result. For example, the swallowing choking detection device 460 may determine that choking is occurring frequently when the number of times choking per unit time exceeds a threshold. In this way, the situation regarding choking can be automatically determined.

[0158] The choking detection device 460 may also calculate the swallowing time from when the person being assisted opens their mouth to when they swallow, based on the swallowing detection result and the result of determining whether the person being assisted has opened or closed their mouth. This makes it possible to determine specific situations, such as whether the person is not putting food into their mouth, or whether the person has put food in their mouth but is not swallowing it, when the number of swallows has decreased. For example, the choking detection device 460 may start counting up a timer when the mouth transitions from a closed state to an open state based on the captured image of the terminal device 462, and stop counting the timer when swallowing is detected by the throat microphone 461. The time when the timer stops counting represents the swallowing time. This makes it possible to accurately determine whether the risk of aspiration is high during meals and whether the caregiver should take some kind of action.

[0159] Choking detection device 460 may also determine the pace of eating based on the swallowing time. Choking detection device 460 may also determine whether the swallowing time is long based on a change in the swallowing time during one meal (for example, the increase or ratio compared to the swallowing time at the beginning). Alternatively, the processor may calculate the average swallowing time for each of multiple meals for the same person receiving care, and determine whether the swallowing time has become long based on the change in the average swallowing time.

[0160] Furthermore, by using the results of determining whether the mouth is open or closed based on the captured image of the terminal device 462, it can be determined whether the mouth no longer opens even when the caregiver approaches a spoon or the like. In this way, if the swallowing time becomes longer when the person being assisted is reluctant to open their mouth, it can be estimated that food is remaining in the mouth and becoming stagnant. Furthermore, by using the captured image to recognize whether food is coming out of the mouth and whether the food is being chewed, it can be determined whether the person being assisted is no longer able to chew the food. For example, if the number of chews is normal but the swallowing time is long, it can be estimated that the person being assisted is no longer able to chew the food. Furthermore, if it is determined that the eyes are closed based on the captured image, it can be determined whether the person being assisted is becoming sleepy.

[0161] Furthermore, by performing a recognition process on chopsticks, spoons, etc. using the captured image, it may be determined whether the person is playing with food, unable to hold a dish, doing nothing, etc. For example, if an object such as a spoon is overlapping the hand of the person being assisted, but the time it takes for the person to bring the object to their mouth is equal to or greater than a predetermined threshold, it is determined that the person is unable to hold a dish or is playing with food. Furthermore, if an object such as a spoon is not overlapping the hand of the person being assisted, and the time the person's gaze is directed toward food (food) is equal to or greater than a predetermined threshold, it is determined that the person is looking at the food without doing anything.

[0162] It is also known that the feature amount of the voice data detected by the throat microphone 461 changes depending on various circumstances. The feature amount here may be the amplitude of the waveform of the voice data, the frequency, or the number of waves included in one swallow. For example, the swallowing sound changes depending on factors such as the hardness or thickness of the food, the pace of eating, etc. It has also been found that the swallowing sound changes depending on the physical condition of the person being assisted, even when the hardness, etc., are the same.

[0163] Therefore, the choking detection device 460 may obtain features representing the swallowing sound based on the output of the throat microphone 461, and estimate the physical condition of the person being assisted based on the features. For example, the choking detection device 460 stores data that associates the features of the swallowing sound with the physical condition of the person being assisted based on past history. The choking detection device 460 then estimates the physical condition of the person being assisted by comparing the data with the actually acquired features. Note that the data that associates the features of the swallowing sound with the physical condition of the person being assisted may be a trained model created by machine learning.

[0164] 3. Use of positioning applications As described above, in the setting mode, an algorithm (update data or a new positioning application AP1) based on the setting input of the second user is transmitted to the first terminal device 200. Therefore, the first terminal device 200 can operate the positioning application AP1 in the use mode. As described above, the positioning application AP1 in the use mode may perform a process of transparently superimposing and displaying image information, which is training data, or may determine whether the posture of the person being assisted is acceptable or unacceptable based on a comparison process of image information and sensing data. The positioning application AP1 may also determine whether the posture of the caregiver (family member of the person being assisted) is acceptable or unacceptable. When simply using the positioning application AP1 stored in the first terminal device 200, the second user does not need to intervene using the second terminal device 300 in the use mode.

[0165] However, even when the same person receiving care is being used, the appropriate training data may differ depending on the situation. Therefore, if the first terminal device 200 is capable of executing processing corresponding to multiple sets of training data, it is necessary to select the training data to be used. Furthermore, the information processing system 10 of this embodiment stores a large amount of training data (algorithms), and it is possible that training data set for a first person receiving care may be suitable for a second person receiving care who is different from the first person receiving care. For example, if the attributes of the first person receiving care and the second person receiving care are similar, training data suitable for one person may also be suitable for the other.

[0166] Therefore, the second terminal device 300 may accept a selection operation by the second user, which is an operation to select teacher data to be used from multiple teacher data. As described above, teacher data is the tacit knowledge of an experienced caregiver, and the selection of which tacit knowledge to use in what situation can also be tacit knowledge. According to the method of this embodiment, the judgment of the expert second user is also used in the selection of tacit knowledge, making it possible to achieve position adjustment suitable for the first user. For example, in the usage mode, the positioning application AP1 may display multiple tabs on a screen that displays transparently as shown in FIG. 4B and switch teacher data based on the selection operation of the tab. For example, one tab may correspond to one teacher data, and teacher data may be selected based on manual switching of the tab by the second user.

[0167] 22 is a sequence diagram illustrating a process when teacher data (tacit knowledge, algorithm) is selected using the second terminal device 300. The process in FIG. 22 is executed, for example, before the start of a specific position adjustment.

[0168] First, in step S201, the first terminal apparatus 200 and the second terminal apparatus 300 are connected. In step S202, the first terminal apparatus 200 transmits information about surrounding sensing devices 400 to the second terminal apparatus 300. The processes in steps S201 and S202 are similar to steps S101 and S102 in FIG. 6 .

[0169] In step S203, the second terminal device 300 accepts a selection input of a connection target device by the second user. This process is similar to step S103 in Fig. 6. Furthermore, in step S203, the second terminal device 300 accepts a selection input of an algorithm (teaching data) to be used.

[0170] For example, the second terminal device 300 may perform a process not shown in Fig. 22 to acquire a list of algorithms registered in the server system 100 and display the list. For example, the server system 100 may perform a process to transmit to the second terminal device 300 a list of algorithms that can be executed using sensing devices 400 that have been introduced in the vicinity of the first terminal device 200, from among the algorithms registered in the storage unit 120. The second user performs an operation to select an algorithm that is suitable for the person being assisted from the list according to his or her own tacit knowledge.

[0171] In step S204, the second terminal device 300 transmits connection information and information identifying the algorithm to the first terminal device 200. The connection information is the same as the example described above in step S104 of FIG. 6. The information identifying the algorithm may be, for example, an ID uniquely assigned to the algorithm. The information identifying the algorithm may also include information such as the name of the algorithm, attributes of the person being assisted when the algorithm was created, the ID of the second user who created the algorithm, assistance history, qualifications held, and affiliated facilities.

[0172] In step S205, the first terminal apparatus 200 establishes a connection with the sensing device 400 based on the connection information. The process of step S205 is similar to the process of step S105 in FIG.

[0173] In step S206, the first terminal device 200 performs a process of identifying an algorithm to be used based on the information received in step S204. In step S207, it is determined whether the identified algorithm has been installed in the first terminal device 200. For example, the first terminal device 200 may perform a process of comparing the positioning application AP1 stored in the storage unit 220 with the identified algorithm. This process may be a determination of whether the same positioning application AP1 has been installed. Furthermore, if the algorithm is implemented as update data (e.g., an add-in) for the positioning application AP1, the process of step S206 may be a determination of whether an add-in corresponding to the identified algorithm has been installed.

[0174] If it is determined in step S207 that the algorithm selected by the second user has already been installed, the processes from step S208 onwards are omitted, and the process according to that algorithm is executed.

[0175] If it is determined in step S207 that the algorithm selected by the second user has not been installed, in step S208, the first terminal device 200 requests the algorithm from the server system 100. In step S209, the server system 100 transmits the requested algorithm to the first terminal device 200. In step S210, the first terminal device 200 installs the transmitted algorithm. This enables the first terminal device 200 to execute processing in accordance with the algorithm.

[0176] In the above, an example has been described in which sensing data from the sensing device 400 is used even in the usage mode of the positioning application AP1. However, the positioning application AP1 may be, for example, an algorithm that superimposes image information representing training data by making it transparent, or that judges pass / fail by comparing the image information with the captured image, and the use of the sensing device 400 is not essential. In this case, the processes of steps S202, S204, and S205 related to the sensing device 400 can be omitted. Also, the process of accepting input from the target device in step S203 can be omitted.

[0177] 4. Changing operating modes based on load The above describes an example in which the first terminal device 200 establishes a connection with the sensing device 400. That is, sensing data from sensing devices 400 installed in a home or the like is collected in the first terminal device 200. Then, in the setting mode, the first terminal device 200 transmits data including the sensing data to the second terminal device 300, thereby remotely setting the teacher data. In addition, in the usage mode, the first terminal device 200 acquires sensing data from the sensing device 400 and performs processing in accordance with the sensing data (for example, comparison with teacher data, etc.). However, if the processing load on the first terminal device 200 or the communication load related to sending and receiving the sensing data is large, there is a possibility that processing on the first terminal device 200 will be delayed.

[0178] Therefore, the first terminal device 200 may compare the processing load of the positioning application AP1 and the communication load of the sensing result with the processing capacity of the first terminal device 200, and perform processing to switch the operation mode of the first terminal device 200 based on the comparison result. Note that the processing described below may be considered as processing to switch the operation mode of the sensing device 400 (processing to switch the destination of sensing data). Below, an example in the setting mode and an example in the usage mode will be described.

[0179] 23A is a sequence diagram illustrating switching of the operation mode in the setting mode. First, in step S301, the first terminal device 200 calculates the processing load of the positioning application AP1. The processing load of the positioning application AP1 may be set in advance based on the algorithm of the positioning application AP1, or may be determined based on the CPU usage and memory usage during execution.

[0180] In step S301, the first terminal device 200 acquires the communication load of the sensing data. If the algorithm is known, the type of sensing data to be communicated and the expected amount of data to be communicated per unit time are known. Therefore, the communication load may be set in advance based on the algorithm. Alternatively, the first terminal device 200 may determine the communication load based on an actual communication log.

[0181] Also, in step S301, the first terminal device 200 acquires the processing capabilities of the processing unit 210 and the communication unit 230. The processing capabilities may be set in advance based on the configuration of the first terminal device 200 or a communication device (such as a wireless LAN access point). Alternatively, the first terminal device 200 may dynamically determine the processing capabilities in consideration of the processing load of applications running other than the positioning application AP1.

[0182] If it is determined in step S301 that the processing load and communication load do not exceed the processing capacity of the first terminal device 200, the processes of steps S302 to S307 are executed.

[0183] In step S302, a connection is established between the first terminal apparatus 200 and the sensing device 400. For example, in step S302, the same processes as steps S102 to S105 in FIG. 6 are executed. The subsequent processes are also the same as steps S107 to S110 in FIG. 6. Specifically, in step S303, the sensing device 400 transmits sensing data to the first terminal apparatus 200. In step S304, the first terminal apparatus 200 acquires a captured image of the person being assisted. In step S305, the first terminal apparatus 200 performs emotion analysis of the first user. In step S306, the first terminal apparatus 200 transmits the first captured image, the sensing data, and the emotion analysis result to the second terminal apparatus 300. The second terminal apparatus 300 displays a screen such as FIG. 9C based on the received information. As described above, when there is a margin of processing capacity for the load, the same process as that of FIG. 6 may be executed.

[0184] On the other hand, if it is determined in step S301 that the processing load and communication load exceed the processing capacity of the first terminal device 200, the processes of steps S308 to S314 are executed.

[0185] First, in step S308, a connection between the sensing device 400 and the server system 100 is established. For example, if it is determined in step S301 that the processing load and communication load exceed the processing capacity, the first terminal device 200 may instruct the sensing device 400, directly or via the server system 100, to switch the destination of sensing data to the server system 100. For example, the sensing device 400 connects to the server system 100 by connecting to a wireless LAN access point installed in the home. This connection may be supported by a second user, as in the example of FIG. 6.

[0186] In step S309, the sensing device 400 acquires sensing data and transmits the sensing data to the server system 100. In step S310, the server system 100 transmits the sensing data to the second terminal device 300.

[0187] The processes of steps S311 and S312 are similar to those of steps S304 and S305. In step S306, the first terminal device 200 transmits the captured image and the emotion analysis result to the second terminal device 300.

[0188] Through the above processing, the second terminal device 300 can acquire sensing data from the server system 100, and acquire captured images and emotion analysis results from the first terminal device 200. Therefore, in step S314, the second terminal device 300 displays a screen such as that shown in FIG. 9C based on the received information.

[0189] When the processing of steps S308 to S314 is executed, unlike the processing of steps S302 to S307, the reception of sensing data from the sensing device 400 and the transmission of the sensing data to the second terminal apparatus 300 can be executed in the server system 100. Furthermore, when the sensing data is stored in the server system 100 as log data, the transmission of the log data from the first terminal apparatus 200 to the server system 100 can also be omitted. Therefore, the load on the first terminal apparatus 200 can be reduced.

[0190] Furthermore, in the method of this embodiment, sensing devices 400 that are not used for the tacit knowledge setting may operate. For example, in the method of this embodiment, if there is a margin of processing capacity for the load, a first mode may be used in which both the sensing device 400 used for the tacit knowledge setting and the sensing device 400 that is not used are connected to the first terminal device 200. Furthermore, if there is not a margin of processing capacity for the load, a second mode may be used in which the sensing device used for the tacit knowledge setting is connected to the first terminal device 200 and the other sensing devices 400 are connected to the server system 100. Furthermore, even if the sensing device 400 that is not used for the tacit knowledge setting is directly connected to the server system 100, if the load exceeds the processing capacity, a third mode may be used in which both the sensing device 400 used for the tacit knowledge setting and the sensing device 400 that is not used for the tacit knowledge setting are connected to the server system 100 without going through the first terminal device 200.

[0191] Furthermore, the mode switching is not limited to the two-stage switching shown in FIG. 23A or the three-stage switching using the first to third modes described above. For example, a priority may be set for each sensing device, and if the processing capacity is insufficient for the load, the sensing devices may be switched to direct connection to the server system 100, starting with the sensing device with the lowest priority. For example, a sensing device 400 not used in the tacit knowledge setting may be set to a lower priority than a sensing device 400 used in the tacit knowledge setting. Using the priority enables flexible switching between the sensing device 400 connected to the first terminal device 200 and the sensing device 400 directly connected to the server system 100. Note that this priority itself may be tacit knowledge, and the priority of each sensing device 400 may be changeable by the second user. As described above, various modifications are possible regarding which sensing device 400 is connected to which equipment.

[0192] 23B is a sequence diagram illustrating an example of a process for switching the operation mode in the usage mode. The process in step S401 is the same as step S301 in FIG.

[0193] If it is determined in step S401 that the processing load and communication load do not exceed the processing capacity of the first terminal device 200, the processes of steps S402 to S406 are executed.

[0194] In step S402, the sensing device 400 acquires sensing data and transmits the sensing data to the first terminal apparatus 200. In step S403, the first terminal apparatus 200 makes a determination based on the sensing data. For example, the first terminal apparatus 200 performs a process of comparing the teacher data with the sensing data, and makes a posture OK / NG determination based on the processing result. In addition, in step S404, the first terminal apparatus 200 acquires a first captured image of the first user using the imaging unit 260.

[0195] In step S405, the first terminal device 200 performs display processing and the like in accordance with the positioning application AP1. For example, the first terminal device 200 displays superimposed display of training data on the captured image, and displays the result of an OK / NG judgment based on sensing data and the like. Although not shown in FIG. 23B, the first terminal device 200 may also perform skeletal tracking and emotion analysis and display the judgment result based on these.

[0196] In step S406, the first terminal device 200 transmits the processing result to the second terminal device 300. This makes it possible to notify the second user of the result of the position adjustment. Although not shown in FIG. 23B, the first terminal device 200 may transmit the processing result to the server system 100. In this case, the result of the position adjustment can be accumulated as log data in the server system 100.

[0197] On the other hand, if it is determined in step S401 that the processing load and communication load exceed the processing capacity of the first terminal device 200, the processes of steps S407 to S412 are executed.

[0198] In step S407, the sensing device 400 acquires sensing data and transmits the sensing data to the server system 100. In step S408, the server system 100 makes a determination based on the sensing data. For example, the server system 100 performs an OK / NG determination based on the teacher data and the sensing data, etc. As described above, the memory unit 120 of the server system 100 stores the positioning application AP1 (algorithm), and therefore is able to execute the processing of step S408. In step S409, the server system 100 transmits the processing result to the first terminal device 200. The information transmitted in step S409 is, for example, flag data indicating either OK or NG.

[0199] In step S410, the first terminal device 200 acquires a first captured image of the first user using the imaging unit 260. In step S411, the first terminal device 200 performs display processing and the like in accordance with the positioning application AP1. For example, the first terminal device 200 displays superimposed display of teacher data on the captured image, and a result of an OK / NG judgment based on sensing data and the like. In step S412, the first terminal device 200 transmits the processing result to the second terminal device 300.

[0200] When the processes of steps S407 to S412 are executed, unlike the processes of steps S402 to S406, reception of sensing data and determination processing based on the sensing data can be executed in the server system 100. Therefore, the processing load and communication load on the first terminal device 200 can be reduced.

[0201] 23A and 23B, if the processing load of emotion analysis or skeleton tracking is large, the first terminal device 200 may transmit the captured image to the server system 100, and the server system 100 may perform processing such as emotion analysis. In this case, although the load related to communication of the captured image increases, it is possible to reduce the processing load related to emotion analysis, etc.

[0202] Note that Figures 23A and 23B are examples of switching the operating mode of the first terminal device 200, and the specific processing in each case where the processing load and communication load do not exceed the processing capacity and where they do exceed the processing capacity is not limited to the above example and can be implemented in various modified forms.

[0203] 5. Reliability Furthermore, the information processing system 10 of this embodiment may determine the trustworthiness of the second user from the first user based on the emotion analysis result of the first user using the first captured image. The process of determining the trustworthiness may be performed by any of the first terminal device 200, the second terminal device 300, and the server system 100, or may be realized by distributed processing of two or more devices. An example in which the first terminal device 200 performs emotion analysis will be described below.

[0204] For example, in emotion analysis, each emotion may be associated with information indicating whether the emotion is positive or negative. For example, emotions such as joy and happiness are positive, and anger and sadness are negative. If the first terminal device 200 determines through emotion analysis that the first user has a positive emotion, it determines that the first user has a high degree of trust in the second user. On the other hand, if the first terminal device 200 determines through emotion analysis that the first user has a negative emotion, it determines that the first user has a low degree of trust in the second user. Furthermore, each emotion may be associated with a reliability score indicating the level of reliability, and the reliability may be calculated based on the reliability score.

[0205] For example, suppose that in the setting mode, a second user using the second terminal device 300 is instructing a first user, such as a person being assisted and their family member, on posture. If the first user's emotions at this time are positive, the first user is likely to view the second user's instruction content, etc. favorably, and therefore has a high level of trust in the second user. On the other hand, if the first user's emotions in the setting mode are negative, the first user may not be happy with the instruction from the second user, and therefore has a low level of trust in the second user.

[0206] Furthermore, emotion analysis is not limited to being performed in the setting mode. For example, in the usage mode, in order to provide feedback, the first user and the second user may have a conversation or a video conference using the first terminal device 200 and the second terminal device 300. In this case, even if emotion analysis of the first user is performed, it is possible to estimate the first user's degree of trust in the second user.

[0207] Based on the result of the emotion analysis, the way in which the second user is involved in providing assistance to the first user may be changed. Examples in each of the setting mode and the use mode will be described below.

[0208] For example, when there are multiple second users, the degree of trust from the first user of the first terminal device 200 may be determined for each second user based on the result of the emotion analysis. For example, it is conceivable that multiple second users with different roles, such as caregivers, nurses, doctors, and care managers, are involved as experts in charge of one person being assisted. It is also conceivable that multiple caregivers are in charge of one person being assisted. In this case, the degree of trust from the first user is determined for each second user.

[0209] In addition, when a second user inputs settings to set teacher data, a second user with a higher reliability may be given priority over a second user with a lower reliability. For example, consider a situation in which a first user requests a second user to set teacher data for performing some position adjustment. In this case, the first terminal device 200 performs a process of preferentially selecting a user with a higher reliability from among multiple second users associated with the first user. For example, in step S101 of FIG. 6, the first terminal device 200 may attempt to connect to the second terminal device 300 of the second user with the highest reliability, starting with the second user with the highest reliability. In this way, the probability that a second user with a higher reliability will set teacher data increases, making it possible to digitize tacit knowledge that is more suitable for the first user.

[0210] In addition, in the usage mode, training data set by a second user with high reliability may be used preferentially over training data set by a second user with low reliability. As described above with reference to Figure 22, the method of this embodiment is assumed to have multiple second users, each of whom will register their own tacit knowledge as training data (positioning application, algorithm). Therefore, the selection of the tacit knowledge to be used is also important.

[0211] In this regard, by using reliability as a criterion for selecting tacit knowledge, it becomes possible to preferentially use tacit knowledge of a second user who is considered to be compatible with the first user. For example, the input of the selection of the algorithm to be used in step S203 in FIG. 22 may be omitted, and the second terminal device 300 or the server system 100 may automatically select the training data to be used based on reliability. Alternatively, the algorithm itself may be selected by the second user, but when displaying a list of candidate algorithms, algorithms created by second users who have a high level of reliability from the first user may be preferentially displayed. In addition, various modifications of the specific processing are possible.

[0212] In addition, there may be situations in which a second user with low reliability must be involved with the first user during caregiving. For example, even if a doctor's reliability is low, if a doctor's diagnosis is required for some kind of response, it is difficult for a caregiver or nurse to respond on behalf of the first user.

[0213] Therefore, in the method of this embodiment, if it is determined based on the results of emotion analysis that the reliability of the second user from the first user of the first terminal device 200 is below a predetermined threshold, at least one of the first terminal device 200 and the second terminal device 300 may present information encouraging the participation of a family member of the person being assisted or a second user with a relatively high reliability when communicating between the first terminal device 200 and the second terminal device 300.

[0214] For example, if the person being assisted has low trust in the doctor, the information processing system 10 may facilitate communication by suggesting the intervention of a caregiver, nurse, care manager, or the like, rather than having a conversation or video conference between the person being assisted and the doctor alone. Alternatively, the information processing system 10 may facilitate communication between the person being assisted and the doctor by suggesting the intervention of a family member of the person being assisted. For example, the suggestion to have a caregiver or the like intervene may be made by the second terminal device 300, and the suggestion to have a family member of the person being assisted intervene may be made by the first terminal device 200. However, various modifications of the presentation method are possible, such as proposing the intervention of a caregiver or the like by the first terminal device 200.

[0215] Furthermore, if a second user with low reliability exists, a suggestion to change the second user to another user may be made. For example, the second user is assumed to be a user with some kind of qualification, such as a physical therapist, occupational therapist, or speech-language-hearing therapist. Therefore, if a second user with low reliability from the first user exists, the information processing system 10 may perform a process of suggesting to the first user another user with the same qualification as the second user as a new second user. For example, the server system 100 may maintain a database of experts who are candidates for the second user for each qualification and propose a new second user based on the database. In this way, even if the second user is changed, the qualifications held by the second user can be maintained. In assistance situations, the expert support that should be received varies depending on the situation, such as when the knowledge of a physical therapist, occupational therapist, or speech-language-hearing therapist is effective. However, the above method makes it possible to receive appropriate support after the change.

[0216] Furthermore, if there is a second user with high reliability and the reliability of other second users is low, a proposal may be made to change the second user with low reliability to a person related to the second user with high reliability. For example, if a doctor is highly reliable but the care manager is low, the care manager may be changed to another care manager related to the doctor. Since related parties are likely to have similar care policies to a certain extent, changing the second user to a related party of a highly reliable person makes it possible to maintain a good relationship between the second user and the first user. Note that when changing a second user with low reliability, a determination based on qualifications may be made as described above. For example, if the care manager is to be changed, a new second user is selected from among people qualified as care managers. The same applies when a user with low reliability is a physical therapist, occupational therapist, speech-language-hearing therapist, etc.

[0217] 6. Remote configuration of tacit knowledge in applications other than positioning The above describes a method for remotely setting tacit knowledge used in the positioning application AP1. For example, if the training data in the positioning application AP1 is the tacit knowledge of a second user, the above-described method makes it possible to appropriately digitize the tacit knowledge even when the second user is located far away from the first user.

[0218] However, the target for remote configuration of tacit knowledge is not limited to the positioning application AP1, and may be other devices or applications. For example, the method of this embodiment may be applied to an information processing system 10 including an electronic device that acquires sensing results regarding at least one of a person and an object during assistance, and a second terminal device 300 used by a second user who supports the assistance of a first user using the electronic device. The electronic device transmits the sensing results to the second terminal device 300 via a network. The second terminal device 300 presents the sensing results to the second user and accepts setting input for the electronic device from the second user. The setting input here is information representing the second user's tacit knowledge regarding the operation of the electronic device. The electronic device then acquires the setting input and operates in accordance with the setting input.

[0219] The electronic device here may be, for example, any of the sensing devices 400 described above. For example, the seat sensor 440 may distinguish between a plurality of states, including forward displacement, right displacement, left displacement, and normal, as described above. The determination process by the seat sensor 440 may be executed as part of the operation of the positioning application AP1 as described above, or may be executed independently of the positioning application AP1. For example, the seat sensor 440 may determine whether the seat sensor 440 is forward displaced or not based on a pressure value, and notify the determination result using an indicator (e.g., a light-emitting element such as an LED) provided in the control box 442 or the like. Note that the determination based on the pressure value may be performed by a processor included in the control box 442, or may be executed by the processing unit 110 of the server system 100. In this way, the state of the person being assisted using the wheelchair 630 can be presented to an assistant or the like located near the wheelchair 630 in an easy-to-understand manner, without the intervention of an application such as the positioning application AP1.

[0220] However, the criteria for determining forward slippage and the like performed by the seat sensor 440 are implicitly known by an experienced caregiver. Therefore, the seat sensor 440 may generate the criteria by receiving a setting input from a second user. For example, the seat sensor 440 may be connected to a first terminal device 200 used by a first user who is the user of the seat sensor 440. The first terminal device 200 and the second terminal device 300 may then display a screen corresponding to the area RE4 in FIG. 16 or the area RE6 in FIG. 17. In this case, the first terminal device 200 and the second terminal device 300 may communicate using a web conference function, as described above with reference to FIG. 9. In this way, even if a second user, an expert, is located in a different location from where the seat sensor 440 is installed, the settings for the seat sensor 440 can be performed using the second user's implicit knowledge.

[0221] The electronic device here may be, for example, the choking detection device 460. As described above, the choking detection device 460 can perform various determinations regarding the eating of the person being assisted, and these determinations can be performed independently of the positioning application AP1.

[0222] For example, which of these determinations is useful varies depending on the condition of the person being assisted and the type of intervention the caregiver can perform, and therefore becomes the tacit knowledge of the expert. Therefore, the choking stool detection device 460 may transmit the output (audio data or waveform signal) of the throat microphone 461, images captured by the camera of the terminal device 462, etc. to the second terminal device 300. The second user may perform setting input to determine whether each determination is on or off based on the transmitted information. In this way, the choking stool detection device 460 can appropriately use the second user's tacit knowledge regarding which determination should be performed.

[0223] Furthermore, the tacit knowledge used in the choking detection device 460 is not limited to this. For example, the choking detection device 460 may determine whether intervention by a caregiver is necessary based on the swallowing time, the frequency of choking, and the results of image recognition by a camera. In this case, the criteria for the degree of abnormality detected that makes intervention necessary may be the tacit knowledge of an expert. Therefore, the choking detection device 460 may receive setting input regarding the intervention criteria from the second user, and the choking detection device 460 may determine the intervention criteria based on the setting input.

[0224] Furthermore, in the first terminal device 200 of this embodiment, an application related to assistance other than the positioning application AP1 may be running. For example, the first terminal device 200 may run a meal application for supporting meal assistance, an agitation application for determining whether or not agitated behavior due to dementia is occurring, or another assistance application. In addition, in the determination of these applications, setting input based on the tacit knowledge of the second user may be performed.

[0225] For example, the technique of this embodiment may be applied to an information processing system 10 including a first terminal device 200 that stores an application for performing processing in assistance and operates in accordance with the application, and a second terminal device 300 used by a second user who supports assistance for a first user using the first terminal device 200. The first terminal device 200 transmits sensing results regarding at least one of a person and an object involved in assistance to the second terminal device 300 via a network. The sensing results here may be the output of a sensor included in the first terminal device 200 or the output of a sensing device 400. The second terminal device 300 presents the sensing results to the second user and accepts application setting inputs from the second user. The setting inputs here are parameters that determine the operation of the application, and are, for example, information representing the second user's appropriate judgment results (e.g., appropriate assistance behavior in meal assistance for a meal application, or the second user's judgment results regarding agitated behavior for an agitated application). The first terminal device 200 operates in accordance with the application corresponding to the setting input. In this way, even when an application other than the positioning application AP1 is targeted, the second user can remotely configure the application, thereby making it possible to appropriately utilize the tacit knowledge of the second user.

[0226] Although the present embodiment has been described in detail above, those skilled in the art will readily understand that many modifications are possible without substantially departing from the novel features and advantages of the present embodiment. Therefore, all such modifications are intended to be within the scope of the present disclosure. For example, a term described at least once in the specification or drawings with a different term having a broader or equivalent meaning may be replaced with that different term anywhere in the specification or drawings. Furthermore, all combinations of the present embodiment and modifications are also within the scope of the present disclosure. Furthermore, the configurations and operations of the information processing system, server system, first terminal device, second terminal device, and sensing device are not limited to those described in the present embodiment, and various modifications are possible. [Explanation of symbols]

[0227] 10...information processing system, 100...server system, 110...processing unit, 120...storage unit, 130...communication unit, 200...first terminal device, 200-1, 200-2...terminal device (first terminal device), 210...processing unit, 220...storage unit, 230...communication unit, 240...display unit, 250...operation unit, 260...imaging unit, 300...second terminal device, 310...processing unit, 320...storage unit, 330...communication unit, 340...display unit, 350...operation unit, 360...imaging unit, 400...sensing device, 420...vehicle Head side sensor, 430...detection device, 440...seat sensor, 441...cushion, 442...control box, 460...choking detection device, 461...throat microphone, 462...terminal device, 610...bed, 620...mattress, 630...wheelchair, AP1...positioning application, DP...display, NW...network, OB1 to OB12, EM...object, RE1 to RE7...area, Se1 to Se4...pressure sensor, S...shape, PO1 to PO2...pointer

Claims

1. a first terminal device that stores a positioning application that performs processing based on training data that indicates a correct position of at least one of a person and an object during assistance, and operates in accordance with the positioning application; a second terminal device used by a second user to support the assistance of the first user using the first terminal device; Including, The first terminal device transmitting a sensing result relating to at least one of the person and the object during the assistance to the second terminal device via a network; The second terminal device presenting the sensing result to a second user and accepting a setting input of the teacher data by the second user; The first terminal device An information processing system that operates according to the positioning application corresponding to the teaching data created by the setting input.

2. In claim 1, The second terminal device An information processing system that outputs connection information used for connecting a sensing device that outputs the sensing result to the first terminal device via the network.

3. In claim 1, The first terminal device transmitting a first captured image of the first user of the first terminal device to the second terminal device during a setting period of the positioning application; The second terminal device An information processing system that transmits a second captured image of the second user of the second terminal device to the first terminal device during the set period.

4. In claim 3, The first terminal device An information processing system that stores the second captured image as log data and displays the second captured image during a period in which the positioning application is used.

5. In claim 3 or 4, A feeling analysis of the first user of the first terminal device is performed based on the first captured image; The second terminal device An information processing system that displays the first captured image transmitted from the first terminal device in association with a result of the emotion analysis.

6. In claim 5, When there are a plurality of second users, a degree of trust from the first user is determined for each of the second users based on a result of the emotion analysis; An information processing system in which, in the setting input of the teaching data, the second user with a high reliability is given priority over the second user with a low reliability.

7. In claim 5, When there are a plurality of second users, a degree of trust from the first user is determined for each of the second users based on a result of the emotion analysis; An information processing system in which the positioning application corresponding to the teacher data set by the second user with a high level of reliability is used preferentially over the positioning application corresponding to the teacher data set by the second user with a low level of reliability.

8. In claim 5, When it is determined based on the result of the emotion analysis that the reliability of the second user from the person being assisted who is the first user is equal to or less than a predetermined threshold, At least one of the first terminal device and the second terminal device An information processing system that presents information encouraging the participation of a family member of the person being assisted or a second user with a relatively high level of trust when the first terminal device and the second terminal device communicate with each other.

9. In any one of claims 1 to 4, The first terminal device Execute a process corresponding to the plurality of training data; The second terminal device An information processing system that accepts a selection operation by the second user to select the teacher data to be used from multiple teacher data.

10. In any one of claims 1 to 4, The first terminal device An information processing system that compares the processing load of the positioning application and the communication load of the sensing result with the processing capacity of the first terminal device, and switches the operation mode of the first terminal device based on the comparison result.

11. An information processing method in an information processing system including: a first terminal device that stores a positioning application that performs processing based on teacher data that represents a correct answer for the position of at least one of a person and an object during assistance, and that operates in accordance with the positioning application; and a second terminal device that is used by a second user that supports assistance of a first user using the first terminal device, transmitting a sensing result relating to at least one of the person and the object during the assistance from the first terminal device to the second terminal device via a network; In the second terminal device, the sensing result is presented to a second user, and a setting input of the teacher data is received from the second user; operating the first terminal device in accordance with the positioning application corresponding to the teacher data created by the setting input; Information processing methods.

Images