Thought-and-reason system, inference model generation system, thought-and-reason device, inference model generation method, and computer program

Abstract

[Problem] To alleviate, more than heretofore, the burden on a caregiver when an ICT device is used by a person who has difficulty speaking, such as a severely mentally / physically handicapped child or a dementia patient. [Solution] In the present invention, a caregiving assistance server 2 generates an inference model by: acquiring a plurality of data sets indicating a first condition at the time of a first bodily reaction of a person who has difficulty speaking, and indicating a first thought of the person who has difficulty speaking at that time; and performing machine learning by using the first condition as an explanatory variable and the first thought as a target variable, which are indicated in each of the plurality of data sets. By inputting, to the inference model, input data indicating a second condition at the time of a second reaction, a second thought at the time of the second reaction is inferred, and a caregiver terminal device outputs the inference results.

Description

[Technical field]

[0001] The present invention relates to a technique for inferring the thoughts, such as desires, of a person with speech impairment. [Background technology]

[0002] Electrical appliances such as air conditioners, televisions, lighting fixtures, and microwave ovens that can be controlled by the user's voice are becoming widespread. Electrical appliances that could only respond to a limited number of commands were already widespread at the end of the 20th century, but in recent years, thanks to the technologies of AI (Artificial Intelligence), IoT (Internet of Things), and ICT (Information and Communications Technology), electrical appliances that can be controlled by voice in a more human or interactive manner by the user have begun to spread. Electrical appliances that can be controlled by voice using these technologies are called "smart home appliances," "IoT home appliances," or "AI home appliances."

[0003] Smart speakers also use AI, IoT, and ICT technologies to search for and present information or control electrical appliances based on the user's voice. Mobile devices such as smartphones, tablet computers, and smartwatches also have such capabilities.

[0004] Hereinafter, such smart home appliances, smart speakers, and mobile devices will be referred to as "ICT devices."

[0005] As mentioned above, users can use ICT devices in a more human or interactive manner. Therefore, ICT devices are easier to use for physically disabled people than conventional devices, improving their quality of life (QOL).

[0006] However, it remains difficult for ICT devices to directly improve the QOL of people with speech difficulties, such as children and adults with severe physical and mental disabilities or dementia patients, because the voices of people with speech difficulties are still difficult to recognize with the voice recognition functions used in ICT devices. Therefore, ICT devices have traditionally been used through the intervention of family members or caregivers such as medical professionals. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent No. 6315744 Summary of the Invention [Problem to be solved by the invention]

[0008] The conversation assistance terminal described in Patent Document 1 has an input device and a display device, and when an image displayed on the display device is selected, a message corresponding to the selected image is output by voice. With this conversation assistance terminal, a user can select a display image to have the message spoken on his / her behalf. This allows the voice to be heard by the ICT device.

[0009] However, conventional devices such as this conversation assistance terminal require the user to select the image to be displayed, which can be difficult for severely physically and mentally disabled children and adults or dementia patients to use without the assistance of a caregiver, which places a burden on the caregiver.

[0010] In view of these problems, the present invention aims to reduce the burden on caregivers when severely physically and mentally disabled children or adults or dementia patients use ICT devices. [Means for solving the problem]

[0011] A thinking inference system according to one embodiment of the present invention includes a dataset acquisition means for acquiring a plurality of datasets indicating a first condition when a first physical reaction occurs of a person with speech difficulty and a first thought that the person with speech difficulty has when the first reaction occurs, an inference model generation means for generating an inference model by performing machine learning using the first condition indicated in each of the plurality of datasets as an explanatory variable and the first thought as a target variable, an inference means for inferring a second thought when a second reaction occurs by inputting input data indicating a second condition when the second reaction occurs into the inference model, and an output means for outputting the second thought. Effect of the Invention

[0012] According to the present invention, the burden on caregivers when a severely mentally and physically disabled child or dementia patient uses an ICT device can be reduced more than ever before. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating an example of an overall configuration of a thought inference system. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a care support server. [Diagram 3] 2 is a diagram illustrating an example of a functional configuration of a care support server. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a caregiver terminal device. [Diagram 5] FIG. 2 is a diagram illustrating an example of a functional configuration of a caregiver terminal device. [Figure 6] FIG. 13 is a diagram showing an example of a thought input screen. [Figure 7] FIG. 13 is a diagram showing an example of a selection list. [Figure 8] FIG. 13 is a diagram illustrating an example of raw data. [Figure 9] FIG. 1 is a diagram illustrating an example of a data set. [Figure 10] FIG. 13 is a diagram illustrating an example of a classification table. [Figure 11]A diagram illustrating an example of the process flow for testing an inference model. [Figure 12] FIG. 13 is a diagram illustrating an example of raw data. [Figure 13] FIG. 13 is a diagram illustrating an example of the flow of an inference process. [Figure 14] FIG. 13 is a diagram showing an example of an inference result screen. [Figure 15] 11 is a flowchart showing an example of the overall flow of processing by an inference service program. [Figure 16] FIG. 13 is a diagram showing a modified example of the functional configuration of the care support server. [Figure 17] FIG. 13 is a diagram showing a modified example of the functional configuration of the caregiver terminal device. [Figure 18] FIG. 13 illustrates an example of an event table. [Figure 19] FIG. 13 is a diagram showing a modified example of the option list. [Figure 20] FIG. 13 is a diagram illustrating an example of an explanatory variable table. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] [Overall structure] Fig. 1 is a diagram showing an example of the overall configuration of a thinking and inference system 1. Fig. 2 is a diagram showing an example of the hardware configuration of a care support server 2. Fig. 3 is a diagram showing an example of the functional configuration of the care support server 2. Fig. 4 is a diagram showing an example of the hardware configuration of a caregiver terminal device 3. Fig. 5 is a diagram showing an example of the functional configuration of the caregiver terminal device 3.

[0015] The thinking and inference system 1 shown in FIG. 1 is a system that infers the thoughts or ideas of a care recipient 81 by AI (Artificial Intelligence) and teaches them to a caregiver 82. For example, the thinking and inference system 1 infers and teaches various thoughts or ideas, such as desires such as "I want to go to school" or "I want to watch TV," answers such as "yes," "no," "this way," "that way," or "I don't know" to questions for the care recipient 81, feelings such as "I'm looking forward to it," "I'm angry," "I'm surprised," or "I'm sad," calls such as "Did you see that, teacher?", "Hurry up!", or "Hello," preferences such as "like" or "dislike," physical conditions such as "sleepy," "hungry," "tired," or "headache," sensations such as "hot," "cold," "smelly," "delicious," "dazzling," or "sour." Hereinafter, such thoughts or ideas are referred to as "thoughts." The caregiver 82 is a family member or teacher of the care recipient 81, a veteran supporter, a person with nursing qualifications, or a medical professional.

[0016] The thinking and inference system 1 is composed of a care support server 2, a plurality of caregiver terminal devices 3, a plurality of video cameras 41, a plurality of indoor measuring devices 42, a plurality of smart speakers 43, a plurality of biometric measuring devices 44, and a communication line 5.

[0017] The care support server 2 and each caregiver terminal device 3 can exchange data via a communication line 5. As the communication line 5, the Internet or a public line, etc., is used.

[0018] The care support server 2 generates an inference model by machine learning and infers the thoughts of the care recipient 81 based on the inference model. A personal computer, a workstation, a cloud server, or the like is used as the care support server 2. The following describes an example in which a personal computer is used as the care support server 2.

[0019] As shown in FIG. 2, the care support server 2 includes a processor 20, a RAM (Random Access Memory) 21, a ROM (Read Only Memory) 22, an auxiliary storage device 23, a network adapter 24, a keyboard 25, a pointing device 26, and a display 27.

[0020] In addition to an operating system, computer programs such as an inference service program 2P are stored in the ROM 22 or the auxiliary storage device 23. The inference service program 2P is a program for realizing the functions of a learning unit 201, an inference model storage unit 202, an inference unit 203, and the like shown in Fig. 3. As the auxiliary storage device 23, a hard disk or an SSD (Solid State Drive) or the like is used.

[0021] The RAM 21 is a main memory of the care support server 2. Computer programs such as the inference service program 2P are loaded into the RAM 21 as necessary.

[0022] The processor 20 executes a computer program loaded into the RAM 21. As the processor 20, a GPU (Graphics Processing Unit) or a CPU (Central Processing Unit) is used.

[0023] The network adapter 24 communicates with devices such as the caregiver terminal device 3 or a web server using protocols such as TCP / IP (Transmission Control Protocol / Internet Protocol). As the network adapter 24, a NIC (Network Interface Card) or a communication device for Wi-Fi is used.

[0024] The keyboard 25 and the pointing device 26 are input devices for an operator to input commands, data, and the like.

[0025] The display 27 displays a screen for inputting commands or data, a screen showing the results of calculations performed by the processor 20, and the like.

[0026] Returning to FIG. 1, the caregiver terminal device 3 is used to provide data for machine learning to the care support server 2, to request inference, and to receive an inference result from the care support server 2. A tablet computer, a smartphone, a personal computer, or the like is used as the caregiver terminal device 3. It is preferable that one caregiver terminal device 3 is provided for each caregiver 82, but one device may be shared by multiple caregivers 82. Below, an example will be described in which a tablet computer is used as the caregiver terminal device 3 and one caregiver terminal device 3 is provided for each caregiver 82.

[0027] As shown in FIG. 4, the caregiver terminal device 3 is composed of a processor 30, RAM 31, ROM 32, flash memory 33, a touch panel display 34, a network adapter 35, a short-range wireless board 36, a video camera 37, a wired input / output board 38, and an audio processing unit 39.

[0028] In addition to an operating system, computer programs such as a client program 3P are stored in the ROM 32 or the flash memory 33. The client program 3P is a program for realizing the functions of the data providing unit 301 and the client unit 302 shown in Fig. 5. The client program 3P is downloaded from an application server on the Internet to the caregiver terminal device 3 and installed in the flash memory 33 by an installer. The installer may be downloaded from the application server together with the client program 3P, or may be prepared in advance in the ROM 32 or the flash memory 33.

[0029] The RAM 31 is a main memory of the caregiver terminal device 3. Computer programs such as the client program 3P are loaded into the RAM 31 as necessary. The processor 30 executes the computer programs loaded into the RAM 31.

[0030] The touch panel display 34 is configured with a touch panel for the caregiver 82 to input commands or data, a display for displaying a screen, and the like.

[0031] The network adapter 35 communicates with other devices such as the care support server 2 using a protocol such as TCP / IP. As the network adapter 35, a communication device for Wi-Fi is used.

[0032] The short-distance wireless board 36 communicates with devices located within a few meters from the caregiver terminal device 3. In particular, in this embodiment, it communicates with a video camera 41, an indoor measuring device 42, and a biometric measuring device 44. As the short-distance wireless board 36, a communication device for Bluetooth is used.

[0033] The video camera 37 captures moving images and generates moving image data. In particular, in this embodiment, the video camera 37 is used to generate moving image data of the care recipient 81.

[0034] The wired input / output board 38 is a device that communicates with peripheral devices via a wire. The wired input / output board 38 is provided with a connection port, and the peripheral devices are connected to the connection port via a cable or directly. As the wired input / output board 38, for example, an input / output device conforming to the USB (Universal Serial Bus) standard is used.

[0035] The audio processing unit 39 is composed of an audio processing device, a microphone, a speaker, etc. Audio captured by the microphone is encoded into audio data by the audio processing device, audio data transmitted from another device is decoded into audio by the audio processing device, and the audio is reproduced by the speaker.

[0036] Returning to Fig. 1, video camera 41 is used to capture an image of care recipient 81. Video camera 41 is further provided with a microphone and an audio processing device, and is also capable of capturing audio and generating audio data. As video camera 41, a commercially available small video camera having a communication function of Bluetooth, Wi-Fi, or USB and a sound collection function is used.

[0037] In order to reduce the burden on the caregiver 82, it is desirable that one video camera 41 is assigned to each care recipient 81 and that the video cameras 41 are positioned in advance so that the care recipient 81 is within the shooting range. Of course, images may be captured and audio collected by the video camera 37 and the microphone of the audio processing unit 39 of the caregiver terminal device 3. For example, when the care recipient 81 and the caregiver 82 are out, images may be captured and audio collected by the caregiver terminal device 3 instead of the video camera 41. An example in which images are captured and audio collected by the video camera 41 will be described below.

[0038] The indoor measuring device 42 measures the condition in the room where the care recipient 81 is present. Specifically, it measures the temperature, humidity, air pressure, illuminance, amount of UV (ultraviolet), and amount of pollen in the room. A commercially available small measuring device for measuring the environment having a communication function of Bluetooth, Wi-Fi, or USB is used as the indoor measuring device 42. A device combining a plurality of commercially available sensors may be used as the indoor measuring device 42. For example, a device combining various sensors such as a temperature and humidity sensor, an air pressure sensor, an illuminance sensor, and a UV sensor from Alps Alpine may be used. The amount of pollen is the amount of pollen per unit volume.

[0039] An indoor measuring device 42 is installed in each room in which a care recipient 81 resides. When there are multiple care recipients 81 in one room, only one indoor measuring device 42 may be installed in the room for sharing, or one indoor measuring device 42 may be installed near each person. In addition, when going out, the care recipient 81 or the caregiver 82 may carry the indoor measuring device 42 with them.

[0040] The smart speaker 43 recognizes voice and performs tasks such as searching for information or controlling electrical appliances based on the recognition result. In this embodiment, these tasks are performed particularly for the care recipient 81. As the smart speaker 43, a commercially available smart speaker having a communication function of Bluetooth, Wi-Fi, or USB is used.

[0041] The biometric device 44 measures bioinformation such as the heart rate, pulse wave, blood oxygen concentration, blood pressure, cardiac potential, body temperature, myoelectric potential, and sweat rate of the care recipient 81. As the biometric device 44, a device having a communication function of Bluetooth, Wi-Fi, or USB and a function of measuring bioinformation, for example, a general-purpose wearable terminal such as Apple's Apple Watch, a wearable terminal with a health function, or a wearable terminal for medical use, is used.

[0042] 3 and 5, it is possible to infer the thoughts of the care recipient 81. These functions will be described below.

[0043] [Preparing the dataset] Fig. 6 is a diagram showing an example of a thought input screen 71. Fig. 7 is a diagram showing an example of a choice list 712. Fig. 8 is a diagram showing an example of raw data 61. Fig. 9 is a diagram showing an example of a data set 62. Fig. 10 is a diagram showing examples of classification tables 69A and 69B.

[0044] The operator of the care support server 2 must collect a lot of data on the care recipients 81 in order to have the care support server 2 perform machine learning and generate an inference model. Therefore, data is collected with the cooperation of the caregiver 82, with the care recipients 81 as subjects of data sampling. Each of the care recipients 81 (81a, 81b, ...) is given a unique user code in advance.

[0045] Although one inference model may be shared by multiple care recipients 81, in order to generate a highly accurate inference model with a small amount of data, it is preferable to generate a dedicated inference model for each care recipient 81. Below, a data collection method will be described using as an example a case in which data is collected to generate an inference model dedicated to a care recipient 81a when the care recipient 81a is being cared for by a caregiver 82a.

[0046] The caregiver 82a starts the client program 3P in advance on his / her own caregiver terminal device 3 and sets the mode to the data collection mode. Then, the data providing unit 301 (see FIG. 5) becomes active. The data providing unit 301 is composed of a moving image data extracting unit 30A, an environmental data acquiring unit 30B, a biological data acquiring unit 30C, a thought data generating unit 30D, and a raw data transmitting unit 30E.

[0047] Generally, when a human being has a thought such as a desire or emotion, the thought may be expressed in a movement or facial expression. This phenomenon is sometimes called a "desire expression reaction." Below, we will explain the case where a desire expression reaction is expressed as a hand movement as an example.

[0048] When the care recipient 81a wants to communicate a thought such as his / her desire, he / she moves his / her hands in accordance with the thought. Then, the movement of the care recipient 81a is filmed by a video camera 41 installed near the care recipient 81a, and the sound around the care recipient 81a is collected. If the care recipient 81a speaks while moving, the collected sound includes the voice of the care recipient 81a. Then, the video camera 41 outputs a moving image of the filmed movement and moving image / audio data 601 of the collected sound, and inputs it to the caregiver terminal device 3 of the caregiver 82a. The format of the moving image / audio data 601 is MP4, MOV, or the like.

[0049] In the caregiver terminal device 3, when the data collection mode is set and moving image and audio data 601 is input from the video camera 41, the moving image data extraction unit 30A, the environmental data acquisition unit 30B, the biometric data acquisition unit 30C, the thought data generation unit 30D, and the raw data transmission unit 30E perform processing as follows.

[0050] The moving image data extraction unit 30A extracts the data of the moving image portion from the moving image and audio data 601 as moving image data 611. In other words, the data of the audio portion is cut.

[0051] The environmental data acquiring unit 30B performs a process of acquiring data on the environment around the care recipient 81a as follows: The environmental data acquiring unit 30B issues a command to an indoor measuring device 42 near the care recipient 81a to measure the current temperature and the like.

[0052] The indoor measuring device 42 then measures the temperature, humidity, air pressure, illuminance, amount of UV light, amount of pollen, and the like, and transmits the space condition data 612 indicating the measurement results to the caregiver terminal device 3.

[0053] The environmental data acquiring unit 30B also accesses an API (Application Program Interface) of a website that transmits weather-related information, for example, an OpenWeatherMap API, via the Internet to acquire weather data 613 indicating information on the current weather, cloud cover, wind direction, and wind speed in the area where the care recipient 81a is currently located, as well as the maximum temperature, minimum temperature, and sunshine hours of the day in the area. The maximum temperature, minimum temperature, and sunshine hours may be actually observed or may be predicted. The sunshine hours may be calculated by the environmental data acquiring unit 30B from the times of sunrise and sunset.

[0054] Furthermore, the environmental data acquiring unit 30B acquires current location data 614 indicating the current location of the care recipient 81a. Specifically, the current location data 614 indicates the latitude and longitude of the care recipient 81a's current location of the care recipient 81a as the latitude and longitude of the care recipient terminal device 3 itself. The latitude and longitude can be acquired by a GPS (Global Positioning System) function of the care recipient terminal device 3 itself.

[0055] Furthermore, the current location data 614 indicates the location attribute of the care recipient 81a. The "location attribute" of the care recipient 81a is the attribute of the facility where the care recipient 81a is currently located. Examples of such attributes include "home", "classroom", "dining hall", "toilet", "workplace", "convenience store", "hospital", and "park". The location attribute may be acquired by the following method.

[0056] For example, a table in which latitude and longitude are associated with attributes for each facility is prepared in advance in the client program 3P. Then, the environmental data acquiring unit 30B checks the latitude and longitude acquired by the GPS function against the table, and acquires attributes that match the latitude and longitude as the location attributes of the care recipient 81a.

[0057] Alternatively, an oscillator that transmits radio waves indicating a unique identifier is installed in advance for each facility, and a table that associates the identifier with an attribute is prepared in advance in the client program 3P. Then, the environmental data acquisition unit 30B checks the identifier indicated in the radio waves received by the short-range wireless board 36 against the table, and acquires an attribute that matches this identifier as the location attribute of the care recipient 81a. For example, an iBeacon oscillator is used as such an oscillator. iBeacon is a registered trademark.

[0058] The environmental data acquiring unit 30B acquires time data 615 indicating the current time from an SNTP (Simple Network Time Protocol) server. The time data 615 may be acquired from the OS (Operating System) or clock of the caregiver terminal device 3 itself. This time can be said to be the time when the desire expression reaction occurred.

[0059] The biometric data acquiring unit 30C performs a process for acquiring the current biometric information of the care recipient 81a as follows. The biometric data acquiring unit 30C commands the biometric device 44 of the care recipient 81a to measure the current biometric information. The biometric device 44 of the care recipient 81a then measures biometric information such as heart rate, pulse wave, blood oxygen concentration, blood pressure, body temperature, myoelectric potential, sweat rate, and cardiac potential, and transmits biometric data 616 indicating the measurement results to the caregiver terminal device 3.

[0060] The thought data generating unit 30D generates the thought data 617 indicating the thought that the care receiver 81a wants to convey as the correct thought, for example, as follows.

[0061] The thought data generating unit 30D displays a thought input screen 71 as shown in FIG. 6 on the touch panel display 34. A list box 711 is arranged on the thought input screen 71. When the caregiver 82a touches the list box 711, the thought data generating unit 30D arranges an option list 712 directly below the list box 711 as shown in FIG. 7. The option list 712 shows character strings representing various thoughts previously selected by a researcher, developer, or caregiver 82a. Hereinafter, the previously selected thoughts are referred to as "candidate thoughts." Due to the size constraints of the touch panel display 34, the option list 712 cannot be displayed in its entirety at the same time, but the caregiver 82a can view the option list 712 part by part by scrolling the option list 712.

[0062] The caregiver 82a judges which of these candidate thoughts matches or is closest to the thought that the care recipient 81a wants to convey based on the actions of the care recipient 81a and the current context (environment such as time, place, and weather).The caregiver 82a then scrolls through the list of choices 712 and touches and selects the candidate thought that matches or is closest from the list of choices 712, thereby annotating the candidate thought, and touches the done button 713.

[0063] Then, the thought data generator 30D generates thought data 617 that indicates the selected thought candidate as a correct thought.

[0064] If the caregiver 82a does not know or is uncertain about the thought that the care recipient 81a wants to convey, the caregiver 82a may check with the care recipient 81a before making a selection. Even if the caregiver 82a knows the thought, the caregiver 82a may check with the care recipient 81a just to be sure before making a selection. If there is no suitable candidate thought, the caregiver 82a may add the thought that the care recipient 81a wants to convey as a new candidate thought, and the thought data generating unit 30D may generate the thought data 617 indicating the candidate thought as a correct thought.

[0065] Alternatively, when the care recipient 81a performs an action, the caregiver 82a may vocalize a thought candidate that is determined to match or be closest to the thought that the care recipient 81a wants to convey. Then, the voice of the caregiver 82a is recorded in the video / audio data 601. The thought data generating unit 30D extracts the voice of the caregiver 82a from the video / audio data 601 and distinguishes the vocalized thought candidate by a voice recognition function. Then, it generates thought data 617 indicating the distinguished thought candidate as a correct thought. Note that, if the vocalized content (thought) does not correspond to any of the existing thought candidates, the vocalized thought may be added as a new thought candidate, and thought data 617 indicating the added thought candidate as a correct thought may be generated. Alternatively, the vocalized thought may be classified into any of the existing thought candidates by AI, and data indicating the classified thought candidate may be generated as thought data 617.

[0066] The raw data transmission unit 30E associates the video data 611, spatial condition data 612, weather data 613, current location data 614, time data 615, biometric data 616, and thought data 617 with the user code of the care recipient 81a and transmits them to the care support server 2 as a set of raw data 61 as shown in Figure 8.

[0067] In the care support server 2, the learning unit 201 is composed of a movement pattern discrimination unit 20A, a dataset generation unit 20B, a dataset memory unit 20C, a machine learning processing unit 20D, and an accuracy testing unit 20E as shown in Figure 3, and performs processing to generate an inference model as follows.

[0068] When raw data 61 is transmitted from the caregiver terminal device 3, the movement pattern discrimination unit 20A discriminates which of a plurality of predefined patterns the movement of the care recipient 81a appearing in the moving image shown in the moving image data 611 in the raw data 61 corresponds to, for example, as follows.

[0069] The motion pattern discrimination unit 20A extracts the hand position from each frame of the moving image shown in the moving image audio data 601, and identifies the change in the hand position, i.e., the trajectory. Then, it is determined that the pattern most similar to the identified trajectory among the multiple patterns corresponds to the current motion pattern of the care recipient 81a.

[0070] The trajectory can be determined by known techniques, for example, Kinovea, which is motion analysis software.

[0071] It is also possible to determine which pattern the identified trajectory corresponds to by a known technique. For example, the determination may be made by pattern matching. Alternatively, the determination may be made by AI. Specifically, a large number of trajectories are prepared in advance, and a data set is prepared by an operator labeling which pattern each trajectory corresponds to. A classifier for classifying the trajectories is generated by supervised learning. Then, the motion pattern determination unit 20A determines which pattern the trajectory identified from the video data 611 corresponds to by using this classifier.

[0072] The multiple predefined patterns are, for example, as follows: "One-way" is a pattern in which the hand moves linearly from one position to another only once. "Circle" is a pattern in which the hand moves in a circular motion. "Round trip" is a pattern in which the hand moves linearly back and forth between one position and another only once. "Weak" is a pattern in which the hand moves tremblingly. "Repetition" is a pattern in which the same motion is performed two or more times in succession. However, to distinguish from "weak," the condition for "repetition" is that the hand moves a certain distance or more in one motion (for example, 20 centimeters or more). "Random" is a pattern in which the hand moves irregularly, and does not fall into any of the above five patterns.

[0073] The data set generating unit 20B generates motion pattern data 618 indicating the discrimination result by the motion pattern discriminator 20A, i.e., the discriminated pattern, and replaces the moving image data 611 of the raw data 61 with the generated motion pattern data 618 to generate a data set 62 as shown in Fig. 9. Then, the generated data set 62 is stored in the data set storage unit 20C in association with the user code of the care recipient 81a.

[0074] In this way, the above-mentioned processes and tasks are performed using a motion of the care recipient 81a as a trigger, whereby one set of data set 62 is generated and stored in the data set storage unit 20C. In the preparation phase, each time the care recipient 81a performs a motion, one set of data set 62 is generated by the above-mentioned processes and tasks and stored in the data set storage unit 20C.

[0075] Just as the movements of the care recipient 81 are classified into one of a plurality of patterns, other information may also be classified into one of a plurality of classes and shown in the data set 62.

[0076] For example, the data set generating unit 20B may classify the temperature indicated in the space situation data 612 into one of five categories according to the category table 69A shown in Fig. 10(A). Then, the temperature may be replaced with the classified category. Alternatively, the classified category may be added to the space situation data 612.

[0077] Alternatively, the data set generating unit 20B may calculate a discomfort index based on the temperature and humidity indicated in the spatial situation data 612, and classify the discomfort index into one of five categories according to the category table 69B shown in Fig. 10(B). Then, the classified category may be added to the spatial situation data 612.

[0078] Alternatively, the data set generating unit 20B may classify the time indicated in the time data 615 into one of time periods such as "sleeping time period", "wake-up time period", "breakfast time period", "class time period", "lunch time period", "rest time period", "free time period", "dinner time period", and "bedtime period" based on a timetable prepared in advance in accordance with the life style of the care recipient 81a. Then, this time may be replaced with the classified time period. Alternatively, the classified time period may be added to the time data 615. For more accurate classification, a timetable may be prepared for each day of the week, and the time may be classified based on the timetable corresponding to the day of the week on which the time data 615 was acquired.

[0079] Additionally, the data set generating unit 20B may calculate a temperature difference from the maximum temperature and the minimum temperature indicated in the weather data 613 and add the temperature difference to the weather data 613.

[0080] Similarly, when a care recipient 81 (81b, ...) other than the care recipient 81a performs an action, a data set 62 for each of the care recipients 81b, ... is generated by the above-mentioned processing and operations, and is stored in the data set memory unit 20C in correspondence with each user code.

[0081] [Generation of inference models] FIG. 10 is a diagram showing an example of a process flow for testing an inference model 68.

[0082] When a certain number (e.g., 1000 or more) of datasets 62 for a certain care recipient 81 are stored in the dataset storage unit 20C, the machine learning processing unit 20D performs machine learning based on these datasets 62 to generate an inference model for the care recipient 81. Hereinafter, an example will be described in which an inference model for a care recipient 81a is generated.

[0083] The machine learning processing unit 20D reads out the datasets 62 associated with the user code of the care recipient 81a from the dataset storage unit 20C. A predetermined percentage (e.g., 70%) of these datasets 62 are used as learning data, and the remaining (e.g., 30%) are used as test data. Hereinafter, the datasets 62 used as learning data will be referred to as "learning data 6A," and the datasets 62 used as test data will be referred to as "test data 6B."

[0084] The machine learning processing unit 20D generates an inference model 68 based on a known machine learning algorithm by using the thought data 617 of the data constituting each of the learning data 6A as a target variable (correct answer data, label) and the remaining data, i.e., the spatial situation data 612, the weather data 613, the current location data 614, the time data 615, and the biological data 616 as explanatory variables (input data). For example, the inference model 68 is generated by a machine learning algorithm such as SVM (Support Vector Machine), Random Forest, or Deep Learning. Alternatively, the inference model 68 may be generated using both SVM and BORUTA.

[0085] When the inference model 68 is generated, the accuracy test unit 20E tests the accuracy of the inference model 68 using the test data 6B. That is, as shown in FIG. 10, the spatial situation data 612, the weather data 613, the current location data 614, the time data 615, the biological data 616, and the motion pattern data 618 of a certain test data 6B are input as target data to the inference model 68, and the thought that the care recipient wants to convey is inferred (#801, #802). Specifically, any one of the various thought candidates described above is output as the inference result. However, when the inference model 68 is generated by deep learning, since there is one output node for each thought candidate, it is inferred that the thought candidate of the output node that outputs the largest value is the thought that the care recipient wants to convey. The same is true for the inference processing unit 20H described later.

[0086] The accuracy test unit 20E determines whether or not the inferred thought (candidate thought) matches the correct thought shown in the thought data 617 (#803). Similarly, the processes of steps #801 to #803 are performed for the other test data 6B.

[0087] If the number of times that the correct thought and the inferred thought in step #803 match with respect to the number of times the processes of steps #801 to #803 are performed is equal to or greater than a predetermined rate (e.g., 90%), the accuracy testing unit 20E certifies that the inference model 68 has passed, and the inference model is stored in the inference model storage unit 202 in association with the user code of the care recipient 81a. This completes preparations for inferring the thoughts of the care recipient 81a.

[0088] On the other hand, if the ratio is less than the predetermined ratio, the accuracy test unit 20E discards the inference model 68 and continues to collect the data set 62. Alternatively, the administrator may adjust the target data by changing the value of a weight parameter of any of the items constituting each of the target data (spatial situation data 612, weather data 613, current location data 614, time data 615, biometric data 616, and motion pattern data 618) (e.g., the weather in the weather data 613 or the location attribute of the current location data 614) or by deleting any of the items (e.g., the cloud cover in the weather data 613 or the latitude and longitude in the current location data 614), and then rerun the test.

[0089] Inference models 68 for care recipients 81 (81b, . . . ) other than care recipient 81a are also generated in a similar manner and stored in the inference model storage unit 202 in association with the respective user codes.

[0090] [Inference of Thought] Fig. 12 is a diagram showing an example of raw data 63. Fig. 13 is a diagram showing an example of the flow of inference processing. Fig. 14 is a diagram showing an example of an inference result screen 72.

[0091] When the inference model 68 of a certain care recipient 81 is stored in the data set storage unit 20C, the inference unit 203 of the care support server 2 and the client unit 302 of the caregiver terminal device 3 can infer the thoughts of the care recipient 81. Hereinafter, the inference method will be described taking as an example a case where the care recipient 81a is photographed by the video camera 41 while being cared for by a caregiver 82a, and the thoughts of the care recipient 81a are inferred.

[0092] The caregiver 82a starts the client program 3P in advance on his / her own caregiver terminal device 3 and sets the mode to the inference mode. Then, the client unit 302 becomes active. As shown in Fig. 5, the client unit 302 is composed of a moving image data extraction unit 30F, an environmental data acquisition unit 30G, a biological data acquisition unit 30H, a raw data transmission unit 30J, and an inference result output unit 30K.

[0093] When the care recipient 81a makes a motion to communicate his / her thoughts to the caregiver 82a, the motion of the care recipient 81a is photographed by a video camera 41 installed near the care recipient 81a, and sounds around the care recipient 81a are collected, in the same manner as in data collection. This generates video / audio data 601. The video / audio data 601 is then output from the video camera 41 and input to the caregiver terminal device 3 of the caregiver 82a.

[0094] In the caregiver terminal device 3, when the data collection mode is set and moving image and audio data 601 is input from the video camera 41, the moving image data extraction unit 30F, the environmental data acquisition unit 30G, the biometric data acquisition unit 30H, and the raw data transmission unit 30J process the data as follows.

[0095] Moving image data extraction unit 30F extracts the data of the moving image portion from the currently input moving image audio data 601 as moving image data 631. That is, similar to moving image data extraction unit 30A of data provider 301, the data of the audio portion is cut off.

[0096] The environmental data acquisition unit 30G acquires the following data in a manner similar to the processing method used by the environmental data acquisition unit 30B. By issuing a command to an indoor measuring device 42 in the vicinity of the care recipient 81a to measure the current temperature, etc., the spatial situation data 632 indicating the current temperature, humidity, air pressure, illuminance, UV amount, pollen amount, etc. is acquired. Weather data 633 indicating information on the current weather, cloud cover, wind direction, and wind speed in the area where the care recipient 81a is currently located, as well as information on the maximum and minimum temperatures of the day in the area, is acquired from a website. Current location data 634 indicating the latitude and longitude of the current location of the care recipient 81a and location attributes is acquired by the GPS function or iBeacon. Time data 635 indicating the current time is acquired from an SNTP server or the like.

[0097] The biometric data acquisition unit 30H acquires biometric data 636 indicating biometric information such as the current heart rate, pulse wave, blood oxygen concentration, blood pressure, cardiac potential, body temperature, myoelectric potential, and sweat rate of the care recipient 81a from the biometric measurement device 44 of the care recipient 81a in a manner similar to the processing method used by the biometric data acquisition unit 30C.

[0098] The raw data transmission unit 30J associates the video data 631, spatial condition data 632, weather data 633, current location data 634, time data 635, and biometric data 636 with the user code of the care recipient 81a and transmits them to the care support server 2 as a set of raw data 63 as shown in Figure 12.

[0099] In the care support server 2, the inference unit 203 is composed of a motion pattern discrimination unit 20F, a target data generation unit 20G, an inference processing unit 20H, and an inference result response unit 20J as shown in Figure 3, and infers the thoughts of the care recipient 81a as follows.

[0100] When the raw data 63 is transmitted from the caregiver terminal device 3, the movement pattern discrimination unit 20F discriminates to which of a plurality of predefined patterns the current movement of the care recipient 81a appearing in the moving image shown in the moving image data 631 in the raw data 63 corresponds. The discrimination method is the same as the discrimination method by the movement pattern discrimination unit 20A.

[0101] The target data generation unit 20G generates motion pattern data 637 indicating the discrimination result, i.e., the discriminated pattern, by the motion pattern discrimination unit 20F, and generates target data 64 by replacing the moving image data 631 of the raw data 63 with the generated motion pattern data 637.

[0102] 13, the inference processing unit 20H inputs the generated target data 64 to the inference model 68 stored in the inference model storage unit 202 in association with the user code of the care recipient 81a, and obtains an output value from the inference model 68, thereby inferring the thoughts of the care recipient 81a. Note that, as described above, if the inference model 68 is generated by deep learning, it infers that the candidate thought of the output node that outputs the largest value is the thought of the care recipient.

[0103] The inference result reply unit 20J transmits the inference result data 65 indicating the thoughts of the care recipient 81a inferred by the inference processing unit 20H to the caregiver terminal device 3 of the caregiver 82a.

[0104] In the caregiver terminal device 3, when the inference result output unit 30K (see Figure 5) receives the inference result data 65, it displays a string of characters representing the thought shown in the inference result data 65 on the touch panel display 34 on an inference result screen 72 as shown in Figure 14, and plays back a sound representing the thought by the sound processing unit 39.

[0105] Furthermore, the voice reproduced by the voice processing unit 39 may be played back to the smart speaker 43 to address the needs of the care recipient 81a. For example, when the voice "It's hot" is input, the smart speaker 43 turns on the air conditioner in the room where the care recipient 81a is present or lowers the set temperature. Or, when the voice "Teacher, where are you?" is input, the smart speaker 43 generates an e-mail with the content "Teacher, where are you?" and sends it to the teacher at the school of the care recipient 81a.

[0106] The thoughts of care recipients 81 (81b, ...) other than care recipient 81a are also inferred in a similar manner based on their respective raw data 63 and inference model 68, and the inference results are transmitted to the caregiver terminal device 3 of the caregiver 82 who is caring for each of them.

[0107] [Overall processing flow of the care support server 2] FIG. 15 is a flowchart showing an example of the overall flow of processing by the inference service program 2P.

[0108] Next, the overall flow of processing in the care support server 2 will be described with reference to a flowchart.

[0109] The care support server 2 executes processing according to the procedure shown in the flowchart of FIG. 15 based on the inference service program 2P.

[0110] In the data set preparation phase, the care support server 2 receives the raw data 61 (see FIG. 8) from the caregiver terminal device 3 (#811), calculates the trajectory of the hand of the care recipient 81 based on the moving image data 611 in the raw data 61 (#812), and identifies the movement pattern represented by the trajectory (#813). Note that the raw data 61 is associated with a user code.

[0111] Then, the care support server 2 generates movement pattern data 618 indicating the identified pattern (#814), generates a data set 62 (see FIG. 9) by replacing the video data 611 with the movement pattern data 618 (#815), and stores the data set 62 in association with this user code (#816).

[0112] Every time the care support server 2 receives the raw data 61, it executes the processes of steps #812 to #816.

[0113] In the machine learning phase, when the number of data sets 62 associated with the same user code reaches a predetermined number, the care support server 2 generates an inference model 68 by performing machine learning based on these data sets 62 (#821). Then, when it is confirmed by a test that the inference model 68 has a certain degree of accuracy, the inference model 68 is stored in association with the user code (#822).

[0114] In the inference phase, when the care support server 2 receives the raw data 63 (see FIG. 12) from the caregiver terminal device 3 (#831), it calculates the trajectory of the hand of the care recipient 81 based on the moving image data 631 in the raw data 63 (#832) and identifies the movement pattern represented by the trajectory (#833). Note that the raw data 63 is associated with a user code.

[0115] The care support server 2 generates motion pattern data 637 indicating the identified pattern (#834), and replaces the moving image data 631 with the motion pattern data 637 to generate object data 64 (#835).

[0116] The care support server 2 then inputs the target data 64 into the inference model 68 associated with this user code to infer the thoughts of the care recipient 81 (#836), and returns inference result data 65 indicating the inference result to the caregiver terminal device 3 (#837). The inference result is then output in the caregiver terminal device 3 as a character string or voice.

[0117] Every time the care support server 2 receives the raw data 63, it executes the processes of steps #832 to #837.

[0118] According to this embodiment, when the care recipient 81 has thoughts such as what he or she wants to do or communicate, the thoughts can be inferred and output even when the caregiver 82 is not present. Therefore, the care recipient 81 can use the ICT device while reducing the burden on the caregiver 82 more than ever before. Even if the care recipient 81 is a severely mentally and physically disabled child or adult or a dementia patient, he or she can use the ICT device more easily than ever before.

[0119] [Modifications] Fig. 16 is a diagram showing a modified example of the functional configuration of the care support server 2. Fig. 17 is a diagram showing a modified example of the functional configuration of the caregiver terminal device 3. Fig. 18 is a diagram showing an example of the event table 605. Fig. 19 is a diagram showing a modified example of the option list 732. Fig. 20 is a diagram showing an example of the explanatory variable table 606.

[0120] In this embodiment, the care support server 2 performs machine learning using the data set 62, i.e., the data of the items shown in Fig. 9 as learning data, but machine learning may be performed by adding data of other items to the data set 62. The same applies to inference.

[0121] For example, the care support server 2 may perform machine learning and inference by including data indicating three-axis geomagnetism and three-axis acceleration at the current location of the care recipient 81 in the data set 62 and the target data 64, respectively.

[0122] In this embodiment, the data set 62 and the target data 64 respectively include the motion pattern data 618 and the motion pattern data 637 as data indicating the desire expression reaction of the care recipient 81. That is, data indicating the hand motion pattern of the care recipient 81 is included, and the care support server 2 performs machine learning and inference based on this pattern. However, machine learning and inference may be performed based on other desire expression reactions. For example, machine learning and inference may be performed based on the eye movement pattern, eyelid movement (opening and closing) pattern, voice pitch change pattern, vocalization, or facial expression pattern of the care recipient 81. Alternatively, machine learning and inference may be performed based on a plurality of desire expression reactions (for example, hand motion patterns and facial expression patterns) arbitrarily selected from these desire expression reactions. The same applies to the modified examples described later with reference to FIG. 16 to FIG. 20.

[0123] The eye movements, eyelid movements, and facial expressions may be extracted from video images captured by the video camera 41 or the video camera 37 of the caregiver terminal device 3. Alternatively, the care recipient 81 may wear a glasses-type wearable terminal, and the eye movements and eyelid movements may be captured by the wearable terminal. The pitch of the voice and the presence or absence of vocalization may be identified from the voice data included in the video and voice data 601.

[0124] In this embodiment, the care support server 2 identifies the trajectory of the hand of the care recipient 81 based on a moving image, but it may also identify the trajectory based on posture, angular velocity, or angular acceleration data acquired by a gyro sensor of the biometric device 44 worn on the wrist of the care recipient 81. The same applies to the modified example described next with reference to Figs. 16 to 20.

[0125] The care support server 2 may include only some of the data items shown in the raw data 61 (see FIG. 8) in the data set 62 (see FIG. 9) instead of including all the data items shown in the raw data 61 (see FIG. 8) in the data set 62. For example, the data set 62 may be generated so as to include only the temperature and humidity data in the spatial situation data 612, the wind direction and wind speed data in the weather data 613, and the body temperature data in the biological data 616 in addition to the current location data 614, the time data 615, and the thought data 617 in the raw data 61.

[0126] In this embodiment, the care support server 2 generates one inference model 68 for one care recipient 81, but may generate an inference model 68 for each time period and location for one care recipient 81. Hereinafter, a generation method will be described using an example of generating an inference model 68 for a care recipient 81a.

[0127] An inference service program 2Q is installed in place of the inference service program 2P in the care support server 2. According to the inference service program 2Q, the functions of a learning unit 211, an inference model storage unit 212, and an inference unit 213 shown in FIG.

[0128] The learning unit 211 is composed of a motion pattern discrimination unit 21A, a data set generation unit 21B, a data set storage unit 21C, a machine learning processing unit 21D, and an accuracy testing unit 21E, and performs processing for generating an inference model.

[0129] The inference section 213 is made up of a motion pattern discrimination section 21F, a target data generation section 21G, an inference processing section 21H, and an inference result response section 21J, and performs processing to infer thoughts.

[0130] A client program 3Q is installed in place of the client program 3P in the caregiver terminal device 3. The client program 3Q realizes functions of a data providing unit 311 and a client unit 312 shown in FIG.

[0131] The data providing unit 311 is composed of a desire expression detecting unit 31A, an environmental data acquiring unit 31B, a biological data acquiring unit 31C, a thought data generating unit 31D, and a raw data transmitting unit 31E, and provides the care support server 2 with data for learning.

[0132] The client unit 312 is composed of a desire expression detection unit 31F, an environmental data acquisition unit 31G, a biological data acquisition unit 31H, a raw data transmission unit 31J, and an inference result output unit 31K, and causes the care support server 2 to perform inference.

[0133] When the care recipient 81a moves his / her hands to communicate his / her desires or other thoughts, the movement of the care recipient 81a is photographed and the voice is collected by a video camera 41 located near the care recipient 81a, and the moving image and voice data 601 is output to the caregiver terminal device 3 of the caregiver 82a.

[0134] In the caregiver terminal device 3, when moving image and audio data 601 is input from the video camera 41 during data collection mode, the desire expression detection unit 31A determines whether a desire expression reaction has occurred based on the moving image and audio data 601, for example, as follows.

[0135] If the hand movement shown in the video / audio data 601 is a predetermined movement (for example, a movement of continuously moving back and forth between two points), it is determined that a desire expression reaction has occurred. Alternatively, if the sound shown in the video / audio data 601 continues for a predetermined time (for example, 2 seconds) or more, it may be determined that a desire expression reaction has occurred.

[0136] When the desire expression detection section 31A determines that a desire expression reaction has occurred, the following process is carried out by the environmental data acquisition section 31B and the biological data acquisition section 31C.

[0137] The environmental data acquiring unit 31B acquires data related to the environment around the care recipient 81a, that is, spatial situation data 612, weather data 613, current location data 614, and time data 615. The acquisition method is similar to the method of acquiring these data by the environmental data acquiring unit 30B (see FIG. 5).

[0138] The biometric data acquiring unit 31C acquires the current biometric information of the care recipient 81a, that is, the biometric data 616. The acquisition method is the same as the data acquisition method by the biometric data acquiring unit 30C.

[0139] It should be noted that the desire expression detection section 31A extracts moving image data 611 from the moving image and audio data 601 in the same manner as the moving image data extraction section 30A.

[0140] The thought data generating unit 31D has an event table 605 for the care recipient 81a. As shown in FIG. 18, the event table 605 shows the time and place where the event occurs for each event (occasion, activity) of the care recipient 81a, such as daily break, school lunch, class, free time, dinner, bath, walk, and bedtime. The place may be represented by latitude and longitude, or may be represented by location attributes (for example, "home", "classroom", "dining room", "toilet", etc.). Furthermore, the event table 605 shows multiple thoughts that may occur to the care recipient 81a during the event as thought candidates. These thought candidates are determined in advance by the researcher or developer in consultation with the caregiver 82a. The thought data generating unit 31D also has an event table 605 for each of the other care recipients 81 (81b, ...).

[0141] The thought data generating unit 31D generates thought data 607 indicating the thought that the care receiver 81a wants to convey as the correct thought based on the event table 605 and the like as follows.

[0142] The thought data generator 31D determines an event corresponding to the current location indicated in the current location data 614 and the time indicated in the time data 615 based on the event table 605. For example, if the current location data 614 indicates "classroom" and the time data 615 indicates "11:50", the thought data generator 31D determines that the event is "class".

[0143] When the thought data generator 31D determines the event, it displays the thought input screen 73 (see FIG. 6) on the touch panel display 34. When the caregiver 82a touches the list box 731, a choice list 732 is placed directly below the list box 711, as shown in FIG. 19. In the choice list 732, a plurality of thought candidates that are associated with this event in the event table 605 are placed as choices.

[0144] The caregiver 82a judges which of the arranged thought candidates matches or is closest to the thought that the care recipient 81a wants to convey based on the actions and appearance of the care recipient 81a. Then, the caregiver 82a touches and selects the thought candidate judged to be closest from the option list 712, and touches the done button 713.

[0145] Then, the thought data generating unit 31D generates thought data 607 that indicates the selected thought candidate as a correct thought. If there is no suitable thought candidate, it may be possible to add a new thought candidate, as in the thought data generating unit 30D.

[0146] The raw data transmission unit 31E associates the video data 611, spatial condition data 612, weather data 613, current location data 614, time data 615, biometric data 616, and thought data 607 with the event code of the current event and the user code of the care recipient 81a, and transmits them to the care support server 2 as a set of raw data 61.

[0147] In the care support server 2, when the raw data 61 is transmitted from the caregiver terminal device 3, the motion pattern discrimination unit 21A discriminates which of a plurality of predefined patterns corresponds to the motion of the care recipient 81a appearing in the moving image shown in the moving image data 611 in the raw data 61. The discrimination method is the same as the discrimination method by the motion pattern discrimination unit 20A in FIG. 3.

[0148] The data set generating unit 21B generates a data set 62 (see FIG. 9) based on the discrimination result by the motion pattern discriminator 21A, similar to the data set generating unit 20B. Then, the data set generating unit 21B stores the generated data set 62 in the data set storage unit 21C in association with the user code of the care recipient 81a and the current event code.

[0149] The machine learning processing unit 21D has an explanatory variable table 606 for the care recipient 81a. As shown in FIG. 20, the explanatory variable table 606 shows one or more explanatory variables and their respective weighting coefficients used for machine learning for each daily event of the care recipient 81a. These explanatory variables are determined in advance by a researcher or developer in consultation with the caregiver 82a based on the daily state of the care recipient 81a from among variables included in the spatial situation data 612, the weather data 613, the biological data 616, and the motion pattern data 618. In addition, the time period and the place where each event occurs are shown, as in the event table 605 (see FIG. 18). The machine learning processing unit 21D also has explanatory variable tables 606 for each of the other care recipients 81 (81b, ...).

[0150] When a certain number (e.g., 100 or more) of datasets 62 associated with the user code of the care recipient 81a and the same event code are stored in the dataset storage unit 21C, the machine learning processing unit 21D reads out these datasets 62 from the dataset storage unit 21C. A predetermined proportion of these datasets 62 is used as learning data 6A, and the remainder is used as test data 6B.

[0151] The machine learning processing unit 21D generates an inference model 68 of this event of the care recipient 81a based on a known machine learning algorithm using the learning data 6A. The point that the thought data 617 of the data constituting each of the learning data 6A is used as a target variable is the same as the machine learning by the machine learning processing unit 20D. However, among the variables included in each of the spatial situation data 612, the weather data 613, the current location data 614, the time data 615, the biological data 616, and the motion pattern data 618, the variables associated with this event in the explanatory variable table 606 of the care recipient 81a are selected and used as explanatory variables. That is, the machine learning processing unit 21D performs machine learning using the thought data 617 and the data indicating the selected explanatory variables as a data set. In addition, at this time, the weighting coefficient of each variable (explanatory variable) indicated in the explanatory variable table 606 is used.

[0152] The explanatory variables used in the machine learning may be selected by the data set generating unit 21B instead of the machine learning processing unit 21D. That is, when the raw data 61 is transmitted from the caregiver terminal device 3, the data set generating unit 21B selects explanatory variables corresponding to the event code of the raw data 61 based on the explanatory variable table 606 of the care recipient 81a. Then, the thought data 617 and the selected explanatory variable data are combined and stored as a data set 62 in the data set storage unit 20C.

[0153] When the inference model 68 is generated, the accuracy test unit 21E uses the test data 6B to test the accuracy of the inference model 68. The test method is basically the same as the test method by the accuracy test unit 20E, except that the explanatory variables shown in the explanatory variable table 606 are used.

[0154] Then, if the accuracy testing unit 21E certifies that the inference model 68 has passed, it stores the inference model 68 in the inference model storage unit 212 in association with the user code of the care recipient 81a and the event code of this event.

[0155] Each part of the inference unit 213 of the care support server 2 and each part of the client unit 312 of the caregiver terminal device 3 perform processing for inferring the thoughts of the care recipient 81. Hereinafter, this processing will be described taking as an example a case where the thoughts of the care recipient 81a are inferred.

[0156] In the caregiver terminal device 3, when the moving image and audio data 601 is input from the video camera 41 during the inference mode, the desire expression detection unit 31F judges whether or not a desire expression reaction has occurred based on the moving image and audio data 601. The judgment method is the same as the judgment method in the desire expression detection unit 31A. Furthermore, the desire expression detection unit 31F extracts moving image data 631 in the same manner as the moving image data extraction unit 30F (see FIG. 3).

[0157] When it is determined that a desire expression response has occurred, the environmental data acquisition section 31G and the biological data acquisition section 31H perform the following processes.

[0158] The environmental data acquisition unit 31G acquires spatial situation data 632, weather data 633, current location data 634, and time data 635 in a manner similar to the processing method used by the environmental data acquisition unit 30B.

[0159] The biometric data acquiring section 31H acquires the biometric data 636 in a manner similar to the processing method used by the biometric data acquiring section 30C.

[0160] The raw data transmitting unit 31J transmits the moving image data 631, the spatial situation data 632, the weather data 633, the current location data 634, the time data 635, and the biometric data 636 to the care support server 2 as a set of raw data 61 in association with the user code of the care recipient 81a.

[0161] In the care support server 2, the movement pattern discrimination unit 21F discriminates to which pattern the movement of the care recipient 81a corresponds, in the same manner as the movement pattern discrimination unit 20A.

[0162] The target data generating unit 21G generates motion pattern data 637 indicating the discrimination result by the motion pattern discriminator 21F, and generates target data 64 by replacing the moving image data 631 of the raw data 61 with the generated motion pattern data 637.

[0163] The inference processing unit 21H has an explanatory variable table 606 (see FIG. 20) like the machine learning processing unit 21D.

[0164] The inference processing unit 21H determines an event corresponding to the current location indicated in the current location data 634 and the time indicated in the time data 635 in the target data 64, based on the explanatory variable table 606. For example, when the current location data 634 indicates "classroom" and the time data 635 indicates "11:55", it determines that the event is "break".

[0165] The inference processing unit 21H inputs the value of a candidate thought corresponding to the identified event in the target data 64 into an inference model 68 corresponding to the user code of the care recipient 81a and the event code of this event, and obtains an output value from the inference model 68, thereby inferring the thoughts of the care recipient 81a.

[0166] The inference result reply unit 21J transmits data indicating the thoughts of the care recipient 81a inferred by the inference processing unit 21H as inference result data 65 to the caregiver terminal device 3 of the caregiver 82a.

[0167] In the caregiver terminal device 3, when the inference result output unit 31K receives the inference result data 65, it displays a character string representing the thought shown in the inference result data 65 on the touch panel display 34 as an inference result screen 72 (see FIG. 14), and also plays a sound representing the thought by the sound processing unit 39. Then, the sound may be played through the smart speaker 43 to address the desire of the care recipient 81a.

[0168] In the caregiver terminal device 3, when the inference result output unit 31K (see Figure 17) receives the inference result data 65, it displays a string of characters representing the thought shown in the inference result data 65 on the touch panel display 34, and plays a sound representing the thought by the sound processing unit 39.

[0169] The thoughts of care recipients 81 (81b, ...) other than care recipient 81a are also inferred in a similar manner based on their respective target data 64 and inference model 68, and the inference results are transmitted to the caregiver terminal device 3 of the caregiver 82 who is caring for each of them.

[0170] According to such a method of performing machine learning by selecting explanatory variables to be used for each event in advance, an inference model 68 that fits each event can be generated, and the accuracy of inference can be improved. One of the major factors that enabled the inventor to devise this method is that he was able to identify by experiment those variables that have a high correlation with the desire expression response from among the various variables mentioned above.

[0171] The inventors calculated the correlation between the thoughts of the care recipient 81 and various variables related to the environment or body of the care recipient 81 (temperature, humidity, air pressure, illuminance, amount of UV, amount of pollen, weather, cloud cover, wind direction, wind speed, maximum temperature, minimum temperature, sunshine hours, temperature difference, current location, time, time zone, three-axis geomagnetism, three-axis acceleration, heart rate, pulse wave, blood oxygen concentration, blood pressure, body temperature, amount of sweat, cardiac potential, muscle potential, movement pattern, eye movement, eyelid movement, change in voice pitch, presence or absence of vocalization, facial expression pattern, etc.) through experiments. As a result, it was found that the thoughts of the care recipient 81 are easily changed when the current location, i.e., place, is different, and the thoughts of the care recipient 81 are easily changed when the time or time zone is different. In other words, it was found that the correlation between the place and the desire expression reaction is large.

[0172] The inventors were also able to identify that the combination of place and time corresponded to the daily events of the care recipient 81. Thus, as in the above-described modified example, an inference model 68 could be generated for each combination of place and time, i.e., each event.

[0173] Furthermore, as a result of the experiment, the inventor found that among these various variables, those with a high correlation with the desire expression reaction differ depending on the place or time. Some examples of explanatory variables used in machine learning are shown in the event table 605 (see FIG. 18) and the explanatory variable table 606 (FIG. 20), but these examples were selected based on the results of an experiment and investigation conducted by the inventor at a special needs school. The explanatory variables used in machine learning may be set for each care recipient 81 according to the context (environment) of the care recipient 81. For example, blood oxygen concentration may be included as an explanatory variable for an event in which the care recipient 81 has an opportunity to call someone. This is particularly suitable when the care recipient 81 is on an artificial respirator.

[0174] 3, 5, 16, and 17, the process of generating the data set 62 from the raw data 61 is performed by the care support server 2, but it may be performed by the caregiver terminal device 3. Similarly, the process of generating the target data 64 from the raw data 63 may be performed by the caregiver terminal device 3.

[0175] In the examples of FIG. 3 and FIG. 16, the machine learning and inference are performed by the care support server 2, but each may be performed by a different server. For example, the machine learning server may perform the machine learning and the inference server may perform the inference. In this case, after generating the inference model 68, the machine learning server transmits the inference model 68 to the inference server and installs it in the inference server. Alternatively, if the inference model 68 is a trained model generated by deep learning, only the parameters of the inference model 68 may be transmitted to the inference server and the parameters may be applied to the inference model prepared in the inference server. The caregiver terminal device 3 transmits the raw data 61 to the machine learning server and transmits the raw data 63 to the inference server.

[0176] 3 and 16, the data set 62 is stored in the care support server 2, but it may be stored in another server. For example, it may be stored in a cloud server such as JGN (Japan Gigabit Network) of the National Institute of Information and Communications Technology (NICT). Machine learning may be executed by a cloud server having a machine learning engine.

[0177] In this embodiment, the inference model 68 is generated for each care recipient 81 and used separately, but a common inference model 68 may be generated and shared by a plurality of care recipients 81.

[0178] In addition, the overall or individual configurations of the thinking and inference system 1, the care support server 2, and the caregiver terminal device 3, the processing content, the processing order, the data set configuration, etc. can be modified as appropriate in accordance with the spirit of the present invention.

Claims

1. A data set acquisition means for acquiring a plurality of data sets indicating a first condition when a first physical reaction occurs in the person with speech difficulty and a first thought that the person with speech difficulty has when the first reaction occurs; an inference model generating means for generating an inference model by performing machine learning using, for each of a plurality of combinations of a time period and a location, the first condition shown in the data set acquired at the time period and the location of the combination as an explanatory variable and the first thought as a target variable; an inference means for inferring a second thought when a second reaction occurs by inputting input data indicating a second condition when the second reaction occurs into the inference model generated for a combination of a time period and a place when the second reaction occurs among the plurality of combinations; an output means for outputting the second thought; having A thought and reasoning system characterized by the above.

2. A data set acquisition means for acquiring a plurality of data sets indicating the condition of the person with speech difficulty when there is a physical reaction and the thoughts of the person with speech difficulty when there is the reaction; an inference model generating means for generating an inference model by performing machine learning using, for each of a plurality of combinations of a time period and a location, the conditions shown in the data set acquired for the time period and the location of the combination as explanatory variables and the thoughts as objective variables; An inference model generation system comprising:

3. The data set acquisition means acquires, as the data set, biometric information of the person with speech difficulty or data indicating a surrounding condition. The inference model generation system according to claim 2.

4. The data set acquisition means acquires data indicating the surrounding climate as the data set. The inference model generation system according to claim 3.

5. The data set acquisition means acquires data indicating information on each of a plurality of items as the data set, the inference model generating means generates the inference model using weighting coefficients set for each of the plurality of items; An inference model generation system as claimed in claim 3 or claim 4.

6. The data set acquisition means acquires data indicating information on each of a plurality of items as the data set, a weighting coefficient for each of the plurality of items is determined according to each of the plurality of combinations; the inference model generating means generates the inference model for each of the plurality of combinations by using the weighting coefficient corresponding to the combination; An inference model generation system as claimed in claim 3 or claim 4.

7. A plurality of candidate thoughts are prepared in advance in accordance with each of the plurality of combinations, The data set acquisition means acquires, as the data set for each of the plurality of combinations, data indicating a thought candidate selected by a caregiver of the person with speech difficulty based on the reaction from among the plurality of thought candidates corresponding to the combination. An inference model generation system according to any one of claims 2 to 6.

8. an inference means for inferring a second thought at the time of the second reaction by inputting input data indicating a second condition at the time of the second reaction into an inference model generated by the inference model generation system according to any one of claims 2 to 7 for the time period and place at which the second reaction occurred; an output means for outputting the second thought; having A thinking and inference device characterized by the above.

9. Obtaining a plurality of data sets indicative of the condition of the person with speech difficulty when a physical reaction occurs and the thoughts of the person with speech difficulty when the reaction occurs; A computer-implemented inference model generation method, comprising: generating an inference model for each of a plurality of combinations of time periods and locations by performing machine learning using the conditions indicated in the dataset acquired at the time periods and locations of the combination as explanatory variables and the thoughts as objective variables.

10. a computer is caused to execute a process of acquiring a plurality of data sets indicative of a condition when a physical reaction occurs of the person with speech difficulty and a thought that the person with speech difficulty has when the reaction occurs; and executing a process of generating an inference model by performing machine learning for each of a plurality of combinations of time periods and locations using the conditions shown in the data set acquired at the time periods and locations of the combinations as explanatory variables and using the thoughts as objective variables for each of the combinations of time periods and locations, the computer; A computer program comprising:

Images