Information processing method, information processing program, and information processing device

Abstract

To provide an information processing method etc. capable of watching using an apparatus which can be simply installed.SOLUTION: An information processing method causes a computer to execute the processing of: acquiring face image data of a user acquired through an interactive robot; acquiring first abnormality information by inputting the acquired face image data, to a first learning model learned so as to output a first abnormal state on abnormality of the user when the face image data is inputted; acquiring interaction data obtained by interacting with the user through the interactive robot; acquires second abnormality information by inputting the acquired interaction data, to a second learning model learned so as to output the second abnormality information on the abnormality of the user when the interaction data is inputted; and outputting the abnormality information to other terminal devices, when both of the first abnormality information and the second abnormality information are outputted within a predetermined time.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present invention relates to an information processing method for outputting abnormality information. [Background technology]

[0002] There have been proposed monitoring systems for the elderly. For example, Patent Document 1 proposes a medical system that can grasp the living conditions of those being monitored, such as the elderly, at an early stage, take appropriate measures, prevent deterioration, encourage independence, maintain quality of life, and reduce costs for the service provider. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-201851 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional systems such as that proposed in Patent Document 1 require the installation of multiple sensors in the homes of elderly people. This requires time and effort for installation and maintenance. The present invention has been made in light of these circumstances. Its purpose is to provide an information processing method and the like that detects abnormalities in a person being monitored using an easily installed terminal and outputs abnormality information. [Means for solving the problem]

[0005] An information processing method according to one aspect of the present application includes a computer executing a process in which facial image data of a user is acquired through an interactive robot, the acquired facial image data is input into a first learning model trained to output a first abnormal state related to the user's abnormality when the facial image data is input, and first abnormality information is acquired by inputting the acquired facial image data into a first learning model trained to output a first abnormality state related to the user's abnormality when the facial image data is input, dialogue data exchanged between the user and the interactive robot is acquired, and the acquired dialogue data is input into a second learning model trained to output second abnormality information related to the user's abnormality when the dialogue data is input, and if both the first abnormality information and the second abnormality information are output within a predetermined time, the computer outputs abnormality information to another terminal device. [Effects of the Invention]

[0006] According to one aspect of the present application, it is possible to detect an abnormality in a person being watched over and output abnormality information using a terminal that is easy to install. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of the configuration of a monitoring system. [Figure 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a management server. [Figure 3] FIG. 2 is a block diagram showing an example of the hardware configuration of a monitoring terminal. [Figure 4] FIG. 2 is a block diagram showing an example of the hardware configuration of a family terminal. [Figure 5] FIG. 2 is a block diagram showing an example of the hardware configuration of an expert terminal. [Figure 6] FIG. 10 is an explanatory diagram showing an example of a subject DB. [Figure 7] FIG. 2 is an explanatory diagram illustrating an example of a scenario DB. [Figure 8] FIG. 10 is an explanatory diagram illustrating an example of a determination result DB. [Figure 9] FIG. 10 is an explanatory diagram illustrating an example of a notification destination DB. [Figure 10] FIG. 10 is an explanatory diagram illustrating an example of a diagnostic model. [Figure 11]FIG. 10 is an explanatory diagram illustrating an example of an analytical model. [Figure 12] 10 is a flowchart illustrating an example of a procedure for interactive processing. [Figure 13] 10 is a flowchart illustrating an example of a procedure for interactive processing. [Figure 14] 10 is a flowchart illustrating an example of a procedure for an abnormality handling process. [Figure 15] 10 is a flowchart illustrating an example of a procedure for remote handling processing. [Figure 16] FIG. 2 is a block diagram illustrating an example of a functional configuration of a management server. DETAILED DESCRIPTION OF THE INVENTION

[0008] (Embodiment 1) An embodiment will be described below with reference to the drawings. Fig. 1 is an explanatory diagram showing an example of the configuration of a monitoring system. The monitoring system 100 includes a management server 1, a monitoring terminal 2, a family terminal 3, and an expert terminal 4. The management server 1, the monitoring terminal 2, the family terminal 3, and the expert terminal 4 are communicably connected via a network N. Although Fig. 1 shows one each of the monitoring terminal 2, family terminal 3, and expert terminal 4, there may be two or more of each.

[0009] The management server 1 is composed of a server computer, a workstation, a PC (Personal Computer), etc. The management server 1 analyzes the images and audio of the person being monitored collected by the monitoring terminal 2 and determines the health condition of the person being monitored. Depending on the determination result, the management server 1 notifies the family terminal 3 and the expert terminal 4. The management server 1 may also be composed of a multi-computer consisting of multiple computers, a virtual machine virtually constructed by software, or a quantum computer. The functions of the management server 1 may also be realized by a cloud service. The person being monitored is assumed to be an elderly person living alone. In the following explanation, the "person being monitored" may also be simply referred to as the "target person."

[0010] The monitoring terminal 2 is composed of a pet robot, a communication robot, a mobile robot, a smart speaker with a camera, etc. The monitoring terminal 2 is installed in the home of the person being monitored. The monitoring terminal 2 engages in voice dialogue with the person being monitored. The monitoring terminal 2 also acquires facial images of the person being monitored.

[0011] The family terminal 3 is configured as a smartphone, tablet computer, notebook PC, etc. The family terminal 3 is used by family members who live separately from the person being monitored. The family terminal 3 communicates with the management server 1 and displays the health status of the person being monitored.

[0012] The expert terminal 4 is configured as a smartphone, tablet computer, notebook PC, etc. The expert terminal 4 is used by experts who are involved with the person being monitored, such as doctors, nurses, care managers, and caregivers. If the management server 1 determines that an abnormality has occurred in the person being monitored, the expert terminal 4 receives a notification of this from the management server 1 and urges the expert to take action. The expert terminal 4 used by a doctor is an example of a doctor terminal.

[0013] 2 is a block diagram showing an example of the hardware configuration of the management server 1. The management server 1 includes a control unit 11, a main memory unit 12, an auxiliary memory unit 13, a communication unit 15, and a reading unit 16. The control unit 11, the main memory unit 12, the auxiliary memory unit 13, the communication unit 15, and the reading unit 16 are connected by a bus B.

[0014] The control unit 11 has one or more arithmetic processing devices such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), etc. The control unit 11 reads and executes a control program 1P (information processing program, program product) stored in the auxiliary storage unit 13, thereby performing various information processing, control processing, etc. related to the management server 1 and realizing various functional units.

[0015] The main memory unit 12 is a static random access memory (SRAM), a dynamic random access memory (DRAM), a flash memory, etc. The main memory unit 12 mainly temporarily stores data required for the control unit 11 to execute arithmetic processing.

[0016] The auxiliary storage unit 13 is a hard disk or an SSD (Solid State Drive), etc., and stores a control program 1P and various DBs (Databases) required for the control unit 11 to execute processing. The auxiliary storage unit 13 stores a subject DB 131, a scenario DB 132, a judgment result DB 133, and a notification destination DB 134. The auxiliary storage unit 13 also stores a diagnostic model 141 and an analysis model 142. The auxiliary storage unit 13 may be a storage device that is separate from the management server 1 and externally connected. The various DBs, etc. stored in the auxiliary storage unit 13 may also be stored in a database server or cloud storage different from the management server 1.

[0017] The communication unit 15 communicates with the monitoring terminal 2, the family terminal 3, the expert terminal 4, etc. via the network N. In addition, the control unit 11 may use the communication unit 15 to download the control program 1P from another computer via the network N, etc., and store it in the auxiliary storage unit 13.

[0018] The reading unit 16 reads the portable storage medium 1a including a CD (Compact Disc)-ROM and a DVD (Digital Versatile Disc)-ROM. The control unit 11 may read the control program 1P from the portable storage medium 1a via the reading unit 16 and store it in the auxiliary storage unit 13. The control unit 11 may also read the control program 1P from the semiconductor memory 1b.

[0019] 3 is a block diagram showing an example of the hardware configuration of the monitoring terminal 2. The monitoring terminal 2 includes a control unit 21, a main memory unit 22, an auxiliary memory unit 23, a communication unit 24, an input unit 25, a display unit 26, an imaging unit 27, a detection unit 28, and an audio input / output unit 29. Each component is connected by a bus B.

[0020] The control unit 21 has one or more arithmetic processing units such as a CPU, an MPU, a GPU, etc. The control unit 21 provides various functions by reading and executing a control program 2P (program, program product) stored in the auxiliary storage unit 23.

[0021] The main memory unit 22 is an SRAM, a DRAM, a flash memory, etc. The main memory unit 22 mainly temporarily stores data necessary for the control unit 21 to execute arithmetic processing.

[0022] The auxiliary storage unit 23 is a hard disk or SSD, etc., and stores various data necessary for the control unit 21 to execute processing. The auxiliary storage unit 23 may be an external storage device connected to the watching terminal 2. The various DBs, etc. stored in the auxiliary storage unit 23 may be stored in a database server or cloud storage.

[0023] The communication unit 24 communicates with the management server 1 via the network N. In addition, the control unit 21 may use the communication unit 24 to download the control program 2P from another computer via the network N or the like, and store it in the auxiliary storage unit 23.

[0024] The input unit 25 is a keyboard and a mouse. The display unit 26 includes a liquid crystal display panel, etc. The display unit 26 displays messages output by the management server 1, etc. The input unit 25 and the display unit 26 may be integrated to form a touch panel display. The input unit 25 and the display unit 26 are not essential components of the monitoring terminal 2. If the monitoring terminal 2 is configured as a smart speaker, the input unit 25 and the display unit 26 may not be provided.

[0025] The imaging unit 27 includes an image sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, and obtains image data such as a facial image of the subject by photoelectrically converting an optical signal input via the image sensor. The imaging unit 27 may be equipped with a zoom lens capable of optical zoom. The imaging unit 27 may also be equipped with a pan / tilt mechanism.

[0026] The detection unit 28 detects the environment of the location where the watching terminal 2 is installed. The detection unit 28 also detects that a target person is present within a predetermined range from the watching terminal 2. The detection unit 28 is equipped with, for example, an illuminance sensor, and detects the brightness of the location where it is installed. The detection unit 28 is equipped with, for example, an infrared sensor, and detects that a person (assuming the target person) is present within a predetermined range. Note that the detection unit 28 is not an essential component of the watching terminal 2, and may not be provided.

[0027] The voice input / output unit 29 includes a microphone and a speaker. The voice input / output unit 29 acquires the voice of the target person through the microphone. The voice input / output unit 29 outputs a voice message to the target person from the speaker.

[0028] 4 is a block diagram showing an example of the hardware configuration of the family terminal 3. The family terminal 3 includes a control unit 31, a main memory unit 32, an auxiliary memory unit 33, a communication unit 34, an input unit 35, and a display unit 36. Each component is connected by a bus B.

[0029] The control unit 31 has one or more arithmetic processing units such as a CPU, an MPU, a GPU, etc. The control unit 31 provides various functions by reading and executing a control program 3P stored in the auxiliary storage unit 33.

[0030] The main memory unit 32 is an SRAM, a DRAM, a flash memory, etc. The main memory unit 32 mainly temporarily stores data required for the control unit 31 to execute arithmetic processing.

[0031] The auxiliary storage unit 33 is a hard disk or an SSD, etc., and stores various data necessary for the control unit 31 to execute processing. The various DBs, etc. stored in the auxiliary storage unit 33 may be stored in a database server or cloud storage.

[0032] The communication unit 34 communicates with the management server 1 via the network N. In addition, the control unit 31 may use the communication unit 34 to download the control program 3P from another computer via the network N or the like and store it in the auxiliary storage unit 33.

[0033] The input unit 35 is a keyboard and a mouse. The display unit 36 ​​includes a liquid crystal display panel or the like. The display unit 36 ​​displays messages output by the management server 1. The input unit 35 and the display unit 36 ​​may be integrated to form a touch panel display.

[0034] The family terminal 3 may include an imaging unit and an audio input / output unit, similar to the watching terminal 2. When the family terminal 3 is configured as a smartphone, the family terminal 3 includes an imaging unit and an audio input / output unit.

[0035] 5 is a block diagram showing an example of the hardware configuration of the expert terminal 4. The expert terminal 4 includes a control unit 41, a main memory unit 42, an auxiliary memory unit 43, a communication unit 44, an input unit 45, a display unit 46, an imaging unit 47, and an audio input / output unit 48. Each component is connected by a bus B. The control unit 41, the main memory unit 42, the auxiliary memory unit 43, the communication unit 44, the input unit 45, the display unit 46, the imaging unit 47, and the audio input / output unit 48 are similar to the control unit 21, the main memory unit 22, the auxiliary memory unit 23, the communication unit 24, the input unit 25, the display unit 26, the imaging unit 27, and the audio input / output unit 29 of the watching terminal 2, respectively, and therefore description thereof will be omitted.

[0036] FIG. 6 is an explanatory diagram showing an example of a subject DB. The subject DB 131 stores information about the subjects. The subject DB 131 includes a subject ID column, a name column, an age column, a gender column, a height column, a weight column, an activity time column, and a health condition column. The subject ID column stores a subject ID that uniquely identifies the subject. The name column stores the name of the subject. The age column stores the age of the subject. The gender column stores the gender of the subject. The height column stores the height of the subject. The weight column stores the weight of the subject. The activity time column stores the activity time of the subject. The activity time indicates the time from waking up to going to bed. The health condition column stores the health condition of the subject. It is desirable to update the values ​​of the height column, weight column, activity time column, and health condition column regularly or irregularly in accordance with fluctuations.

[0037] FIG. 7 is an explanatory diagram showing an example of a scenario DB. The scenario DB 132 stores a scenario for the monitoring terminal 2 to converse with the subject. The scenario DB 132 includes an ID string, an utterance string, a response string, and a transition ID string. The ID string stores an ID that identifies each utterance uttered by the monitoring terminal 2. The utterance string stores the content of the utterance. The response string stores the content of the subject's response to the utterance. Since multiple responses are expected from the subject to each utterance, there is a one-to-many relationship between the utterance and the response. The transition ID string stores the ID of the next utterance that the monitoring terminal 2 will utter after receiving a response. In the transition ID string, "None" indicates that the dialogue is to end. The value indicating the end of the dialogue may be other than "None".

[0038] FIG. 8 is an explanatory diagram showing an example of a judgment result DB. The judgment result DB 133 stores the judgment results of the subject's current condition obtained through dialogue, etc. The judgment result DB 133 includes a subject ID column, a date and time column, a health column, and a dialogue column. The subject ID column stores the subject ID. The date and time column stores the date and time when the judgment was made. The health column stores the judgment results of the subject's health condition. The dialogue column stores the classification results of the dialogue content. In FIG. 8, the judgment results of the health condition are expressed as numbers, but the possible values ​​may be divided into multiple intervals and a word indicating the condition may be assigned to each interval. Furthermore, the classification results of the dialogue content may be expressed as character strings indicating the classification instead of numbers.

[0039] FIG. 9 is an explanatory diagram showing an example of a notification destination DB. The notification destination DB 134 stores information on who to notify when an abnormal situation occurs to a target person. The notification destinations include family members living separately, a family doctor, and a responsible care manager. The notification destination DB 134 includes a target person ID column, a number column, a name column, a relationship column, an email address column, and a phone number column. The target person ID column stores the target person ID. The number column stores the order of notifications, priority, etc. The name column stores the names of people who will be notification destinations. The relationship column stores the relationship between the notification destination and the target person. The email address column stores the email address of the notification destination. The phone number column stores the phone number of the notification destination.

[0040] FIG. 10 is an explanatory diagram showing an example of a diagnostic model. When facial image data of a subject is input, the diagnostic model 141 (first learning model) outputs an evaluation score indicating the subject's health level. The evaluation score is, for example, a value between 0 and 1, with the closer to 1 the healthier the subject and the closer to 0 the unhealthier (abnormal) the subject. In the following explanation, the diagnostic model 141 is described as a CNN (Convolutional Neural Network), but the diagnostic model 141 is not limited to a CNN and may be a model constructed with other learning algorithms, such as a neural network other than CNN, a Bayesian network, or a decision tree. The evaluation score is an example of first abnormality information.

[0041] The management server 1 inputs facial images included in the training data into the input layer, undergoes arithmetic processing in the intermediate layer, and obtains evaluation scores from the output layer. The management server 1 compares the evaluation scores output from the output layer with the labels included in the training data, i.e., the correct values, and optimizes the parameters used in the arithmetic processing in the intermediate layer so that the output values ​​of the output nodes approach the correct values. These parameters include, for example, the weights (coupling coefficients) between neurons and the coefficients of the activation functions used in each neuron. There are no particular limitations on the method for optimizing the parameters, but for example, the management server 1 optimizes various parameters using the backpropagation method. The management server 1 performs the above learning process using all of the training data and generates a trained diagnostic model 141.

[0042] FIG. 11 is an explanatory diagram showing an example of an analysis model. When dialogue data between a subject and the monitoring terminal 2 is input, the analysis model 142 (second learning model) determines to which of a plurality of categories the content of the dialogue falls and outputs the determined classification. The analysis model 142 includes a voice recognition model 1421 and a classification model 1422. The voice recognition model 1421 is configured, for example, by a GMM-HMM, a DNN-HMM, or an end-to-end model. GMM is a Gaussian Mixture Model: a mixed normal distribution model. HMM is a Hidden Markov Model: a hidden Markov model. DNN is a deep neural network. The classification model 1422 is configured, for example, by an RNN (recurrent neural network), an LSTM (long short-term memory), a naive Bayes classifier, or the like.

[0043] When voice data is input, the voice recognition model 1421 performs voice recognition and outputs text. When text is input, the classification model 1422 classifies the content of the text. The classification is, for example, normal, slurred speech, incomplete conversation, angry, forgetful, etc. The classification model 1422 outputs a classification value (classification information) indicating the classification. The classification value is an example of second abnormality information. Note that the classification may also include olfactory abnormalities such as not being able to smell. To detect olfactory abnormalities, the monitoring terminal 2 asks questions in the dialogue, such as "Can you smell coffee?" or "Can you smell miso soup?" Alternatively, the monitoring terminal 2 may be equipped with an aroma sprayer that periodically emits aromas and asks whether the user can smell them.

[0044] The training data for the classification model 1422 is prepared by associating dialogue text with content classification values. The dialogue text that constitutes the training data is input to the classification model 1422, and parameters are adjusted so that the classification value output by the classification model 1422 matches the classification value (= correct value) that constitutes the training data.

[0045] Next, information processing performed in the monitoring system 100 will be described. FIGS. 12 and 13 are flowcharts showing an example of the procedure of the dialogue processing. The control unit 21 of the monitoring terminal 2 determines whether or not a target person has been detected (step S1). The target person is detected, for example, by the detection unit 28. If the control unit 21 determines that the target person has not been detected (NO in step S1), it repeats step S1. If the control unit 21 determines that the target person has been detected (YES in step S1), it determines whether or not to call out to the target person (step S2). For example, if the control unit 21 detects the target person for the first time in a certain day, it determines that the target person will be called out to. If it is within a time period that has been set in advance for dialogue, the control unit 21 determines that the target person will be called out to. However, if a predetermined time (e.g., three hours) has not passed since the last time the control unit 21 called out to the target person, the control unit 21 may determine that the target person will not be called out to. If the control unit 21 determines that the target person will not be called out to (NO in step S2), it ends the dialogue processing. If the control unit 21 determines that a call should be made (YES in step S2), it selects a message to be output as a call (step S3). The selected message may be, for example, a greeting, and an appropriate message is selected depending on the time of day and the season. The message is stored in advance in the auxiliary storage unit 23. The control unit 21 outputs the message as audio from the audio input / output unit 29 (step S4). The control unit 21 captures an image of the target person's face using the imaging unit 27 (step S5). At this time, if the control unit 21 determines that it is difficult to capture an image of the target person's face, for example, because the target person is far from the monitoring terminal 2, it outputs a message from the audio input / output unit 29 encouraging the target person to approach the monitoring terminal 2 and turn their face toward it. This message is also stored in advance in the auxiliary storage unit 23. The control unit 21 transmits an image of the target person's face to the management server 1 (step S6). The control unit 11 of the management server 1 receives the image of the face (step S7). The control unit 11 inputs the image of the face to the diagnostic model 141 to obtain an evaluation score, and stores the image of the face and the evaluation score (step S8). The control unit 11 selects a message based on the evaluation score (step S9). The control unit 11 transmits the message to the watching terminal 2 (step S10). The control unit 21 of the watching terminal 2 receives the message (step S11). The control unit 21 outputs the message as voice from the voice input / output unit 29 (step S12).The control unit 21 collects the target person's response to the message (step S13). For example, the voice input / output unit 29 collects the target person's response voice. The control unit 21 determines whether the target person has responded (step S14). For example, the control unit 21 analyzes the waveform of the collected response voice to determine whether the target person has responded. If the control unit 21 determines that the target person has responded (YES in step S14), it transmits the collected response voice to the management server 1 (step S15). The control unit 11 of the management server 1 receives the response voice (step S16). The control unit 11 inputs the response voice into the analysis model 142 to analyze the content of the dialogue, and stores the resulting classification value (step S17). The classification value is stored in the judgment result DB 133. The control unit 11 determines whether the target person is normal based on the classification value (step S18). If the control unit 11 determines that the target person is normal (YES in step S18), it selects the next message to be output (step S19). The control unit 11 transmits the message to the watching terminal 2 (step S20). The control unit 21 of the watching terminal 2 receives the message (step S21). The control unit 21 outputs the message as voice from the voice input / output unit 29 (step S22). The control unit 21 determines whether to end the process (step S23). If the output message is a message to end the dialogue, the control unit 21 determines to end the dialogue. If the control unit 21 determines not to end the process (NO in step S23), the control unit 21 returns the process to step S13. If the control unit 21 determines to end the process (YES in step S23), the control unit 21 ends the process. If the control unit 11 of the management server 1 determines that the target person is not normal (NO in step S18), the control unit 11 performs an abnormality handling process (step S26) and ends the process. The abnormality handling process will be described later. If the control unit 21 of the watching terminal 2 determines that the target person has not responded (NO in step S14), the control unit 21 transmits a message indicating that there has been no response to the management server 1 (step S24). The control unit 11 of the management server 1 receives the message that there is no response (step S25). The control unit 11 performs an abnormality handling process (step S26) and ends the process. It is assumed that the dialogue process is repeated several times a day. If the imaging unit 27 has a zoom lens or a pan-tilt mechanism, it may capture images other than facial images.For example, a camera can follow a subject moving around a room and capture their body movements as they move. It can also capture basic daily activities such as eating and changing clothes. By capturing more images other than facial images, the amount of data available for analyzing the subject's condition increases, making it possible to improve the accuracy of detecting abnormalities and increase the number of detectable abnormalities.

[0046] FIG. 14 is a flowchart showing an example of the procedure for handling an abnormality. The control unit 11 of the management server 1 transmits an operation instruction to the monitoring terminal 2 (step S41). For example, if there is a possibility that an abnormality has occurred in the target person, the control unit 11 instructs the monitoring terminal 2 to call out to the target person again. For example, the control unit 11 instructs the monitoring terminal 2 to output messages such as "Are you OK?", "Can you hear me?", and "Can you speak?" as the content of the message. Furthermore, to confirm whether the reason the target person did not respond to the message is due to an abnormality, the control unit 11 instructs the monitoring terminal 2 to perform an analysis by sensing. The control unit 21 of the monitoring terminal 2 receives the instruction (step S42). The control unit 21 determines whether the instruction is a message (step S43). If the control unit 21 determines that the instruction is not a message (NO in step S43), it collects environmental information (step S44). The control unit 21 performs sensing using the detection unit 28 and collects the brightness of the room and the amount of infrared light using the infrared sensor. The control unit 21 performs an analysis based on the collected information (step S45). Based on the analysis result, the control unit 21 determines whether the subject is normal or not (step S46). For example, if the time period during which the subject is expected to be sleeping is dark, the infrared center predicts that the subject is not present within a predetermined range, and the installation location is set to a location other than the bedroom, the control unit 21 determines that the subject is not abnormal. Also, if the time period during which the subject is expected to be sleeping is bright, the infrared center predicts that the subject is present within a predetermined range, and the installation location is set to the living room, the control unit 21 determines that the subject is not normal. The installation location of the monitoring terminal 2 is stored in advance in the auxiliary storage unit 13. If the control unit 21 determines that the subject is not normal (NO in step S46), the control unit 21 changes the mode of the monitoring terminal 2 (step S47). For example, the control unit 21 transitions from a normal mode to an abnormal mode. For example, in the abnormal mode, video calls are enabled by external control. The control unit 21 notifies the management server 1 that the monitoring terminal 2 is determined to be abnormal (step S48). After receiving the notification, the control unit 11 of the management server 1 notifies the family terminal 3 or the expert terminal 4 (step S59). The notification may include details of the detected abnormality and an image taken by the watching terminal 2. The notification is an example of abnormality information.The control unit 11 determines the notification destination by referring to the notification destination DB 134. The control unit 11 returns the process to the caller. If the control unit 21 determines that the target person is normal (YES in step S46), it transmits a message indicating that the target person is normal to the management server 1 (step S57). The control unit 11 of the management server 1 receives the message indicating that the target person is normal (step S58). The control unit 11 returns the process to the caller. If the control unit 21 determines that the instruction is a call (YES in step S43), it outputs a call message and then collects the target person's response to the message (step S49). For example, the voice input / output unit 29 collects the target person's response voice. The control unit 21 determines whether or not there is a response from the target person (step S50). If the control unit 21 determines that there is no response from the target person (NO in step S50), it moves the process to step S44.

[0047] When the control unit 21 determines that there is a response from the subject (YES in step S50), it transmits the collected response voice to the management server 1 (step S51). The control unit 11 of the management server 1 receives the response voice (step S52). The control unit 11 inputs the response voice into the analysis model 142 to analyze the dialogue content, and stores the resulting classification value (step S53). The control unit 11 determines whether the subject is normal or not based on the classification value (step S54). When the control unit 11 determines that the subject is normal (YES in step S54), the control unit 11 returns the process to the caller. When the control unit 11 determines that the subject is not normal (NO in step S54), it transmits a mode change instruction to the watching terminal 2 (step S55). The control unit 21 of the watching terminal 2 receives the mode change instruction (step S56). The control unit 21 moves the process to step S47.

[0048] FIG. 15 is a flowchart showing an example of the procedure for the remote response processing. The remote response processing is triggered when the expert terminal 4 receives a notification in step 59 of the abnormality handling processing shown in FIG. 14. The control unit 41 of the expert terminal 4 receives the notification from the management server 1 and displays it on the display unit 46 (step S71). After confirming the notification, the expert instructs the expert terminal 4 to connect to the monitoring terminal 2. The control unit 41 sends a connection request to the monitoring terminal 2 (step S72). The connection request may include the expert's ID and password. The control unit 21 of the monitoring terminal 2 receives the connection request (step S73). The control unit 21 determines whether to permit the connection (step S74). For example, if the mode is abnormal mode, the control unit 21 determines that the connection is permitted. If the connection request includes an ID and password, the control unit 21 performs authentication and determines whether to permit the connection. The combination of ID and password that permits the connection is stored in advance in the auxiliary storage unit 23. If the control unit 21 determines not to permit the connection (NO in step S74), it sends a connection denial to the expert terminal 4 (step S75). The control unit 41 of the expert terminal 4 receives the connection denial (step S76). The remote response process ends. If the control unit 21 determines to permit the connection (YES in step S74), it sends a connection permission to the expert terminal 4 (step S77). The control unit 41 of the expert terminal 4 receives the connection permission (step S78). The expert terminal 4 and the watching terminal 2 send and receive video and audio (steps S79, S80) and provide a video call. The expert grasps the condition of the subject through the video call. The control unit 41 of the expert terminal 4 determines whether to end (step S81). If the control unit 41 determines not to end (NO in step S81), the process returns to step S79. If the control unit 41 determines to end (YES in step S81), it sends an end command to the watching terminal 2 (step S82). The control unit 41 determines that the video call has ended, for example, when the expert instructs the expert to end the video call. The control unit 21 of the watching terminal 2 receives the end command (step S83). The control unit 21 disconnects communication with the expert terminal 4 (step S84). The control unit 41 of the expert terminal 4 detects the disconnection of communication (step S85). The control unit 41 transmits a message to the management server 1 that the video call has ended (step S86).At this time, it is desirable to transmit the subject's condition and diagnosis results created by the expert. The control unit 11 of the management server 1 receives the notification of completion (step S87). The control unit 11 stores the fact that the video call has been made as history in the auxiliary storage unit 13 (step S88). The control unit 11 transmits a notification of receipt to the expert terminal 4 (step S89). The control unit 41 of the expert terminal 4 receives the notification of receipt (step S90) and terminates the processing. Here, we have described a case where the expert terminal 4 executes the remote response process, but the remote response process performed when the family terminal 3 receives a notification is similar. When the doctor uses the expert terminal 4 to video call with the subject, online consultation is also possible.

[0049] In the above-described embodiment, monitoring is possible by installing a monitoring terminal 2 in the home of the person being monitored. Abnormalities in the person being monitored can be detected based on the person's facial expressions and responses during conversations. If an abnormality is detected, a notification is sent to the family terminal 3 or the expert terminal 4. The family member or expert who receives the notification can check the condition of the person being monitored via a video call using the family terminal 3 or the expert terminal 4 and the monitoring terminal 2. If necessary, emergency measures, such as requesting an ambulance, can be taken. Furthermore, because the person being monitored can interact with the monitoring terminal 2, a soothing effect similar to interacting with a pet can be expected. Even if the person being monitored lives alone, this can help reduce feelings of loneliness and prevent dementia and mental illness. Furthermore, by configuring the monitoring terminal 2 with a pet robot or other device with a tracking function, it is possible to capture the movements of the person being monitored as they move. Additionally, during conversations, the monitoring terminal 2 can react by combining speech and movement, providing a more soothing effect to the person being monitored compared to voice-only communication.

[0050] 16 is a block diagram showing an example of the functional configuration of the management server 1. The management server 1 includes a first data acquisition unit 11a, a first abnormality acquisition unit 11b, a second data acquisition unit 11c, a second abnormality acquisition unit 11d, and an abnormality output unit 11e. When the control unit 11 executes the control program 1P, the management server 1 operates as follows.

[0051] The first data acquisition unit 11a acquires facial image data of the user acquired through the interactive robot. The first abnormality acquisition unit 11b acquires first abnormality information by inputting the acquired facial image data into a first learning model that has been trained to output a first abnormal state related to the user's abnormality when the facial image data is input. The second data acquisition unit 11c acquires dialogue data exchanged with the user through the interactive robot. The second abnormality acquisition unit 11d acquires second abnormality information by inputting the acquired dialogue data into a second learning model that has been trained to output second abnormality information related to the user's abnormality when the dialogue data is input. The abnormality output unit 11e outputs the abnormality information to another terminal device when both the first abnormality information and the second abnormality information are output within a predetermined time.

[0052] The technical features (constituent elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed. The embodiments disclosed herein are to be considered as illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0053] 100 Monitoring System 1 Management Server 11 Control section 11a First data acquisition unit 11b 1st abnormality acquisition part 11c Second data acquisition unit 11d 2nd abnormality acquisition part 11e Abnormal output section 12 Main memory 13 Auxiliary storage 131 Target Person DB 132 Scenario DB 133 Judgment result DB 134 Notification destination DB 141 Diagnostic Model 142 Analytical Model 1421 Speech Recognition Model 1422 Classification Model 15 Communications Department 16 Reading unit 1P control program 1a Portable storage media 1b semiconductor memory 2. Monitoring device 2P control program 3 Family devices 3P control program 4 Expert terminal B Bus N Network

Claims

1. Acquire face image data of the user acquired through the interactive robot; inputting the acquired facial image data into a first learning model that has been trained to output first abnormality information regarding an abnormality of a user when the facial image data is input, and acquiring first abnormality information; making a speech in response to the first abnormality information through the interactive robot; Acquire conversation data between the user and the interactive robot through the interactive robot; inputting the acquired dialogue data into a second learning model that has been trained to output second anomaly information related to an anomaly of the user when dialogue data is input, and acquiring second anomaly information; anomaly information is output to a terminal device in response to the second anomaly information, The second abnormality information has classification information including slurred speech, inability to hold a conversation, anger, or forgetfulness, If the second abnormality information is classified as slurred speech or incomplete conversation, the interactive robot calls out to the user again. An information processing method characterized in that the processing is executed by a computer.

2. When the interactive robot detects the user using a sensor, if the interactive robot does not receive a response from the user when it speaks to the user again, it executes a notification process.

2. The information processing method according to claim 1,

3. When the interactive robot detects the user through a sensor, the interactive robot outputs a greeting voice; acquiring face image data of the user responding to the output voice via an imaging unit of the interactive robot; The acquired face image data is input to the first learning model.

3. The information processing method according to claim 1 or 2.

4. When the first learning model outputs the first abnormality information, the dialogue data between the first learning model and the user in response to the first abnormality information is acquired.

4. The information processing method according to claim 1, wherein the first and second inputs are input to the first and second inputs.

5. The classification information includes olfactory abnormality, When the second abnormality information is classified as an olfactory abnormality, the abnormality information is output to the terminal device.

5. The information processing method according to claim 1, wherein the first and second inputs are input to the first and second inputs.

6. outputting the abnormality information to a doctor's terminal; The doctor terminal and the interactive robot are communicably connected to start an online consultation.

6. The information processing method according to claim 1, wherein:

7. Acquire face image data of the user acquired through the interactive robot; inputting the acquired facial image data into a first learning model that has been trained to output first abnormality information regarding an abnormality of a user when the facial image data is input, and acquiring first abnormality information; making a speech in response to the first abnormality information through the interactive robot; Acquire conversation data between the user and the interactive robot through the interactive robot; inputting the acquired dialogue data into a second learning model that has been trained to output second anomaly information related to an anomaly of the user when dialogue data is input, and acquiring second anomaly information; anomaly information is output to a terminal device in response to the second anomaly information, The second abnormality information has classification information including slurred speech, inability to hold a conversation, anger, or forgetfulness, If the second abnormality information is classified as slurred speech or incomplete conversation, the interactive robot calls out to the user again. An information processing program that causes a computer to execute a process.

8. An information processing device having a control unit, The control unit Acquire face image data of the user acquired through the interactive robot; inputting the acquired facial image data into a first learning model that has been trained to output first abnormality information regarding an abnormality of a user when the facial image data is input, and acquiring first abnormality information; making a speech in response to the first abnormality information through the interactive robot; Acquire conversation data between the user and the interactive robot through the interactive robot; inputting the acquired dialogue data into a second learning model that has been trained to output second anomaly information related to an anomaly of the user when dialogue data is input, and acquiring second anomaly information; anomaly information is output to a terminal device in response to the second anomaly information, The second abnormality information has classification information including slurred speech, inability to hold a conversation, anger, or forgetfulness, The control unit If the second abnormality information is classified as slurred speech or incomplete conversation, the interactive robot calls out to the user again.

1. An information processing device comprising:

Images