PROGRAM, INFORMATION PROCESSING APPARATUS AND INFORMATION PROCESSING METHOD

Abstract

This program causes a computer to perform the processes of: outputting a first question by voice; accepting an answer to the first question from a subject; outputting a second question in text; accepting an answer to the second question from the subject; determining whether the answers to the first and second questions are correct or not; and estimating the possibility of brain dysfunction of the subject on the basis of whether the answers to the first and second questions are correct or not.

Description

[Technical field]

[0001] The present invention relates to a program, an information processing device, and an information processing method. [Background technology]

[0002] There are technologies that support the diagnosis of brain dysfunction such as dementia. For example, Patent Literature 1 discloses a dementia diagnosis device that converts a speech voice of a subject responding to a given question into text data, calculates an edit distance with a comparison text data, and determines whether or not the subject may have developed dementia. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-483 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the invention of Patent Document 1 does not estimate the possibility of brain dysfunction by taking into consideration the differences in symptoms of various brain dysfunctions such as dementia and aphasia.

[0005] In one aspect, an object of the present invention is to provide a program or the like that can suitably estimate the possibility of brain dysfunction. [Means for solving the problem]

[0006] A program according to one aspect outputs a first question by voice, receives an answer to the first question from a subject, outputs a second question by text, receives an answer to the second question from the subject, determines whether the answers to the first and second questions are correct, and estimates a possibility of brain dysfunction of the subject based on the correctness of the answers to the first and second questions. The program causes a computer to execute a process, and when the answer to the first question is incorrect and the answer to the second question is correct, it is inferred that there is a possibility of aphasia, and when the answers to both the first and second questions are incorrect, it is inferred that there is a possibility of dementia.The processing is executed by a computer. Effect of the Invention

[0007] In one aspect, the possibility of brain dysfunction can be suitably estimated. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of the configuration of a dialogue system. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of a server. [Diagram 3] FIG. 2 is a block diagram showing a configuration example of a mobile terminal. [Figure 4] FIG. 2 is a block diagram showing a configuration example of a speaker terminal. [Diagram 5] FIG. 1 is an explanatory diagram showing an overview of a first embodiment. [Figure 6] FIG. 13 is an explanatory diagram showing an example of a message display screen. [Figure 7] 13 is a flowchart showing a processing procedure executed by a server. [Figure 8] FIG. 11 is a block diagram showing an example of the configuration of a server according to the second embodiment. [Figure 9] FIG. 13 is an explanatory diagram showing an example of a record layout of an answer history DB. [Figure 10A] FIG. 4 is an explanatory diagram showing an example of a display screen of a speaker terminal. [Figure 10B] FIG. 4 is an explanatory diagram showing an example of a display screen of a speaker terminal. [Figure 11A] FIG. 11 is an explanatory diagram showing an example of a message display screen according to the second embodiment. [Figure 11B] FIG. 11 is an explanatory diagram showing an example of a message display screen according to the second embodiment. [Figure 12] FIG. 13 is an explanatory diagram showing another example of the chat screen when the estimation result is displayed. [Figure 13] FIG. 13 is an explanatory diagram showing an example of a history screen. [Figure 14] 11 is a flowchart illustrating an example of a processing procedure executed by a server according to the second embodiment. [Figure 15] 11 is a flowchart illustrating an example of a processing procedure executed by a server according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present invention will now be described in detail with reference to the drawings showing embodiments thereof. (Embodiment 1) Fig. 1 is an explanatory diagram showing a configuration example of a dialogue system. In this embodiment, a dialogue system that judges the presence or absence of an abnormality, preferably a brain dysfunction, of a subject based on a speech voice input by the subject will be described. The dialogue system includes an information processing device 1, a mobile terminal 2, and a speaker terminal 3. Each device is communicatively connected to a network N such as the Internet.

[0010] The information processing device 1 is an information processing device capable of various information processing and information transmission and reception, such as a server computer, a personal computer, etc. In this embodiment, the information processing device 1 is a server computer, and will be referred to as server 1 below for simplicity. The server 1 determines whether or not there is a possibility of brain dysfunction from a speech voice input by the subject. Specifically, as described later, the server 1 detects an abnormality suspected to be a brain dysfunction from a speech voice input by the subject as a message to a chat group in which multiple users including the subject participate, or a speech voice input by the subject to a chatbot system that operates based on a predetermined dialogue engine.

[0011] The target brain dysfunction is not particularly limited, but examples include dementia, aphasia, etc. The server 1 detects abnormal speech (unclear words, slip of the tongue, etc.) caused by dementia, aphasia, etc., and presents the abnormal part to other users (for example, users related to the subject, such as the subject's family members and medical professionals treating the subject).

[0012] The mobile terminal 2 is an information processing terminal used by each user including the subject, and is, for example, a smartphone, a tablet terminal, etc. Although only one mobile terminal 2 is illustrated in FIG. 1, it is assumed that the mobile terminals 2 of the subject and each of the other users are connected to the server 1. The server 1 acquires from the mobile terminal 2 a speech voice input by the subject as a message to a chat group or the like, and converts it into text. The server 1 then detects an abnormality from the converted text.

[0013] The speaker terminal 3 is a voice input / output terminal installed in the subject's home or the like, and is a so-called smart speaker. The speaker terminal 3 is not limited to what is called a smart speaker, and may be any terminal that has a voice input / output function and an image display function. The speaker terminal 3 is not limited to being installed in the subject's home, and may be installed in a facility other than the home (e.g., a care facility). The speaker terminal 3 functions as a terminal device of a chatbot system and engages in a dialogue with the subject. As described later, the server 1 may obtain the subject's speech from the speaker terminal 3 and detect an abnormality.

[0014] In this embodiment, the terminal devices that cooperate with the server 1 include the mobile terminal 2 and the speaker terminal 3, but other types of terminal devices (for example, robot-type devices) may also be used. The terminal device may be a local terminal equipped with a voice input / output function, an image display function, etc., and its type is not particularly limited.

[0015] 2 is a block diagram showing an example of the configuration of the server 1. The server 1 includes a control unit 11, a main storage unit 12, a communication unit 13, and an auxiliary storage unit . The control unit 11 has one or more arithmetic processing devices such as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), etc., and performs various information processing, control processing, etc. by reading and executing a program P1 stored in the auxiliary storage unit 14. The main storage unit 12 is a temporary storage area such as a static random access memory (SRAM), a dynamic random access memory (DRAM), a flash memory, etc., and temporarily stores data required for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing processing related to communication, and transmits and receives information to and from the outside. The auxiliary storage unit 14 is a non-volatile storage area such as a large-capacity memory, a hard disk, etc., and stores the program P1 and other data required for the control unit 11 to execute processing.

[0016] The auxiliary storage unit 14 may be an external storage device connected to the server 1. The server 1 may be a multi-computer consisting of a plurality of computers, or may be a virtual machine virtually constructed by software.

[0017] In the present embodiment, the server 1 is not limited to the above configuration, and may include, for example, an input unit for accepting operation input, a display unit for displaying images, etc. The server 1 may also include a reading unit for reading a portable storage medium 1a such as a CD (Compact Disk)-ROM or a DVD (Digital Versatile Disc)-ROM, and may read and execute the program P1 from the portable storage medium 1a. Alternatively, the server 1 may read the program P1 from a semiconductor memory 1b.

[0018] 3 is a block diagram showing an example of the configuration of the portable terminal 2. The portable terminal 2 includes a control unit 21, a main memory unit 22, a communication unit 23, a display unit 24, an input unit 25, an audio output unit 26, an audio input unit 27, an imaging unit 28, and an auxiliary memory unit 29. The control unit 21 has one or more arithmetic processing units such as a CPU, an MPU, etc., and performs various information processing, control processing, etc. by reading and executing a program P2 stored in the auxiliary storage unit 29. The main storage unit 22 is a temporary storage area such as a RAM, and temporarily stores data necessary for the control unit 21 to execute arithmetic processing. The communication unit 23 is a communication module for performing processing related to communication, and transmits and receives information to and from the outside. The display unit 24 is a display screen such as a liquid crystal display, and displays images.

[0019] The input unit 25 is an operation interface such as a touch panel, and accepts operation input from the user. The audio output unit 26 is a speaker, and outputs audio. The audio input unit 27 is a microphone, and accepts audio input from the user. The imaging unit 28 is a camera equipped with an imaging element such as a CMOS (Complementary MOS), and captures images. The auxiliary storage unit 29 is a non-volatile storage area such as a hard disk or large-capacity memory, and stores the program P2 and other data necessary for the control unit 21 to execute processing.

[0020] The mobile terminal 2 may include a reading unit for reading a portable storage medium 2a such as a CD-ROM, and may read and execute the program P2 from the portable storage medium 2a. Alternatively, the mobile terminal 2 may read the program P2 from the semiconductor memory 2b.

[0021] 4 is a block diagram showing an example of the configuration of the speaker terminal 3. The speaker terminal 3 includes a control unit 31, a main memory unit 32, a communication unit 33, a display unit 34, an input unit 35, an audio output unit 36, an audio input unit 37, an imaging unit 38, and an auxiliary memory unit 39. The control unit 31 has one or more arithmetic processing units such as a CPU, an MPU, etc., and performs various information processing, control processing, etc. by reading and executing a program P3 stored in the auxiliary storage unit 39. The main storage unit 32 is a temporary storage area such as a RAM, and temporarily stores data necessary for the control unit 31 to execute arithmetic processing. The communication unit 33 is a communication module for performing processing related to communication, and transmits and receives information to and from the outside. The display unit 34 is a display screen such as a liquid crystal display, and displays images.

[0022] The input unit 35 is an operation interface such as a touch panel, and receives operation input from the user. The audio output unit 36 ​​is a speaker, and outputs audio. The audio input unit 37 is a microphone, and receives audio input from the user. The imaging unit 38 is a camera equipped with an imaging element such as a CMOS, and captures images. The auxiliary storage unit 39 is a non-volatile storage area such as a hard disk or large-capacity memory, and stores the program P3 and other data necessary for the control unit 31 to execute processing.

[0023] The speaker terminal 3 may include a reading unit for reading the portable storage medium 2a such as a CD-ROM, and may read and execute the program P3 from the portable storage medium 3a. Alternatively, the speaker terminal 3 may read the program P3 from the semiconductor memory 3b.

[0024] Fig. 5 is an explanatory diagram showing an overview of the first embodiment. The overview of the present embodiment will be described with reference to Fig. 5.

[0025] As described above, the server 1 judges whether the subject's condition is abnormal or not based on messages to a chat group in which multiple users including the subject participate. Fig. 5 shows an image of a conversation in a chat group. The right side of Fig. 5 shows messages from the subject, and the left side shows messages from other users (e.g., family members) and the system (server 1). The subject can also input messages by voice using text input or a voice recognition function.

[0026] The server 1 converts the speech input by the subject into text and detects abnormalities from the converted text. The example in Fig. 5 illustrates a case where the server 1 converts the subject's speech into text "I went to the park today" in response to a message from another user, "Where did you go today?" In this case, the server 1 detects the abnormal part "park" from the text.

[0027] A specific method for detecting an abnormal portion is not particularly limited, but for example, the server 1 performs morphological analysis to divide the text into a plurality of character strings (words), and compares each character string with each word in the word dictionary by referring to a word dictionary (database not shown) that stores a large number of words. Note that in this embodiment, the text is divided into words, but the text may be divided into units longer than words (e.g., phrases) or units shorter than words. The server 1 detects character strings that are not stored in the word dictionary as abnormal portions. Note that, for example, the server 1 may prescribe words that occur infrequently (e.g., words other than common words) in the word dictionary, and detect words that occur infrequently as abnormal portions.

[0028] The server 1 may also store text related to the speech input by the subject and detect abnormal parts based on past text. For example, the server 1 stores (registers) character strings obtained by dividing text by morphological analysis as new words in a word dictionary, and builds a word dictionary for each subject. Then, when the server 1 receives speech input from a subject and converts it into text, it detects abnormal parts by referring to the word dictionary corresponding to the subject. This makes it possible to improve the accuracy of detecting abnormal parts by taking into account the tendency of the subject's speech.

[0029] In the above, the abnormal part is detected using a word dictionary, but the method of detecting the abnormal part is not limited to this. For example, the server 1 may also detect the abnormal part by performing syntactic analysis, semantic analysis, etc. of the text. The detection method is not limited to the rule-based method. For example, the server 1 may prepare a machine learning model (e.g., a neural network) that has been trained to detect the abnormal part when text is input, and input the text converted from the spoken voice into the model to detect the abnormal part. In this way, the method of detecting the abnormal part is not particularly limited.

[0030] When an abnormality is detected, the server 1 generates a question asking about the abnormality and outputs it to the mobile terminal 2 of the target person. The question is preferably in any one of the 6W3H (Who, Whom, When, Where, What, Why, How, How many, How Much) formats. The server 1 generates the question by applying a character string corresponding to the abnormality to a question template in any one of the 6W3H formats. The server 1 outputs the question as a message in the chat group and displays it on the mobile terminal 2. For example, the server 1 may convert the question into voice and output it to the mobile terminal 2.

[0031] The server 1 receives an answer to the question from the subject. The answer is received by voice, just like when a message is input. The server 1 converts the input voice answer into text and judges whether the subject's condition is abnormal, specifically, whether there is a possibility of brain dysfunction.

[0032] FIG. 5 illustrates patterns 1 to 3 as response examples. In the case of pattern 1, the correct word "park" is recognized from the voice, so the server 1 determines that the subject's condition is normal. On the other hand, in the case of pattern 2, the abnormal part "touen" is recognized again from the voice, so the server 1 determines that the subject's condition is abnormal. Also, in the case of pattern 3, although the character string "touen" is included, a correct sentence is recognized from the voice in terms of the context before and after, so the server 1 determines that the subject's condition is normal.

[0033] In this way, the server 1 determines the state of the subject from the answer to the question. In this case, the server 1 may determine the state of the subject from data other than voice. For example, the mobile terminal 2 captures an image of the subject when inputting the answer to the above question, and the server 1 determines the state of the subject from the captured image (e.g., a moving image).

[0034] Specifically, the server 1 recognizes the subject's face from the image and judges the subject's condition from the asymmetry of the face. For example, when brain dysfunction occurs due to cerebral infarction, cerebral hemorrhage, etc., asymmetrical states and movements of the face are observed, such as different movements of the left and right sides of the face, one side drooping, and distortion on one side. The server 1 divides the face area in the image into two areas, left and right, and specifies the state (coordinates of each feature point such as the eyes and the corners of the mouth) and movement (movement of the feature points) of each area to judge whether the state and / or movement of the left and right sides of the face are asymmetrical. If it is judged to be asymmetrical, the server 1 judges that the subject's condition is abnormal.

[0035] In the above, the subject is imaged when the answer to the question is input, but the subject may be imaged when the initial message (the message in which the abnormal part is detected) is input by voice, and the left-right asymmetry of the face may be determined from the image at the time of input of the message (voice). In other words, the time when the image is captured is not limited to the time when the answer to the question is input, but may be the time when the message is input by voice.

[0036] Furthermore, in this embodiment, image and sound are combined to determine whether there is something wrong with the subject, but it is also possible to determine whether there is something wrong with the subject from sound (text) alone.

[0037] As described above, the server 1 detects abnormal areas suspected to be brain dysfunction from the text of the voice input by the subject as a message to the chat group, asks for the abnormal areas again, and judges the subject's condition from the voice response to the question and / or the image at the time the response was input.

[0038] In the above, the case where the subject person has a group chat with other users has been described as an example, but the present embodiment is not limited to this. For example, the server 1 may detect an abnormal part from an input voice when the subject person talks to a chatbot realized based on a predetermined dialogue engine.

[0039] The chatbot may input and output voice using a mobile terminal 2 such as a smartphone, or may input and output voice using a speaker terminal 3 (smart speaker) installed in the target person's home, etc. Here, the speaker terminal 3 is described as accepting voice input from the target person and outputting a response voice.

[0040] The speaker terminal 3 accepts various voice inputs, such as daily greetings (such as "Good morning"), requests to output information (such as today's weather, schedules, etc.), and requests to operate devices (such as home appliances). The speaker terminal 3 performs various information processing on these input voices (for example, when a greeting is input, it outputs a greeting response voice, and when a voice for operating a device is input, it outputs a device operation signal, etc.). The server 1 acquires the voice input to the speaker terminal 3, converts it into text, and detects any abnormalities. The method of detecting the abnormalities is the same as described above.

[0041] Furthermore, the server 1 may call out to the subject from the system side via the speaker terminal 3 and accept voice input. For example, the server 1 outputs a voice asking a specific question (such as "What's the weather today?") to the speaker terminal 3 at regular intervals and accepts a voice input of the answer from the subject. This makes it possible to encourage the subject, for example, if the subject is an elderly person living alone, to talk to the subject periodically and at the same time to detect abnormalities in the subject from the content of the conversation.

[0042] In this way, the server 1 may detect an abnormal part from the dialogue voice with the chatbot. That is, the voice to be used for detecting an abnormal part is not limited to a message to another user, and may be any utterance voice.

[0043] Fig. 6 is an explanatory diagram showing an example of a message display screen. Fig. 6 illustrates a state in which a message (text) indicating that an abnormality has been detected is displayed in a group chat. Specifically, Fig. 6 illustrates a chat screen displayed on the mobile terminal 2 of another user (family member, etc.) who exchanges messages with the subject. In Fig. 6, messages from the subject and the system are shown on the left, and messages from other users are shown on the right.

[0044] When the server 1 detects an abnormality in the message of the subject, it displays the character string corresponding to the abnormality in a display mode different from other character strings. For example, the server 1 changes the display color of the character string corresponding to the abnormality and changes (highlights) the background color of the abnormality. For convenience of illustration in FIG. 6, the change in the display color of the character string is shown in bold, and the change in the background color is shown in hatching. In addition, the server 1 displays the question output from the system side (server 1) together with the subject's answer to the question.

[0045] Furthermore, the server 1 displays an object 61 corresponding to the abnormal part in the message of the subject. The object 61 may be an example of a display mode showing a character string corresponding to the abnormal part, or may be an icon for playing back a voice input by the subject. When an operation input to the object 61 is accepted, the server 1 outputs the input voice to the mobile terminal 2 and plays it back. This allows other users (family members, etc.) to check the condition of the subject by listening to the input voice. Note that the server 1 may be capable of playing back not only the initial voice when the abnormal part is detected, but also a voice that answers a subsequent question.

[0046] Furthermore, when the server 1 judges that the subject may have a brain dysfunction, it notifies other users of the judgment result. For example, as shown in FIG. 6, the server 1 outputs a comment (information) to the subject that the subject may have a brain dysfunction, and displays it on the mobile terminal 2. Specifically, the server 1 displays a comment encouraging the subject to visit a medical institution and encouraging the subject to take a test to check whether or not the subject has a brain dysfunction. For example, the server 1 displays a button for selecting whether or not to take the test in the comment, and when the server 1 receives an operation input to the "TEST" button, it outputs test data to the mobile terminal 2 of the subject (or the mobile terminal 2 of another user). The test data is, for example, a calculation test such as addition and subtraction, a test to guess an object in a picture, etc., but is not particularly limited thereto.

[0047] Of course, the server 1 may notify the subject himself / herself of the determination result indicating the possibility of brain dysfunction, as well as other users (family members, etc.) related to the subject.

[0048] From the above, by detecting abnormalities from the subject's everyday conversational voice (messages to chat groups, voice input to chatbots, etc.), it is possible to easily detect abnormalities in the subject and encourage them to visit a medical institution, etc.

[0049] When displaying a message indicating that an abnormality has been detected, the server 1 may change the display mode of the abnormality depending on the state of the subject determined from the answer to the question and / or the image at the time of answer input. For example, if the server 1 determines that the state of the subject is abnormal from the voice answering the question, it displays the character string corresponding to the abnormality in red. On the other hand, if an abnormality has been detected from the message but the state of the subject is normal from the voice answering the question, the server 1 displays the character string corresponding to the abnormality in blue. This makes it possible to present the abnormality to other users by assigning a high or low level of abnormality, such as in the case of a simple slip of the tongue.

[0050] 7 is a flowchart showing the process steps executed by the server 1. The process contents executed by the server 1 will be described with reference to FIG. The control unit 11 of the server 1 receives voice input from the subject (step S11). As described above, the voice is, for example, a message to a chat group in which multiple users including the subject participate, but may also be voice input to a chatbot based on a predetermined dialogue engine. The control unit 11 converts the input voice into text (step S12). The control unit 11 detects an abnormality from the converted text (step S13). For example, the control unit 11 performs a morphological analysis of the text to divide it into multiple character strings, and compares each character string with each word stored in a predetermined word dictionary to detect the character string corresponding to the abnormality.

[0051] The control unit 11 judges whether an abnormality has been detected from the text (step S14). If it is judged that an abnormality has not been detected (S14: NO), the control unit 11 outputs the converted text to the mobile terminal 2 of the other user as a message for the subject, and displays it (step S15). If it is judged that an abnormality has been detected (S14: YES), the control unit 11 outputs a question to the mobile terminal 2 of the subject, asking for a character string corresponding to the abnormality (step S16). The control unit 11 accepts a voice input of a response to the question from the subject (step S17). The control unit 11 also acquires an image of the subject at the time of inputting the response from the mobile terminal 2 (step S18).

[0052] The control unit 11 judges whether the condition of the subject is abnormal based on the voice input in step S17 and / or the image acquired in step S18 (step S19). Specifically, the control unit 11 judges whether the subject may have a brain dysfunction. For example, the control unit 11 converts the input voice into text as in step S14, divides it into multiple character strings, and compares it with each word in the word dictionary to judge whether there is a character string corresponding to an abnormal part. In addition, the control unit 11 recognizes the face of the subject from the image captured in step S18, and judges whether the state and / or movement of the left and right sides of the face are asymmetric. If it is judged that there is no abnormality (S19: NO), the control unit 11 shifts the process to step S15.

[0053] When it is determined that there is an abnormality (S19: YES), the control unit 11 causes a message (text) showing the character string corresponding to the abnormal part in a display mode different from other character strings to be displayed on the mobile terminal 2 of the other user (step S20). Specifically, as described above, the control unit 11 changes the display color of the character string corresponding to the abnormal part and changes the background color of the character string to be displayed. In addition, the control unit 11 causes an object 61 for playing back the voice input by the subject user to be displayed.

[0054] The control unit 11 determines whether or not to play back the input voice in response to an operation input to the object 61 (step S21). If it is determined that the input voice is to be played back (S21: YES), the control unit 11 plays back the voice input by the target user on the mobile terminal 2 of the other user (step S22). After executing the process of step S22, or if the result of step S21 is NO, the control unit 11 ends the series of processes.

[0055] For the sake of convenience in the present embodiment, the server 1 is assumed to perform processes such as converting input voice into text and detecting abnormalities, but a part or all of the processes may be performed by a local mobile terminal 2 (or a speaker terminal 3). For example, the mobile terminal 2 may perform the text conversion and the server 1 may detect abnormalities. In this way, the entity that performs the series of processes is not particularly limited.

[0056] As described above, according to the first embodiment, the voice input by the subject is converted into text to detect the abnormal part, and the character string corresponding to the abnormal part is displayed in a different display mode from other character strings and presented to other users. This allows other users to easily understand the abnormality of the subject.

[0057] Furthermore, according to the first embodiment, abnormalities in a subject can be detected from the subject's usual conversational voice, such as conversational messages in a chat group or voice input to a chatbot.

[0058] Furthermore, according to the first embodiment, by referring to text related to the subject's previously input voice, it is possible to improve the accuracy of detecting an abnormal portion.

[0059] Furthermore, according to this embodiment 1, when an abnormality is detected, a question is output to ask about the abnormality and an answer is received, thereby making it possible to more appropriately determine whether or not the subject's condition is abnormal.

[0060] Furthermore, according to the first embodiment, by determining the left-right asymmetry of the face from an image captured of the subject when entering a response, it is possible to more appropriately determine abnormalities in the subject related to brain dysfunction.

[0061] Furthermore, according to this embodiment 1, by changing the display mode of the character string corresponding to the abnormal area depending on the answer to the question and / or the condition of the subject determined from an image capturing the subject, it is possible to present the abnormal area to other users with a rating of the degree of abnormality.

[0062] Furthermore, according to the first embodiment, by reproducing the voice input by the subject, other users can easily understand the state of the subject.

[0063] (Embodiment 2) In the first embodiment, an embodiment in which an abnormal part is detected from a voice input by a subject is described. In the present embodiment, an embodiment in which, when an abnormal part is detected, questions are asked by voice and text to estimate the possibility of brain dysfunction is described. Note that the same reference numerals are used for the contents that overlap with the first embodiment, and the description is omitted.

[0064] 8 is a block diagram showing a configuration example of the server 1 according to the embodiment 2. The auxiliary storage unit 14 of the server 1 according to the embodiment stores an answer history DB 141. The answer history DB 141 is a database that stores answers of the subject to questions described later and an estimation result of the possibility of brain dysfunction based on the answers.

[0065] FIG. 9 is an explanatory diagram showing an example of a record layout of the answer history DB 141. The answer history DB 141 includes a date and time column, a subject column, a voice column, a text column, a reaction column, an estimated result column, and an image column. The date and time column stores the answer date and time when the subject answers the question. The subject column, the voice column, the text column, the reaction column, the estimated result column, and the image column store, in association with the answer date and time, the name of the subject who answered, the correctness or incorrectness of the answer to the voice question (first question described later), the correctness or incorrectness of the answer to the text question (second question described later), the subject's reaction to the question, the estimation result of the possibility of brain dysfunction estimated based on the answer, and a captured image (e.g., a video) of the subject at the time of answering. The reaction column stores the determination results of the left and right symmetry of the face, the movement of the fingers or the line of sight, and facial expression, etc., determined from the captured image of the subject, as described later, as well as the answer time from the output of the question to the input of the answer.

[0066] Figures 10A and 10B are explanatory diagrams showing an example of a display screen of the speaker terminal 3. Figures 11A and 11B are explanatory diagrams showing an example of a display screen of a message according to embodiment 2. An outline of this embodiment will be described with reference to Figures 10A, 10B, 11A, and 11B.

[0067] As described in the first embodiment, the server 1 detects an abnormal part from the voice input by the subject and presents it to other users. In this embodiment, when an abnormal part is detected, the server 1 asks the subject questions by voice and text. Then, the server 1 estimates the possibility of brain dysfunction based on the subject's answers to the questions.

[0068] Specifically, the server 1 outputs a first question by voice and a second question by text to the speaker terminal 3, and causes the speaker terminal 3 to output voice and display an image corresponding to each question. Fig. 10A and B respectively show example screens when the first question is asked and when the second question is asked. The server 1 displays answer options for each question on the speaker terminal 3, and accepts input of an answer by accepting a screen operation to select one of the displayed options.

[0069] In this embodiment, the inquiry is made via the speaker terminal 3, but the inquiry may be made via the mobile terminal 2.

[0070] Before describing Fig. 10A and Fig. 10B, a description will be given based on Fig. 11A. Fig. 11A illustrates a chat screen displayed on another user's mobile terminal 2, similar to Fig. 6. When an abnormal part is detected from text related to the voice input by the subject, the mobile terminal 2 displays a message from the subject in which the abnormal part was detected, similar to the first embodiment.

[0071] In this embodiment, when the server 1 detects an abnormality, it accepts an input of a message to the subject from another user via the screen. The content of the message is not particularly limited, but it is preferable to accept an input of a message including an image. In the example of FIG. 11A, a message including an image of a close relative (e.g., a grandchild) of the subject is input as a message to the subject.

[0072] The server 1 analyzes messages input by other users and extracts data for generating the first and second questions. For example, the server 1 extracts proper nouns (e.g., a person's name, in the example of FIG. 11A and FIG. 11B, the grandchild's name "Taro") and images from the text. The server 1 generates the first and second questions based on the extracted data and outputs them to the speaker terminal 3.

[0073] 10A and 10B, the explanation will be continued. First, the server 1 generates a first question by voice and outputs it to the speaker terminal 3. For example, as shown in Fig. 10A, the server 1 outputs an image extracted from a message of another user and another image different from the extracted image to the speaker terminal 3 and displays them, and outputs a voice prompting a screen operation to select one of the images.

[0074] For example, the server 1 extracts an image area in which a person (grandchild) appears from the image extracted from the message, generates a thumbnail image, and displays it on the speaker terminal 3. The server 1 also displays unrelated images prepared in advance as another option. In the example of FIG. 10A, two images are displayed, but three or more images may be displayed. In the present embodiment, images input by other users are displayed, but for example, images may be prepared (registered) in a database in advance for each target person, and the images prepared in the database may be displayed. The server 1 applies a proper noun (grandchild's name) extracted from the message to a question in a template, generates a voice that prompts the user to select an image of a person corresponding to the proper noun, and outputs the voice to the speaker terminal 3.

[0075] The server 1 accepts input of a response to the first question. Specifically, the server 1 accepts a screen operation for selecting one of a plurality of images displayed on the speaker terminal 3. Note that the response may be input by voice or the like.

[0076] When the server 1 receives an input of a response to the first question, it outputs a second question in the form of text. For example, as shown in FIG. 10B, the server 1 displays a question asking whether or not to view an image (photo), and also displays an object (button) for selecting whether or not to view the image. Note that FIG. 10B illustrates a case where the correct image (image of a grandchild) is selected on the screen of FIG. 10A, and in this case, the question displayed is "Do you want to view the photo?", but if an incorrect image is selected on the screen of FIG. 10A, the question displayed is "Do you want to view the photo?".

[0077] The server 1 accepts a screen operation to select one of two options, "View photos" or "Don't view photos." If "View photos" is selected, the server 1 outputs messages from other users to the speaker terminal 3. Specifically, the server 1 causes the speaker terminal 3 to display images input by other users. Of course, text other than images may also be displayed. If "Don't view photos" is selected (or if no button is operated), the server 1 waits for processing for a predetermined time, and when the predetermined time has elapsed, the series of processes ends without displaying a message.

[0078] The server 1 judges whether the answers to the first and second questions are correct. The server 1 estimates the possibility of brain dysfunction based on the answers to the first and second questions being correct. Specifically, the server 1 estimates whether there is a possibility of brain dysfunction based on a combination of correct answers and incorrect answers, and estimates the type of possible brain dysfunction.

[0079] Although the brain dysfunction to be estimated is not particularly limited, in this embodiment, aphasia and dementia (or a temporary decline in cognitive function due to cerebral infarction, etc.) are estimated. The server 1 estimates whether or not there is a possibility of aphasia and whether or not there is a possibility of dementia based on the combination of correct and incorrect answers.

[0080] Specifically, the server 1 estimates that there is a possibility of aphasia if the answer to the first question by voice is incorrect and the answer to the second question by text is correct. Also, the server 1 estimates that there is a possibility of dementia if the answers to both the first and second questions are incorrect. Note that if both the first and second questions are correct, it estimates that there is a normal state, and if only the second question is incorrect, it treats it as an accidental answer mistake.

[0081] Aphasia and dementia are often confused, but aphasia is a disorder that impairs language ability, while dementia is a disorder that impairs general cognitive ability, including non-verbal ability. Depending on which symptom it is, the response to voice and text is different. Therefore, in this embodiment, a first question is asked by voice and a second question is asked by text, and aphasia and dementia are identified according to the combination of correct and incorrect answers to each question.

[0082] The server 1 estimates the possibility of brain dysfunction based on the answers to the first and second questions as well as images of the subject at the time of answering. For example, the speaker terminal 3 simultaneously images the subject when outputting the first question and / or the second question. The server 1 obtains answers to each question from the speaker terminal 3 and also obtains an image at the time of answering to perform the estimation.

[0083] For example, the server 1 estimates the possibility of brain dysfunction based on the left-right asymmetry of the subject's face, as in the first embodiment. That is, the server 1 divides the face area in the image into two areas, left and right, and identifies the state (coordinates of each feature point such as the eyes and the corners of the mouth) and movement (movement of the feature points) of each area to determine whether the left-right state and / or movement of the face are asymmetric. This allows the server 1 to detect a situation in which brain dysfunction is occurring due to cerebral infarction, etc.

[0084] In this embodiment, the server 1 estimates the possibility of brain dysfunction by determining from the image whether the subject is in a difficult state to answer, in addition to the asymmetry of the face. Specifically, the server 1 detects a specific event that corresponds to a difficult state from the image as follows.

[0085] For example, the server 1 detects the subject's hand (fingers) or the subject's gaze direction from the image, and determines whether the movement of the subject's hand or gaze direction corresponds to a specific movement. Specifically, the server 1 detects the movement of the subject's hand or gaze direction going back and forth between each option (image for the first question, button for the second question) because the subject is unsure which option to choose. For example, in the case of Broca's aphasia, when verbally commanded to select the correct option from multiple options, the subject is observed to have difficulty in answering due to a lack of understanding of the command, and to be unsure which option to choose. Therefore, the server 1 detects the hand or gaze direction from the image when answering the first question by voice, and determines whether the hand or gaze direction goes back and forth between the images, thereby estimating the possibility of aphasia.

[0086] Also, for example, the server 1 may recognize the facial expression of the subject and determine whether or not the facial expression corresponds to a specific expression (worried, anxious, troubled, etc.). In this case, it is possible to determine whether or not the subject is in a difficult state to answer, similarly to the above.

[0087] Also, for example, the server 1 may determine the distressed state by estimating the biological information of the subject from the image. The biological information may be, for example, pupil dilation, pulse rate, facial temperature (body temperature), blood flow rate, etc. The server 1 estimates this biological information from the image and detects changes in the biological information (for example, pupil dilation, faster pulse rate, etc.) to determine whether the subject is in a distressed state to answer.

[0088] In the above, whether or not the user is in distress is determined from the image, but for example, the server 1 may determine whether or not the user is in distress based on the response time to a question in addition to the image. Specifically, the server 1 measures the response time from outputting a question (for example, the first question) to inputting a response, and determines whether or not the response time is equal to or greater than a predetermined threshold. This makes it possible to detect a situation in which a response takes a long time because the user is in distress.

[0089] As described above, the server 1 estimates the possibility of brain dysfunction from the image captured of the subject and / or the response time, in addition to the responses to the first and second questions. For example, even if the responses to both the first and second questions are correct and the subject is estimated to be normal, the server 1 estimates that there is a possibility of brain dysfunction if the left and right movement of the face and / or the state is asymmetric or if the subject is determined to be in a distressed state. Or, even if the response to the first question is correct and the response to the second question is incorrect and processed as an accidental response error by the subject, if the left and right movement of the face and / or the state is asymmetric or if the subject is determined to be in a distressed state, the server 1 may repeat the process of asking questions by changing the question text, etc.

[0090] The server 1 outputs the estimation result to the mobile terminal 2 of the other user and displays it. Fig. 11B shows a chat screen when the estimation result is displayed. For example, the server 1 displays text indicating the estimation result (judgment result) and also displays a score that quantifies the estimation result.

[0091] The server 1 calculates a score for each of the "voice" corresponding to the first question, the "text" corresponding to the second question, and the "response" corresponding to the image and the answering time, and displays them on the mobile terminal 2. There is no particular limitation on the method of calculating the scores, but for example, the server 1 tallys up the correct answers to the first and second questions given in a past predetermined period (e.g., one week), calculates scores evaluating the voice recognition ability and the character recognition ability, respectively (e.g., the rate of correct answers in a predetermined period), and outputs them as the scores for "voice" and "text". In addition, for example, the server 1 calculates the degree of distress from the image and / or the answering time, and outputs it as the score for "response".

[0092] Fig. 12 is an explanatory diagram showing another example of the chat screen when the estimation result is displayed. Fig. 12 shows the chat screen when it is estimated that there is a high possibility of brain dysfunction. When it is estimated that there is a high possibility of brain dysfunction, the server 1 notifies the other user's mobile terminal 2 of the estimation result and causes it to be displayed on the chat screen.

[0093] Specifically, the server 1 displays a score obtained by converting the estimation result into a numerical value, as in the case of Fig. 11B, and also displays text indicating that the possibility of a brain dysfunction is high. For example, as shown in Fig. 12, the server 1 displays a comment encouraging the user to visit a medical institution, together with indicating the type of brain dysfunction that is estimated to be highly likely.

[0094] Of course, the server 1 may notify the subject himself / herself of the estimation result indicating the possibility of brain dysfunction, as well as other users (family members, etc.) related to the subject.

[0095] The server 1 further displays a link 121 on the chat screen for viewing (checking) the subject's response history. The link 121 is an object for outputting (displaying) history information indicating the subject's past responses to the first and second questions and the history of the estimation results of the possibility of brain dysfunction, and is an object for transitioning to the history screen of Fig. 13. When an operation input to the link 121 is accepted, the mobile terminal 2 transitions to the history screen of Fig. 13.

[0096] Note that the history information may be made viewable even when the subject's condition is estimated to be normal (FIG. 11B). In addition, it is of course possible to make the history information viewable at any time, not just by transitioning from the chat screen.

[0097] 13 is an explanatory diagram showing an example of a history screen. The history screen is a display screen that displays a series of history information such as the correctness or incorrectness of the subject's answers to the first and second questions, the captured image of the subject taken at the time of answering, and the estimated result of brain dysfunction based on the answers and the images. The server 1 stores various types of history information in an answer history DB 141, and outputs the history information in response to a request from the mobile terminal 2. For example, the history screen includes an answer history table 131, an image display field 132, and a score graph 133.

[0098] The answer history table 131 is a table showing whether the answers to the first and second questions ("audio" and "text") were correct at each past point in time, the degree of distress of the subject judged based on the captured image at the time of answering ("response" score), and the estimated result of brain dysfunction ("judgment"). In addition, the answer history table 131 displays a play button 1311 for playing the captured image (video) in association with each point in time.

[0099] The image display field 132 is a display field that displays an image of the subject captured when the subject answers the first and / or second question. When an operation input to the play button 1311 is accepted, the mobile terminal 2 displays an image (video) captured at the corresponding time.

[0100] 11B and 12 in time series. The mobile terminal 2 displays a graph (e.g., a line graph) showing the scores for "voice" that evaluates the voice recognition ability based on the correctness of the answer to the first question, "text" that evaluates the character recognition ability based on the correctness of the answer to the second question, and "response" that evaluates the subject's condition based on a captured image or the like, and presents the change in the subject to the user.

[0101] As described above, when the server 1 detects an abnormal part from the speech of the subject, it outputs the first and second questions and estimates the possibility of brain dysfunction from the correctness of the answers to each question and the images at the time of answering, etc. This makes it possible to discover abnormalities in the subject early and analyze the brain dysfunction.

[0102] 14 and 15 are flowcharts showing an example of a process procedure executed by the server 1 according to embodiment 2. After executing the process of step S22, or if NO in step S21, the server 1 executes the following process. The control unit 11 of the server 1 accepts an input of a message including an image from another user (step S23). The control unit 11 analyzes the message, extracts the image included in the message, and extracts proper nouns and the like from the text (step S24).

[0103] Based on the message analyzed in step S24, the control unit 11 outputs a first question by voice to the speaker terminal 3 (step S25). For example, the control unit 11 causes the speaker terminal 3 to display a thumbnail image obtained by extracting a part of the image and another image different from the image as options, and outputs a voice prompting the user to operate the screen to select one of the images. The control unit 11 accepts input of a response to the first question (step S26). Specifically, as described above, the control unit 11 accepts an operation input to select one of the multiple images (options) displayed.

[0104] The control unit 11 outputs a second question in the form of text to the speaker terminal 3 (step S27). For example, the control unit 11 causes buttons for selecting whether or not to view the image to be displayed as options, and also causes the speaker terminal 3 to display text asking whether or not to view the image selected in step S26. The control unit 11 accepts input of a response to the second question (step S28). For example, the control unit 11 accepts an operation input for selecting one of the multiple displayed buttons (options).

[0105] The control unit 11 judges whether the answer to the second question is correct or not (step S29). For example, the control unit 11 judges whether a selection input to view an image has been received or not. If it is judged to be correct (S29: YES), the control unit 11 outputs a message (image) from another user to the speaker terminal 3 (step S30).

[0106] If it is determined that the answer is not correct (S29: NO), the control unit 11 determines whether or not a predetermined time has elapsed since the second question was output (step S31). If it is determined that the predetermined time has not elapsed (S31: NO), the control unit 11 returns the process to step S29. After executing the process of step S30, or if the result of step S31 is YES, the control unit 11 acquires an image of the subject at the time of answering in step S26 and / or step S28 from the speaker terminal 3 (step S32).

[0107] The control unit 11 estimates the possibility of brain dysfunction of the subject based on the answers to the first and second questions and the image of the subject at the time of answering and / or the answering time (step S33). Specifically, the control unit 11 estimates whether or not there is a possibility of brain dysfunction, and estimates the type of brain dysfunction (aphasia and dementia). For example, when the answer to the first question is incorrect and the answer to the second question is correct, the control unit 11 estimates that there is a high possibility of aphasia. Furthermore, when the answers to both the first and second questions are incorrect, the control unit 11 estimates that there is a high possibility of dementia.

[0108] Furthermore, the control unit 11 determines whether the left and right state and / or movement of the face are asymmetrical from the image of the subject at the time of answering. The control unit 11 also determines whether the subject is in a distressed state from the image of the subject and / or the answering time. Even if the answers to the first and second questions are estimated to be normal, the control unit 11 estimates that there is a possibility of brain dysfunction depending on the determination result of the left and right asymmetry of the face and / or the distressed state. The control unit 11 stores the correctness or incorrectness of the answers to the first and second questions, the captured image of the subject at the time of answering, the estimation result of the possibility of brain dysfunction, etc. in the answer history DB 141 (step S34).

[0109] The control unit 11 outputs the estimation result to the portable terminal 2 of the other user (step S35). For example, the control unit 11 causes the portable terminal 2 to display the estimation result as to whether or not there is a possibility of brain dysfunction, and calculates a score evaluating the subject based on the answer to the first question (audio), the answer to the second question (text), and the image of the subject at the time of answering and / or the answering time, and causes the portable terminal 2 to display the score.

[0110] The control unit 11 judges whether or not to output history information indicating the past answers by the subject to the first and second questions and the history of the estimation result of the possibility of brain dysfunction based on the answers (step S36). For example, the control unit 11 judges whether or not an operation input to the link 121 is received on the chat screen exemplified in FIG. 12. When it is judged that the history information is to be output (S36: YES), the control unit 11 outputs the history information to the mobile terminal 2 of the other user and displays it (step S37). Specifically, as described above, the control unit 11 displays the answers to the first and second questions at each time point in the past, the estimation result related to the brain dysfunction, and images of the subject as history information. After executing the process of step S36, or if NO in step S36, the control unit 11 ends the series of processes.

[0111] In the above description, the first and second questions are output in response to a message from another user, but the present embodiment is not limited to this. For example, the server 1 may output the first and second questions to the speaker terminal 3 at regular intervals and accept answer inputs regardless of the presence or absence of messages from other users. In this case, the server 1 may prepare images for questions (images of a grandchild in the above example) in a database in advance, and generate the first and second questions using the images. In this way, the first and second questions may be output regardless of the presence or absence of messages from other users.

[0112] As described above, according to the second embodiment, the possibility of brain dysfunction can be suitably estimated by asking a first question by voice and a second question by text.

[0113] Furthermore, according to the second embodiment, it is possible to estimate the type of brain dysfunction (preferably aphasia and dementia) based on a combination of correct and incorrect answers to each question.

[0114] Furthermore, according to the second embodiment, answer options are displayed on the speaker terminal 3 and answer input is accepted by operating the screen, so that even in a case where cognitive ability has been reduced due to brain dysfunction, it is possible to appropriately prompt the user to input an answer.

[0115] Furthermore, according to the second embodiment, when an abnormal part is detected from the subject's conversational voice, a question is started, which allows early detection of brain dysfunction.

[0116] Furthermore, according to the second embodiment, the first and second questions are generated from messages from other users who are conversation partners of the target person, making it possible to ask questions appropriate to the target person.

[0117] Furthermore, according to the second embodiment, in addition to the answer itself, the possibility of brain dysfunction is estimated based on the image of the subject at the time of answering and / or the answering time. This makes it possible to detect a state where cerebral infarction or the like has occurred (asymmetry of the left and right sides of the face) or a state where the subject is having difficulty answering, and more appropriately estimates the possibility of brain dysfunction.

[0118] The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0119] 1. Server (information processing device) 11 Control section 12 Main memory 13. Communications Department 14 Auxiliary storage P1 Program 141 Answer History DB 2. Mobile devices 21 Control section 22 Main memory 23 Communications Department 24 Display section 25 Input section 26 Audio output section 27 Audio input section 28 Imaging unit 29 Auxiliary storage P2 Program 3 Speaker terminals 31 Control Unit 32 Main memory 33 Communications Department 34 Display section 35 Input section 36 Audio output section 37 Audio input section 38 Imaging section 39 Auxiliary storage P3 Program

Claims

1. outputting a first question by voice; accepting a response to the first question from the subject; outputting a second textual prompt; accepting a response to the second question from the subject; determining whether the answers to the first and second questions are correct; A possibility of brain dysfunction of the subject is estimated based on the correctness of the answers to the first and second questions. A program for causing a computer to execute a process, If the answer to the first question is incorrect and the answer to the second question is correct, it is assumed that there is a possibility of aphasia; If the answers to both the first and second questions are incorrect, it is assumed that there is a possibility of dementia. A program that causes a computer to carry out processing.

2. Displaying answer options to the first and second questions on a display unit; and accepting input of answers to the first and second questions by accepting a screen operation to select one of the displayed options. The program according to claim 1.

3. Obtaining text obtained by converting the speech input from the subject; Detecting anomalies from the text; When the abnormal portion is detected, the first and second queries are output.

3. The program according to claim 1 or 2.

4. receiving, from the target person, a voice input of a message to be sent to a chat group in which a plurality of users including the target person participates; converting the message into the text and detecting the anomaly; When the abnormality is detected, the text showing the character string corresponding to the abnormality in a display mode different from other character strings is displayed on the terminal device of another user. The program according to claim 3.

5. accepting an input of a message from the terminal device of the other user that displays the text; generating the first and second questions based on the messages of the other users; Outputting the generated first and second questions The program according to claim 4.

6. acquiring an image of the subject at the time of answering the first or second question; The possibility of the brain dysfunction is estimated based on the correctness of the answers to the first and second questions and the image. The program according to any one of claims 1 to 5.

7. determining whether the subject's facial movement or state is asymmetric based on the image; If it is determined that the brain function is asymmetric, it is presumed that there is a possibility of the brain function disorder. The program according to claim 6.

8. determining whether the subject is in a difficult state to respond based on the image; If it is determined that the subject is in a state of difficulty in responding, it is presumed that the subject may have a brain dysfunction. The program according to claim 6 or 7.

9. measuring a response time from outputting the first or second question to receiving an input of a response; The possibility of the brain dysfunction is estimated based on the correctness of the answers to the first and second questions and the answering time. The program according to any one of claims 1 to 8.

10. Notifying the subject or another user related to the subject of the estimated result of the possibility of the brain dysfunction. The program according to any one of claims 1 to 9.

11. The answers to the first and second questions and the estimation result of the brain dysfunction based on the correctness of the answers are stored in a storage unit; Outputting history information indicating the history of the answers and estimation results The program according to any one of claims 1 to 10.

12. a first output unit that outputs a first question by voice; a first reception unit that receives an answer to the first question from the subject; a second output unit that outputs a second question in the form of text; a second reception unit that receives an answer to the second question from the subject; a determination unit that determines whether or not the answers to the first and second questions are correct; and an estimation unit that estimates a possibility of brain dysfunction of the subject based on the correctness of the answers to the first and second questions, The early estimation part is If the answer to the first question is incorrect and the answer to the second question is correct, it is assumed that there is a possibility of aphasia; If the answers to both the first and second questions are incorrect, it is assumed that there is a possibility of dementia. Information processing device.

13. outputting a first question by voice; accepting a response to the first question from the subject; outputting a second textual prompt; accepting a response to the second question from the subject; determining whether the answers to the first and second questions are correct; A possibility of brain dysfunction of the subject is estimated based on the correctness of the answers to the first and second questions. An information processing method in which a process is executed by a computer, comprising the steps of: If the answer to the first question is incorrect and the answer to the second question is correct, it is assumed that there is a possibility of aphasia; If the answers to both the first and second questions are incorrect, it is assumed that there is a possibility of dementia. An information processing method in which processing is performed by a computer.

Images