Programs, information processing devices, and information processing methods

Abstract

The program causes the computer to perform the following processes: outputting a first question using sound, receiving an answer to the first question from the subject, outputting a second question using text, receiving an answer to the second question from the subject, determining whether the answers to the first and second questions are correct, and inferring the possibility that the subject has a brain dysfunction based on whether the answers to the first and second questions are correct.

Description

Technical Field

[0001] This invention relates to programs, information processing apparatus, and information processing methods. Background Technology

[0002] There are technologies that support the diagnosis of brain dysfunctions such as dementia. For example, Patent Document 1 discloses a dementia diagnostic device that converts the spoken voice of a subject answering a prescribed question into text data, calculates the edit distance between the text data and comparison text data, and determines whether the subject is likely to have symptoms of dementia.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: JP 2020-483 Summary of the Invention

[0006] However, the invention in Patent Document 1 did not infer the possibility of having a brain dysfunction based on the different symptoms of various brain dysfunctions such as dementia and aphasia.

[0007] In one aspect, the aim is to provide a procedure that can appropriately infer the likelihood of having a brain dysfunction, etc.

[0008] One aspect of the procedure involves a computer performing the following processing: outputting a first question using sound, receiving an answer to the first question from the subject, outputting a second question using text, receiving an answer to the second question from the subject, determining whether the answers to the first and second questions are correct, and inferring the likelihood that the subject has a brain dysfunction based on the correctness of the answers to the first and second questions.

[0009] Invention Effects

[0010] In one respect, it is possible to properly infer the likelihood of having a brain dysfunction. Attached Figure Description

[0011] Figure 1 This is an explanatory diagram showing an example of the structure of a dialogue system.

[0012] Figure 2 This is a block diagram illustrating an example of server configuration.

[0013] Figure 3 This is a block diagram illustrating an example of the configuration of a portable terminal.

[0014] Figure 4 This is a block diagram showing an example of the configuration of a speaker terminal.

[0015] Figure 5 This is an explanatory diagram showing an outline of Embodiment 1.

[0016] Figure 6 This is an explanatory diagram showing an example of a message display screen.

[0017] Figure 7 This is a flowchart illustrating the processing sequence executed by the server.

[0018] Figure 8 This is a block diagram illustrating an example of the configuration of the server according to Embodiment 2.

[0019] Figure 9 This is an illustrative diagram showing an example of the record layout for answering historical records in a DB.

[0020] Figure 10A This is an explanatory diagram showing an example of a display screen on a speaker terminal.

[0021] Figure 10B This is an explanatory diagram showing an example of a display screen on a speaker terminal.

[0022] Figure 11A This is an explanatory diagram showing an example of a message display screen for Embodiment 2.

[0023] Figure 11B This is an explanatory diagram showing an example of a message display screen for Embodiment 2.

[0024] Figure 12 This is an illustrative diagram showing another example of a chat screen displaying the inference results.

[0025] Figure 13 This is an explanatory diagram showing an example of a historical record screen.

[0026] Figure 14 This is a flowchart illustrating an example of the processing sequence executed by the server in Implementation 2.

[0027] Figure 15 This is a flowchart illustrating an example of the processing sequence executed by the server in Implementation 2. Detailed Implementation

[0028] Hereinafter, the present invention will be described in detail with reference to the accompanying drawings illustrating embodiments thereof.

[0029] (Implementation Method 1)

[0030] Figure 1This is an explanatory diagram showing an example of the configuration of a dialogue system. In this embodiment, a dialogue system that judges abnormalities in a user based on the user's spoken words, and more accurately determines whether there is a brain dysfunction, will be described. The dialogue system includes an information processing device 1, a portable terminal 2, and a speaker terminal 3. Each device is connected to a network N such as the Internet for communication.

[0031] Information processing device 1 is an information processing device capable of performing various information processing and sending / receiving information, such as a server computer or a personal computer. In this embodiment, information processing device 1 is a server computer, which will be referred to as server 1 for simplicity below. Server 1 determines whether there is a possibility of brain dysfunction from the speech voice input by the subject. Specifically, as described later, server 1 detects abnormalities suspected of indicating possible brain dysfunction from the speech voice input by the subject as a message sent to a chat group including the subject and other users, or the speech voice input by the subject to a chatbot system that operates based on a prescribed dialogue engine.

[0032] There are no specific limitations on the brain dysfunction that can be considered as a subject, such as dementia or aphasia. Server 1 detects abnormal speech (unclear words, mispronunciations, etc.) caused by dementia, aphasia, etc., and alerts other users (such as family members of the subject, medical professionals who treat the subject, and other users associated with the subject) to the abnormalities.

[0033] Portable terminal 2 is an information processing terminal used by various users, including the target user, such as a smartphone or tablet. Furthermore, in Figure 1 The diagram only shows one portable terminal 2, but the portable terminals 2 of the target user and other users are connected to server 1. Server 1 obtains the target user's spoken voice from the portable terminal 2 for purposes such as sending messages to the chat group, and converts it into text. Then, server 1 detects anomalies from the converted text.

[0034] Speaker terminal 3 is a sound input / output terminal installed in the user's home or similar location, and is a so-called smart speaker. However, speaker terminal 3 is not limited to being called a smart speaker; it only needs to have sound input / output functions and image display functions. Furthermore, the location of speaker terminal 3 is not limited to the user's home; it can also be installed in facilities other than the user's home (such as a nursing home). Speaker terminal 3 functions as a terminal device in the chatbot system, engaging in dialogue with the user. As explained later, server 1 can also obtain the user's voice from speaker terminal 3 to detect anomalies.

[0035] Furthermore, in this embodiment, portable terminal 2 and speaker terminal 3 are examples of terminal devices that work in conjunction with server 1, but other types of terminal devices (such as robot-type devices) may also be used. The terminal device can be any local terminal with sound input / output functions, image display functions, etc., and its form is not particularly limited.

[0036] Figure 2 This is a block diagram showing an example of the configuration of server 1. Server 1 includes a control unit 11, a main storage unit 12, a communication unit 13, and an auxiliary storage unit 14.

[0037] The control unit 11 has one or more processing devices such as CPUs (Central Processing Units), MPUs (Micro-Processing Units), and GPUs (Graphics Processing Units) to read and execute the program P1 stored in the auxiliary storage unit 14, thereby performing various information processing and control processing. The main storage unit 12 is a temporary storage area for SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), flash memory, etc., temporarily storing data required by the control unit 11 for processing. The communication unit 13 is a communication module for communication-related processing, transmitting and receiving information with external devices. The auxiliary storage unit 14 is a non-volatile storage area such as a large-capacity memory or hard disk, storing the program P1 and other data required by the control unit 11 for processing.

[0038] Furthermore, the auxiliary storage unit 14 can be an external storage device connected to the server 1. Additionally, the server 1 can be multiple computers, or it can be a virtual machine built using software.

[0039] Furthermore, in this embodiment, the server 1 is not limited to the above-described configuration, and may also include, for example, an input unit for receiving operation input and a display unit for displaying images. Additionally, the server 1 has a reading unit for reading portable storage media 1a such as CD (Compact Disk)-ROM and DVD (Digital Versatile Disc)-ROM, and can read and execute program P1 from portable storage media 1a. Alternatively, the server 1 may also read program P1 from semiconductor memory 1b.

[0040] Figure 3This is a block diagram showing an example of the configuration of a portable terminal 2. The portable terminal 2 includes a control unit 21, a main storage unit 22, a communication unit 23, a display unit 24, an input unit 25, a sound output unit 26, a sound input unit 27, a camera unit 28, and an auxiliary storage unit 29.

[0041] The control unit 21 has one or more arithmetic processing devices such as CPUs and MPUs, which read and execute program P2 stored in the auxiliary storage unit 29, thereby performing various information processing and control processing. The main storage unit 22 is a temporary storage area such as RAM, which temporarily stores the data required by the control unit 21 for performing arithmetic processing. The communication unit 23 is a communication module for performing communication-related processing, and for sending and receiving information with external devices. The display unit 24 is a display screen such as a liquid crystal display, which displays images.

[0042] Input unit 25 is an operation interface such as a touch panel, receiving operation input from the user. Sound output unit 26 is a speaker, outputting sound. Sound input unit 27 is a microphone, receiving sound input from the user. Image capture unit 28 is a camera with an image capture element such as a CMOS (Complementary MOS), capturing images. Auxiliary storage unit 29 is a non-volatile storage area such as a hard disk or mass storage device, storing program P2 and other data required for processing by control unit 21.

[0043] Furthermore, the portable terminal 2 is equipped with a reading unit that can read portable storage media 2a such as CD-ROMs, and can also read and execute program P2 from portable storage media 2a. Alternatively, the portable terminal 2 can also read program P2 from semiconductor memory 2b.

[0044] Figure 4 This is a block diagram showing an example of the configuration of a speaker terminal 3. The speaker terminal 3 includes a control unit 31, a main storage unit 32, a communication unit 33, a display unit 34, an input unit 35, a sound output unit 36, a sound input unit 37, a camera unit 38, and an auxiliary storage unit 39.

[0045] The control unit 31 has one or more processing units such as CPUs and MPUs, which read and execute the program P3 stored in the auxiliary storage unit 39, thereby performing various information processing and control processing. The main storage unit 32 is a temporary storage area such as RAM, which temporarily stores the data required by the control unit 31 for performing processing. The communication unit 33 is a communication module for performing communication-related processing, and for sending and receiving information with external devices. The display unit 34 is a display screen such as a liquid crystal display, which displays images.

[0046] Input unit 35 is an operation interface such as a touch panel, receiving operation input from the user. Sound output unit 36 ​​is a speaker, outputting sound. Sound input unit 37 is a microphone, receiving sound input from the user. Image capture unit 38 is a camera with an image capture element such as a CMOS sensor, capturing images. Auxiliary storage unit 39 is a non-volatile storage area such as a hard disk or mass storage device, storing the program P3 and other data required for processing by control unit 31.

[0047] Furthermore, the speaker terminal 3 has a reading unit that can read portable storage media 2a such as CD-ROMs, and can read and execute program P3 from portable storage media 3a. Alternatively, the speaker terminal 3 can read program P3 from semiconductor memory 3b.

[0048] Figure 5 This is an explanatory diagram showing an outline of Embodiment 1. Based on Figure 5 This section provides an overview of the implementation method.

[0049] As described above, server 1 determines whether the target's status is abnormal based on messages sent to the chat group by multiple users, including the target user. Figure 5 The image shows the dialogue patterns in a chat group. Figure 5 The right side represents messages from the recipient, while the left side represents messages from other users (such as family members) and the system (server 1). Furthermore, the recipient can also input messages using text input or voice recognition.

[0050] Server 1 converts the user's input voice into text and detects anomalies in the converted text. Figure 5 In the example, the illustration shows a situation where, in contrast to the message "Where did you go today?" from another user, server 1 converts the recipient's spoken voice into the text "went to taoyuan today". In this case, server 1 detects an anomaly in the text at "taoyuan".

[0051] The specific method for detecting anomalies is not particularly limited. For example, server 1 performs morphological analysis to divide the text into multiple strings (vocabularies), and compares each string with each word in the vocabulary dictionary (database not shown) by referring to a vocabulary dictionary containing many words. Furthermore, in this embodiment, the unit of text division is set as a word; the computer can also divide the text using units longer than words (e.g., sentence segments) or units shorter than words. Server 1 detects strings not stored in the vocabulary dictionary as anomalies. Additionally, for example, server 1 can pre-define low-frequency words (e.g., words other than common phrases) in the vocabulary dictionary, and detect low-frequency words as anomalies, etc.

[0052] In addition, the server 1 can also store in advance the text of the voice input by the subject, and detect abnormal points based on the past text. For example, the server 1 saves (enters) the strings that can divide the text by morphological analysis as new words in the word dictionary, and constructs a word dictionary for each subject. Then, when the server 1 receives the voice input from the subject and converts it into text, it refers to the word dictionary corresponding to the subject to detect abnormal points. Thus, considering the speaking tendency of the subject, the detection accuracy of abnormal points can be improved.

[0053] In addition, although the above uses the word dictionary to detect abnormal points, the detection method of abnormal points is not limited to this. For example, the server 1 can also perform syntactic structure analysis and semantic analysis of the text to detect abnormal points. In addition, the detection method is not limited to the rule base. For example, the server 1 can also prepare in advance a machine learning model (such as a neural network) that has completed learning in a way that can detect abnormal points when text is input, and input the text converted into speech sound into this model to detect abnormal points. In this way, the detection method of abnormal points is not particularly limited.

[0054] When an abnormal point is detected, the server 1 generates a question that repeatedly asks about the abnormal point and outputs it to the portable terminal 2 of the subject. This question preferably adopts one of the forms of 6W3H (Who, Whom, When, Where, What, Why, How, How many, How much). The server 1 applies the string corresponding to the abnormal point to the template of a question in one of the forms of 6W3H to generate a question. The server 1 outputs the question as a message in the chat group and displays it on the portable terminal 2. In addition, for example, the server 1 can also convert the question into voice and output it to the portable terminal 2.

[0055] The server 1 receives the input of an answer corresponding to the above question from the subject. This answer is input using voice in the same way as when inputting a message. The server 1 converts the input answer voice into text and determines whether the state of the subject is abnormal. Specifically, it determines whether there is a possibility of suffering from a brain dysfunction.

[0056] In Figure 5 Patterns 1 to 3 are illustrated as answer examples. In the case of Pattern 1, the correct word "park (gongyuan)" is recognized from the voice, so the server 1 determines that the state of the subject is normal. On the other hand, in the case of Pattern 2, the abnormal point "taoyuan" is recognized again from the voice, so the server 1 determines that the state of the subject is abnormal. In addition, in the case of Pattern 3, although the string "taoyuan (Taoyuan)" is included, a correct sentence is recognized from the voice considering the context, so the server 1 determines that the state of the subject is normal.

[0057] In this way, server 1 determines the state of the user based on their answer to the question. In this case, server 1 can also determine the state of the user from data other than sound. For example, if portable terminal 2 captures a picture of the user answering the question, server 1 can determine the state of the user from the captured image (e.g., a moving image).

[0058] Specifically, server 1 identifies the subject's face from an image and judges the subject's condition based on the asymmetry of the face. In cases of brain dysfunction, such as cerebral infarction or cerebral hemorrhage, the left and right sides of the face may exhibit different dynamics, with one side drooping or tilting, resulting in asymmetry and dynamics. The server divides the facial region in the image into left and right regions, determines the state (coordinates of feature points such as the eyes and mouth) and dynamics (movement of feature points) of each region, and judges whether the left and right state and / or dynamics of the face are asymmetrical. If asymmetry is determined, server 1 determines that the subject's condition is abnormal.

[0059] Furthermore, the subject can be photographed while answering the aforementioned questions, but also when the subject initially receives the audio as a message (a message indicating an anomaly is detected), and the asymmetry of the face can be determined from the image taken when the message (audio) is received. In other words, the image can be captured not only when answering the questions, but also when the audio of the message is received.

[0060] In addition, while this embodiment combines images and sounds to determine the abnormality of the subject, it is also possible to determine the abnormality of the subject solely from sound (text).

[0061] As described above, server 1 detects anomalies in the text of the voice input sent by the subject to the chat group, which may indicate a possible brain dysfunction, and asks questions in response to the anomalies, and judges the subject's state from the voice of the response to the questions and / or the image of the response input.

[0062] Furthermore, the above description uses the scenario of a target user engaging in group chat with other users as an example, but this implementation is not limited to this. For example, server 1 can detect anomalies in the input voice when a chatbot based on a defined dialogue engine is conversing with a target user.

[0063] Chatbots can input and output sound using portable terminals 2 such as smartphones, but they can also use speaker terminals 3 (smart speakers) located in the user's home or elsewhere. Here, we will explain how speaker terminals 3 receive sound input from the user and output response sound.

[0064] Speaker terminal 3 accepts various sound inputs, such as daily greetings ("Good morning," etc.), information output requests (e.g., today's weather, schedules, etc.), device (home appliances, etc.) operation requests, etc. Speaker terminal 3 performs various information processing on these sound inputs (e.g., outputting a greeting response sound after a greeting is input, outputting a device operation signal after a device operation sound is input, etc.). Server 1 receives the sound input to speaker terminal 3, converts it into text, and detects anomalies. The anomaly detection method is the same as described above.

[0065] Additionally, server 1 can call the recipient from the system side via speaker terminal 3 to receive audio input. For example, server 1 can periodically output a question ("How's the weather today?") to speaker terminal 3 and receive a response from the recipient. Thus, in cases where the recipient is, for example, an elderly person living alone, regular conversations can be facilitated, and abnormalities in the recipient can be detected from the content of the conversation.

[0066] In this way, server 1 can detect anomalies from the audio of conversations with the chatbot. That is, the audio that becomes the object of anomaly detection is not limited to messages sent to other users, but can be any speaking voice.

[0067] Figure 6 This is an explanatory diagram showing an example of a message display screen. In Figure 6 The image illustrates how messages (text) indicating an anomaly are displayed in a group chat. Specifically, in... Figure 6 The image shows a chat window displayed on a portable terminal 2 belonging to another user (family member, etc.) exchanging messages with the recipient. Figure 6 In the diagram, messages from the target and the system are shown on the left, while messages from other users are shown on the right.

[0068] When Server 1 detects an anomaly in the message from the target, it displays the string corresponding to the anomaly in a different format than other strings. For example, Server 1 changes the display color of the string corresponding to the anomaly and changes (highlights) the background color of the anomaly. Furthermore, in Figure 6 For illustration purposes, bold text is used to show when the display color of the string changes, and shading is used to show when the background color changes. Additionally, server 1 also displays the question output from the system side (server 1) and the respondent's answer to that question.

[0069] Additionally, server 1 displays object 61 corresponding to the anomaly in the message between the server and the recipient. Object 61 can be, for example, a display format representing the string corresponding to the anomaly, or it can be an icon used to reproduce the sound input by the recipient. Upon receiving operation input to object 61, server 1 outputs the input sound to portable terminal 2 and reproduces it. Thus, other users (family members, etc.) can hear the input sound to confirm the recipient's status. Furthermore, server 1 can reproduce not only the initial sound that detected the anomaly, but also the sound of responses to subsequent questions.

[0070] Furthermore, if server 1 determines that the target user may have a brain dysfunction, it will notify other users of the determination. For example, if... Figure 6 As shown, server 1 outputs an assessment (information) suggesting the possibility of the subject having a brain dysfunction and displays it on portable terminal 2. Specifically, server 1 displays an assessment that encourages the subject to seek medical attention and undergo tests to check for brain dysfunction. For example, server 1 displays a button in the assessment for selecting whether or not to undergo the test. Upon receiving input to the "test" button, server 1 outputs test data to the subject's portable terminal 2 (or another user's portable terminal 2). The test data may include, for example, calculation tests such as addition and subtraction, or tests involving touching objects reflected in a photograph, but is not specifically limited to these.

[0071] In addition, server 1 can notify not only other users associated with the subject (family members, etc.), but also the subject himself of the judgment result that he may have a brain dysfunction.

[0072] In summary, by detecting anomalies in the subject's normal conversational voice (messages sent to chat groups, voice input to chatbots, etc.), it is possible to easily detect abnormalities in the subject and encourage them to seek medical attention.

[0073] Furthermore, when displaying a message indicating an anomaly, server 1 can change the display of the anomaly in accordance with the state of the user determined from the answer to the question and / or the image at the time of the answer input. For example, if server 1 determines that the user's state is abnormal based on the audio of the answer to the question, it displays the string corresponding to the anomaly in red. On the other hand, if an anomaly is detected in the message but the user's state is normal based on the audio of the answer to the question, server 1 displays the string corresponding to the anomaly in blue. Thus, in cases such as simply making a mistake, the severity of the anomaly can be graded to alert other users to the anomaly.

[0074] Figure 7 This is a flowchart illustrating the processing sequence executed by server 1. Based on Figure 7This describes the processing performed by server 1.

[0075] The control unit 11 of server 1 receives voice input from the target user (step S11). As described above, the voice input may be a message sent to a chat group consisting of multiple users, including the target user, or it may be voice input to a chatbot based on a defined dialogue engine. The control unit 11 converts the input voice input into text (step S12). The control unit 11 detects anomalies in the converted text (step S13). For example, the control unit 11 performs morphological analysis of the text to divide it into multiple strings, compares each word stored in a defined vocabulary dictionary with each string, and detects the string corresponding to the anomaly.

[0076] The control unit 11 determines whether an anomaly is detected from the text (step S14). If no anomaly is detected (S14: No), the control unit 11 outputs the converted text as a message from the target user to the portable terminal 2 of another user and displays it (step S15). If an anomaly is detected (S14: Yes), the control unit 11 outputs a question that asks a question corresponding to the string of the anomaly to the target user's portable terminal 2 (step S16). The control unit 11 receives audio input from the target user regarding the answer to the question (step S17). In addition, the control unit 11 acquires an image of the target user at the moment the answer is input from the portable terminal 2 (step S18).

[0077] Based on the sound input in step S17 and / or the image acquired in step S18, control unit 11 determines whether the subject's state is abnormal (step S19). Specifically, control unit 11 determines whether the subject may have a brain dysfunction. For example, similar to step S14, control unit 11 converts the input sound into text and divides it into multiple strings, compares them with each word in a vocabulary dictionary, and determines whether there is a string corresponding to the abnormality. In addition, control unit 11 identifies the subject's face from the image captured in step S18 and determines whether the left and right sides of the face are asymmetrical and / or dynamic. If it is determined that there is no abnormality (S19: No), control unit 11 moves the processing to step S15.

[0078] If an anomaly is detected (S19: Yes), the control unit 11 displays a message (text) corresponding to the anomaly in a different display format than other strings on the other user's portable terminal 2 (step S20). Specifically, as described above, the control unit 11 changes the display color of the string corresponding to the anomaly and also changes the background color of the string. Additionally, the control unit 11 displays an object 61 for the sound input by the user who is reproducing the anomaly.

[0079] The control unit 11 determines whether to reproduce the input sound in accordance with the operation input to the target 61 (step S21). If it is determined that the input sound should be reproduced (S21: Yes), the control unit 11 reproduces the sound input by the target on the other user's portable terminal 2 (step S22). After performing the processing in step S22, or if it is determined that it is no in step S21, the control unit 11 ends the series of processing.

[0080] Furthermore, in this embodiment, for ease of explanation, the server 1 performs the text conversion of the input sound, anomaly detection, and other processing. However, the local portable terminal 2 (or speaker terminal 3) may also perform some or all of the processing. For example, the portable terminal 2 may perform text conversion, while the server 1 may perform anomaly detection. Thus, the processing entity in this series of processes is not particularly limited.

[0081] According to Embodiment 1, the voice input by the user is converted into text to detect anomalies. The string corresponding to the anomaly is displayed in a different format than other strings and then displayed to other users. This allows other users to easily understand the user's anomaly.

[0082] Furthermore, according to Embodiment 1, it is possible to detect abnormalities in the subject from the subject's normal conversational voice, such as conversation messages in a chat group or voice input to a chatbot.

[0083] Furthermore, according to Embodiment 1, by referring to the text of the target's past input voice, the detection accuracy of anomalies can be improved.

[0084] Furthermore, according to Embodiment 1, when an anomaly is detected, a question about the anomaly is output and an answer is received, which can more accurately determine whether the state of the subject is abnormal.

[0085] Furthermore, according to Embodiment 1, by judging the left-right asymmetry of the face from the image of the subject from the time of shooting to the time of responding to input, it is possible to more accurately judge the abnormalities of the subject related to brain dysfunction.

[0086] Furthermore, according to this embodiment 1, by changing the display format of the string corresponding to the abnormality in accordance with the answer to the question and / or the state of the subject determined from the image of the subject, it is possible to indicate the severity of the abnormality to other users.

[0087] Furthermore, according to Embodiment 1, other users can easily grasp the state of the subject by reproducing the voice input by the subject.

[0088] (Implementation Method 2)

[0089] In Embodiment 1, the method of detecting anomalies from the voice input of the subject was described. In this embodiment, when anomalies are detected, the method of inferring the possibility of having a brain dysfunction based on questions based on voice and text is described. Furthermore, the same reference numerals are used for content repeated in Embodiment 1, and descriptions are omitted.

[0090] Figure 8 This is a block diagram illustrating a configuration example of server 1 in Embodiment 2. The auxiliary storage unit 14 of server 1 in this embodiment stores a response history DB141. The response history DB141 is a database that stores the responses to the respondents who asked the questions, as described later, and inferences based on those responses regarding the likelihood of the respondents having a brain dysfunction.

[0091] Figure 9 This is an explanatory diagram showing an example of the record layout of the Response History DB141. The Response History DB141 includes a Date and Time column, a Subject column, a Voice column, a Text column, a Response column, an Inference Result column, and an Image column. The Date and Time column stores the date and time when the subject responded to the question. The Subject column, Voice column, Text column, Response column, Inference Result column, and Image column store, respectively, in association with the response date and time, the name of the subject who responded, whether the response to the voice-based question (the first question described later) was correct or not, whether the response to the text-based question (the second question described later) was correct or not, the subject's response to the question, the inference result of the possibility of brain dysfunction inferred from the response, and an image (e.g., video) of the subject taken during the response. In the Response column, in addition to the judgment results such as the left-right symmetry of the face, the dynamics of the fingers or the direction of the gaze, and facial expressions judged from the subject's image as described later, the response time from the output of the question to the input of the response is also stored.

[0092] Figure 10A as well as Figure 10B This is an explanatory diagram showing an example of the display screen of speaker terminal 3. Figure 11A as well as Figure 11B This is an explanatory diagram showing an example of a message display screen in Embodiment 2. Based on Figure 10A , Figure 10B , Figure 11A as well as Figure 11B This section provides an overview of the implementation method.

[0093] As described in Implementation 1, server 1 detects anomalies in the voice input from the user and alerts other users. In this implementation, server 1, upon detecting anomalies, asks questions based on both voice and text. Then, based on the user's response to the questions, server 1 infers the likelihood of the user having a brain dysfunction.

[0094] Specifically, server 1 outputs a first question based on sound and a second question based on text to speaker terminal 3, performing sound output and image display corresponding to each question. Figure 10A , Figure 10B The diagram illustrates screen examples for the first question and the second question. Server 1 displays answer options on speaker terminal 3 for each question and accepts answer input by accepting a screen operation that selects one of the displayed options.

[0095] In addition, questions can be asked via speaker terminal 3 in this embodiment, but they can also be asked via portable terminal 2.

[0096] In explanation Figure 10A as well as Figure 10B Previously, based on Figure 11A Explanation will be provided. In Figure 11A Zhongyu Figure 6 Similarly, the chat screen displayed on the portable terminal 2 of another user is illustrated. If an anomaly is detected in the voice text input by the target user, the portable terminal 2, as in Embodiment 1, displays the message from the target user whose anomaly was detected.

[0097] In this embodiment, when an anomaly is detected, server 1 receives input from other users via this screen, including messages intended for the target. The message content is not particularly limited, but it is preferable to accept input including images. Figure 11A In the example, as a message sent to the recipient, the input included images of the recipient's close relatives (such as grandchildren).

[0098] Server 1 parses messages input from other users and extracts data to generate the first and second questions. For example, Server 1 extracts proper nouns from the text (such as names, etc.). Figure 11A as well as Figure 11B The example extracts the grandson's name "Taro" and an image. Server 1 generates the first and second questions based on the extracted data and outputs them to speaker terminal 3.

[0099] Return to Figure 10A as well as Figure 10B Continuing the explanation. Server 1 first generates a first sound-based question and outputs it to speaker terminal 3. For example, server 1... Figure 10A As shown, images extracted from messages from other users and other images different from those images are output to speaker terminal 3 for display, and a sound is output to prompt the selection of which image to use for the screen operation.

[0100] For example, server 1 extracts the image area of ​​the person (grandson) from the image retrieved from the message and generates a thumbnail image, which is then displayed on speaker terminal 3. Alternatively, server 1 can display a pre-prepared, unrelated image as an alternative. Furthermore, in Figure 10A In this example, two images are displayed, but more than three are also possible. Additionally, in this embodiment, images input by other users are displayed, but for example, images could be prepared (entered) in the database beforehand for each user and displayed as prepared images. Server 1 will generate a sound that facilitates the selection of an image of the person corresponding to the proper noun (grandson's name) extracted from the message, using a template, and output it to speaker terminal 3.

[0101] Server 1 accepts input in response to the first question. Specifically, server 1 accepts a screen operation that allows selection of one of multiple images displayed on speaker terminal 3. Alternatively, input can be received in the form of sound.

[0102] Given an answer to the first question, server 1 outputs a text-based second question. For example, server 1... Figure 10B As shown, a prompt appears asking whether you want to view the image (photo), and a button is displayed for selecting whether to view it. Additionally, in Figure 10B The diagram in the middle shows Figure 10A In cases where the correct answer image (the image of the grandson) is selected in the screen, the question "Want to see the picture?" is displayed, but... Figure 10A If the wrong image is selected in the picture, the question "Don't you want to see the picture?" will be displayed as a prompt.

[0103] Server 1 accepts screen input from users choosing between "view photos" or "don't view photos." If "view photos" is selected, Server 1 outputs messages from other users to speaker terminal 3. Specifically, Server 1 displays images input by other users on speaker terminal 3. In addition to processing images, text can also be displayed. If "don't view photos" is selected (or no button is pressed), Server 1 waits for a specified time before processing, ending the process without displaying any messages after the specified time.

[0104] Server 1 determines whether the answers to the first and second questions are correct. Then, based on the correctness of the answers to the first and second questions, Server 1 infers the likelihood of having a brain dysfunction. Specifically, Server 1 infers the likelihood of having a brain dysfunction based on combinations of correct and incorrect answers, and infers the possible types of brain dysfunction.

[0105] There are no particular limitations on the brain dysfunction that can be used as the object of inference. In this embodiment, aphasia and dementia (or transient cognitive impairment based on cerebral infarction, etc.) are used as the objects of inference. Server 1 infers the possibility of having aphasia and the possibility of having dementia based on the combination of correct and incorrect answers.

[0106] Specifically, if server 1 provides an incorrect answer to the first voice-based question but a correct answer to the second text-based question, it infers the possibility of aphasia. Conversely, if server 1 provides incorrect answers to both the first and second questions, it infers the possibility of dementia. Furthermore, if both the first and second questions are answered correctly, it infers normalcy; if only the second question is answered incorrectly, it is treated as an occasional error.

[0107] Aphasia and dementia are often confused, but aphasia is a disorder that impairs language ability, while dementia is a disorder that usually includes cognitive abilities, including non-verbal abilities. Correspondingly, responses to sound and text differ for each symptom. Therefore, in this embodiment, a first sound-based question and a second text-based question are asked, and aphasia and dementia are identified by the combination of correct and incorrect answers to each question.

[0108] In addition to answering the first and second questions, server 1 infers the likelihood of the subject having a brain dysfunction based on images captured during the answering process. For example, speaker terminal 3 captures an image of the subject while outputting the first and / or second questions. Server 1 obtains the answers to each question from speaker terminal 3 and uses images captured during the answering process for inference.

[0109] For example, similar to Implementation 1, Server 1 infers the likelihood of a brain dysfunction based on the left-right asymmetry of the subject's face. That is, Server 1 divides the facial region in the image into left and right regions, determines the state (coordinates of feature points such as the eyes and mouth) and dynamics (movement of feature points) of each region, and judges whether the left-right state and / or dynamics of the face are asymmetrical. Thus, Server 1 can detect situations of brain dysfunction caused by conditions such as cerebral infarction.

[0110] In this embodiment, in addition to the asymmetry of the face, server 1 also determines from the image whether the subject is having difficulty answering questions, thereby inferring the possibility of having a brain dysfunction. Specifically, as described below, server 1 detects specific phenomena corresponding to a difficult state from the image.

[0111] For example, server 1 detects the subject's hand (finger) or gaze direction from an image and determines whether the subject's hand or gaze movement corresponds to a specific movement. Specifically, server 1 detects the subject's hand or gaze movement as they hesitate between options (images in the first question, buttons in the second question) because they are hesitant in choosing an option. In the case of, for example, Broca's aphasia, it is observed that when given a verbal command to correctly select from multiple options, the subject hesitates due to difficulty understanding the command. Therefore, server 1 detects the hand or gaze direction from the image during an answer to, for example, a voice-based first question, and determines whether the hand or gaze movement is moving between images, thereby inferring the possibility of aphasia.

[0112] Additionally, server 1 can also recognize the facial expressions of the subject and determine whether they correspond to specific expressions (discomfort, anxiety, annoyance, etc.). Similarly, in this case, it can determine whether the subject is experiencing difficulty answering questions.

[0113] Alternatively, server 1 can also determine a difficult state by inferring the subject's biometric information from an image. Biometric information includes, for example, the degree of pupil dilation, pulse, facial temperature (body temperature), and blood flow velocity. Server 1 infers this biometric information from the image and detects changes in this information (such as pupil dilation, increased pulse rate, etc.) to determine whether the subject is in a difficult state.

[0114] The above method determines whether a difficult state is being experienced based on a pattern. However, server 1, for example, can also determine this based on the response time to a question, in addition to an image. Specifically, server 1 measures the response time from the output of a question (e.g., the first question) to the input of a response to determine whether the response time exceeds a predetermined threshold. Thus, since a difficult state is being experienced, the situation of time spent responding can be detected.

[0115] As described above, in addition to the answers to the first and second questions, server 1 infers the possibility of brain dysfunction based on the captured image of the subject and / or the timing of the responses. For example, even if server 1 infers that the subject is normal because both answers to the first and second questions are correct, it may still infer the possibility of brain dysfunction if there is asymmetry in the left and right facial movements and / or state, or if the subject is judged to be in a difficult state. Alternatively, even if the subject's answer to the first question is correct but the answer to the second question is incorrect, and this is treated as an occasional error, server 1 may still modify the question and re-ask if there is asymmetry in the left and right facial movements and / or state, or if the subject is judged to be in a difficult state.

[0116] Server 1 outputs the inference results to other users' portable terminals 2 and displays them. Figure 11B The diagram illustrates the chat window when the inference results are displayed. For example, server 1 displays text representing the inference result (judgment result) and also shows a score that represents the numerical value of the inference result.

[0117] Server 1 calculates scores for "voice" corresponding to the first question, "text" corresponding to the second question, and "response" corresponding to the image and response time, and displays these scores on portable terminal 2. The method for calculating these scores is not particularly limited. For example, Server 1 may statistically analyze the correctness of responses to the first and second questions over a specified period (e.g., one week), calculate scores evaluating voice recognition and text recognition capabilities (e.g., the correct response rate within the specified period), and output these scores as "voice" and "text" scores, respectively. Additionally, Server 1 may calculate the degree of difficulty based on the image and / or response time, and output this as a "response" score.

[0118] Figure 12 This is another illustrative diagram showing a chat screen when the inference results are displayed. In Figure 12 The image illustrates a chat window display when a high probability of brain dysfunction is inferred. In such a case, server 1 notifies other users' portable terminals 2 of the inference result, which is then displayed on the chat window.

[0119] Specifically, server 1 and Figure 11B Similarly, the numerical score of the inference result is displayed, along with text indicating a high probability of having a brain dysfunction. For example, server 1... Figure 12 As shown, the types of brain dysfunctions that are presumed to be highly probable are illustrated, along with evaluations that encourage seeking medical attention.

[0120] In addition, server 1 can send notifications not only to other users (family members, etc.) associated with the subject, but also to the subject himself / herself, informing him / her of the inference that he / she may have a brain dysfunction.

[0121] Server 1 also displays link 121 in the chat window for viewing (confirming) the recipient's answer history. Link 121 is for outputting (displaying) historical information, showing the recipient's past answers to the first and second questions, as well as inferences about the possibility of having a brain dysfunction. Link 121 is used for migration to... Figure 13 The object of the history screen. Upon receiving operation input to link 121, portable terminal 2 migrates to... Figure 13 Historical footage.

[0122] Furthermore, it can be set so that even if the state of the inferred object is normal ( Figure 11B You can also view the history information under the current screen. In addition, you can not only switch from the chat screen, but also set it so that you can view the history information at any time.

[0123] Figure 13 This is an explanatory diagram showing an example of a history display screen. The history display screen shows a series of historical information, including whether the responses to the first and second questions were correct, images of the respondent taken during the responses, and inferences about brain dysfunction based on the responses and images. Server 1 stores various historical information in the response history DB141 and outputs the historical information in accordance with requests from the portable terminal 2. For example, the history display screen includes an response history table 131, an image display bar 132, and a score chart 133.

[0124] The answer history table 131 is a table that displays at each past time point whether the answers to the first and second questions ("audio" and "text") were correct or not, the degree of difficulty of the subject judged based on the images taken at the time of the answers ("response" score), and the inference result of brain dysfunction ("judgment"). In addition, the answer history table 131 displays a play button 1311 for playing back the captured images (video) in association with each time point.

[0125] Image display bar 132 is a display bar for showing images of the subject taken when answering the first and / or second questions. Upon receiving input to the playback button 1311, the portable terminal 2 displays the image (video) taken at the corresponding time point.

[0126] Fractional chart 133 is presented as a time series. Figure 11B, Figure 12 The portable terminal 2 displays a graph (e.g., a line graph) showing the scores for "voice" (evaluating voice recognition ability based on the correctness of the answer to the first question), "text" (evaluating character recognition ability based on the correctness of the answer to the second question), and "reaction" (evaluating the subject's state based on captured images, etc.), thus indicating changes in the subject to the user.

[0127] As described above, server 1 outputs a first and a second question when it detects an anomaly in the subject's voice. Based on the correctness of the answers to each question and the images accompanying the answers, it infers the likelihood of brain dysfunction. This allows for early detection of abnormalities in the subject and the analysis of brain dysfunction.

[0128] Figure 14 as well as Figure 15 This is a flowchart illustrating an example of the processing sequence performed by server 1 in Embodiment 2. After performing the processing in step S22, or if no such processing occurred in step S21, server 1 performs the following processing.

[0129] The control unit 11 of server 1 receives input messages, including images, from other users (step S23). The control unit 11 parses the message, extracts the image contained in the message, and extracts proper nouns, etc., from the text (step S24).

[0130] Based on the message parsed in step S24, control unit 11 outputs a first voice-based question to speaker terminal 3 (step S25). For example, control unit 11 displays a thumbnail image containing a portion of an image and another image different from that image as options on speaker terminal 3, and outputs a sound prompting the user to select which image to choose. Control unit 11 accepts input for the answer to the first question (step S26). Specifically, as described above, control unit 11 accepts input to select one of the multiple displayed images (options).

[0131] The control unit 11 outputs a text-based second question to the speaker terminal 3 (step S27). For example, the control unit 11 displays a button for selecting whether to view an image as an option, and displays text asking whether to view the image selected in step S26 on the speaker terminal 3. The control unit 11 accepts input for the answer to the second question (step S28). For example, the control unit 11 accepts operation input to select one of the multiple buttons (options) displayed.

[0132] The control unit 11 determines whether the answer to the second question is a correct answer (step S29). For example, the control unit 11 determines whether the selection input indicating viewing an image has been accepted. If the answer is determined to be correct (S29: Yes), the control unit 11 outputs the message (image) from the other user to the speaker terminal 3 (step S30).

[0133] If the answer is determined to be incorrect (S29: No), the control unit 11 determines whether a predetermined time has elapsed since the second question was output (step S31). If the predetermined time has not elapsed (S31: No), the control unit 11 returns to step S29. After executing the processing in step S30, or if the result in step S31 is yes, the control unit 11 captures an image of the recipient when obtaining the answer from the speaker terminal 3 in steps S26 and / or S28 (step S32).

[0134] Based on the answers to the first and second questions, the image of the subject at the time of the answers, and / or the time of the answers, the control unit 11 infers the likelihood that the subject has a brain dysfunction (step S33). Specifically, the control unit 11 infers the likelihood of having a brain dysfunction and infers the type of brain dysfunction (aphasia and dementia). For example, if the control unit 11 answers the first question incorrectly but answers the second question correctly, it infers a high probability that the subject has aphasia. Conversely, if the control unit 11 answers both the first and second questions incorrectly, it infers a high probability that the subject has dementia.

[0135] Furthermore, the control unit 11 determines whether the left and right sides of the face and / or dynamics are asymmetrical based on the image of the person answering. Additionally, the control unit 11 determines whether the person is in a difficult state based on the image of the person and / or the time of answering. Even if the answers to the first and second questions are inferred to be normal, the control unit 11 infers the possibility of a brain dysfunction based on the left and right asymmetry of the face and / or the judgment result of a difficult state. The control unit 11 stores the correctness of the answers to the first and second questions, the image of the person answering, and the inference result of the possibility of a brain dysfunction in the answer history DB141 (step S34).

[0136] The control unit 11 outputs the inference result to the portable terminal 2 of other users (step S35). For example, the control unit 11 displays the inference result of the possibility of whether the person has a brain dysfunction, and calculates a score for the subject based on the answer to the first question (voice), the answer to the second question (text), the image of the subject at the time of the answer, and / or the answer time, and displays it on the portable terminal 2.

[0137] Control unit 11 determines whether to output historical information showing the subject's past answers to the first and second questions, and the inference of the possibility of having a brain dysfunction based on the answers (step S36). For example, control unit 11 determines whether to accept... Figure 12 The example shown is the operation input to link 121 in the chat screen. If it is determined that historical information should be output (S36: Yes), the control unit 11 outputs and displays the historical information to the portable terminal 2 of other users (step S37). Specifically, as described above, in addition to past answers to the first and second questions at various points in time, and the inference results of brain dysfunction, the control unit 11 also displays images of the subject as historical information. After executing step S36, or if step S36 is not performed, the control unit 11 ends the series of processes.

[0138] Furthermore, while the above describes accepting messages from other users as input and outputting the first and second questions, this embodiment is not limited to this. For example, server 1 could also be configured to output the first and second questions to speaker terminal 3 at regular intervals and accept responses, regardless of whether there are messages from other users. In this case, as long as server 1 has prepared images for the questions in the database beforehand (in the above example, an image of a grandson), the first and second questions can be generated using those images. In this way, the first and second questions can be output regardless of whether there are messages from other users.

[0139] According to this embodiment 2, by asking a first question based on sound and a second question based on text, the likelihood of having a brain dysfunction can be appropriately inferred.

[0140] Furthermore, according to this embodiment 2, it is possible to infer the type of brain dysfunction (appropriately, aphasia and dementia) based on the combination of correct and incorrect answers to each question.

[0141] Furthermore, according to Embodiment 2, by displaying answer options on the speaker terminal 3 and accepting answer input via screen operation, answer input can be appropriately facilitated even in cases of impaired recognition ability due to brain dysfunction.

[0142] Furthermore, according to Embodiment 2, questioning begins when an abnormality is detected in the subject's voice. This allows for the early detection of brain dysfunction.

[0143] Furthermore, according to this embodiment 2, first and second questions are generated from messages from other users who are the counterparts in the dialogue. This allows for the creation of questions tailored to the target audience.

[0144] Furthermore, according to Embodiment 2, in addition to answering the question itself, the likelihood of having a brain dysfunction is inferred based on the image of the person answering and / or the time of answering. Therefore, it is possible to detect states such as cerebral infarction (facial asymmetry) or difficulty answering, and to more accurately infer the likelihood of a brain dysfunction.

[0145] The embodiments described herein should be considered illustrative in all respects and not limiting. The scope of the invention is not limited by the foregoing, but is shown by the claims and includes all modifications within the meaning and scope equivalent to the claims.

[0146] Explanation of reference numerals in the attached figures

[0147] 1. Server (Information Processing Device)

[0148] 11 Control Department

[0149] 12 main storage units

[0150] 13Ministry of Communications

[0151] 14 Auxiliary Storage Section

[0152] P1 program

[0153] 141 answers history DB

[0154] 2 portable terminals

[0155] 21 Control Department

[0156] 22 Main Storage Section

[0157] 23 Ministry of Communications

[0158] 24 Display Unit

[0159] 25 Input Section

[0160] 26 Audio Output Section

[0161] 27. Audio Input Section

[0162] 28th Filming Department

[0163] 29 Auxiliary Storage Section

[0164] P2 program

[0165] 3-speaker terminal

[0166] 31 Control Department

[0167] 32 main storage units

[0168] 33 Ministry of Communications

[0169] 34 Display Unit

[0170] 35 Input Section

[0171] 36 sound output section

[0172] 37. Audio Input Section

[0173] 38th Photography Department

[0174] 39 Auxiliary Storage Section

[0175] P3 program.

Claims

1. A program product comprising a program that causes a computer to perform the following processes: Use sound to output the first question. Receive the answer to the first question from the recipient. The second question is output using text. The user receives input from the object to answer the second question. Determine whether the answers to the first question and the second question are correct. Based on the correctness of the answers to the first and second questions, the likelihood of the subject suffering from a brain dysfunction is inferred. If the answer to the first question is incorrect and the answer to the second question is correct, it is inferred that there is a possibility of aphasia. If both the first and second questions are answered incorrectly, it is inferred that there is a possibility of dementia.

2. The program product according to claim 1, wherein, The options for answering the first question and the second question will be displayed on the display unit. Input is received to answer the first question and the second question by accepting a screen operation that allows you to select one of the displayed options.

3. The program product according to claim 1 or 2, wherein, Obtain the text converted from the voice input by the subject. Anomalies were detected in the text. If the anomaly is detected, output the first question and the second question.

4. The program product according to claim 3, wherein, The target user receives audio input for messages sent to a chat group containing multiple users, including the target user. Convert the message into the text and detect any anomalies. If the anomaly is detected, the text corresponding to the anomaly will be displayed on other users' terminal devices in a display format different from other strings.

5. The program product according to claim 4, wherein, Receive message input from the terminal device of the other user that displays the text. The first question and the second question are generated based on the messages from the other users. Output the first question and the second question generated.

6. The program product according to claim 1 or 2, wherein, Acquire images of the person responding to the first question or the second question. Based on the correctness of the answers to the first and second questions, and the image, the likelihood of having the brain dysfunction is inferred.

7. The program product according to claim 6, wherein, Based on the image, determine whether the left and right sides of the subject's face are asymmetrical in terms of movement or state. If the condition is determined to be asymmetrical, it is inferred that there is a possibility of having the aforementioned brain dysfunction.

8. The program product according to claim 7, wherein, Based on the image, determine whether the subject is having difficulty answering. If a person is judged to be having difficulty answering questions, it is inferred that they may have the aforementioned brain dysfunction.

9. The program product according to claim 6, wherein, Based on the image, determine whether the subject is having difficulty answering. If a person is judged to be having difficulty answering questions, it is inferred that they may have the aforementioned brain dysfunction.

10. The program product according to claim 1 or 2, wherein, The response time is measured from the time the first question or the second question is output until the input for a response is received. Based on the correctness of the answers to the first and second questions, and the time taken to answer them, the likelihood of having the brain dysfunction is inferred.

11. The program product according to claim 1 or 2, wherein, The inference of the likelihood of the subject having the aforementioned brain dysfunction shall be communicated to the subject or other users related to the subject.

12. The program product according to claim 1 or 2, wherein, The answers to the first and second questions, as well as the inferences inferred from the correctness of the answers, indicating the presence of the aforementioned brain dysfunction, are stored in the storage unit. The output represents historical information that shows the history of the answer and the inference results.

13. An information processing apparatus comprising: The first output unit of the first question is output using sound. The first receiving unit receives the answer to the first question from the recipient; Use text to output the second output part of the second question; The second receiving unit receives operational input from the recipient regarding the answer to the second question; The judgment unit determines whether the answers to the first question and the second question are correct; as well as The inference unit, based on the correctness of the answers to the first and second questions, infers the likelihood that the subject suffers from a brain dysfunction. If the inference unit determines that the person has aphasia when the answer to the first question is incorrect and the answer to the second question is correct, the inference unit infers that the person may have aphasia. If the inference unit provides incorrect answers to both the first and second questions, it infers that the person may have dementia.

14. An information processing method, wherein a computer performs the following processing: Use sound to output the first question. Receive the answer to the first question from the recipient. The second question is output using text. The user receives input from the object to answer the second question. Determine whether the answers to the first question and the second question are correct. Based on the correctness of the answers to the first and second questions, the likelihood of the subject suffering from a brain dysfunction is inferred. If the answer to the first question is incorrect and the answer to the second question is correct, it is inferred that there is a possibility of aphasia. If both the first and second questions are answered incorrectly, it is inferred that there is a possibility of dementia.

Images