Estimation device and estimation method

Abstract

This study not only estimates the interpersonal responsiveness of individuals with ASD in conversations that include those with ASD, but also estimates the presence or absence of behaviors that cause miscommunication during conversations. [Solution] The estimation device receives input of utterance information from a group conversation including the person to be estimated. The estimation device then uses the utterance information from the group conversation as input and estimates the behavioral characteristics of the person to be estimated using a model trained to estimate at least one of the following behavioral characteristics: the low level of engagement of the speakers in the conversation, the degree to which turn-taking is not smooth, the degree to which statements are out of context, and the presence or absence of ASD characteristics in nonverbal and verbal behaviors. The estimation device then outputs the behavioral characteristics of the person to be estimated in the conversation. Furthermore, based on the utterance information from the group conversation and the estimation results of the behavioral characteristics of the speakers in the conversation, the estimation device estimates and outputs the interpersonal responsiveness of the person to be estimated in the conversation.

Description

Technical Field

[0001] The present invention relates to an estimation device and an estimation method for estimating the presence or absence of behavioral characteristics causing miscommunication in individuals with ASD (Autism Spectrum Disorder) and interpersonal responsiveness.

Background Art

[0002] Conventionally, there is a technique for estimating the value of SRS-2 (Social Responsiveness Scale Second Edition, a score for interpersonal responsiveness for determining ASD) using video data of monologues.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in order to provide practical support for individuals with ASD (for example, developing conversational intervention agents that participate in conversations including individuals with ASD, and identifying miscommunicative behaviors towards others arising from the characteristics of individuals with ASD), it is necessary not only to estimate the interpersonal responsiveness of individuals with ASD, but also to detect behaviors that cause miscommunication during conversations. However, until now, no technology has been proposed to estimate not only the interpersonal responsiveness of individuals with ASD, but also the presence or absence of behaviors that cause miscommunication during conversations.

[0005] Therefore, the present invention aims to solve the aforementioned problems and to estimate not only the interpersonal responsiveness of individuals with ASD, but also the presence or absence of behaviors that cause miscommunication during conversations. [Means for solving the problem]

[0006] To solve the aforementioned problems, the present invention is characterized by comprising: an input unit that receives input of utterance information of a group conversation including a target person; a first estimation unit that takes the utterance information of the group conversation as input and estimates and outputs the behavioral characteristics of the target person in the input conversation using a model that has been trained to output estimation results of at least one of the following behavioral characteristics: low engagement of the speaker in the conversation, degree to which turn changes are not smooth, degree to which statements are out of context, and presence or absence of ASD characteristics in nonverbal and verbal behavior; and a second estimation unit that takes the utterance information of the group conversation and the estimation results of the behavioral characteristics of the speaker in the conversation as input and estimates and outputs the interpersonal responsiveness of the target person in the conversation using a model that has been trained to output estimation results of the interpersonal responsiveness of the speaker. [Effects of the Invention]

[0007] According to the present invention, in conversations involving individuals with ASD, it is possible not only to estimate the interpersonal responsiveness of individuals with ASD, but also to detect behaviors that cause miscommunication during the conversation. [Brief explanation of the drawing]

[0008] [Figure 1]Figure 1 is a diagram illustrating the overview of the estimation device. [Figure 2] Figure 2 shows an example of the configuration of the estimation device. [Figure 3] Figure 3 shows an example of a network for realizing the behavioral characteristics estimation unit. [Figure 4] Figure 4 shows an example of a network for realizing a human response estimation unit. [Figure 5] Figure 5 shows an example of prosodic features. [Figure 6] Figure 6 shows an example of facial features. [Figure 7] Figure 7 is a flowchart showing an example of the processing steps performed by the estimation device. [Figure 8] Figure 8 shows an example of a computer running the estimation program. [Modes for carrying out the invention]

[0009] The following describes embodiments for carrying out the present invention with reference to the drawings. The present invention is not limited to these embodiments.

[0010] [overview] The overview of the estimation device of this embodiment will be explained using Figure 1. The estimation device uses speech information from a conversation of a group including an ASD person (for example, voice data and text data of the ASD person in the conversation, and video data of each member of the group including the ASD person) to estimate the presence or absence of behavioral characteristics that cause miscommunication in the ASD person. Furthermore, the estimation device uses the speech information from the conversation of the group including the ASD person and the estimation result of the presence or absence of the behavioral characteristics that cause miscommunication in the ASD person to estimate the interpersonal responsiveness score of the ASD person.

[0011] Here, the estimation device of this embodiment characterizes the communication behavior of an ASD person by, for example, three aspects of conversation interaction (low engagement, degree of lack of smooth turn-taking, degree of non-contextual speech) and two ASD characteristics (ASD characteristics in verbal behavior and ASD characteristics in non-verbal behavior).

[0012] For example, from the utterance information of a conversation of a group including an ASD person, the estimation device estimates, as the behavior characteristics (hereinafter, appropriately referred to as "behavior characteristics") that cause miscommunication of the ASD person, the low engagement in the conversation of the ASD person, the degree of lack of smooth turn-taking, the degree of non-contextual speech, the presence or absence of ASD characteristics in non-verbal behavior and verbal behavior, and other behavior characteristics.

[0013] By outputting the estimation results of the behavior characteristics of an ASD person as described above, for example, it can be used for the development of a conversation intervention agent for a conversation including an ASD person, the development of an application for pointing out miscommunication behavior to an ASD person, and the like.

[0014] In addition, the estimation device can improve the estimation accuracy of the interpersonal responsiveness score by using not only the utterance information of a conversation of a group including an ASD person but also the estimation results of the behavior characteristics of the ASD person in the estimation of the interpersonal responsiveness score of the ASD person.

[0015] Note that the estimation device uses, for example, a machine learning technique such as a deep learning technique to estimate the above-mentioned interpersonal responsiveness and behavior characteristics of an ASD person.

[0016] Here, in the estimation of the behavior characteristics of an ASD person, the estimation device uses, for example, a machine learning model that considers not only the behavior of the ASD person himself / herself but also the behavior of others (for example, the facial expressions of other members of the conversation group). Thereby, the estimation device can improve the estimation accuracy of the behavior characteristics of an ASD person.

[0017] [Configuration Example] Next, a configuration example of the estimation device 10 will be described using FIG. 2. The estimation device 10 includes, for example, an input / output unit 11, a storage unit 12, and a control unit 13.

[0018] The input / output unit 11 is an interface that controls the input and output of various data. The input / output unit 11 receives, for example, the utterance information of conversations of a group including ASD individuals (e.g., the utterance data (voice data), text data, and video data of the faces of each member of the group) of ASD individuals in the conversation. Further, the input / output unit 11 outputs the estimation results of the interpersonal responsiveness and behavioral characteristics of ASD individuals.

[0019] The storage unit 12 and the control unit 13 are realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. For example, the storage unit 12 stores the utterance information of conversations of a group including ASD individuals received by the input / output unit 11, the parameters of the machine learning model used by the control unit 13, and the like.

[0020] The control unit 13 controls the entire estimation device 10. The function of the control unit 13 is realized, for example, when a CPU (Central Processing Unit) executes a program stored in the storage unit 12.

[0021] The control unit 13 includes an input unit 131, a behavioral characteristic estimation unit (first estimation unit) 132, and an interpersonal response estimation unit (second estimation unit) 133.

[0022] The input unit 131 receives the input of the utterance information of conversations of a group including an ASD individual (estimation target). The utterance information includes, for example, the voice information of the estimation target (e.g., the prosodic feature amount in each utterance), the language information of the estimation target (the text data of each utterance), the video data of the faces of each member of the conversation group, and the like.

[0023] The behavioral characteristics estimation unit 132 estimates and outputs the behavioral characteristics of the target person in the input group conversation from the utterance information of the conversation. For example, the behavioral characteristics estimation unit 132 takes the utterance information of the group conversation as input and uses a model trained to output the results of estimating the behavioral characteristics of the speaker in the conversation (e.g., low level of engagement in the conversation, degree of unsmooth turn transitions, degree of incontextual remarks, presence or absence of ASD characteristics in nonverbal and verbal behavior) to estimate and output the behavioral characteristics of the target person in the input conversation.

[0024] Furthermore, the training of the above model will use a dataset that includes not only the behavior of the individual with ASD but also the verbal and nonverbal behaviors of others. For example, the dataset used to train the above model will be video data of the faces of all members, including the individual with ASD, during a conversation. By doing so, it is possible to construct a model with high performance in estimating the behavioral characteristics of the target individual.

[0025] The interpersonal response estimation unit 133 uses the utterance information of the input group conversation and the behavioral characteristics of the estimated subject estimated by the behavioral characteristics estimation unit 132 to estimate the interpersonal responsiveness of the estimated subject and output it.

[0026] For example, the interpersonal response estimation unit 133 takes the utterance information of the group conversation and the estimated results of the behavioral characteristics of the speakers in the conversation as input, and uses a model trained to output the estimated result of the interpersonal responsiveness of the speaker to estimate and output the interpersonal responsiveness score of the person to be estimated in the conversation. The interpersonal responsiveness score is, for example, an evaluation value of interpersonal responsiveness from an existing psychological test, such as the SRS-2 test score.

[0027] [Specific example of the behavioral characteristic estimation unit] A specific example of the behavioral characteristics estimation unit 132 is described below. The information (speech information) input to the behavioral characteristics estimation unit 132 includes, for example, the following three pieces of information. • Speech information: Prosodic features in each utterance of the estimated subject. • Language information: Transcript of each utterance of the estimated subject. • Facial information: Facial images of all participants in the conversation (group members).

[0028] The behavioral characteristics estimation unit 132 performs, for example, the following processes. Here, we will explain using the case where the behavioral characteristics estimation unit 132 is implemented by the network shown in Figure 3 as an example.

[0029] The network shown in Figure 3 takes the actions of all participants in a conversation (for example, three people) as input and determines whether the target behavioral characteristics were detected for each utterance of the estimated target person.

[0030] First, the behavioral characteristic estimation unit 132 vectorizes each input information (language information, voice information, and facial information) shown in Figure 3 using an encoder.

[0031] For example, linguistic information is applied to a pre-trained deep learning language model for transcribing each utterance, and the resulting representation of each utterance is converted into a 768-dimensional vector. The language model used is, for instance, RoBERTa, which has been trained on Japanese text.

[0032] The linguistic information may be vectorized using methods other than those described above. For example, BERT may be used as a pre-trained linguistic model. Alternatively, if a pre-trained model is not used, the transcript of utterances may be converted into vectors using part-of-speech information such as the number of nouns and verbs. Furthermore, the number of dimensions is not limited to 768 dimensions, but may depend on the number of dimensions of the vectors obtained by vectorizing the linguistic data.

[0033] The speech information is converted into a 384-dimensional vector of prosodic features (including pitch, power, MFCC, etc.) for each utterance. For example, OpenSMILE can be used to extract the speech features. When using OpenSMILE, a configuration file such as IS09_emotion.conf is used. Alternatively, other toolkits besides OpenSMILE may be used for extracting the speech features. Furthermore, the speech information may be vectorized using a pre-trained model such as HuBERT or wav2vec.

[0034] Facial information is processed, for example, by a video-based facial tracking tool and converted into a 709-dimensional feature vector for each frame. The behavioral characteristic estimation unit 132 then combines the feature vectors of all participants to obtain an n-person × 709-dimensional vector. For example, if there are a total of 3 conversation participants, the feature vectors of all participants will be a 2127-dimensional vector. Alternatively, the behavioral characteristic estimation unit 132 may vectorize the facial information using a pre-trained model such as VGGFace or ArcFace.

[0035] Furthermore, the behavioral characteristic estimation unit 132 extracts data contained within each utterance interval from the start and end time information of the utterance and calculates the average value to obtain an average vector of feature quantities for each utterance interval. As a result, the facial features of all participants (regardless of whether they are speaking or not) in each utterance interval are represented as a 2127-dimensional vector.

[0036] Next, the behavioral characteristic estimation unit 132 applies a fully connected layer (FC layer) to each of the vectorized language, voice, and facial information, converting them into, for example, 200-dimensional vectors. These three vectors are then input to a Transformer layer, encoded, and then passed through a softmax layer to output a binary classification result (behavior detected / behavior not detected).

[0037] For example, the estimation device 10 uses the network described above to create a learning model for each of the five types of behavioral characteristics. The behavioral characteristic estimation unit 132 then inputs the speech information into each of the above models, integrates the results output from each model, and creates a five-dimensional output vector.

[0038] The five behavioral characteristics described above are defined as follows. The behavioral characteristics related to conversational interaction include, for example, the following three types. 1. Engagement: The speaker's engagement with the conversation is low, and their participation is inappropriate. Examples include: the speaker continuing to talk without any prior interaction, lack of response, not looking at the listener while speaking, etc. 2. Turn-taking: The speaker is not smoothly taking turns in the conversation. Examples include: the speaker interrupting another participant mid-sentence, starting to speak over them, or starting to speak abruptly (without any preamble). 3. Conversational context: The speaker's utterance does not fit the context of the conversation. Examples include: the speaker giving irrelevant responses or making unrelated statements to others' questions or comments, or drawing conclusions (making decisions unilaterally) without sufficient discussion.

[0039] Furthermore, behavioral characteristics related to ASD traits include, for example, the following two types: 4. ASD characteristics in nonverbal behavior: There is something unusual about the speaker's nonverbal behavior. Examples include: the speaker speaking too quickly to understand, having an unnatural tone of voice (for example, speaking in an angry manner), not hiding negative emotions, or speaking too loudly or too softly. 5. ASD characteristics in language behavior: The content of the speaker's speech is understandable, but it feels out of context. Examples include: frequent repetition by the speaker, unfinished speech, utterances consisting only of single words, and grammatically incorrect speech.

[0040] The behavioral characteristic estimation unit 132 outputs a 5-dimensional vector representing the detection results of the five types of behaviors described above. Each dimension has a value of 0 if the corresponding behavior is not detected, and 1 if the corresponding behavior is detected.

[0041] The behavioral characteristics estimation unit 132 may estimate all five types of behaviors as the behavioral characteristics of the person being estimated, or it may estimate only some of them.

[0042] [Specific example of the interpersonal response estimation unit] A specific example of the interpersonal response estimation unit 133 is described below. The information input to the interpersonal response estimation unit 133 includes, for example, the following four pieces of information. • Speech information: Prosodic features in each utterance of the estimated subject. • Language information: Transcript of each utterance of the estimated subject. • Facial information: Facial images of all participants in the conversation (group members). • Estimation results of the behavioral characteristics of the subject, output by the behavioral characteristics estimation unit 132.

[0043] The human response estimation unit 133 performs the following processes, for example. Here, we will explain using the case where the human response estimation unit 133 is implemented by the network shown in Figure 4 as an example.

[0044] The interpersonal response estimation unit 133 vectorizes each input piece of information (language information, voice information, facial information, and estimated results of the behavioral characteristics of the person being estimated) using an encoder.

[0045] For example, linguistic information is obtained by applying a transcription of each utterance of the person to a pre-trained language model using deep learning, similar to the method described in the specific example of the behavioral characteristic estimation unit 132, and converting each utterance into a 768-dimensional vector. For example, RoBERTa, trained on Japanese text, is used as the language model. The interpersonal response estimation unit 133 then calculates a vector by averaging the vectors of all utterances in the conversation.

[0046] Furthermore, the interpersonal response estimation unit 133 extracts, for example, 20-dimensional prosodic features (including pitch, power, MFCC, etc.) from the speech information for each utterance, as shown in Figure 5. For this extraction of speech features, for example, OpenSMILE is used. In this case as well, for example, IS09_emotion.conf is used as the OpenSMILE configuration file. The interpersonal response estimation unit 133 then calculates a vector obtained by averaging the speech feature vectors (feature vectors) of all utterances in the conversation. It should be noted that the interpersonal response estimation unit 133 can be expected to achieve better estimation performance by using features that correlate with interpersonal responsiveness as the above-mentioned speech features.

[0047] Furthermore, facial information is processed by a video-based face tracking tool, similar to the behavioral characteristic estimation unit 132 described above, and converted for each frame into a feature vector consisting of, for example, 55 feature quantities as shown in Figure 6. The interpersonal response estimation unit 133 then calculates a vector by averaging the feature vectors of all frames of the conversation. It should be noted that the interpersonal response estimation unit 133 can be expected to achieve better estimation performance by using features that correlate with interpersonal responsiveness as the facial features.

[0048] The estimated behavioral characteristics of the target individual are, for example, represented as a 5-dimensional vector, which can be transformed to any dimension (e.g., 10 dimensions) using any arbitrary method. This arbitrary method could be, for example, a transformation using a fully connected layer of a neural network.

[0049] The interpersonal response estimation unit 133 uses the vectorized language, voice, facial information, and estimated behavioral characteristics of the estimated subject as feature vectors for that subject. The interpersonal response estimation unit 133 then uses a pre-trained regression model (e.g., linear regression, XGBoost, etc.) to obtain an estimated interpersonal responsiveness score from the above-mentioned feature vectors of the estimated subject. The interpersonal response estimation unit 133 then outputs the estimated interpersonal responsiveness score for the estimated subject.

[0050] As explained in the specific example of the behavioral characteristics estimation unit 132, the vectorization method used by the interpersonal response estimation unit 133 is not limited to the method described above.

[0051] [Example of processing procedure] Next, an example of the processing procedure performed by the estimation device 10 will be explained using Figure 7. First, the input unit 131 of the estimation device 10 receives input of speech information from a group conversation including the person to be estimated (for example, language information and voice information of the person to be estimated, and facial information of all members of the group) (S1). Then, the behavioral characteristics estimation unit 132 estimates the behavioral characteristics of the person to be estimated based on the speech information received in S1 (S2). Next, the interpersonal response estimation unit 133 estimates the interpersonal responsiveness score of the person to be estimated based on the speech information received in S1 and the behavioral characteristics of the person to be estimated estimated in S2 (S3). Then, the behavioral characteristics estimation unit 132 outputs the estimated result of the behavioral characteristics of the person to be estimated obtained in S2, and the interpersonal response estimation unit 133 outputs the estimated result of the interpersonal responsiveness score of the person to be estimated obtained in S3 (S4).

[0052] By performing the above processing, the estimation device 10 can not only estimate the interpersonal responsiveness of individuals with ASD in conversations that include individuals with ASD, but also detect behaviors that cause miscommunication in individuals with ASD from the conversation.

[0053] [Other embodiments] Although the models used by the behavioral characteristic estimation unit 132 and the interpersonal response estimation unit 133 were described as pre-constructed models, they are not limited to this. For example, the estimation device 10 may have a function to learn the models used by the behavioral characteristic estimation unit 132 and the interpersonal response estimation unit 133 using a predetermined dataset.

[0054] [System configuration, etc.] Furthermore, the components of each part shown in the diagram are functional concepts and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown in the diagram, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. Moreover, all or any part of the processing functions performed by each device can be realized by a CPU and the program executed on that CPU, or by hardware using wired logic.

[0055] Furthermore, among the processes described in the embodiments described above, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified.

[0056] [program] The aforementioned estimation device 10 can be implemented by installing a program (estimation program) as packaged software or online software on a desired computer. For example, by having the above program run on an information processing device, the information processing device can function as the estimation device 10. The information processing device referred to here includes mobile communication terminals such as smartphones, mobile phones and PHS (Personal Handyphone System), as well as terminals such as PDA (Personal Digital Assistant).

[0057] Figure 8 shows an example of a computer running an estimated program. Computer 1000 has, for example, memory 1010 and a CPU 1020. Computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0058] Memory 1010 includes ROM (Read Only Memory) 1011 and RAM (Random Access Memory) 1012. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0059] The hard disk drive 1090 stores, for example, the OS 1091, application program 1092, program module 1093, and program data 1094. That is, the program that defines each process executed by the estimation device 10 is implemented as a program module 1093 in which executable code for a computer is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing a process similar to the functional configuration in the estimation device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0060] Furthermore, the data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes them.

[0061] Furthermore, the program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via a network interface 1070. [Explanation of symbols]

[0062] 10 Estimation device 11 Input / output section 12 Storage section 13 Control Unit 131 Input section 132 Behavioral Characteristics Estimation Unit 133 Interpersonal response estimation unit

Claims

1. An input unit that accepts input of spoken information from a group conversation including the target person, A first estimation unit takes utterance information from a group conversation as input and uses a model trained to output estimation results of the speaker's behavioral characteristics, indicating at least one of the following: low engagement of the speaker in the conversation, degree of poor turn-taking, degree of incongruity of statements, and presence or absence of ASD characteristics in nonverbal and verbal behavior, to estimate and output the behavioral characteristics of the target person in the input conversation. A second estimation unit takes the utterance information of the group conversation and the estimated results of the behavioral characteristics of the utterancers in the conversation as input, and uses a model trained to output the estimated results of the interpersonal responsiveness of the utterancers to estimate and output the interpersonal responsiveness of the target person in the conversation. An estimation device characterized by comprising:

2. The model used by the first estimation unit to estimate the behavioral characteristics of the person in question in the conversation is: This model is trained to take speech information, including video footage of each group member's face in the aforementioned conversation, as input, and to output an estimated result of the behavioral characteristics of the speaker in the aforementioned conversation. The estimation device according to feature 1.

3. An estimation method performed by an estimation device, A process for receiving input of spoken information from a group conversation including the target person, The process involves taking utterance information from a group conversation as input and using a model trained to output estimation results of the speaker's behavioral characteristics, such as the degree of low engagement, the degree of poor turn-taking, the degree of incongruity of statements, and the presence or absence of ASD characteristics in nonverbal and verbal behavior, to estimate and output the behavioral characteristics of the target person in the input conversation. The process involves using a model trained to take utterance information from a group conversation and estimated behavioral characteristics of the speakers in the conversation as input, and outputting an estimated result of the speaker's interpersonal responsiveness, to estimate and output the interpersonal responsiveness of the target person in the conversation. An estimation method characterized by including the following.

Images