Information processing device, information processing method, and program

Abstract

To propose an information processing device, an information processing method, and a program, which can accurately estimate the state of the other party.SOLUTION: In a communication support system, a terminal TA functioning as an information processing device has a stimulus recognition unit ST, a reaction prediction unit PR, a reaction recognition unit RE, and a state estimation unit ES. The stimulus recognition unit ST recognizes words and actions of a speaker that stimulate the other party in a conversation. The reaction prediction unit PR recognizes a reaction of the other party expected to a stimulus as an expected reaction. The reaction recognition unit RE recognizes a reaction of the other party actually performed to the stimulus as an actual reaction, and the state estimation unit ES estimates the state of the other party based on differences between the expected reaction and the actual reaction.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In real face-to-face communication, various information of the other party can be read from the speech, intonation, and expression of the other party in the conversation. The speaker can infer the state of the other party from the read information and pay attention to the other party to conduct smooth communication. In remote communication using a communication terminal, since the conversation is conducted through a screen, the speaker has difficulty reading the information of the other party for the following reasons.

[0003] (i) When the face of the other party reflected on the screen is small, it is difficult to read the expression and gestures of the other party. (ii) When a delay occurs in the transmission of information, the exact timing of the conversation cannot be grasped. (iii) There is a possibility of missing the expression of the other party due to being distracted by the operation of the communication terminal.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] There is a technology for inferring the state of the other party from the expression and intonation of the other party's words. However, in this type of technology, the state of the other party is inferred based on the level of absolute expressions and the like. Since there are individual differences in the level of expressions and the like to be detected, it is difficult to obtain accurate estimation results. For example, a person who has a habit of wrinkling their eyebrows when trying to understand an explanation is inferred to be angry.

[0006] Therefore, this disclosure proposes an information processing device, an information processing method, and a program that can accurately predict the state of the other party. [Means for solving the problem]

[0007] According to this disclosure, an information processing device is provided, comprising: a stimulus recognition unit that recognizes the speaker's words and actions that stimulate the other party in a dialogue; a response prediction unit that recognizes the expected response of the other party to the stimulus as a predicted response; a response recognition unit that recognizes the actual response of the other party to the stimulus as an actual response; and a state prediction unit that predicts the state of the other party based on the difference between the predicted response and the actual response. Furthermore, according to this disclosure, an information processing method is provided in which the information processing of the information processing device is performed by a computer, and a program is provided in which the information processing of the information processing device is performed by a computer.

[0008] Furthermore, the present disclosure provides an information processing device having: a data transmission unit that transmits the speaker's words and actions that stimulate the other party to the conversation partner; a data receiving unit that receives the state of the other party inferred from the stimulus; and a partner state presentation unit that presents the state of the other party to the speaker. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram of a conventional communication support system. [Figure 2] This is a block diagram of the communication support system disclosed herein. [Figure 3] This diagram shows an example of communication between an insurance agent and a customer. [Figure 4] This figure shows an example of the processing flow of the communication support system disclosed herein. [Figure 5] This figure shows an example of how to calculate the positive / negative score. [Figure 6] This figure shows an example of the hardware configuration of a communication terminal. [Modes for carrying out the invention]

[0010] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each of the following embodiments, the same parts will be denoted by the same reference numerals, and redundant descriptions will be omitted.

[0011] The explanation will proceed in the following order. [1. Communication support services] [1-1. Examples of conventional system configurations] [1-2. Example of System Configuration in This Disclosure] [2. Examples of communication] [3. Information Processing Methods] [4. Hardware Configuration Examples] [5. Effects]

[0012] [1. Communication support services] Figures 1 and 2 illustrate an overview of the communication support service. Figure 1 is a block diagram of the communication support system CSC applied to a conventional communication support service. Figure 2 is a block diagram of the communication support system CS applied to the communication support service of this disclosure.

[0013] Communication support services are services that facilitate remote communication between users. Users communicate remotely using communication devices™ such as smartphones, tablets, laptops, and desktop computers.

[0014] The communication support system CSC has multiple communication terminals TM corresponding to the number of users. Each communication terminal TM uses sensors to sense the user and transmits the sensing results to other users' communication terminals TM. The communication support system visualizes and presents the state (emotions, etc.) of the conversation partner PA to the speaker SK. The speaker SK can then engage in conversation while recognizing the PA's state in real time.

[0015] [1-1. Examples of conventional system configurations] In the example of FIG. 1, the speaker SK is an insurance diplomat CN, and the conversation partner PA is a customer CU. The speaker SK and the partner PA communicate via the terminal TK and the terminal TA. The terminals TK and TA are communication terminals TM capable of transmitting and receiving video and audio. The terminals TK and TM function as information processing devices that process video and audio.

[0016] The terminal TK has a camera input unit CM, a microphone input unit MC, a content display unit DC, a partner video display unit DP, a partner voice output unit SP, a partner status presentation unit PS, a data transmission unit D TK, and a data reception unit DRK. The terminal TA has a camera input unit CM, a microphone input unit MC, a shared content display unit DS, a partner video display unit DP, a partner voice output unit SP, a reaction recognition unit RE, a data transmission unit DTA, and a data reception unit DRA.

[0017] For example, known cameras, microphones, and speakers are used for the camera input unit CM, the microphone input unit MC, and the partner voice output unit SP. Known displays such as LCD (Liquid Crystal Display) are used for the content display unit DC, the shared content display unit DS, and the partner video display unit DP.

[0018] The camera input unit CM inputs the video of the speaker SK or the partner PA taken by the camera. The microphone input unit MC inputs the voice of the speaker SK or the partner PA collected by the microphone. The content display unit DC and the shared content display unit DS display the content CT shared among users on the display.

[0019] The video, voice, and content CT of the speaker SK are supplied to the partner video display unit DP, the partner voice output unit SP, and the shared content display unit DS of the terminal TA via the data transmission unit D TK and the data reception unit DRA, respectively. The video and voice of the partner PA are supplied to the partner video display unit DP and the partner voice output unit SP of the terminal TK via the data transmission unit DTA and the data reception unit DRK.

[0020] The reaction recognition unit RE recognizes the other party PA's reactions from their video and audio. The reaction recognition unit RE analyzes the other party PA's reactions and infers the other party PA's state based on the analysis results. For example, the analysis is performed using positive-negative analysis, a type of sentiment analysis. In positive-negative analysis, the emotional state is represented by a positive-negative score PN. The positive-negative score is a numerical representation of the degree of positivity. The reaction recognition unit RE notifies the other party state presentation unit PS of the inferred state of the other party PA (e.g., positive-negative score PN) via the data transmission unit DTA and data reception unit DRK. The other party state presentation unit PS displays the state of the other party PA on the terminal TK's display SCK.

[0021] In the example shown in Figure 1, the reaction recognition unit RE is mounted on the terminal TA. However, as shown by the dotted line in Figure 1, the reaction recognition unit RE may also be mounted on the terminal TK. In this case, the reaction recognition unit RE recognizes the reaction of the other PA from the video and audio of the other PA acquired via the data receiving unit DRK. Also, since the example in Figure 1 is an application example for insurance consulting, only the state of one user (customer CU) is analyzed. However, in other application examples such as business negotiations, it is preferable that both users can recognize each other's state. In this case, the reaction recognition unit RE may also be mounted on the terminal TK so that the states of both users engaging in the dialogue can be analyzed.

[0022] The lower part of Figure 1 shows an example of the information displayed on terminal TK and terminal TA. Terminal TA's display SCA shows content CT and speaker SK's video IMA. Terminal TK's display SCK shows content CT, the other party PA's video IMK, and the other party PA's positive / negative sentiment PN. Speaker SK uses the positive / negative sentiment PN to infer whether the other party PA has favorable feelings, or to what extent the other party PA understands the content of the conversation.

[0023] [1-2. Example of System Configuration in This Disclosure] Figure 2 shows an example of the communication support system CS of this disclosure. What distinguishes this disclosure from conventional systems is the method for inferring the state of the other party PA. In the conventional example shown in Figure 1, the state of the other party PA is inferred using only the responses of the other party PA in the dialogue. In this disclosure, the state of the other party PA is inferred based on the correlation between the speaker SK's actions and the other party PA's response to those actions (information on the difference between the expected response of the other party PA to the speaker SK's actions and the actual response of the other party PA).

[0024] The data transmission unit DTK transmits the speaker SK's words and actions that stimulate the conversation partner PA to the terminal TA (opponent PA). The terminal TA infers the state of the opponent PA based on the correlation between the speaker SK's words and actions and the opponent PA's response. The data reception unit DRK receives the inferred state of the opponent PA in response to the stimulus from the terminal TA. The opponent state presentation unit PS presents the state of the opponent PA to the speaker SK. The following explanation will focus on the differences from the conventional example shown in Figure 1.

[0025] In addition to the configuration shown in Figure 1, the terminal TA includes a stimulus recognition unit ST, a response prediction unit PR, a state prediction unit ES, a stimulus database unit DBS, and a response database unit DBR.

[0026] The stimulus recognition unit ST recognizes the words and actions of speaker SK that stimulate the conversation partner PA. A stimulus refers to something that triggers a response from the other party PA. Stimulating words and actions include statements, gestures, facial expressions, and the presentation of content CT that encourage the other party PA's response (understanding, agreement, action, etc.). The stimulus recognition unit ST extracts stimulating words and actions from the video of speaker SK, the audio of speaker SK, and the content CT presented by speaker SK.

[0027] The behaviors to be extracted are pre-defined by the system designer or other relevant personnel. The Stimulus Database (DBS) stores the content of the behaviors to be extracted as stimuli as stimulus information. The content of the behavior refers to the type and characteristics of the behavior. For example, if the stimulus is a gesture, the type of gesture and the characteristics of the skeletal movement are stored as stimulus information. If the stimulus is a facial expression, the type of expression and the arrangement of facial feature points are stored as stimulus information.

[0028] The stimulus recognition unit ST extracts stimulus-related verbal and auditory behavior from the speaker SK's video, audio, and content CT using known image and audio analysis techniques. The stimulus recognition unit ST outputs the content of the stimulus-related verbal and auditory behavior, as well as the time (timestamp) when the stimulus was recognized by the other party PA, as stimulus recognition information.

[0029] The stimulus recognition unit (ST) can recognize stimuli based on the speaker SK's unique behavioral patterns. The stimulus database unit (DBS) stores the speaker SK's unique behaviors (such as habits) used to encourage understanding and responses from others as the speaker SK's behavioral patterns. Examples of unique behavioral patterns include "checking the other person's reaction after each utterance," "repeating important terms that the speaker wants to be understood two or more times," "circling important parts with the mouse cursor," and "remaining silent for a while to draw attention." The stimulus recognition unit (ST) extracts behaviors similar to the speaker's unique behavioral patterns from the speaker SK's video, audio, and content CT.

[0030] The response prediction unit (PR) recognizes the expected response of the other party (PA) to a stimulus as a predicted response. For example, if the stimulus is a "question," then actions such as "nodding" or "answering" are recognized as predicted responses. If the stimulus is "presentation of materials," then actions such as "attention" or "nodding" are recognized as predicted responses.

[0031] The correspondence between stimuli and expected responses is defined in the Response Database Unit (DBR). The Response Database Unit (DBR) defines one or more expected responses for each stimulus. For each expected response, the Response Database Unit (DBR) stores the content of the expected behavior, the timing of the behavior, and the expected positive / negative sentiment (expected positive / negative sentiment) as expected response information.

[0032] For example, if the expected response is a gesture, the type of gesture, the characteristics of the skeletal movement, the timing of the gesture, and the expected positive / negative sentiment from the gesture are stored as expected response information. If the expected response is a facial expression, the type of expression, the arrangement of facial feature points, the timing of the expression's appearance, and the expected positive / negative sentiment from the expression are stored as expected response information. The value of the expected positive / negative sentiment can be arbitrarily set by the system designer. For example, for a "question," "nodding" is assumed as one of the expected responses. Since "nodding" is a positive response that shows understanding of the other party (PA), a high value is set for the expected positive / negative sentiment.

[0033] The response prediction unit PR identifies the content of the stimulus's verbal or physical action from the stimulus recognition information. The response prediction unit PR extracts one or more predicted response pieces of information linked to the corresponding verbal or physical action from the response database unit DBR and supplies them to the state prediction unit ES.

[0034] The response prediction unit (PR) can predict the other party's response to a stimulus by taking into account the other party's unique behavioral patterns. The response database unit (DBR) stores the other party's unique verbal and physical behaviors (such as habits) that they exhibit when stimulated, as their behavioral patterns. Examples of unique behavioral patterns include "nodding several times with their mouth rounded" in response to "presentation of materials" to indicate understanding, or "thinking deeply with fingers on their temples" to indicate deep thought. The response prediction unit (PR) extracts verbal and physical behaviors similar to the other party's unique behavioral patterns from the other party's video and audio.

[0035] The response recognition unit RE recognizes the other PA's response from the other PA's video and audio. The response recognition unit RE recognizes the other PA's actual response to a stimulus as the actual response. The state estimation unit ES estimates the state of the other PA based on the difference between the predicted response and the actual response. The state of the other PA is calculated, for example, as a positive / negative score PN. The positive / negative score PN is calculated by correcting the predicted positive / negative score based on the difference between the predicted response and the actual response. The method of correction can be arbitrarily set by the system developer.

[0036] For example, the state estimation unit ES calculates a provisional positive / negative score PN based on the type of response from the opposing PA. The response database unit DBR stores information about individual actions that should be detected as responses as response information. The response information includes the content of the actions detected as responses, and the positive / negative score (provisional positive / negative score) expected from those actions. The value of the provisional positive / negative score is arbitrarily set by the system designer. For example, turning in a direction unrelated to the conversation is a negative response indicating indifference, so the provisional positive / negative score is set to a low value.

[0037] The reaction prediction unit PR extracts reaction information corresponding to the reaction of the counterpart PA from the reaction database unit DBR and supplies it to the state prediction unit ES. The state prediction unit ES extracts a provisional positive / negative score from the reaction information and calculates it as a positive / negative score PN indicating the state of the counterpart PA. If the reaction of the counterpart PA is of the same type as the predicted reaction, the state prediction unit ES corrects the positive / negative score PN with a correction value corresponding to the difference between the predicted positive / negative score extracted from the predicted reaction information and the provisional positive / negative score. The state prediction unit ES calculates the corrected positive / negative score PN as the state of the counterpart PA.

[0038] The state prediction unit ES can predict the state of the opposing PA by taking into account the difference in the magnitude of the reaction and the difference in the timing of the reaction between the predicted reaction and the actual reaction. For example, the reaction database unit DBR has "nodding" registered as one of the predicted reactions to the stimulus "questioning". If "nodding" is recognized as the actual reaction, the reaction prediction unit PR extracts predicted reaction information about "nodding" from the reaction database unit DBR and supplies it to the state prediction unit ES. From the predicted reaction information, the state prediction unit ES extracts the magnitude of a standard "nod", the timing (the time from "questioning" to "nodding"), and the predicted positive / negative score.

[0039] The reaction recognition unit RE analyzes the actual reaction to detect the actual size and timing of the "nod". The state prediction unit ES compares the actual size and timing of the "nod" with the size and timing of the "nod" predicted by the reaction prediction unit PR (extracted from predicted reaction information), and calculates a correction value to be applied based on the comparison result. For example, the larger the size of the "nod" is compared to the predicted value, or the earlier the timing of the "nod" is compared to the predicted value, the larger the correction value will be.

[0040] If the other party's (PA's) response is of a different type than the expected response, the state prediction unit (ES) calculates a provisional positive / negative score extracted from the response information as the positive / negative score (PN) indicating the other party's (PA's) state. If the most recent stimulus from the speaker (SK) may influence the current behavior of the other party (PA), the state prediction unit (ES) can correct the positive / negative score (PN) with a correction value that takes the most recent stimulus into account. For example, if a positive response is expected due to the most recent stimulus, a positive correction value is applied to the positive / negative score (PN).

[0041] The state estimation unit ES notifies the state of the estimated counterpart PA (positive / negative degree PN) to the counterpart state indication unit PS via the data transmission unit DTA and data reception unit DRK. The counterpart state indication unit PS displays the state of the counterpart PA on the terminal TK's display SCK.

[0042] The example in Figure 2 is an application to insurance consulting, so only the state of one user (customer CU) is analyzed. However, in other application examples such as business negotiations, it is preferable that both users can recognize each other's state. In this case, terminal TK may also be equipped with the same configuration as terminal TA (stimulus recognition unit ST, response recognition unit RE, response prediction unit PR, state prediction unit ES, stimulus database unit DBS, and response database unit DBR) to enable analysis of the states of both users engaging in the dialogue.

[0043] [2. Examples of communication] Figure 3 shows an example of communication between an insurance agent (CN) and a customer (CU).

[0044] Insurance agent CN asks customer CU via terminal TK, "Why is it better to diversify my assets?" Customer CU's terminal TA analyzes the video and audio transmitted from terminal TK by insurance agent CN and detects the insurance agent CN's "question" as a stimulus.

[0045] Terminal TA anticipates a response of "immediately (after 0.1 seconds)" to the stimulus of "a question." In reality, customer CU is unable to find an appropriate answer and turns their head to the side, deep in thought. Terminal TA analyzes the customer CU's video and audio and recognizes the response as "thinking for a while (1 second) with their head turned to the side." Terminal TA determines the Pos / Negative Score PN to be 10 and notifies Terminal TK. The Pos / Negative Score PN can range from 0 to 100. A lower Pos / Negative Score PN indicates a greater degree of negativity.

[0046] Terminal TK updates the Positive / Negative Score PN on the display SCK based on a notification from Terminal TA. Insurance agent CN sees the value of Positive / Negative Score PN and realizes that customer CU does not understand. To help customer CU understand, insurance agent CN says, "Here it is," and shows the document "Don't put all your eggs in one basket" to the camera of Terminal TK. Terminal TK sends the document to Terminal TA as content CT for sharing.

[0047] Terminal TA recognizes the update of content CT and displays the document on display SCA. Terminal TA anticipates a response to the stimulus of "presentation of document" as "looking at the document for a while (0.5 seconds) and nodding slightly." In reality, customer CU understands the significance of diversified investment from the presented document and nods emphatically, saying "I see." Terminal TA analyzes customer CU's video and audio and recognizes a response of "immediately (0.1 seconds later) nodding emphatically and saying 'I see'." Terminal TA determines the positive / negative sentiment index PN to be 90 and notifies Terminal TK.

[0048] Terminal TK updates the positive / negative score PN on the display SCK based on the notification from Terminal TA. Insurance agent CN sees the value of the positive / negative score PN and recognizes that customer CU has understood.

[0049] [3. Information Processing Methods] Figure 4 shows an example of the processing flow of the communication support system CS.

[0050] The stimulus recognition unit ST analyzes the video and audio (hereinafter referred to as "viewed video, etc.") that the customer CU views via the terminal TA (Step S1). The viewed video, etc. includes the facial image and audio of the insurance agent CN, as well as shared document video, etc. Based on the analysis results, the stimulus recognition unit ST determines whether or not there are any words or actions in the viewed video, etc. that would stimulate the customer CU (Step S2).

[0051] The verbal and physical actions to be extracted as stimuli are registered in the Stimulus Database Unit (DBS). The Stimulus Database Unit (DBS) stores stimulus information that defines the content (type, characteristics) of each verbal or physical action that constitutes a stimulus. The Stimulus Recognition Unit (ST) compares the customer's (CU) viewing video, etc., with the Stimulus Database Unit (DBS). If the viewing video, etc., contains verbal or physical actions registered in the Stimulus Database Unit (DBS), the Stimulus Recognition Unit (ST) determines that a stimulus is present.

[0052] If there is any stimulating behavior in the viewed video, etc. (Step S2: Yes), the stimulus recognition unit ST generates stimulus recognition information indicating the content of the stimulating behavior and the time (timestamp) when the stimulus was recognized by the customer CU (Step S3). The content of the stimulating behavior is obtained from the stimulus information. The time information is obtained by analyzing the viewed video, etc. "The time when the stimulus was recognized by the customer CU" means the time when the customer CU actually acquired the viewed video, etc., taking into account network delays, packet loss, etc., rather than the transmission time of the viewed video, etc.

[0053] The response prediction unit PR predicts the customer CU's response to a stimulus (step S4). The predicted responses are registered in the response database unit DBR. The response database unit DBR stores predicted response information for each stimulus, which defines the content of the expected response, the timing of the expected response, and the expected positive / negative degree based on the expected response. The response prediction unit PR identifies the content of the stimulus from the stimulus recognition information and extracts the predicted response information corresponding to the stimulus from the response database unit DBR.

[0054] The reaction recognition unit RE analyzes the video and audio of the customer CU (hereinafter referred to as "monitoring video, etc.") monitored by the terminal TA (step S5). The monitoring video, etc. includes the customer CU's facial video and audio.

[0055] The state prediction unit ES calculates the positive / negative sentiment PN of the customer CU based on the analysis results of the response recognition unit RE (step S6). For example, when the state prediction unit ES obtains information on the predicted response from the response prediction unit PR, it calculates the positive / negative sentiment PN based on the difference between the expected response to the stimulus (predicted response) and the actual response of the customer CU (actual response) (step S6). The calculated positive / negative sentiment PN is transmitted to the terminal TK and displayed on the display SCK.

[0056] In step S2, if there are no stimulating words or actions in the viewed video, etc. (step S2: No), the calculation process of the predicted response shown in steps S3 and S4 is not performed. The response recognition unit RE analyzes the monitoring video, etc., and recognizes customer CU's words or actions that are unrelated to stimuli (step S5). The state prediction unit ES calculates the customer CU's positive / negative score PN based on the customer CU's words or actions that are unrelated to stimuli (step S6).

[0057] Terminal TK determines the end of processing based on an operation such as pressing the exit button (step S7). Terminal TK repeats the above process until the exit operation is performed.

[0058] Figure 5 shows an example of a method for calculating the positive / negative degree (PN).

[0059] The reaction recognition unit RE analyzes the monitoring video and other data to determine whether any behavior has been detected from the customer CU (step S11). The behaviors to be detected are registered in the reaction database unit DBR. The behaviors to be detected include both behaviors that are in response to a stimulus (responses to a stimulus) and behaviors that are unrelated to a stimulus. If the reaction recognition unit RE detects any behavior registered in the reaction database unit DBR from the monitoring video and other data, it determines that some behavior has been detected from the customer CU.

[0060] If any behavior is detected from the customer CU (Step S11: Yes), the state estimation unit ES calculates the positive / negative degree PN based on the detected behavior.

[0061] First, the state estimation unit ES calculates a provisional positive / negative score based on the customer's words and actions unrelated to the stimulus (step S12). The response recognition unit RE determines whether or not a predicted response to the stimulus exists in the time period before and after (step S13). If a predicted response exists (step S13: Yes), the state estimation unit ES corrects the provisional positive / negative score based on the difference between the predicted response and the actual response and outputs the resulting positive / negative score PN as the customer CU's state.

[0062] For example, the state prediction unit ES determines whether the actual customer CU's response to the stimulus (actual response) is more positive than the expected response (step S14). If the actual response is more positive than the expected response (step S14: Yes), the state prediction unit ES changes the hypothetical positive / negative score to positive based on the difference between the expected and actual responses (step S15). If the actual response is more negative than the expected response (step S14: No), the state prediction unit ES changes the hypothetical positive / negative score to negative based on the difference between the expected and actual responses (step S16).

[0063] If no behavior is detected from the customer CU (Step S11: No), the state estimation unit ES calculates the current positive / negative score PN by taking into account the most recent stimuli. The most recent stimuli refer to stimuli that are likely to cause a predicted response behavior at the current time. For example, the response database unit DBR has periods with a predetermined time range registered as the timing when a predicted response may occur. If the current time falls within that period, it is considered that the most recent stimuli may influence the current behavior of the customer CU. The state estimation unit ES calculates the current positive / negative score PN by taking into account such most recent stimuli that may influence the current behavior of the customer CU.

[0064] First, the state prediction unit ES determines whether or not there are past stimuli that could influence the current customer CU's behavior (step S18). The state prediction unit ES determines that there are past stimuli that could influence the current customer CU's behavior if the current time falls within the period registered as the timing of the expected response.

[0065] If there is a past stimulus that could influence the current behavior of customer CU (Step S18: Yes), the state prediction unit ES determines whether the expected response to the past stimulus is a positive response or not (Step S19).

[0066] The state prediction unit ES extracts the predicted positive / negative score corresponding to the predicted reaction from the reaction database unit DBR. If the predicted reaction is a positive reaction (step S19: Yes), the state prediction unit ES changes the provisional positive / negative score to negative according to the predicted positive / negative score (step S20). If the predicted reaction is a negative reaction (step S19: No), the state prediction unit ES changes the provisional positive / negative score to positive according to the predicted positive / negative score (step S21).

[0067] The provisional positive / negative score used in steps S20 and S21 is, for example, the most recently calculated provisional positive / negative score or positive / negative score PN. This allows the state estimation unit ES to calculate the current positive / negative score PN by taking into account the most recently calculated positive / negative score PN. It is assumed that the customer CU's state (positive, negative) will be maintained for a while and gradually return to neutral. Therefore, the positive / negative score PN may also change accordingly to return to neutral. For example, the state estimation unit ES may gradually bring the positive / negative score PN closer to the neutral value as time passes.

[0068] The state estimation unit ES notifies the insurance agent CN of the provisional positive / negative score PN, which is corrected in steps S15, S16, S20, and S21, as the positive / negative score PN that indicates the state of the customer CU (step S17).

[0069] [4. Hardware Configuration Examples] Figure 6 shows an example of the hardware configuration of a communication terminal TM.

[0070] Information processing for the communication terminal TM is implemented, for example, by computer 1000. Computer 1000 has a CPU (Central Processing Unit) 1100, RAM (Random Access Memory) 1200, ROM (Read Only Memory) 1300, HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The various parts of computer 1000 are connected by bus 1050.

[0071] The CPU 1100 operates based on programs (program data 1450) stored in the ROM 1300 or HDD 1400, and controls various parts. For example, the CPU 1100 loads the programs stored in the ROM 1300 or HDD 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0072] ROM1300 stores boot programs such as the BIOS (Basic Input Output System) executed by CPU1100 when computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0073] The HDD1400 is a computer-readable non-temporary recording medium that non-temporarily records programs executed by the CPU1100 and data used by such programs. Specifically, the HDD1400 is a recording medium that records an information processing program according to an embodiment, which is an example of program data 1450.

[0074] The communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (for example, the Internet). For example, the CPU 1100 can receive data from other devices or transmit data it has generated to other devices via the communication interface 1500.

[0075] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard or mouse via the input / output interface 1600. The CPU 1100 also transmits data to output devices such as display devices, speakers, or printers via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, or semiconductor memory.

[0076] For example, when computer 1000 functions as a communication terminal TM according to the embodiment, the CPU 1100 of computer 1000 realizes the functions of each of the parts described above by executing an information processing program loaded on RAM 1200. The HDD 1400 stores the information processing program, various models, and various data according to this disclosure. The CPU 1100 reads and executes the program data 1450 from HDD 1400, but as another example, these programs may be obtained from other devices via an external network 1550.

[0077] [5. Effects] Terminal TA comprises a stimulus recognition unit ST, a response prediction unit PR, a response recognition unit RE, and a state prediction unit ES. The stimulus recognition unit ST recognizes the words and actions of speaker SK that stimulate the conversation partner PA. The response prediction unit PR recognizes the expected response of the conversation partner PA to the stimulus as the expected response. The response recognition unit RE recognizes the actual response of the conversation partner PA to the stimulus as the actual response. The state prediction unit ES predicts the state of the conversation partner PA based on the difference between the expected response and the actual response. In the information processing method of this disclosure, the processing of terminal TA is executed by computer 1000. The program of this disclosure causes computer 1000 to implement the processing of terminal TA.

[0078] In this configuration, the state of the other party PA is inferred from how PA reacts to the speaker SK's words and actions (stimuli). If PA's state is inferred solely from information about PA (at the level of absolute facial expressions, etc.), accurate inferences will not be possible due to individual differences. By inferring PA's state based on the correlation between speaker SK's words and actions and PA's reactions, the accuracy of the inference is improved.

[0079] The state estimation unit ES estimates the state of the opponent PA by taking into account the difference in the magnitude of the reaction and the difference in the timing of the reaction between the predicted reaction and the actual reaction.

[0080] This configuration improves the accuracy of predicting the opponent's PA (Player Area) status.

[0081] The stimulus recognition unit ST recognizes the speaker SK's unique behavioral patterns and actions as stimuli.

[0082] This configuration allows for the extraction of appropriate stimuli that take into account the speaker SK's behavioral patterns.

[0083] The response prediction unit (PR) predicts the opponent PA's response to a stimulus by taking into account the opponent PA's unique behavioral patterns.

[0084] This configuration allows for the prediction of an appropriate response, taking into account the opponent's PA's behavior patterns.

[0085] The state estimation unit ES calculates a provisional positive / negative score based on the words and actions of the opposing PA, which are unrelated to the stimulus. The state estimation unit ES corrects the provisional positive / negative score based on the difference between the expected response and the actual response, and outputs the resulting positive / negative score PN as the state of the opposing PA.

[0086] This configuration allows for the estimation of an appropriate state by taking into account the opponent's actions and words, in addition to their actual reactions.

[0087] The state estimation unit ES calculates the current positive / negative score PN by taking into account recent stimuli that may influence the current behavior of the opposing PA.

[0088] This configuration allows for the effects of recent stimuli to be appropriately reflected in the current state of the opposing PA.

[0089] The state estimation unit ES calculates the current positive / negative score PN by taking into account the most recently calculated positive / negative score PN.

[0090] This configuration allows for accurate prediction of the state of data by considering the contextual relationships between data arranged in a time series.

[0091] The state estimation unit ES gradually adjusts the positive / negative degree PN to a neutral value as time progresses.

[0092] According to this configuration, the positive / negative score (PN) shows a gradual change that corresponds to changes in emotions.

[0093] Terminal TK comprises a data transmission unit DTK, a data reception unit DRK, and a partner state presentation unit PS. The data transmission unit DTK transmits the words and actions of speaker SK that stimulate the conversation partner PA. The data reception unit DRK receives the inferred state of partner PA in response to the stimulus. The partner state presentation unit PS presents the state of partner PA to speaker SK.

[0094] With this configuration, speaker SK can conduct a dialogue while checking the status of the other party PA.

[0095] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0096] [Note] Furthermore, this technology can also be configured as follows. (1) A stimulus recognition unit that recognizes the speaker's words and actions that stimulate the other party in the conversation, A reaction prediction unit that recognizes the expected response of the other party to a stimulus as a predicted response, A reaction recognition unit that recognizes the actual response of the other party to the aforementioned stimulus as an actual response, A state prediction unit that predicts the state of the opponent based on the difference between the predicted reaction and the actual reaction, An information processing device having (2) The state estimation unit estimates the state of the other party by taking into account the difference in the magnitude of the reaction and the difference in the timing of the reaction between the predicted reaction and the actual reaction. The information processing device described in (1) above. (3) The stimulus recognition unit recognizes the speaker's specific behavioral patterns as the stimulus. The information processing device described in (1) or (2) above. (4) The response prediction unit predicts the opponent's response to the stimulus by taking into account the opponent's unique behavioral patterns. An information processing device as described in any one of (1) to (3) above. (5) The state estimation unit calculates a provisional positive / negative score based on the other party's words and actions unrelated to the stimulus, corrects the provisional positive / negative score based on the difference between the predicted response and the actual response, and outputs the resulting positive / negative score as the other party's state. An information processing device as described in any one of (1) through (5) above. (6) The state estimation unit calculates the current positive / negative degree by taking into account the most recent stimuli that may influence the current words and actions of the other party. The information processing device described in (5) above. (7) The state estimation unit calculates the current positive / negative degree by taking into account the most recently calculated positive / negative degree. The information processing device described in (5) or (6) above. (8) The state estimation unit gradually brings the positive / negative degree closer to the neutral value as time progresses. The information processing device described in (7) above. (9) A data transmission unit that transmits the speaker's words and actions that stimulate the other party to the conversation partner, A data receiving unit that receives the state of the other party inferred from the aforementioned stimulus, A unit that presents the state of the other party to the speaker, An information processing device having (10) Recognize the speaker's words and actions that provoke the other person in the conversation. The expected response of the other party to the stimulus is recognized as the expected response. The actual response of the other party to the aforementioned stimulus is recognized as the actual response. Based on the difference between the predicted reaction and the actual reaction, the state of the opponent is inferred. An information processing method performed by a computer, which includes the ability to perform the following actions. (11) Recognize the speaker's words and actions that provoke the other person in the conversation. The expected response of the other party to the stimulus is recognized as the expected response. The actual response of the other party to the aforementioned stimulus is recognized as the actual response. Based on the difference between the predicted reaction and the actual reaction, the state of the opponent is inferred. A program that allows a computer to accomplish something. [Explanation of Symbols]

[0097] DRK Data Receiver DTK Data Transmission Unit ES state estimation unit PA opponent PN Positive / Negative Degree PR Reaction Prediction Department PS Opponent Status Display Unit RE Reaction Recognition Unit SK speaker ST stimulus recognition unit TA (Terminal Adapter) TK Terminal (Information Processing Device) TM Communication Terminal (Information Processing Device)

Claims

1. A stimulus recognition unit that recognizes the speaker's words and actions that stimulate the other party in the conversation, A reaction prediction unit that recognizes the expected response of the other party to a stimulus as a predicted response, A reaction recognition unit that recognizes the actual response of the other party to the aforementioned stimulus as an actual response, A state prediction unit that predicts the state of the opponent based on the difference between the predicted reaction and the actual reaction, An information processing device having

2. The state estimation unit estimates the state of the other party by taking into account the difference in the magnitude of the reaction and the difference in the timing of the reaction between the predicted reaction and the actual reaction. The information processing apparatus according to claim 1.

3. The stimulus recognition unit recognizes the speaker's specific behavioral patterns as the stimulus. The information processing apparatus according to claim 1.

4. The response prediction unit predicts the opponent's response to the stimulus by taking into account the opponent's unique behavioral patterns. The information processing apparatus according to claim 1.

5. Recognize the speaker's words and actions that provoke the other person in the conversation. The expected response of the other party to the stimulus is recognized as the expected response. The actual response of the other party to the aforementioned stimulus is recognized as the actual response. Based on the difference between the predicted reaction and the actual reaction, the state of the opponent is inferred. An information processing method performed by a computer, which includes the ability to perform the following actions.

6. Recognize the speaker's words and actions that provoke the other person in the conversation. The expected response of the other party to the stimulus is recognized as the expected response. The actual response of the other party to the aforementioned stimulus is recognized as the actual response. Based on the difference between the predicted reaction and the actual reaction, the state of the opponent is inferred. A program that allows a computer to accomplish something.

Images