Information processing apparatus, information processing method, and recording medium

Abstract

One of the objects is to provide an information processing device capable of appropriately specifying a type of communication of a person who is a communication partner without being easily affected by an emotion of the person.SOLUTION: An information processing device includes an acoustic feature amount acquisition unit that acquires an acoustic feature amount of a voice uttered by one person, an emotion specifying unit that specifies an emotion of the one person on the basis of the acoustic feature amount, and an estimation unit that estimates a communication type of the one person on the basis of information indicating the emotion and information indicating the acoustic feature amount by using a learning model.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, an information processing program, and a recording medium, and more particularly to an information processing device, an information processing method, an information processing program, and a recording medium that can be used when a user communicates, including having a conversation, with a single person. [Background technology]

[0002] Communication between people, especially that required for the work of call center operators, requires dealing with various types of people, which can be difficult for some people or make it difficult to get good results. For this reason, methods for improving communication have been proposed.

[0003] For example, Patent Document 1 discloses a call center device that analyzes the voice of a customer and the voice of an operator, calculates a value of a first evaluation axis for evaluating the customer's customer type based on the sound pressure and / or frequency of the customer's voice, calculates a value of a second evaluation axis for evaluating the customer's customer type based on the overlap between the customer's voice and the operator's voice, identifies the customer's customer type based on the value of the first evaluation axis and the value of the second evaluation axis, and displays it on the screen of an operator terminal. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6567729 Summary of the Invention [Problem to be solved by the invention]

[0005] For example, as mentioned above, when analyzing the voice of a communication partner to identify their type, even if the partner is the same person, their emotions may differ depending on their physical condition or mood, or on the situation at hand, such as whether they are tense or relaxed. In such cases, one of the challenges is that the partner's emotions may affect their speaking style, making it difficult to identify their type.

[0006] The present invention has been made in consideration of the above points, and one of its objectives is to provide an information processing device that is less influenced by the emotions of the person with whom you are communicating and that is able to appropriately identify the communication type of the person with whom you are communicating. [Means for solving the problem]

[0007] The invention described in claim 1 is characterized by comprising an acoustic feature acquisition unit that acquires acoustic features of a voice uttered by one person, an emotion identification unit that identifies the emotion of the one person based on the acoustic features, and an estimation unit that uses a learning model to estimate the communication type of the one person based on information indicating the emotion and information indicating the acoustic features.

[0008] The invention described in claim 2 is characterized by comprising an acoustic feature acquisition unit that acquires acoustic features of a voice uttered by one person, a situation acquisition unit that acquires information indicating a situation when the voice was uttered, an emotion identification unit that identifies the emotion of the one person based on the information indicating the situation, and an estimation unit that estimates the communication type of the one person based on the information indicating the emotion and the information indicating the acoustic features using a learning model.

[0009] The invention described in claim 8 is characterized by comprising: an acoustic feature acquisition unit that acquires acoustic features of a voice uttered by one person; an emotion identification unit that identifies the emotion of the one person based on the acoustic features; and an estimation unit that estimates the communication type of the one person from the information indicating the emotion of the one person and the information indicating the acoustic features using a learning model that receives as input information indicating a person's emotion and information indicating the acoustic features of the voice uttered by the person and outputs information indicating a person's communication type.

[0010] The invention described in claim 9 is characterized by comprising: an acoustic feature acquisition unit that acquires acoustic features of a voice uttered by one person; an emotion identification unit that identifies the emotion of the one person based on the acoustic features; and an estimation unit that estimates the communication type of the one person from information indicating the emotion of the one person and information indicating the acoustic features using a plurality of learning models constructed for a predetermined plurality of items that indicate types of human emotions, so as to receive information indicating the acoustic features of the voice uttered by the person as input and output information indicating the communication type of the person.

[0011] The invention described in claim 10 is an information processing method in which an information processing device processes information, comprising: an acoustic feature acquisition step in which an acoustic feature acquisition unit acquires acoustic features of a voice uttered by one person; an emotion identification step in which an emotion identification unit identifies the emotion of the one person based on the acoustic features; and an estimation step in which an estimation unit uses a learning model to estimate the communication type of the one person based on information indicating the emotion and information indicating the acoustic features.

[0012] The invention described in claim 11 is an information processing program executed by an information processing device having a computer, characterized in that the computer is caused to execute the following steps: an acoustic feature acquisition step in which an acoustic feature acquisition unit acquires acoustic features of a voice uttered by one person; an emotion identification step in which an emotion identification unit identifies the emotion of the one person based on the acoustic features; and an estimation step in which an estimation unit uses a learning model to estimate the communication type of the one person based on information indicating the emotion and information indicating the acoustic features. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram schematically illustrating an example of communication using an information processing device according to an embodiment. [Figure 2] FIG. 2 is a functional block diagram illustrating a configuration of a terminal according to an embodiment. [Figure 3] FIG. 2 is a functional block diagram illustrating a configuration of a server according to an embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a learning model according to an embodiment. [Figure 5] 10 is a flowchart illustrating an example of a routine executed by the terminal of the embodiment. [Figure 6] 10 is a flowchart illustrating an example of a routine executed by a server of an embodiment. [Figure 7] 10 is a flowchart illustrating an example of a subroutine executed by the server of the embodiment. [Figure 8] 10 is a flowchart illustrating an example of a routine executed by the terminal of the embodiment. [Figure 9] FIG. 10 is a diagram showing an example of an operation screen according to a first modified example of the embodiment. [Figure 10] 10 is a flowchart showing an example of a routine executed in a terminal according to a first modification. [Figure 11] 10 is a flowchart showing an example of a routine executed in a server of the first modification. [Figure 12]FIG. 10 is a diagram schematically illustrating an example of a learning model according to Modification 2. [Figure 13] 10 is a flowchart showing an example of a subroutine executed in a server of Modification 2. DETAILED DESCRIPTION OF THE INVENTION

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following description and accompanying drawings, substantially the same or equivalent parts are designated by the same reference numerals. [Example]

[0015] 1 is a diagram showing an example of communication using an information processing system 10 as an information processing device according to an embodiment of the present invention, in which a user UR is engaged in conversation-based communication with a customer CL as a person. Specifically, FIG. 1 shows a situation in which the user UR, who is a salesperson, is negotiating with the customer CL. In the negotiation, for example, the conversation proceeds by first exchanging greetings, followed by the user UR explaining the product, and the customer CL indicating whether or not they intend to purchase.

[0016] The characteristics of communication styles may differ from person to person. For example, the characteristics of communication styles of people can be classified into several predetermined types. In this specification, the characteristics of communication styles that differ from person to person will be described as communication types.

[0017] In the following description, four communication types based on social style theory will be used as an example of a method for classifying communication types. Specifically, in this embodiment, four communication types, "Driver," "Analytical," "Amiable," and "Expressive," will be used for the description.

[0018] As mentioned above, communication types vary from person to person, and therefore the appropriate response may also differ depending on the other person's communication type. For example, it may be effective to quickly convey the conclusion to one person, while it may be more effective to explain things step by step to another. For example, in communication that involves negotiations, the outcome of the negotiation may change depending on how you respond. Therefore, knowing the appropriate response for the customer CL's communication type is useful for users UR in promoting smoother communication with the customer CL.

[0019] The information processing system 10 is a system that can estimate the communication type of a customer CL in communication between a user UR and a customer CL, and can present a message including advice regarding the communication to the user UR.

[0020] As shown in FIG. 1, the information processing system 10 includes a server 11 and a terminal 13. The server 11 is, for example, a server device installed in a location different from where the user UR is located.

[0021] The terminal 13 is, for example, a terminal device such as a smartphone or tablet used by the user UR. The terminal 13 has a touch panel 15. The touch panel 15 is a touch panel display configured to be able to display an operation screen and messages and to accept operation inputs on the displayed screen.

[0022] The server 11 and the terminal 13 can transmit and receive data to and from each other via a network. For example, a communication protocol such as TCP / IP is used for the transmission and reception. The network can be constructed by internet communication including wireless communication such as a mobile communication network, Wi-Fi (registered trademark), and wired communication.

[0023] The server 11 performs information processing on the estimation of the communication type based on the information received from the terminal 13. Specifically, the terminal 13 acquires voice data from the voice uttered by the customer CL, for example, using a microphone (not shown) built into the terminal 13, and transmits the voice data to the server 11. The server 11 estimates the communication type of the customer CL using the voice data. The server 11 generates advice information including advice on communication with the customer CL and transmits the advice information to the terminal 13. The terminal 13 presents the message to the user UR by displaying the message on the touch panel 15 based on the advice information.

[0024] 2 is a block diagram showing the configuration of the terminal 13. The terminal 13 is configured by connecting each unit via a system bus 17, for example.

[0025] The transmitting / receiving unit 19 is a network adapter such as a network interface card (NIC) connected to the wireless device. The transmitting / receiving unit 19 transmits and receives data to and from the server 11. For example, the transmitting / receiving unit 19 transmits data necessary for estimating the communication type to the server 11.

[0026] Furthermore, the transmitting / receiving unit 19 receives information relating to the estimation result of the communication type from the server 11. For example, the transmitting / receiving unit 19 receives from the server 11 advice information that is information including advice that reflects the estimation result of the communication type of the customer CL.

[0027] The storage unit 21 is a storage device configured, for example, by a hard disk drive, a solid state drive (SSD), a flash memory, etc. The storage unit 21 stores various programs for performing information processing in the terminal 13. Note that the various programs may be acquired, for example, from another server device or the like via a network.

[0028] The audio input unit 23 is connected to the microphone 24. The microphone 24 is, for example, a microphone built into the terminal 13. The audio input unit 23 is an interface that acquires audio data from the microphone 24. For example, the audio data acquired by the audio input unit 23 is transmitted to the server 11 by the transmitting / receiving unit 19.

[0029] The input / output unit 26 is connected to the touch panel 15. The input / output unit 26 is an interface that accepts operation input via the touch panel 15. The input / output unit 26 is also an interface that supplies image data to be displayed on the touch panel 15. For example, the input / output unit 26 supplies image data for displaying an operation screen to the touch panel 15. For example, the input / output unit 26 accepts operation input by the user UR to the touch panel 15. For example, the input / output unit 26 accepts operation input by the user UR to instruct the start of estimation of the communication type of the customer CL.

[0030] The terminal control unit 28 is a computer that includes a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory), and controls the operation of the terminal 13. In the terminal control unit 28, the CPU reads out and executes various programs stored in the ROM and the storage unit 21, thereby realizing various functions.

[0031] The display control unit 29 is a functional unit of the terminal control unit 28. The display control unit 29 controls the display of an image on the touch panel 15. For example, the display control unit 29 controls the display of an operation screen on the touch panel 15. The display control unit 29 also controls the display of a message for the user UR on the touch panel 15 as a display screen, based on advice information acquired from the server 11 via the transmission / reception unit 19.

[0032] 3 is a block diagram showing the configuration of the server 11. The server 11 is configured by connecting each unit via a system bus 31, for example.

[0033] The transmitter / receiver 33 is a network adapter such as a network interface card (NIC) connected to a wireless device. The transmitter / receiver 33 transmits and receives data to and from the terminal 13. For example, the transmitter / receiver 33 receives data necessary for estimating the communication type from the terminal 13. The transmitter / receiver 33 also transmits data based on the estimation result of the communication type to the terminal 13. For example, the transmitter / receiver 33 transmits advice information including advice regarding communication with the customer CL to the terminal 13.

[0034] The server control unit 35 is a computer that includes a CPU (Central Processing Unit), a ROM (Read Only Memory), and a Random Access Memory (Random Access Memory) and controls the operation of the server 11. The CPU realizes various functions by reading and executing various programs stored in the ROM.

[0035] The acoustic feature extraction unit 37, which serves as an acoustic feature acquisition unit, is a functional unit of the server control unit 35. The acoustic feature extraction unit 37 acquires voice data from the terminal 13 via the transmission / reception unit 33. The acoustic feature extraction unit 37 extracts, from the acquired voice data, a portion uttered by the customer CL as customer voice data. The acoustic feature extraction unit 37 calculates, from the customer voice data, acoustic features that digitize physical features and characteristics contained in the voice of the customer CL.

[0036] For example, the acoustic feature quantity extraction unit 37 calculates numerical values ​​indicating characteristics such as the volume, pitch, and tone of the voice as the features or characteristics of the voice of the customer CL.

[0037] For example, the acoustic feature extraction unit 37 extracts the time intervals of phonemes from the customer voice data and calculates a numerical value indicating the speaking speed of the customer CL. Also, for example, the acoustic feature extraction unit 37 calculates a numerical value indicating the pitch of the voice from the fundamental frequency (F0) of the customer voice data.

[0038] In this embodiment, an example will be described in which the acoustic feature extraction unit 37 calculates numerical values ​​indicating each of nine items as acoustic features: "speaking rate," "frequency deviation of fundamental frequency," "mean or deviation of skewness of fundamental frequency," "mean or deviation of kurtosis of fundamental frequency," "mean or deviation of peak length of fundamental frequency," "mean or deviation of peak decibel," "mean or deviation of peak time," "zero-crossing rate (or number of zero-crossings)," and "deviation and difference for each dimension of MFCC (Mel Frequency Cepstrum)."

[0039] The emotion identification unit 39 is a functional unit of the server control unit 35. The emotion identification unit 39 identifies a person's emotion based on the acoustic features extracted by the acoustic feature extraction unit 37. Specifically, the emotion identification unit 39 identifies the emotion that the person who uttered the voice that is the basis of the acoustic features felt when they uttered the voice. Therefore, the emotion identification unit 39 identifies the emotion that the customer CL felt when they uttered the voice based on the acoustic features of the voice uttered by the customer CL.

[0040] For example, in identifying the emotion, the emotion identification unit 39 identifies which of a plurality of predetermined items indicating the type of human emotion the acquired acoustic feature corresponds to. For example, the plurality of predetermined items indicating the type of human emotion may include items such as "calm," "tension," and "anger." Furthermore, the plurality of predetermined items indicating the type of human emotion may include items such as "sadness," "confusion," "dissatisfaction," "disgust," "contempt," "resignation," "joy," "expectation," and "trust."

[0041] For example, the emotion identification unit 39 identifies the emotion of the customer CL according to a predetermined criterion. For example, a correspondence relationship between acoustic features and items indicating types of emotions may be determined in advance, and a table describing the correspondence relationship may be stored in the emotion identification unit 39 or the mass storage device 47. For example, the emotion identification unit 39 identifies the emotion by referring to the table based on the acoustic features.

[0042] Furthermore, the emotion identification unit 39 may identify a person's emotion based on information indicating the situation when the voice was uttered. The situation when the voice was uttered may be, for example, a situation in which communication is taking place, such as a conversation between a salesperson and a customer in a store, a job interview, a business negotiation, or a conversation between a call center operator and a customer. Furthermore, the situation when the voice was uttered may be, for example, the state of the person with whom the user is communicating, such as a tense state or a relaxed state.

[0043] For example, the information indicating the situation may be input from the touch panel 15 of the terminal 13. For example, a selection screen for selecting the situation in which the voice was generated may be displayed on the touch panel 15, and a selection operation by the user may be accepted. In this case, the emotion identification unit 39 acquires the information indicating the situation via the transmission / reception unit 33.

[0044] The type estimation unit 41 as an estimation unit estimates the communication type of one person (i.e., customer CL) based on the acoustic features of the one person extracted by the acoustic feature extraction unit 37 and the emotion of the one person identified by the emotion identification unit 39. The type estimation unit 41 estimates the communication type using a learning model (i.e., a trained model), which is a mathematical model constructed using a deep learning technique.

[0045] The inference unit 43 is included in the type estimation unit 41. The inference unit 43 inputs necessary information into a learning model and causes the learning model to output an estimation result of the communication type. For example, the estimation result may be output as a numerical value indicating the degree of compatibility between each of a predetermined plurality of communication types and the communication type of the customer CL.

[0046] Furthermore, for example, the estimation results output from the learning model may be output as a numerical value indicating the probability that the communication type of customer CL matches each of the communication types, or may be output as a numerical value indicating one matching communication type.

[0047] The type estimation unit 41 identifies an estimation result of the communication type based on the output from the learning model. For example, when the learning model outputs numerical values ​​indicating the compatibility between each of a predetermined plurality of communication types and the communication type of the customer CL, the type estimation unit 41 selects the communication type corresponding to the largest numerical value and identifies it as the estimation result of the communication type.

[0048] The advice generation unit 45 generates advice information including advice for the user UR regarding communication with the customer CL based on the estimation result by the type estimation unit 41. For example, the advice information includes information indicating recommended topics and ways of proceeding in a conversation with the customer CL. For example, the advice generation unit 45 generates advice information by selecting advice previously associated with each communication type.

[0049] The mass storage device 47 is a storage device configured, for example, by a hard disk drive, a solid state drive (SSD), a flash memory, etc. The mass storage device 47 stores various programs for performing information processing in the server 11. The server control unit 35 realizes various functions by reading and executing the various programs stored in the mass storage device 47.

[0050] The various programs may be acquired, for example, from another server device via a network. Alternatively, the various programs may be recorded on a recording medium and read via a drive. The various programs stored in the mass storage device 47 can be transmitted via a network, or can be recorded on a computer-readable recording medium and transferred.

[0051] The mass storage device 47 includes a learning model storage unit 47A. The learning model storage unit 47A stores a learning model that receives information indicating a person's emotion and information indicating the acoustic features of a person's voice as input and outputs information indicating the person's communication type. The learning model stored in the learning model storage unit 47A is a mathematical model constructed using a deep learning technique.

[0052] The inference unit 43 of the type estimation unit 41 inputs information indicating the acoustic features extracted by the acoustic feature extraction unit 37 and information indicating the emotion of the customer CL identified by the emotion identification unit 39 to a learning model stored in the learning model storage unit 47A. For example, the learning model outputs a numerical value indicating the probability that the customer CL matches each communication type or a numerical value indicating the communication type of the customer CL.

[0053] Fig. 4 is a diagram schematically illustrating a learning model M1, which is an example of a learning model stored in the learning model storage unit 47A. Fig. 4 shows an example in which, as the output of the learning model M1, a numerical value indicating the probability that the communication type of customer CL conforms to each of the four communication types described above according to social communication theory is output.

[0054] The learning model M1 shown in Figure 4 is a neural network constructed by deep learning. The neural network is composed of an input layer, two or more intermediate layers (hidden layers), and an output layer.

[0055] For example, in the input layer of the learning model M1, nine numerical values ​​AU1 to AU9 indicating nine items of acoustic features, respectively, and a numerical value EM indicating an emotion are input. For example, the numerical value indicating an emotion may be a numerical value assigned in advance to an item indicating a predetermined type of emotion.

[0056] When this input is made, the output layer outputs four numerical values ​​indicating the probability that the communication type of customer CL, the person related to the input information, matches each of the four communication types based on social style theory: "Driver," "Analytical," "Amiable," and "Expressive."

[0057] That is, the learning model M1 is constructed so as to receive information indicating a person's acoustic features and information indicating a person's emotions as input, and to output information indicating a person's communication type.

[0058] Therefore, when information indicating the acoustic features of the speech uttered by customer CL and information indicating the emotion of customer CL are input to learning model M1, information indicating the communication type of customer CL is output. Note that learning model M1 may be configured to output, for example, a single numerical value indicating the communication type of customer CL in the output layer.

[0059] The learning model M1 is constructed, for example, by supervised learning as follows. For example, the communication type of one collaborator is identified from the results of a questionnaire administered to the one collaborator and used as the correct answer data. Also, voice data of the one collaborator conversing is acquired, and acoustic features are extracted from the voice data. Furthermore, the emotion of the one collaborator when he or she spoke when acquiring the voice data is identified. Information indicating the acoustic features and information indicating the emotion of the collaborator are used as input data.

[0060] The input data is input to a neural network with multiple intermediate layers, and learning is performed by updating the neural network parameters using the backpropagation method so that the output is the communication type of correct data. By performing this type of learning for a large number of collaborators, a learning model M1 can be constructed.

[0061] The emotion used for learning is identified based on, for example, the acoustic features described above. Alternatively, the emotion may be identified based on the situation when the collaborator uttered the voice.

[0062] Furthermore, when preparing the correct answer data or input data to be used for learning, image data may be used in addition to the questionnaire, voice data, etc., as described above. For example, when identifying a collaborator's communication type, accuracy may not be ensured with a questionnaire alone. In such cases, information such as facial expressions and body movements may be extracted from the image data and used as information for determining the communication type. For example, a collaborator's communication type may be identified by combining a questionnaire, voice data, and image data.

[0063] In this way, by combining image data, it is possible to select the communication type that best matches the communication tendencies of the collaborators as the correct answer data. In other words, the accuracy of the correct answer data can be improved. Therefore, the accuracy of communication type estimation can be improved by using a learning model constructed using such correct answer data.

[0064] Similarly, image data may be combined with the data used for learning to identify emotions as input data. For example, by combining acoustic features and image data to identify emotions, high accuracy in identifying emotions can be ensured.

[0065] Furthermore, learning using correct answer data or input data combined with image data may be performed in addition to a learning model that has already been constructed.

[0066] Furthermore, the learning model stored in the learning model storage unit 47A may be updated based on the estimation result by the type estimation unit.

[0067] 5 is a flowchart showing a voice data transmission routine RT1, which is an example of a routine executed by the terminal control unit 28 of the terminal 13. For example, when the power of the terminal 13 is turned on, the terminal control unit 28 starts the voice data transmission routine RT1.

[0068] When the voice data transmission routine RT1 starts, the terminal control unit 28 determines whether or not the start button has been pressed via the input / output unit 26 (step S11). In step S11, the terminal control unit 28 determines whether or not an operation to instruct the start of information processing for estimating the communication type has been accepted on the operation screen displayed on the touch panel 15.

[0069] If the terminal control unit 28 determines in step S11 that the start button has not been pressed (step S11: NO), it repeats step S11 and determines again whether or not the start button has been pressed.

[0070] When the terminal control unit 28 determines in step S11 that the start button has been pressed (step S11: YES), it starts transmitting the voice data acquired by the microphone 24 and the voice input unit 23 (step S12). In step S12, the terminal control unit 28 starts transmitting the voice data to the server 11 via the transmission / reception unit 19.

[0071] After executing step S12, the terminal control unit 28 determines whether or not the end button has been pressed via the input / output unit 26 (step S13). In step S13, the terminal control unit 28 determines whether or not an operation to instruct the end of information processing for estimating the communication type has been accepted on the operation screen displayed on the touch panel 15.

[0072] If the terminal control unit 28 determines in step S13 that the end button has not been pressed (step S13: NO), it repeats step S13 while continuing to transmit the voice data, and determines again whether the end button has been pressed.

[0073] When the terminal control unit 28 determines in step S13 that the end button has been pressed (step S13: YES), it ends the transmission of voice data from the transceiver unit 19 (step S14). After executing step S14, the terminal control unit 28 ends the voice data transmission routine RT1 and repeatedly executes the voice data transmission routine RT1.

[0074] 6 is a flowchart showing a type estimation routine RT2, which is an example of a routine executed by the server control unit 35 of the server 11. For example, the server control unit 35 starts the type estimation routine RT2 when the server 11 is powered on.

[0075] When the server control unit 35 starts the type estimation routine RT2, it waits for reception of voice data from the terminal 13 via the transmitting / receiving unit 33 (step S21).

[0076] After executing step S21, the server control unit 35 determines whether or not it has received voice data of a predetermined time or more (step S22). In step S22, the server control unit 35 determines whether or not it has received voice data in which the voice is emitted for a predetermined time or more, for example, via the transmitting / receiving unit 33. The predetermined time may be, for example, 15 seconds or 30 seconds.

[0077] In step S22, if the server control unit 35 determines that voice data for a predetermined time or longer has not been acquired (step S22: NO), the process returns to step S21 and waits again for reception of voice data.

[0078] In step S22, when the server control unit 35 determines that voice data of a predetermined time or more has been acquired (step S22: YES), it causes the acoustic feature extraction unit 37 to extract acoustic features of customer voice data from the received voice data (step S23 as an acoustic feature acquisition step). In step S23, the acoustic feature extraction unit 37 calculates, for example, numerical values ​​indicating characteristics such as voice volume, voice pitch, and timbre as features or characteristics of the voice of the customer CL based on the received voice data.

[0079] In step S23, the acoustic feature extraction unit 37 calculates a numerical value indicating the speaking speed of the customer CL, for example, from the intervals between phonemes in the voice waveform of the customer voice data. Also, in step S23, the acoustic feature extraction unit 37 calculates a numerical value indicating a voice pitch feature, for example, from the fundamental frequency of the customer voice data.

[0080] After executing step S23, the server control unit 35 causes the emotion identification unit 39 to identify the emotion of the customer CL based on the acoustic feature extracted in step S23 (step S24 as an emotion identification step). In step S24, for example, the emotion identification unit 39 identifies to which of a plurality of predetermined items indicating types of human emotions the acoustic feature extracted in step S23 corresponds.

[0081] For example, the predetermined plurality of items indicating the type of human emotion may include items such as "calm," "tension," "anger," etc. Furthermore, the predetermined plurality of items indicating the type of human emotion may include items such as "sadness," "confusion," "dissatisfaction," "disgust," "contempt," "resignation," "joy," "expectation," "trust," etc.

[0082] After executing step S24, the server control unit 35 causes the type estimation unit 41 to execute a type estimation subroutine to estimate the communication type of customer CL (step S25 as an estimation step). In step S25, for example, the type estimation unit 41 identifies the communication type of customer CL using a learning model based on the acoustic features of customer CL extracted in step S23 and the emotion of customer CL identified in step S24. In step S25, for example, the type estimation unit 41 identifies which of the four communication types the communication type of customer CL belongs to.

[0083] After executing step S25, the server control unit 35 causes the advice generation unit 45 to generate advice information including advice for the user UR regarding communication with the customer CL based on the estimation result in step S25 (step S26). In step S26, for example, the advice generation unit 45 selects advice corresponding to the communication type identified in step S25 from advice previously associated with each communication type, and generates advice information. For example, the advice information includes information indicating recommended topics and ways of proceeding in a conversation with the customer CL.

[0084] After executing step S26, the server control unit 35 transmits the advice information generated in step S25 to the terminal 13 via the transmitting / receiving unit 33 (step S27).

[0085] After executing step S27, the server control unit 35 ends the type estimation routine RT2, and then repeatedly executes the type estimation routine RT2.

[0086] FIG. 7 is a flowchart showing the type estimation subroutine RT3 executed by the type estimation unit 41 of the server control unit 35 in step S25 of the type estimation routine RT2 (FIG. 6).

[0087] When the type estimation subroutine starts, the type estimation unit 41 inputs information indicating the acoustic features of the voice uttered by the customer CL and information indicating the emotion of the customer CL to a learning model (step S31). In step S31, for example, the type estimation unit 41 inputs the information indicating the acoustic features and the information indicating the emotion to a neural network stored in the learning model storage unit 47A as shown in FIG.

[0088] The information indicating the acoustic feature may be a numerical value obtained by quantifying each item such as "speaking rate" or "standard deviation of fundamental frequency" using a predetermined method. The information indicating the emotion may be a numerical value assigned in advance to an item indicating a predetermined type of emotion.

[0089] After executing step S31, the type estimation unit 41 acquires information output from the learning model (step S32). In step S32, for example, the type estimation unit 41 acquires a numerical value indicating the probability that the communication type of the customer CL matches each of the four communication types.

[0090] After executing step S32, the type estimation unit 41 identifies which communication type the communication type of customer CL corresponds to (step S33). In step S33, for example, the type estimation unit 41 identifies the communication type corresponding to the largest numerical value among the four numerical values ​​acquired in step S32 as the communication type of customer CL.

[0091] After executing step S33, the type estimation unit 41 ends the type estimation subroutine.

[0092] 8 is a flowchart showing a message display routine RT4, which is an example of a routine executed by the terminal control unit 28. The terminal control unit 28 starts the message display routine RT4 when, for example, the power of the terminal 13 is turned on.

[0093] When the message display routine RT4 starts, the terminal control unit 28 determines whether or not the start button has been pressed (step S41). In step S41, the terminal control unit 28 determines via the input / output unit 26 whether or not an operation to instruct the start of information processing for estimating the communication type has been accepted on the operation screen displayed on the touch panel 15.

[0094] If the terminal control unit 28 determines in step S41 that the start button has not been pressed (step S41: NO), it returns to the beginning of the message display routine RT4 and determines again whether the start button has been pressed.

[0095] When the terminal control unit 28 determines in step S41 that the start button has been pressed (step S41: YES), the terminal control unit 28 waits for reception of advice information from the server 11 (step S42).

[0096] After executing step S42, the terminal control unit 28 determines whether or not advice information has been received from the server 11 via the transmitting / receiving unit 19 (step S43).

[0097] When the terminal control unit 28 determines in step S43 that advice information has been received (step S43: YES), it causes a message to be displayed on the touch panel 15 via the display control unit 29 based on the received advice information (step S44). For example, in step S44, messages such as "Let's get to the conclusion first," "Let's make small talk first," and "Let's talk in a respectful manner" are displayed on the touch panel 15.

[0098] If the terminal control unit 28 determines in step S43 that advice information has not been received (step S43: NO) or after executing step S44, the terminal control unit 28 determines whether the end button has been pressed (step S45). In step S45, the terminal control unit 28 determines whether an operation to instruct the end of information processing for estimating the communication type has been accepted on the operation screen displayed on the touch panel 15.

[0099] If the terminal control unit 28 determines in step S45 that the end button has not been pressed (step S45: NO), the process returns to step S42 and waits again for reception of advice information.

[0100] When the terminal control unit 28 determines in step S45 that the end button has been pressed (step S45: YES), it ends waiting for reception of advice information from the server 11 (step S46).

[0101] After step S46 is completed, the terminal control unit 28 ends the message display routine RT4, and then repeatedly executes the message display routine RT4.

[0102] As described above, the information processing system 10 as an information processing device of this embodiment can extract acoustic features from a voice of one person with whom a user is communicating, and identify the emotion of the one person from the acoustic features. Furthermore, the communication type of the one person can be estimated using a learning model based on information indicating the acoustic features and information indicating the emotion of the one person.

[0103] Therefore, even if the emotion of a person influences and changes the characteristics of the voice uttered by that person, the influence of the emotion can be reduced and the communication type can be appropriately estimated. Therefore, it is possible to provide an information processing device that is less influenced by the emotion of the person with whom the user is communicating and that can appropriately identify the communication type of the person with whom the user is communicating.

[0104] Furthermore, the information processing system 10 can provide the user with a message containing appropriate advice according to the communication type by appropriately estimating the communication type, thereby enabling the user to communicate more smoothly.

[0105] [Variation 1] 9 to 11, an information processing system 10 according to a first modified example of the present embodiment will be described. The information processing system 10 according to the first modified example has the same configuration as the information processing system 10 described in the embodiment, and differs only in the information processing relating to the identification of emotions.

[0106] Specifically, Modification 1 is common to the embodiment in that information indicating acoustic features and information indicating emotions are used when estimating the communication type of the other person (customer CL). In the embodiment, the emotions of customer CL are identified based on acoustic features. In contrast, in Modification 1, the emotions of customer CL are identified based on situation information, which is information indicating the situation of communication between user UR and customer CL.

[0107] For example, the situation of communication between the user UR and the customer CL is the situation when the speech from which the acoustic features used to estimate the communication type are extracted is uttered by the customer CL.

[0108] The context of communication is, for example, the context itself, such as a business meeting, an interview, or customer service at a store, or is identified based on the context. In addition to context information, other information that contributes to identifying the context of communication includes the age, gender, hometown, language used, occupation, educational background, career history, and other information of the people involved in the communication.

[0109] Since people's emotions are often influenced by the above-mentioned situations when communicating, in this modification 1, the emotions of the communication partner are identified based on information indicating the communication situation. Similar to the above-mentioned embodiment, this modification 1 will be described below using as an example a situation in which a user UR is negotiating with a customer CL.

[0110] Fig. 9 shows an operation screen 15A, which is an example of an operation screen displayed on the touch panel 15 of the terminal 13. The operation screen shown in Fig. 9 is displayed, for example, when a user UR performs an operation to instruct the start of information processing related to a communication type (for example, when the start button is pressed).

[0111] 9, a message prompting the user UR to select a communication situation is displayed on the operation screen 15A. Below the message, buttons for "Customer Service," "Interview," and "Business Negotiation" are displayed. The user UR selects, for example, the item that is closest to the communication situation that will take place.

[0112] For example, if the communication situation in which user UR is engaged is a customer service situation in a retail store or department store, "customer service" would be the item closest to the communication situation. For example, if the communication situation in which user UR is engaged is a job interview and user UR is the interviewer, "interview" would be the item closest to the communication situation.

[0113] In this first modification, the communication situation that the user UR is in is a business negotiation situation, so the user UR selects the "business negotiation" item. Note that the communication situation options displayed on the operation screen 15A are not limited to the example shown in Fig. 9. For example, many more items may be included in addition to the items shown in Fig. 9.

[0114] 10 is a flowchart showing a status information transmission routine RT5, which is an example of a routine executed by the terminal control unit 28 of the terminal 13 in this modification 1. For example, when the power of the terminal 13 is turned on, the terminal control unit 28 starts the status information transmission routine RT5.

[0115] When the situation information transmission routine RT5 starts, the terminal control unit 28 determines whether or not the start button has been pressed via the input / output unit 26 (step S51). In step S51, the terminal control unit 28 determines whether or not an operation to instruct the start of information processing for estimating the communication type has been accepted on the operation screen displayed on the touch panel 15.

[0116] If the terminal control unit 28 determines in step S51 that the start button has not been pressed (step S51: NO), it repeats step S51 and determines again whether or not the start button has been pressed.

[0117] When the terminal control unit 28 determines in step S51 that the start button has been pressed (step S51: YES), it waits for input of information indicating the status of communication (status information) (step S52).

[0118] After executing step S52, the terminal control unit 28 determines whether or not situation information has been input via the input / output unit 26 (step S53). In step S53, for example, the terminal control unit 28 determines that situation information has been input when any one of the items indicating situations displayed on the operation screen 15A as shown in Fig. 9 is selected by an operation on the touch panel 15.

[0119] When the terminal control unit 28 determines that the status information has been input (step S53: YES), it transmits the status information to the terminal 13 (step S54).

[0120] If it is determined that the situation information has not been input (step S53: NO) or after executing step S54, the terminal control unit 28 determines whether the end button has been pressed (step S55).

[0121] In step S55, the terminal control unit 28 determines via the input / output unit 26 whether or not an operation to instruct the end of information processing for estimating the communication type has been accepted on the operation screen displayed on the touch panel 15, for example.

[0122] If the terminal control unit 28 determines in step S55 that the end button has not been pressed (step S55: NO), the process returns to step S52 and waits again for input of status information.

[0123] When the terminal control unit 28 determines in step S55 that the end button has been pressed (step S55: YES), it ends the status information transmission routine RT5, and thereafter repeatedly executes the status information transmission routine RT5.

[0124] 11 is a flowchart showing a type estimation routine RT6, which is an example of a routine executed by the server control unit 35 of the server 11 in this modified example 1. In Fig. 11, steps that are substantially the same as or equivalent to steps in the type estimation routine RT2 shown in Fig. 6 are given the same reference numerals.

[0125] Specifically, the type estimation routine RT2 includes step S24 of identifying an emotion (FIG. 6). The emotion identification in step S24 is performed based on acoustic features. In contrast, in this first modification, the emotion identification is performed based on situation information.

[0126] The type estimation routine RT6 differs from the type estimation routine RT2 in that it includes steps S61 to S63 instead of step S24, but is otherwise the same routine as RT2.

[0127] When the server control unit 35 starts the type estimation routine RT6, it waits for the reception of situation information from the terminal 13 (step S61).

[0128] After executing step S61, the server control unit 35 determines whether or not status information has been received from the terminal 13 via the transmitting / receiving unit 33 (step S62). In step S62, the server control unit 35 functions as a status acquisition unit. If the server control unit 35 determines in step S62 that status information has not been received (step S62: NO), the process returns to step S61 and waits for status information again.

[0129] In step S62, if the server control unit 35 determines that the situation information has been received (step S62: YES), it causes the emotion identification unit 39 to identify the emotion of the customer CL based on the situation information received from the terminal 13 in step S62 (step S63 as an emotion identification step).

[0130] In step S63, for example, the emotion identification unit 39 identifies to which of a plurality of predetermined items indicating types of human emotions the situation information indicating the communication situation received from the terminal 13 in step S62 corresponds. For example, the plurality of predetermined items indicating types of human emotions are associated in advance with a plurality of situations indicating the communication situation. For example, the predetermined plurality of items indicating the type of human emotion may include items such as "calm," "tension," "anger," etc. Furthermore, the predetermined plurality of items indicating the type of human emotion may include items such as "sadness," "confusion," "dissatisfaction," "disgust," "contempt," "resignation," "joy," "expectation," "trust," etc.

[0131] After executing step S63, the server control unit 35 executes steps S21 to S23 to extract acoustic features, similar to the type estimation routine RT2, and then proceeds to step S25 as an estimation step, where the server control unit 35 estimates the communication type of the customer CL based on the information indicating emotion and the acoustic features in the type estimation subroutine.

[0132] Thereafter, the server control unit 35 causes the advice generation unit 45 to generate advice information including advice for the user UR regarding communication with the customer CL based on the estimation result in step S25 (step S26), and transmits the advice information generated in step S25 to the terminal 13 via the transmission / reception unit 33 (step S27). After executing step S27, the server control unit 35 ends the type estimation routine RT6, and then repeatedly executes the type estimation routine RT6.

[0133] In the type estimation routine RT6, steps S61 to S63 may be executed after steps S21 to S23. Furthermore, steps S61 to S63 and steps S21 to S23 may be executed in parallel.

[0134] As described above, the information processing system as an information processing device of the present modified example 1 can identify the emotion of the customer CL based on the situation, such as the scene in which the communication by the user UR is taking place. Furthermore, the information processing system of the present modified example 1 can estimate the communication type of the customer CL using a learning model based on the information indicating the emotion of the customer CL identified based on the situation in which the communication is taking place and the acoustic features of the voice uttered by the customer CL.

[0135] [Variation 2] 12 and 13, the information processing system 10 of Modification 2 of the present embodiment will be described. The information processing system 10 of Modification 2 has the same configuration as the information processing system 10 described in the embodiment, and only the information processing related to the estimation of the communication type is different.

[0136] Specifically, Modification 2 is common to the above-mentioned embodiment in that a learning model is used to estimate a communication type. As described above, the learning model used in the embodiment is constructed so as to receive numerical values ​​indicating acoustic features and emotions as inputs and to output numerical values ​​indicating an estimation result of a communication type. In contrast, in Modification 2, a separate learning model is constructed for each emotion, and each learning model is constructed so as to receive numerical values ​​indicating acoustic features as inputs and to output an estimation result of a communication type.

[0137] 12 is an example of a learning model used to estimate the communication type of customer CL in Modification 2, and is a diagram schematically showing learning models ME1 to ME3 stored in the learning model storage unit 47A. In Modification 2, an example will be described in which learning models ME1 to ME3 are constructed for each of "calm," "tension," and "anger," which indicate types of human emotions.

[0138] Each of the learning models ME1 to ME3 is a neural network constructed by deep learning, similar to the learning model M1 shown in Fig. 4, and is composed of an input layer, three intermediate layers (hidden layers), and an output layer. Note that the number of intermediate layers is not limited to three, and may be two or more.

[0139] As shown in FIG. 12, each of the learning models ME1 to ME3 receives as input AU1 to AU9, which are nine numerical values ​​indicating nine items of acoustic features, respectively.

[0140] As shown in FIG. 12, each of the learning models ME1 to ME3 outputs a numerical value indicating the probability that the communication type of the customer CL conforms to each of the four communication types according to the social communication theory.

[0141] For example, if the emotion of customer CL is determined to be "calm," numerical values ​​AU1 to AU9 indicating acoustic features based on the speech uttered by customer CL are input into the input layer of learning model ME1 for the emotion "calm" among learning models ME1 to ME3.

[0142] When this input is made, the output layer outputs four numerical values ​​indicating the probability that the communication type of customer CL, the person related to the input information, matches each of the four communication types based on social style theory: "Driver," "Analytical," "Amiable," and "Expressive."

[0143] That is, each of the learning models ME1 to ME3 is constructed so as to receive the acoustic features of a person as input and output information indicating the communication type of the person.

[0144] For example, a learning model ME1 for the emotion "calm" is constructed as follows. For example, the emotion felt by one collaborator during a conversation is identified, and audio data from the conversation is acquired and acoustic features are extracted from the audio data. Furthermore, the communication type of the one collaborator is identified from the results of a questionnaire administered to the one collaborator. In this way, many sets of data that associate emotions, acoustic features, and communication types are collected.

[0145] From the collected data sets, only the data sets for which the emotion was "neutral" were extracted, and the communication types included in the extracted data sets were set as the correct answer data. The acoustic features corresponding to each correct answer data were set as the input data.

[0146] Input data for the "neutral" emotion is fed into a neural network with multiple hidden layers, and learning is performed while updating the neural network parameters using the backpropagation method so that the output is the communication type of the correct data. By performing this type of learning for a large number of pairs of correct data and input data, learning model ME1 can be constructed.

[0147] Similarly, a learning model ME2 can be constructed by learning when the emotion is "tension," and a learning model ME3 can be constructed by learning when the emotion is "anger."

[0148] The emotion to be used for learning may be identified, for example, based on the acoustic features as described in the examples, or may be identified based on the communication situation when the collaborator uttered the voice, as in the second variant example.

[0149] 13 is a flowchart showing the type estimation subroutine RT7 executed by the type estimation unit 41 of the server control unit 35 in this Modification 2. In this Modification 2, the type estimation routine RT2 (FIG. 6) of the above embodiment and the type estimation routine RT6 (FIG. 11) of Modification 1 are applied. The type estimation subroutine RT7 is executed in step S25 of the type estimation routine RT2 (FIG. 6) or step S25 of the type estimation routine RT6 (FIG. 11).

[0150] When the type estimation subroutine starts, the type estimation unit 41 selects a learning model corresponding to the emotion of customer CL (step S71). For example, in step S71, the type estimation unit 41 selects a learning model corresponding to the emotion of customer CL identified in step S24 of the type estimation routine RT2 or step S63 of the type estimation routine RT6 from a plurality of learning models constructed for each type of emotion. For example, in step S71, if the emotion of customer CL is "calm," the learning model ME1 (FIG. 12) constructed to correspond to the emotion "calm" is selected.

[0151] After executing step S71, the type estimation unit 41 inputs numerical values ​​indicating acoustic features to the selected learning model (step S72). In step S72, for example, the type estimation unit 41 inputs numerical values ​​AU1 to AU9 indicating acoustic features to the neural network stored in the learning model storage unit 47A as shown in FIG.

[0152] After executing step S72, the type estimation unit 41 acquires information output from the learning model (step S73). In step S73, for example, the type estimation unit 41 acquires a numerical value indicating the probability that the communication type of the customer CL matches each of the four communication types.

[0153] After executing step S73, the type estimation unit 41 identifies which communication type the communication type of customer CL corresponds to (step S74). In step S74, for example, the type estimation unit 41 identifies the communication type corresponding to the largest numerical value among the four numerical values ​​acquired in step S73 as the communication type of customer CL.

[0154] After executing step S74, the type estimation unit 41 ends the type estimation subroutine.

[0155] As described above, the information processing system 10 as an information processing device of the present modified example 2 can extract acoustic features of a voice of one person who is a communication partner of a user based on the voice of the person, and can estimate the communication type of the one person from the acoustic features. When estimating the communication type, a learning model corresponding to the emotion of the one person can be selected from learning models constructed for each predetermined type of emotion and used for the estimation.

[0156] Therefore, even if the emotions of a person affect the characteristics of the voice uttered by that person, the influence of emotions can be reduced by using a learning model constructed for each emotion, and the communication type can be appropriately estimated. Therefore, it is possible to provide an information processing device that is less affected by the emotions of the person with whom the user is communicating and that is able to appropriately identify the communication type of the person with whom the user is communicating.

[0157] Furthermore, the user can receive a message containing appropriate advice according to the communication type based on the estimation result of the communication type via the terminal 13, which allows the user to communicate more smoothly.

[0158] In the above-described second modification, an example has been described in which emotions are identified from communication situations and a learning model constructed for each emotion is used to estimate a communication type, but this is not limiting. For example, a learning model constructed for each piece of information indicating a communication situation may be used, and a learning model corresponding to the communication situation may be selected to estimate a communication type.

[0159] In the above embodiment and modified examples, the information processing system 10 as an information processing device of the present invention includes the server 11 and the terminal 13. However, the present invention is not limited to this. For example, the terminal 13 may include all of the components corresponding to the acoustic feature extraction unit 37, emotion identification unit 39, type estimation unit 41, advice generation unit 45, and learning model storage unit 47A of the server 11.

[0160] Although the example in which there is one customer CL has been described, type estimation may be performed for multiple people. For example, information processing may be performed in parallel on each of the voice data of multiple people.

[0161] In the above-described embodiment and modification examples, a learning model may be used to identify emotions. For example, in the embodiment or modification example 2, emotions may be identified using a learning model constructed so that numerical values ​​indicating acoustic features are input and numerical values ​​indicating types of emotions are output. Furthermore, for example, in the first modification, emotions may be identified using a learning model constructed so that a numerical value indicating the state of communication is input and a numerical value indicating the type of emotion is output.

[0162] The configurations in the above-described embodiments and modifications are merely examples, and can be appropriately selected and modified depending on the application, etc. [Explanation of symbols]

[0163] 10 Information Processing Systems 11 Server 13 Terminals 15 Touch Panel 17, 31 System Bus 19, 33 Transmitter / receiver 21 Memory section 23 Audio input section 24. Mike 26 Input / output section 28 Terminal control unit 29 Display control unit 35 Server control unit 37 Acoustic feature extraction unit 39 Emotion identification part 41 Type Estimation Section 43 Reasoning part 45 Advice Generation Unit 47 Mass storage 47A Learning model memory section

Claims

1. an acoustic feature acquisition unit that acquires acoustic features of a voice uttered by one person; a situation acquisition unit that acquires situation information input by a user and indicates a communication scene involving the one person or a state of the one person when the voice is uttered; an estimation unit that estimates the communication type of the one person based on the situation information and information indicating the acoustic feature amount using a learning model.

2. an acoustic feature acquisition unit that acquires acoustic features of a voice uttered by one person; a situation acquisition unit that acquires situation information input by a user and indicates a communication scene involving the one person or a state of the one person when the voice is uttered; an emotion identification unit that identifies the emotion of the one person based on the situation information; an estimation unit that estimates the communication type of the one person based on the information indicating the emotion and the information indicating the acoustic feature using a learning model.

3. The information processing device described in claim 1 or 2, characterized in that when information indicating the emotion of the one person identified based on the situation information and information indicating the acoustic features are input, the learning model outputs a numerical value indicating the degree of compatibility between each of a plurality of communication types and the communication type of the one person.

4. 4. The information processing apparatus according to claim 1, further comprising an advice generating unit that generates advice information including advice regarding communication with the one person based on the estimation result by the estimation unit.

5. 5. The information processing apparatus according to claim 4, further comprising a display control unit that controls displaying a message for the user on a display screen based on the advice information.

6. The information processing apparatus according to claim 2 , wherein the emotion identification unit identifies the emotion of the one person using a learning model that outputs information indicating the emotion when the situation information is input.

7. An information processing method in which an information processing device processes information, an acoustic feature acquisition step in which an acoustic feature acquisition unit acquires acoustic features of a voice uttered by one person; a situation acquisition step in which a situation acquisition unit acquires situation information input by a user, the situation information indicating a communication scene involving the one person or a state of the one person when the voice was uttered; an estimation step in which an estimation unit uses a learning model to estimate the communication type of the one person based on the situation information and information indicating the acoustic features.

8. An information processing program executed by an information processing device including a computer, the computer comprising: an acoustic feature acquisition step in which an acoustic feature acquisition unit acquires acoustic features of a voice uttered by one person; a situation acquisition step in which a situation acquisition unit acquires situation information input by a user, the situation information indicating a communication scene involving the one person or a state of the one person when the voice was uttered; an estimation step in which an estimation unit estimates a communication type of the one person based on the situation information and information indicating the acoustic feature using a learning model; An information processing program for executing the above.

9. 9. A computer-readable recording medium storing the information processing program according to claim 8.

Images