Emotion estimation method, information processing device, and program

Abstract

Improve emotion estimation technology. [Solution] An emotion estimation method executed by an information processing device 10, comprising: acquiring audio data; determining whether or not the audio data contains linguistic information; if the audio data contains linguistic information, inputting the audio data into a first estimation model that estimates emotions based on linguistic information to estimate the emotion corresponding to the audio data; and if the audio data does not contain linguistic information, inputting the audio data into a second estimation model that estimates emotions without relying on linguistic information to estimate the emotion corresponding to the audio data.

Description

Technical Field

[0001] The present disclosure relates to an emotion estimation method, an information processing apparatus, and a program.

Background Art

[0002] Conventionally, techniques for analyzing the content of business negotiations have been known. For example, Patent Document 1 discloses a dialogue analysis system that checks whether a salesperson explains what should be explained and does not mention what should not be mentioned in a business negotiation with a customer.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The success or failure of a business negotiation may be related to the customer's emotion. Patent Document 1 does not disclose estimating the emotion of a customer or the like by utilizing machine learning or the like. Also, not limited to business negotiations, if emotion can be estimated from voice data, it can lead to improvements in customer response and the quality of customer support, but emotion estimation technology has not been sufficiently studied so far. Thus, there has been room for improvement in emotion estimation technology.

[0005] An object of the present disclosure made in view of such circumstances is to improve emotion estimation technology.

Means for Solving the Problems

[0006] An emotion estimation method according to an embodiment of the present disclosure is an emotion estimation method executed by an information processing apparatus, acquiring voice data, determining whether the voice data includes language information, If the audio data includes linguistic information, the audio data is input into a first estimation model that estimates emotions based on linguistic information to estimate the emotion corresponding to the audio data; if the audio data does not include linguistic information, the audio data is input into a second estimation model that estimates emotions without relying on linguistic information to estimate the emotion corresponding to the audio data. Includes. [Effects of the Invention]

[0007] According to one embodiment of this disclosure, emotion estimation technology is improved. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing the schematic configuration of the system according to this embodiment. [Figure 2] This is a flowchart showing the operation of an information processing device. [Figure 3] This is a flowchart showing the operation of an information processing device. [Modes for carrying out the invention]

[0009] The embodiments of this disclosure will be described below.

[0010] (Summary of the embodiment) Referring to Figure 1, the overview and configuration of System 1 according to this embodiment will be described. System 1 according to this embodiment comprises an information processing device 10 and a terminal device 20. The information processing device 10 and the terminal device 20 are communicated with a network 30, which includes, for example, a mobile communication network and the Internet.

[0011] The information processing device 10 is, for example, a server device installed in a data center or the like. For example, the information processing device 10 is a server belonging to a cloud computing system or other computing system. Although Figure 1 shows an example where system 1 has one information processing device 10, it is not limited to this. System 1 may have two or more information processing devices 10.

[0012] The terminal device 20 is any device used by each user. For example, general-purpose electronic devices such as smartphones, tablet devices, and wearable devices, or dedicated electronic devices, can be used as the terminal device 20. Although Figure 1 shows an example where System 1 has one terminal device 20, it is not limited to this. System 1 may have two or more terminal devices 20.

[0013] In emotion estimation techniques, supervised learning models that use features extracted from audio data and emotion labels estimate the emotion associated with the audio data using a single emotion estimation model based on information related to the audio data (linguistic information, paralinguistic information, and nonverbal information). Here, linguistic information is information indicating the content of the utterance based on the audio data. Paralinguistic information is information such as emotion, attitude, and intention based on the audio data. Nonverbal information is information such as the speaker's age and gender based on the audio data.

[0014] In emotion estimation techniques using supervised learning models, a single general-purpose emotion estimation model is employed, but depending on the content of the audio data, the optimal estimation result is not always obtained. The emotion estimation technique according to this embodiment is characterized in that, in general terms, the estimation process is performed by switching between multiple estimation models rather than using just one estimation model.

[0015] Hereinafter, the outline of the emotion estimation technology according to this embodiment will be described, and the details will be described later. The emotion estimation technology according to this embodiment is executed by the information processing apparatus 10. First, the information processing apparatus 10 acquires voice data such as in a negotiation. The information processing apparatus 10 determines whether the voice data includes language information. When the voice data includes language information, the information processing apparatus 10 inputs the voice data into a first estimation model that estimates emotion based on the language information, and estimates the emotion corresponding to the voice data. On the other hand, when the voice data does not include language information, the information processing apparatus 10 inputs the voice data into a second estimation model that estimates emotion without relying on language information, and estimates the emotion corresponding to the voice data.

[0016] As described above, according to this embodiment, the information processing apparatus 10 determines whether the voice data includes language information, and switches between the first estimation model and the second estimation model according to the content of the voice data to estimate emotion. In this way, the emotion estimation technology is improved in that the optimal estimation process according to the content of the voice data can be executed.

[0017] Next, each configuration of the information processing apparatus 10 and the terminal apparatus 20 will be described in detail.

[0018] (Configuration of Information Processing Apparatus) As shown in FIG. 1, the information processing apparatus 10 includes a control unit 11, a storage unit 12, an input unit 13, an output unit 14, and a communication unit 15.

[0019] The control unit 11 includes at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a CPU (central processing unit) or GPU (graphics processing unit), or a dedicated processor specialized for specific processing. The dedicated circuit is, for example, an FPGA (field-programmable gate array) or an ASIC (application specific integrated circuit). While controlling each part of the information processing device 10, the control unit 11 executes processing related to the operation of the information processing device 10.

[0020] The storage unit 12 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a RAM (random access memory) or a ROM (read only memory). The RAM is, for example, a SRAM (static random access memory) or a DRAM (dynamic random access memory). The ROM is, for example, an EEPROM (electrically erasable programmable read only memory). The storage unit 12 functions as, for example, a main storage device, an auxiliary storage device, or a cache memory. In the storage unit 12, data used for the operation of the information processing device 10 and data obtained by the operation of the information processing device 10 are stored.

[0021] For example, the memory unit 12 stores the first estimation model and the second estimation model. As described above, the first estimation model is a model that estimates emotion based on linguistic information. Specifically, the first estimation model is a model created by machine learning using a machine learning algorithm. The first estimation model may be a machine learning model constructed on the basis of a decision tree, for example. Machine learning models constructed on the basis of a decision tree include, but are not limited to, Light GBM and XGBoost. Alternatively, the machine learning model may be a model generated based on a machine learning algorithm such as a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), or other deep learning methods. The first estimation model according to this embodiment outputs the emotion corresponding to the speech data as the target variable, based on explanatory variables relating to linguistic information, paralinguistic information, and non-linguistic information of the speech data.

[0022] On the other hand, the second estimation model is a model that estimates emotion without relying on linguistic information. Specifically, the second estimation model is a model created by machine learning using a machine learning algorithm. The second estimation model may be a machine learning model constructed based on a decision tree, for example. Examples of machine learning models constructed based on a decision tree include Light GBM and XGBoost, but are not limited to these. Alternatively, the machine learning model may be a model generated based on a machine learning algorithm such as a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), or other deep learning methods. The second estimation model outputs the emotion corresponding to the speech data as the target variable, based on explanatory variables relating to at least one of the paralinguistic and nonlinguistic information of the speech data.

[0023] When audio data contains linguistic information, it is preferable to use a first estimation model that estimates emotions based at least on the linguistic information of the audio data. This is because emotion estimation that takes into account the linguistic information of the audio data is likely to be highly accurate. On the other hand, when audio data does not contain linguistic information, it is preferable to use a second estimation model that estimates emotions without relying on the linguistic information of the audio data. This is because the second estimation model can perform emotion estimation that is specialized for at least one of paralinguistic information and non-linguistic information. As described above, in the accounting estimation technology according to this embodiment, it is determined whether or not the audio data contains linguistic information, and the first estimation model and the second estimation model are switched according to the content of the audio data to estimate emotions. In this way, the optimal estimation process is performed according to the content of the audio data.

[0024] The input unit 13 includes at least one input interface. The input interface may be, for example, a physical key, a capacitive key, a pointing device, or a touchscreen integrated with a display. Alternatively, the input interface may be, for example, a sound sensor that accepts voice input, or a camera that accepts gesture input. The input unit 13 accepts operations to input data used for the operation of the information processing device 10. Instead of being integrated into the information processing device 10, the input unit 13 may be connected to the information processing device 10 as an external input device. Any connection method can be used, for example, USB (Universal Serial Bus), HDMI (registered trademark) (High-Definition Multimedia Interface), or Bluetooth (registered trademark).

[0025] The output unit 14 includes at least one output interface. The output interface is, for example, a display that outputs information as video, or a speaker that outputs information as sound. The display is, for example, an LCD (liquid crystal display) or an organic EL (electroluminescence) display. The output unit 14 outputs data obtained by the operation of the information processing device 10. Instead of being provided in the information processing device 10, the output unit 14 may be connected to the information processing device 10 as an external output device. Any connection method can be used, for example, USB, HDMI (registered trademark), or Bluetooth (registered trademark).

[0026] The communication unit 15 includes at least one external communication interface. The communication interface may be either a wired or wireless communication interface. In the case of wired communication, the communication interface may be, for example, a LAN (Local Area Network) interface or a USB (Universal Serial Bus) interface. In the case of wireless communication, the communication interface may be, for example, an interface compatible with mobile communication standards such as LTE (Long Term Evolution), 4G (4th generation), or 5G (5th generation), or an interface compatible with short-range wireless communication such as Bluetooth (registered trademark). The communication unit 15 receives data used for the operation of the information processing device 10 and transmits data obtained by the operation of the information processing device 10.

[0027] The functions of the information processing device 10 are realized by executing the program according to this embodiment on a processor corresponding to the control unit 11. In other words, the functions of the information processing device 10 are realized by software. The program causes the computer to perform the operations of the information processing device 10, thereby causing the computer to function as the information processing device 10. That is, the computer functions as the information processing device 10 by performing the operations of the information processing device 10 according to the program.

[0028] In this embodiment, the program can be recorded on a computer-readable recording medium. The computer-readable recording medium includes non-temporary computer-readable media, such as magnetic recording devices, optical discs, magneto-optical recording media, or semiconductor memory. The program can be distributed, for example, by selling, transferring, or lending portable recording media such as DVDs (digital versatile discs) or CD-ROMs (compact disc read-only memory) on which the program is recorded. Alternatively, the program may be distributed by storing it on the storage of an external server and transmitting it from the external server to other computers. The program may also be provided as a program product.

[0029] Some or all of the functions of the information processing device 10 may be implemented by a dedicated circuit corresponding to the control unit 11. In other words, some or all of the functions of the information processing device 10 may be implemented by hardware.

[0030] (Terminal device configuration) As shown in Figure 1, the terminal device 20 comprises a control unit 21, a storage unit 22, an input unit 23, an output unit 24, and a communication unit 25.

[0031] The control unit 21 includes at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a CPU (central processing unit) or GPU (graphics processing unit), or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). The control unit 21 controls each part of the terminal device 20 and executes processes related to the operation of the terminal device 20.

[0032] The storage unit 22 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or at least two combinations thereof. The semiconductor memory is, for example, RAM (random access memory) or ROM (read-only memory). The RAM is, for example, SRAM (static random access memory) or DRAM (dynamic random access memory). The ROM is, for example, EEPROM (electrically erasable programmable read-only memory). The storage unit 22 functions, for example, as a main memory, auxiliary memory, or cache memory. The storage unit 22 stores data used for the operation of the terminal device 20 and data obtained by the operation of the terminal device 20.

[0033] The input unit 23 includes at least one input interface. The input interface may be, for example, a physical key, a capacitive key, a pointing device, or a touchscreen integrated with a display. Alternatively, the input interface may be, for example, a sound sensor that accepts voice input, or a camera that accepts gesture input. The input unit 23 accepts operations to input data used for the operation of the terminal device 20. Instead of being provided in the terminal device 20, the input unit 23 may be connected to the terminal device 20 as an external input device. Any connection method can be used, for example, USB (Universal Serial Bus), HDMI (registered trademark) (High-Definition Multimedia Interface), or Bluetooth (registered trademark).

[0034] The output unit 24 includes at least one output interface. The output interface is, for example, a display that outputs information as video, or a speaker that outputs information as sound. The display is, for example, an LCD (liquid crystal display) or an organic EL (electroluminescence) display. The output unit 24 outputs data obtained by the operation of the terminal device 20. Instead of being provided in the terminal device 20, the output unit 24 may be connected to the terminal device 20 as an external output device. Any connection method can be used, for example, USB, HDMI (registered trademark), or Bluetooth (registered trademark).

[0035] The communication unit 25 includes at least one external communication interface. The communication interface may be either a wired or wireless communication interface. In the case of wired communication, the communication interface may be, for example, a LAN (Local Area Network) interface or a USB (Universal Serial Bus) interface. In the case of wireless communication, the communication interface may be, for example, an interface compatible with mobile communication standards such as LTE (Long Term Evolution), 4G (4th generation), or 5G (5th generation), or an interface compatible with short-range wireless communication such as Bluetooth (registered trademark). The communication unit 25 receives data used for the operation of the terminal device 20 and transmits data obtained by the operation of the terminal device 20.

[0036] The functions of the terminal device 20 are realized by executing the program according to this embodiment on a processor corresponding to the control unit 21. In other words, the functions of the terminal device 20 are realized by software. The program causes the computer to perform the operations of the terminal device 20, thereby causing the computer to function as the terminal device 20. That is, the computer functions as the terminal device 20 by performing the operations of the terminal device 20 according to the program.

[0037] Some or all of the functions of the terminal device 20 may be implemented by a dedicated circuit corresponding to the control unit 21. In other words, some or all of the functions of the terminal device 20 may be implemented by hardware.

[0038] (Operation of information processing device) Referring to Figure 2, the operation of the information processing device 10 according to this embodiment will be described.

[0039] Step S10: The control unit 11 of the information processing device 10 acquires audio data.

[0040] Any method can be used to acquire the audio data. For example, the control unit 11 may acquire audio data from external devices, including the terminal device 20, via the communication unit 15 and the network 30. Alternatively, the control unit 11 may acquire audio data via the input unit 13. The audio data includes the voice of a specific speaker in a business negotiation, meeting, etc. However, the audio data is not limited to this and may include any data such as presentations, telephone conversations with customers, customer support, interactions in educational settings, interviews, everyday conversations, and audio posts on social media. The specific speaker may be any speaker, such as a customer or staff member in a business negotiation. In this embodiment, the business negotiation is, for example, a negotiation related to the sale of a vehicle, but is not limited to this. For example, the business negotiation may be a meeting aimed at concluding various types of contracts, such as the buying and selling of real estate, insurance product contracts, or the sale of financial products. In this embodiment, the emotions of one specific speaker in the audio data are estimated, but are not limited to this. For example, the emotions of multiple speakers in the audio data may be estimated.

[0041] Step S20: The control unit 11 determines whether or not the audio data contains language information.

[0042] Any method can be used to determine whether or not the audio data contains language information. For example, the control unit 11 may determine whether or not the audio data contains language information by performing speech recognition processing, transcription processing, etc. on the audio data. If the audio data contains language information, the process proceeds to step S30. If the audio data does not contain language information, the process proceeds to step S40.

[0043] Step S30: If the voice data contains linguistic information, the control unit 11 inputs the voice data into the first estimation model to estimate the emotion corresponding to the voice data. As described above, the first estimation model is a model that estimates emotions based on linguistic information.

[0044] Step S40: If the voice data does not contain linguistic information, the control unit 11 inputs the voice data into the second estimation model to estimate the emotion corresponding to the voice data. As described above, the second estimation model is a model that estimates emotions without relying on linguistic information.

[0045] Step S50: The control unit 11 outputs the emotion estimation result corresponding to the voice data. Specifically, the control unit 11 outputs the estimation result estimated by the first estimation model in step S30, or the estimation result estimated by the second estimation model in step S40.

[0046] Any method can be used for outputting the estimation results. For example, the control unit 11 may transmit data related to the estimation results to the terminal device 20 via the communication unit 15, and the terminal device 20 may output the estimation results via the output unit 24. The control unit 21 may output the estimation results via a user interface displayed by the output unit 24. Alternatively, the control unit 11 may output the estimation results via a user interface displayed by the output unit 14.

[0047] As described above, the information processing device 10 according to this embodiment acquires voice data, determines whether the voice data contains language information, inputs the voice data to a first estimation model that estimates emotions based on language information if the voice data contains language information, and estimates the emotion corresponding to the voice data if the voice data does not contain language information, inputs the voice data to a second estimation model that estimates emotions without relying on language information, and estimates the emotion corresponding to the voice data.

[0048] With this configuration, the information processing device 10 determines whether or not the audio data contains linguistic information, and switches between the first estimation model and the second estimation model according to the content of the audio data to estimate emotion. In this way, emotion estimation technology is improved because it can perform the optimal estimation process according to the content of the audio data.

[0049] The first estimation model may perform a process to estimate two types of emotions: expressed emotions and intrinsic emotions. Expressed emotions are emotions expressed through linguistic information. In this embodiment, expressed emotions are also called verbal emotions. Intrinsic emotions are emotions or sensations that reside in the mind and are not expressed as linguistic information. In other words, intrinsic emotions are emotions expressed through at least one of paralinguistic information and nonlinguistic information. In this embodiment, intrinsic emotions are also called actual emotions. For example, the first estimation model may separate audio data into first vector data corresponding to paralinguistic and nonlinguistic information and second vector data corresponding to linguistic information, and estimate the difference in emotions (hereinafter also called the emotion gap) based on the first vector data and the second vector data. In this embodiment, this model is also called the emotion gap model. When the first estimation model is an emotion gap model, the actual emotion can be estimated by performing an emotion estimation process based on the first vector data, and the verbal emotion can be estimated by performing an emotion estimation process based on the second vector data. This makes it possible to estimate the emotion gap between the actual emotion and the verbal emotion.

[0050] The emotion gap model may be trained based on a first loss function based on the difference between linguistic information based on speech data and the first vector data, a second loss function based on symmetric or asymmetric learning of the second vector data, and a third loss function that minimizes the mutual information between the first and second vector data. Symmetric learning may include simCLR. Asymmetric learning may include BYOL, SimSiam, or DINO. CLUB or DiCy may be used in training related to the third loss function.

[0051] When the first estimation model estimates two emotions—the emotion expressed in the words and the actual emotion—and the two do not match, the system may perform emotion estimation using a second estimation model and output the result estimated by the second estimation model. This is because if the estimated emotion expressed in the words differs from the estimated actual emotion, there is a high risk that the emotion expressed in the words does not reflect the speaker's true emotion. In this case, the system may output the result of a second estimation model that estimates emotions without relying on linguistic information. Figure 3 is a flowchart of the operation of the information processing device 10 that performs the aforementioned process. The same reference numerals are used for operations identical to those in Figure 2, and their explanations are omitted.

[0052] Step S41: Following step S30 in Figure 3, the control unit 11 of the information processing device 10 determines whether the emotion expressed in the words matches the actual emotion. If the emotion expressed in the words matches the actual emotion, the process proceeds to step S50. If the emotion expressed in the words does not match the actual emotion, the process proceeds to step S42.

[0053] Step S42: If the emotion expressed in the words does not match the actual emotion, the control unit 11 inputs the voice data into the second estimation model to estimate the emotion corresponding to the voice data.

[0054] While this disclosure has been described based on the drawings and embodiments, it should be noted that those skilled in the art may make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of this disclosure. For example, the functions, etc., included in each component or step can be rearranged in a logically consistent manner, and multiple components or steps can be combined into one or divided into two.

[0055] For example, in the above-described embodiment, it is also possible to distribute the configuration and operation of the information processing device 10 across multiple computers that can communicate with each other. Similarly, it is also possible to distribute the configuration and operation of the terminal device 20 across multiple computers that can communicate with each other.

[0056] Some embodiments of the present disclosure are described below. However, it should be noted that the embodiments of the present disclosure are not limited to these. [Note 1] A method for estimating emotions performed by an information processing device, Acquiring audio data and, Determining whether the aforementioned audio data contains language information, If the audio data includes linguistic information, the audio data is input into a first estimation model that estimates emotions based on linguistic information to estimate the emotion corresponding to the audio data; if the audio data does not include linguistic information, the audio data is input into a second estimation model that estimates emotions without relying on linguistic information to estimate the emotion corresponding to the audio data. A method for estimating emotions, including the following: [Note 2] The emotion estimation method described in Appendix 1, The first estimation model is a model that estimates emotions based on linguistic information, paralinguistic information, and nonverbal information, and is a method for estimating emotions. [Note 3] A method for estimating emotions as described in Appendix 1 or Appendix 2, The first estimation model is an emotion estimation method that separates the speech data into first vector data corresponding to paralinguistic and nonlinguistic information and second vector data corresponding to linguistic information, and estimates emotion based on the first vector data and the second vector data. [Note 4] A method for estimating emotions as described in any one of the items in Appendix 1 to Appendix 3, The second estimation model is a model that estimates emotions based on at least one of paralinguistic information and nonlinguistic information, and is used as an emotion estimation method. [Note 5] A method for estimating emotions as described in any one of the items in Appendix 1 to Appendix 4, A method for estimating emotion, wherein, if the emotion expressed in the words does not match the emotion expressed in the results estimated by the first estimation model, the voice data is input into a second estimation model to estimate the emotion corresponding to the voice data. [Note 6] An information processing device comprising a control unit, The control unit, Acquire audio data, Determine whether the aforementioned audio data contains language information, An information processing device that, if the audio data includes linguistic information, inputs the audio data into a first estimation model that estimates emotions based on linguistic information to estimate the emotion corresponding to the audio data, and if the audio data does not include linguistic information, inputs the audio data into a second estimation model that estimates emotions without relying on linguistic information to estimate the emotion corresponding to the audio data. [Note 7] The information processing device described in Appendix 6, The first estimation model is an information processing device that estimates emotions based on linguistic information, paralinguistic information, and nonverbal information. [Note 8] An information processing device as described in Appendix 6 or Appendix 7, The first estimation model is an information processing device that separates the audio data into first vector data corresponding to paralinguistic and nonlinguistic information and second vector data corresponding to linguistic information, and estimates emotion based on the first vector data and the second vector data. [Note 9] An information processing device described in any one of the items 6 to 8 of the appendix, The second estimation model is an information processing device that estimates emotions based on at least one of paralinguistic information and nonlinguistic information. [Note 10] A method for estimating emotions as described in any one of the items in Appendix 6 to Appendix 9, An information processing device that, if the emotion expressed in the words does not match the emotion expressed in the results estimated by the first estimation model, inputs the voice data into a second estimation model to estimate the emotion corresponding to the voice data. [Note 11] On the computer, Acquiring audio data and, Determining whether the aforementioned audio data contains language information, If the audio data includes linguistic information, the audio data is input into a first estimation model that estimates emotions based on linguistic information to estimate the emotion corresponding to the audio data; if the audio data does not include linguistic information, the audio data is input into a second estimation model that estimates emotions without relying on linguistic information to estimate the emotion corresponding to the audio data. A program that executes the command. [Note 12] The program described in Appendix 11, The first estimation model is a program that estimates emotions based on linguistic information, paralinguistic information, and nonverbal information. [Note 13] The program described in Appendix 11 or Appendix 12, The first estimation model is a program that separates the audio data into first vector data corresponding to paralinguistic and nonlinguistic information and second vector data corresponding to linguistic information, and estimates emotion based on the first vector data and the second vector data. [Note 14] A program described in any one of the items in Appendix 11 to Appendix 13, The second estimation model is a program that estimates emotions based on at least one of paralinguistic information and nonlinguistic information. [Note 15] A program described in any one of the items in Appendix 11 to Appendix 14, A program that, if the emotion expressed in the words does not match the emotion expressed in the results estimated by the first estimation model, inputs the audio data into a second estimation model to estimate the emotion corresponding to the audio data. [Explanation of Symbols]

[0057] 10 Information Processing Devices 11 Control Unit 12 Storage section 13 Input section 14 Output section 15 Communications Department 20 Terminal devices 21 Control Unit 22 Memory section 23 Input section 24 Output section 25 Communications Department 30 Networks

Claims

1. A method for estimating emotions performed by an information processing device, Acquiring audio data and, Determining whether the aforementioned audio data contains language information, If the audio data includes linguistic information, the audio data is input into a first estimation model that estimates emotions based on linguistic information to estimate the emotion corresponding to the audio data; if the audio data does not include linguistic information, the audio data is input into a second estimation model that estimates emotions without relying on linguistic information to estimate the emotion corresponding to the audio data. A method for estimating emotions, including the following:

2. A method for estimating emotions according to claim 1, The first estimation model is a model that estimates emotions based on linguistic information, paralinguistic information, and nonverbal information, and is an emotion estimation method.

3. A method for estimating emotions according to claim 2, The first estimation model is an emotion estimation method that separates the speech data into first vector data corresponding to paralinguistic and nonlinguistic information and second vector data corresponding to linguistic information, and estimates emotion based on the first vector data and the second vector data.

4. A method for estimating emotions according to claim 1, The second estimation model is a model that estimates emotions based on at least one of paralinguistic information and nonlinguistic information, and is used as an emotion estimation method.

5. A method for estimating emotions according to claim 1, A method for estimating emotion, wherein, if the emotion expressed in the words does not match the emotion expressed in the results estimated by the first estimation model, the voice data is input into a second estimation model to estimate the emotion corresponding to the voice data.

6. An information processing device comprising a control unit, The control unit, Acquire audio data, Determine whether the aforementioned audio data contains language information, An information processing device that, if the audio data includes linguistic information, inputs the audio data into a first estimation model that estimates emotions based on linguistic information to estimate the emotion corresponding to the audio data, and if the audio data does not include linguistic information, inputs the audio data into a second estimation model that estimates emotions without relying on linguistic information to estimate the emotion corresponding to the audio data.

7. An information processing apparatus according to claim 6, The first estimation model is an information processing device that estimates emotions based on linguistic information, paralinguistic information, and nonverbal information.

8. An information processing apparatus according to claim 7, The first estimation model is an information processing device that separates the audio data into first vector data corresponding to paralinguistic and nonlinguistic information and second vector data corresponding to linguistic information, and estimates emotion based on the first vector data and the second vector data.

9. An information processing apparatus according to claim 6, The second estimation model is an information processing device that estimates emotions based on at least one of paralinguistic information and nonlinguistic information.

10. A method for estimating emotions according to claim 6, An information processing device that, if the emotion expressed in the words does not match the emotion expressed in the result estimated by the first estimation model, inputs the voice data into a second estimation model to estimate the emotion corresponding to the voice data.

11. On the computer, Acquiring audio data and, Determining whether the aforementioned audio data contains language information, If the audio data includes linguistic information, the audio data is input into a first estimation model that estimates emotions based on linguistic information to estimate the emotion corresponding to the audio data; if the audio data does not include linguistic information, the audio data is input into a second estimation model that estimates emotions without relying on linguistic information to estimate the emotion corresponding to the audio data. A program that executes the command.

12. The program according to claim 11, The first estimation model is a program that estimates emotions based on linguistic information, paralinguistic information, and nonverbal information.

13. The program according to claim 12, The first estimation model is a program that separates the audio data into first vector data corresponding to paralinguistic and nonlinguistic information and second vector data corresponding to linguistic information, and estimates emotion based on the first vector data and the second vector data.

14. The program according to claim 11, The second estimation model is a program that estimates emotions based on at least one of paralinguistic information and nonlinguistic information.

15. The program according to claim 11, A program that, if the emotion expressed in the words does not match the emotion expressed in the results estimated by the first estimation model, inputs the audio data into a second estimation model to estimate the emotion corresponding to the audio data.

Images