Information processing device and information processing program

Abstract

To provide an information processing device and an information processing program capable of accurately estimating features exhibited by voice data in a case where multiple data are extracted from one voice data.SOLUTION: An information processing device includes: an acquisition unit that acquires one voice data of a user; an extraction unit that extracts a plurality of voice data extracted from the one voice data at each predetermined period by transferring a period at each predetermined unit time; an estimation unit that estimates each of feature quantities indicating the user's emotion from each of the plurality of voice data using an estimation model in which machine learning is executed to estimate the feature quantities indicating the user's emotion from the plurality of extracted voice data; and a determination unit that determines the user's emotion expressed by the one voice data using the feature quantities corresponding to each of the plurality of voice data.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing program for estimating emotions from voice data. [Background technology]

[0002] Patent Document 1 discloses a trauma screening device that estimates emotions from voice data by converting voice data into image data as preprocessing and performing machine learning using the image data as training data. The trauma screening device according to Patent Document 1 is characterized in that when generating multiple image data for each predetermined period from a single image data converted by preprocessing, the image data is amplified by shifting the image data by a predetermined time unit and extracting the multiple image data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-079446 Summary of the Invention [Problem to be solved by the invention]

[0004] When estimating features such as emotions from voice data using the technology of Patent Document 1, different features may be estimated for each of the extracted multiple data. Therefore, when multiple data are extracted from one voice data, it is not always possible to accurately estimate the features indicated by the voice data.

[0005] An object of the present invention is to provide an information processing device and an information processing program that can accurately estimate features indicated by audio data when multiple data are extracted from one audio data. [Means for solving the problem]

[0006] The information processing device according to claim 1 includes an acquisition unit that acquires one voice data of a user; an extraction unit that extracts a plurality of voice data extracted from the one voice data at predetermined intervals by shifting the intervals at predetermined unit times; an estimation unit that estimates a feature quantity indicating the user's emotion from each of the plurality of voice data using an estimation model that has been subjected to machine learning to estimate a feature quantity indicating the user's emotion from the extracted plurality of voice data; and a determination unit that determines the user's emotion expressed by the one voice data using the feature quantities corresponding to the plurality of voice data. The predetermined period is set to be longer than the longest period among periods showing an emotion different from the user's emotion set as a label of the one piece of voice data acquired in the past, and is set to be shorter than the shortest period among periods showing an emotion corresponding to the user's emotion set as a label of the one piece of voice data, and the determination unit performs a majority vote using feature amounts corresponding to each of the plurality of voice data, and determines the emotion shown most frequently as the user's emotion expressed by the one piece of voice data. .

[0007] The information processing device described in claim 1 acquires one voice data of a user, moves a predetermined period that is an extraction range at predetermined unit time intervals, extracts multiple voice data from the one voice data, estimates feature values ​​indicating emotion from each of the extracted multiple voice data using an estimation model that has been subjected to machine learning to estimate the user's emotion from the voice data, and determines the user's emotion indicated by the one voice using the estimated multiple feature values. As a result, when multiple data are extracted from the one voice data, it is possible to accurately estimate the features indicated by the voice data.

[0008] The information processing device of claim 2 is the information processing device of claim 1, wherein the predetermined period is set according to the feature indicating an emotion corresponding to the user's emotion set as a label of the one piece of voice data acquired in the past, and the feature indicating an emotion different from the user's emotion set as a label of the one piece of voice data.

[0009] According to the information processing device of claim 2, when a piece of audio data contains a feature that is different from the feature indicated by the entire piece of audio data, the influence of the feature that is different from the feature indicated by the audio data can be suppressed.

[0010] The information processing device of claim 3 is the information processing device of claim 1 or claim 2, wherein the extraction unit sets the unit time so that the number of pieces of audio data extracted from the one piece of audio data is a predetermined number, and extracts the multiple pieces of audio data.

[0011] According to the information processing device of claim 3, it is possible to estimate the user's emotions with high accuracy regardless of the length of the acquired voice data.

[0012] The information processing device of claim 4 is the information processing device of any one of claims 1 to 3, wherein the estimation unit uses, as the estimation models, an individual user estimation model that has learned one piece of voice data for each individual user among a plurality of users, and an overall user estimation model that has learned one piece of voice data related to all of the plurality of users, to estimate features indicating the emotions of the users, and the determination unit determines the user emotion represented by the one piece of voice data using the features estimated by the individual user estimation model and the overall user model, respectively.

[0013] According to the information processing device of claim 4, it is possible to estimate the user's emotion with higher accuracy compared to when estimation is performed using either the individual user estimation model or the overall user estimation model.

[0014] The information processing program according to claim 5, wherein the information processing program acquires one voice data of a user, A plurality of pieces of voice data extracted from the voice data for each predetermined period, extracting the plurality of audio data by transferring the period to the We performed machine learning to estimate features that indicate user emotions from speech data. using a model to estimate a feature quantity indicating an emotion of the user from each of the plurality of pieces of voice data; A majority vote is performed using feature amounts corresponding to each of the plurality of voice data, and the emotion expressed most frequently is determined as the emotion of the user represented by the one voice data. Let the computer perform the process The predetermined period is set to be greater than the longest period among periods showing an emotion different from the user's emotion set as a label of the one voice data acquired in the past, and is set to be smaller than the shortest period among periods showing an emotion corresponding to the user's emotion set as a label of the one voice data. .

[0015] A computer executing the information processing program according to claim 5 acquires one voice data of a user, moves a predetermined period as an extraction range at predetermined unit time intervals, extracts multiple voice data from the one voice data, estimates feature quantities indicating emotion from each of the extracted multiple voice data using an estimation model that has been subjected to machine learning to estimate the user's emotion from the voice data, and determines the user's emotion indicated by the one voice using the estimated multiple feature quantities. This makes it possible to accurately estimate the features indicated by the voice data when multiple data are extracted from the one voice data. [Effects of the Invention]

[0016] According to the present invention, when a plurality of data are extracted from one piece of speech data, the features indicated by the speech data can be estimated with high accuracy. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a diagram showing a schematic configuration of an information processing system according to an embodiment of the present invention; [Figure 2] FIG. 2 is a block diagram showing the hardware configuration of a center server according to the present embodiment. [Figure 3] FIG. 2 is a block diagram showing the functional configuration of a center server according to the present embodiment. [Figure 4] FIG. 2 is a data flow diagram illustrating the emotion estimation process according to the present embodiment. [Figure 5] 1 is a schematic diagram illustrating a window size and a frame shift according to an embodiment of the present invention. [Figure 6] 3 is a schematic diagram illustrating a feature included in the voice data of the present embodiment. FIG. [Figure 7] 10 is a flowchart showing the flow of emotion estimation processing executed in the center server of the present embodiment. [Figure 8] 10 is a flowchart showing the flow of a process for generating an estimation model executed in the center server of the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] An information processing system including an information processing device of the present invention will be described below. The information processing system is a system that estimates a user's emotions using voice data of the user acquired from a terminal used by the user.

[0019] (Overall composition) 1, an information processing system 10 according to an embodiment of the present invention includes a center server 20 as an information processing device and a terminal 30 operated by a user. The center server 20 and the terminal 30 are connected to each other via a network N.

[0020] Although FIG. 1 illustrates one terminal 30 for one center server 20, the number of center servers 20 and terminals 30 is not limited to this.

[0021] The center server 20 is a device that acquires user voice data from the terminal 30 and estimates the user's emotions indicated by the acquired voice data. Note that the information processing device according to the present embodiment has been described as being a center server. However, the information processing device is not limited to this. The information processing device may also be a personal computer such as a terminal, a mobile terminal, or a tablet.

[0022] The terminal 30 is an in-vehicle terminal installed in a vehicle that has the function of storing the voice uttered by the user and transmitting the voice as voice data to the center server 20, a mobile terminal owned by the user, a tablet terminal, etc.

[0023] (Center server) FIG. 2 is a block diagram showing an example of the hardware configuration of the center server 20 according to this embodiment.

[0024] 2, the center server 20 according to this embodiment includes a CPU (Central Processing Unit) 20A, a ROM (Read Only Memory) 20B, a RAM (Random Access Memory) 20C, a storage 20D, an input unit 20E, and a communication I / F 20F. The CPU 20A, the ROM 20B, the RAM 20C, the storage 20D, the input unit 20E, and the communication I / F 20F are connected to each other via an internal bus 20G so as to be able to communicate with each other.

[0025] The CPU 20A is a central processing unit that executes various programs and controls each part. That is, the CPU 20A reads programs from the ROM 20B and the storage 20D, and executes the programs using the RAM 20C as a working area.

[0026] The ROM 20B stores various programs and various data. In this embodiment, the ROM 20B stores an information processing program 100 for estimating emotions from voice data acquired from the terminal 30. When the information processing program 100 is executed, the center server 20 acquires voice data from the terminal 30 and executes various processes including a process for estimating emotions from the voice data. The RAM 20C temporarily stores programs or data as a working area.

[0027] The storage 20D is, for example, a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The storage 20D stores user voice data, trained models, various programs, and the like. The storage 20D according to this embodiment stores an estimation model 110 as a trained model, and a voice information database (hereinafter referred to as "voice information DB") 130 that stores voice data.

[0028] The input unit 20E is a pointing device and a keyboard that accepts input of characters and instructions to execute processing.

[0029] The communication I / F 20F is a communication module for communicating with the terminal 30. The communication module uses communication standards such as 5G, LTE, and Wi-Fi (registered trademark). The communication I / F 20F is connected to a network N. Note that the communication I / F 20F may perform wired communication.

[0030] The information processing program 100 as a program is a program for controlling the center server 20. When the information processing program 100 is executed, the center server 20 executes various processes including a process of acquiring voice data and a process of estimating the user's emotions from the voice data.

[0031] The estimation model 110 is a trained model generated by executing machine learning to estimate a user's emotion from voice data. The estimation model 110 estimates and outputs the user's emotion indicated by input voice data. Note that a decision tree model, a k-means method, an SVM (Support Vector Machine) model, etc. can be applied as the estimation model 110 according to this embodiment.

[0032] The voice information DB 120 stores voice data that has been acquired in the past.

[0033] As shown in Figure 3, in the center server 20 of this embodiment, the CPU 20A executes the information processing program 100 to function as an acquisition unit 200, an extraction unit 210, an estimation unit 220, a judgment unit 230, a memory unit 240, and a learning unit 250.

[0034] The acquisition unit 200 has a function of acquiring user voice data 300 transmitted from the terminal 30, as shown in FIG.

[0035] The extraction unit 210 has a function of extracting extracted data 310 from the acquired voice data 300. Specifically, the extraction unit 210 extracts a predetermined number of fragment data (hereinafter referred to as the "number of fragments") from the acquired voice data 300 as extracted data 310. As an example, as shown in FIG. 5, the extraction unit 210 extracts extracted data 310 corresponding to the number of fragments from the voice data 300 in accordance with a window size 400 and a frame shift 410. Here, the window size 400 is an example of a "predetermined period," and the frame shift 410 is an example of a "predetermined unit time."

[0036] The window size 400 is the period of data to be extracted from the audio data 300 as the extracted data 310. For example, if the window size 400 is set to 2 seconds, the extraction unit 210 extracts 2 seconds of extracted data 310 from the audio data 300.

[0037] The frame shift 410 is the amount by which the start and end positions of each piece of extracted data 310 are shifted when extracting multiple pieces of extracted data 310 from the audio data 300. For example, if the frame shift 410 is set to 0.1 seconds, the extraction unit 210 extracts each piece of extracted data 310 by shifting the start and end positions by 0.1 seconds. The amount of the frame shift 410 is set according to the size of the audio data 300, the number of fragments, and the size of the window size 400. For example, if the audio data 300 is 20 seconds long, the number of fragments is 100, and the window size 400 is 2 seconds, the extraction unit 210 sets the frame shift 410 to 0.18 seconds and extracts 100 pieces of extracted data 310 each for 2 seconds, shifted by 0.18 seconds.

[0038] The size of window size 400 is set according to the learned learning data. For example, even if "positive" is set as the label in voice data 300, emotions other than "positive" (e.g., emotions such as "negative" or "neutral") may be included in a portion of the data. The size of window size 400 is set during the learning phase so as to correspond to the period of data indicating emotions other than "positive" that is included in voice data labeled "positive."

[0039] As an example, as shown in FIG. 6, audio data 300 to which the label "positive" is set includes a period (hereinafter referred to as the "corresponding period") indicating "positive" (an emotion corresponding to the emotion set in the label) and a period (hereinafter referred to as the "different period") indicating an emotion other than "positive" (an emotion different from the emotion set in the label).

[0040] The window size 400 is set to be larger than the maximum of the different periods and smaller than the minimum of the corresponding periods. This reduces the number of extracted data 310 containing emotions different from the set label when extracting multiple pieces of extracted data 310 from the audio data 300, thereby reducing the influence of majority vote in the determining unit 230, which will be described later. Note that the maximum of the different periods according to this embodiment is smaller than the minimum of the corresponding periods.

[0041] The estimation unit 220 has a function of estimating the user's emotion indicated by the extracted data 310 using the estimation model 110. Here, the estimation model 110 according to this embodiment includes an individual model 110A trained on the voice data 300 of each individual user, and an overall model 110B trained on the voice data 300 of all users. Here, the individual model 110A is an example of an "individual user estimation model," and the overall model 110B is an example of an "overall user estimation model."

[0042] The estimation unit 220 estimates the user's emotion as an estimation result 320 for each piece of extracted data 310 using the individual model 110A and the overall model 110B.

[0043] The determination unit 230 has a function of determining the user's emotion indicated by the voice data 300 as a determination result 330 using the estimation result 320 estimated by the estimation unit 220, and outputting the determination result 330. Specifically, the determination unit 230 performs a majority vote using the estimation results 320 estimated for the multiple pieces of extracted data 310, and determines the emotion indicated most frequently as the emotion indicated by the voice data 300.

[0044] Here, the determination unit 230 integrates the estimation result 320 estimated by the individual model 110A for one piece of extracted data 310 with the estimation result 320 estimated by the overall model 110B, and performs determination on the resulting single estimation result 320. For example, the determination unit 230 weights the corresponding estimation results 320 among the multiple estimation results 320 estimated by the individual model 110A and the multiple estimation results 320 estimated by the overall model 110B, and integrates the corresponding estimation results 320 to perform determination. Note that in the present embodiment, a form in which each estimated estimation result 320 is weighted and integrated has been described. However, this is not limiting. The estimated estimation results 320 may be integrated by averaging.

[0045] The storage unit 240 has a function of storing the acquired voice data 300 in the voice information DB 120. Here, a label is set for the stored voice data, and the data is stored as learning data. The set label may be set by the user, or the determination result determined by the determination unit 230 may be set as the label. Furthermore, the voice data 300 may be stored in association with the user's characteristics.

[0046] The learning unit 250 has a function of executing machine learning using previously acquired voice data 300 as learning data, and generating an individual model 110A and an overall model 110B as the estimation model 110.

[0047] (Flow of Control) The flow of each process executed in the information processing system 10 of this embodiment will be described using the flowchart in Fig. 7. Each process in the center server 20 is executed by the CPU 20A of the center server 20 functioning as an acquisition unit 200, an extraction unit 210, an estimation unit 220, a determination unit 230, a storage unit 240, and a learning unit 250. The process of estimating a user's emotion shown in Fig. 7 is executed, for example, when voice data 300 is input and an instruction to estimate a user's emotion is input.

[0048] In step S100, the CPU 20A acquires the voice data 300 input from the terminal 30.

[0049] In step S101, CPU 20A extracts a plurality of extracted data 310 from acquired audio data 300.

[0050] In step S102, the CPU 20A estimates the user's emotion for each extracted piece of extracted data 310. Here, the CPU 20A inputs one piece of extracted data 310 to the individual model 110A and the overall model 110B, and obtains an estimation result 320 from each of the individual model 110A and the overall model 110B. The CPU 20A also selects, as the individual model 110A, an estimation model 110 corresponding to the user related to the input voice data 300, and estimates the emotion.

[0051] In step S103, the CPU 20A integrates the corresponding estimation results 320 from the multiple estimation results 320 estimated by the individual model 110A and the multiple estimation results 320 estimated by the overall model 110B, and outputs the estimation result 320 for each extracted data 310.

[0052] In step S104, CPU 20A uses the integrated plurality of estimation results 320 to perform a majority vote, determines the most prevalent emotion as the user's emotion in voice data 300, and outputs it.

[0053] In step S105, the CPU 20A determines whether or not to end the process of estimating the user's emotion. If the process of estimating the user's emotion is to be ended (step S105: YES), the CPU 20A ends the process of estimating the user's emotion. On the other hand, if the process of estimating the user's emotion is not to be ended (step S105: NO), the CPU 20A proceeds to step S100 and acquires the input voice data 300.

[0054] Next, a process for generating a trained model executed in the information processing system 10 of this embodiment will be described with reference to the flowchart of Fig. 8. The generation process shown in Fig. 8 is executed, for example, when an instruction to execute a process for generating a trained model is input.

[0055] In step S200, CPU 20A acquires previously acquired voice data 300 as learning data.

[0056] In step S201, the CPU 20A performs machine learning using the acquired learning data to generate an estimation model 110. Here, the CPU 20A generates, as the estimation model 110, an individual model 110A using the voice data 300 for each user, and an overall model 110B using the voice data 300 for all users.

[0057] In step S202, CPU 20A inputs voice data 300 to the generated estimation model 110, and evaluates estimation model 110 using the user's emotion output from estimation model 110.

[0058] In step S203, the CPU 20A determines whether or not to end the process of generating the estimation model 110. If the process of generating the estimation model 110 is to be ended (step S203: YES), the CPU 20A proceeds to step S204. On the other hand, if the process of generating the estimation model 110 is not to be ended (step S203: NO), the CPU 20A proceeds to step S200 and acquires training data.

[0059] In step S204, CPU 20A stores the generated estimation model 110.

[0060] (summary) The center server 20 as an information processing device of this embodiment acquires one voice data of a user, shifts the extraction range of the one voice data for a predetermined period at predetermined unit time intervals to extract multiple voice data, estimates features indicating emotions from each of the extracted multiple voice data using an estimation model that has performed machine learning to estimate the user's emotions from the voice data, and determines the user's emotions indicated by the one voice using the estimated multiple features.

[0061] As described above, according to this embodiment, when a plurality of data are extracted from one piece of speech data, the features indicated by the speech data can be estimated with high accuracy.

[0062] In the above embodiment, the window size 400 is set to be larger than the maximum period of the difference periods and smaller than the minimum period of the corresponding periods. However, the present invention is not limited to this. The window size 400 may be set to be larger than the maximum period of the difference periods or smaller than the minimum period of the corresponding periods.

[0063] Furthermore, the personal model 110A according to the above embodiment has been described as learning the voice data 300 for each user. However, this is not limiting. Voice data for each user feature may be learned. For example, the user features such as gender, age, height, and weight may be associated with the voice data 300 and stored, and the voice data 300 related to similar features may be used as training data to perform machine learning to generate the personal model 110A. Furthermore, when selecting the personal model 110A, the personal model 110A may be selected using the user features associated with the voice data 300. [remarks] In the above embodiment, various processes executed by the CPU 20A after reading software (programs) may be executed by various processors other than a CPU. Examples of such processors include programmable logic devices (PLDs) (such as field-programmable gate arrays (FPGAs)) whose circuit configuration can be changed after fabrication, and application-specific integrated circuits (ASICs) that are dedicated electrical circuits designed specifically to execute specific processes. Each of the above processes may be executed by one of these processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0064] In the above embodiment, each program is described as being pre-stored (installed) in a computer-readable non-transitory recording medium. For example, the information processing program 100 in the center server 20 is pre-stored in the ROM 20B. However, the present invention is not limited to this. Each program may be provided in a form recorded on a non-transitory recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0065] The processing flow described in the above embodiment is an example, and unnecessary steps may be deleted, new steps may be added, or the processing order may be changed within the scope of the gist of the invention. [Explanation of symbols]

[0066] 20 Center server (information processing device) 200 Acquisition Department 210 Extraction part 220 Estimation Department 230 Judgment section

Claims

1. an acquisition unit that acquires one voice data of a user; an extracting unit that extracts a plurality of pieces of audio data from the one piece of audio data for each predetermined period by shifting the predetermined period for each predetermined unit time; an estimation unit that estimates a feature quantity indicating a user's emotion from each of the plurality of extracted voice data using an estimation model that has been subjected to machine learning for estimating a feature quantity indicating a user's emotion from the plurality of voice data; a determination unit that determines a user's emotion expressed by the one piece of voice data by using feature amounts corresponding to the plurality of voice data, the predetermined period is set to be longer than a longest period among periods indicating an emotion different from a user's emotion set as a label of the one piece of voice data acquired in the past, and is set to be shorter than a shortest period among periods indicating an emotion corresponding to the user's emotion set as a label of the one piece of voice data; The determination unit performs majority voting using feature amounts corresponding to the plurality of voice data, and determines the most frequently expressed emotion as the user's emotion represented by the one voice data. Information processing device.

2. The predetermined period is set according to the feature amount indicating an emotion corresponding to the emotion of the user set as a label of the one piece of voice data acquired in the past and the feature amount indicating an emotion different from the emotion of the user set as a label of the one piece of voice data. The information processing device according to claim 1 .

3. The extraction unit sets the unit time so that the number of pieces of audio data to be extracted from the one piece of audio data is a predetermined number, and extracts the plurality of pieces of audio data. The information processing device according to claim 1 .

4. the estimation unit estimates, as the estimation models, feature quantities indicating emotions of the users using an individual user estimation model that has learned one piece of voice data for each individual user among a plurality of users, and an overall user estimation model that has learned one piece of voice data related to all of the plurality of users; The determination unit determines the user's emotion expressed by the one piece of voice data using feature amounts estimated by the individual user estimation model and the overall user estimation model. The information processing device according to claim 1 .

5. Acquire one voice data of the user; a plurality of pieces of voice data extracted from the one piece of voice data for each predetermined period, the plurality of pieces of voice data being extracted by shifting the predetermined period for each predetermined unit time; using an estimation model obtained by performing machine learning to estimate feature quantities indicating user emotions from the extracted plurality of voice data, to estimate feature quantities indicating user emotions from each of the plurality of voice data; A majority vote is performed using feature amounts corresponding to each of the plurality of voice data, and the emotion expressed most frequently is determined as the emotion of the user represented by the one voice data. Have the computer execute the process, The predetermined period is set to be greater than the longest period among periods showing an emotion different from the user's emotion set as a label of the one piece of voice data acquired in the past, and is set to be smaller than the shortest period among periods showing an emotion corresponding to the user's emotion set as a label of the one piece of voice data. Information processing program.

Images