Annotation request device, annotation request method, and annotation request program

Abstract

To provide an annotation request device, an annotation request method, and an annotation request program that can reduce a load on an annotator.SOLUTION: A speech emotion estimation section 22 estimates an emotion value of speech data using a learned model generated by machine learning, an annotation data estimation section 24 identifies the emotion value estimated by the speech emotion estimation section 22, and extracts values near positive and negative threshold values. And when the annotation data estimation section 24 has been able to extract data near the threshold values, an annotation request processing section 26 sends a total of three pieces of speech data corresponding to the values near the threshold values (upper and lower two values) held until just before and input speech to an annotator to request annotation.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an annotation request device, an annotation request method, and an annotation request program.

Background Art

[0002] Patent Document 1 proposes a learning data creation method for creating learning data for risk prediction, which includes positive data and negative data, by a computer.

[0003] Specifically, a plurality of still image data or moving image data are acquired as a plurality of event data in which an event that is an accident or an incident is reflected respectively, and a plurality of non-event data in which an event is not reflected respectively. A first data, which is still image data or moving image data included in one of the acquired plurality of event data and is still image data or moving image data before a predetermined time from the event, is presented, and one of the acquired plurality of non-event data is presented as second data. A determination result as to whether or not the first data and the second data are similar is received, and learning data is created by storing the event data and the non-event data in a storage device. At the time of storage, the event data is stored as positive data, and when the received determination result indicates that the first data and the second data are similar, the non-event data is stored as positive data, and when the received determination result indicates that the first data and the second data are not similar, the non-event data is stored as negative data.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The technology described in Patent Document 1 can be applied not only to distinguishing between event data and non-event data, but also, for example, to estimating the speaker's emotions. However, it requires presenting two sets of data to the annotator uniformly, which increases the annotator's workload, so there is room for improvement.

[0006] This invention has been made in consideration of the above facts, and aims to provide an annotation request device, an annotation request method, and an annotation request program that can reduce the load on the annotator. [Means for solving the problem]

[0007] An annotation request device according to the first embodiment includes an estimation unit that estimates the speaker's emotions, including positive and negative, from voice data, and an estimation unit that determines whether the emotion estimated by the estimation unit is a threshold for the positive or negative emotion. Corresponds to plus or minus 5% When located within a predetermined range, the speaker's voice data and the location within the predetermined range and within 5% above or below the threshold. The system includes a request unit that presents the annotator with two other estimated or annotated audio data sets that cross the threshold, each corresponding to the threshold, and requests annotation.

[0008] According to the first embodiment, the estimation unit estimates the speaker's emotions, including positive and negative emotions, from the speech data.

[0009] The request unit determines whether the emotion estimated by the estimation unit is positive or negative. Corresponds to plus or minus 5% When located within a predetermined range, the speaker's voice data and the location within a predetermined range And within 5% above and below the threshold The annotator is presented with two other estimated or annotated audio data points that cross the threshold, corresponding to each of the two thresholds, and is asked to perform the annotation. This reduces the load on the annotator compared to simply presenting two data points, one positive and one negative, to each annotator, as the annotator is only presented with audio data near the threshold.

[0012] The 2 An annotation request method relating to this embodiment involves a computer estimating the speaker's emotions, including positive and negative, from the audio data, and the estimated emotion being a threshold for either the positive or negative emotion. Corresponds to plus or minus 5% When located within a predetermined range, the speaker's voice data and the location within the predetermined range and within 5% above or below the threshold. The process involves presenting the annotator with two other estimated or annotated audio data sets that cross the threshold, corresponding to each of the thresholds, and requesting annotation.

[0013] According to the third embodiment, an annotation request method that can reduce the load on the annotator can be provided.

[0014] The 3 The annotation request program according to the embodiment estimates the speaker's emotions, including positive and negative, from the audio data, and the estimated emotion is determined by a threshold for either the positive or negative emotion. Corresponds to plus or minus 5% When located within a predetermined range, the speaker's voice data and the location within the predetermined range and within 5% above or below the threshold. The process involves presenting the annotator with two other estimated or annotated audio data sets that cross the threshold, corresponding to each of the thresholds, and requesting annotation from them.

[0015] According to the fourth embodiment, an annotation request program can be provided that can reduce the load on the annotator. [Effects of the Invention]

[0016] As described above, the present invention provides an annotation request device, an annotation request method, and an annotation request program that can reduce the load on the annotator. [Brief explanation of the drawing]

[0017] [Figure 1]This is a diagram showing the schematic configuration of the information processing system according to this embodiment. [Figure 2] This is a block diagram showing the schematic configuration of the server of the information processing system according to this embodiment. [Figure 3] This is a flowchart showing an example of the processing flow performed in the annotation request section of the server of the information processing system according to this embodiment. [Figure 4] This is a flowchart showing an example of the processing flow performed in the emotion value estimation model learning section of the server of the information processing system according to this embodiment.

Mode for Carrying Out the Invention

[0018] Hereinafter, an example of an embodiment of the present invention will be described in detail with reference to the drawings. An information processing system including an annotation request device will be described. FIG. 1 is a diagram showing the schematic configuration of the information processing system according to this embodiment.

[0019] The information processing system 10 according to this embodiment is a system that requests an annotator 62 to annotate voice data used for learning a machine learning model that estimates the emotion of the speaker 60 from the voice of a conversation.

[0020] The information processing system 10 according to this embodiment includes a server 11 connected to a network 50 such as a local area network (LAN) or the Internet, and the server 11 functions as an annotation request device as an example.

[0021] The server 11 receives voice data obtained by recording the voice of the speaker 60 via the network 50, and performs a process of requesting an annotation from the annotator 62 via the network 50. Further, the server 11 receives an annotation result from the annotator 62 via the network 50 and performs a process of learning a machine learning model for estimating the emotion of the speaker 60.

[0022] As shown in Figure 2, the server 11 has a typical computer configuration including a CPU (Central Processing Unit) 11A, ROM (Read Only Memory) 11B, RAM (Random Access Memory) 11C, storage 11D, interface (I / F) 11E, and bus 11F. The CPU 11A loads programs such as annotation request programs stored in ROM 11B into RAM 11C and executes them, thereby functioning as an annotation request unit 12, a sentiment value estimation model learning unit 30, and a database (DB) 40. Figure 2 is a block diagram showing the schematic configuration of the server 11 of the information processing system 10 according to this embodiment.

[0023] As shown in Figure 1, the annotation request unit 12 has the functions of an input unit 14, a speech splitting unit 16, a speech content analysis unit 18, a speaker estimation unit 20, a speech emotion value estimation unit 22 as an example of an estimation unit, an annotation data estimation unit 24, and an annotation request processing unit 26 as an example of a request unit.

[0024] The input unit 14 receives audio-only audio information or video information including audio, acquired from a cloud-based or on-premise system, as audio data. For example, the speaker 60 sends recorded audio data from an information processing terminal such as a personal computer to the server 11, which is then received by the input unit 14.

[0025] The audio splitting unit 16, when the audio data acquired by the input unit 14 is video information that includes audio, performs a process to split the audio information from the video information and extract only the audio information as audio data.

[0026] The speech content analysis unit 18 performs transcription based on the audio data extracted by the audio splitting unit 16. If the input unit 14 receives audio information consisting only of audio as audio data, the speech content analysis unit 18 performs transcription based on the audio data received by the input unit 14. For example, transcription is performed from audio data using a well-known technique for transcribing from audio. A well-known technique involves generating a learning model that estimates text from audio using machine learning, and then performing transcription by inputting audio data into the learning model.

[0027] The speaker estimation unit 20 performs a process to estimate the speaker 60 by comparing it with the past speaker database 42. For example, it uses a method to identify the speaker 60 of the voice data, such as a personal identification technology like voiceprint authentication.

[0028] The speech emotion value estimation unit 22 performs processing to estimate the emotion value, intonation, and speaking speed of the speech data. For example, a learning model that estimates the emotion value, intonation, and speaking speed from the speech data is generated by machine learning, and the emotion value, intonation, and speaking speed are estimated by inputting the speech data into the learning model. In this embodiment, the speech emotion value estimation unit 22 estimates the emotion value of the speaker 60 using the learning model learned by machine learning. As an example of an emotion value, a value from 0 to 100 is used.

[0029] The annotation data estimation unit 24 checks the emotion values ​​estimated by the speech emotion value estimation unit 22 and extracts values ​​near the positive or negative threshold. For example, the emotion values ​​are set to a range of 0 to 100, with 0 to 30 being negative, 70 to 100 being positive, and 31 to 69 being normal. As an example of values ​​near the threshold, values ​​above and below a predetermined range of the threshold (e.g., 30 or 70) (e.g., ±5%) are applied, and values ​​near the threshold are extracted from the emotion values ​​estimated by the speech emotion value estimation unit 22.

[0030] The annotation request processing unit 26, when it has extracted data near the threshold by the annotation data estimation unit 24, that is, when the data falls within a predetermined range of the positive or negative threshold, sends a total of three pieces of audio data to the annotator 62: the audio data corresponding to the previously held values ​​near the threshold (two values, upper and lower) and the input audio, and requests annotation. In this embodiment, three pieces of data are sent to the annotator 62: the audio data corresponding to the previously held values ​​near the threshold (two values, upper and lower) and the input audio. However, this is not the only way, and the audio data near the threshold may be only one of the upper or lower values ​​within the predetermined range of the threshold. Furthermore, the values ​​near the threshold may be audio data for which the sentiment value has already been estimated by the learning model, or annotated audio data. In addition, the data is sent to the annotator 62, for example, by sending the data to an information processing terminal such as a personal computer operated by the annotator 62.

[0031] Furthermore, the emotion value estimation model learning unit 30 also has the functions of an annotation data input unit 32 and a model learning unit 34.

[0032] The annotation data input unit 32 receives the annotation results for the audio data that it has requested from the annotation request processing unit 26 to the annotator 62.

[0033] The model learning unit 34 uses the annotation results received by the annotation data input unit 32 to train a learning model for estimating sentiment values. For example, it uses the annotation results received by the annotation data input unit 32 to update thresholds through ranking learning or the like.

[0034] On the other hand, database 40 includes speaker DB 42, sentiment value DB 44, annotation data management DB 46, and learning model management DB 48, among others.

[0035] Speaker DB42 creates a speaker database from past audio data and is used when estimating the speaker 60.

[0036] The emotion value DB44 manages the emotion value of each utterance as a database. For example, it manages data including the emotion value estimated by the utterance emotion value estimation unit 22, as well as intonation, speaking speed, etc.

[0037] The annotation data management DB46 manages past audio data by including the annotation results within the audio data itself.

[0038] The learning model management DB48 manages learning models that estimate the emotional value of speech using speech data as input. For example, it manages learning models trained by the model learning unit 34.

[0039] Next, we will describe the specific processing performed by the server 11 of the information processing system 10 according to this embodiment, which is configured as described above.

[0040] First, the processing performed by the annotation request unit 12 of the server 11 of the information processing system 10 according to this embodiment will be described. Figure 3 is a flowchart showing an example of the processing flow performed by the annotation request unit 12 of the server 11 of the information processing system 10 according to this embodiment. Note that the processing in Figure 3 starts, for example, when an annotation request for audio data is issued.

[0041] In step 100, the CPU 11A performs audio input and proceeds to step 102. That is, the input unit 14 receives audio information or video information containing audio acquired from a cloud-based or on-premise system as audio data.

[0042] In step 102, the CPU 11A determines whether or not the data is video data. This determination determines whether or not the audio data received by the input unit 14 is video information that includes audio. If the determination is affirmative, the process proceeds to step 104; if it is only audio information, the determination is denied and the process proceeds to step 106.

[0043] In step 104, the CPU 11A splits and extracts the audio and proceeds to step 106. Specifically, the audio splitting unit 16, if the audio data acquired by the input unit 14 is video information that includes audio, splits the audio information and video information and extracts only the audio information as audio data.

[0044] In step 106, the CPU 11A performs speech analysis on the speech data and proceeds to step 108. Specifically, the speech content analysis unit 18 performs transcription based on the speech data extracted by the speech splitting unit 16, or the speech-only speech data received by the input unit 14.

[0045] In step 108, the CPU 11A identifies speaker 60 and proceeds to step 110. That is, the speaker estimation unit 20 performs a process to estimate speaker 60 by comparing it with the past speaker DB 42.

[0046] In step 110, the CPU 11A estimates the speech emotion value and proceeds to step 112. That is, the speech emotion value estimation unit 22 uses a learning model generated by machine learning to estimate the emotion value of the speech data.

[0047] In step 112, the CPU 11A determines whether the estimated emotion value is normal or not. In this embodiment, the annotation data estimation unit 24 checks the emotion value estimated by the speech emotion value estimation unit 22 and determines, for example, whether the estimated emotion value is between 31 and 69. If the determination is affirmative, the series of processes ends there; if it is negative, the process proceeds to step 114.

[0048] In step 114, the CPU 11A determines whether the emotion value is positive or not. In this embodiment, the annotation data estimation unit 24 checks the emotion value estimated by the speech emotion value estimation unit 22 and determines, for example, whether the estimated emotion value is between 70 and 100. If the determination is negative, i.e., if the emotion value is between 0 and 30, it is negative and the process proceeds to step 116; if it is positive, the process proceeds to step 118.

[0049] In step 116, the CPU 11A determines whether the value is near the negative threshold. In this embodiment, the annotation data estimation unit 24 checks the emotion value estimated by the speech emotion value estimation unit 22 and determines, for example, whether the value is near the negative threshold and is above or below a predetermined range of the negative threshold. If the determination is negative, the series of processes ends there; if it is positive, the process proceeds to step 120.

[0050] On the other hand, in step 118, the CPU 11A determines whether the value is near the positive threshold. In this embodiment, the annotation data estimation unit 24 checks the emotion value estimated by the speech emotion value estimation unit 22 and determines, for example, whether the value is near the positive threshold and is above or below a predetermined range of the positive threshold. If the determination is negative, the series of processes ends there; if it is positive, the process proceeds to step 120.

[0051] In step 120, the CPU 11A completes the series of processes by submitting an annotation request. Specifically, if the annotation data estimation unit 24 has been able to extract data near the threshold, the annotation request processing unit 26 submits a request for annotation to the annotator 62 by sending a total of three pieces of data—the threshold values ​​(two values, upper and lower) that were held up until the previous step, and the input audio.

[0052] Thus, in the server 11 of the information processing system 10 according to this embodiment, audio data near a threshold is requested to be annotated by the annotator 62, which reduces the load on the annotator 62 compared to when two pieces of data, positive and negative, are presented uniformly.

[0053] Furthermore, by sending audio data corresponding to values ​​above and below a predetermined threshold range, along with the input audio, to the annotator 62, it becomes possible to select what the user considers more positive or negative. This allows the degree of positivity or negativity to be calculated using ranking learning, etc., and effective data for the model (hard examples, threshold boundaries, etc.) can be presented to the user.

[0054] Furthermore, when requesting annotation, audio data corresponding to values ​​above and below the threshold is sent as audio data near the threshold. This allows the annotator 62 to be presented with the upper and lower limits of the predetermined range of the threshold, enabling more accurate annotation than sending other audio data that does not cross the threshold.

[0055] Next, the processing performed in the emotion value estimation model learning unit 30 of the server 11 of the information processing system 10 according to this embodiment will be described. Figure 4 is a flowchart showing an example of the processing flow performed in the emotion value estimation model learning unit 30 of the server 11 of the information processing system 10 according to this embodiment. Note that the processing in Figure 4 starts, for example, when annotation results are received from the annotator 62.

[0056] In step 200, the CPU 11A inputs the annotated data and proceeds to step 202. That is, the annotation data input unit 32 receives the annotation results for the audio data that it requested from the annotation request processing unit 26 to the annotator 62.

[0057] In step 202, the CPU 11A completes the series of processes by calculating a positive or negative threshold. For example, the model learning unit 34 updates the threshold using ranking learning or the like, using the annotation results received by the annotation data input unit 32. That is, it recalculates the threshold for estimating sentiment values ​​by retraining the learning model that estimates sentiment values. This improves the accuracy of sentiment value estimation of the retrained learning model.

[0058] In the above embodiment, the annotation request unit 12, the sentiment value estimation model learning unit 30, and the database 40 were described as functions of the same server 11, but this is not limited to this. For example, they may each be functions of a different server, or any one of the functions may be functions of a different server.

[0059] Furthermore, although the processing performed by the server 11 in each of the above embodiments has been described as software processing performed by executing a program, it is not limited to this. For example, it may be processing performed by hardware such as a GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), and FPGA (Field-Programmable Gate Array). Alternatively, it may be processing that combines both software and hardware. In the case of software processing, the program may be stored on various storage media and distributed.

[0060] Furthermore, the present invention is not limited to the above, and it is of course possible to implement it in various modified forms without departing from its spirit. [Explanation of Symbols]

[0061] 10. Information Processing Systems 11. Server (Annotation Request Device) 12 Annotation Request Department 14 Input section 22. Speech emotion value estimation unit (estimation unit) 24 Annotation Data Estimation Unit 26 Annotation Request Processing Unit (Request Department) 60 speakers 62 Annotators

Claims

1. An estimation unit that estimates the speaker's emotions, including positive and negative ones, from the audio data, If the emotion estimated by the estimation unit falls within a predetermined range corresponding to 5% above or below the positive or negative threshold, the request unit presents the speaker's voice data and two other estimated or annotated voice data that fall within the predetermined range and correspond to 5% above or below the threshold, respectively, to the annotator and requests annotation. An annotation request device equipped with the following features.

2. Computers The speaker's emotions, including both positive and negative, are estimated from the audio data. An annotation request method that, when the estimated emotion falls within a predetermined range corresponding to 5% above or below the positive or negative threshold, presents the speaker's voice data and two other estimated or annotated voice data that fall within the predetermined range and straddle the threshold, each corresponding to 5% above or below the threshold, to an annotator and requests annotation.

3. On the computer, The speaker's emotions, including both positive and negative, are estimated from the audio data. An annotation request program for causing an annotator to perform the process of requesting annotation when the estimated emotion falls within a predetermined range corresponding to 5% above or below the positive or negative threshold, by presenting the speaker's voice data and two other estimated or annotated voice data that fall within the predetermined range and straddle the threshold, each corresponding to 5% above or below the threshold.

Images