Information processing device, program, and information processing method
The information processing device evaluates speech recognition accuracy by controlling audio output and calculating recognition accuracy across multiple states, enhancing performance in varied usage scenarios.
Patent Information
- Application Number
- JP2021007078
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-01-20
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-01-20
AI Technical Summary
Existing speech recognition systems fail to evaluate accuracy based on varying states of speech output, such as location and speaking style, leading to inconsistent performance in different usage scenarios.
An information processing device and method that controls audio output based on conditions, recognizes the output, and calculates recognition accuracy, allowing evaluation across multiple speech states.
Enables accurate evaluation of speech recognition accuracy in diverse environments by adjusting output conditions, improving performance in various usage scenarios.
Smart Images

Figure 0007586718000001 
Figure 0007586718000002 
Figure 0007586718000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing device, a program, and an information processing method. [Background technology]
[0002] Conventionally, there are technologies for evaluating speech recognition accuracy. For example, a speech recognition accuracy estimation device disclosed in the following Patent Document 1 performs speech recognition on an input speech, and estimates the speech recognition accuracy of each recognized word based on a word alignment network that indicates the probability of the word being correct, as an evaluation of the speech recognition result. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2018-25717 A Summary of the Invention [Problem to be solved by the invention]
[0004] Meanwhile, there are various possible scenarios in which a voice recognition device is used (hereinafter also referred to as "usage scenarios"), such as creating minutes at a meeting and controlling a device at home. It is considered that the state of the voice output to be recognized differs for each usage scenario, such as the position of the speaker and how the speaker speaks. However, the above-mentioned patent document has a problem in that it is not possible to evaluate the voice recognition accuracy based on multiple states of the voice output.
[0005] In view of the above, an object of the present invention is to provide an information processing device, a program, and an information processing method that are capable of evaluating speech recognition accuracy in consideration of a plurality of states of speech output. [Means for solving the problem]
[0006] An information processing device according to one embodiment of the present invention includes an audio output control unit that controls the output of audio from an audio output device based on audio output conditions, an acquisition unit that acquires recognition result text information indicating the recognition result of the audio from a audio recognition device that recognizes the audio output from the audio output device, a calculation unit that refers to a text memory unit that stores test text information indicating the content of the test audio contained in the audio, and calculates the recognition accuracy of the recognition result based on the test text information and the recognition result text information, and an output unit that outputs the calculated recognition accuracy in association with the conditions, and the audio output control unit changes the conditions based on the recognition accuracy calculated by the calculation unit, and controls the output of audio from the audio output device based on the changed conditions.
[0007] A program according to one aspect of the present invention causes a computer to realize an audio output control function that controls the output of audio from an audio output device based on audio output conditions, an acquisition function that acquires recognition result text information indicating the recognition result of the audio from a audio recognition device that recognizes the audio output from the audio output device, a calculation function that refers to a text memory unit that stores test text information indicating the content of the test audio contained in the audio, and calculates the recognition accuracy of the recognition result based on the test text information and the recognition result text information, and an output function that outputs the calculated recognition accuracy in association with the conditions, and the audio output control function changes the conditions based on the recognition accuracy calculated by the calculation function, and controls the output of audio from the audio output device based on the changed conditions.
[0008] An information processing method according to one aspect of the present invention includes a computer that controls the output of voice from a voice output device based on voice output conditions, obtains recognition result text information indicating the recognition result of the voice from a voice recognition device that recognizes the voice output from the voice output device, refers to a text memory unit that stores test text information indicating the content of the test voice contained in the voice, calculates the recognition accuracy of the recognition result based on the test text information and the recognition result text information, outputs the calculated recognition accuracy in association with conditions, changes the conditions based on the calculated recognition accuracy, and controls the output of voice from the voice output device based on the changed conditions. Effect of the Invention
[0009] According to the present invention, it is possible to provide an information processing device, a program, and an information processing method that are capable of evaluating speech recognition accuracy in consideration of a plurality of states of speech output. [Brief description of the drawings]
[0010] [Figure 1] FIG. 1 is a diagram for explaining an example of a system configuration of an evaluation system according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a diagram for explaining an overview of an evaluation system according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram illustrating an example of a functional configuration of a control device according to the present embodiment. [Figure 4] 11 is a table showing an example of output conditions including a plurality of parameters of the evaluation system according to the present embodiment. [Diagram 5] FIG. 2 is a diagram showing an example of a screen of the evaluation system according to the embodiment. [Figure 6] FIG. 2 is a diagram showing an example of a screen of the evaluation system according to the embodiment. [Figure 7] 5 is a diagram illustrating an example of a relationship between the recognition accuracy of the evaluation system according to the embodiment and the volume of a sound output device. FIG. [Figure 8] 11 is a diagram illustrating an example of a relationship between the recognition accuracy of the evaluation system according to the embodiment and the distance from the audio output device. FIG. [Figure 9] FIG. 4 is a diagram illustrating an example of the operation of the control device according to the embodiment. [Figure 10] FIG. 2 is a diagram illustrating an example of a hardware configuration of a control device according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] A preferred embodiment of the present invention (hereinafter, referred to as "the present embodiment") will be described with reference to the accompanying drawings. In each drawing, the same reference numerals denote the same or similar configurations.
[0012] In this embodiment, a "unit", "means", "apparatus", or "system" does not simply mean a physical means, but also includes cases where the functions of the "unit", "means", "apparatus", or "system" are realized by software. Furthermore, the functions of one "unit", "means", "apparatus", or "system" may be realized by two or more physical means or devices. Furthermore, the functions of two or more "units", "means", "apparatus", or "system" may be realized by one physical means or device.
[0013] <1. System configuration> With reference to FIG. 1, an example of the system configuration of an evaluation system 1 according to this embodiment will be described. The evaluation system 1 is a system for users such as developers and customers to evaluate the speech recognition accuracy of a speech recognition device. For example, in the development of a speech recognition device, the evaluation system 1 can be used to evaluate the accuracy of a current product and a next-generation product of the same model to check whether the recognition accuracy has been improved. The evaluation system 1 can also be used, for example, to evaluate and compare the recognition accuracy of one's own speech recognition device with that of another company's speech recognition device.
[0014] 1, the evaluation system 1 includes a control device 100, a speech recognition device 200 to be evaluated, and speech output devices 300a to 300c that output speech. The evaluation system 1 may be connected to a speech recognition system 500 via a network N, for example. The speech output devices 300a to 300c may be collectively referred to as "speech output devices 300" unless there is a particular need to distinguish between them.
[0015] The network N is composed of a wireless network or a wired network. Examples of the network include a mobile phone network, a PHS (Personal Handy-phone System) network, a wireless LAN (Local Area Network), 3G (3rd Generation), LTE (Long Term Evolution), 4G (4th Generation), 5G (5th Generation), WiMax (registered trademark), infrared communication, Bluetooth (registered trademark), a wired LAN, a telephone line, a power line network, and a network conforming to IEEE1394, etc.
[0016] The control device 100 is an information processing device that controls the output of sound from the sound output device 300. The control device 100 may be connected to the sound output device 300, for example, by wire (for example, USB, HDMI (registered trademark), plug jack, etc.) or wirelessly (for example, Wifi, Bluetooth, etc.). The control device 100 acquires recognition result text information from the voice recognition device 200. For the sake of simplicity, an example will be described in which the control device 100 is one terminal device, but the control device 100 is not limited to this. The control device 100 may be configured, for example, by distributing the functions it has among a plurality of terminal devices.
[0017] The "recognition result text information" is information that indicates the speech recognition result output from the speech output device 300 in the form of text (character string).
[0018] The voice recognition device 200 is a device to be evaluated by the evaluation system 1. The voice recognition device 200 is, for example, an information processing device capable of communicating with the control device 100 and the voice recognition system 500. The voice recognition device 200 acquires the voice of a speaker, converts the acquired voice into text by voice recognition, and records the text.
[0019] The voice recognition device 200 may be, for example, a so-called smart speaker that responds to an acquired voice by dialogue or the like. As another example, the voice recognition device 200 may be a general-purpose tablet terminal or a smartphone. For example, the voice recognition device 200 may use a general-purpose tablet terminal as the voice recognition device 200 by installing a dedicated program in the general-purpose tablet terminal and executing the program.
[0020] The audio output device 300 is a device including an amplifier circuit and a speaker, and is a device that outputs audio. The audio output device 300 is controlled by the control device 100 to output audio. The audio output device 300 may be, for example, a speaker device, or may be an information processing device such as a smartphone or a laptop terminal that can communicate with the control device 100. There may be one audio output device 300, or there may be multiple audio output devices 300. The audio data of the audio (hereinafter, simply referred to as "audio data") that the control device 100 causes the audio output device 300 to output may be linked from the control device 100 during control, or may be stored in a storage unit (not shown) of the audio output device 300 or an external storage device.
[0021] The voice that the control device 100 causes the voice output device 300 to output includes a test voice for evaluating the recognition accuracy. In other words, the evaluation system 1 evaluates to what extent the voice recognition device 200 has recognized the contents of the test voice. The output voice may include, for example, noise in addition to the test voice. The test voice may include a first voice and a second voice. The second voice is a voice that is different from the first voice in terms of length, volume, frequency, etc.
[0022] The voice recognition system 500 is a system capable of communicating with the voice recognition device 200. The voice recognition system 500 receives voice data acquired from the voice recognition device 200, and recognizes the voice based on the received data.
[0023] <2. System Overview> An overview of the evaluation system 1 will be described with reference to Fig. 2. In this example, a plurality of voice output devices 300 are placed around a voice recognition device 200 to be evaluated, and a test is performed to evaluate the recognition accuracy of voice output from the plurality of voice output devices 300. In addition, when performing a test in the evaluation system 1, the voice recognition device 200 and the voice output device 300 may be placed in a room for evaluation (a measurement room in this example) so that external voice is prevented from entering the voice recognition device 200 as much as possible.
[0024] (1) As shown in FIG. 2, first, the control device 100 instructs the audio output device 300 to output audio based on audio output conditions (hereinafter also referred to as "output conditions").
[0025] (2) The audio output device 300 outputs audio based on this instruction.
[0026] The "audio output condition (output condition)" is, for example, a condition on how to output audio from the audio output device 300. The audio output condition may include, for example, a plurality of parameters related to audio output (hereinafter, simply referred to as "multiple parameters" or "parameters"). The parameters will be described in detail later.
[0027] (3) The voice recognition device 200 acquires the voice output from the voice output device 300 and recognizes the acquired voice. This voice recognition process may be executed by the function of the voice recognition device 200 itself, or may be partially or entirely delegated to the voice recognition system 500. The voice recognition device 200 records the result of the voice recognition as recognition result text information.
[0028] (4) The control device 100 acquires the recorded recognition result text information via the API of the voice recognition device 200.
[0029] (5) The control device 100 calculates the recognition accuracy of the recognition result based on the test text information and the recognition result text information. Here, the "test text information" is information that indicates, in text form, the content of the test voice included in the voice to be output by the voice output device 300.
[0030] (6) The control device 100 associates the recognition accuracy calculated in (5) above with the output condition applied in (1) above and outputs the result to a screen, a file, or the like.
[0031] (7) The control device 100 feeds back the recognition accuracy calculated in (5) above, and instructs the audio output device 300 to output audio again. Specifically, the control device 100 changes the output conditions based on this recognition accuracy, and instructs the audio output device 300 to output audio based on the changed output conditions.
[0032] In the evaluation system 1, the above steps (1) to (7) may be repeated a predetermined number of times. Specifically, the voice output by the voice output device 300 and the voice recognition by the voice recognition device 200 for this voice output may be performed multiple times. Then, when the steps are performed multiple times, the control device 100 evaluates the recognition accuracy of the voice recognition device 200 while changing the output conditions each time.
[0033] According to the above configuration, the control device 100 can create multiple voice output states by changing the output conditions when evaluating the voice recognition accuracy. Therefore, according to the above configuration, the control device 100 can allow the user to evaluate the voice recognition accuracy based on multiple voice output states. Therefore, the control device 100 can allow the user to evaluate the voice recognition accuracy in environments assuming various usage scenarios of the voice recognition device 200 by creating various voice output states.
[0034] <3. Functional configuration> The functional configuration of the control device 100 according to this embodiment will be described with reference to Fig. 3. As shown in Fig. 3, the control device 100 includes a control unit 110, a storage unit 130, and a communication unit 140.
[0035] The control unit 110 includes an audio output control unit 111 , an acquisition unit 112 , a calculation unit 113 , an output unit 114 , and a generation unit 115 .
[0036] <Audio output control section> The audio output control unit 111 controls the output of audio from the audio output device 300 based on the output condition. For example, when the output condition is "output audio data A", the audio output control unit 111 may refer to the storage unit 130 that stores the audio data A and transmit the audio data A to the audio output device 300 via the communication unit 140. For example, when the output condition is "output audio data with a volume of XX [db] and an audio length of XX [s] twice at yy / mm / dd h1h1:m1m1 and h1h1:m2m2 at a normal playback speed", the audio output control unit 111 may acquire audio data that matches the output condition from the storage unit 130, connect to the audio output device 300 so as to be able to communicate with the audio output device 300, and transmit the acquired audio data and an instruction to output audio (including parameters included in the output condition (in this example, the timing of output, the number of times of output, and the playback speed)) to the audio output device 300.
[0037] The output conditions may be stored, for example, as condition information in the storage unit 130. In addition, the output conditions may be stored as condition information by combining a plurality of output conditions as a test scenario.
[0038] The voice data may include, for example, test voice data of a test voice and noise data that generates noise. The test voice data and the noise data may be, for example, different individual data, or may be mixed. The test voice data may also include first test voice data of a first test voice and second test voice data of a second test voice having a length different from that of the first test voice.
[0039] The audio output control unit 111 changes the output conditions based on the recognition accuracy calculated by the calculation unit 113. Then, the audio output control unit 111 controls the output of audio from the audio output device 300 based on the changed output conditions.
[0040] According to the above configuration, the voice output control unit 111 can create multiple voice output states by changing the output conditions when evaluating the voice recognition accuracy. Therefore, according to the above configuration, the voice output control unit 111 can have the user evaluate the voice recognition accuracy based on multiple voice output states. Therefore, the voice output control unit 111 can have the user evaluate the voice recognition accuracy in environments assuming various usage scenarios of the voice recognition device 200 by creating various voice output states.
[0041] For example, when the audio output by the audio output device 300 and the voice recognition by the voice recognition device 200 of the audio output are executed multiple times, the audio output control unit 111 may control the audio output by changing one or more parameters used among the multiple parameters for each of the multiple times.
[0042] According to the above configuration, the voice output control unit 111 can vary the parameters to be used among the multiple parameters related to voice output, thereby controlling the variation of where to make the difference for each output condition. Therefore, according to the above configuration, the voice output control unit 111 can allow the user to evaluate the voice recognition accuracy in a varied environment.
[0043] The multiple parameters may include at least two of the following: information for identifying the voice data of the voice to be output by the voice output device 300; a voice source localization position (including the distance and direction from each of the voice output devices 300 with respect to the voice recognition device 200); a voice speaking speed (including the speed at which the speaker reads and speaks the contents of the test voice, as well as the playback speed of the voice); a voice volume; a voice length; information about the speaker of the voice (for example, information for identifying gender, age, and whether the voice is a human voice or a synthetic voice); an output timing; or an output count). The "information for identifying the voice data" may be, for example, a file name of a file that stores the voice data, or information for accessing the file (such as a URL). The multiple parameters may be combined for each output condition, as shown in FIG. 4.
[0044] The parameters may include, for example, device selection information, where "device selection information" refers to information for selecting a device to be turned on among the multiple audio output devices 300 for each of the multiple times.
[0045] The parameters may include, for example, device designation information, which is information that designates whether the audio output device 300 is to output a test voice or noise for each of the multiple times.
[0046] The parameters may include, for example, the length of the audio to be output by the audio output device 300 .
[0047] For example, when the voice data is data obtained by mixing test voice data and noise data, the parameters may be parameters related to this mixture (hereinafter, also referred to as "mixture parameters").
[0048] The parameter may be, for example, the difference between the volume of the test sound and the volume of noise, or the parameter may be, for example, the difference between the volume of the first test sound and the volume of the second test sound.
[0049] The audio output control unit 111 may control the audio output by, for example, changing the value of each parameter. The audio output control unit 111 may control the audio output by, for example, changing the value of each parameter for each preset value (for example, increasing the volume by 5 [db] at a time). The audio output control unit 111 may change the value of each parameter for each threshold value of the threshold information or the reference value of the reference value information.
[0050] For example, the audio output control unit 111 may control the audio output by switching on / off the audio output of each of the multiple audio output devices 300 each time, further based on the device selection information.
[0051] According to the above configuration, the audio output control unit 111 can switch on / off the audio output of each of the multiple audio output devices 300 to provide variation in sound source localization and enable evaluation of the recognition accuracy.
[0052] The audio output control unit 111 may control the audio output by switching the audio to be output from each of the multiple audio output devices 300 between a test audio and noise each time, further based on the device designation information, for example.
[0053] For example, depending on the usage scene, such as outdoors or in a commercial facility, noise such as ambient noise may affect the recognition accuracy of the voice recognition device 200. According to the above configuration, the voice output control unit 111 can evaluate the recognition accuracy taking into account the influence of such noise. Therefore, according to the above configuration, the voice output control unit 111 can perform evaluation assuming various usage scenes including noisy scenes.
[0054] The audio output control section 111 may control the audio output by, for example, switching the audio to be output from each of the audio output devices 300 between the first test audio and the second test audio for each of a plurality of times.
[0055] For example, the speech recognition device 200 may be used in a situation where multiple voices are generated from different sound sources, such as a conversation in a conference room, where multiple speakers are speaking from their respective seats. According to the above configuration, the speech output control unit 111 can control the output so as to be output under output conditions suited to such a usage situation.
[0056] <Acquisition part> The acquisition unit 112 acquires recognition result text information indicating the recognition result of the voice from the voice recognition device 200 that recognizes the voice output from the voice output device 300. The acquisition by the acquisition unit 112 may be in any manner, and may, for example, receive a text file of the recognition result text information transmitted from the voice output device 300. As another example, the acquisition unit 112 may acquire the recognition result text information by remotely accessing the voice recognition device 200 to acquire the text file of the recognition result text information, or by instructing an API implemented by the voice recognition device 200 to refer to the recognition result text information and acquiring the recognition result text information as a result.
[0057] <Calculation section> The calculation unit 113 refers to the text storage unit 131 that stores the test text information, and calculates the recognition accuracy of the recognition result based on the test text information and the recognition result text information acquired by the acquisition unit 112. The calculation unit 113 stores the calculated recognition accuracy in the storage unit 130 as accuracy information. For example, the calculation unit 113 may associate the calculated recognition accuracy with the applied output condition and store it in the storage unit 130 as accuracy information. For example, the calculation unit 113 may calculate the degree to which the text of the test text information and the text of the recognition result text information match as the recognition accuracy. For example, when the text of the test text information and the text of the recognition result text information match 80% of the time, the calculation unit 113 may calculate the recognition accuracy to be "0.8" or "80%".
[0058] The calculation unit 113 may specify a threshold or a reference value for each of the multiple parameters based on, for example, a combination of an output condition applied during a predetermined learning period and the recognition accuracy calculated under the output condition. The calculation unit 113 stores the specified threshold as threshold information in the threshold storage unit 132. The calculation unit 113 may also store the specified reference value as reference value information in the storage unit 130.
[0059] <Volume threshold or reference value> For example, when specifying the threshold value of the volume, the calculation unit 113 may construct a first pattern model of the volume and the recognition accuracy as shown in FIG. 7 by inputting the volume of the voice during a predetermined learning period and the corresponding recognition accuracy of the voice as learning data. For example, the calculation unit 113 may construct the first pattern model using statistical processing by regression analysis with the volume as an explanatory variable (feature amount) and the recognition accuracy as a target variable (feature amount). The calculation unit 113 may input the volume of the voice to the constructed first pattern model to calculate the recognition accuracy. For example, the calculation unit 113 divides the possible range of the volume for each usage scene into multiple stages ("high: volume at which sufficient recognition accuracy is obtained" and "medium: volume at which acceptable recognition accuracy is obtained") according to multiple recognition accuracy values (in this example, sufficient recognition accuracy (0.9) and acceptable recognition accuracy (0.7)). The calculation unit 113 may specify, for example, the upper limit value and / or the lower limit value of either of the two divided ranges (R1 or R2) as the threshold value of the volume. Furthermore, the calculation unit 113 may specify, for example, the volume value (d1) at which the recognition accuracy in the first pattern model is maximized as the reference volume value.
[0060] <distance threshold or reference value> The calculation unit 113 may construct a second pattern model of the distance from the voice output device 300 and the recognition accuracy as shown in FIG. 8 by inputting, as learning data, the distance between the voice recognition device 200 and the voice output device 300 during a predetermined learning period and the corresponding recognition accuracy. The calculation unit 113 may construct the second pattern model using statistical processing by regression analysis, with the distance from the voice output device 300 as an explanatory variable (feature amount) and the recognition accuracy as an objective variable (feature amount). The calculation unit 113 may input the distance from the voice output device 300 to the constructed second pattern model to calculate the recognition accuracy. The calculation unit 113 may divide, for example, the possible range of the distance from the speaker into multiple stages according to multiple recognition accuracy values, as in the example of the volume above. The calculation unit 113 may specify, for example, the upper limit value and / or the lower limit value of either of the two divided ranges (R3 or R4) as the threshold value of the distance from the voice output device 300. Furthermore, the calculation unit 113 may specify, for example, the volume value (d2) at which the recognition accuracy in the second pattern model is maximized as the reference value of the distance.
[0061] <Frequency threshold or reference value> The calculation unit 113 may calculate the recognition accuracy based on, for example, the frequency of the voice output by the voice output device 300 during a predetermined learning period. The calculation unit 113 may calculate, for example, a statistical value (average value or median value) of the frequency and specify the calculated statistical value as the reference value of the frequency. The calculation unit 113 may also calculate, for example, a frequency band and specify an upper limit or lower limit of the calculated frequency band as the frequency threshold value.
[0062] The calculation unit 113 may extract, as a feature, for example, the power (or sound pressure level) of at least one of consonants or a high frequency range equal to or higher than a predetermined threshold, contained in the voice of the voice data during a predetermined learning period. The "power" referred to here is so-called acoustic power, which indicates the weight (power) for each frequency in a sound frequency analysis, and differs from the loudness or strength (volume) of a sound sensed by human hearing. The power is the strength of a consonant or a high frequency range equal to or higher than a predetermined threshold. The calculation unit 113 may calculate the recognition accuracy based on the extracted feature. For example, the calculation unit 113 may weight the recognition accuracy calculated above by the power of the consonant, and classify the weighted recognition accuracy into two stages ("high" and "medium") similar to the above volume example.
[0063] For example, the calculation unit 113 may plot the sound pressure level and frequency of the voice on a graph with the sound pressure level on the vertical axis and the frequency on the horizontal axis. As in the above example of volume, the calculation unit 113 divides the possible range of the plotted data into two recognition accuracy areas ("high" and "medium"). For example, the calculation unit 113 may calculate the recognition accuracy depending on which of the two divided areas the sound pressure level and frequency of the voice belong to.
[0064] The calculation unit 113 may calculate a composite recognition accuracy (hereinafter also referred to as a "composite recognition accuracy") based on a combination of recognition accuracies calculated based on at least one of (a) volume, (b) distance from the speaker, (c) frequency, and (d) power of consonants or a high frequency range equal to or higher than a predetermined threshold, as described above. The calculation unit 113 may specify a threshold or a reference value for each parameter based on the composite recognition accuracy. For example, the calculation unit 113 may specify the value of each parameter when the composite recognition accuracy is maximized as the threshold or the reference value.
[0065] The calculation unit 113 may, for example, calculate a weighted average of the recognition accuracies (a) to (e) above, and calculate the calculated weighted average as the composite recognition accuracy. For example, in calculating this weighted average, the calculation unit 113 may set the importance of (a) and (b) above higher than the other (c) to (e). The calculation unit 113 may, for example, weight each recognition accuracy by a coefficient proportional to the importance. Specifically, the calculation unit 113 may calculate the composite recognition accuracy by the following formula.
[0066] Composite recognition accuracy = (α × recognition accuracy of (E) above + β × recognition accuracy of (A) above + θ × recognition accuracy of (B) above + δ + recognition accuracy of (C) above) / (α + β + θ + δ)
[0067] "α" is the weighting coefficient of (E) above, "β" is the weighting coefficient of (A) above, i.e., the volume, "θ" is the weighting coefficient of (B) above, i.e., the distance, and "δ" is the weighting coefficient of (C) above, i.e., the frequency. β and θ may be greater than α and δ depending on the set importance.
[0068] The calculation unit 113 may calculate, for example, a correlation degree (correlation coefficient) indicating the degree of correlation between at least two or more parameters among the multiple parameters. For example, the calculation unit 113 may calculate the correlation coefficient between the volume and the recognition accuracy by dividing the covariance between the volume and the recognition accuracy by the product of the standard deviation of the volume and the standard deviation of the recognition accuracy as the correlation degree between the volume and the recognition accuracy. As another example, the calculation unit 113 may calculate the correlation coefficient between two or more parameters as the correlation degree between two or more parameters by using a multivariate analysis technique (such as multiple regression analysis or logistic regression).
[0069] The calculation unit 113 may calculate, for example, the reliability of the recognition result. The calculation unit 113 may calculate, for example, the reliability of each word included in the recognition result, and calculate the recognition accuracy of the recognition result by aggregating the calculated reliability of each word.
[0070] The confidence for each word may have a value within a predetermined range (for example, a range of 0.0 to 1.0). The closer the value is to 1.0, i.e., the upper limit, within the predetermined range, the less the confidence for each word indicates that there are relatively few other competitive candidates having a score similar to that word. On the other hand, the closer the value is to 0.0, i.e., the lower limit, within the predetermined range, the more the confidence for each word indicates that there are relatively many other competitive candidates having a score similar to that word. In other words, the closer the value is to the upper limit, within the predetermined range, the less the confidence for each word indicates that there are no other candidates similar to the word of the top candidate in the recognition result, and the recognition result is output with confidence (certainty).
[0071] There are several methods for calculating the reliability of a word. For example, Komatani and Kawahara, "Dialogue processing for efficient confirmation and guidance using reliability of speech recognition results" (Transactions of Information Processing Society of Japan, Vol. 43, No. 10, pp. 3078-3086) is known.
[0072] <Output section> The output unit 114 refers to the storage unit 130 that stores the accuracy information, and outputs the recognition accuracy calculated by the calculation unit 113 in association with the output condition. The output unit 114 may output this association in any manner, and may, for example, output it to a screen, or may output it to a csv file or a table-format file.
[0073] Now, with reference to FIG. 5 and FIG. 6, an example of the output of the output section 114 will be described.
[0074] 5, the output unit 114 may output (display) a recognition accuracy evaluation screen A1 on a user terminal (not shown) for each speech recognition device 200 to be evaluated. The recognition accuracy evaluation screen A1 includes, for each output condition to be applied to the speech recognition device 200 to be evaluated, information for identifying the output condition (No. in this example), the contents of the output condition, the recognition accuracy corresponding to the output condition, and an execute button for outputting voice according to the output condition. When the execute button is pressed by the user, the control device 100 instructs the voice output device 300 to output voice.
[0075] As shown in FIG. 6, the output unit 114 may output the recognition accuracy evaluation screen A2 to the user terminal for each speech recognition device to be evaluated and for each usage scenario (in this example, "creating minutes in a conference room (small size)"). In the recognition accuracy evaluation screen A2, output conditions that do not exceed the parameter thresholds of the usage scenario are extracted and output as output conditions to be applied to the speech recognition device 200 to be evaluated and the usage scenario. For each extracted output condition, the recognition accuracy evaluation screen A2 includes information for identifying the output condition (No in this example), the contents of the output condition, the recognition accuracy corresponding to the output condition, and an execute button for outputting voice according to the output condition. When the execute button is pressed by the user, the control device 100 instructs the voice output device 300 to output voice.
[0076] The output unit 114 may, for example, refer to the threshold storage unit 132 that stores the threshold information, and extract and output, for each usage screen, a screen whose recognition accuracy is equal to or higher than a predetermined accuracy and whose associated parameters do not exceed a threshold, based on the threshold information. Also, the output unit 114 may, for example, extract and output, for each usage screen, a screen whose parameters associated with the recognition accuracy do not exceed a threshold, based on the threshold information. The "usage scene" is a scene in which the voice recognition device 200 is used.
[0077] The "threshold information" is information indicating the thresholds of a plurality of parameters for each usage scenario. The thresholds indicated in the threshold information may be, for example, a preset value (fixed value) or a value determined by the calculation unit 113.
[0078] Generally, the recognition accuracy tends to improve with an increase in volume, but there are some usage situations, such as meetings, where a high volume is not appropriate or acceptable. According to the above configuration, the output unit 114 can extract only those parameters that fall within an appropriate or acceptable range according to the usage situation and output them to the user. Therefore, according to the above configuration, the user can efficiently evaluate each usage situation that he or she wishes to evaluate.
[0079] The output unit 114 may, for example, refer to a reference value storage unit (not shown) that stores reference value information, and based on the reference value information, extract and output, for each screen in use, a screen whose recognition accuracy is equal to or higher than a predetermined accuracy and whose associated parameters are within a predetermined range from the reference value. Also, the output unit 114 may, for example, extract and output, for each screen in use, a screen whose parameters associated with the recognition accuracy are within a predetermined range from the reference value, based on the reference value information.
[0080] The "reference value information" is information indicating the reference value of each of a plurality of parameters for each usage scenario. The threshold value indicated in the reference value information may be, for example, a preset value (fixed value) or a value specified by the calculation unit 113.
[0081] Generally, the recognition accuracy tends to improve as the volume increases, but there are some usage situations, such as meetings, where a high volume is not appropriate or acceptable. According to the above configuration, the output unit 114 can extract each parameter based on a reference value according to the usage situation and output it to the user. Therefore, according to the above configuration, the user can efficiently evaluate each usage situation that he or she wants to evaluate.
[0082] For example, when outputting the output conditions, the output unit 114 may rearrange (sort) a plurality of parameters included in the output conditions in descending order of correlation with the recognition accuracy and output the rearranged (sorted) parameters. Also, when outputting the output conditions, the output unit 114 may output a parameter having the highest correlation with the recognition accuracy among the plurality of parameters included in the output conditions in a identifiable manner (for example, by displaying in a highlighted manner). With this configuration, the output unit 114 allows the user to understand which parameter has influenced the recognition accuracy and to make an evaluation.
[0083] <Generation part> The generating unit 115 may generate voice data including a test voice based on the test text information using a voice synthesis process.
[0084] The generating unit 115 may process the first test voice data and the second test voice data into stereo voice data, for example. Here, the "stereo voice data" refers to voice data for stereophonically reproducing the first test voice and the second test voice. The generating unit 115 may localize the sound image of the voice of the first test voice data or the second test voice data, for example, as a pre-processing of the stereo voice data. The processing unit 122 may localize the sound image of the first test voice at the position of a virtual sound source for each speaker (channel) for the first test voice data, for example. The position of this virtual sound source may be set according to the sound source localization position of the output condition, for example.
[0085] <Storage section> The storage unit 130 stores voice data, reference value information, and accuracy information. The storage unit 130 may store the above various information and data by utilizing a database management system (DBMS), or may store the above information by utilizing a file system. When a DBMS is utilized, a table may be provided for each of the above information, and the information may be managed by associating the tables with each other. The storage unit 130 further includes a text storage unit 131 and a threshold storage unit 132.
[0086] The text storage unit 131 stores test text information and recognition result text information. The threshold storage unit 132 stores threshold information.
[0087] <Communications Department> The communication unit 140 transmits and receives various information and data such as voice data and text information to and from the voice recognition device 200 and the voice output device 300 via a wired network or a wireless network.
[0088] <4. Example of operation> An example of the operation of the control device 100 will be described with reference to Fig. 8. Note that the order of processing in the example of the operation of Fig. 8 shown below is just an example, and may be changed as appropriate. In this example, an example will be described in which voice output and voice recognition are executed multiple times based on the 1st to Nth (N is a natural number) output conditions as a test scenario in sequence, and the recognition accuracy of the voice recognition device 200 is calculated each time.
[0089] 8, the audio output control unit 111 of the control device 100 controls the output of audio from the audio output device 300 based on the Nth output condition (S10). Next, the acquisition unit 112 acquires recognition result text information indicating the recognition result of the audio output from the audio recognition device 200 that recognized the audio output from the audio output device 300 (S11).
[0090] Next, the calculation unit 113 refers to the text storage unit 131 that stores the test text information, and calculates the recognition accuracy of the recognition result based on the test text information and the recognition result text information (S12). Next, the output unit 114 outputs the calculated recognition accuracy and the Nth output condition in association with each other (S13).
[0091] Next, if the test scenario includes the (N+1)th output condition (Yes in S14), the audio output control unit 111 increments N (S15). The flow returns to before step S10.
[0092] Next, if the N+1th output condition does not exist in the test scenario (No in S14), the flow ends.
[0093] <5. Hardware configuration> 9, an example of a hardware configuration in which the above-mentioned control device 100 is realized by a computer 800 will be described. Note that the functions of each device can also be realized by dividing them into a plurality of devices.
[0094] As shown in FIG. 9, a computer 800 includes a processor 801, a memory 803, a storage device 805, an input I / F unit 807, a data I / F unit 809, a communication I / F unit 811, and a display device 813.
[0095] The processor 801 controls various processes in the computer 800 by executing programs stored in the memory 803. For example, each functional unit of the control unit 110 of the control device 100 can be realized by the processor 801 executing a program temporarily stored in the memory 803.
[0096] The memory 803 is a storage medium such as a RAM (Random Access Memory), etc. The memory 803 temporarily stores the program code of the program executed by the processor 801 and data required when the program is executed.
[0097] The storage device 805 is a non-volatile storage medium such as a hard disk drive (HDD) or a flash memory. The storage device 805 stores an operating system and various programs for implementing the above-mentioned configurations. In addition, the storage device 805 can also store tables for registering voice data, text information (test text information and recognition result text information), threshold information, and the like, and a DB for managing these tables. Such programs and data are loaded into the memory 803 as necessary, and are referred to by the processor 801.
[0098] The input I / F unit 807 is a device for receiving input from a user. Specific examples of the input I / F unit 807 include a keyboard, a mouse, a touch panel, various sensors, a wearable device, etc. The input I / F unit 807 may be connected to the computer 800 via an interface such as a USB (Universal Serial Bus).
[0099] The data I / F unit 809 is a device for inputting data from outside the computer 800. A specific example of the data I / F unit 809 is a drive device for reading data stored in various storage media. The data I / F unit 809 may be provided outside the computer 800. In that case, the data I / F unit 809 is connected to the computer 800 via an interface such as a USB.
[0100] The communication I / F unit 811 is a device for performing data communication via the Internet N, either wired or wirelessly, with an external device of the computer 800. The communication I / F unit 811 may be provided outside the computer 800. In that case, the communication I / F unit 811 is connected to the computer 800 via an interface such as a USB.
[0101] The display device 813 is a device for displaying various types of information. Specific examples of the display device 813 include a liquid crystal display, an organic EL (Electro-Luminescence) display, a display of a wearable device, and the like. The display device 813 may be provided outside the computer 800. In that case, the display device 813 is connected to the computer 800 via, for example, a display cable. In addition, when a touch panel is adopted as the input I / F unit 807, the display device 813 can be configured integrally with the input I / F unit 807.
[0102] It should be noted that the present embodiment is an example for explaining the present invention, and is not intended to limit the present invention to only the embodiment. In addition, the present invention can be modified in various ways without departing from the gist of the invention. Furthermore, a person skilled in the art can adopt an embodiment in which each element described above is replaced with an equivalent, and such an embodiment is also included in the scope of the present invention.
[0103] [Variations] Although the present invention has been described based on the above embodiment, the following cases are also included in the present invention.
[0104] [Variation 1] At least a part of each component in the control device 100 according to the above embodiment may be provided by a program dedicated to the evaluation system 1 to be installed in the voice recognition device 200. For example, this program may be provided with each functional unit of the control unit 110 of the control device 100, and the evaluation system 1 may be configured with the voice recognition device 200 and the voice output device 300 without providing a separate control device 100.
[0105] [Variation 2] Although an example has been described in which the text storage unit and the threshold storage unit according to the above embodiment are provided in the storage unit 130 of the control device 100, the text storage unit and the threshold storage unit are not limited to this. The text storage unit and the threshold storage unit according to the above embodiment may be provided in the voice recognition device 200, or may be provided in a device of an external system, for example.
[0106] [Variation 3] Although not shown in the above embodiment, the audio output device 300 may be manually arranged, or the control device 100 and the audio output device 300 may be linked by an arm or the like. In the evaluation system 1, the audio output control unit 111 may control (for example, rotate or extend) this arm based on the output condition to be applied, thereby arranging the audio output device 300. [Explanation of symbols]
[0107] 1...evaluation system, 100...control device, 110...control unit, 111...audio output control unit, 112...acquisition unit, 113...calculation unit, 114...output unit, 115...generation unit, 130...storage unit, 140...communication unit, 200...speech recognition device, 300...speech output device, 500...speech recognition system, 800...computer, 801...processor, 803...memory, 805...storage device, 807...input I / F unit, 809...data I / F unit, 811...communication I / F unit, 813...display device, 817...audio input device, 819...audio output device.
Claims
1. an audio output control unit that controls an output of audio from an audio output device based on an audio output condition; an acquisition unit that acquires, from a voice recognition device that recognizes the voice output from the voice output device, recognition result text information indicating a recognition result of the voice; a calculation unit that refers to a text storage unit that stores test text information indicating the contents of a test voice included in the voice, and calculates a recognition accuracy of the recognition result based on the test text information and the recognition result text information; an output unit that outputs the calculated recognition accuracy and the condition in association with each other, the audio output control unit changes the condition based on the recognition accuracy calculated by the calculation unit, and controls output of audio from the audio output device based on the changed condition; The speech output by the speech output device and the speech recognition by the speech recognition device for the speech output are executed a plurality of times; the condition includes a plurality of parameters related to the audio output; There are a plurality of the audio output devices, The audio data of the audio to be output by the audio output device includes test audio data of the test audio and noise data for generating noise; the plurality of parameters include device designation information for designating whether the test voice or the noise is to be output for each of the plurality of times, The audio output control unit controls the audio output by switching the audio to be output from each of the plurality of audio output devices between the test audio and the noise for each of the times based on the device designation information. Information processing device.
2. The audio output control unit controls the audio output by changing one or more parameters used among the plurality of parameters in each of the times. The information processing device according to claim 1 .
3. The output unit outputs the recognition accuracy of each of the speech recognitions performed multiple times in association with one or more parameters used to control the speech output in each of the speech recognitions. The information processing device according to claim 2 .
4. the output unit refers to a threshold storage unit that stores threshold information indicating thresholds for each of the plurality of parameters for each usage scenario in which the voice recognition device is used, and further based on the threshold information, extracts and outputs, for each usage scenario, a parameter whose recognition accuracy is equal to or higher than a predetermined accuracy and whose associated parameter does not exceed the threshold. The information processing device according to claim 3 .
5. the plurality of parameters include at least two of information regarding a sound source localization position of the sound to be output by the sound output device, an utterance speed of the sound, a volume of the sound, or information regarding a speaker of the sound, The audio output control unit controls the audio output by changing the value of the parameter. The information processing device according to claim 1 .
6. The parameters include device selection information for selecting a device to be turned on among the plurality of audio output devices at each of the times, The audio output control unit controls the audio output by switching on / off the audio output of each of the plurality of audio output devices at each time, further based on the device selection information. The information processing device according to claim 1 .
7. the parameters include a length of the audio to be output by the audio output device; The test voice data includes a first test voice data of a first test voice and a second test voice data of a second test voice having a length different from that of the first test voice, The audio output control unit controls the audio output by switching the audio to be output from each of the audio output devices between the first test audio and the second test audio for each of the times. The information processing device according to claim 1 .
8. On the computer, an audio output control function for controlling audio output from an audio output device based on an audio output condition; an acquisition function for acquiring, from a voice recognition device that recognizes the voice output from the voice output device, recognition result text information indicating a recognition result of the voice; a calculation function of calculating a recognition accuracy of the recognition result based on the test text information and the recognition result text information by referring to a text storage unit that stores test text information indicating the content of the test voice included in the voice; an output function of outputting the calculated recognition accuracy in association with the condition; the audio output control function changes the condition based on the recognition accuracy calculated by the calculation function, and controls the output of audio from the audio output device based on the changed condition; The speech output by the speech output device and the speech recognition by the speech recognition device for the speech output are executed a plurality of times; the condition includes a plurality of parameters related to the audio output; There are a plurality of the audio output devices, The audio data of the audio to be output by the audio output device includes test audio data of the test audio and noise data for generating noise; the plurality of parameters include device designation information for designating whether the test voice or the noise is to be output for each of the plurality of times, The audio output control function controls the audio output by switching the audio to be output from each of the plurality of audio output devices between the test audio and the noise in each round based on the device designation information. program.
9. The computer Controlling audio output from an audio output device based on the audio output conditions; obtaining recognition result text information indicating a recognition result of the voice from a voice recognition device that recognizes the voice output from the voice output device; calculating a recognition accuracy of the recognition result based on the test text information and the recognition result text information by referring to a text storage unit that stores test text information indicating the content of the test voice included in the voice; outputting the calculated recognition accuracy in association with the condition; changing the condition based on the calculated recognition accuracy; controlling an output of audio from the audio output device based on the changed condition; The speech output by the speech output device and the speech recognition by the speech recognition device for the speech output are executed a plurality of times; the condition includes a plurality of parameters related to the audio output; There are a plurality of the audio output devices, The audio data of the audio to be output by the audio output device includes test audio data of the test audio and noise data for generating noise; the plurality of parameters include device designation information for designating whether the test voice or the noise is to be output for each of the plurality of times, and controlling the audio output by switching the audio to be output from each of the plurality of audio output devices between the test audio and the noise for each of the times based on the device designation information. Information processing methods.
Citation Information
Patent Citations
Voice recognizing device and process condition setting method in voice recognizing device
JP2002328696A
Speech recognizing device
JP2003280678A
Apparatus, method, and program for speech recognition evaluation
JP2005084102A
Signal processing method and signal processing device
JP2005258215A
Voice recognition accuracy estimation device, voice recognition accuracy estimation method and voice recognition accuracy estimation program
JP2018025717A
