Information processing device, information processing method, and computer program

Abstract

To inhibit the number of evaluations about each pair from being unequal and to inhibit the number of evaluations about each pair from becoming excessive in subjective comparative evaluations.SOLUTION: An information processing device for executing asynchronous and parallel subjective comparative evaluations by a plurality of evaluators, includes an evaluation object setting part, an evaluation request part, and an evaluation state update part. The evaluation object setting part dynamically sets an evaluation object pair to be an object of subjective comparative evaluations on the basis of prediction values of subjective comparative evaluations about each pair of an evaluation object. The evaluation request part requests the evaluators to perform a subjective comparative evaluation about the evaluation object pair. The evaluation state update part dynamically updates evaluation states about the evaluation object pair on the basis of evaluation results by the evaluators.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The technology disclosed in this specification relates to information processing for performing asynchronous and parallel subjective comparative evaluations by a plurality of evaluators. [Background technology]

[0002] For example, subjective evaluation is used when evaluating generated media such as synthetic speech. For example, in subjective evaluation of synthetic speech, a value evaluation is performed in which human evaluators listen to a sample of the synthetic speech and assign a score. In a value evaluation, each evaluator assigns a score on a five-point scale, for example from "1: Very bad" to "5: Very good," and the evaluation value is the mean opinion score (abbreviated as "MOS"), which is the average of all the evaluators' scores.

[0003] In value evaluation, only the evaluation scale is given, and in many cases, clear evaluation criteria are not determined, so the bias and noise of the evaluation by the evaluator is likely to be large. Therefore, in order to distinguish the difference in quality, it is necessary to collect a large number of evaluations. In addition, for example, the quality of generated media has improved significantly due to recent advances in machine learning technology, and it is becoming difficult to distinguish the difference in quality using value evaluation.

[0004] Comparative evaluation is a method for evaluating items with similar quality. Comparative evaluation is a subjective evaluation method in which two samples are presented to the evaluator and the evaluator is asked to choose which one is better. Compared to value evaluation, comparative evaluation has less bias among evaluators and makes it easier to distinguish quality differences with a small number of evaluations. However, in comparative evaluation, the number of pairs of items being evaluated is enormous, and it is difficult to evaluate all pairs. For this reason, a two-stage evaluation method is often adopted in which rough trends are observed using value evaluation, and then only pairs with small differences are extracted and a comparative evaluation is performed, which can be redundant and inefficient.

[0005] Online learning is a method for efficiently allocating evaluation targets and the number of evaluations to achieve a certain evaluation accuracy. Based on the evaluation results by the evaluators, the online learning algorithm allocates the number of evaluations so that the confidence intervals of those with similar quality are farther apart, thereby ensuring the accuracy of evaluations of similar quality. Conversely, by cutting off the evaluation based on the confidence interval, the number of evaluations is minimized for evaluation targets whose quality is not similar to others.

[0006] The MergeRank algorithm is one of the online learning methods (see, for example, Non-Patent Document 1). The MergeRank algorithm efficiently determines whether the quality is high or low by inferring the total order of the objects to be evaluated based on a sorting algorithm, and optimizes the combination of pairs for comparative evaluation and the number of evaluations for each pair by changing the comparison function of the merge sort to the evaluation value by the evaluator. When online learning is introduced into subjective evaluation, it is expected to realize more efficient evaluation by dynamically determining the evaluation target based on the evaluation by the evaluator. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Moein Falahatgar et al., "Maximum Selection and Ranking under Noisy Comparisons," Proceedings of Machine Learning Research, 2017, Vol. 70, pp. 1088-1096. Summary of the Invention [Problem to be solved by the invention]

[0008] Conventional online learning algorithms are based on the premise of sequential execution on offline data. Therefore, conventional online learning algorithms cannot be applied to subjective comparative evaluation in a crowdsourcing environment as is. That is, in a crowdsourcing environment, evaluation is performed asynchronously and in parallel by multiple evaluators, so it takes time from when a pair to be evaluated is presented to the evaluator until the evaluator returns the evaluation result. Therefore, a simple algorithm that sets the pair with the worst evaluation accuracy among each pair of objects to be evaluated as the pair to be evaluated may result in an uneven number of evaluations for each pair. In addition, the number of evaluations for each pair may be excessive.

[0009] For example, after issuing an evaluation request for a certain pair (pair A) to a certain evaluator (evaluator E1), if the evaluation result of pair A by evaluator E1 has not been returned at the time of setting the next pair to be evaluated, pair A will remain the pair with the worst evaluation accuracy, and an evaluation request will be issued to another evaluator (evaluator E2) with pair A as the evaluation target. As a result, there is a risk that the number of evaluations for each pair will become excessively uneven. Also, depending on the relationship between the timing of issuing the evaluation request and the timing of receiving the evaluation result, the evaluation state for pair A may have converged at the time the evaluation result for pair A is received. In that case, the number of evaluations for pair A will be excessive.

[0010] Thus, if conventional online learning algorithms are simply applied to subjective comparative evaluation in a crowdsourcing environment, there is a risk that the number of evaluations for each pair will be overly uneven, and that the number of evaluations for each pair will be excessively large, which may result in a decrease in the overall accuracy and efficiency of the subjective comparative evaluation.

[0011] Note that these issues are not limited to evaluations in a crowdsourcing environment, but are common when performing asynchronous and parallel subjective comparative evaluations by multiple evaluators.

[0012] This specification discloses a technique that can solve the above-mentioned problems. [Means for solving the problem]

[0013] The technology disclosed in this specification can be realized, for example, in the following forms.

[0014] (1) The information processing device disclosed in this specification is a device that executes asynchronous and parallel subjective comparative evaluations by multiple evaluators, and includes an evaluation target setting unit, an evaluation request unit, and an evaluation status update unit. The evaluation target setting unit dynamically sets evaluation target pairs to be subjected to the subjective comparative evaluation based on a predicted value of the subjective comparative evaluation for each pair of objects to be evaluated. The evaluation request unit requests the evaluators to perform the subjective comparative evaluation for the evaluation target pairs. The evaluation status update unit dynamically updates the evaluation status for the evaluation target pairs based on the evaluation results by the evaluators.

[0015] In this way, in the information processing device, the evaluation target setting unit dynamically sets evaluation target pairs to be subjected to the subjective comparative evaluation based on the predicted value of the subjective comparative evaluation for each pair of evaluation objects. Therefore, it is possible to prevent the number of evaluation requests for a certain pair from becoming disproportionately large, or the number of evaluations for a certain pair from becoming disproportionately large. Therefore, according to the information processing device, even when performing asynchronous and parallel subjective comparative evaluations by multiple evaluators, it is possible to prevent the number of evaluations for each pair from becoming uneven, and to prevent the number of evaluations for each pair from becoming excessive.

[0016] (2) In the information processing device, the evaluation target setting unit may be configured to set, among the pairs of the evaluation objects, a pair having the worst predicted value of the accuracy of the subjective comparative evaluation as the evaluation target pair. With this configuration, it is possible to effectively prevent the number of evaluations for each pair from becoming uneven and to effectively prevent the number of evaluations for each pair from becoming excessive.

[0017] (3) In the information processing device, the evaluation target setting unit may be configured to calculate a predicted value of the accuracy of the subjective comparative evaluation for each pair of the evaluation objects based on the number of requests for the subjective comparative evaluation. With this configuration, the predicted value of the accuracy of the subjective comparative evaluation can be calculated more appropriately, and the number of evaluations for each pair can be effectively prevented from becoming uneven, and the number of evaluations for each pair can be effectively prevented from becoming excessive.

[0018] (4) In the information processing device, the evaluation state update unit may be configured not to update the evaluation state of the pair to be evaluated when the evaluation state of the pair to be evaluated is in a predetermined convergence state at the time of receiving the evaluation result. With this configuration, even when performing asynchronous and parallel subjective comparative evaluations by multiple evaluators, it is possible to avoid the occurrence of inconsistencies and suppress deterioration of evaluation performance without introducing a complex mechanism (e.g., a mechanism using a queue that executes updates sequentially) that simultaneously prevents inconsistencies in the evaluation states of each pair and updates the evaluation states.

[0019] (5) In the information processing device, the evaluation state update unit may be configured to hold the evaluation result instead of discarding it if the evaluation state for the pair to be evaluated is in a predetermined convergence state at the time of receiving the evaluation result. With this configuration, it is possible to prevent the occurrence of inconsistencies in the evaluation state and to suppress a decrease in usability for the evaluator.

[0020] (6) The information processing device further includes an upper limit setting unit that sets an upper limit for the total number of evaluations in the subjective comparative evaluation, and an accuracy setting unit that sets a minimum guaranteed accuracy of the subjective comparative evaluation that can be achieved by the evaluation of the number of times of the upper limit, and if the total number of evaluations has not reached the upper limit at the time when the minimum guaranteed accuracy is achieved, the evaluation target setting unit may set the evaluation target pair based on the evaluation state for each pair of the evaluation objects until the total number of evaluations reaches the upper limit. According to this configuration, even if the total number of evaluations is fixed under a fixed budget in a crowdsourcing environment, for example, it is possible to obtain an evaluation result with a minimum guaranteed accuracy within the budget, and the overall evaluation accuracy can be improved by evaluations continued after the minimum guaranteed accuracy is achieved.

[0021] (7) In the information processing device, the evaluation target setting unit may be configured to obtain an initial order of the evaluation targets and set the evaluation target pairs based on the initial order. With this configuration, it is possible to efficiently perform subjective comparative evaluation based on the initial order.

[0022] The technology disclosed in this specification can be realized in various forms, such as an information processing device, an information processing method, a computer program for realizing those methods, a non-transitory recording medium on which that computer program is recorded, etc. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 is an explanatory diagram showing a configuration of an information processing system 10 according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a block diagram showing a schematic configuration of a server device 100. [Diagram 3] Flowchart showing evaluation process [Figure 4] A conceptual diagram of subjective comparative evaluation of evaluation target pairs. [Diagram 5] Another explanatory diagram conceptually showing subjective comparative evaluation of evaluation target pairs [Figure 6]FIG. 1 is an explanatory diagram conceptually illustrating a rating request process and a rating status update process according to an embodiment of the present invention. [Figure 7] FIG. 13 is an explanatory diagram conceptually illustrating an evaluation request process and an evaluation status update process in a comparative example. [Figure 8] FIG. 1 is an explanatory diagram showing the results of the evaluation process of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] A. Embodiment: A-1. Configuration of information processing system 10: FIG. 1 is an explanatory diagram showing the configuration of an information processing system 10 in this embodiment. The information processing system 10 in this embodiment is a system for performing asynchronous and parallel subjective comparative evaluations by multiple evaluators for three or more objects to be evaluated. The objects to be evaluated in the subjective comparative evaluation may be tangible or intangible. In this embodiment, the information processing system 10 performs subjective comparative evaluation of synthetic voice (voice synthesis system), which is a generated media, in a crowdsourcing environment.

[0025] The information processing system 10 includes a server device 100 and terminal devices 200 used by multiple evaluators E (E1, E2, E3, ...). The terminal devices 200 are, for example, PCs, tablet terminals, smartphones, etc. Each device constituting the information processing system 10 is connected to each other so as to be able to communicate with each other via a communication network NET.

[0026] The server device 100 is a device that executes and manages the subjective comparative evaluation in the information processing system 10. The server device 100 is an example of an information processing device.

[0027] 2 is a block diagram showing a schematic configuration of the server device 100. The server device 100 includes a control unit 110, a storage unit 120, a display unit 130, an operation input unit 140, and an interface unit 150. These units are connected to each other via a bus 190 so as to be able to communicate with each other.

[0028] The display unit 130 of the server device 100 is, for example, a liquid crystal display, and displays various images and information. The operation input unit 140 is, for example, a keyboard, a mouse, buttons, a microphone, and the like, and receives operations and instructions from an administrator. The display unit 130 may be provided with a touch panel so as to function as the operation input unit 140. At least one of the display unit 130 and the operation input unit 140 may be omitted. The interface unit 150 is, for example, a LAN interface, a USB interface, and the like, and communicates with other devices and databases (for example, a relational database) via wired or wireless communication.

[0029] The storage unit 120 of the server device 100 is composed of, for example, a ROM, a RAM, a hard disk drive (HDD), a solid state drive (SSD), etc., and is used to store various programs and data, and as a work area when executing various programs, and as a temporary storage area for data. For example, the storage unit 120 stores an evaluation processing program CP, which is a computer program for executing the evaluation processing described below. The evaluation processing program CP is provided in a state stored in a computer-readable recording medium (not shown), such as a CD-ROM, a DVD-ROM, or a USB memory, or is provided in a state in which it can be obtained from an external device via the interface unit 150, and is stored in the storage unit 120 in a state in which it can operate on the server device 100.

[0030] The control unit 110 of the server device 100 is configured with, for example, a CPU, and controls the operation of the server device 100 by executing a computer program read from the storage unit 120. For example, the control unit 110 functions as an evaluation processing unit 111 that executes the evaluation processing described below by reading an evaluation processing program CP from the storage unit 120 and executing it. The evaluation processing unit 111 includes an upper limit setting unit 112, an accuracy setting unit 113, an evaluation target setting unit 114, an evaluation request unit 115, and an evaluation status update unit 116. The functions of each of these units will be described in conjunction with the explanation of the evaluation processing described below.

[0031] A-2. Evaluation process: Next, the evaluation process executed by the server device 100 of this embodiment will be described. FIG. 3 is a flowchart showing the evaluation process. The evaluation process is a process of subjectively comparing and evaluating synthetic voices in a crowdsourcing environment. In the subjective comparison evaluation in a crowdsourcing environment, a subjective comparison evaluation of an evaluation target pair set from a plurality of synthetic voices generated by a plurality of voice synthesis systems is performed asynchronously and in parallel by a plurality of evaluators E. Each evaluator E performs a subjective evaluation to select which of the pairs of synthetic voices presented as evaluation targets is superior. In addition, in the subjective comparison evaluation in a crowdsourcing environment, the budget is fixed in advance, and therefore the total number of evaluations is fixed.

[0032] FIG. 4 is an explanatory diagram conceptually showing a subjective comparative evaluation of an evaluation target pair. FIG. 4 illustrates an example of an evaluation state for a pair of systems i and j, both of which are speech synthesis systems. The winning rate (average preference score) WR ij If exceeds the threshold 0.5, system i is the winner. The error bias EB, which represents the accuracy of the subjective comparative evaluation of this pair, is calculated by the confidence interval CI (= δ / 2, δ is the confidence level) and the winning rate WR ij and the threshold value of 0.5. The smaller the error bias EB, the better the accuracy of the subjective comparative evaluation of this pair. When the error bias EB falls below the preset allowable bias TB, statistical significance has been established in the subjective comparative evaluation of this pair (i.e., the algorithm for performing the subjective comparative evaluation of this pair has converged).

[0033] Note that each pair of evaluation targets only needs to be evaluated up to a maximum number of evaluations m. The maximum number of evaluations m determines the budget that can guarantee the convergence of the algorithm. The maximum number of evaluations m for each pair of evaluation targets can be calculated in advance as the number of evaluations when the confidence interval CI is equal to the allowable bias TB, and when the maximum number of evaluations m is reached, the error bias EB will always be less than or equal to the allowable bias TB. In other words, as shown in Figure 5, the worst case in comparative evaluation is when both comparison targets are of the same quality (average preference score WR ij = 0.5). In order to keep the error bias EB below the tolerable bias TB even in this worst case, evaluations must be performed until the confidence interval CI becomes equal to the tolerable bias TB. Since the confidence interval CI depends only on the number of evaluations and not on the evaluation value, the maximum number of evaluations m in the worst case shown in Figure 5 can be calculated by determining the number of evaluations that satisfies the relationship of confidence interval CI = tolerable bias TB.

[0034] The evaluation processing unit 111 of the server device 100 executes evaluation processing according to an algorithm that is based on the well-known MergeRank algorithm and is improved in consideration of the fact that evaluations are performed asynchronously and in parallel in a crowdsourcing environment and that the total number of evaluations (budget) is fixed, as described above. The following description will focus on these improvements.

[0035] First, the upper limit setting unit 112 (FIG. 2) of the evaluation processing unit 111 sets an upper limit of the total number of evaluations based on a fixed budget determined in the crowdsourcing environment (S110). The upper limit of the total number of evaluations can be calculated, for example, by dividing the budget by the unit price per evaluation.

[0036] Next, the accuracy setting unit 113 of the evaluation processing unit 111 sets a minimum guaranteed accuracy that can be reliably achieved by the total number of evaluations of the upper limit value set in S110 (S120). As described above, in this embodiment, the accuracy of the subjective comparative evaluation is represented by the error bias EB (see FIG. 4), so the accuracy setting unit 113 sets the allowable bias TB as the minimum guaranteed accuracy. The accuracy setting unit 113 sets the allowable bias TB so that the upper limit of the total number of evaluations for the convergence of the algorithm that performs the subjective comparative evaluation for each pair, calculated based on the allowable bias TB, is equal to or smaller than the upper limit of the total number of evaluations set based on the budget in S110.

[0037] Next, the evaluation processing unit 111 executes an evaluation request process and an evaluation state update process (S130). Specifically, the evaluation target setting unit 114 of the evaluation processing unit 111 dynamically sets an evaluation target pair to be a subject of subjective comparative evaluation from each pair of synthetic voices, the evaluation request unit 115 requests the evaluator E to perform a subjective comparative evaluation on the evaluation target pair, and the evaluation state update unit 116 dynamically updates the evaluation state of the evaluation target pair based on the evaluation result by the evaluator E, thereby sorting (ranking) a plurality of synthetic voices. At this time, the evaluation target setting unit 114 calculates a predicted value of the accuracy of the subjective comparative evaluation (error bias EB) based on the number of evaluation requests for each pair, and sets the pair with the largest predicted value of the error bias EB (the actual value of the error bias EB when evaluation results for all evaluation requests have been returned) as the evaluation target pair.

[0038] FIG. 6 is an explanatory diagram conceptually illustrating the evaluation request process and the evaluation status update process in this embodiment. FIG. 6 conceptually illustrates the evaluation request process and the evaluation status update process for three pairs (pair A, pair B, pair C) of synthetic voices to be evaluated, which are the objects to be evaluated, to three evaluators E (E1, E2, E3). At timing t1 in the figure, the number of evaluation requests for each pair is 4, and the number of evaluations (the number of times the evaluation results have been returned to the server device 100) for each pair is also 4. That is, all the evaluation results requested to each evaluator E have been returned to the server device 100. At this time, the error bias EB has the smallest value for pair C (pair C has the best evaluation accuracy) and the largest value for pair A (pair A has the worst evaluation accuracy).

[0039] At timing t1 in Fig. 6, all the evaluation results requested to each evaluator E have been returned, so the evaluation state for each pair is determined, and the predicted value of the error bias EB is equal to the actual value. Therefore, the evaluation target setting unit 114 sets pair A, which has the largest error bias EB (worst evaluation accuracy), as the pair to be evaluated. The evaluation requesting unit 115 requests, for example, evaluator E1 to make a subjective comparative evaluation of pair A.

[0040] Next, at timing t2 in FIG. 6, the number of evaluation requests for pair A has increased by 1 compared to timing t1 to 5, but since the evaluation result for pair A by evaluator E1 has not returned, the number of evaluations for pair A remains at 4. Therefore, the error bias EB for pair A remains larger than the error bias EB of pairs B and C. In this case, if a rule that sets the pair with the largest error bias EB as the pair to be evaluated is applied, pair A will be set as the pair to be evaluated again. However, as described above, in this embodiment, the evaluation target setting unit 114 calculates a predicted value of the error bias EB based on the number of evaluation requests for each pair. After issuing an evaluation request, for the portion for which the evaluation result has not returned, the evaluation target setting unit 114 calculates an average value (the average preference score WR ij), but calculates a predicted value of the error bias EB reflecting the narrowing of the confidence interval CI with an increase in the number of evaluations. The evaluation target setting unit 114 sets the pair with the largest predicted value of the error bias EB as the pair to be evaluated. In the example of FIG. 6, the predicted value of the error bias EB calculated for pair A is smaller than the actual error bias EB for pairs B and C. Therefore, the evaluation target setting unit 114 sets pair B with the largest error bias EB as the pair to be evaluated. The evaluation requesting unit 115 requests a subjective comparative evaluation of pair B from, for example, evaluator E2.

[0041] Similarly, at timing t3 in FIG. 6, the number of evaluation requests for pair B has increased by 1 compared to timing t2 to 5, but since the evaluation result for pair B by evaluator E2 has not returned, the number of evaluations for pair B remains 4, and the error bias EB for pair B remains large. However, the evaluation target setting unit 114 calculates a predicted value of the error bias EB for pair B, reflecting the narrowing of the confidence interval CI accompanying the increase in the number of evaluations. In the example of FIG. 6, the predicted value of the error bias EB calculated for pair B is smaller than the actual error bias EB for pair C. Therefore, the evaluation target setting unit 114 sets pair C, which has the largest error bias EB, as the evaluation target pair. The evaluation request unit 115 requests, for example, evaluator E3 to make a subjective comparative evaluation of pair C.

[0042] At timing t4 in FIG. 6, the evaluation result of pair A by evaluator E1 is returned, and the number of evaluations for pair A increases by one to 5 compared to timing t3. The evaluation status update unit 116 updates the evaluation status of pair A based on the evaluation result by evaluator E1. Specifically, the evaluation status update unit 116 calculates an average value (average preference score WR shown in FIG. 4) based on the received evaluation results including the new evaluation result. ij ) and calculate the error bias EB to reflect the narrowing of the confidence interval CI as the number of evaluations increases.

[0043] Similarly, at timing t5 in Fig. 6, the evaluation result of pair B by evaluator E2 is returned, so the number of evaluations for pair B increases by 1 compared to timing t4 to 5. The evaluation status update unit 116 updates the evaluation status for pair B based on the evaluation result by evaluator E2. Also, at timing t6 in Fig. 6, the evaluation result of pair C by evaluator E3 is returned, so the number of evaluations for pair C increases by 1 compared to timing t5 to 5. The evaluation status update unit 116 updates the evaluation status for pair C based on the evaluation result by evaluator E3.

[0044] Evaluation processing unit 111 repeats such processing to sort (rank) a plurality of synthetic voices.

[0045] During the evaluation request process and the evaluation state update process of S130, the evaluation processing unit 111 monitors whether the minimum guaranteed accuracy set in S120 has been achieved (S140). More specifically, the evaluation processing unit 111 monitors whether the error bias EB for each pair to be evaluated has become equal to or less than the allowable bias TB set as the minimum guaranteed accuracy, and for a pair for which the error bias EB has become equal to or less than the allowable bias TB, the evaluation processing unit 111 determines that statistical significance has been established (i.e., the algorithm for performing subjective comparative evaluation for the pair has converged), and excludes the pair from candidates for subsequent evaluation target pairs. When the error bias EB for all pairs to be evaluated has become equal to or less than the allowable bias TB, the evaluation processing unit 111 determines that the minimum guaranteed accuracy has been achieved, and ends the evaluation request process and the evaluation state update process of S130.

[0046] Depending on the relationship between the timing of issuing the evaluation request and the timing of receiving the evaluation result, the evaluation state of a certain pair may have converged (the error bias EB is equal to or less than the allowable bias TB) at the time when the evaluation result of the certain pair is received. Even if the evaluation state update unit 116 receives the evaluation result of the certain pair after the evaluation state of the certain pair has converged, the evaluation state update unit 116 does not update the evaluation state of the certain pair. However, the evaluation state update unit 116 does not discard the evaluation result received after the evaluation state has converged, but holds it. Also, for example, when two or more pairs converge at the same time, if the evaluation states of the respective pairs are updated at the same time, the update of one pair overwrites the update of the other pair, and it becomes as if there was no update of the overwritten pair. Even when such a state (a conflict state) occurs, the evaluation state update unit 116 does not update the evaluation state of each pair.

[0047] After the minimum guaranteed accuracy is achieved, the evaluation processing unit 111 continues the evaluation until the total number of evaluations reaches the upper limit of the total number of evaluations set in S110. For example, the evaluation processing unit 111 issues an evaluation request for the pair with the largest error bias EB for the multiple synthetic voices sorted according to the evaluations in S130, thereby improving the overall evaluation accuracy. When the total number of evaluations reaches the upper limit, the evaluation processing unit 111 ends the evaluation process.

[0048] A-3. Advantages of this embodiment: As described above, the server device 100 of this embodiment is an information processing device that executes asynchronous and parallel subjective comparative evaluations by multiple evaluators, and includes an evaluation target setting unit 114, an evaluation request unit 115, and an evaluation status update unit 116. The evaluation target setting unit 114 dynamically sets evaluation target pairs to be subjected to subjective comparative evaluations based on predicted values ​​of subjective comparative evaluations for each pair of evaluation targets. The evaluation request unit 115 requests the evaluators to perform subjective comparative evaluations for the set evaluation target pairs. The evaluation status update unit 116 dynamically updates the evaluation status for the evaluation target pairs based on the evaluation results by the evaluators. Since the server device 100 of this embodiment has such a configuration, as described below, it is possible to prevent the number of evaluations for each pair from becoming uneven, and also to prevent the number of evaluations for each pair from becoming excessive.

[0049] FIG. 7 is an explanatory diagram conceptually illustrating the evaluation request process and the evaluation status update process in the comparative example. In the comparative example shown in FIG. 7, unlike the present embodiment shown in FIG. 6, a simple algorithm is adopted in which a pair having the worst evaluation accuracy (maximum error bias EB) among each pair of evaluation objects is set as the evaluation target pair. Therefore, in the comparative example shown in FIG. 7, at timing t1, pair A having the largest error bias EB is set as the evaluation target pair, and a request for subjective comparison evaluation of pair A is issued to, for example, evaluator E1. Next, at timing t2, since the evaluation result of pair A by evaluator E1 has not returned, pair A remains as the pair having the largest error bias EB, pair A is set as the evaluation target pair again, and a request for subjective comparison evaluation of pair A is issued to, for example, evaluator E2. Next, at timing t3, since the evaluation result of pair A by evaluators E1 and E2 has not returned, pair A remains as the pair having the largest error bias EB, pair A is set as the evaluation target pair again, and a request for subjective comparison evaluation of pair A is issued to, for example, evaluator E3. As a result, the number of evaluation requests for pair A will be disproportionately high.

[0050] After that, at timing t4, the evaluation result of pair A by evaluator E1 is returned, and the evaluation status of pair A is updated based on the evaluation result. Next, at timing t5, the evaluation result of pair A by evaluator E2 is returned, and the evaluation status of pair A is updated based on the evaluation result. Next, at timing t6, the evaluation result of pair A by evaluator E3 is returned, and the evaluation status of pair A is updated based on the evaluation result. As a result, the number of evaluations for pair A becomes disproportionately high.

[0051] Also, assume that at timing t4, the evaluation state of pair A is updated based on the evaluation result by evaluator E1, and the evaluation of pair A converges. In this case, if the evaluation state of pair A is updated based on the evaluation result by evaluator E2 at timing t5, an inconsistency occurs in the evaluation state of pair A.

[0052] In contrast, in the server device 100 of this embodiment, the evaluation target setting unit 114 dynamically sets evaluation target pairs to be subjected to subjective comparative evaluation based on the predicted value of the subjective comparative evaluation for each pair of evaluation targets (see FIG. 6). Therefore, it is possible to prevent the number of evaluation requests for a certain pair from becoming disproportionately large, or the number of evaluations for a certain pair from becoming disproportionately large. Therefore, according to the server device 100 of this embodiment, even when performing asynchronous and parallel subjective comparative evaluations by multiple evaluators, it is possible to prevent the number of evaluations for each pair from becoming uneven, and also to prevent the number of evaluations for each pair from becoming excessive.

[0053] Furthermore, in the server device 100 of this embodiment, the evaluation target setting unit 114 sets, among each pair of evaluation objects, the pair with the worst predicted value of the accuracy of the subjective comparative evaluation as the evaluation target pair. Therefore, according to the server device 100 of this embodiment, it is possible to effectively prevent the number of evaluations for each pair from becoming uneven, and also to effectively prevent the number of evaluations for each pair from becoming excessive.

[0054] Furthermore, in the server device 100 of this embodiment, the evaluation target setting unit 114 calculates a predicted value of the accuracy of the subjective comparative evaluation for each pair of evaluation targets based on the number of requests for subjective comparative evaluation. Therefore, according to the server device 100 of this embodiment, it is possible to more appropriately calculate a predicted value of the accuracy of the subjective comparative evaluation, and it is possible to effectively prevent the number of evaluations for each pair from becoming uneven, and also to effectively prevent the number of evaluations for each pair from becoming excessive.

[0055] In addition, in the server device 100 of this embodiment, when the evaluation state of the pair to be evaluated is in a predetermined convergence state at the time of receiving the evaluation result, the evaluation state update unit 116 does not update the evaluation state of the pair to be evaluated. Therefore, according to the server device 100 of this embodiment, even when performing asynchronous and parallel subjective comparative evaluation by multiple evaluators, it is possible to avoid the occurrence of inconsistency in the evaluation state of each pair. Furthermore, in the server device 100 of this embodiment, when the evaluation state of the pair to be evaluated is in a predetermined convergence state at the time of receiving the evaluation result, the evaluation state update unit 116 holds the evaluation result without discarding it. Therefore, according to the server device 100 of this embodiment, it is possible to prevent the occurrence of inconsistency in the evaluation state and suppress the deterioration of the usability of the evaluator.

[0056] The server device 100 of this embodiment further includes an upper limit setting unit 112 that sets an upper limit for the total number of evaluations in the subjective comparative evaluation, and an accuracy setting unit 113 that sets the minimum guaranteed accuracy of the subjective comparative evaluation that can be achieved by the evaluation of the maximum number of times. If the total number of evaluations has not reached the upper limit when the minimum guaranteed accuracy is achieved, the evaluation target setting unit 114 sets evaluation target pairs based on the evaluation state for each pair of evaluation targets until the total number of evaluations reaches the upper limit. Therefore, according to the server device 100 of this embodiment, even if the total number of evaluations is fixed under a fixed budget in a crowdsourcing environment, for example, it is possible to obtain evaluation results with a minimum guaranteed accuracy within the budget, and the overall evaluation accuracy can be improved by evaluations continued after the minimum guaranteed accuracy is achieved.

[0057] In addition, in the server device 100 of this embodiment, the evaluation target setting unit 114 may obtain an initial order of the evaluation objects and set the evaluation target pairs based on the initial order. According to this configuration, the subjective comparative evaluation can be efficiently performed based on the initial order.

[0058] A-4. Example: An example of the evaluation process performed by the server device 100 of the embodiment described above will be described. Fig. 8 is an explanatory diagram showing the results of the evaluation process example. Table 1 is a table showing the results of the evaluation process example.

[0059] [Table 1]

[0060] In this example, synthetic speech generated by 13 speech synthesis systems (GTPAC, GTPA, ..., etc. shown in Fig. 8 and Table 1) was used as the evaluation target. The evaluators were 304 people recruited by crowdsourcing.

[0061] For each system, 500 samples were evaluated. A subjective comparative evaluation of 39 pages and 500 sets was designed, with a total budget of 19,500. Each page containing two samples was presented to the evaluators, and they were asked to select the one that sounded more natural. The pair combinations and the evaluation amounts of the pairs were automatically set by the server device 100 based on the evaluation results by the evaluators.

[0062] The allowable bias TB was set to 0.06, and the confidence level δ was set to 0.05. In this configuration, the maximum number of evaluations m for each pair is 527. In such a configuration, the worst total number of evaluations ranges from 11,594 (all pairs were evaluated up to the maximum count) to 19,499, which, given the total budget mentioned above (19,500), ensures the reliable convergence of the algorithm performing subjective comparative evaluations for each pair.

[0063] Note that we used the MOS-based initial ordering for the following two reasons. (1) To avoid the worst case sorting complexity. (2) To investigate how different the sorting order based on subjective comparative evaluation is from the sorting order based on MOS.

[0064] Figure 8 shows the error bias EB after convergence for pairs that were actually evaluated out of a total of 78 pairs of pairs of 13 systems. Table 1 also shows the number of evaluations, win rate (average preference score), and WR for the pairs that were actually evaluated. ij , confidence interval CI, and error bias EB are shown.

[0065] As shown in FIG. 8, in this embodiment, evaluation was not performed for all 78 pairs, but rather, ranking (sorting) of 13 systems was completed by evaluation of 23 pairs.

[0066] Also, as shown in Table 1, fewer evaluations were assigned to pairs with relatively large quality differences, and more evaluations were assigned to pairs with relatively small quality differences. For example, the DFPAC:DFPA pair had a winning rate (average preference score) WR ij ,3122 evaluations were assigned because the difference in quality was relatively small, i.e., ,0.51. On the other hand, the VSPAC:DFVAT pair had a winning rate (average preference score) WR ij Because of the relatively large quality difference, only 21 ratings were assigned.

[0067] As shown in FIG. 8 and Table 1, for almost all pairs, the error bias EB was below the preset allowable bias TB value (0.06). Also, for some pairs, the error bias EB was even smaller than the allowable bias TB. This is because after the error bias EB was below the allowable bias TB (i.e., the minimum guaranteed accuracy was achieved), the evaluation continued until the total number of evaluations reached the upper limit value defined by the fixed budget. Also, no pairs were observed for which the error bias EB was excessively low. From the above, it can be said that this embodiment can prevent the number of evaluations for each pair from becoming uneven, and can prevent the number of evaluations for each pair from becoming excessive.

[0068] B. Variations: The technology disclosed in this specification is not limited to the above-described embodiments, and can be modified in various forms without departing from the spirit of the invention. For example, the following modifications are also possible.

[0069] The configurations of the information processing system 10 and the server device 100 in the above embodiment are merely examples and can be modified in various ways. The contents of the evaluation process in the above embodiment are merely examples and can be modified in various ways. For example, some steps of the evaluation process in the above embodiment may be omitted, or the contents of some steps may be changed.

[0070] In the above embodiment, the predicted value of the error bias EB is calculated based on the number of requests for subjective comparative evaluations, but the predicted value of the error bias EB may be calculated based on another index value instead of or in addition to the number of requests for subjective comparative evaluations.

[0071] In the above embodiment, the error bias EB is used as an index value representing the accuracy of the subjective comparative evaluation, but other index values ​​may be used.

[0072] In the above-described embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. [Explanation of symbols]

[0073] 10: Information processing system 100: Server device 110: Control unit 111: Evaluation processing unit 112: Upper limit setting unit 113: Precision setting unit 114: Evaluation target setting unit 115: Evaluation request unit 116: Evaluation status update unit 120: Memory unit 130: Display unit 140: Operation input unit 150: Interface unit 190: Bus 200: Terminal device

Claims

1. An information processing device that executes asynchronous and parallel subjective comparative evaluations by a plurality of evaluators, an evaluation target setting unit that dynamically sets an evaluation target pair to be a target of the subjective comparative evaluation based on a predicted value of the subjective comparative evaluation for each pair of evaluation targets; an evaluation requesting unit that requests the evaluator to perform the subjective comparative evaluation of the pair to be evaluated; an evaluation state update unit that dynamically updates an evaluation state of the pair to be evaluated based on an evaluation result by the evaluator; An information processing device comprising:

2. 2. The information processing device according to claim 1, The evaluation target setting unit sets, among each pair of the evaluation objects, a pair having the worst predicted value of accuracy of the subjective comparative evaluation as the evaluation target pair.

3. 3. The information processing device according to claim 1, The evaluation target setting unit calculates a predicted value of accuracy of the subjective comparative evaluation for each pair of the evaluation objects based on the number of requests for the subjective comparative evaluation.

4. 3. The information processing device according to claim 1, An information processing device, wherein the evaluation state update unit does not update the evaluation state for the pair to be evaluated if the evaluation state for the pair to be evaluated is in a predetermined convergence state at the time of receiving the evaluation result.

5. 5. The information processing device according to claim 4, The information processing device, wherein the evaluation state update unit, when receiving the evaluation result, if the evaluation state for the pair to be evaluated is in a predetermined convergence state, holds the evaluation result without discarding it.

6. 3. The information processing device according to claim 1, further comprising: an upper limit setting unit that sets an upper limit of the total number of evaluations in the subjective comparative evaluation; an accuracy setting unit that sets a minimum guaranteed accuracy of the subjective comparative evaluation that can be achieved by the upper limit number of evaluations; Equipped with In the case where the total number of evaluations has not reached the upper limit value at the time when the minimum guaranteed accuracy is achieved, the evaluation target setting unit sets the evaluation target pair based on the evaluation state for each pair of the evaluation objects until the total number of evaluations reaches the upper limit value.

7. 3. The information processing device according to claim 1, The evaluation target setting unit obtains an initial order of the objects to be evaluated, and sets the evaluation target pair based on the initial order.

8. An information processing method for performing asynchronous and parallel subjective comparative evaluations by a plurality of evaluators, comprising: A step of dynamically setting evaluation target pairs to be subjected to the subjective comparative evaluation based on a predicted value of the subjective comparative evaluation for each pair of evaluation targets; requesting the evaluator to perform the subjective comparative evaluation of the evaluation target pair; dynamically updating an evaluation state for the pair to be evaluated based on an evaluation result by the evaluator; An information processing method comprising:

9. A computer program for performing asynchronous and parallel subjective comparative evaluations by multiple evaluators, comprising: On the computer, A process of dynamically setting evaluation target pairs to be subjected to the subjective comparative evaluation based on a predicted value of the subjective comparative evaluation for each pair of evaluation targets; A process of requesting the evaluator to perform the subjective comparative evaluation of the evaluation target pair; a process of dynamically updating an evaluation state for the pair to be evaluated based on an evaluation result by the evaluator; A computer program that executes the following:

Images