Calculation device and calculation method

Abstract

In subjective evaluation experiments, the number of evaluations required to achieve a target confidence interval can be calculated for each evaluator and each subject being evaluated. [Solution] The calculation device is a calculation device for calculating the number of times an evaluator evaluates an evaluation target in a subjective evaluation experiment, and comprises an estimation unit that estimates the standard deviation of the evaluator's score for the evaluation target based on the evaluator's evaluation data, and a calculation unit that calculates and outputs the number of times the evaluator evaluates necessary to achieve a target confidence interval based on the standard deviation estimated by the estimation unit.

Description

Technical Field

[0001] The present invention relates to a calculation device and a calculation method.

Background Art

[0002] For example, in various services such as voice or video, the average (MOS: Mean Opinion Score) of the perceived quality of multiple users is known as an index for quality design and control. However, since preferences and feelings vary from person to person, it becomes possible to satisfy the requirements (intentions) of service providers and users by providing services tailored to each individual's characteristics, and it is expected to enhance the retention effect.

[0003] In Non-Patent Document 1, a study considering individual differences has been conducted in a subjective evaluation experiment targeting encoding degradation in a video distribution service.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In order to accurately grasp individual evaluation characteristics, it is desirable to perform multiple evaluations under the same conditions rather than only one evaluation for one evaluation target as in the past. In the technique disclosed in Non-Patent Document 1, the number of repetitions under the same conditions is uniformly set to 16 times.

[0006] However, some evaluators can consistently perform evaluations with fewer trials, while others cannot. Similarly, some evaluation subjects can be evaluated relatively consistently, while others cannot. Furthermore, due to factors such as the evaluator's burden and cost, there is a limit to the number of evaluations an evaluator can perform. Therefore, if the number of evaluations for an evaluation subject is high, the number of evaluation subjects (the number of evaluable conditions) decreases. Consequently, it is important to determine the number of evaluations required to achieve the target confidence interval for each evaluator and evaluation subject.

[0007] Embodiments of the present invention have been made in view of the above problems, and enable the calculation of the number of evaluations required to achieve a target confidence interval for each evaluator and evaluation subject in a subjective evaluation experiment. [Means for solving the problem]

[0008] To solve the above problems, the calculation device according to an embodiment of the present invention is a calculation device for calculating the number of times an evaluator evaluates an evaluation target in a subjective evaluation experiment, and comprises: an estimation unit that estimates the standard deviation of the evaluator's score for the evaluation target based on the evaluator's evaluation data, and a calculation unit that calculates and outputs the number of times the evaluator evaluates necessary to achieve a target confidence interval based on the standard deviation estimated by the estimation unit. [Effects of the Invention]

[0009] According to embodiments of the present invention, in a subjective evaluation experiment, it becomes possible to calculate the number of evaluations required to achieve a target confidence interval for each evaluator and evaluation subject. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of the configuration of the calculation device according to this embodiment. [Figure 2] Figure (1) illustrates the standard deviation database according to this embodiment. [Figure 3]It is a diagram (2) for explaining the standard deviation DB according to the present embodiment. [Figure 4] It is a flowchart showing an example of a calculation process of the number of evaluations according to the present embodiment. [Figure 5] It is a flowchart showing an example of an update process of the standard deviation DB according to the present embodiment. [Figure 6] It is a diagram for explaining an estimation process of the standard deviation according to Example 1. [Figure 7] It is a diagram showing an image of grouping evaluators according to Example 2. [Figure 8] It is a diagram showing an example of the hardware configuration of a computer. [Figure 9] It is a diagram (1) for explaining the effects of the present embodiment. [Figure 10] It is a diagram (2) for explaining the effects of the present embodiment. [Figure 11] It is a diagram (3) for explaining the effects of the present embodiment.

MODE FOR CARRYING OUT THE INVENTION

[0011] Hereinafter, embodiments of the present invention (the present embodiment) will be described with reference to the drawings. The embodiments described below are merely examples, and the embodiments to which the present invention is applied are not limited to the following embodiments.

[0012] (Definition of Terms) In the present embodiment, the terms used in the following description are defined as follows. MOS: Average of scores of evaluations by multiple evaluators SOS: Standard deviation of scores of evaluations by multiple evaluators μ: Average of scores of multiple evaluations by one evaluator σ: Standard deviation of scores of multiple evaluations by one evaluator CI: Confidence interval

[0013] [[ID=5l]] <Configuration of Calculation Device> FIG. 1 is a diagram showing a configuration example of the calculation device according to the present embodiment. The calculation device 100 is an information processing device having a computer configuration or a system including a plurality of computers. The calculation device 100 realizes each functional configuration as shown in FIG. 1, for example, by executing a predetermined program on one or more computers included in the calculation device 100. In the example of FIG. 1, the calculation device 100 has each functional configuration such as an input unit 101, an estimation unit 102, a calculation unit 103, a storage unit 104, an update unit 105, and a standard deviation DB 110. Note that at least a part of the above functional configurations may be realized by hardware.

[0014] The input unit 101 executes an input process of inputting input data such as evaluation data of a target evaluator, identification information for identifying an evaluation target, and a target CI which is a target confidence interval, for example, to the estimation unit 102. The input data input by the input unit 101 may be data input from outside the calculation device 100, or may be data stored in advance in the storage unit 104 or the like.

[0015] The estimation unit 102 executes an estimation process of estimating the standard deviation of the score given by the evaluator to the evaluation target based on the evaluation data of the evaluator included in the input data. For example, the estimation unit 102 refers to the standard deviation DB 110 of the evaluation target group and estimates the standard deviation of the score given by the evaluator to the evaluation target based on the relationship between the standard deviation of the score for the evaluation target and the standard deviation of the scores for other evaluation targets.

[0016] Alternatively, the estimation unit 102 refers to the standard deviation DB 110 of the evaluation target group and estimates the standard deviation of the score given by the evaluator to the evaluation target based on the relationship between the standard deviation of the scores in the group including the evaluator or the evaluator and the average of the scores. Specific processing contents will be described later by exemplifying a plurality of embodiments.

[0017] The calculation unit 103 calculates and outputs the number of evaluations by the evaluator required to achieve the target confidence interval (target CI) based on the standard deviation estimated by the estimation unit 102.

[0018] The memory unit 104 is implemented, for example, by a program executed by the computer provided in the computing device 100, and a storage device provided by the computer, and stores various data, information, and programs.

[0019] The standard deviation DB110 is a database that stores the relationship (regression line) between two factors, along with its coefficient of determination, slope (a), and intercept (b) for each pre-calculated combination of quality conditions. The standard deviation DB110 may be located outside the calculation device 100.

[0020] The update unit 105 performs an update process to update the standard deviation DB 110 based on the newly entered evaluation data. The update unit 105 may be located outside the calculation device 100.

[0021] With the above configuration, the calculation device 100 starts operation using the evaluation data of the target evaluator, the identification information of the subject to be evaluated, and the target CI as input data, and outputs the number of evaluations for the target evaluator by referring to the standard deviation DB 110 of the subject group to be evaluated.

[0022] Furthermore, the update unit 105 receives new evaluation data as input and starts its operation, and stores the standard deviation of the scores for each evaluator and evaluation subject in the standard deviation DB 110.

[0023] In the following explanation, examples involving the evaluation of video distribution services may be given, but this does not limit the services to be evaluated in this invention.

[0024] (Regarding the standard deviation database) Figures 2 and 3 illustrate the standard deviation database according to this embodiment. For example, in an evaluation experiment targeting a video distribution service, there are five quality conditions: PVS (Processed Video Sequence) 1-5 (PVS1: high quality to PVS5: low quality). Furthermore, when n evaluators perform m evaluations (n ​​and m are integers greater than or equal to 1), the regression line of σ and the coefficient of determination are calculated in advance for each combination of PVS.

[0025] In Graph 200 of Figure 2, the horizontal axis represents the standard deviation σ of each evaluator's m evaluations for each of PVS2-5 (PVS2 to PVS5). There are n points for each PVS. The vertical axis represents the standard deviation σ of each evaluator for PVS1. For example, as shown in Graph 200, the relationship (regression line) between the standard deviation σ of PVS1 and the standard deviations σ of PVS2-5, and their coefficient of determination are derived in advance and registered in the standard deviation DB110 as shown in Figure 3.

[0026] Here, the coefficient of determination represents the degree of goodness (or fit) of the regression line, and is, for example, the square of the correlation coefficient. In this embodiment, it is assumed that in combinations of PVS with a larger coefficient of determination (in the example of Figure 3, the combination of PVS1 and PVS3), the quality degradation of both is similar (for example, both only show encoding degradation).

[0027] Furthermore, the combination of PVS is not limited to PVS1 and PVS2-5; other combinations such as PVS2 and PVS1, 3, and 4 are also acceptable. Additionally, the relationship between the two does not have to be linear, and may be determined by multiple regression analysis based on multiple PVSs.

[0028] For example, if evaluation data from a new evaluator is added, the relevant evaluation data is appropriately input into the update unit 105 to update the standard deviation DB 110. Increasing the sample size enables more reliable interval estimation.

[0029] <Processing flow> Next, we will explain the processing flow of the method for calculating the number of evaluations according to this embodiment.

[0030] (Calculation process for the number of evaluations) Figure 4 is a flowchart illustrating an example of the evaluation count calculation process according to this embodiment. This process shows an overview of the evaluation count calculation process performed by the calculation device 100 described in Figure 1.

[0031] In step S401, the input unit 101 inputs input data such as the evaluation data of the target evaluator, identification information of the subject to evaluation, and target CI to the estimation unit 102. For example, the input unit 101 inputs input data received from an external source to the estimation unit 102. Alternatively, the input unit 101 may input input data previously stored in the storage unit 104 to the estimation unit 102 based on an operation by an operator or the like.

[0032] In step S402, the estimation unit 102, based on the input evaluation data of the evaluators, refers to the standard deviation DB 110 and estimates the standard deviation by the evaluators for the evaluation target corresponding to the input identification information of the evaluation target. A specific example of the estimation method will be described later.

[0033] In step S403, the calculation unit 103 calculates and outputs the number of evaluators' evaluations required to achieve the input target CI, based on the standard deviation calculated by the estimation unit 102. The output destination of the calculated number of evaluations is not limited to outside the calculation device 100; for example, it may be output (stored) to the storage unit 104, or output (displayed) to a display device of the computer equipped with the calculation device 100.

[0034] (Standard deviation database update process) Figure 5 is a flowchart showing an example of the standard deviation DB update process according to this embodiment. This process shows, for example, an overview of the evaluation count calculation process performed by the update unit 105 described in Figure 1.

[0035] In step S501, when the update unit 105 receives the input of evaluation data, it executes the process in step S502.

[0036] In step S502, the update unit 105 registers the standard deviation of the scores for each evaluator and evaluation subject in the standard deviation DB 110 from the evaluation data. For example, the update unit 105 adds the input evaluation data to the existing evaluation data to create a graph 200 as shown in Figure 2, derives the relationship (regression line) between the standard deviation σ of PVS1 and the standard deviation σ of PVS2-5 and its coefficient of determination, and registers it in the standard deviation DB 110 as shown in Figure 3.

[0037] <Examples> Next, we will explain the process for calculating the number of evaluations, as described in Figure 4, by illustrating it with a specific example.

[0038] [Example 1] Figure 6 is a diagram illustrating the estimation process related to Example 1. For example, in the video distribution service described in Figure 2, an example is given of calculating the number of evaluations required to make the 95% confidence interval of the average score of a certain evaluator X for PVS1 0.3. It is assumed that the calculation device 100 already has evaluation data for PVS2-5 (PVS2 to PVS5).

[0039] Furthermore, the input unit 101 shall input the following input data to the estimation unit 102. • Evaluation data of the target evaluators: PVS2-5 evaluation data • Identification information to be evaluated: PVS1 ·Target CI:0.3

[0040] In this case, as shown in Figure 6, the coefficient of determination R for PVS1 vs. PVS3 is R 2 Since the value is 0.99, the estimation unit 102 can estimate the standard deviation of PVS1 with high accuracy from the standard deviation of PVS3. For example, as shown in Figure 6, the estimation unit 102 can estimate the estimated value 602 of evaluator X's PVS1 from the standard deviation 601 of evaluator X's PVS3.

[0041] Next, the calculation unit 103 uses the input target CI (0.3) and the estimated standard deviation σ (estimated standard deviation of PVS1 for evaluator X) to calculate the required number of evaluations N, for example, using the following formula (1).

[0042]

number

[0043] [Example 2] Example 2 describes an example in which the number of evaluations is calculated based on the relationship between the standard deviation SOS and the average score MOS. Except for this point, the configuration of the calculation device 100 and the processing procedure in Example 2 are the same as in Example 1.

[0044] Here, we will explain the case of a 5-point rating scale from 1 to 5 as a specific example. In this case, it is known that the standard deviation SOS can be approximated by the following equation (2) using the average rating MOS.

[0045]

number

[0046] Furthermore, the following approximation formula (3) also holds for the standard deviation σ among evaluators and the average score μ.

[0047]

number

[0048] This parameter C can be calculated using the least squares method or similar for a series of average scores μ and standard deviation σ. The average scores μ and standard deviation σ used to calculate parameter C may be obtained by evaluating all videos a certain number of times (e.g., 8 times), or by evaluating a portion of the videos (e.g., about half) a certain number of times (e.g., 16 times).

[0049] Figure 7 shows an image of the grouping of evaluators in Example 2. This graph 700 shows an image of grouping evaluators by parameter C, with the horizontal axis being the average score μ and the vertical axis being the standard deviation σ. In the example of Figure 7, graph 700 shows an image of grouping into three groups: group G1-1 with a "large" value of parameter C, group G1-2 with a "medium" value of parameter C, and group G1-3 with a "small" value of parameter C.

[0050] Such grouping can be done, for example, by percentile values, or by using Ward's method. Furthermore, the average value of parameter C of the evaluators belonging to each group can be used as the parameter C for that group.

[0051] The estimation unit 102 estimates the standard deviation σ using equation (3) described above, with respect to the object for which the number of evaluations required is to be calculated, using the average score μ obtained from a certain number of evaluations and the parameter C of the group to which the evaluators of the object belong. Alternatively, the estimation unit 102 may assume μ = MOS and estimate the standard deviation σ using equation (3) described above, with respect to the average score μ and the parameter C of the group to which the evaluators of the object belong.

[0052] The calculation unit 103 uses the standard deviation σ estimated by the estimation unit 102 to calculate the number of evaluations N for the input target CI using the aforementioned formula (1). Note that formula (1) is just one example, and the calculation unit 103 may use any other formula that is appropriate for calculating the number of evaluations N.

[0053] In Example 2, the standard deviation DB110 has the standard deviation σ of the scores for each evaluator and evaluation subject, as well as the average score μ, pre-registered. The update unit 105 then registers the standard deviation σ of the scores for each evaluator and evaluation subject, as well as the average score μ, into the standard deviation DB110 from the input evaluation data.

[0054] [Example 3] Example 3 describes an example where the method for calculating parameter C differs from that of Example 2. In Example 2, the evaluators were divided into groups and parameter C was calculated for each group, but in Example 3, parameter C is calculated for each individual evaluator. Except for this point, the configuration of the calculation device 100 and the processing procedure in Example 3 are the same as in Example 2.

[0055] Similar to Example 2, the parameter C in equation (3) described above can be calculated using the least squares method or the like for the series of mean score μ and standard deviation σ. The mean score μ and standard deviation σ for calculating parameter C may be obtained by evaluating all videos a certain number of times (e.g., 8 times), or by evaluating a portion of all videos (e.g., about half) a certain number of times (e.g., 16 times).

[0056] As a result, the estimation unit 102 can estimate the standard deviation σ using the above-mentioned equation (3) with respect to the target for which the number of evaluations to be calculated is the average score μ obtained after a certain number of evaluations and the parameter C calculated for the evaluator of the target. Alternatively, the estimation unit 102 may assume μ = MOS and estimate the standard deviation σ using the above-mentioned equation (3) with respect to the average score μ and the parameter C calculated for the evaluator of the target.

[0057] Furthermore, the calculation unit 103 uses the standard deviation σ estimated by the estimation unit 102 to calculate the number of evaluations N for the input target CI using the aforementioned formula (1). Note that formula (1) is just one example, and the calculation unit 103 may use any other formula that is appropriate for calculating the number of evaluations N.

[0058] [Example 4] Example 4 describes an example in which the calculation device 100 does not include the update unit 105, as in Examples 1 to 3. In this case, the calculation device 100 refers to the standard deviation DB 110 of the evaluation target group which is held in advance, and calculates the number of evaluations N to achieve the target CI using one of the methods in Examples 1 to 3.

[0059] Thus, the calculation device 100 according to this embodiment does not necessarily have to have an update unit 105.

[0060] <Hardware Configuration> The calculation device 100 according to this embodiment has a hardware configuration of a computer 800 as shown in Figure 8, for example. The calculation device 100 may be composed of multiple computers 800. Furthermore, the computers constituting the calculation device 100 are not limited to physical machines, but may be, for example, virtual machines on the cloud.

[0061] Figure 8 shows an example of a computer hardware configuration. In the example in Figure 8, the computer 800 includes a drive device 801, an auxiliary storage device 802, a memory device 803, a CPU 804, an interface device 805, a display device 806, an input device 807, and an output device 1308, all of which are interconnected by bus B. The computer 800 may also include other processors such as a GPU (Graphics Processing Unit).

[0062] The program that enables processing on the computer 800 is provided on a recording medium 811, such as a CD-ROM or memory card. When the recording medium 811 containing the program is set in the drive device 801, the program is installed from the recording medium 811 to the auxiliary storage device 802 via the drive device 801. However, the program does not necessarily have to be installed from the recording medium 811; it may also be downloaded from another computer via a network. The auxiliary storage device 802 is a storage device that stores the installed program as well as necessary files and data.

[0063] The memory device 803 reads and stores a program from the auxiliary storage device 802 when a program startup command is received. The CPU 804 implements the functions related to the calculation device 100 according to the program stored in the memory device 803. The interface device 805 is used as an interface for connecting to a communication network, etc. The display device 806 displays a GUI (Graphical User Interface) etc., generated by a program. The input device 807 consists of a keyboard, mouse, buttons, and / or touch panel etc., and is used to input various operation commands. The output device 1308 outputs the calculation results.

[0064] Note that CPU804 may be other processors such as a DSP (Digital Signal Processor), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array).

[0065] <Effects of the Embodiment> According to this embodiment, in a subjective evaluation experiment, the number of evaluations required to achieve the target confidence interval can be calculated for each evaluator and each subject being evaluated. As a result, compared to a situation without the calculation device 100 according to this embodiment, it is possible to reduce evaluation costs or increase the number of subjects being evaluated.

[0066] The following three examples illustrate the specific effects.

[0067] (Example 1) As shown in Figure 9, in an evaluation experiment of a certain service, two evaluators (evaluator x, evaluator y) will perform the evaluation.

[0068] The calculation device 100 according to this embodiment calculates the number of evaluations N so that, for example, if evaluator x's standard deviation σ is large, the same object is evaluated more times. Also, if, for example, evaluator y's standard deviation is small, the calculation device 100 calculates the number of evaluations N so that the same object is evaluated fewer times. By adjusting the number of evaluations N for each evaluator in this way, it becomes possible to collect evaluation data efficiently without waste. For example, evaluator x can evaluate the same object many times, while evaluator y can handle many objects with fewer evaluations, thus enabling optimization.

[0069] (Example 2) As shown in Figure 10, in an evaluation experiment of a certain service, the quality conditions are set to PVS1 to PVS100, and the evaluation is conducted by three groups, A, B, and C. The number of evaluations performed by each group will be explained below.

[0070] On the first day of the experiment, Group A will evaluate PVS1-50 (PVS1-PVS50), Group B will evaluate PVS25-74, and Group C will evaluate PVS1-24 and PVS75-100. On the first day, all PVS will be evaluated a predetermined number of times (e.g., 16 times). The relationships obtained on the first day (e.g., coefficient of determination, slope, intercept, etc. for PVS1 vs. PVS) will be stored in the standard deviation DB110.

[0071] On the second day of the experiment, the calculation device 100 calculates the number of evaluations N that each group will perform by referring to the standard deviation DB 110. For example, group A performs the following processing for the evaluations of PVS51-74 and PVS75-100 out of PVS1-100. • PVS51-74: Based on the relationships established from Group B (PVS25-74) on Day 1, the number of evaluations N required to achieve the target CI is calculated using the values ​​for Group S (PVS25-50) obtained on Day 1. • PVS75-100: Based on the relationships established from Group C (PVS1-24, PVS75-100) on Day 1, the number of evaluations N required to achieve the target CI is calculated using the values ​​of Group A (PVS1-24) obtained on Day 1.

[0072] In this way, the calculation device 100 can optimize the number of evaluations N on the second day by utilizing the data from the first day.

[0073] (Example 3) As shown in Figure 11, the calculation device 100 estimates the standard deviation σ for each quality of the item being evaluated using parameter C for the evaluator or the group to which the evaluator belongs, and calculates the number of evaluations for the estimated standard deviation σ and the target CI. This allows the calculation device 100 to optimally set the number of evaluations to the minimum necessary, according to the item being evaluated.

[0074] <Summary of Embodiments> This specification discloses at least the following calculation devices, calculation methods, and programs. (Section 1) A calculation device for calculating the number of times an evaluator has evaluated an object in a subjective evaluation experiment, An estimation unit that estimates the standard deviation of the scores given by the evaluators to the subject of evaluation based on the evaluators' evaluation data, Based on the standard deviation estimated by the estimation unit, the calculation unit calculates and outputs the number of evaluations by the evaluator required to achieve the target confidence interval. A calculating device having the following features. (Section 2) The calculation device according to paragraph 1, wherein the estimation unit estimates the standard deviation of the scores given by the evaluator for the subject to evaluation based on the relationship between the standard deviation of the scores given for the subject to evaluation and the standard deviation of the scores given for other subjects to evaluation. (Section 3) The calculation device according to paragraph 1, wherein the estimation unit estimates the standard deviation of the scores given by the evaluators for the subject of evaluation based on the relationship between the standard deviation of the scores given by the evaluators or a group including the evaluators and the mean of the scores. (Section 4) The calculation device described in paragraph 1, which takes the evaluation data already evaluated by the evaluator and the target CI as input and estimates the number of evaluations required to achieve the target CI. (Section 5) The calculation device according to paragraph 1, which takes evaluation data as input and has an update unit that registers evaluation data, or statistical quantities including the standard deviation and / or mean of the scores calculated from the evaluation data, for each evaluator and evaluation subject into a database. (Section 6) In a subjective evaluation experiment, a computer calculates the number of times an evaluator evaluates an object. An estimation process is performed to estimate the standard deviation of the scores given by the evaluators to the subject of evaluation, based on the evaluators' evaluation data. Based on the standard deviation estimated in the estimation process, a calculation process is performed to calculate and output the number of evaluations by the evaluator required to achieve the target confidence interval. The calculation method used to perform this calculation. (Section 7) A program that causes a computer to execute the calculation method described in paragraph 6, or a storage medium that stores such a program.

[0075] Although this embodiment has been described above, the present invention is not limited to this specific embodiment, and various modifications and changes are possible within the scope of the gist of the invention as described in the claims. [Explanation of Symbols]

[0076] 100 Calculation device 101 Input Section 102 Estimation part 103 Calculation Unit 104 Storage section 105 Update section 110 Standard deviation DB 800 Computers

Claims

1. A calculation device for calculating the number of times an evaluator has evaluated an object in a subjective evaluation experiment, An estimation unit that estimates the standard deviation of the scores given by the evaluators to the subject of evaluation based on the evaluators' evaluation data, Based on the standard deviation estimated by the estimation unit, the calculation unit calculates and outputs the number of evaluations by the evaluator required to achieve the target confidence interval. A calculating device having the following features.

2. The calculation device according to claim 1, wherein the estimation unit estimates the standard deviation of the scores given by the evaluator to the subject of evaluation based on the relationship between the standard deviation of the scores given to the subject of evaluation and the standard deviation of the scores given to other subjects of evaluation.

3. The calculation device according to claim 1, wherein the estimation unit estimates the standard deviation of the scores given by the evaluators for the subject to be evaluated, based on the relationship between the standard deviation of the scores given by the evaluators or a group including the evaluators and the mean of the scores.

4. In a subjective evaluation experiment, a computer calculates the number of times an evaluator evaluates an object. An estimation process is performed to estimate the standard deviation of the scores given by the evaluators to the subject of evaluation, based on the evaluators' evaluation data. Based on the standard deviation estimated in the estimation process, a calculation process is performed to calculate and output the number of evaluations by the evaluator required to achieve the target confidence interval. The calculation method used to perform this calculation.

Images