Evaluation count derivation device

Abstract

This evaluation count derivation device is provided with: an input unit for receiving input of multiple evaluation values obtained by multiple subjective evaluations of a service; and an evaluation count derivation unit for deriving, on the basis of the variance of the multiple evaluation values, an optimal evaluation count that is the minimum count of evaluations required for obtaining a desired statistical reliability.

Description

Evaluation number derivation device

[0001] The present invention relates to a technique for acquiring evaluation data such as a user's Quality of Experience (QoE) value through subjective quality evaluation of a service.

[0002] Traditionally, various services have used the average of multiple users' perceived quality (MOS: Mean Opinion Score) as an indicator for quality design and control. However, with the recent advancement of ICT, the communication environment has become more diverse, and users' values ​​have become more diverse as well. This has created a demand for quality design and control that takes into account the differences in values ​​between users, such as "personalization" (providing optimal information and services according to user attributes, interests, hobbies, preferences, behavior, etc.).

[0003] Conventionally, methods for evaluating quality of experience (subjective evaluation methods) have been recommended by international standardization organizations such as ITU-T (video viewing: Non-Patent Document 1, audio listening: Non-Patent Document 2).

[0004] ITU-T Rec. P.910 : Subjective video quality assessment methods for multimedia applicationsITU-T Rec. P.800 : Methods for subjective determination of transmission quality

[0005] In conventional subjective evaluation methods, multiple evaluators evaluate each condition only once, and the average quality (MOS) of all evaluators is calculated. In order to accurately capture the impact of personalization, it is necessary to obtain accurate quality for each individual user.

[0006] However, because evaluations by the same evaluator under the same conditions involve a certain degree of variability, a single evaluation is insufficient from the standpoint of reliability. To reduce the effects of this variability, it is necessary to obtain results from each evaluator evaluating the same conditions multiple times. However, conventional subjective evaluation methods do not define a method for deriving the optimal number of evaluations.

[0007] If the number of evaluations is insufficient, the subjective evaluation results will be unreliable. On the other hand, if the number of evaluations is more than necessary, the subjective evaluation will become excessively costly and time-consuming.

[0008] That is, the prior art has a problem in that highly reliable evaluation data cannot be efficiently obtained in the subjective evaluation method.

[0009] The present invention has been made in view of the above points, and an object of the present invention is to provide a technique that enables highly reliable evaluation data to be efficiently obtained in a subjective evaluation method.

[0010] According to the disclosed technology, there is provided an evaluation count derivation device comprising: an input unit that inputs multiple evaluation values ​​obtained by multiple subjective evaluations of a service; and an evaluation count derivation unit that derives an optimal evaluation count, which is the minimum number of evaluations required to obtain desired statistical reliability, based on the variance of the multiple evaluation values.

[0011] The disclosed technology provides a technology that enables highly reliable evaluation data to be obtained efficiently in a subjective evaluation method.

[0012] It is a diagram showing an example of the configuration of an evaluation system. It is a diagram showing an example of the configuration of an evaluation count deriving device. It is a flowchart for explaining the operation of the evaluation system. It is a diagram showing an example of the hardware configuration of the device.

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0014] Hereinafter, a technology will be described that enables efficient acquisition of reliable and stable evaluation data in a system that acquires evaluation data such as a user's quality of experience value through subjective quality evaluation of a service.

[0015] (Outline of the embodiment) In the present embodiment, data on the evaluation of the service to be evaluated, in which evaluators have evaluated the same conditions a certain number of times, is acquired in advance by using an evaluation experiment device 1. Next, an evaluation count deriving device 2 utilizes the property that the evaluation results follow a probability distribution such as a normal distribution or a t-distribution, and calculates the number of evaluations at which the size of a predetermined confidence interval is equal to or smaller than a desired threshold from the variance of the evaluation values, thereby deriving the optimal number of evaluations as the minimum number of evaluations necessary to sufficiently guarantee reliability.

[0016] (System Configuration and Operation Overview) Fig. 1 shows an example of the overall configuration of an evaluation system according to this embodiment. As shown in Fig. 1, this evaluation system includes an evaluation experiment device 1 and an evaluation count derivation device 2. The operation of this evaluation system is outlined below.

[0017] The evaluation experiment device 1 collects evaluation result information consisting of a plurality of evaluation values, and inputs the collected evaluation result information to the evaluation count deriving device 2. The evaluation count deriving device 2 calculates (derives) an optimal evaluation count B from the input evaluation result information, and inputs the optimal evaluation count B to the evaluation experiment device 1. The functions and operations of each device will be explained in more detail below.

[0018] (Evaluation Experiment Apparatus 1) The evaluation experiment apparatus 1 allows an evaluator to experience a service to be evaluated (such as watching video or listening to audio). The evaluator evaluates the quality of the service and inputs the quality of experience as a numerical value (evaluation value) into the evaluation experiment apparatus 1.

[0019] The service to be evaluated is not limited to a specific service. Furthermore, the quality evaluation method is not limited to a specific method, and may be any method that allows an evaluator to determine an arbitrary evaluation value for the evaluation target, such as the methods defined in ITU-T Recommendations P.910 (Non-Patent Document 1) and P.800 (Non-Patent Document 1). The following description will be given taking the example of a service to be evaluated being video viewing.

[0020] The configuration of the evaluation experiment device 1 shown in Fig. 1 is a configuration assuming that the service to be evaluated is video viewing. Note that the specific numerical values ​​used to explain the calculations in the following explanation are merely examples for the purpose of explanation.

[0021] As shown in Fig. 1, the evaluation experiment device 1 has an evaluation target video DB 11 and an evaluation unit 12. The evaluation target video DB 11 stores a plurality of evaluation target videos. The evaluation unit 12 reads out the evaluation target videos from the evaluation target video DB 11, displays them, and asks an evaluator to evaluate the evaluation target videos. The evaluator inputs the evaluation results (evaluation values) to the evaluation unit 12.

[0022] More specifically, the evaluation unit 12 has each of the N evaluators select A evaluation videos from all the evaluation videos to be evaluated, and have them evaluate each of the A evaluation videos X times in random order. Each evaluator inputs the evaluation results for each evaluation target video to the evaluation unit 12. As a result, the evaluation unit 12 obtains N×A×X evaluation results, where N, A, and X are each an integer greater than or equal to 1.

[0023] The evaluation unit 12 of the evaluation experiment device 1 transmits N×A×X evaluation results as evaluation result information to the evaluation count deriving device 2, and then acquires the optimal evaluation count B from the evaluation count deriving device 2 and outputs the optimal evaluation count B. Note that the evaluation result information transmitted to the evaluation count deriving device 2 may use the results of past evaluation experiments.

[0024] In the evaluation experiment device 1 that has obtained the optimum number of evaluations B, for example, when a subjective quality evaluation of a certain video to be evaluated is carried out for a certain evaluator, the evaluation is carried out B times.

[0025] (Evaluation Number Deriving Device 2) Next, the function and operation of the evaluation number deriving device 2 will be described. The evaluation number deriving device 2 receives as input the N×A×X evaluation values ​​output from the evaluation experiment device 1 as evaluation result information, and calculates and outputs the optimal evaluation number B. Here, the optimal evaluation number B refers to the minimum number of evaluations required to obtain statistically reliable statistical values ​​for evaluation values ​​that vary for each evaluation trial. In other words, the optimal evaluation number is the minimum number of evaluations required to obtain the desired statistical reliability for the evaluation results.

[0026] Here, it is assumed that there is no difference in the variability of evaluations by the same evaluator due to differences in evaluators or videos, and by analyzing the combined evaluation values ​​of A videos evaluated by N people, the optimal number of evaluations B can be estimated using a small number of evaluations per person.

[0027] As shown in FIG. 1 , the evaluation count deriving device 2 includes an evaluation result information analysis unit 21 , a threshold determination unit 22 , an input unit 23 , and an output unit 24 .

[0028] The input unit 23 inputs the evaluation result information output from the evaluation experiment device 1, and outputs the evaluation result information (N×A×X evaluation values) to the evaluation result information analysis unit 21. The output unit 24 outputs the optimal number of evaluations B obtained by the threshold determination unit 22 to the evaluation experiment device 1.

[0029] The evaluation result information analysis unit 21 calculates, from the evaluation result information, a variance value v of the evaluation results of the evaluator i (1 to N) for the video j (1 to A). ｉｊ The variance value v is calculated and output. ｉｊ is calculated as follows:

[0030] Here, q ｉｊｋ means the kth (1 to X) evaluation value for video j by rater i, and Aveq ｉｊ means the average of X evaluation scores for video j by evaluator i. Σ is the sum of k=1 to X.

[0031] The threshold determination unit 22 determines the variance value v output from the evaluation result information analysis unit 21. ｉｊ (i=1 to N, j=1 to A) and a threshold T are input, and the optimum number of evaluations B is calculated and output. Here, the threshold T indicates the minimum necessary degree of variation. The threshold determination unit 22 may store the threshold T in advance in a storage device such as a memory. A method for calculating the optimum number of evaluations B by the threshold determination unit 22 will be described below.

[0032] The threshold determination unit 22 calculates v for each evaluation target video j for each evaluator i. ｉｊ Regarding the average value of the top 25% of the evaluators and the video to be evaluated, v ａｖｅFor example, if N=4 and A=10, the threshold determination unit 22 calculates the average value v of the top 25%. ａｖｅ Then, the average value of the top 25% of the 10 variance values ​​is calculated from 4×10=40 variance values.

[0033] The use of the top 25% data as the data used to calculate the average value is an example, and is not limited to the top 25%. In this evaluation, the variation of the evaluation values ​​within the evaluators is assumed to be a normal distribution (or t-distribution), and the variance is v ａｖｅ If we assume that the average of the Y evaluation values ​​is taken, the size of one side of the 95% confidence interval that is expected when dealing with the average of the Y evaluation values ​​is as follows:

[0034] Since this value Z needs to be less than T, the threshold value determination unit 22 selects the smallest integer that satisfies the following formula as the optimal number of evaluations B:

[0035] The value 1.96 in the above formula is a value for determining a 95% confidence interval, and 1.65 is used for a 90% confidence interval, and 2.58 is used for a 99% confidence interval. T is set to, for example, 5% of the range of the evaluation value (0.2 for a 5-level ACR (Absolute Category Rating)), but is not limited to this value.

[0036] The variance value v in the above formula ａｖｅ The square root of is the average value v ａｖｅ This corresponds to the standard deviation of the data (evaluation value) for which the optimum number of evaluations B is to be calculated. More specifically, it corresponds to the standard deviation of a normal distribution (or a t-distribution), which is a predetermined probability distribution that the data is assumed to follow. In this embodiment, as described above, the optimum number of evaluations B is derived using a formula for deriving a confidence interval of a probability distribution such as a normal distribution (or a t-distribution), but the present invention is not limited to this.

[0037] The optimal number of evaluations may be derived using a formula for deriving the standard deviation of a probability distribution such as a normal distribution (or a t-distribution), instead of the confidence interval. For example, ａｖｅ The optimum evaluation count B may be the evaluation count at which the square root of ( ) is equal to or smaller than a predetermined threshold value.

[0038] The evaluation count deriving device 2 may have the configuration shown in Fig. 2. The evaluation count deriving device 2 shown in Fig. 2 has an input unit 23, an output unit 24, and an evaluation count deriving unit 25. The evaluation count deriving unit 25 includes the function of the evaluation result information analysis unit 21 and the function of the threshold determination unit 22.

[0039] The input unit 23 inputs a plurality of evaluation values ​​obtained by subjective evaluation of a service multiple times. The evaluation count derivation unit 25 derives an optimal evaluation count, which is the minimum number of evaluations required to obtain a desired statistical reliability, based on the variance of the plurality of evaluation values. Specific examples of the derivation method are as described above in relation to the operations of the evaluation result information analysis unit 21 and the threshold determination unit 22.

[0040] (Processing Procedure) Hereinafter, the processing procedure of the evaluation system shown in FIG. 1 will be described with reference to the flowchart of FIG.

[0041] <S11> In S11, the evaluation experiment device 1 acquires X evaluation results for each of the A videos to be evaluated from each of the N evaluators, and transmits these to the evaluation result information analysis unit 21 as evaluation result information.

[0042] <S12> In S12, the evaluation result analysis unit 21 calculates the variance value v of the evaluation result information. ｉｊ (i = 1 to N, j = 1 to A) and calculate the variance value v ｉｊ (i=1 to N, j=1 to A) is sent to the threshold determination unit 22.

[0043] <S13> In S13, the threshold determination unit 22 calculates and outputs the optimum evaluation count B by the method already described.

[0044] (Hardware Configuration Example) Any of the devices described in this embodiment (evaluation experiment device 1, evaluation count derivation device 2) can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0045] That is, the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0046] Fig. 4 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 4 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B. The computer may further include a GPU.

[0047] The program that realizes the processing on the computer is provided by a recording medium 1001, such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0048] When an instruction to start the program is received, the memory device 1003 reads out and stores the program from the auxiliary storage device 1002. The CPU 1004 realizes functions related to the device in accordance with the program stored in the memory device 1003. Furthermore, when the computer is the evaluation count deriving device 2, for example, the memory device 1003 stores evaluation result information received from the evaluation experiment device 1. The CPU 1004 derives the optimal evaluation count B by reading out the evaluation result information from the memory device 1003.

[0049] The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to a program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0050] (Summary, Effects, etc. of the Embodiments) As described above, the technology described in the present embodiment provides a technology that enables highly reliable evaluation data to be efficiently obtained in a subjective evaluation method.

[0051] Furthermore, the technology described in this embodiment makes it possible to determine the optimal number of evaluations in advance, thereby enabling an experiment design that minimizes the scale of evaluation experiments while still obtaining highly reliable evaluation data.

[0052] The following additional notes are provided regarding the above-described embodiments.

[0053] <Additional Notes> (Additional Item 1) A rating count deriving device including a memory and at least one processor connected to the memory, wherein the processor inputs a plurality of rating values ​​obtained by subjectively rating a service multiple times, and derives an optimal rating count, which is the minimum number of ratings required to obtain a desired statistical reliability, based on a variance of the multiple rating values. (Additional Item 2) The rating count deriving device according to Additional Item 1, wherein the processor derives the optimal rating count based on the variance by utilizing a statistical property that the multiple rating values ​​follow a predetermined probability distribution. (Additional Item 3) The rating count deriving device according to Additional Item 2, wherein the processor derives the optimal rating count by utilizing a mathematical formula for deriving a standard deviation of the predetermined probability distribution or a mathematical formula for deriving a confidence interval of the predetermined probability distribution. (Additional Item 4) The rating count deriving device according to any one of Additional Items 1 to 3, wherein the multiple rating values ​​are multiple rating values ​​obtained by multiple subjective ratings of each of a plurality of videos to be rated by a plurality of raters. (Supplementary Item 5) A non-transitory storage medium storing a program for causing a computer to function as the evaluation count deriving device according to any one of Supplementary Items 1 to 4.

[0054] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0055] REFERENCE SIGNS LIST 1 Evaluation experiment device 11 Evaluation target video DB 12 Evaluation unit 2 Evaluation count derivation device 21 Evaluation result information analysis unit 22 Threshold determination unit 23 Input unit 24 Output unit 25 Evaluation count derivation unit 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. A rating count derivation device comprising: an input unit for inputting multiple evaluation values ​​obtained by multiple subjective evaluations of a service; and a rating count derivation unit for deriving an optimal rating count, which is the minimum number of evaluations required to obtain a desired statistical reliability, based on the variance of the multiple evaluation values.

2. The evaluation count deriving device according to claim 1, wherein the evaluation count deriving unit derives the optimal evaluation count based on the variance value by utilizing the statistical property that the multiple evaluation values ​​follow a predetermined probability distribution.

3. The evaluation count deriving device according to claim 2, wherein the evaluation count deriving unit derives the optimal evaluation count by utilizing a formula for deriving a standard deviation of the specified probability distribution or a formula for deriving a confidence interval of the specified probability distribution.

4. The evaluation count deriving device according to any one of claims 1 to 3, wherein the multiple evaluation values ​​are multiple evaluation values ​​obtained by multiple subjective evaluations by multiple evaluators for each of multiple evaluation target videos.

Images