Evaluation count derivation device

Abstract

This evaluation count derivation device is provided with: an input unit to which a plurality of evaluation values obtained through a plurality of subjective evaluations of a service is input; and an evaluation count derivation unit that derives an optimal evaluation count, which is the minimum number of evaluations required to obtain a desired statistical reliability, on the basis of the plurality of evaluation values.

Description

Evaluation number derivation device

[0001] The present invention relates to a technique for acquiring evaluation data such as a user's Quality of Experience (QoE) value through subjective quality evaluation of a service.

[0002] Traditionally, various services have used the average of multiple users' perceived quality (MOS: Mean Opinion Score) as an indicator for quality design and control. However, with the recent advancement of ICT, the communication environment has become more diverse, and users' values ​​have become more diverse as well. This has created a demand for quality design and control that takes into account the differences in values ​​between users, such as "personalization" (providing optimal information and services according to user attributes, interests, hobbies, preferences, behavior, etc.).

[0003] Conventionally, methods for evaluating quality of experience (subjective evaluation methods) have been recommended by international standardization organizations such as ITU-T (video viewing: Non-Patent Document 1, audio listening: Non-Patent Document 2).

[0004] ITU-T Rec. P.910 : Subjective video quality assessment methods for multimedia applicationsITU-T Rec. P.800 : Methods for subjective determination of transmission quality

[0005] In conventional subjective evaluation methods, multiple evaluators evaluate each condition only once, and the average quality (MOS) of all evaluators is calculated. In order to accurately capture the impact of personalization, it is necessary to obtain accurate quality for each individual user.

[0006] However, because evaluations by the same evaluator under the same conditions involve a certain degree of variability, a single evaluation is insufficient from the standpoint of reliability. To reduce the effects of this variability, it is necessary to obtain results from each evaluator performing evaluations under the same conditions multiple times. However, conventional subjective evaluation methods do not define a method for deriving the optimal number of evaluations. As a result, there are problems such as an insufficient number of evaluations, resulting in unreliable subjective quality evaluation results, or conducting more evaluations than necessary, resulting in excessive subjective evaluation costs and time.

[0007] That is, the prior art has a problem in that highly reliable evaluation data cannot be efficiently obtained in the subjective evaluation method.

[0008] The present invention has been made in view of the above points, and an object of the present invention is to provide a technique that enables highly reliable evaluation data to be efficiently obtained in a subjective evaluation method.

[0009] According to the disclosed technology, there is provided an evaluation count derivation device comprising: an input unit that inputs multiple evaluation values ​​obtained by multiple subjective evaluations of a service; and an evaluation count derivation unit that derives an optimal evaluation count, which is the minimum number of evaluations required to obtain the desired statistical reliability, based on the multiple evaluation values.

[0010] The disclosed technology provides a technology that enables highly reliable evaluation data to be obtained efficiently in a subjective evaluation method.

[0011] It is a diagram showing an example of the configuration of an evaluation system. It is a diagram showing an example of the configuration of an evaluation count deriving device. It is a flowchart for explaining the operation of the evaluation system. It is a diagram showing an example of the hardware configuration of the device.

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0013] Hereinafter, a technology will be described that enables efficient acquisition of reliable and stable evaluation data in a system that acquires evaluation data such as a user's quality of experience value through subjective quality evaluation of a service.

[0014] (Outline of the embodiment) In this embodiment, an evaluation experiment device 1 is used in which each evaluator evaluates the same conditions multiple times for the service to be evaluated. The evaluation results are input sequentially to an evaluation count deriving device 2 until the degree of variation falls within a threshold, and the evaluation count deriving device 2 derives the number of evaluations when the degree of variation falls within the threshold as the minimum number of evaluations necessary to sufficiently guarantee reliability. Subjective quality evaluation is performed based on this number of evaluations.

[0015] (System Configuration and Operation Overview) Fig. 1 shows an example of the overall configuration of an evaluation system according to this embodiment. As shown in Fig. 1, this evaluation system includes an evaluation experiment device 1 and an evaluation count derivation device 2. The operation of this evaluation system is outlined below.

[0016] The evaluation experiment device 1 collects evaluation result information consisting of a plurality of evaluation values ​​and inputs the collected evaluation result information to the evaluation count deriving device 2. The evaluation count deriving device 2 calculates (derives) the optimal evaluation count from the input evaluation result information, but if a sufficient number of pieces of evaluation result information is not obtained to calculate the optimal evaluation count, an additional evaluation request is input to the evaluation experiment device 1. In response to this request, the evaluation experiment device 1 collects additional evaluation result information and inputs the additional evaluation result information to the evaluation count deriving device 2. The evaluation experiment device 1 calculates the optimal evaluation count based on the evaluation result information obtained so far. The functions and operations of each device will be explained in more detail below.

[0017] (Evaluation Experiment Apparatus 1) The evaluation experiment apparatus 1 allows an evaluator to experience a service to be evaluated (such as watching video or listening to audio). The evaluator evaluates the quality of the service and inputs the quality of experience as a numerical value (evaluation value) into the evaluation experiment apparatus 1.

[0018] The service to be evaluated is not limited to a specific service. Furthermore, the quality evaluation method is not limited to a specific method, and may be any method that allows an evaluator to determine an arbitrary evaluation value for the evaluation target, such as the methods defined in ITU-T Recommendations P.910 (Non-Patent Document 1) and P.800 (Non-Patent Document 1). The following description will be given taking the example of a service to be evaluated being video viewing.

[0019] The configuration of the evaluation experiment device 1 shown in Fig. 1 is a configuration assuming that the service to be evaluated is video viewing. Note that the specific numerical values ​​used to explain A, X, etc. in the following explanation are merely examples for the purpose of explanation.

[0020] As shown in FIG. 1, the evaluation experiment device 1 includes a dummy image DB 11 , an evaluation target image DB 12 , and an evaluation unit 13 .

[0021] The evaluation target video DB 12 stores A evaluation target videos that are evaluation targets. The dummy video DB 11 stores one or more dummy videos.

[0022] The evaluation unit 13 reads out and displays an evaluation target video from the evaluation target video DB 12, or reads out and displays a dummy video from the dummy video DB 11.

[0023] More specifically, the evaluation unit 13 displays the dummy video so that the evaluator evaluates one dummy video every time A / X evaluation target videos are randomly evaluated. For example, if A=30 and X=3, the evaluator evaluates one dummy video every time 30 / 3=10 evaluation target videos are randomly evaluated. The evaluator inputs the evaluation results of each evaluation target video to the evaluation unit 13.

[0024] As a result, when the evaluators have evaluated all A videos to be evaluated once each, the evaluation unit 13 obtains X evaluation results for the dummy videos. The evaluation unit 13 of the evaluation experiment device 1 transmits the X evaluation results for the dummy videos to the evaluation count deriving device 2.

[0025] The evaluation unit 13 of the evaluation experiment device 1 transmits X evaluation results for the dummy video to the evaluation count derivation device 2, and then obtains either an additional evaluation request or an optimal evaluation count B from the evaluation count derivation device 2. When the evaluation unit 13 obtains the additional evaluation request, it performs evaluations of A evaluation target videos and X evaluations of the dummy video in the same procedure as above, and transmits these evaluation results again to the evaluation count derivation device 2. When the evaluation unit 13 obtains the optimal evaluation count B, it has the evaluators evaluate each evaluation target video a number of times calculated by subtracting the number of evaluations of the evaluation target video up to now from B, and completes the subjective evaluation.

[0026] (Evaluation count derivation device 2) Next, the function and operation of the evaluation count derivation device 2 will be described. The evaluation count derivation device 2 receives X evaluation results for dummy videos acquired by the evaluation experiment device 1 as input, and outputs an optimal evaluation count. Here, the optimal evaluation count refers to the minimum number of evaluations required to obtain statistically reliable statistical values ​​for evaluation values ​​that vary for each evaluation trial. In other words, the optimal evaluation count is the minimum number of evaluations required to obtain the desired statistical reliability for the evaluation results.

[0027] As shown in FIG. 1, the evaluation count deriving device 2 includes an evaluation result information analysis unit 21, a threshold determination unit 22, an input unit 23, and an output unit 24.

[0028] The input unit 23 inputs X evaluation results for the dummy video acquired by the evaluation experiment device 1, and outputs the X evaluation results to the evaluation result information analysis unit 21. The evaluation result information analysis unit 21 calculates a variance v of the evaluation results using the X evaluation results. When the evaluation result information analysis unit 21 acquires multiple "X evaluation results," it calculates the variance v for all of these evaluation results. For example, when two "X evaluation results" are acquired, it calculates the variance v for 2×X evaluation values. The variance v is calculated using the following formula:

[0029] Here, q ｉ is the i-th evaluation value, and q ｑｖｅis the average value of all evaluation values, and N is the number of evaluation values ​​in the evaluation result information, that is, the cumulative number of evaluation results for the dummy video. For example, if "X evaluation results" are obtained twice, N = 2 × X. Σ is the sum of i = 1 to N.

[0030] The evaluation result information analysis unit 21 outputs the variance value v and the number N of evaluation values ​​used in calculating the variance value v to the threshold determination unit 22. The threshold determination unit 22 receives the variance value v and the number N of evaluation values ​​output from the evaluation result information analysis unit 21 as well as a threshold T, and calculates the optimal number of evaluations. Here, the threshold T indicates the minimum necessary degree of variation. The threshold determination unit 22 may store the threshold T in advance in a storage device such as a memory. A method for calculating the optimal number of evaluations by the threshold determination unit 22 will be described below.

[0031] The evaluation results can be assumed to follow a normal distribution, and the standard error s of a probability distribution such as a normal distribution or a t-distribution can be expressed by the following formula using the number of evaluation value data, i.e., the number of evaluations, N. Note that the standard error represents the variance of the estimated quantity, and a small standard error means that the accuracy of the estimated quantity is high.

[0032] In this embodiment, the standard error s of this normal distribution (or t-distribution) is defined as the dispersion value (which may also be called the degree of dispersion). If the standard error s is greater than the threshold T, it means that there is not enough data to ensure the reliability of the input evaluation result. Therefore, the threshold determination unit 22 transmits an additional evaluation request for the dummy video to the output unit 24, and the output unit 24 transmits the additional evaluation request to the evaluation experiment device 1. If the standard error s is equal to or less than the threshold T, it means that there is enough data to ensure the reliability of the input evaluation result.

[0033] In this embodiment, the threshold determination unit 22 calculates the smallest N at which s is less than T as B, and transmits B to the evaluation experiment device 1 via the output unit 24. Alternatively, the threshold determination unit 22 may remove 1, 2, ..., X-1 evaluation results from the X evaluation results last received by the evaluation count calculation device 2, compare the standard error s with T, and transmit the smallest N at which s is less than or equal to T as the optimal evaluation count B to the evaluation experiment device 1. Note that, as will be explained in the flow described later, removing 1, 2, ..., X-1 evaluation results from the X evaluation results last received means comparing the standard error s with T while increasing the number of evaluation results to be removed by one.

[0034] For example, when the range of the evaluation scale is R, T is set to 0.05×R. Note that T=0.05×R is just an example and is not limited to this.

[0035] Furthermore, the variation value s may be substituted for a confidence interval in an arbitrary confidence coefficient α of a probability distribution such as a normal distribution or a t distribution, as follows:

[0036] Here, μ is a value determined by the reliability coefficient α, and is a value that is uniquely determined from α using a distribution table.

[0037] Furthermore, as the variation value s, the standard deviation based on the variance value v output by the evaluation result information analysis unit 21 may be used as is, as shown in the following formula: Also, the variance value v itself may be used as the variation value s.

[0038] The evaluation count deriving device 2 may have the configuration shown in Fig. 2. The evaluation count deriving device 2 shown in Fig. 2 has an input unit 23, an output unit 24, and an evaluation count deriving unit 25. The evaluation count deriving unit 25 includes the function of the evaluation result information analysis unit 21 and the function of the threshold determination unit 22.

[0039] The input unit 23 inputs a plurality of evaluation values ​​obtained by subjective evaluation of a service multiple times. The evaluation count derivation unit 25 derives an optimal evaluation count, which is the minimum number of evaluations required to obtain a desired statistical reliability, based on the plurality of evaluation values. Specific examples of the derivation method are as described above in connection with the operations of the evaluation result information analysis unit 21 and the threshold determination unit 22.

[0040] (Processing Procedure) Hereinafter, the processing procedure of the evaluation system shown in FIG. 1 will be described with reference to the flowchart of FIG.

[0041] <S11> In S11, the evaluation experiment device 1 obtains the evaluation results of one evaluation for each of the A videos to be evaluated, as well as X evaluation results for the dummy videos, and transmits information on the X evaluation results for the dummy videos to the evaluation result information analysis unit 21.

[0042] <S12> In S12, the evaluation result information analysis unit 21 calculates the variance v for the data obtained by removing m pieces of data (initial value 0) from the last X pieces of data received from the cumulative information of the evaluation results for the dummy video, and transmits this value and the number of pieces of data used to calculate the variance v to the threshold determination unit 22.

[0043] For example, let us say that X=3 and that X (=3) pieces of data have been received twice at present. In other words, the evaluation result information analysis unit 21 currently holds six pieces of data, and these are sorted into q １ , q ２ , q ３ , q ４ , q ５ , q ６ Let us assume that: ｉ The index i in q represents the order of evaluation. １ is the first evaluation result, and q ６ is the latest evaluation result obtained at this point. Also, assume that m=2 at this point.

[0044] In the above situation, in S12, the evaluation result information analysis unit 21 receives q ４ , q ５ , q ６The data q after removing two data ４ and, q １ , q ２ , q ３ Calculate the variance from

[0045] In the above example, when removing m pieces of data from the last received X pieces of data, the most recent m pieces of data are removed. However, this is just an example, and when removing m pieces of data from the last received X pieces of data, any m pieces of data from the X pieces of data may be removed.

[0046] In the above example, the evaluation result information analysis unit 21 transmits the variance v and the number of data N, which is 6−2=4, to the threshold determination unit 22 .

[0047] <S13> In S13, the threshold determination unit 22 calculates a variation value s based on the variance v and N received from the evaluation result information analysis unit 21, and performs threshold determination by comparing the variation value s with a threshold value T. If s is equal to or less than T, the process proceeds to S14, and if s is greater than T, the process proceeds to S15.

[0048] <S14> In S14, where s is equal to or smaller than T, the threshold determination unit 22 sets m=m+1 and returns to S12. This makes it possible to search for the optimal number of evaluations.

[0049] <S15, 16> In S15, where s exceeds T, if m=0, the threshold determination unit 22 sends an additional evaluation request to the evaluation experiment device 1, and the process transitions to S11. In other words, if s exceeds T and m=0, the number of data items for the current number of times, "X pieces of data," is not sufficient, so an additional evaluation is requested.

[0050] If m>0, the process proceeds to S16, where the threshold determination unit 22 adds 1 to the current number of evaluation values ​​N and transmits this value as the optimum number of evaluations B to the evaluation experiment device 1. Upon receiving B, the evaluation experiment device 1 has the evaluators continue evaluations until the number of evaluations for each evaluation target video reaches B. In other words, the evaluation experiment device 1 has the evaluators evaluate each evaluation target video a number of times obtained by subtracting the number of evaluations that have been made for each evaluation target video up to that point from the received number of evaluations B, and outputs the evaluation results for all evaluation target videos.

[0051] For example, if B = 10, and seven evaluations have been made on a certain video P to be evaluated at the time the evaluation experimental device 1 receives B, then the evaluation experimental device 1 will make an additional 10-7 = 3 evaluations on the video P to be evaluated.

[0052] The meaning of adding 1 to the current number of evaluation values ​​N to set the optimum number of evaluations B when m>0 in S15 will be explained.

[0053] Let m at the time of S15 be m=M. "m>0" in S15 means that "variation value s≦threshold T" was true at the time of m=M-1, but "variation value s>threshold T" was true at the time of m=M. Here, m is the number of data to be removed. Therefore, the optimum number of evaluations B is the value obtained by adding 1 to the current number N of evaluation values.

[0054] As described above, in this embodiment, the optimum number of evaluations B is derived while evaluating the evaluation target video, but this process is merely an example. For example, first, video evaluation (and derivation of the optimum number of evaluations) may be performed solely for the purpose of deriving the optimum number of evaluations B, and then the optimum number of evaluations B may be evaluated for each evaluation target video.

[0055] (Hardware Configuration Example) Any of the devices described in this embodiment (evaluation experiment device 1, evaluation count derivation device 2) can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0056] That is, the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0057] Fig. 4 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 3 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B. The computer may further include a GPU.

[0058] The program that realizes the processing on the computer is provided by a recording medium 1001, such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0059] When an instruction to start the program is received, the memory device 1003 reads out and stores the program from the auxiliary storage device 1002. The CPU 1004 realizes the functions of the device in accordance with the program stored in the memory device 1003. Furthermore, when the computer is the evaluation count derivation device 2, for example, the memory device 1003 stores evaluation data of dummy images received from the evaluation experiment device 1. The CPU 1004 derives the optimal evaluation count B by reading out the evaluation data from the memory device 1003.

[0060] The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to a program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0061] (Summary of the embodiment, effects, etc.) As described above, the technology described in this embodiment makes it possible to clarify the optimal number of evaluations, and as a result, it becomes possible to efficiently obtain highly reliable evaluation data.

[0062] The following additional notes are provided regarding the above-described embodiments.

[0063] <Additional Notes> (Additional Item 1) A rating count deriving device including a memory and at least one processor connected to the memory, wherein the processor inputs a plurality of rating values ​​obtained by a plurality of subjective ratings of a service, and derives an optimal rating count, which is the minimum number of ratings required to obtain a desired statistical reliability, based on the plurality of rating values. (Additional Item 2) The processor determines a degree of variation of the plurality of rating values ​​based on a variance of the plurality of rating values, compares the degree of variation with a threshold, and derives as the optimal rating count the number of ratings at which the degree of variation does not exceed the threshold. (Additional Item 3) The rating count deriving device according to Additional Item 2, wherein the degree of variation is a standard error in a probability distribution of the rating values, a standard deviation in the probability distribution of the rating values, a confidence interval in the probability distribution of the rating values, or the variance. (Supplementary Item 4) The evaluation count deriving device according to Supplementary Item 2 or 3, wherein the processor requests an additional evaluation from an evaluation experiment device that executes the subjective evaluation when the degree of variation exceeds the threshold, and derives the optimal evaluation count based on a plurality of evaluation values ​​obtained by adding one or more evaluation values ​​obtained by the additional evaluation to the plurality of evaluation values. (Supplementary Item 5) A non-transitory storage medium that stores a program for causing a computer to function as the evaluation count deriving device according to any one of Supplementary Item 1 to 4.

[0064] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0065] REFERENCE SIGNS LIST 1 Evaluation experiment device 11 Dummy video DB 12 Evaluation target video DB 13 Evaluation unit 2 Evaluation number deriving device 21 Evaluation result information analysis unit 22 Threshold value determination unit 23 Input unit 24 Output unit 25 Evaluation number deriving unit 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. A rating count derivation device comprising: an input unit for inputting multiple evaluation values ​​obtained by multiple subjective evaluations of a service; and a rating count derivation unit for deriving an optimal rating count, which is the minimum number of evaluations required to obtain a desired statistical reliability, based on the multiple evaluation values.

2. The evaluation count deriving device according to claim 1, wherein the evaluation count deriving unit determines a degree of variation of the multiple evaluation values ​​based on a variance value of the multiple evaluation values, compares the degree of variation with a threshold, and derives as the optimal evaluation count a number of evaluations at which the degree of variation does not exceed the threshold.

3. The evaluation count deriving device according to claim 2, wherein the degree of variability is a standard error in the probability distribution of the evaluation values, a standard deviation in the probability distribution of the evaluation values, a confidence interval in the probability distribution of the evaluation values, or the variance value.

4. The evaluation count derivation device according to claim 2 or 3, wherein the evaluation count derivation unit requests an additional evaluation from an evaluation experiment device that executes the subjective evaluation when the degree of variability exceeds the threshold value, and derives the optimal evaluation count based on a plurality of evaluation values ​​obtained by adding one or more evaluation values ​​obtained by the additional evaluation to the plurality of evaluation values.

Images