Video processing device, method and program

Abstract

A video processing device according to an embodiment includes a detection unit for detecting at least one type of event that possibly influences the quality of experience of a user with a video displayed by an application program and a determination unit for determining the degree of influence of the event on the quality of experience with the video on the basis of a detection result of the detection unit.

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to a video processing device, method, and program. [Background technology]

[0002] In recent years, in order to improve user satisfaction with video (moving images) and to improve resource efficiency, methods for controlling communication resources etc. based on intent have been studied and implemented. Specifically, the user's quality of experience (QoE) requirements for video are considered as the user's intent, and network (NW) resources and server resources are appropriately allocated to satisfy these requirements. Therefore, it is necessary to accurately grasp the user's QoE and extract the intent from changes in QoE.

[0003] Technologies for quantifying the quality of experience in video distribution systems are classified into subjective quality assessment and objective quality. In the subjective evaluation method, users of an application program (hereinafter sometimes referred to as an application or app) evaluate the quality of their experience by being asked to rate the quality or by completing a questionnaire (see, for example, Non-Patent Document 1). In addition, in objective evaluation methods, it is common to estimate the user's quality of experience based on parameters related to the communication environment and situation (e.g., packet loss and jitter, etc.) (see, for example, Non-Patent Documents 2 to 4). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] T. Hoβfeld, R. Schatz, M. Seufert, M. Hirth, T. Zinner, &P. Tran-Gia, (2011) “Quantification of YouTube(R) QoE via Crowdsourcing,” in IEEE International Workshop on Multimedia Quality of Experience - Modeling, Evaluation, and Directions (MQoE), Dana Point, CA, USA. [Non-patent document 2] MH Pinson, & S.Wolf, (2004) “A New Standardized Method for Objectively Measuring Video Quality”, IEEE Transactions on Broadcasting, Vol. 50, pp. 312-322. [Non-patent document 3] F. Agboma &A. Liotta, (2008) “QoE-aware QoS management,” in Proceedings of the 6th International Conference on Advances in Mobile Computing and Multimedia, pp. 111-116 [Non-patent document 4] VA Machado et al., "A new proposal to provide estimation of QoS and QoE over WiMAX networks: An approach based on computational intelligence and discrete-event simulation," 2011 IEEE Third Latin-American Conference on Communications, 2011, pp. 1-6 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the above-mentioned subjective evaluation method, if the type of application program changes, it is necessary to have users re-assign quality ratings to evaluate the QOE, which raises concerns about enormous costs.Furthermore, even in the above-mentioned objective evaluation method, since the parameters obtained from the communication environment and the like are the base, the parameters must be revised when the application program or communication method changes. In other words, existing technologies have not yet achieved a solution to universally evaluate the QoE of applications and extract user intent.

[0006] The present invention has been made in light of the above-mentioned circumstances, and its object is to provide a video processing device, method, and program that enable appropriate evaluation of the quality of experience of video. [Means for solving the problem]

[0007] A video processing device according to one aspect of the present invention includes a detection unit that detects at least one type of event that may affect a user's quality of experience for a video displayed by an application program, and a determination unit that determines the degree of impact that the event has on the quality of experience for the video based on a detection result by the detection unit. the detection unit detects an object appearing in the video, pixelation appearing in the video, and a wait display appearing in the video, and the determination unit calculates a first score indicating the degree of impact that the stoppage of the behavior of the object detected by the detection unit has on the quality of experience when the behavior of the object detected by the detection unit remains stopped over time, a second score indicating the degree of impact that the pixelation detected by the detection unit has on the quality of experience, and a third score indicating the degree of impact that the wait display detected by the detection unit has on the quality of experience, The degree of impact of the event on the quality of experience is determined based on the calculated first, second, and third scores. When the position and size of the object detected by the detection unit do not change over time, the determination unit determines the duration of the state in which the behavior of the object has stopped, and calculates the first score based on the position of the object, the size of the object, and the duration of the state in which the behavior of the object has stopped. The position and size of the object are determined based on an area of ​​the object and a distance from a center of the object to a center of a display screen of the video. .

[0008] A video processing method according to one aspect of the present invention is performed by a video processing device. circle A method for The detection unit of the video processing device Detecting at least one type of event that may affect a user's quality of experience with respect to a video displayed by an application program; The determination unit of the video processing device determining a degree of influence of the event on the quality of experience for the video based on the detected result. The detecting includes detecting an object appearing in the video, pixelation appearing in the video, and a wait display appearing in the video, respectively; the determining includes calculating, when the behavior of the detected object remains stationary over time, a first score indicating the degree of impact that this stoppage has on the quality of experience, a second score indicating the degree of impact that the detected pixelation has on the quality of experience, and a third score indicating the degree of impact that the detected wait display has on the quality of experience, respectively; and determining the degree of impact that the event has on the quality of experience based on the calculated first, second, and third scores; the determining includes, when the position and size of the detected object do not change over time, determining a duration for which the behavior of the object remains stationary, and calculating the first score based on the position of the object, the size of the object, and the duration for which the behavior of the object remains stationary, . [Effects of the Invention]

[0009] According to the present invention, it is possible to appropriately evaluate the quality of experience of a video. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an application example of a video processing device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of a subject detected by the subject detection unit. [Figure 3] FIG. 3 is a diagram illustrating an example of a processing procedure performed by the stop determination unit. [Figure 4] FIG. 4 is a diagram illustrating an example of calculation of a stop time for one subject. [Figure 5] FIG. 5 is a diagram showing an example of updating the subject's position and subject's size in the past time period. [Figure 6] FIG. 6 is a diagram illustrating an example of a processing procedure performed by the wait determination unit. [Figure 7] FIG. 7 is a diagram illustrating an example of calculation of a waiting duration for a waiting display. [Figure 8] FIG. 8 is a diagram showing an example of updating the position and size of the waiting display in the past time period. [Figure 9] FIG. 9 is a diagram illustrating an example of a processing procedure performed by a pixelation determination unit. [Figure 10] FIG. 10 is a diagram illustrating an example of calculating pixelation duration for pixelation. [Figure 11] FIG. 11 is a diagram showing an example of updating the pixelation position and pixelation size for a past time period. [Figure 12] FIG. 12 is a diagram illustrating an example of a processing procedure performed by the sensation calculation unit and the intent derivation unit. [Figure 13] FIG. 13 is a diagram showing an example of subjects, pixelation, and wait displays detected from the screen of a Web conference application. [Figure 14]FIG. 14 is a diagram showing a first example of subjects, pixelations, and wait displays detected at each time from the screen of a video distribution application, as well as score calculation. [Figure 15] FIG. 15 is a diagram showing a second example of subjects, pixelations, and wait displays detected at each time from the screen of a video distribution application, as well as score calculation. [Figure 16] FIG. 16 is a diagram showing a third example of subjects, pixelations, and wait displays detected at each time from the screen of a video distribution application, as well as score calculation. [Figure 17] FIG. 17 is a diagram showing a fourth example of subjects, pixelations, and wait displays detected at each time from the screen of a video distribution application, as well as score calculation. [Figure 18] FIG. 18 is a block diagram showing an example of the hardware configuration of a video processing device according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a diagram showing an application example of a video processing device according to an embodiment of the present invention. As shown in FIG. 1, a video processing device 100 according to one embodiment of the present invention has a subject detection unit 11, a pixelation detection unit 12, a wait display detection unit 13, a stop determination unit 14, a pixelation determination unit 15, a wait determination unit 16, a bodily sensation calculation unit 17, and an intent derivation unit 18.

[0012] The subject detection unit 11 acquires the application screen as images (moving images) at each of multiple timings from the current time to a specified time in the past, and detects the position and size of the subject that appears in the images at the target timing at which the subject, an object in which the user is interested, appears in this image and at past timings that are chronologically connected to that timing, and outputs the results of these detections to the stop determination unit 14 together with information on the target timing.

[0013] The pixelation detection unit 12 acquires the screen of the application as images at each of a plurality of timings, detects the position and size of pixelation appearing in this image at the target timing at which pixelation appears in this image, and outputs the results of this detection together with information on the target timing to the pixelation determination unit 15.

[0014] The wait display detection unit 13 acquires the screen of the application as images at each of a plurality of timings, detects the position and size of the wait display appearing in this image at the target timing at which the wait display appears in this image, and outputs the results of this detection together with information on the target timing to the wait determination unit 16. The subject detection unit 11, pixelation detection unit 12, and wait display detection unit 13 detect events that may affect the user's quality of experience with respect to the video displayed by the application program.

[0015] The operations of the stop determination unit 14, pixelation determination unit 15, wait determination unit 16, bodily sensation calculation unit 17, and intent derivation unit 18 will be described in order below. The subject detector 11, pixelation detector 12, and waiting indicator detector 13 may be implemented by utilizing or retraining a public object detection library (eg, yolo).

[0016] Next, a description will be given of the processing by the stop determination unit 14. Fig. 2 is a diagram showing an example of a subject detected by the subject detection unit. Fig. 3 is a diagram showing an example of a processing procedure by the stop determination unit. First, the stop determination unit 14 receives input of the current time (present point in time) t, information on the position of the subject, and information on the size of the subject from the subject detection unit 11.

[0017] The position of the subject is represented by the horizontal coordinate x of the upper left vertex of the subject to be detected, for example, "subject A" shown in FIG. 2, and the vertical coordinate y of the upper left vertex of the subject. The size of the subject is made up of the subject's width w and height h. These x, y, w, and h are all assumed to be normalized.

[0018] Next, when the subject is detected by the subject detection unit 11 in the video at the current time t, the stop determination unit 14 calculates the length of the stop time, which is the time during which the subject's position and size do not change from the target timing between the past time period [t-INT to tN*INT] and the current time t to the timing going back from that timing, i.e., the time during which the subject's behavior continues to stop (S11). Here, when a subject of the same position and size is detected at a target timing and at a past timing chronologically connected to the target timing, it is determined that the subject's behavior has stopped at the time corresponding to these timings, and this time corresponds to the subject's stop time related to the target timing.

[0019] INT is the time interval between detection of the subject, pixelation, or wait display, and can be set by a system administrator of the video processing device 100. t~N*INT is the interval from the current time (t) to N*INT, and is, for example, the range of detection time for the main body, pixelation, or wait display on the screen, and can be set by the system administrator. For example, if the detection time interval INT is 5 [ms] and N is 100, the stop time, pixelation duration, or wait duration, which will be described later, will be calculated based on the subject, pixelation, or wait display within 500 [ms] (5 [ms] x 100 times) from the current time.

[0020] The stop determination unit 14 updates the position and size of the past time period [t-INT to tN*INT] (S12). The stop determination unit 14 calculates a stop score indicating the degree of impact that the subject whose behavior has stopped has on the user's quality of experience of the video, taking into account the subject's stopping position, the size of the stopped subject, and the length of the stop time calculated in S11, and outputs this score (S13). This stop score can be calculated for each timing from the current time t to the past time "t-INT*N" at which the subject is detected by the subject detection unit 11 and the behavior of the subject is determined to have stopped.

[0021] FIG. 4 is a diagram illustrating an example of calculation of a stop time for one subject. 4, the parameters (x, y, w, h) of the subject's position abscissa, subject's position ordinate, subject's width, and subject's height are the same (0.1, 0.1, 0.3, 0.4) from the current time t to "t-INT" to "t-INT*2", and the parameters at time "t-INT*3", which is the time interval "INT" prior to time "t-INT*2", change to (0.2, 0.1, 0.3, 0.5) from time "t-INT*2". In S11, the stop determination unit 14 calculates the time "INT*2" during which the parameters do not change from the current time t as the stop time of the subject's behavior related to the current time t.

[0022] FIG. 5 is a diagram showing an example of updating the subject's position and subject's size in the past time period. In the example shown in Figure 5, since the new parameters (x, y, w, h) at the current time t shown in Figure 4 are (0.1, 0.1, 0.3, 0.4), at S12 the stop determination unit 14 updates the parameters at the time "t-INT" shown in Figure 4 to the parameters at the current time t.

[0023] With this update, the parameters at time "t-INT*2" shown in Figure 4 are updated to the parameters at time "t-INT*1" which is the time interval "INT" after the time "t-INT*2" before the update, and the parameters at each other time are updated to the parameters at time "t-INT*1" which is the time interval "INT" after the time.

[0024] After the update, in S13, the stop determination unit 14 calculates a stop score using the following formula (1) in consideration of the subject's position, the subject's size, and the calculated length of the stop time.

[0025] Stopping score = Σ (subject area * distance from the center of the subject to the center of the screen * stopping time * weight) ... Equation (1) Using this formula (1), the stopping score is calculated, for example, as follows: The coordinates of the center of the subject can be calculated from (x, y, w, h). The horizontal coordinate of the center of the subject and the vertical coordinate of the center of the subject are calculated by the following equations (2) and (3), respectively. The abscissa of the center of the subject = x + w / 2 ... Equation (2) The ordinate of the center of the subject = yh / 2 ... Equation (3) When the maximum abscissa and ordinate of the screen are 1, the coordinates of the center of the screen are (0.5, 0.5). The distance dis from the center of the subject to the center of the screen is calculated using the following equation (4).

number

[0026] Next, a description will be given of the processing performed by the wait determination unit 16. Fig. 6 is a diagram showing an example of a processing procedure performed by the wait determination unit. First, the wait determination unit 16 receives input of the current time t from the wait display detection unit 13, and parameters (x1, y1, w1, h1) that are information on the position of the wait display and information on the size of the wait display.

[0027] The position of the waiting display is determined by the horizontal coordinate x1 of the top left corner of the detection target and the vertical coordinate y1 of the top left corner of the detection target. The size of the waiting display is determined by the horizontal width w1 of the detection target and the vertical width h1 of the detection target. These x1, y1, w1, and h1 are all assumed to be normalized.

[0028] Next, the wait determination unit 16 calculates the length of the wait duration, which is the time during which the wait display continues to appear in the video from the target timing between the past time period [t-INT to tN*INT] and the current time t to the timing going back from that timing (S21). The wait determination unit 16 updates the position and size of the wait display in the past time period [t-INT to tN*INT] (S22). The wait determination unit 16 calculates a wait score indicating the degree of influence that the wait display has on the user's quality of experience with the video, taking into account the wait duration calculated in S21, and outputs the score (S23). This waiting score can be calculated for each timing at which the waiting display is detected by the waiting display detection unit 13, among each timing from the current time t to the past time "t-INT*N".

[0029] FIG. 7 is a diagram illustrating an example of calculation of a waiting duration for a waiting display. 7, the parameters (x1, y1, w1, h1) of the abscissa of the wait display, the ordinate of the wait display, the width of the wait display, and the height of the wait display are the same (0.4, 0.6, 0.1, 0.1) at the current time t, the time "t-INT", and the time "t-INT*2", and the parameters at the time "t-INT*3" which is the time interval "INT" prior to the time "t-INT*2" change to (null) as seen from the time "t-INT*2". In S21, the wait determination unit 16 calculates the time "INT*2" during which the parameters do not change as seen from the current time t as the wait duration for the current time t.

[0030] FIG. 8 is a diagram showing an example of updating the position and size of the waiting display in the past time period. In the example shown in Figure 8, since the parameters (x1, y1, w1, h1) at the current time t shown in Figure 7 are (0.4, 0.6, 0.1, 0.1), at S22 the waiting determination unit 16 indicates that the parameters at the time "t-INT" shown in Figure 7 will be updated to the parameters at the current time t.

[0031] With this update, the parameters at time "t-INT*2" shown in Figure 7 are updated to the parameters at time "t-INT*1" which is the time interval "INT" after the time "t-INT*2" before the update, and the parameters at each other time are updated to the parameters at time "t-INT*1" which is the time interval "INT" after the time.

[0032] After the update, in S23, the waiting determination unit 16 calculates the waiting score by taking into account the waiting duration, using the following formula (6). Waiting score = presence or absence of waiting display + waiting duration * weight ... Equation (6)

[0033] When the wait display appears on the screen, the presence / absence of the wait display is "1", and when the wait display does not appear on the screen, the presence / absence of the wait display is "0". Using the above formula (6), the waiting score is calculated, for example, as follows: Wait score = 1 + (INT * 2) * weight

[0034] Next, a description will be given of the processing performed by the pixelation determination unit 15. Fig. 9 is a diagram showing an example of the processing procedure performed by the pixelation determination unit. First, the pixelation determination unit 15 receives as input from the pixelation detection unit 12 the current time t, information on the position of the pixelation, and parameters (x2, y2, w2, h2) that are information on the size of the pixelation.

[0035] The position of the pixelation is the horizontal coordinate x2 of the top left corner of the target, and the vertical coordinate y2 of the top left corner of the target. The size of the pixelation is the horizontal width w2 of the target, and the vertical width h2 of the target. These x2, y2, w2, and h2 are all assumed to be normalized.

[0036] Next, the pixelation determination unit 15 calculates the length of pixelation duration, which is the time during which pixelation continues to appear in the image from the target timing between the past time period [t-INT to tN*INT] and the current time t to the timing going back from that timing (S31). The pixelation determination unit 15 updates the position and size of the pixelation in the past time period [t-INT to tN*INT] (S32).

[0037] The pixelation determination unit 15 calculates a pixelation score indicating the degree of impact that pixelation has on the user's quality of experience of the video, taking into account the pixelation size and the pixelation duration calculated in S31, and outputs this (S33). This pixelation score can be calculated for each timing at which pixelation is detected by the pixelation detection unit 12, from the current time t to the past time "t-INT*N".

[0038] FIG. 10 is a diagram illustrating an example of calculating pixelation duration for pixelation. In the example shown in Figure 10, at the current time t and time "t-INT", the parameters (x2, y2, w2, h2) which are the horizontal coordinate of the pixelation, the vertical coordinate of the pixelation, the horizontal width of the pixelation, and the vertical width of the pixelation are the same (0.2, 0.2, 0.3, 0.5), and it is shown that the parameters at time "t-INT*2", which is the time interval "INT" prior to time "t-INT*1", change to (null) from time "t-INT*1". In S31, the pixelation determination unit 15 calculates the time "INT" during which the parameters do not change as viewed from the current time t as the pixelation duration related to the current time t.

[0039] FIG. 11 is a diagram showing an example of updating the pixelation position and pixelation size for a past time period. In the example shown in Figure 11, since the new parameters (x2, y2, w2, h2) at the current time t shown in Figure 10 are (0.2, 0.2, 0.3, 0.5), at S32 the pixelation determination unit 15 updates the parameters at the time "t-INT" shown in Figure 10 to the parameters at the current time t.

[0040] With this update, the parameters at time "t-INT*2" shown in Figure 10 are updated to the parameters at time "t-INT*1" which is the time interval "INT" after the time "t-INT*2" before the update, and the parameters at each other time are updated to the parameters at time "t-INT*1" which is the time interval "INT" after the time.

[0041] After the update, in S33, the pixelation determination unit 15 calculates the pixelation score by taking into account the pixelation duration time using the following equation (7). Pixelation score = presence or absence of pixelation + pixelation duration * pixelation size * weight ... Equation (7)

[0042] When pixelation appears on the screen, the presence or absence of pixelation is "1", and when pixelation does not appear on the screen, the presence or absence of pixelation is "0". Using this equation (7), the pixelation score is calculated, for example, as follows: Pixelation Score = 1 + INT * (0.3 * 0.5) * Weight

[0043] Next, a description will be given of the processing performed by the bodily sensation calculation unit 17 and the intent derivation unit 18. Fig. 12 is a diagram showing an example of the processing procedure performed by the bodily sensation calculation unit and the intent derivation unit. First, the sensation calculation unit 17 receives the stop score output from the stop determination unit 14, the pixelation score output from the pixelation determination unit 15, and the wait score output from the wait determination unit 16.

[0044] The experience calculation unit 17 multiplies the input stop score, pixelation score, and wait score by a predetermined weight according to the type of score, and then calculates the sum of each score to calculate an experience score that indicates the degree to which the stopping of the subject's behavior, pixelation, and wait display affect the user's experience quality (S41). This experience score can be calculated for each timing from the current time t to the past time "t-INT*N", at which the stop score is calculated, the waiting score is calculated, and the pixelation score is calculated.

[0045] If the calculated experience score exceeds a threshold, the intent derivation unit 18 determines that the quality experienced by the user for the video has deteriorated, and generates and outputs an intent requesting improvement of the experience quality (S42). This intent can be generated for each of the timings from the current time t to the past time "t-INT*N" at which the experience score was calculated.

[0046] Next, an example in which the video processing device 100 according to this embodiment is applied to a Web conference application will be described. FIG. 13 is a diagram showing an example of subjects, pixelation, and wait displays detected from the screen of a Web conference application. 13, it is shown that subjects (subject A, subject B, subject C), pixelation, and wait display (Wait) are detected at time "t-INT" and the current time t. For simplicity of explanation, it is assumed that N in the above N*INT is 1.

[0047] 13, the stop determination unit calculates the time from "t-INT" to the current time t as the stop time of the behavior of the subjects at the current time t, and then calculates the stop score for the current time t by taking this stop time into consideration. In this example, subject B is relatively large in size and is close to the center of the screen, so it has a relatively large impact on the stop score.

[0048] In the example shown in Figure 13, pixelation is continuously detected between the current time t and the time "t-INT", so the pixelation determination unit 15 calculates this time as the pixelation duration for the current time t, and takes this time and the size of the pixelation into account to calculate the pixelation score for the current time t.

[0049] In the example shown in Figure 13, since the waiting display is continuously detected between the current time t and the time "t-INT", the waiting determination unit 16 calculates this time as the waiting duration for the current time t, and then takes this time into account to calculate the waiting score for the current time t.

[0050] The experience calculation unit 17 calculates the experience score for the same timing based on the stop score, pixelation score, and waiting score calculated for this timing. In addition, the intent derivation unit 18 compares this calculated experience score with a set threshold, and when it determines that the user's experience quality has deteriorated, it issues an intent to request an improvement in the user's experience quality.

[0051] Next, an example in which the video processing device 100 according to this embodiment is applied to a video distribution application will be described. FIG. 14 is a diagram showing a first example of subjects, pixelations, and wait displays detected at each time from the screen of a video distribution application, as well as score calculation. In the example shown in FIG. 14, since a wait display (Wait) is detected at time "t-INT*3", the wait determination unit 16 calculates a wait score related to this time "t-INT*3".

[0052] FIG. 15 is a diagram showing a second example of subjects, pixelations, and wait displays detected at each time from the screen of a video distribution application, as well as score calculation. In the example shown in FIG. 15, the subject and pixelation are detected at time "t-INT*1", indicating that the subject has been detected at the current time t.

[0053] In the example shown in Figure 15, "Subject A" is detected at time "t-INT*2", but the subject is not detected at the time just before this, "t-INT*3", so the stop determination unit 14 determines that there is no stop in the subject's behavior at this time "t-INT*2". Therefore, the user experience score for this time "t-INT*2" is not calculated, and no intent is output.

[0054] FIG. 16 is a diagram showing a third example of subjects, pixelations, and wait displays detected at each time from the screen of a video distribution application, as well as score calculation. 16, "subject A" is detected at time "t-INT", and "subject A" is also detected at the time "t-INT*2" chronologically preceding this time, and the position and size of "subject A" do not change between these times. Therefore, the stop determination unit 14 determines that the behavior of the subject has stopped at time "t-INT", and calculates a stop score for this time "t-INT".

[0055] In the example shown in FIG. 16, pixelation is detected at time "t-INT", so the pixelation determination unit 15 calculates the pixelation score for this time "t-INT".

[0056] FIG. 17 is a diagram showing a fourth example of subjects, pixelations, and wait displays detected at each time from the screen of a video distribution application, as well as score calculation. In the example shown in Figure 17, "Subject A" is detected at the current time t, and "Subject A" is also detected at the time "t-INT" chronologically preceding this time, and the position and size of "Subject A" change between the current time t and the time "t-INT". Therefore, the stop determination unit 14 determines that there is no stop in the behavior of the subject at this time "t-INT". Therefore, the user experience score for this current time t is not calculated. The experience calculation unit 17 calculates the experience score for the same timing based on the stop score, pixelation score, and waiting score calculated for this timing. In addition, the intent derivation unit 18 compares this calculated experience score with a set threshold, and when it determines that the user's experience quality has deteriorated, it issues an intent to request an improvement in the user's experience quality.

[0057] In one embodiment of the present invention as described above, an index that affects the user's quality of experience is determined from the image displayed on the screen by a video application program, so that the user's quality of experience for the image can be evaluated in a general manner for various types of application programs.

[0058] FIG. 18 is a block diagram showing an example of the hardware configuration of a video processing device according to an embodiment of the present invention. 18, the video processing device 100 according to the above embodiment is configured, for example, by a server computer or a personal computer, and has a hardware processor 111A such as a CPU (Central Processing Unit). A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111A via a bus 115.

[0059] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from a communication network NW. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0060] The input / output interface 113 is connected to an input device 200 and an output device 300 that are attached to the video processing device 100 and used by a user or the like. The input / output interface 113 takes in operation data input by a user or the like via an input device 200 such as a keyboard, a touch panel, a touchpad, or a mouse, and outputs output data to an output device 300 including a display device using a liquid crystal or an organic electroluminescence (EL) display, etc. Note that the input device 200 and the output device 300 may be devices built into the video processing device 100, or may be input devices and output devices of other information terminals that can communicate with the video processing device 100 via a network NW.

[0061] The program memory 111B is a non-transitory tangible storage medium that is a combination of a non-volatile memory that can be written to and read from at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), and a non-volatile memory such as a read only memory (ROM), and stores programs necessary to execute various control processes, etc., according to one embodiment.

[0062] The data memory 112 is a tangible storage medium that is a combination of, for example, the above-mentioned nonvolatile memory and a volatile memory such as RAM (Random Access Memory), and is used to store various data acquired and created during various processing steps.

[0063] The video processing device 100 according to one embodiment of the present invention can be configured as a data processing device having software-based processing function units, including a subject detection unit 11, a pixelation detection unit 12, a wait display detection unit 13, a stop determination unit 14, a pixelation determination unit 15, a wait determination unit 16, a sensation calculation unit 17, and an intent derivation unit 18, as shown in FIG. 1.

[0064] Each information storage unit used as a working memory by each unit of the video processing device 100 can be configured using the data memory 112 shown in Fig. 18. However, these configured storage areas are not essential components within the data analysis device 1, and may be areas provided in an external storage medium such as a USB (Universal Serial Bus) memory, or a storage device such as a database server located in the cloud.

[0065] The processing function units in each of the above-mentioned subject detection unit 11, pixelation detection unit 12, wait display detection unit 13, stop determination unit 14, pixelation determination unit 15, wait determination unit 16, bodily sensation calculation unit 17, and intent derivation unit 18 can all be realized by having the above-mentioned hardware processor 111A read and execute a program stored in the program memory 111B. Note that some or all of these processing function units may be realized in various other forms, including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0066] The methods described in each embodiment may be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (e.g., a floppy disk, a hard disk, etc.), an optical disk (e.g., a CD-ROM, a DVD, an MO, etc.), or a semiconductor memory (e.g., a ROM, a RAM, a flash memory, etc.), or may be transmitted and distributed via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only execution programs but also tables and data structures) that the computer executes. The computer that realizes this device reads the program stored on the recording medium and, in some cases, configures the software means using the configuration program, and executes the above-described processing by having the operation controlled by this software means. The term "recording medium" as used herein is not limited to a storage medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory installed inside the computer or in a device connected via a network.

[0067] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention. [Explanation of symbols]

[0068] 100...Video processing device 11...Subject detection unit 12...Pixelation detection unit 13...Waiting display detection unit 14...Stop judgment section 15...Pixelation determination unit 16...Waiting decision section 17…Experience calculation section 18...Intent derivation part

Claims

1. a detection unit that detects at least one type of event that may affect a user's quality of experience with respect to an image displayed by an application program; a determination unit that determines a degree of impact that the event has on the quality of experience for the video based on a detection result by the detection unit; Equipped with The detection unit Detecting an object appearing in the image, a pixelation appearing in the image, and a wait indication appearing in the image, respectively; The determination unit When the behavior of the object detected by the detection unit remains stopped over time, a first score is calculated to indicate the degree of impact that this stoppage has on the quality of experience, a second score is calculated to indicate the degree of impact that the pixelation detected by the detection unit has on the quality of experience, and a third score is calculated to indicate the degree of impact that the wait display detected by the detection unit has on the quality of experience. determining a degree of impact of the event on the quality of experience based on the calculated first, second, and third scores; The determination unit When the position and size of the object detected by the detection unit do not change over time, a duration of the state in which the behavior of the object remains stationary is determined; calculating the first score based on a position of the object, a size of the object, and a duration of time during which the object is motionless; The position of the object and the size of the object are based on the area of ​​the object and the distance from the center of the object to the center of the display screen of the image. Image processing device.

2. The determination unit determining a duration for which the pixelation detected by the detection unit continuously appears in the video, and calculating the second score based on a size of the pixelation and a duration for which the pixelation continuously appears in the video; The video processing device according to claim 1 .

3. The determination unit determining a duration for which the wait display detected by the detection unit continuously appears in the video, and calculating the third score based on the duration for which the wait display continuously appears in the video; The video processing device according to claim 1 .

4. A method performed by a video processing device, comprising: detecting, by a detection unit of the video processing device, at least one type of event that may affect a user's quality of experience with respect to video displayed by an application program; determining, by a determination unit of the video processing device, a degree of influence of the event on the quality of experience for the video based on the detected result; Equipped with The detecting step includes: detecting an object appearing in the image, a pixelation appearing in the image, and a wait indication appearing in the image, The determining step comprises: When the behavior of the detected object remains stationary over time, calculating a first score indicating the degree of impact that this stoppage has on the quality of experience, a second score indicating the degree of impact that the detected pixelation has on the quality of experience, and a third score indicating the degree of impact that the detected wait display has on the quality of experience; determining a degree of impact of the event on the quality of experience based on the calculated first, second, and third scores; The determining step comprises: determining a duration of a state in which the behavior of the object is stopped when the position and size of the detected object do not change over time; calculating the first score based on a position of the object, a size of the object, and a duration of a state in which the object is motionless; The position of the object and the size of the object are based on the area of ​​the object and the distance from the center of the object to the center of the display screen of the image. Image processing method.

5. The determining step comprises: determining a duration for which the detected pixelation appears in the image, and calculating the second score based on a size of the pixelation and a duration for which the pixelation appears in the image. The video processing method according to claim 4 .

6. The determining step comprises: determining a duration for which the detected wait indication continuously appears in the image, and calculating the third score based on the duration for which the wait indication continuously appears in the image. The video processing method according to claim 4 .

7. A video processing program that causes a processor to function as each unit of the video processing device according to any one of claims 1 to 3.

Images