Quality of Experience Calculation Device, Quality of Experience Calculation Method, and Quality of Experience Calculation Program

Abstract

A quality-of-experience calculation device according to an embodiment of the present invention is provided with an operation-event acquisition unit, a screen acquisition unit, a latency extraction unit, a waiting-screen determination unit, and a quality-of-experience calculation unit. The operation-event acquisition unit acquires an operation event of a user who uses an application for which quality of experience is to be calculated. The screen acquisition unit acquires a series of app screens provided by the application. The latency extraction unit, through image analysis of the app screens acquired by the screen acquisition unit, extracts the latency from the timing of acquisition of the operation event by the operation-event acquisition unit to a change to an app screen indicating a response to the operation event. The waiting-screen determination unit, through image analysis of the app screens acquired by the screen acquisition unit, determines whether or not each of the app screens is a waiting screen. The quality-of-experience calculation unit calculates quality of experience for the user on the basis of the latency extracted by the latency extraction unit and the result of waiting-screen determination by the waiting-screen determination unit.

Description

[Technical field]

[0001] An embodiment of the present invention relates to a quality of experience calculation device, a quality of experience calculation method, and a quality of experience calculation program. [Background technology]

[0002] In recent years, methods of controlling communication resources based on intent have been studied and implemented to improve user satisfaction and improve resource efficiency. Specifically, the quality of experience (QoE) requirements, which are the quality of the service experienced by the user, are taken as the user's intent, and network resources, server resources, etc. are appropriately allocated to satisfy the requirements. For this reason, it is necessary to accurately grasp the quality of experience of users who use application programs (hereinafter abbreviated as apps) that provide services to users on user terminals, and to extract intent from the changes in the quality of experience.

[0003] Among the applications used by users, there are operational applications that are frequently operated by users and whose screens transition in response to the operations. Examples of such operational applications include virtual desktop (VDI: Virtual Desktop Infrastructure) / remote desktop (RDI: Remote Desktop Infrastructure) services, e-commerce (Electric commerce) services, some gaming services, etc.

[0004] A technology for quantifying the quality of experience of such an operational application is proposed in, for example, Non-Patent Document 1. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] J. Hosek et al., "Predicting user QoE satisfaction in current mobile networks," 2014 IEEE International Conference on Communications (ICC), 2014, pp. 1088-1093, doi: 10.1109 / ICC.2014.6883466. Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the quality of experience quantification technology proposed in Non-Patent Document 1, when the target application changes, for example, from a web browsing application for using an e-commerce service to a VDI application, the parameters for evaluating the quality of experience and the API (Application Programming Interface) specifications are different, so the quality of experience evaluation method cannot be reused.

[0007] As a result, it was necessary to consider and design a method for measuring the quality of experience for each app, and it took time and cost to implement resource control based on the quality of experience requirements (intents).

[0008] The present invention aims to provide a quality of experience calculation technique that can calculate a quality of experience that is accurate, that is, close to the actual user experience, without having to redesign the quality of experience measurement method, even if the application for which the quality of experience is to be calculated is different. [Means for solving the problem]

[0009] In order to solve the above problem, a quality of experience calculation device according to one aspect of the present invention includes an operation event acquisition unit, a screen acquisition unit, a delay time extraction unit, a wait screen determination unit, and a quality of experience calculation unit. The operation event acquisition unit acquires an operation event of a user who uses an application for which the quality of experience is to be calculated. The screen acquisition unit acquires a series of application screens provided by the application. The delay time extraction unit extracts a delay time from the acquisition of an operation event acquired by the operation event acquisition unit to the change to an application screen responding to the operation event, by performing image analysis on the application screen acquired by the screen acquisition unit. The wait screen determination unit determines whether the application screen is a wait screen, by performing image analysis on the application screen acquired by the screen acquisition unit. The quality of experience calculation unit calculates the quality of experience of the user based on the delay time extracted by the delay time extraction unit and the determination result of the wait screen by the wait screen determination unit. Effect of the Invention

[0010] According to one aspect of the present invention, the user's quality of experience is calculated using generic data, namely, operation events and application screens. Therefore, even if the application for which the quality of experience is to be calculated is different, a quality of experience calculation technique can be provided that can calculate a quality of experience that is accurate, that is, close to the actual user experience, without having to redesign the quality of experience measurement method. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an intent extraction device to which a quality of experience calculation device according to an embodiment of the present invention is applied. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of the intent extraction device. [Diagram 3] FIG. 3 is a flowchart illustrating an example of an intent extraction processing operation performed by the intent extraction device. [Figure 4] FIG. 4 is a flowchart showing an example of the operation of the operation event acquisition processing in FIG. [Diagram 5]FIG. 5 is a flowchart showing an example of the abnormal operation detection processing operation in FIG. [Figure 6] FIG. 6 is a flowchart showing an example of an abnormal operation detection process corresponding to the operation event in FIG. 5 in the case where the operation event is a click operation event of a mouse button. [Figure 7] FIG. 7 is a flowchart showing an example of an abnormal operation detection process corresponding to an operation event in FIG. 5 when the operation event is a double click operation event of a mouse button or a close button press operation event which is a click operation on the close button of a mouse button. [Figure 8] FIG. 8 is a flowchart showing an example of the application screen acquisition processing operation in FIG. [Figure 9] FIG. 9 is a flowchart showing an example of the subject detection processing operation in FIG. [Figure 10] FIG. 10 is a flowchart showing an example of the response time determination process operation in FIG. [Figure 11] FIG. 11 is a flowchart showing an example of the operation of the operation event response time determination process in FIG. 10 when the operation event is a double-click operation of a mouse. [Figure 12] FIG. 12 is a flowchart showing an example of the operation of the response time determination process corresponding to the operation event in FIG. 10 when the operation event is a close button pressing operation event. [Figure 13] FIG. 13 is a flowchart showing an example of the wait display detection processing operation in FIG. [Figure 14] FIG. 14 is a flowchart showing an example of the wait determination process operation in FIG. [Figure 15] FIG. 15 is a flowchart showing an example of the bodily sensation calculation processing operation in FIG. [Figure 16] FIG. 16 is a flowchart showing an example of the intent derivation processing operation in FIG. [Figure 17] FIG. 17 is a schematic diagram showing a display example at time t0 when a VDI application is executed. [Figure 18] FIG. 18 is a schematic diagram showing a display example at time t0+3 when a VDI application is executed. [Figure 19] FIG. 19 is a schematic diagram showing a display example from time t0+3 to time t0+10 when a VDI application is executed. [Figure 20] FIG. 20 is a schematic diagram showing a display example from time t0+15 to time t0+20 when a VDI application is executed. [Figure 21] FIG. 21 is a schematic diagram showing a display example at time t0 when a Web browsing application is executed while an E-commerce service is being used. [Figure 22] FIG. 22 is a schematic diagram showing a display example from time t0 to time t0+5 when a Web browsing application is executed. [Diagram 23] FIG. 23 is a schematic diagram showing a display example at time t0+6 when a Web browsing application is executed. [Figure 24] FIG. 24 is a schematic diagram showing a display example at time t0 when a gaming application is executed. [Diagram 25] FIG. 25 is a schematic diagram showing a display example at time t0+3 when a gaming application is executed. [Figure 26] FIG. 26 is a schematic diagram showing a display example from time t0+7 ​​to time t0+10 when a gaming application is executed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] [One embodiment] An embodiment of the present invention will now be described with reference to the drawings.

[0013] (Configuration example) 1 is a block diagram showing an example of the configuration of an intent extraction device 10 to which a quality of experience calculation device 1 according to an embodiment of the present invention is applied. The quality of experience calculation device 1 includes an operation event acquisition unit 11, an abnormal operation detection unit 12, an application screen acquisition unit 13, a subject detection unit 14, a response time determination unit 15, a wait display detection unit 16, a wait determination unit 17, and a quality of experience calculation unit 18. The intent extraction device 10 includes the quality of experience calculation device 1 and an intent derivation unit 19.

[0014] The operation event acquisition unit 11 has a function of acquiring a user's input event as an operation event. The input event includes, for example, a keyboard event, a mouse event, an operation of a gaming device such as a controller, etc., in an application for which the quality of experience is calculated and detected in an OS (Operation System). The keyboard event includes a downward key operation (KeyDown) event, a key press operation (KeyPress) event, an upward key operation (KeyUp), etc. The mouse event includes a mouse button click operation (MouseClick) event, a mouse button double click operation (DoubleClick) event, etc.

[0015] The abnormal operation detection unit 12 has a function of detecting, as an abnormal operation, an unusual behavior that is often performed when a user feels a deterioration in physical sensation. Such behavior includes, for example, repeated tapping of the mouse or keyboard in a short period of time, repeated tapping of a specific key combination such as pressing the control ("Ctrl") key and the "S" key simultaneously (Ctrl+S), successive launching and closing of apps, etc. Abnormal operations detected by the abnormal operation detection unit 12 may be customized according to the characteristics of the apps, in addition to such general-purpose combinations of operations.

[0016] The application screen acquisition unit 13 has a function of acquiring an application screen, which is a display screen of a target application, as a moving image.

[0017] The subject detection unit 14 has a function of detecting a subject (for example, a window, a product subject, an explanatory text, etc.) on the screen by performing image analysis on a series of application screens acquired as a video by the application screen acquisition unit 13. The subject detection unit 14 can detect the detection time, position, and size of the subject. This subject detection unit 14 can be realized by utilizing or retraining an existing object detection library available in the market.

[0018] The response time judgment unit 15 has a function of measuring the time from the occurrence of an operation event leading to a screen transition, such as a mouse double click event, to the change of the subject of the screen as a response time, based on the subject detected by the subject detection unit 14 and the operation event acquired by the operation event acquisition unit 11. Note that the response time judgment unit 15 is based on the following assumptions. · The app screen transitions in response to user operations (double-clicking an icon on the screen (DoubleClick), clicking a mouse to press a button on the screen (MouseClick), etc.). The operations that lead to screen transitions may vary depending on the type of app. For example, in the case of a gaming app, events such as pressing keys to move left or right also lead to screen transitions and can be treated as operation events. The operations that the response time determination unit 15 targets can be customized depending on the application.

[0019] The wait display detection unit 16 has a function of detecting the occurrence of a wait display on the screen by performing image analysis on a series of application screens acquired as a video by the application screen acquisition unit 13. The wait display detection unit 16 can detect the detection time of the occurrence of a wait display, and the position and size of the wait display. This wait display detection unit 16 can be realized by utilizing or retraining an existing object detection library available on the market.

[0020] The waiting determination unit 17 has a function of calculating a waiting score based on the presence or absence of a waiting display from the waiting display detection unit 16 and the duration of the waiting display.

[0021] The experience calculation unit 18 has a function of calculating the user experience quality by applying weighting to the presence or absence of an abnormal operation detected by the abnormal operation detection unit 12 and the degree of the abnormality, the response time determined by the response time determination unit 15, and the waiting score calculated by the waiting determination unit 17.

[0022] The intent derivation unit 19 has a function of generating quality of experience requirements (intents) based on the user quality of experience calculated by the experience calculation unit 18.

[0023] FIG. 2 is a diagram illustrating an example of a hardware configuration of the intent extraction device 10. As shown in FIG.

[0024] The intent extraction device 10 is configured by a computer such as a personal computer or a smartphone, which is a user terminal at which a user uses a service. As shown in FIG. 2, the intent extraction device 10 has a hardware processor 100 such as a CPU (Central Processing Unit). The CPU can execute multiple information processes simultaneously by using a multi-core and multi-threaded one. The processor 100 may also include multiple CPUs. In the intent extraction device 10, a program memory 200, a data memory 300, a communication interface 400, and an input / output interface (referred to as input / output IF in FIG. 2) 500 are connected to the processor 100 via a bus 600.

[0025] The communication interface 400 may include, for example, one or more wired or wireless communication modules. The communication interface 400 may communicate with other computers, servers, or other data processing devices that provide services, which are connected via a cable or a network such as a LAN (Local Area Network) or the Internet.

[0026] The input unit 700 and the display unit 800 are connected to the input / output interface 500. The input unit 700 is an input device such as a pointing device such as a keyboard or a mouse, or a gaming device such as a controller. The display unit 800 is a display device such as a liquid crystal display or a CRT (Cathode Ray Tube) display. The input unit 700 and the display unit 800 may be a so-called tablet-type input / display device in which an input detection sheet using an electrostatic method or a pressure method is arranged on a display screen of a display device using, for example, liquid crystal or organic EL (Electro Luminescence). The input / output interface 500 inputs operation information inputted in the input unit 700 to the processor 100, and causes the display unit 800 to display display information generated by the processor 100.

[0027] The input unit 700 and the display unit 800 do not have to be connected to the input / output interface 500. The input unit 700 and the display unit 800 can transmit and receive information to and from the processor 100 by including a communication unit for connecting to the communication interface 400 directly or via a network.

[0028] The input / output interface 500 may also have a function for reading / writing a recording medium such as a semiconductor memory such as a flash memory, or may have a function for connecting to a reader / writer having a function for reading / writing such a recording medium. Furthermore, the input / output interface 500 may have a function for connecting to other devices.

[0029] The program memory 200 is a non-transient tangible computer-readable storage medium that is a combination of a non-volatile memory that can be written and read at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), and a non-volatile memory such as a ROM. The program memory 200 stores programs required for the processor 100 to execute various control processes according to an embodiment, such as an intent extraction program including a quality of experience calculation program. That is, the processing function units in each of the above-mentioned operation event acquisition unit 11, abnormal operation detection unit 12, application screen acquisition unit 13, subject detection unit 14, response time determination unit 15, wait display detection unit 16, wait determination unit 17, quality of experience calculation unit 18, and intent derivation unit 19 can all be realized by having the processor 100 read and execute the intent extraction program including the quality of experience calculation program stored in the program memory 200. It should be noted that some or all of these processing functions may be realized in a variety of other forms, including integrated circuits such as an Application Specific Integrated Circuit (ASIC) or a field-programmable gate array (FPGA).

[0030] The data memory 300 is a tangible computer-readable storage medium, for example, a combination of the above-mentioned nonvolatile memory and a volatile memory such as a RAM (Random Access Memory). The data memory 300 is used to store various data acquired and created in the process of performing various processes. That is, in the data memory 300, an area for appropriately storing various data in the process of performing various processes is secured. As such areas, the data memory 300 may be provided with, for example, a setting value storage unit 301, an operation event recording unit 302, an abnormality recording unit 303, an application screen recording unit 304, a subject recording unit 305, a response time recording unit 306, a waiting display recording unit 307, a waiting recording unit 308, a quality of experience recording unit 309, an intent storage unit 310, and a temporary storage unit 311.

[0031] The setting value storage unit 301 is used to store various setting values ​​used when the processor 100 executes the intent extraction program including the quality of experience calculation program. The setting values ​​include, for example, a time setting value used when the processor 100 executes the operations as the abnormal operation detection unit 12, the response time determination unit 15, the wait determination unit 17, etc., and a threshold value used when the processor 100 operates as the intent derivation unit 19. These setting values ​​can be set in advance by the user. Furthermore, if the intent extraction program including the quality of experience calculation program has these setting values ​​as default values, they can be set in the setting value storage unit 301 when the program starts to be executed.

[0032] The operation event recording unit 302 is used to record the operation event acquired when the processor 100 operates as the above-mentioned operation event acquiring unit 11 .

[0033] The abnormality recording section 303 is used to record abnormal operations detected when the processor 100 operates as the abnormal operation detection section 12, such as the presence or absence of abnormal operation detection, the degree of abnormality, and the like.

[0034] The application screen recording unit 304 is used to record the application screen acquired as a moving image when the processor 100 operates as the application screen acquisition unit 13 described above.

[0035] The subject recording section 305 is used to record the detection time, position and size of a subject detected when the processor 100 operates as the subject detection section 14 described above.

[0036] The response time recording unit 306 is used to record the response time measured when the processor 100 operates as the response time determining unit 15 described above.

[0037] The wait display recording section 307 is used to record the detection time of the occurrence of a wait display, the position and the size of the wait display, which are detected when the processor 100 operates as the wait display detection section 16 described above.

[0038] The waiting recorder 308 is used to record the waiting score calculated when the processor 100 operates as the waiting determiner 17 described above.

[0039] The quality of experience recording unit 309 is used to record the user quality of experience calculated when the processor 100 operates as the quality of experience calculation unit 18 described above.

[0040] The intent storage unit 310 is used to store quality of experience requirements (intents) generated when the processor 100 operates as the intent derivation unit 19 described above.

[0041] The temporary memory unit 311 is used to store various intermediate data that is generated during operation of the processor 100 and is not stored in the setting value memory unit 301, the operation event recording unit 302, the abnormality recording unit 303, the subject recording unit 305, the response time recording unit 306, the waiting display recording unit 307, the waiting recording unit 308, the quality of experience recording unit 309, and the intent memory unit 310.

[0042] (operation) Next, the operation of the intent extraction device 10 having the quality of experience calculation device 1 will be described.

[0043] 3 is a flowchart showing an example of an intent extraction processing operation by the intent extraction device 10. The processor 100 of the intent extraction device 10 starts the operation of the intent extraction device 10 shown in this flowchart by reading and executing an intent extraction program including a quality of experience calculation program stored in the program memory 200. Execution of the intent extraction program in the processor 100 is started when an instruction to perform intent extraction is received from the input unit 700 via the input / output interface 500 or from an external data processing device via the communication interface 400.

[0044] The processor 100 can operate as an operation event acquisition unit 11, an abnormal operation detection unit 12, an application screen acquisition unit 13, a subject detection unit 14, a response time judgment unit 15, a waiting display detection unit 16, a waiting judgment unit 17, a bodily sensation calculation unit 18, and an intent derivation unit 19, each of which is an independent process.

[0045] That is, the processor 100 operates as the operation event acquisition unit 11 and performs an operation event acquisition process (step S11). The processor 100 also operates as the abnormal operation detection unit 12 and performs an abnormal operation detection process (step S12). The processor 100 also operates as the application screen acquisition unit 13 and performs an application screen acquisition process (step S13). The processor 100 also operates as the subject detection unit 14 and performs a subject detection process (step S14). The processor 100 also operates as the response time determination unit 15 and performs a response time determination process (step S15). The processor 100 also operates as the wait display detection unit 16 and performs a wait display process (step S16). The processor 100 also operates as the wait determination unit 17 and performs a wait determination process (step S17). The processor 100 also operates as the bodily sensation calculation unit 18 and performs a bodily sensation calculation process (step S18). Moreover, the processor 100 operates as the intent derivation unit 19 and performs an intent derivation process (step S19). The processor 100 executes the processes of steps S11 to S19 in parallel.

[0046] When any of the processes executing the processes of steps S11 to S19 ends, the processor 100 judges whether or not an instruction to end the intent extraction has been received from the input unit 700 via the input / output interface 500, or from an external data processing device via the communication interface 400 (step S20). If it is determined that an instruction to end has not been received, the processor 100 executes the process again.

[0047] Then, if it is determined that an instruction to end the intent extraction has been given in any process, the processor 100 ends the operation of the intent extraction device 10.

[0048] Next, the processing executed in each process will be described. FIG. 4 is a flowchart showing an example of an operation event acquisition processing operation executed as step S11 in FIG. 3. In this operation event acquisition processing, the processor 100 detects the presence or absence of a user input event at a current time t (step S111). Note that the current time t is the time at which this operation event acquisition processing operation is executed, and does not refer to the same time as the current time t in the description of each processing operation described later. This also applies to the description of each processing operation described later. If it is determined that there is no user input event, the processor 100 ends this operation event acquisition processing. On the other hand, if it is determined that there is a user input event, the processor 100 acquires the user input event (step S112). Then, the processor 100 records the acquired user input event as an operation event in the operation event recording unit 302 of the data memory 300 (step S113). Then, the processor 100 ends this operation event acquisition processing.

[0049] Fig. 5 is a flowchart showing an example of an abnormal operation detection processing operation executed as step S12 in Fig. 3. In this abnormal operation detection processing, the processor 100 reads out an operation event at the current time t from the operation event recording unit 302 (step S121). Then, the processor 100 determines what the read operation event is (step S122). The processor 100 executes an operation event corresponding abnormal operation detection processing operation to detect an abnormal operation corresponding to the determined operation event (step S123). Then, when the operation event corresponding abnormal operation detection processing operation is completed, the processor 100 ends this abnormal operation detection processing operation.

[0050] Here, as an example of the operation of the operation event-responsive abnormal operation detection process in step S123, a case where an abnormal operation of repeated mouse taps is detected and a case where an abnormal operation of successive application launches and closures is detected will be described.

[0051] 6 is a flowchart showing an example of the operation event-responsive abnormal operation detection processing operation of step S123 when the determined operation event is a mouse button click operation (MouseClick) event. The processor 100 determines a click operation (MouseClick) event from among operation events recorded in the operation event recording unit 302, which are recorded between t-T1, which is a time T1 before the current time t, and the current time t (step S12311). The time t-T1 to the time t is a target time period for abnormal operation detection, and a specific value of T1 is stored in the set value storage unit 301 of the data memory 300. That is, T1 can be customized. Then, the processor 100 determines whether the number of times that the interval time of this click operation (MouseClick) event is less than m seconds is n or more (step S12312). The variables m and n are also customizable values ​​stored in the set value storage unit 301. If the processor 100 determines that the number of times that the interval time of the event is m seconds or less is less than n, the processor 100 ends the operation of the operation event-related abnormal operation detection process. On the other hand, if the processor 100 determines that the number of times that the interval time of the event is m seconds or less is n or more, the processor 100 records the operation as an operation abnormality in the abnormality recording unit 303 of the data memory 300 (step S12313). Then, the processor 100 ends the operation event-related abnormal operation detection process.

[0052] 7 is a flowchart showing an example of the operation event-responsive abnormal operation detection processing operation of step S123 when the determined operation event is a mouse button double click operation (DoubleClick) event or a close button press operation (CloseButtonDown) event which is a mouse button click operation on the close button. The processor 100 determines a double click operation (DoubleClick) event and a close button press operation (CloseButtonDown) event from among the operation events recorded in the operation event recording unit 302 during the period from time t-T1 to the current time t (step S12321). Then, the processor 100 determines whether the number of times of the double click operation (DoubleClick) event and the close button press operation (CloseButtonDown) event is n or more (step S12322). If it is determined that the number of times of the events is less than n, the processor 100 ends the operation event-responsive abnormal operation detection processing operation. On the other hand, if it is determined that the number of occurrences of those events is n or more, the processor 100 records it as an operation abnormality in the abnormality recording unit 303 of the data memory 300 (step S12323). Then, the processor 100 ends the operation of the operation event corresponding abnormal operation detection processing.

[0053] Fig. 8 is a flowchart showing an example of an application screen acquisition process executed as step S13 in Fig. 3. In this application screen acquisition process, the processor 100 acquires a screen image of a target application at a current time t (step S131). Then, the processor 100 adds this acquired screen image to a video, which is a series of screen images acquired up to the time t, and records the video in the application screen recording unit 304 of the data memory 300 (step S132). Then, the processor 100 ends this application screen acquisition process.

[0054] FIG. 9 is a flowchart showing an example of the subject detection processing operation executed as step S14 in FIG. 3. In this subject detection processing, the processor 100 reads out a screen image at the current time t and a screen image at the previous time t-1 from the application screen recording unit 304 (step S141). Here, the time interval between the current time t and the previous time t-1 is stored in the setting value storage unit 301 of the data memory 300. That is, one time interval can be customized. The processor 100 detects each subject from the read screen images at the times t and t-1 (step S142). Then, the processor 100 judges whether or not the subject has been changed, that is, whether or not the two detected subjects are different (step S143). If it is judged that there has been no subject change, the processor 100 ends this subject detection processing operation. On the other hand, if it is determined that the subject has been changed, the processor 100 records the subject detected from the screen image at the current time t, that is, the subject detection time, the subject's position and size, in the subject recording section 305 of the data memory 300 (step S144). Then, the processor 100 ends this subject detection processing operation.

[0055] Fig. 10 is a flowchart showing an example of the response time judgment processing operation executed as step S15 in Fig. 3. In this response time judgment processing, the processor 100 reads out an operation event at the current time t from the operation event recording unit 302 (step S151). Then, the processor 100 determines what the read operation event is (step S152). The processor 100 executes the operation event corresponding response time judgment processing operation to determine the response time corresponding to the determined operation event (step S153). Then, when the operation event corresponding response time judgment processing operation is completed, the processor 100 ends this response time judgment processing operation.

[0056] Here, as an example of the operation event response time determination processing operation of step S153 above, we will explain the case where the response time for a mouse button double click event (DoubleClick) and the case where the response time for a close button press event (CloseButtonDown) are determined.

[0057] 11 is a flowchart showing an example of the operation event response time determination process operation of step S153 when the determined operation event is a double-click operation (DoubleClick) event of a mouse button. The processor 100 reads out the subjects recorded in the subject recording unit 305 from the current time t to time t+T2, which is the time T2 after the current time t (step S15311). The time t to time t+T2 is the target time period for subject detection, and the specific value of T2 is stored in the set value storage unit 301 of the data memory 300. That is, T2 can be customized. Then, the processor 100 determines whether a new subject has appeared between the time t and time t+T2 (step S15312). This new subject is the subject (for example, a window, a product, a description, etc.) that is the transition destination specified by the double-click. This new subject may be any subject set in the set value storage unit 301. By specifying the type and subject of the operation event to be targeted based on the characteristics of the application, the accuracy of response time judgment can be improved. Then, when it is determined that a new subject has appeared, the processor 100 records the time interval L from the current time t to the appearance time of the new subject as the response time in the response time recording unit 306 of the data memory 300 (step S15313). On the other hand, when it is determined that a new subject has not appeared, the processor 100 records the response time as null in the response time recording unit 306 (step S15314). Thus, when the recording of the response time is completed in step S15313 or step S15314, the processor 100 terminates the operation event response time judgment processing operation.

[0058] 12 is a flowchart showing an example of the operation event response time determination process operation of step S153 when the determined operation event is a close button pressing operation (CloseButtonDown) event. The processor 100 reads out the subject recorded in the subject recording unit 305 from the current time t to time t+T2 (step S15321). Then, the processor 100 judges whether the subject at the current time t has disappeared between this time t and time t+T2 (step S15322). This subject at the current time t is the subject (for example, a window, a product, a description, etc.) that is specified to be closed by pressing the close button. This subject may also be an arbitrary subject set in the setting value storage unit 301. Then, when it is judged that the subject at the current time t has disappeared, the processor 100 records the time interval L from the current time t to the disappearance time of the subject as the response time in the response time recording unit 306 of the data memory 300 (step S15323). On the other hand, if it is determined that the subject at the current time t has not disappeared, the processor 100 records the response time as null in the response time recording unit 306 (step S15324). When the recording of the response time is thus completed in step S15323 or step S15324, the processor 100 ends the operation of this operation event response time determination process.

[0059] FIG. 13 is a flowchart showing an example of a wait display detection process executed as step S16 in FIG. 3. In this wait display detection process, the processor 100 reads out a screen image at the current time t from the application screen recording unit 304 (step S161). Then, the processor 100 detects a wait display from the read screen image at the time t (step S162). Here, the processor 100 judges whether or not a wait display has been detected (step S163). If it is judged that a wait display has not been detected, the processor 100 ends this wait display detection process. On the other hand, if it is judged that a wait display has been detected, the processor 100 records the wait display, that is, the detection time of the occurrence of the wait display, and the position and size of the wait display, in the wait display recording unit 307 of the data memory 300 (step S164). Then, the processor 100 ends this wait display detection process.

[0060] Fig. 14 is a flow chart showing an example of the waiting judgment processing operation executed in step S17 in Fig. 3. In this waiting judgment processing, the processor 100 refers to the waiting display recording unit 307 and detects the presence or absence of a waiting display at the current time t (step S171). The processor 100 also refers to the waiting display recording unit 307 and measures the duration from the current time t to time t-T3, which is the time T3 before the current time t, as the duration of the waiting display (step S172). The time t-T3 is the detection time of the oldest waiting display among the records of consecutive waiting displays up to the current time t in the waiting display recording unit 307. Then, based on the detected presence or absence of a waiting display and the measured duration of the waiting display, the processor 100 Waiting score = Waiting display status * Display duration (step S173). The processor 100 records the calculated waiting score in the waiting recording unit 308 of the data memory 300 (step S174). Then, the processor 100 ends this waiting determination processing operation.

[0061] FIG. 15 is a flowchart showing an example of the bodily sensation calculation process executed as step S18 in FIG. 3. In this bodily sensation calculation process, the processor 100 reads out the abnormal operation, the response time, and the waiting score recorded in the abnormality recording unit 303, the response time recording unit 306, and the waiting recording unit 308 at the current time t as indices (step S181). Then, the processor 100 quantifies the user's quality of experience by calculating the sum of each index according to the weight of each read index (step S182). The weight assigned to each index is stored in the set value storage unit 301. That is, the weight of each index can be set in advance for each application by an application developer or the like. The processor 100 records the calculated user's quality of experience in the bodily sensation recording unit 309 of the data memory 300 (step S183). After that, the processor 100 waits for a certain period of time (step S184). This certain period of time can be set in advance for each application by an application developer or the like, and is stored in the set value storage unit 301. After the waiting period ends, the processor 100 ends the body sensation calculation processing operation.

[0062] FIG. 16 is a flowchart showing an example of the intent derivation processing operation executed as step S19 in FIG. 3. The processor 100 reads out the user's quality of experience at the current time t from the quality of experience recording unit 309 (step S191). Then, the processor 100 judges whether the read out user's quality of experience exceeds a threshold (step S192). This threshold is stored in the set value storage unit 301. That is, the threshold can be set in advance for each application by an application developer or the like. When it is judged that the user's quality of experience does not exceed the threshold, the processor 100 ends this intent derivation processing operation. On the other hand, when it is judged that the user's quality of experience exceeds the threshold, the processor 100 judges that the user's quality of experience is deteriorated, and generates a quality of experience requirement (intent) requesting improvement of the experience (step S193). The processor 100 stores the generated intent in the intent storage unit 310 of the data memory 300 (step S194). Then, the processor 100 ends this intent derivation processing operation.

[0063] (Specific examples) Below, we will explain the specific operations using several apps as examples.

[0064] <VDIアプリ> First, a case where the target application is a VDI application will be described.

[0065] 17 is a schematic diagram showing a display example at time t0 during execution of a VDI application. A plurality of desktop icons 702 indicating connectable desktops are arranged on a VDI application screen 701. When a mouse cursor 703 is moved to any desktop icon 702 and the corresponding desktop is designated by a double-click operation, the processor 100 operating as the operation event acquisition unit 11 acquires a mouse button double-click operation (DoubleClick) event as an operation event at this time t0.

[0066] 18 is a schematic diagram showing a display example at time t0+3 when a VDI application is executed. Time t0+3 is three time intervals after time t0. The VDI application connects to the specified desktop, and at this time t0+3, displays a window 704 for displaying the desktop. At this time t0+3, the processor 100 operating as the subject detection unit 14 detects a new window 704 as the changed subject. Then, the processor 100 operating as the response time determination unit 15 determines that the delay "3" from the time t0 when the double click was made to the time t0+3 when the window 704, which is the new subject, was opened is the response time, and records it in the response time recording unit 306.

[0067] FIG. 19 is a schematic diagram showing a display example at times t0+3 to t0+10 when a VDI application is executed. Time t0+10 is 10 time intervals after time t0. Even if a VDI application opens a window 704 at time t0+3, there are cases where the specified desktop cannot be displayed immediately in the window 704. In such a case, a wait display 705 is displayed. FIG. 19 shows a case where the wait display 705 remains even at time t0+10. In each of the time intervals from time t0+3 to time t0+10, the processor 100 operating as the wait display detection unit 16 detects this wait display 705, and the processor 100 operating as the wait determination unit 17 measures the duration of the wait display and records the wait score in the wait recording unit 308. For example, at time t0+10, the wait display 705 continues for seven time intervals from time t0+3 to time t0+10, so the wait display duration is measured as "7".

[0068] 20 is a schematic diagram showing a display example from time t0+15 to t0+20 when a VDI application is executed. For example, assume that the mouse is clicked continuously between time t0+15 and t0+20. In such a case, the processor 100 operating as the operation event acquisition unit 11 detects a click operation (MouseClick) event of the mouse button between time t0+15 and t0+20, and the processor 100 operating as the abnormal operation detection unit 12 determines that the number of click operation (MouseClick) events with an interval time of m seconds or less is n or more times, and records the operation abnormality in the abnormality recording unit 303.

[0069] Furthermore, the processor 100 operating as the experience calculation unit 18 calculates the user quality of experience from abnormal operations, response times, and waiting scores at regular time intervals (for example, at intervals of five time intervals) during the time period from time t0 to time t0+20. The processor 100 operating as the intent derivation unit 19 then compares this user quality of experience with a set threshold, and if it exceeds the threshold, i.e., if the user quality of experience is determined to be deteriorated, generates a quality of experience requirement (intent) requesting improvement of the experience.

[0070] <Webブラウジングアプリ> Next, a case where the target application is a web browsing application will be described.

[0071] 21 is a schematic diagram showing a display example at time t0 when a Web browsing application is executed while an E-commerce service is being used. A browser screen 706 for Web browsing displays a plurality of product icons 707 to which hyperlinks to product pages for presenting details of each product are assigned. When a mouse cursor 703 is moved to any one of the product icons 707 and a hyperlink to the product page of the corresponding product is selected by a click operation, the processor 100 operating as the operation event acquisition unit 11 acquires a mouse hyperlink click operation (MouseHyperlinkClick) event as an operation event at this time t0.

[0072] FIG. 22 is a schematic diagram showing a display example at times t0 to t0+5 when a Web browsing application is executed. Time t0+5 is five time intervals after time t0. Even if a hyperlink is selected at time t0, the Web browsing application may not be able to immediately display the linked page on the browser screen 706. In such a case, a waiting display 705 is displayed. FIG. 20 shows a case where the waiting display 705 remains even at time t0+5. In each of the time intervals from time t0 to time t0+5, the processor 100 operating as the waiting display detection unit 16 detects the waiting display 705, and the processor 100 operating as the waiting determination unit 17 measures the duration of the waiting display and records the waiting score in the waiting recording unit 308. For example, at time t0+5, the waiting display 705 continues for five time intervals from time t0 to time t0+5, so the waiting display duration is measured as "5".

[0073] FIG. 23 is a schematic diagram showing a display example at time t0+6 when a Web browsing application is executed. Time t0+6 is six time intervals after time t0. The Web browsing application displays the relevant product page at time t0+6 after the wait display 705 from time t0 to time t0+5. In such a case, at this time t0+6, the processor 100 operating as the subject detection unit 14 detects the product 708 and the description 709 as the changed subject. In this way, the subject may include multiple displays. Then, the processor 100 operating as the response time determination unit 15 can determine the delay "6" from time t0 when the product icon 707 is clicked to time t0+6 when the new page is opened as the response time, and record it in the response time recording unit 306.

[0074] The processor 100 operating as the experience calculation unit 18 calculates the user quality of experience from abnormal operations, response times, and waiting scores at regular time intervals (e.g., at intervals of five time intervals) during the time period from time t0 to time t0+6. The processor 100 operating as the intent derivation unit 19 then compares this user quality of experience with a set threshold, and if it exceeds the threshold, i.e., if the user quality of experience is determined to be deteriorated, generates a quality of experience requirement (intent) requesting improvement of the experience.

[0075] <Gaming App> Next, a case where the target application is a gaming application will be described.

[0076] 24 is a schematic diagram showing a display example at time t0 when a gaming application is executed. Car 711, which is the object to be operated, is displayed on gaming application screen 710. Since this display example is during a game, processor 100 operating as subject detection unit 14 has already detected car 711 as the subject. Here, when right direction button 713 of controller 712 is operated as an instruction to move car 711 to the right, processor 100 operating as operation event acquisition unit 11 acquires a right direction button press (RightButtonDown) event as an operation event at this time t0.

[0077] 25 is a schematic diagram showing a display example at time t0+3 when a gaming application is executed. The gaming application displays car 711 moving to the right. At this time t0+3, processor 100 operating as subject detection unit 14 detects a change in the position of car 711, which is the subject, as the changed subject. Then, processor 100 operating as response time determination unit 15 determines that the delay from time t0, when right direction button 713 of controller 712 is operated, to time t0+3, when the subject's position is updated, is "3" as the response time, and records it in response time recording unit 306.

[0078] FIG. 26 is a schematic diagram showing a display example at times t0+7 ​​to t0+10 when a gaming application is executed. Time t0+7 ​​is the time seven time intervals after time t0. Even if there is some operation from controller 712, the gaming application may not be able to immediately update the display to a display corresponding to the operation, for example, to display car 711 moving. In such a case, waiting display 705 is displayed. FIG. 26 shows a case where the waiting display is displayed from time t0+7, and remains as waiting display 705 even at time t0+10. In each of the time intervals between time t0+7 ​​and time t0+10, processor 100 operating as waiting display detection unit 16 detects this waiting display 705, and processor 100 operating as waiting determination unit 17 measures the duration of the waiting display and records the waiting score in waiting recording unit 308. For example, at time t0+10, the wait display 705 continues for three time intervals from time t0+7 ​​to time t0+10, so the wait display duration is measured as "3".

[0079] Furthermore, the processor 100 operating as the experience calculation unit 18 calculates the user quality of experience from abnormal operations, response times, and waiting scores at regular time intervals (for example, at intervals of five time intervals) during the time period from time t0 to time t0+10. The processor 100 operating as the intent derivation unit 19 then compares this user quality of experience with a set threshold, and if it exceeds the threshold, i.e., if the user quality of experience is determined to be deteriorated, generates a quality of experience requirement (intent) requesting improvement of the experience.

[0080] The quality of experience calculation device 1 according to the embodiment described above acquires an operation event of a user who uses an application for which the quality of experience is to be calculated by using the operation event acquisition unit 11, and acquires a series of application screens provided by the application by using the application screen acquisition unit 13. Furthermore, the quality of experience calculation device 1 extracts a delay time from the acquisition of the operation event acquired by the operation event acquisition unit 11 to the change to an application screen responding to the operation event by performing image analysis of the application screen acquired by the application screen acquisition unit 13 by using the subject detection unit 14 and the response time determination unit 15 functioning as delay time extraction units, and determines whether the application screen is a waiting screen by performing image analysis of the application screen acquired by the application screen acquisition unit 13 by using the wait display detection unit 16 and the wait determination unit 17 functioning as wait screen determination units. Then, the quality of experience calculation device 1 calculates the quality of experience of the user by using the experience calculation unit 18 based on the delay time and the wait determination results. Therefore, since the quality of experience calculation device 1 according to an embodiment calculates the user's quality of experience using generic data, such as the operation event and the application screen, it is possible to provide a quality of experience calculation technique that can calculate a quality of experience that is accurate, that is, close to the actual user experience, without redesigning the experience measurement method even if the application to be calculated for the quality of experience is different. In addition, since the calculation basis of the user's quality of experience is extracted from the application screen, it is possible to improve the interpretability of the calculated user's experience. Furthermore, it is possible to calculate a generic user quality of experience in various operation applications. Then, by inputting the calculated user's quality of experience to the intent derivation unit 19 of the intent extraction device 10, the intent derivation unit 19 can generate a quality of experience requirement (intent) based on the user's quality of experience. Therefore, it is possible to extract intent in various operation applications.

[0081] Furthermore, in the quality of experience calculation device 1 according to an embodiment, the abnormal operation detection unit 12 detects whether the operation event acquired by the operation event acquisition unit 11 corresponds to an abnormal operation behavior indicating deterioration of the user's quality of experience, and the quality of experience calculation unit 18 calculates the user's quality of experience based on the abnormal operation behavior detected by the abnormal operation detection unit 12 in addition to the determination results of the delay time and the waiting screen. In this way, by taking abnormal operation behavior into consideration, it is possible to calculate a more accurate quality of experience, that is, a quality of experience that is closer to the actual user experience.

[0082] Here, the quality of experience calculation unit 18 calculates the user's quality of experience by weighting each of the delay time, the determination result of the standby screen, and the abnormal operating behavior, and calculating the sum. Therefore, the importance of each of the delay time, the judgment result of the wait screen, and the abnormal operating behavior can be adjusted to calculate the user's quality of experience.

[0083] The quality of experience calculation device 1 according to an embodiment further includes a setting value storage unit 301 that stores weights for the delay time, the determination result of the wait screen, and the abnormal operation behavior for each of the multiple applications, and the quality of experience calculation unit 18 uses a weight corresponding to the application for which the quality of experience is to be calculated, from among the multiple weights stored in the setting value storage unit 301. In this way, since the emphasis placed on the delay time, the judgment result of the waiting screen, or the abnormal operation behavior differs for each application, by setting this weight for each application, the optimal weight can be used.

[0084] The delay time extraction unit also includes a subject detection unit 14 and a response time determination unit 15. The subject detection unit 14 detects a subject that is the user's main focus on the application screen by image analysis of the application screen acquired by the application screen acquisition unit 13, and the response time determination unit 15 measures the time from the acquisition of the operation event acquired by the operation event acquisition unit 11 to the change in the subject detected by the subject detection unit 14 as the delay time. In this way, the quality degradation duration, which is the time from the acquisition of the operation event to the change of the subject, can be measured as the delay time. The user experience calculation unit 18 calculates the user experience on such a screen by taking into account the user's interest level and the quality degradation duration, thereby making it possible to calculate a user experience that is closer to a real user experience.

[0085] The waiting time extraction unit also includes a waiting display detection unit 16 and a waiting judgment unit 17. The waiting display detection unit 16 detects a waiting display on the application screen by image analysis of the application screen acquired by the application screen acquisition unit 13, and the waiting judgment unit 17 calculates a waiting score indicating the judgment result of the waiting screen based on the presence or absence of a waiting display and the duration of the waiting display detected by the waiting display detection unit 16. In this way, the waiting score can be calculated based on the quality degradation duration, which is the duration of the waiting display. The user experience calculation unit 18 calculates the user experience taking into account such a quality degradation duration, thereby calculating a user experience that is closer to a realistic user experience.

[0086] [Other embodiments] It should be noted that the present invention is not limited to the above embodiment. For example, the flow of each process described above is not limited to the procedures described, and the order of some steps may be changed, or some steps may be performed simultaneously in parallel.

[0087] Furthermore, the intent extraction device 10 does not necessarily have to include the quality of experience calculation device 1. That is, it may be configured separately as a computer constituting the quality of experience calculation device 1 and a computer functioning as the intent derivation unit 19. The computer functioning as the intent derivation unit 19 may be configured as a data processing device providing a service such as a server that can be accessed by multiple quality of experience calculation devices 1 via a network.

[0088] The method described in the embodiment can be stored as a program (software means) that can be executed by a calculator (computer) on a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), an optical disk (CD-ROM, DVD, MO, etc.), a semiconductor memory (ROM, RAM, flash memory, etc.), and can also be distributed by transmitting it via a communication medium. The program stored on the medium side also includes a setting program that configures the software means (including not only execution programs but also tables and data structures) that the calculator executes within the computer. The computer that realizes this device reads the program recorded on the recording medium, and in some cases, configures the software means by the setting program, and executes the above-mentioned processing by controlling the operation of the software means. The recording medium referred to in this specification is not limited to a recording medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory provided inside the computer or in a device connected via a network.

[0089] In short, this invention is not limited to the above-mentioned embodiment, and various modifications can be made in the implementation stage without departing from the gist of the invention. Moreover, each embodiment may be implemented in combination as appropriate as possible, and in that case, the combined effect can be obtained. Furthermore, the above-mentioned embodiment includes inventions at various stages, and various inventions can be extracted by appropriate combinations of the disclosed constituent elements. [Explanation of symbols]

[0090] 1…Quality of experience calculation device 10…Intent extraction device 11...Operation event acquisition section 12…Abnormal operation detection section 13…App screen capture section 14...Subject detection unit 15...Response time judgment section 16...Waiting display detection section 17...Waiting decision section 18…Experience calculation section 19…Intent derivation part 100…Processor 200…Program memory 300…Data memory 301...Setting value storage section 302...Operation event recording section 303…Abnormal Recording Section 304…Application screen recording section 305…Subject Records Division 306…Response time recorder 307... Waiting display recorder 308... Waiting Recording Section 309…Quality of Experience Recording Section 310...Intent storage unit 311…Temporary storage unit 400…Communication interface 500... Input / Output Interface 600…Bus 700…Input section 701…VDI app screen 702…Desktop icon 703…Mouse cursor 704…Window 705...Waiting display 706…Browser screen 707…Product icon 708...Product 709…Description 710…Gaming app screen 711…Car 712…Controller 713…Right button 800...Display section

Claims

1. an operation event acquisition unit that acquires an operation event of a user who uses an application for which a quality of experience is to be calculated; a screen capture unit for capturing a series of application screens provided by the application; a delay time extraction unit that extracts a delay time from a time point when the operation event is acquired by the operation event acquisition unit to a time point when the application screen is changed to a time point when the application screen is changed in response to the operation event by performing an image analysis on the application screen acquired by the screen acquisition unit; a standby screen determination unit that determines whether the application screen acquired by the screen acquisition unit is a standby screen by image analysis of the application screen; a quality of experience calculation unit that calculates a quality of experience of the user based on the delay time extracted by the delay time extraction unit and a determination result of the waiting screen by the waiting screen determination unit; Equipped with the determination result of the standby screen by the standby screen determination unit includes information indicating whether the application screen is a standby screen or not, and a duration for which the application screen is a standby screen. Quality of experience calculation device.

2. The device further includes an abnormal operation detection unit that detects whether the operation event acquired by the operation event acquisition unit corresponds to an abnormal operation behavior indicating a deterioration in a physical sensation of the user, The quality of experience calculation device according to claim 1 , wherein the quality of experience calculation unit calculates the quality of experience of the user based on the abnormal operation behavior detected by the abnormal operation detection unit in addition to the delay time and the determination result of the waiting screen.

3. The quality of experience calculation device according to claim 2 , wherein the experience calculation unit calculates the quality of experience of the user by weighting the delay time, the determination result of the waiting screen, and the abnormal operation behavior, respectively, and calculating a sum of the weighted values.

4. A storage unit is further provided for storing the delay time, the determination result of the waiting screen, and the weight of each of the abnormal operation behaviors for each of a plurality of applications, The quality of experience calculation device according to claim 3 , wherein the quality of experience calculation unit uses a weighting factor corresponding to the application for which the quality of experience is to be calculated, from among a plurality of weighting factors stored in the storage unit.

5. The delay time extraction unit a subject detection unit that detects a subject on the application screen by image analysis of the application screen acquired by the screen acquisition unit; a response time determination unit that measures, as the delay time, a time from a point in time when the operation event acquisition unit acquires the operation event to a point in time when the subject detection unit detects a change in the subject; The quality of experience calculation device according to claim 1 , further comprising:

6. The waiting time extraction unit a wait display detection unit that detects a wait display on the application screen by image analysis of the application screen acquired by the screen acquisition unit; a wait determination unit that calculates a wait score indicating a determination result of the wait screen based on the presence or absence of the wait display and the duration of the wait display detected by the wait display detection unit; The quality of experience calculation device according to claim 1 , further comprising:

7. A quality of experience calculation method in a quality of experience calculation device having a processor and calculating a quality of experience of a user who uses an application, comprising: acquiring an operation event of the user by the processor; The processor acquires a series of application screens provided by the application; The processor performs an image analysis on the acquired application screen to extract a delay time from a time point when the acquired operation event is acquired until the application screen is changed to a time point when the application screen is changed to a time point when the operation event is acquired; The processor performs an image analysis on the acquired application screen to determine whether the application screen is a standby screen; The processor calculates the user's quality of experience based on the delay time and the determination result of the wait screen. Prepare for this. the determination result of the standby screen includes information indicating whether the application screen is a standby screen or not, and a duration for which the application screen is a standby screen. Method for calculating quality of experience.

8. A quality of experience calculation program that causes a processor to function as each of the components of the quality of experience calculation device according to claim 1 .

Images