Quality of experience visualization device, quality of experience visualization method, and quality of experience visualization program

Abstract

To enable appropriate measures to be taken when communication quality deteriorates.SOLUTION: An experience quality visualization device 100 calculates a score indicating user satisfaction with an online conference on the basis of at least one of information regarding communication performance related to the online conference and information regarding the performance of terminal devices used for the online conference, and a learning model that performs a predetermined classification of the online conference. The experience quality visualization device 100 generates recommendation information for improving the calculated score indicating user satisfaction with the online conference.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a quality of experience visualization device, a quality of experience visualization method, and a quality of experience visualization program. [Background technology]

[0002] With recent improvements in Internet technology, online conferences are sometimes held. Online conferences are realized based on technology that connects terminal devices of multiple users via applications installed on the terminal devices and transmits audio, video, and the like in both directions. As described above, online conferences are realized by connecting terminal devices via the Internet, and therefore the quality of the online conference may be affected by the communication environment.

[0003] Therefore, a technique is known for visualizing the quality of an online conference by visualizing the communication speed during the online conference. For example, a conventional technique is known in which the communication speed for transmitting and receiving data between multiple terminal devices and the communication speed during communication are acquired, and the acquired communication speed is compared with a threshold value to display the communication quality based on a preset display method (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2018-023115 Summary of the Invention [Problem to be solved by the invention]

[0005] However, conventional technologies have a problem in that it is difficult to appropriately deal with degradation of communication quality. For example, conventional technologies are technologies for determining whether the communication speed of sending and receiving video, audio, etc. when a user communicates using a terminal device is of sufficient quality, but they only display the communication speed in a predetermined format, making it difficult to identify the cause of degradation of communication quality. [Means for solving the problem]

[0006] Therefore, in order to solve the above-mentioned problems and achieve the objectives, the quality of experience visualization device of the present invention is characterized by having an inference unit that calculates a score indicating user satisfaction with the online conference based on at least one of information regarding communication performance related to the online conference and information regarding the performance of a terminal device used for the online conference, and a learning model that performs a predetermined classification of the online conference, and a generation unit that generates recommendation information to improve the score indicating user satisfaction with the online conference calculated by the inference unit. [Effects of the Invention]

[0007] The present invention has the effect of enabling appropriate measures to be taken against degradation of communication quality. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an overview of the processing performed by the quality of experience visualization device according to this embodiment. [Figure 2] FIG. 2 is a diagram illustrating the quality of experience visualization process according to this embodiment. [Figure 3] FIG. 3 is a diagram showing the configuration of a quality of experience visualization device according to this embodiment. [Figure 4] FIG. 4 is a table diagram showing an example of conference history information according to this embodiment. [Figure 5] FIG. 5 is a table diagram showing an example of terminal information according to this embodiment. [Figure 6]FIG. 6 is a table showing an example of feedback information according to this embodiment. [Figure 7] FIG. 7 is a table illustrating an example of quality of experience score information according to this embodiment. [Figure 8] FIG. 8 is a table diagram showing an example of action information according to this embodiment. [Figure 9] FIG. 9 is a diagram showing an example of conference history information related to the online conference tool according to the present embodiment. [Figure 10] FIG. 10 is a diagram showing an example of an output screen according to this embodiment. [Figure 11] FIG. 11 is a diagram showing an example of an output screen according to this embodiment. [Figure 12] FIG. 12 is a diagram showing an example of an output screen according to this embodiment. [Figure 13] FIG. 13 is a diagram showing an example of an output screen according to this embodiment. [Figure 14] FIG. 14 is a diagram showing an example of the learning process according to this embodiment. [Figure 15] FIG. 15 is a diagram illustrating an example of the quality of experience score calculation process according to this embodiment. [Figure 16] FIG. 16 is a flowchart showing the learning process according to this embodiment. [Figure 17] FIG. 17 is a flowchart showing the quality of experience visualization process according to this embodiment. [Figure 18] FIG. 18 is a diagram illustrating an example of a computer that executes the quality of experience visualization process according to this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention (hereinafter referred to as "embodiments") will be described with reference to the drawings. Note that the embodiments are not limited to the following description.

[0010] <Overview> Fig. 1 is a diagram illustrating an overall view of the processing of a quality of experience visualization device 100 according to this embodiment. The quality of experience visualization device 100 shown in Fig. 1 is an example of a computer that provides technology for visualizing the quality of an online conference experienced by a user of an online conference tool as a predetermined score, recommending problems, and implementing countermeasures.

[0011] In the following description of the present embodiment, the "quality of an online conference experienced by a user using an online conference tool" may be referred to as "quality of experience." Also, the "predetermined score indicating the level of quality of experience" may be referred to as "quality of experience score."

[0012] (background) With recent improvements in Internet technology, online conferences are sometimes held based on technology that connects multiple users' terminal devices via applications installed on the terminal devices and transmits audio, video, and other information in both directions.

[0013] The quality of online meetings can be affected by the communication environment. Therefore, if the quality of an online meeting declines, users of online meeting tools and information system personnel at companies are required to understand the situation and take steps to improve quality, but this can be difficult.

[0014] For example, employees (hereinafter, sometimes referred to simply as "users") who work in a hybrid work environment, combining in-office and remote work, and who participate in meetings using online conferencing tools, may participate in online meetings from various locations. However, it is difficult for such users to grasp the communication conditions at the locations where they participate in online meetings. Therefore, even if users experience delays in communication or frequent disconnections that prevent them from having a comfortable online meeting, it is difficult for them to easily understand the cause. Furthermore, even if a user feels that the operation of an online conferencing tool is slow (i.e., "poor quality of experience"), it is difficult for the user to determine for themselves whether their own quality of experience is better or worse than that of other users.

[0015] Meanwhile, corporate information system personnel (hereafter referred to simply as "administrators") who manage online conferencing tools manage the company's internal information systems, including the network, and manage the online conferencing tools used throughout the company. However, even when monitoring the network and systems, it is difficult for administrators to determine whether users are able to participate in online conferences comfortably (i.e., whether the "quality of experience" is good). Furthermore, because quality of experience is qualitative information perceived by users, it is difficult for administrators to obtain quantitative information about quality of experience, making it difficult to provide appropriate responses to user inquiries.

[0016] Therefore, a reference technology is known that visualizes the quality of an online conference by visualizing the communication speed during the online conference. For example, a reference technology is known that acquires the communication speed for transmitting and receiving data between multiple terminal devices and the communication speed during communication, compares the acquired communication speed with a threshold, and displays the communication quality based on a preset display method.

[0017] However, the above-mentioned reference technology has a problem in that it is difficult to appropriately deal with degradation of communication quality. For example, the reference technology can grasp whether the communication speed of the terminal device for sending and receiving video, audio, etc. is of sufficient quality, but only displays the communication speed in a predetermined format, making it difficult to identify the cause of degradation of communication quality.

[0018] That is, the reference technology can visualize quantitative values ​​such as communication speed, but cannot visualize qualitative indicators such as the user's "quality of experience." Also, the reference technology can visualize the phenomenon of a decrease in communication speed, but cannot provide detailed information on the cause of the phenomenon. Therefore, even when a user inquires about poor response of an online conferencing tool, the administrator cannot grasp who specifically is experiencing the inconvenience and in what situation, making it difficult to take measures.

[0019] (Overview of processing by the quality of experience visualization device 100) Therefore, the quality of experience visualization device 100 inputs information about communication performance related to the online conference and information about the performance of the terminal device used for the online conference into a trained learning model and calculates a score indicating user satisfaction with the online conference. Then, if the calculated score is below a predetermined threshold, the quality of experience visualization device 100 generates recommendation information based on information about the performance of the online conference or the terminal device. Now, returning to FIG. 1, the quality of experience visualization process performed by the quality of experience visualization device 100 will be described.

[0020] In the following sections, information about communication performance related to an online conference may be referred to as "communication performance information." Furthermore, information about the performance of a terminal device used in an online conference may be referred to as "terminal device performance information."

[0021] As shown in (1) of Fig. 1, the quality of experience visualization device 100 calculates a quality of experience score based on at least one of communication performance information and terminal device performance information, and a learning model that performs a predetermined classification of online conferences. As shown in Fig. 1, the quality of experience visualization device 100 receives communication performance information from, for example, a conference system 10. In addition, the quality of experience visualization device 100 receives terminal device performance information from, for example, a terminal device 20.

[0022] As shown in (2) of FIG. 1, the quality of experience visualization device 100 generates recommendation information for improving the calculated quality of experience score.

[0023] Through the above-described processing, the QoE visualization device 100 according to this embodiment generates recommendation information for improving a QoE score calculated by inputting communication performance information and terminal device performance information into a trained learning model. Therefore, the QoE visualization device 100 has the effect of enabling appropriate measures to be taken against degradation of communication quality.

[0024] <Explanation of the quality of experience visualization device 100> Next, we will explain the functions of the quality of experience visualization device 100. First, the flow of quality of experience visualization processing according to this embodiment will be explained using FIG.

[0025] Fig. 2 is a diagram illustrating the quality of experience visualization process according to this embodiment. Fig. 2 shows a conference system 10 that realizes online conferences, a terminal device 20 operated by a user, a terminal device 20n that displays a quality of experience score, recommendation information, etc. to a user or an administrator, and a quality of experience visualization device 100 that visualizes the user's quality of experience.

[0026] The quality of experience visualization device 100 performs a learning phase ((1) in Figure 2) of a learning model used to infer a quality of experience score, and an inference phase ((2) in Figure 2) of inferring a quality of experience score based on the trained learning model and generating recommendation information.

[0027] 2, for convenience of explanation, the explanatory variables (learning data) in the learning phase and the inference data in the inference phase are illustrated as being received from the same conference system 10 and terminal device 20, but this is not limiting. In other words, the explanatory variables (learning data) and the inference data may be collected from different conference systems 10 or terminal devices 20.

[0028] In addition, the details of the learning process of the learning model shown in Figure 2, the inference process based on the trained learning model, and the generation and output process of recommendation information using the inferred quality of experience score will be explained in later sections, so detailed explanations will be omitted in Figure 2.

[0029] First, the learning phase ((1) in FIG. 2) will be described. The quality of experience visualization device 100 receives, as explanatory variables, communication-related metrics information ((1-1) in FIG. 2), which is performance information of communication related to an online conference received from the conference system 10, and performance information of the terminal device ((1-2) in FIG. 2), received from the terminal device 20. The quality of experience visualization device 100 also receives, as a response variable, user feedback information on the online conference ((1-3) in FIG. 2), which is input by users who participate in the online conference, from the conference system 10 and the terminal device 20.

[0030] The quality of experience visualization device 100 uses explanatory variables ((1-1) and (1-2) in FIG. 2) and objective variables ((1-3) in FIG. 2) received as learning data to train a learning model ((1-4) in FIG. 2).

[0031] Next, the inference phase ((2) in FIG. 2) will be described. The quality of experience visualization device 100 performs a predetermined inference process based on the trained learning model ((2-1) in FIG. 2). Specifically, the quality of experience visualization device 100 receives, as inference data, metrics information related to communication received from the conference system 10 ((2-2) in FIG. 2) and performance information of the terminal device received from the terminal device 20 ((2-3) in FIG. 2).

[0032] The quality of experience visualization device 100 uses the inference data ((2-2) and (2-3) in Figure 2) to perform a predetermined inference process ((2-4) in Figure 2) to calculate a quality of experience score based on a trained learning model.

[0033] The quality of experience visualization device 100 uses the quality of experience score calculated based on the inference process to generate recommendation information including improvements and countermeasures for increasing (improving) the quality of experience score ((2-5) in FIG. 2).The quality of experience visualization device 100 then outputs the generated recommendation information to a terminal device 20n operated by an administrator or a user ((2-6) in FIG. 2).

[0034] (Perceived quality visualization device 100) Next, we will explain the configuration of the quality of experience visualization device 100. Fig. 3 is a diagram showing the configuration of the quality of experience visualization device 100 according to this embodiment. As shown in Fig. 3, the quality of experience visualization device 100 has a communication unit 110, a storage unit 120, and a control unit 130.

[0035] 3, the quality of experience visualization device 100 may include an input unit such as a keyboard or a mouse for receiving input such as operations by an administrator, etc. The quality of experience visualization device 100 may also include a display unit such as a display for displaying to an administrator, etc., conference history information including metrics information, terminal information including performance information of terminal devices, feedback information, quality of experience scores, recommendation information, etc.

[0036] (Communication unit 110) The communication unit 110 performs data communication related to conference history information received from the conference system 10, terminal information received from the user's terminal device 20, and input of feedback information from the user. The communication unit 110 also performs data communication related to output of the calculated quality of experience score and generated recommendation information.

[0037] The communication unit 110 is realized by a NIC (Network Interface Card) or the like, and controls communication via an electric communication line such as a LAN (Local Area Network), the Internet, etc. The communication unit 110 is connected to the network by wire or wirelessly as necessary, and can transmit and receive information to and from the conference system 10 and the terminal device 20 in both directions.

[0038] (Storage unit 120) The storage unit 120 stores data and programs used for various processes by the control unit 130, and various data acquired by the operation of the control unit 130. The storage unit 120 is realized by a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 3 , the storage unit 120 has a conference history information DB 121, a terminal information DB 122, a feedback information DB 123, a learning model DB 124, a quality of experience score information DB 125, an action information DB 126, and a recommendation information DB 127.

[0039] (Meeting history information DB121) The conference history information DB 121 is a database that stores conference history information including communication performance and the like received from the conference system 10. Specifically, the conference history information DB 121 stores information such as information about the conference that was held, information about each user who participated in the conference, information about the communication of each user who participated in the conference, and metrics information related to the communication of each user who participated in the conference.

[0040] An example of conference history information stored in the conference history information DB 121 will now be described using a table diagram. Fig. 4 is a table diagram showing an example of conference history information according to this embodiment. As shown in Fig. 4, the conference history information DB 121 stores conference information, user information, communication information, and metrics information in association with "No.", which is information that identifies individual conference history information.

[0041] For example, the conference history information DB 121 stores conference information "A", user information "B", communication information "C", and metrics information "D" associated with No. "1" as conference information in association with each other.

[0042] The above-mentioned conference information "A" includes information for identifying the conference, such as a conference ID (Identification number), the start and end times of the conference, and information about the participants in the conference. User information "B" is information for each user who participated in the conference, and includes, for example, the user name, the device name and application version used by the user, etc.

[0043] The communication information "C" includes information regarding communication between the user's terminal device and the server device for each media such as audio communication, video communication, and video communication using a shared screen, as well as segment information, IP (Internet Protocol) address, port number, device information, network connection type, radio wave strength, link speed, etc.

[0044] Metrics information "D" is performance information for communication related to online conferences, and includes information such as communication direction, maximum / average round-trip time (latency), maximum / average jitter, maximum / average packet loss, frame rate, estimated bandwidth, and number of packets per unit time.

[0045] (Terminal information DB122) The terminal information DB 122 is a database that stores terminal information that serves as an index of the performance of a terminal device, etc., received from the user's terminal device 20. Specifically, the terminal information DB 122 stores information such as information that identifies the terminal device, information about a conference that has been held, information about users who have participated in the conference, and information about the performance of the terminal device.

[0046] Here, an example of terminal information stored in the terminal information DB 122 will be described using a table diagram. Fig. 5 is a table diagram showing an example of terminal information according to this embodiment. As shown in Fig. 5, the terminal information DB 122 stores identification information of a terminal device, conference information, user information, CPU usage rate, memory usage rate, and CPU temperature in association with "No.", which is information identifying individual terminal information.

[0047] The conference information and user information stored in the terminal information DB 122 may be the same information as the information stored in the conference history information DB 121. That is, the terminal information stored in the terminal information DB 122 is associated with the conference information and user information stored in the conference history information DB 121.

[0048] For example, the terminal information DB122 stores the terminal device identification information "E" associated with No. "1", the conference information "A", the user information "B", the CPU usage rate "50%", the memory usage rate "50%", and the CPU temperature "50°C" as terminal information, each associated with the number "1".

[0049] The above-mentioned terminal device identification information is information for identifying the terminal device that participated in the target online conference. CPU usage is an index showing the proportion of CPU processing time occupied by a program running on a terminal device. Memory usage is an index showing the storage capacity of the area of ​​the main memory installed in the terminal device that is occupied by the program running on the terminal device. CPU temperature is the temperature of the CPU and is an index showing the degree of load on the CPU.

[0050] (Feedback information DB123) The feedback information DB 123 is a database that stores information (feedback information) related to feedback about the online conference input by the user. Specifically, the feedback information DB 123 stores information such as information about the conference that was held, information about the users who participated in the conference, and information about feedback about the conference.

[0051] Here, an example of feedback information stored in the feedback information DB 123 will be described using a table diagram. Fig. 6 is a table diagram showing an example of feedback information according to this embodiment. As shown in Fig. 6, the feedback information DB 123 stores conference information, user information, feedback results, and binary conversion results in association with "No.", which is information that identifies individual feedback information.

[0052] The conference information and user information stored in the feedback information DB 123 may be the same information as the information stored in the conference history information DB 121. That is, the terminal information stored in the feedback information DB 123 is associated with the conference information and user information stored in the conference history information DB 121.

[0053] For example, the feedback information DB 123 stores, as feedback information, the meeting information "A," the user information "B," the feedback result "5," and the binary conversion result "good," all of which are associated with No. "1." The binary conversion process will be described later in the section on the processing unit 141.

[0054] The feedback result described above is an evaluation index provided for each online conference tool to allow users to evaluate the online conference they participated in, and includes, for example, an index such as "5 out of 5" as described above. The binary conversion result is the result of converting the feedback result into a binary value based on predetermined conditions.

[0055] (Learning model DB124) The learning model DB 124 is a database that stores learned learning models. Specifically, the learning model DB 124 stores a learning model for voice communication, a learning model for video communication, and a learning model for screen-sharing communication.

[0056] The above-mentioned learning model for voice communication is a learning model for calculating a quality of experience score related to voice communication. The learning model for video communication is a learning model for calculating a quality of experience score related to video communication. The learning model for screen-sharing communication is a learning model for calculating a quality of experience score related to communication when screen sharing is performed in an online conference.

[0057] (Quality of Experience Score Information DB125) The quality of experience score information DB 125 is a database that stores information (quality of experience score information) related to the quality of experience score of a user in a target online conference, which is inferred by the later-described inference unit 151. Specifically, the quality of experience score information DB 125 stores information related to a conference that has been held, information related to users who have participated in the conference, an index indicating the user's satisfaction with the online conference, and the like.

[0058] Here, an example of quality of experience score information stored in the quality of experience score information DB 125 will be described using a table diagram. Fig. 7 is a table diagram showing an example of quality of experience score information according to this embodiment. As shown in Fig. 7, the quality of experience score information DB 125 stores conference information, user information, and quality of experience scores in association with "No." This is information that identifies individual quality of experience score information.

[0059] The conference information and user information stored in the quality of experience score information DB 125 may be the same information as the information stored in the conference history information DB 121. That is, the terminal information stored in the quality of experience score information DB 125 is associated with the conference information and user information stored in the conference history information DB 121.

[0060] For example, the quality of experience score information DB 125 stores the meeting information "A", the user information "B", and the quality of experience score "82" associated with No. "1" as quality of experience score information in association with each other.

[0061] The above-mentioned quality of experience score is a score (numerical value) that indicates user satisfaction with the target online conference calculated by the inference unit 151 described below, and the higher the score, the higher the probability that the user feels that the quality of communication related to the online conference is high.

[0062] (Action Information DB126) The action information DB 126 is a database that stores action information for the generation unit 152, which will be described later, to generate recommendation information based on the quality of experience score, etc. Specifically, the action information DB 126 stores action information in which conditions for determining an action for each communication state in an online conference are associated with actions (countermeasures) for each situation, etc.

[0063] Here, an example of action information stored in the action information DB 126 will be described using a table diagram. Fig. 8 is a table diagram showing an example of action information according to this embodiment. As shown in Fig. 8, the action information DB 126 stores conditions and actions in association with "No.", which is information that identifies individual pieces of recommendation information.

[0064] The above-mentioned conditions are conditions that trigger the execution of a preset action, and when the communication status, operating status, etc. of a terminal device of a user participating in an online conference or the like meets the condition, processing (action) is executed according to the countermeasure for generating recommendation information or improving the QoE score included in the recommendation information. The action is a countermeasure method set in association with the above-mentioned condition in order to improve the communication status, operating status, etc. of the terminal device of a user participating in an online conference or the like.

[0065] For example, the action information DB 126 stores, as action information, a condition "packet loss is equal to or greater than a specified value, and radio wave strength is less than a specified value" and an action "message: radio wave strength is decreasing, so it is recommended to move closer to the communication device," which are associated with No. "1." The above information means that when the communication state of the user's terminal device satisfies "packet loss is equal to or greater than a specified value, and radio wave strength is less than a specified value," a message such as "radio wave strength is decreasing, so it is recommended to move closer to the communication device" is output to the user as recommendation information.

[0066] Furthermore, for example, the action information DB 126 stores, as action information, the condition "CPU usage rate or memory usage rate exceeds a specified value" and the action "terminate unnecessary applications" associated with No. "2." The above information means that when the operating state of the user's terminal device satisfies the condition "usage rate or memory usage rate exceeds a specified value," the quality of experience visualization device 100 executes a response method such as "terminating unnecessary applications."

[0067] Furthermore, for example, the action information DB 126 stores, as action information, the condition "round trip time is equal to or greater than a specified value, or jitter is equal to or greater than a specified value" and the action "change the communication path of the conference tool through the API (Application Programming Interface) of the network service," which are associated with No. "3." The above information means that when "round trip time is equal to or greater than a specified value, or jitter is equal to or greater than a specified value," is satisfied, the quality of experience visualization device 100 executes a response method such as "change the communication path of the conference tool through the API of the network service."

[0068] (control unit 130) Here, we will return to Fig. 3 to continue the explanation. The control unit 130 has an internal memory for temporarily storing programs that define various processing procedures and the like of the quality of experience visualization device 100 and processing data, and is realized by electronic circuits such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit), and integrated circuits such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array). As shown in Fig. 3, the control unit 130 has a reception unit 131, an output unit 132, an improvement implementation unit 133, a learning processing unit 140, and an inference processing unit 150.

[0069] (Reception Department 131) The receiving unit 131 receives information used by the quality of experience visualization device 100 to perform predetermined learning processes, inference processes, recommendation information generation processes, etc. via the communication unit 110, input unit, etc., and stores the information in the memory unit 120.

[0070] Specifically, the reception unit 131 receives conference history information related to an online conference from the conference system 10, etc. via the above-described communication unit 110, etc. Specifically, the reception unit 131 receives conference history information such as conference information, user information, communication information, and metrics information from the conference system 10, etc., and stores the information in the conference history information DB 121.

[0071] An example of conference history information received by the receiving unit 131 will now be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of conference history information related to an online conference tool according to this embodiment. Fig. 9 shows an example of conference history information stored in JSON (JavaScript (registered trademark) Object Notation) format by a predetermined online conference tool.

[0072] The conference history information received by the receiving unit 131 includes information associated with each layer, such as "Call Record," "Session," "Segment," "Media," and "Stream."

[0073] "CallRecord ((1) in FIG. 9)" is data stored for each conference, and includes information such as the conference ID, conference start time, conference end time, and participants.

[0074] "Session ((2) in FIG. 9)" includes the user name, the name of the terminal device (device name), the version of the application (application version), and the feedback result (feedback). Multiple "Sessions ((2) in FIG. 9)" that are data stored for each user are associated with one "CallRecord ((1) in FIG. 9)."

[0075] "Segment ((3) in FIG. 9)" includes information (session information) about the communication between the user's terminal device and the server device. A plurality of "Segments ((3) in FIG. 9)" that are data stored for each communication between the user's terminal device and the server device are associated with one "Session ((2) in FIG. 9)."

[0076] "Media ((4) in Fig. 9)" includes IP addresses, port numbers, device information, network (NW) connection formats, signal strength, link speed, etc. Multiple "Media ((4) in Fig. 9)" that are data stored for each media such as audio communication, video communication, and screen sharing in video communication are associated with one "Segment ((3) in Fig. 9)."

[0077] "Stream ((5) in Fig. 9)" includes the communication direction, maximum / average round trip time (RTT), maximum / average jitter, maximum / average packet loss, frame rate, estimated bandwidth, number of packets, etc. Two "Streams ((5) in Fig. 9)" for inbound and outbound communications, which are information stored for each inbound and outbound communication, are associated with one "Media ((4) in Fig. 9)."

[0078] Furthermore, the reception unit 131 receives terminal information related to the terminal device 20 involved in the online conference from the terminal device 20 or the like via the above-described communication unit 110 or the like. Specifically, the reception unit 131 receives terminal information such as terminal device identification information, CPU usage rate, memory usage rate, CPU temperature, etc. from the user's terminal device 20 or the like, and stores the received information in the terminal information DB 122 in association with the conference information and user information. Note that the terminal information received by the reception unit 131 may be collected by agent software or the like installed on the user's terminal device.

[0079] The receiving unit 131 also receives feedback information of users related to the online conference from the conference system 10, the terminal device 20 operated by the user, etc., via the above-described communication unit 110, etc. For example, the receiving unit 131 receives information such as a feedback result for the target conference from a user who participated in the online conference, such as "5 out of 5," and stores the information in the feedback information DB 123.

[0080] (output unit 132) The output unit 132 outputs the generated quality of experience score and recommendation information to a manager such as an administrator, a user, etc. For example, the output unit 132 can output information to a terminal device operated by the manager, a user, etc. via the communication unit 110. In addition, the output unit 132 can output information to a manager, etc. via a display unit, etc. provided in the quality of experience visualization device 100.

[0081] Specifically, the output unit 132 outputs the quality of experience score calculated by the inference unit 151 (described later) and the recommendation information generated by the generation unit 152 (described later) as an output screen in a predetermined format. Note that when the inference unit 151 (described later) calculates both the "quality of experience score based on communication performance information" and the "quality of experience score based on terminal device performance information," the output unit 132 can display them side by side.

[0082] An example of an output screen output by the output unit 132 will now be described with reference to Fig. 10. Figs. 10 to 13 are diagrams showing examples of output screens according to this embodiment. Fig. 10 shows, as a first example, an example of an output screen that displays the aggregation results of quality of experience scores. Fig. 11 shows, as a second example, an output screen that displays a list of quality of experience scores corresponding to the distribution of quality of experience scores. Figs. 12 and 13 show, as a third example, an example of an output screen that displays a detailed screen of an individual quality of experience score.

[0083] (Example 1: Aggregated results of quality of experience scores) First, as a first example, the results of tallying quality of experience scores will be described with reference to Fig. 10. As shown in (1) of Fig. 10, the output unit 132 can output an output screen that displays the results of a predetermined tally of calculated quality of experience scores of a predetermined granularity. For example, the output unit 132 displays the average value of the quality of experience scores per unit period ((1-1) of Fig. 10). Note that the lower the displayed quality of experience score, the more frequently audio and video interruptions occur during the conference.

[0084] Furthermore, the output unit 132 can qualitatively express the user's satisfaction with the online conference by changing the color tone of the average value of the quality of experience score per unit period, etc. As a specific example, as shown in (1-2) of Fig. 10, the output unit 132 can classify the scores into four levels: 0 to 25, 26 to 50, 51 to 75, and 76 to 100, and display the average value of the quality of experience score per unit period, etc., by changing the color tone for each level.

[0085] For example, "0-25" is a score tier where it is expected that the user has an impression of the online conference tool such as "completely unusable." Also, "26-50" is a score tier where it is expected that the user has an impression of the online conference tool such as "frustrating, don't want to use it." Also, "51-75" is a score tier where it is expected that the user has an impression of the online conference tool such as "comfortable, generally satisfied." And "76-100" is a score tier where it is expected that the user has an impression of the online conference tool such as "I want to use it more, stress-free."

[0086] Furthermore, the output unit 132 can output the distribution of the quality of experience scores for the entire conference in the target organization for each unit period, as shown in (1-3) of FIG.

[0087] When an administrator or the like selects a distribution graph of quality of experience scores in an area shown in (1-3) of Fig. 10, the output unit 132 can output a list of quality of experience scores corresponding to the selected area. The list of quality of experience scores will be explained in the next section on the second example.

[0088] By outputting the aggregated results of the quality of experience scores described above, the quality of experience visualization device 100 makes it easy for managers and others to understand which meetings have low quality of experience scores by checking the quality of experience scores of all meetings within the target organization displayed on the dashboard.

[0089] (Second example: A list of quality of experience scores corresponding to the distribution of quality of experience scores) Next, as a second example, a list of quality of experience scores corresponding to the distribution of quality of experience scores will be described with reference to Fig. 11. As shown in (1) of Fig. 11, the output unit 132 displays a list of quality of experience scores for each region of the selected distribution graph.

[0090] For example, the output unit 132 can display a list of quality of experience scores for each conference ((1-1) in FIG. 11) in response to an administrator or the like selecting an area in the distribution graph of quality of experience scores shown in (1-3) in FIG. 10. Note that if the distribution of quality of experience scores shown in (1-3) in FIG. 10 is not at the granularity of "each conference" but at the granularity of "each user," the output unit 132 can output a list of quality of experience scores for each user ((1-2) in FIG. 11).

[0091] Furthermore, when an administrator or the like selects a conference from the list shown in (1-3) of Fig. 11, the output unit 132 can display a detailed screen of the quality of experience score for the selected conference. The detailed screen of the quality of experience score will be described in the next section on the third example.

[0092] By outputting a list of quality of experience scores corresponding to the distribution of quality of experience scores described above, the quality of experience visualization device 100 makes it easy for managers and others to identify meetings with low quality of experience scores from the distribution of quality of experience scores displayed on the dashboard.

[0093] (Third example: Detailed screen of individual quality of experience scores) Next, as a third example, a detailed screen of an individual quality of experience score will be described with reference to Fig. 12 and Fig. 13. As shown in Fig. 12 (1), the output unit 132 displays a detailed screen of the quality of experience score for each individual granularity, such as for each meeting.

[0094] As described above, the output unit 132 can display a detailed screen of the quality of experience score for each conference selected by an administrator or the like from the list shown in (1-3) of Figure 11 ((1-1) of Figure 12).

[0095] For example, the output unit 132 displays the average quality of experience score of the target conference and the quality of experience scores for audio (audio communication), video (video communication), and screen sharing ((1-2) in FIG. 12). The output unit 132 also displays the conference duration, start time, end time, and number of participants as conference data ((1-3) in FIG. 12).

[0096] The output unit 132 displays information about the users who participated in the target online conference, such as the user name, the quality of experience score for each user, the OS (Operating System) of the terminal device, the NW connection type, and the source NW ((1-4) in FIG. 12). Furthermore, the output unit 132 can display recommendation information generated by the generation unit 152 (described later) ((1-5) in FIG. 12).

[0097] The output unit 132 displays metrics information for communications related to the target online conference ((1-6) in FIG. 12). For example, the output unit 132 displays list information such as the average jitter, maximum jitter, average packet loss rate, maximum packet loss rate, average round trip time, and average complementary sample rate for each of the upstream and downstream directions. The output unit 132 can also display the above-mentioned metrics information for each of audio, video, screen sharing, and network connection.

[0098] Furthermore, the output unit 132 can display recommendation information generated by the generating unit 152 (described later) ((1-7) in FIG. 12).

[0099] Furthermore, the output unit 132 can output the above-mentioned metrics information ((1-6) in Figure 12) in graph format for each individual piece of metrics information ((1) in Figure 13), in addition to displaying it in a list using text, numerical values, etc., as shown in Figure 13.

[0100] For example, the output unit 132 can display the metrics information "jitter" as a graph by arranging "max (downstream)," "average (downstream)," "max (upstream)," "average (upstream)," etc. ((1-1) in FIG. 13).

[0101] By outputting the detailed screen of the individual quality of experience scores described above, the quality of experience visualization device 100 enables a manager or the like to easily check and analyze the quality of experience scores and metrics information for each granularity of each conference or each user. Therefore, the quality of experience visualization device 100 has the effect of enabling a manager or the like to consider measures to improve the quality of experience score.

[0102] (Improvement Implementation Department 133) The improvement implementation unit 133 executes processing according to a countermeasure for improving the QoE score, which is included in the recommendation information generated by the generation unit 152. Specifically, when a user issues an instruction for improvement according to the generated recommendation information, or when the generation unit 152, which will be described later, generates recommendation information such as "changing the communication path of the conference tool through the API of the network service" or "terminating unnecessary applications," the improvement implementation unit 133 can automatically execute the above-mentioned "changing the communication path of the conference tool through the API of the network service" or "processing to terminate unnecessary applications."

[0103] The improvement implementation unit 133 can automatically execute processing according to the countermeasure when a predetermined condition is met, such as when permission to implement the countermeasure is obtained from the user.

[0104] (Learning processing unit 140) The learning processing unit 140 uses communication performance information, terminal device performance information, etc. to train the learning model so as to classify the target online conference into a good conference or a bad conference. The learning processing unit 140 further includes a processing unit 141 and a learning unit 142 for executing each step in the learning process of the learning model described above.

[0105] (Processing section 141) The processing unit 141 performs a predetermined processing on data that is used as learning data by the learning unit 142 described below. Specifically, the processing unit 141 processes feedback from users who participated in an online conference into first identification information such as "GOOD" indicating a good conference and second identification information such as "POOR" indicating a bad conference.

[0106] For example, when the user's feedback is on a five-point scale such as "1: bad," "2: somewhat bad," "3: average," "4: somewhat good," and "5: good," the processing unit 141 converts "3 to 5" into "GOOD." On the other hand, for example, the processing unit 141 converts "1 and 2" into "POOR." Note that when there are multiple pieces of feedback for the target conference, the processing unit 141 may average the values ​​of the feedback and then perform the above-mentioned conversion process.

[0107] Furthermore, the processing unit 141 performs a predetermined processing process on the metrics information and adds it to the explanatory variables.

[0108] Specifically, the processing unit 141 calculates a weighted average of the number of packets for the amount of data communication in the uplink direction of users other than the first user, and adds the calculated average to the explanatory variables.

[0109] Furthermore, when the online conference is video communication and communication in which the display screen of a terminal device is shared in the video communication, the processing unit 141 adds a frame rate ratio calculated using the receiving frame rate of the first user and the transmitting frame rate of the user other than the first user to the explanatory variables. Note that the above-mentioned frame rate ratio is an index representing quality degradation along the path from the sender to the receiver.

[0110] Furthermore, when the user feedback is information about a meeting that is poor among predetermined binary values, the processing unit 141 removes predetermined outliers from the communication metrics information. Specifically, the processing unit 141 uses the interquartile range (IQR), the first quartile (Q1: 1st quartile), and the third quartile (Q3: 3rd quartile) to remove outliers with a lower limit of "Q1-1.5×IQR" and an upper limit of "Q3+1.5×IQR." Note that, for data within the outlier range, the processing unit 141 does not remove data for which user feedback is classified as "POOR" or the like as an outlier but includes it in the learning data.

[0111] Furthermore, if there is a missing value in the communication performance information or the terminal device performance information, the processing unit 141 complements the missing value using a median value or the like.

[0112] (Learning Section 142) The learning unit 142 uses predetermined explanatory variables and objective variables to train a learning model such as a random forest decision tree so as to classify online conferences into good conferences (GOOD) or bad conferences (POOR) using communication performance information as input. Furthermore, the learning unit 142 can use at least one of the communication performance information and the terminal device performance information as the explanatory variables. Furthermore, the learning unit 142 can use, as the objective variable, user feedback on the online conference, converted into a binary value of GOOD (first identification information) indicating a good conference and POOR (second identification information) indicating a bad conference.

[0113] Furthermore, the learning unit 142 learns the learning model using, as an explanatory variable, a value obtained by calculating a weighted average of the number of packets for the amount of upstream data communication of users other than the first user. Furthermore, when the online conference is video communication and communication in which the display screen of a terminal device is shared in the video communication, the learning unit 142 learns the learning model using, as an explanatory variable, a frame rate ratio calculated using the receiving frame rate of the first user and the transmitting frame rate of users other than the first user.

[0114] An example of the learning process by the learning unit 142 will now be described with reference to Fig. 14. Fig. 14 is a diagram showing an example of an output screen according to this embodiment. Fig. 14 shows explanatory variables ((1) in Fig. 14), objective variables ((2) in Fig. 14), and a learning model to be learned ((3) in Fig. 14) as learning data.

[0115] As described above, the learning unit 142 uses explanatory variables including communication performance information, terminal device performance information, etc., and feedback from users to learn a learning model based on supervised machine learning or the like as a classification problem into good meetings / bad meetings.

[0116] The explanatory variables shown in (1) of Fig. 14 include communication metrics information such as round trip time, packet loss, jitter, etc., and performance information of terminal devices such as CPU usage rate, memory usage rate, etc. The objective variables shown in (2) of Fig. 14 include binary-converted user feedback information such as "a good meeting (GOOD) if the user feedback is 3 or more" and "a bad meeting (POOR) if the user feedback is less than 3."

[0117] The learning model shown in (3) of FIG. 14 is learned based on supervised machine learning such as random forest, which is an algorithm that combines two techniques, "decision tree" and "ensemble learning (bagging)." As shown in (3) of FIG. 14, the learning model includes multiple decision trees, and the learning unit 142 learns each decision tree to classify input learning data into "Yes or No (GOOD or POOR)." Note that each of the above-mentioned decision trees uses different metric information, etc.

[0118] Furthermore, the learning unit 142 learns multiple learning models according to the type of conference. For example, the learning unit 142 learns different learning models using metrics information for each type of communication format related to the conference, such as audio communication, video communication, and screen sharing communication in video communication. Then, the learning unit 142 stores the learned learning models in the learning model DB 124.

[0119] (Inference processing unit 150) The inference processing unit 150 executes a predetermined inference process based on the learned learning model. The inference processing unit 150 further includes an inference unit 151 and a generation unit 152 for executing each step in the inference process based on the above-mentioned learning model.

[0120] (Inference part 151) The inference unit 151 executes a predetermined inference process based on the learning model learned by the learning processing unit 140 .

[0121] Specifically, the inference unit 151 inputs at least one of communication performance information and terminal device performance information into a decision tree, which is a learning model trained to classify online conferences into good conferences or bad conferences, and calculates the percentage of decision trees that classify online conferences into good conferences as the quality of experience score.

[0122] In addition, the inference unit 151 calculates the quality of experience score based on a predetermined granularity that combines one or more of each online conference tool used in the online conference, each online conference, and each participant in the online conference.

[0123] An example of the inference processing by the inference unit 151 will now be described with reference to Fig. 15. Fig. 15 is a diagram showing an example of the quality of experience score calculation processing according to this embodiment. Fig. 15 shows explanatory variables ((1) in Fig. 15) as inference data, a trained learning model ((2) in Fig. 15) used in the inference processing, and an inference result ((3) in Fig. 15).

[0124] As described above, the inference unit 151 calculates the quality of experience score as an inference result based on the classification results for each decision tree obtained by inputting explanatory variables including communication performance information, terminal device performance information, etc. into a trained learning model.

[0125] The explanatory variables shown in (1) of FIG. 15 include communication metrics information such as round trip time, packet loss, and jitter, and performance information of terminal devices such as CPU utilization rate and memory utilization rate.

[0126] In addition, in the learning model shown in FIG. 15(2), a "good meeting" or "bad meeting" is classified according to the explanatory variables input for each "decision tree." Based on the classification results for each decision tree described above, the inference unit 151 multiplies the percentage of decision trees classified as "good meeting (GOOD)" by 100 to convert the value and calculates the quality of experience score. Note that a quality of experience score of "100" indicates a high probability that the meeting is of good quality, and a quality of experience score of "0" indicates a high probability that the meeting is of poor quality.

[0127] Furthermore, the inference unit 151 performs inference processing based on multiple learning models tailored to the type of conference. For example, the inference unit 151 selects a learning model corresponding to each media type, such as audio communication, video communication, and screen sharing communication in video communication, involved in the conference, performs inference using metrics information and the like for each media type, and calculates a quality of experience score ((3) in FIG. 15).

[0128] If performance information of the terminal device exists as inference data, the inference unit 151 may calculate the quality of experience score based on the performance information of the terminal device.

[0129] Specifically, the inference unit 151 measures the number of times that performance information of the terminal device, such as CPU usage rate and memory usage rate, collected at a predetermined cycle, falls below a preset threshold. Then, the inference unit 151 displays the ratio (e.g., %) of the measured number of times as a score. For example, if the threshold for CPU usage rate is set to "90%" and 8 out of 10 measurements during the target conference period were below the threshold and 2 out of 10 were above the threshold, the inference unit 151 calculates the score as "80."

[0130] (Generation unit 152) The generation unit 152 generates recommendation information including a response method for improving the score indicating the user satisfaction with the online conference calculated by the inference unit 151. Specifically, the generation unit 152 generates recommendation information including the quality of experience score and a response method for increasing the quality of experience score selected based on at least one of the average value, worst value, distribution, and predetermined variance of the quality of experience score.

[0131] For example, when the average or worst value of the quality of experience score falls below a predetermined threshold or when the distribution or variance of the quality of experience score exceeds a predetermined range, the generation unit 152 refers to the communication performance information or the terminal device performance information related to the quality of experience score. Next, the generation unit 152 identifies the communication performance information or the terminal device performance information that is causing the quality of experience score to decrease.

[0132] Then, when the communication performance information or terminal device performance information that is the cause of the identified decrease in the quality of experience score satisfies the conditions of the action information stored in the action information DB 126, the generation unit 152 generates recommendation information using the action content associated with the conditions.

[0133] (Processing Procedure) Next, we will explain the processing procedures related to the quality of experience visualization device 100 according to this embodiment. Note that the processing by the quality of experience visualization device 100 is divided into two parts, a "learning process" and a "quality of experience visualization process," and therefore the processing procedures will be explained using the respective flowcharts.

[0134] (Learning process procedure) First, the procedure of the "learning process" performed by the quality of experience visualization device 100 will be described with reference to Fig. 16. Fig. 16 is a diagram showing a flowchart of the learning process according to this embodiment.

[0135] The receiving unit 131 receives explanatory variables and objective variables (S101). Next, the learning processing unit 140 uses the explanatory variables and objective variables to learn a learning model (S102).

[0136] If the learning termination condition of the learning model is not satisfied (No in S103), the quality of experience visualization device 100 repeats the learning process. On the other hand, if the learning termination condition of the learning model is satisfied (Yes in S103), the quality of experience visualization device 100 ends the learning process.

[0137] The above-mentioned conditions for terminating the learning are not particularly limited. For example, the quality of experience visualization device 100 may determine that the learning conditions are met when the number of times the target learning model has been learned reaches a predetermined number of times or when the model accuracy of the learning model exceeds a predetermined threshold.

[0138] (Procedure for visualizing quality of experience) Next, the procedure of the "quality of experience visualization process" performed by the quality of experience visualization device 100 will be described with reference to Fig. 17. Fig. 17 is a diagram showing a flowchart of the quality of experience visualization process according to this embodiment.

[0139] The receiving unit 131 receives data for inference (S201). Next, the inference unit 151 performs inference based on the trained learning model and calculates a quality of experience score (S202).

[0140] If the condition for generating recommendation information is met based on the inferred quality of experience score (Yes in S203), the generation unit 152 generates recommendation information using the calculated quality of experience score (S204). Subsequently, the output unit 132 outputs the quality of experience score and the recommendation information (S205). Then, the quality of experience visualization device 100 ends the quality of experience visualization process.

[0141] On the other hand, if the inferred quality of experience score does not satisfy the condition for generating recommendation information (No in S203), the output unit 132 outputs the quality of experience score without including recommendation information (S206).Then, the quality of experience visualization device 100 ends the quality of experience visualization process.

[0142] (effect) Next, the effects achieved by the quality of experience visualization device 100 according to this embodiment will be described. The inference unit 151 of the quality of experience visualization device 100 calculates a quality of experience score based on at least one of communication performance information and terminal device performance information, and a learning model that performs a predetermined classification of online conferences. The generation unit 152 of the quality of experience visualization device 100 generates recommendation information for improving the quality of experience score calculated by the inference unit 151. Therefore, the quality of experience visualization device 100 of this embodiment has the effect of enabling appropriate measures to be taken against degradation of communication quality.

[0143] The inference unit 151 inputs at least one of communication performance information and terminal device performance information into a decision tree, which is a learning model trained to classify online conferences into good conferences or bad conferences. The inference unit 151 calculates the percentage of decision trees that classify online conferences into good conferences as a quality of experience score.

[0144] Therefore, the quality of experience visualization device 100 can calculate a score that represents the quality of an online conference as experienced by a user, rather than simply using an index such as a communication state as a score that represents the quality of the online conference. Therefore, the quality of experience visualization device 100 has the effect of being able to visualize the quality of an online conference that is in line with the user's experience.

[0145] The inference unit 151 calculates the quality of experience score based on a predetermined granularity that is a combination of one or more of each online conference tool used in the online conference, each online conference, and each participant in the online conference.

[0146] Therefore, the quality of experience visualization device 100 is not limited to the granularity of, for example, "meeting," but can calculate the quality of experience score at various granularities. Therefore, when the quality of experience score decreases, the quality of experience visualization device 100 has the effect of making it possible to easily identify the events that are the cause of the decrease.

[0147] The generation unit 152 generates recommendation information including at least one of a quality of experience score and a countermeasure for improving the quality of experience score selected based on at least one of an average value, a worst value, a distribution, and a predetermined variance of the quality of experience score.

[0148] Therefore, the quality of experience visualization device 100 not only outputs a score representing the quality of the online conference, but also recommends measures to improve the score to the administrator, etc. Therefore, the quality of experience visualization device 100 has the effect of not only grasping the quality of the online conference, but also enabling the improvement of the quality.

[0149] The learning processing unit 140 uses at least one of communication performance information and terminal device performance information as explanatory variables, and user feedback on online conferences converted into two values, GOOD (first identification information) indicating a good conference and POOR (second identification information) indicating a bad conference, as learning data, as a target variable, and trains a learning model to classify online conferences into good or bad conferences using communication performance information related to held online conferences as inference data as input.

[0150] Therefore, the quality of experience visualization device 100 can learn a learning model for calculating a score representing the quality of an online conference experienced by a user, rather than simply using an index such as a communication state as a score representing the quality of the online conference. Therefore, the quality of experience visualization device 100 has the effect of enabling appropriate measures to be taken against degradation of communication quality using the learned learning model.

[0151] When the user feedback is POOR (second identification information) indicating a poor conference, the learning processing unit 140 excludes a predetermined outlier from the communication performance information.

[0152] Therefore, the quality of experience visualization device 100 enables learning that is more in line with reality. In other words, if outliers are mechanically excluded, accurate learning would be difficult in cases where the quality of an online conference deteriorates significantly due to unexpected factors. However, the quality of experience visualization device 100 can avoid such events and perform learning with greater accuracy. Therefore, the quality of experience visualization device 100 achieves the effect of enabling appropriate measures to be taken against deterioration in communication quality using a trained model that has been trained with greater accuracy.

[0153] The learning processing unit 140 learns the learning model using, as an explanatory variable, a value obtained by calculating a weighted average of the number of packets for the amount of upstream data communication of users other than the first user. Furthermore, when the online conference is video communication and communication in which the display screen of a terminal device is shared in the video communication, the learning processing unit 140 learns the learning model using, as an explanatory variable, a frame rate ratio calculated using the receiving frame rate of the first user and the transmitting frame rate of users other than the first user.

[0154] Therefore, the quality of experience visualization device 100 has the effect of enabling appropriate measures to be taken against degradation in communication quality by using a trained model that has been trained with higher accuracy.

[0155] The improvement implementation unit 133 executes processing according to the countermeasure for improving the quality of experience score, which is included in the recommendation information generated by the generation unit 152.

[0156] Therefore, the quality of experience visualization device 100 not only presents the calculated quality of experience score and the created recommendation information to the administrator, etc., but also automatically executes the countermeasures included in the recommendation information when the conditions are satisfied. Therefore, the quality of experience visualization device 100 has the effect of enabling more effective and efficient improvement of communication quality.

[0157] As described above, the quality of experience visualization device 100 enables more efficient and appropriate identification of factors that cause degradation in communication quality and the implementation of countermeasures for those factors than conventional methods. For example, the quality of experience visualization device 100 enables an administrator or the like to intuitively understand whether a user is able to comfortably hold an online conference by checking the quality of experience score.

[0158] Furthermore, the quality of experience visualization device 100 has the effect of enabling administrators and the like to appropriately respond to inquiries from users regarding the quality of online meetings by outputting detailed quality of experience scores and recommendation information.

[0159] Furthermore, the QoE visualization device 100 enables users who participate in an online conference to check the QoE score and detailed data of the conference and analyze the causes of poor quality by themselves. As a result, the QoE visualization device 100 can reduce the burden on administrators and other personnel.

[0160] Furthermore, the quality of experience visualization device 100 makes it possible to understand, from the statistical values ​​of the quality of experience score, whether one's own quality of experience is good or bad in comparison with other users. As a result, the quality of experience visualization device 100 makes the user realize that a more comfortable online conference can be held, thereby improving the quality of the online conference.

[0161] Furthermore, the quality of experience visualization device 100 enables reduction and efficiency of computer processing through the above-described processing. For example, the quality of experience visualization device 100 can reduce the processing required for model construction compared to conventional methods by automatically constructing a user quality of experience calculation model using performance information (metrics information) of communication related to online conferences and user feedback.

[0162] Furthermore, the quality of experience visualization device 100 can automatically select a response method corresponding to the quality of experience score calculated by inputting automatically received performance information into a trained learning model, thereby automating system settings for improving the quality of experience score. Therefore, the quality of experience visualization device 100 can improve the quality of experience more efficiently and effectively than ever before.

[0163] <Modification> The following describes modified examples realized by the quality of experience visualization device 100 according to this embodiment.

[0164] (Data, etc.) The communication performance information, terminal device performance information, quality of experience score, conference system, terminal device, learning model, metrics information, voice communication, video communication, screen sharing in video communication, names of functional parts of the quality of experience visualization device 100, steps, processes, names of steps or processes, etc. used in the description of the above embodiments are merely examples and can be changed as desired.

[0165] For example, the conference history information DB 121 stores conference information, user information, communication information, and metrics information in association with "No.", which identifies individual conference history information, but is not limited to this.

[0166] Furthermore, the above-mentioned conference information has been described as including, but is not limited to, information identifying the conference, such as a conference ID, the start and end times of the conference, and information about the participants in the conference. Furthermore, the user information has been described as including, but is not limited to, information about each user who participated in the conference, such as the user name, the device name and application version used by the user. Furthermore, the communication information has been described as including, but is not limited to, information about communication between the user's terminal device and the server device for each medium, such as audio communication, video communication, and video communication using a shared screen, as well as segment information, IP address, port number, device information, network connection type, signal strength, link speed, and the like. Furthermore, the metrics information has been described as performance information about communication related to an online conference, such as communication direction, maximum / average round trip time (latency), maximum / average jitter, maximum / average packet loss, frame rate, estimated bandwidth, and number of packets per unit time.

[0167] For example, it has been explained that the terminal information DB122 stores the identification information of the terminal device, the conference information, the user information, the CPU usage rate, the memory usage rate, and the CPU temperature in association with "No" that identifies individual terminal information, but this is not limited to this.

[0168] For example, the feedback information DB 123 has been described as storing conference information, user information, feedback results, and binary conversion results in association with "No." This does not limit the scope of the present invention. Also, although the binary conversion results have been described, the feedback information DB 123 can store, for example, "five values" other than "binary."

[0169] For example, although the learning model DB 124 has been described as storing a learning model for voice communication, a learning model for video communication, and a learning model for screen-sharing communication, the present invention is not limited to this. Furthermore, the learning model DB 124 may store, as the trained learning model, a trained learning model trained by the learning processing unit 140, or a trained learning model trained by another information processing device or the like.

[0170] For example, although it has been described that the quality of experience score information DB 125 stores conference information, user information, and quality of experience scores in association with "No.", which identifies individual quality of experience score information, the present invention is not limited to this. Also, although it has been described that the quality of experience score information DB 125 stores quality of experience scores as numerical values, the quality of experience scores may be stored in other formats such as text or figures.

[0171] For example, the action information DB 126 has been described as storing conditions and actions in association with "No.", which identifies individual pieces of recommendation information, but the present invention is not limited to this.

[0172] (Output screen layout) Although examples of output screens according to this embodiment have been described in Figures 10 to 13, the configurations of the output screens output by the quality of experience visualization device 100 are not limited to those shown in Figures 10 to 13. In other words, the quality of experience visualization device 100 can arbitrarily change the information displayed on the screen, the display position, display size, color tone, etc. of the information depending on the situation, in addition to the screen layouts shown in Figures 10 to 13.

[0173] (Regarding the timing of the learning and inference phases) In this embodiment, for convenience of explanation, the inference phase is performed after the learning phase, but this is not limiting. For example, the quality of experience visualization device 100 according to this embodiment can perform a learning process for a learning model to build a learning model in advance. Then, the quality of experience visualization device 100 can execute an inference process using the learned learning model at any timing.

[0174] (Data storage location) Although the quality of experience visualization device 100 according to this embodiment has been described as storing information received from the conference system 10 and the terminal device 20 in the storage unit 120, the present invention is not limited to this. For example, the quality of experience visualization device 100 may store the information in a data lake on the cloud, instead of in its own storage unit 120.

[0175] (Generation of recommendation information) In the above description, the quality of experience visualization device 100 generates recommendation information based on information about the performance of an online conference or a terminal device when the calculated score is below a predetermined threshold, but the granularity of the generated recommendation information is not particularly limited. For example, the quality of experience visualization device 100 may generate recommendation information at the granularity of each unit period based on information aggregated during a unit period, in addition to for each conference held.

[0176] (Flowcharts, etc.) The steps in the flowcharts may be interchanged as long as there is no contradiction, and some steps may not be performed. In addition, conjunctions such as "next," "continue," "further," "at this time," and "on this occasion" used in the explanation of the flowcharts do not limit the order or timing of the execution of the processes in the flowcharts.

[0177] <Hardware configuration> The components of each device shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of each device can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

[0178] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can also be performed manually using known methods. In addition, the information including the processing procedures, control procedures, specific names, various data, and parameters shown in the drawings can be changed as desired unless otherwise specified.

[0179] <Program> In one embodiment, the various devices constituting the quality of experience visualization device 100 can be implemented by installing a quality of experience visualization program as package software or online software on a desired computer. For example, by executing the quality of experience visualization program on an information processing device, it is possible to cause the various devices constituting the quality of experience visualization device 100 to function. The information processing device referred to here includes desktop and notebook personal computers. In addition, the information processing device also includes mobile communication terminals such as smartphones and mobile phones, and even slate terminals such as PDAs (Personal Digital Assistants).

[0180] 18 is a diagram showing an example of a computer that executes the quality of experience visualization process according to this embodiment. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected via a bus 1080.

[0181] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0182] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the programs that define the processes of the various devices that make up the quality of experience visualization device 100 are implemented as program modules 1093 in which computer-executable codes are written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, the program modules 1093 for executing processes similar to those of the functional configurations of the various devices that make up the quality of experience visualization device 100 are stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0183] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.

[0184] The program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a LAN or a WAN (Wide Area Network)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0185] <Other> Although the present embodiment has been described above, the present embodiment is not limited by the descriptions and drawings that form part of the disclosure. In other words, other embodiments, examples, operational techniques, etc. that are made by those skilled in the art based on the present embodiment are all included in the scope of the present embodiment. [Explanation of symbols]

[0186] 10. Conference System 20 Terminal equipment 100 Experienced quality visualization device 110 Communications Department 120 Storage section 121 Meeting History Information DB 122 Terminal Information DB 123 Feedback Information DB 124 Learning Model DB 125 Quality of Experience Score Information DB 126 Action Information DB 130 control section 131 Reception 132 Output section 140 Learning processing unit 141 Processing Department 142 Learning Department 150 Inference processing unit 151 Reasoning part 152 Generation part

Claims

1. an inference unit that calculates a score indicating user satisfaction with the online conference based on at least one of information regarding communication performance related to the online conference and information regarding the performance of a terminal device used in the online conference, and a learning model that performs a predetermined classification of the online conference; and a generation unit that generates recommendation information for improving the score that indicates user satisfaction with the online conference calculated by the inference unit; A quality of experience visualization device comprising:

2. The inference unit inputting at least one of information regarding communication performance related to the online conference and information regarding performance of a terminal device used in the online conference into a decision tree, which is the learning model trained to classify the online conference into a good conference or a bad conference; A ratio of the decision trees that classify the online conference as a good conference is calculated as a score indicating user satisfaction with the online conference. The quality of experience visualization device according to claim 1 .

3. The inference unit calculating a score indicating user satisfaction with the online conference based on a predetermined granularity that combines one or more of each online conference tool used for the online conference, each online conference, and each participant in the online conference; 3. The quality of experience visualization device according to claim 1 or 2.

4. The generation unit a score indicating user satisfaction with the online conference; and, a countermeasure for improving the score indicating user satisfaction with the online conference, the countermeasure being selected based on at least one of an average value, a worst value, a distribution, and a predetermined variance of the score indicating user satisfaction with the online conference; and generating the recommendation information including at least one of the following:

3. The quality of experience visualization device according to claim 1 or 2.

5. As an explanatory variable, at least one of information regarding the performance of communication related to the online conference and information regarding the performance of a terminal device used in the online conference; As a response variable, user feedback on the online conference converted into a binary value of first identification information indicating a good conference and second identification information indicating a bad conference; is used as training data, The method further includes a learning processing unit that uses information on communication performance related to a conducted online conference, which is inference data, as input and learns the learning model to classify the online conference into the good conference or the bad conference. The quality of experience visualization device according to claim 1 .

6. The learning processing unit If the user's feedback is the first identification information indicating a good conference, excluding a predetermined outlier from information regarding communication performance related to the online conference. The quality of experience visualization device according to claim 5 .

7. The learning processing unit the learning model is trained using a weighted average calculated for the number of packets of the upstream data communication traffic of users other than the first user as the explanatory variable; 7. The quality of experience visualization device according to claim 5 or 6.

8. The learning processing unit When the online conference is a video communication and a communication in which a display screen of a terminal device is shared in the video communication, training the learning model using, as the explanatory variable, a frame rate ratio calculated using a receiving frame rate of a first user and a transmitting frame rate of a user other than the first user; 7. The quality of experience visualization device according to claim 5 or 6.

9. The method further includes an improvement execution unit that executes processing according to a countermeasure for improving a score indicating user satisfaction with the online conference, the countermeasure being included in the recommendation information generated by the generation unit.

7. The quality of experience visualization device according to claim 1, 2, 5, or 6.

10. A quality of experience visualization method executed by a quality of experience visualization device, an inference process for calculating a score indicating user satisfaction with the online conference based on at least one of information regarding communication performance related to the online conference and information regarding the performance of a terminal device used in the online conference, and a learning model that performs a predetermined classification of the online conference; a generating step of generating recommendation information for improving the score indicating the user satisfaction with the online conference calculated by the inference step; A quality of experience visualization method comprising:

11. an inference procedure for calculating a score indicating user satisfaction with the online conference based on at least one of information regarding communication performance related to the online conference and information regarding the performance of a terminal device used in the online conference, and a learning model that performs a predetermined classification of the online conference; and a generation step of generating recommendation information for improving a score indicating user satisfaction with the online conference calculated by the inference step; A quality of experience visualization program that runs the above on a computer.

Images