Estimation method, estimation program, and communication processor

Abstract

To make it possible to mitigate a processing load in network monitoring.SOLUTION: An estimation method for estimating the quality of a service executed by a first communication device of a communication system including the first communication device for executing the service, a second communication device for performing communication with the first communication device over a network, and a monitoring device for monitoring the network, includes: an acquisition step of acquiring packets to be transmitted in a first direction on the network; a feature amount determination step of classifying the acquired packets into packets of a first type and packets of a second type differing from the packets of the first type to determine a feature amount for each of the types of the classified packets; and an estimation step of the first communication device estimating the quality of the service using a learning model learned using the feature amount and quality of the service at timing at which the packets used for the determination of the feature amount are transmitted.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an estimation method, an estimation program, and a communications processing device. [Background technology]

[0002] In recent years, the emergence of various applications has led to ever-changing network requirements. In order to effectively utilize limited network resources in communication systems, it is necessary to appropriately allocate resources according to the importance and characteristics of applications.

[0003] For example, in a certain communication system, a system administrator may monitor the communication status of the network and dynamically control it. However, the more complex the configuration of the communication system, the more difficult it becomes to manually monitor and control the network appropriately.

[0004] Therefore, network control is sometimes automated to prevent problems from being overlooked due to human monitoring and to execute control with real-timeness.In an automated control system, for example, features (e.g., the number of packets passing within a given time, data volume, etc.) are extracted from packets flowing through the network, and quality is estimated using a machine learning model created in advance.

[0005] Techniques for controlling network resources are described in, for example, the following documents: [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2012-191372 A [Patent Document 2] JP 2011-172150 A [Patent Document 3] US2020-0244490 publication Summary of the Invention [Problem to be solved by the invention]

[0007] However, packet monitoring requires the continuous capture of a large number of packets in order to extract features. Also, the more packets there are to be monitored, the greater the processing load for packet capture and feature extraction, which may make it impossible to control the network in real time.

[0008] In view of this, one disclosure provides an estimation method, an estimation program, and a communications processing device that can reduce the processing load in network monitoring. [Means for solving the problem]

[0009] An estimation method for estimating the quality of a service performed by a first communication device in a communication system having a first communication device that executes a service, a second communication device that communicates with the first communication device via a network, and a monitoring device that monitors the network, the method comprising: an acquisition step of acquiring packets transmitted in a first direction on the network; a feature determination step of classifying the acquired packets into a first type of packets and a second type of packets different from the first type of packets, and determining a feature for each type of classified packet; and an estimation step of estimating the quality of the service performed by the first communication device using a learning model trained using the feature and the quality of the service at the time when the packets used to determine the feature are transmitted. Effect of the Invention

[0010] One aspect of the present invention can reduce the processing load in network monitoring. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a communication system 10. As shown in FIG. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of the monitoring device 100. [Diagram 3] FIG. 3 is a diagram illustrating an example of the software configuration of the monitoring device 100. As shown in FIG. [Figure 4] FIG. 4 is a diagram showing an example of a model learning process in the monitoring device 100. As shown in FIG. [Diagram 5] FIG. 5 is a diagram illustrating an example of feature amounts of an upstream packet. [Figure 6] FIG. 6 is a diagram illustrating an example of a processing flowchart of the teacher data generation process S100. [Figure 7] FIG. 7 is a diagram showing an example of a process flowchart of the supervised data generation process S200. [Figure 8] FIG. 8 is a diagram showing an example of a process flowchart of the service quality estimation model learning process S300. [Figure 9] FIG. 9 is a diagram showing an example of a service quality estimation process in the monitoring device 100. As shown in FIG. [Figure 10] FIG. 10 is a diagram showing an example of a process flowchart of the service quality estimation process S400. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] [First embodiment] A first embodiment will be described.

[0013] <About the communication system 10> 1 is a diagram showing an example of the configuration of a communication system 10. The communication system 10 includes a monitoring device 100, a client machine 200, a server machine 400, and a network 300. The communication system 10 is a client-server system in which the server machine 400 transmits corresponding data to the client machine 200 in response to a request for a service executed by the client machine 200. The service to be executed is, for example, a video distribution application. Note that the service may be rephrased as, for example, an application.

[0014] The client machine 200 is a communication device that executes a service and receives data from the server machine 400, and is, for example, a tablet terminal device, a smartphone, or a computer. In executing a video distribution application, the client machine 200, for example, transmits a request message for video data to the server machine 400, receives the video data from the server machine 400, and transmits an acknowledgment message (ACK: ACKnowledgement) to the server machine 400 indicating that the video data has been normally received. Furthermore, when the client machine 200 fails to normally receive the video data, it transmits a non-acknowledgement message (NACK: Non-ACKnowledgement) to the server machine 400 indicating that the video data has not been normally received.

[0015] The server machine 400 is a communication device, such as a computer or a server machine, that responds to a request from the client machine 200 and transmits data to the client machine 200. When the client machine 200 executes a video distribution application, the server machine 400 receives a request message for video data from the client machine 200, and transmits the video data corresponding to the request to the client machine 200 by including the packet or packets in the packet. When the server machine 400 receives an ACK, the server machine 400 transmits, for example, the subsequent video data to the client machine 200. When the server machine 400 receives a NACK, the server machine 400 retransmits the video data included in the packet corresponding to the NACK to the client machine 200.

[0016] The network 300 is a network that connects the client machine 200, the server machine 400, and the monitoring device 100, and is, for example, the Internet or an infrastructure network. The network 300 can transmit and receive packets between communication devices. The packets can include data and messages. The network 300 performs communication according to a communication protocol such as HTTP (HyperText Transfer Protocol) or HTTPS (HyperText Transfer Protocol Secure).

[0017] The monitoring device 100 is a communication processing device that acquires packets transmitted between the client machine 200 and the server machine 400 in the network 300 and monitors the network, and is, for example, a computer or a server machine. The monitoring device 100, for example, captures packets transmitted in the upstream direction (from the client machine 200 to the server machine 400) on the network 300 and monitors the state of the network. The monitoring device 100 also estimates the quality of the service being executed (hereinafter, may be referred to as service quality) from the state of the network. The service quality is, for example, MOS (Mean Opinion Score). MOS is, for example, a value obtained by averaging the evaluation results by the number of evaluators, in which evaluators watch a distributed video whose image quality changes depending on the state of the network and evaluate it on a five-point scale from 1 (bad) to 5 (good), and is an evaluation value that includes human subjectivity. The service quality used is not limited to MOS. <Configuration Example of Monitoring Device 100> 2 is a diagram showing an example of the configuration of the monitoring device 100. The monitoring device 100 includes a CPU (Central Processing Unit) 110, a storage 120, a memory 130, and a communication circuit 150.

[0018] The storage 120 is an auxiliary storage device that stores programs and data, such as a flash memory, a hard disk drive (HDD), or a solid state drive (SSD). The storage 120 stores a communication monitoring program 121, a model learning program 122, and a service quality estimation model 123.

[0019] The service quality estimation model 123 is an AI (Artificial Intelligence) learning model that estimates the service quality in the client machine 200. The service quality estimation model 123 is a learning model that is trained using, for example, feature quantities related to packets and corresponding service quality as input data. Note that, although there is only one service quality estimation model 123 in FIG. 2, there are multiple service quality estimation models for each type of application when multiple types of applications are supported.

[0020] The memory 130 is an area into which the programs stored in the storage 120 are loaded. The memory 130 may also be used as an area in which the programs store data.

[0021] The communication circuit 150 is a device that transmits and receives packets to and from other communication devices, and is, for example, a network interface circuit. The communication circuit 150 communicates with the client machine 200 and the server machine 400, and obtains packets transmitted and received over the network 300.

[0022] The CPU 110 is a processor that loads a program stored in the storage 120 into the memory 130, executes the loaded program, configures each unit, and realizes each process.

[0023] The CPU 110 executes the communication monitoring program 121 to construct a packet collection unit, a packet acquisition unit, and a packet type identification unit, and performs communication monitoring processing. The communication monitoring processing is processing for acquiring packets transmitted and received over the network 300, and monitoring the state of the network 300. The communication monitoring processing includes processing for acquiring (capturing) packets transmitted and received over the network 300, processing for analyzing and classifying the contents and types of the acquired packets, and processing for calculating the feature quantities of the classified packets.

[0024] The CPU 110 executes a packet collection module 1211 included in the communication monitoring program 121 to construct a packet collection unit and an acquisition unit and perform packet collection processing. The packet collection processing is processing for acquiring packets transmitted between the client machine 200 and the server machine 400 over the network 300. The packet collection processing acquires, for example, upstream packets.

[0025] The CPU 110 executes the packet analysis module 1212 of the communication monitoring program 121 to construct a classification unit, an application estimation unit, and a packet type identification unit, and performs packet analysis processing. The packet analysis processing is processing for analyzing the contents and type of an acquired packet and classifying the packet. The monitoring device 100 classifies the packet, for example, as either an ACK or something else. The packet analysis processing also includes processing for identifying (estimating) the type and name of the application to which the acquired packet is sent or received.

[0026] The CPU 110 executes a feature calculation module 1213 included in the communication monitoring program 121 to construct a feature calculation unit (feature determination unit) and perform feature calculation processing. The feature calculation processing is processing for calculating (extracting) feature amounts, such as the number of packets or the amount of data in a predetermined time, for each classified packet (or packet group).

[0027] The CPU 110 executes the model learning program 122 to construct a model learning unit and perform a model learning process. The model learning process is a process for learning a service quality estimation model 123, which will be described later. The monitoring device 100 inputs answer data and teacher data to the service quality estimation model 123 to perform the model learning process.

[0028] <Example of software configuration of monitoring device 100> 3 is a diagram illustrating an example of a software configuration of the monitoring device 100. The monitoring device 100 executes a program to construct a packet collection unit 1001, an application estimation unit 1002, a packet type identification unit 1003, a feature calculation unit 1004, a model learning unit 1005, and a service quality estimation unit 1006.

[0029] The packet collection unit 1001 performs a packet collection process to acquire packets (packets) transmitted and received over the network 300 between the client machine 200 and the server machine 400 .

[0030] The application estimation unit 1002 analyzes the type and contents of the acquired packet, and estimates (classifies) the type of application being executed on the client machine 200 .

[0031] The packet type identification unit 1003 performs a packet analysis process to analyze the contents and type of each packet in the acquired packet group and classify the packets. The packet type identification unit 1003 classifies the packets, for example, as either ACK or something else.

[0032] The feature amount calculation unit 1004 performs a feature amount calculation process for calculating a feature amount relating to the number of packets or the amount of data in a predetermined time for each classified packet.

[0033] The model learning unit 1005 inputs the correct answer data and the teacher data into the service quality estimation model 123, and performs a model learning process for learning the model.

[0034] The service quality estimation unit 1006 performs a service quality estimation process for estimating the service quality by using the feature amount of the packet. The service quality estimation unit 1006 is, for example, the service quality estimation model 123.

[0035] <Model learning process> A description will be given of a model learning process of the service quality estimation model 123. Fig. 4 is a diagram showing an example of the model learning process in the monitoring device 100.

[0036] 4, the client machine 200 executes, for example, a video distribution application. The video distribution application is an application that requests the server machine 400 to transmit a video, receives packets including divided video data from the server machine 400, and plays the video on a screen on the client machine 200. In the upstream direction of the network 300, for example, a request packet including a request for video data, an ACK packet including an ACK indicating that the video was successfully received, a NACK packet indicating that the video was not successfully received, and the like flow. In the downstream direction of the network 300, for example, a data packet including video data flows.

[0037] The monitoring device 100 performs packet acquisition (S10). Packet acquisition is a process of acquiring a packet group PG10 that is being transmitted in the upstream direction from the network 300 during a predetermined period of time.

[0038] The monitoring device 100 performs packet classification (S11). Packet classification includes a process of dividing the acquired packets into flows. Thereafter, the monitoring device 100 executes model learning, application estimation, feature calculation, and service quality estimation for each divided flow.

[0039] In addition, packet classification involves application estimation, which estimates the type of application being executed by client machine 200 from the type and contents of the acquired packet. Note that in the model learning process, an application may be executed for model learning, and the type of application may be obvious even without estimation. In this case, the estimation process does not need to be executed.

[0040] The application may be estimated using, for example, the destination or source address of a packet, a port number, etc. For example, if the traffic pattern of a certain service has characteristics, a correspondence between the characteristics and the service (application type) may be prepared in advance, and the application may be estimated using the correspondence. For example, a traffic pattern characteristic of a video distribution application is that a large file is intermittently transmitted from the server machine 400 to the client machine 200. Therefore, the monitoring device 100 can analyze the data size included in the packet, the number of packets within a predetermined time, the packet transmission interval, etc., and estimate that it is a video distribution application.

[0041] Thereafter, the monitoring device 100 executes processes such as feature determination and service quality estimation for each estimated type of application. In addition, packet classification is performed by analyzing the contents and type of each acquired packet and classifying the packets. For example, packets are classified into ACK packets and other packets (called non-ACK packets).

[0042] The monitoring device 100 calculates a feature amount for each of the classified packets. For example, the monitoring device 100 calculates a feature amount for an ACK packet (S12), and calculates a feature amount for a non-ACK packet (S13).

[0043] Fig. 5 is a diagram showing an example of the feature quantities of an upstream packet. Fig. 5 shows an example of a packet when a video distribution application is executed. When no error occurs in data transmission, the packets transmitted as upstream packets are a request packet (non-ACK packet) and an ACK packet.

[0044] The monitoring device 100 acquires the packet group G1 for a predetermined time L from the acquisition start time T. The monitoring device 100 acquires packets (packet groups) periodically (every predetermined time L), for example.

[0045] The monitoring device 100 classifies each packet of the packet group G1 as an ACK packet or a non-ACK packet according to, for example, the packet size. The monitoring device 100 classifies packets with a packet size PS or less as ACK packets, and classifies the rest as non-ACK packets. This classification method utilizes the fact that the packet size of an ACK packet is smaller than that of other packets (e.g., request packets) and that the packet size of an ACK packet is generally fixed. The size of an ACK packet may be calculated, for example, by calculating the frequency of occurrence of packet sizes in a measured section (time) and finding the average size of the most frequently occurring packet. Note that the packets may be classified according to information in the header portion of the packet (packet type) or the like. In FIG. 5, the packets are classified into ACK packet groups PG4 and PG5 and non-ACK packets P1 to P3.

[0046] The feature amount is determined (calculated) from, for example, the packet transmission interval, the total number of packets, the total packet capacity, the packet throughput, and values ​​obtained by statistically processing these during a predetermined period. The feature amount used for learning (used as teacher data) may be one type or multiple types of feature amount. The statistical processing includes, for example, the mean, maximum value, minimum value, mode, median, variance, kurtosis, and skewness.

[0047] In addition to the numerical values ​​calculated for each ACK packet and non-ACK packet, the feature may also include, for example, a relational value between the ACK packets and non-ACK packets (for example, the ratio between ACK packets and non-ACK packets, a comparison value of the total amount of data, etc.).

[0048] For example, in the case of a video distribution application, when the communication state is good and there is no error, a plurality (for example, a predetermined number) of downstream video packets are received for one upstream request packet, and a plurality (for example, a predetermined number) of upstream ACK packets are transmitted in response to the received packets. In other words, when the video application is operating well, it can be assumed that a state in which a predetermined number of ACK packets are transmitted for one non-ACK packet occurs periodically and continuously for a certain period of time. In this way, the monitoring device 100 can estimate the execution state of the application (the state of packets in the network) from the feature amount of the upstream packet. The service quality estimation model 123 can further estimate the service quality by inputting the corresponding service quality as correct answer data in addition to the feature amount of the packet and learning it.

[0049] Furthermore, the feature amount to be determined may be, for example, only ACK packets or only non-ACK packets. The monitoring device 100 may determine the feature amount of only packets that have the feature of an ACK packet or a non-ACK packet (the one with the more prominent feature).

[0050] 4, the monitoring device 100 generates training data from the feature amounts of the ACK packets and the feature amounts of the non-ACK packets (S14). The training data is data used for model learning.

[0051] Meanwhile, the monitoring device 100 acquires data from the client machine 200 (S15). The data is data used to generate supervised data to be used for model learning. When, for example, MOS is used as the service quality, the acquired data is, for example, an evaluation by the user of the client machine 200 (for example, an evaluation on a five-level scale from 1 (bad) to 5 (good)). When, for example, MOS is used as the service quality, the acquired data is, for example, image information related to images (for example, image resolution, bit rate, display resolution of the client machine 200, etc.).

[0052] The monitoring device 100 generates correct answer data including the service quality at the timing (period) when the feature was calculated from the acquired data (S16). For example, when the acquired data is user evaluation, the monitoring device 100 regards the average value of the evaluation during the period when the feature was calculated as the correct answer data. Also, when the acquired data is image information, for example, the monitoring device 100 calculates an approximation value of MOS using the image information during the period when the feature was calculated using the calculation formula of "ITU-T P.1203", and regards the calculated approximation value as the correct answer data. "ITU-T P.1203" is a standardized specification that defines a calculation formula for calculating an approximation value of MOS from image information as a function for estimating short-term quality of audio / video.

[0053] The monitoring device 100 performs model learning using the correct answer data and the teacher data (S17) and generates a trained service quality estimation model 123.

[0054] <Flowchart of model learning process> The model learning process will be described in detail below. Here, the following will explain each of the teacher data generation process S100 for generating teacher data, the supervised data generation process S200 for generating supervised data, and the service quality estimation model learning process S300 for generating (learning) the service quality estimation model 123.

[0055] <1. Teacher data generation process S100> 6 is a diagram showing an example of a processing flowchart of the teacher data generation process S100. The monitoring device 100 acquires a packet group in the upstream direction (S100-1). The monitoring device 100 acquires the packet group periodically, for example. Thereafter, the teacher data generation process is executed for each packet flow and application.

[0056] The monitoring device 100 classifies the type of each packet included in the packet group (S100-2). In the process S100-2, the packets are classified into ACK packets and non-ACK packets according to the size of the packets, for example.

[0057] The monitoring device 100 uses the ACK packets (Yes in S100-3) to calculate the feature amounts of the ACK packets (S100-4). On the other hand, the monitoring device 100 uses the non-ACK packets (No in S100-3) to calculate the feature amounts of the non-ACK packets (S100-5).

[0058] The monitoring device 100 generates training data for model learning from the feature amounts of the ACK packets and the non-ACK packets (S100-6). The monitoring device 100 executes the service quality estimation model learning process S300 using the training data, and ends the process.

[0059] <2. Correct Data Generation Process S200> 7 is a diagram showing an example of a process flowchart of the supervised data generation process S200. The monitoring device 100 acquires data from the client machine 200 (S200-1). The data is, for example, data used to calculate a MOS or an approximate value of a MOS. The data may be acquired in response to the acquisition of a packet group in the teacher data generation process S100.

[0060] The monitoring device 100 determines the service quality from the acquired data (S200-2).

[0061] The monitoring device 100 generates correct answer data including the determined service quality (S200-3).

[0062] The monitoring device 100 executes the service quality estimation model learning process S300 using the correct answer data, and ends the process.

[0063] <3. Service quality estimation model learning process S300> 8 is a diagram showing an example of a process flowchart of the service quality estimation model learning process S300. The monitoring device 100 acquires teacher data and supervised data (S300-1).

[0064] The monitoring device 100 associates the time of the teacher data with the time of the correct answer data (S300-2). The monitoring device 100 associates the time when the packet that calculated the feature amount of the teacher data was transmitted or acquired with the time when the data used to generate the correct answer data was acquired or generated. In other words, the monitoring device 100 associates the timing when the packet used to generate the teacher data was flowing on the network 300 with the quality of the service being executed on the client machine 200, and uses it for learning.

[0065] The monitoring device 100 inputs the associated teacher data and correct answer data to the service quality estimation model 123, trains the model (S300-3), and ends the process. For example, an algorithm used in the training may be Random Forest, Multilayer Perceptron, Long Short Term Memory, or Transformer. Through the process S300-3, the service quality estimation model 123 becomes a trained model.

[0066] <Service quality estimation process> A description will be given of a service quality estimation process using the trained service quality estimation model 123. FIG.

[0067] 9, the client machine 200 is executing a video distribution application. The video distribution application is an application that requests the server machine 400 to transmit a video, receives packets including divided video data from the server machine 400, and plays the video on a screen on the client machine 200. In the upstream direction of the network 300, for example, a request packet including a request for video data, an ACK packet including an ACK indicating that the video was successfully received, a NACK packet indicating that the video was not successfully received, and the like flow. In the downstream direction of the network 300, for example, a data packet including video data flows.

[0068] The monitoring device 100 performs packet acquisition (S20). Packet acquisition is a process of acquiring a packet group PG20 that is being transmitted in the upstream direction from the network 300 during a predetermined period of time.

[0069] The monitoring device 100 performs packet classification (S21). The packet classification includes a process of dividing the acquired packets into flows. The packet classification also estimates the application being executed. In this case, the application being executed is estimated to be a video distribution application.

[0070] The monitoring device 100 calculates a feature amount for each classified packet. For example, the monitoring device 100 calculates a feature amount for an ACK packet (S22) and calculates a feature amount for a non-ACK packet (S23). The calculated feature amount includes all or a part of the feature amount used to generate the training data.

[0071] The monitoring device 100 inputs the feature amount of the ACK packet and the feature amount of the non-ACK packet to the service quality estimation model 123, and causes it to output an estimated service quality. The service quality estimation model 123 used is a model that has already been trained and corresponds to the application being executed. The estimated service quality to be output is the service quality used to generate the correct answer data, for example, MOS (including an approximate value).

[0072] <Processing flowchart of service quality estimation process> The service quality estimation process will be described in detail with reference to Fig. 10, which is a diagram showing an example of a processing flowchart of the service quality estimation process S400.

[0073] The monitoring device 100 acquires a group of packets in the upstream direction (S400-1). The monitoring device 100 acquires the group of packets periodically, for example.

[0074] The monitoring device 100 classifies the type of each packet included in the packet group (S400-2). In the process S400-2, the packets are classified into ACK packets and non-ACK packets according to the size of the packets, for example.

[0075] The monitoring device 100 uses the ACK packets (Yes in S400-3) to calculate the feature amounts of the ACK packets (S400-4). On the other hand, the monitoring device 100 uses the non-ACK packets (No in S400-3) to calculate the feature amounts of the non-ACK packets (S400-5).

[0076] The monitoring device 100 inputs the feature amounts of the ACK packets and the non-ACK packets to the service quality estimation model 123 (S400-6).

[0077] The monitoring device 100 acquires the estimated service quality output by the service quality estimation model 123 (S400-7), and ends the process.

[0078] [Second embodiment] In the first embodiment, for example, MOS is used as the service quality. In the second embodiment, for example, KQI (Key Quality Indicator) is used as the service quality. Note that the service quality used is not limited to KQI.

[0079] The KQI includes, for example, the bit rate, resolution, and stall time of a video segment. The client machine 200, for example, periodically acquires the bit rate and resolution and transmits them to the monitoring device 100. The monitoring device 100 may also periodically acquire the bit rate from the client machine 200.

[0080] Also, if the KQI is stall time, the client machine 200 may, for example, measure the time that the video is stopped, save it as the stall time, and transmit the stall time to the monitoring device 100 upon request from the monitoring device 100 or periodically.

[0081] When, for example, KQI is used as the service quality, the process S15 in FIG. 4 and the process S200-1 in FIG. 7 are, for example, stall time, resolution, or bit rate, and the correct answer data is generated using the data.

[0082] [Third embodiment] In the first embodiment, for example, upstream packets are acquired because a video distribution application is being executed, but in the third embodiment, downstream packets are acquired. Alternatively, in the third embodiment, the direction of packet acquisition is switched between upstream and downstream for each type of service.

[0083] For example, the monitoring device 100 may execute a service in advance, acquire upstream and downstream packets, and determine the direction from which the number of packets is small as the direction from which packets are to be acquired.

[0084] For example, a service that transmits video captured by a client machine 200 to a server machine 400 will have fewer downstream packets than upstream packets, so when executing this type of service, the monitoring device 100 may acquire downstream packets.

[0085] [Fourth embodiment] In the first embodiment, the monitoring device 100 classifies acquired packets into, for example, ACK packets and non-ACK packets. In the fourth embodiment, the monitoring device 100 may further extract request packets from, for example, non-ACK packets. The request packets are, for example, HTTP requests in HTTP or HTTPS. The monitoring device 100 checks the packet header and handles the HTTP request as a request packet.

[0086] [Other embodiments] The embodiments may be used in combination with each other. The service quality and the feature amount are merely examples, and may be other than those described above.

[0087] It is possible to assume that the features of an ACK packet are correlated with, for example, the bit rate of the video being played. For example, the lower the standard deviation of the packet transmission interval, the closer the packet interval is to a constant, and the more stable the network is, and the higher the possibility that high-bit rate video is being transmitted, and the higher the service quality.

[0088] In addition, the higher the kurtosis of the packet transmission interval, the smaller the variation and the more stable the distribution. This means that it is possible to assume a stable network, that it is highly likely that high bitrate video is being transmitted, and that the service quality is high.

[0089] In addition, the larger the value of the distortion of packet transmission intervals, the more the distribution of packet intervals is biased to the left (0), which means that packet intervals tend to be small. Since packets are sent and received at high speed, it can be assumed that the network is stable, that high bitrate video is likely being transmitted, and that the service quality is high.

[0090] In this manner, the monitoring device 100 may determine the feature amount assuming various states.

[0091] To summarise, the following is an appendix.

[0092] (Appendix 1) 1. An estimation method for estimating quality of a service executed by a first communication device in a communication system having a first communication device that executes a service, a second communication device that communicates with the first communication device via a network, and a monitoring device that monitors the network, comprising: acquiring packets transmitted in a first direction on the network; a feature determining step of classifying the acquired packets into a first type of packets and a second type of packets different from the first type of packets, and determining a feature for each type of classified packets; an estimation step of estimating a quality of a service performed by the first communication device by using a learning model trained using the feature amount and a quality of service at a timing when the packet used for calculating the feature amount is transmitted; The estimation method has the following structure:

[0093] (Appendix 2) The first type of packet is a packet that includes an acknowledgment indicating successful reception of data. Estimation method described in Appendix 1.

[0094] (Appendix 3) the service is a video distribution application; The first direction is from the first communication device to the second communication device. Estimation method described in Appendix 2.

[0095] (Appendix 4) In the feature determination step, the second type of packets are classified into request packets that the first communication device makes a request to the second communication device and other packets, and features of the request packets are determined. Estimation method described in Appendix 3.

[0096] (Appendix 5) The first direction is a direction in which a smaller number of packets are transmitted within a predetermined time by the execution of the service. Estimation method described in Appendix 1.

[0097] (Appendix 6) Further, the method includes a learning step of learning the learning model using the feature amount determined by executing the acquisition step and the feature amount determination step and the quality of service acquired from the first communication device. Estimation method described in Appendix 1.

[0098] (Appendix 7) The quality of the service is MOS; In the learning step, image information relating to the image being reproduced is acquired from the first communication device, the image information is used to calculate an approximation of the MOS, and the approximation is regarded as the quality of the service. Estimation method described in Appendix 6.

[0099] (Appendix 8) said quality of service is KQI, In the learning step, image information relating to the image being reproduced is acquired from the first communication device, and the image information is regarded as the service quality. Estimation method described in Appendix 6.

[0100] (Appendix 9) An estimation program for estimating quality of a service executed by a first communication device in a communication system having a first communication device that executes an application, a second communication device that communicates with the first communication device via a network, and a monitoring device that monitors the network, the estimation program comprising: acquiring packets transmitted in a first direction on the network; a feature determining step of classifying the acquired packets into a first type of packets and a second type of packets different from the first type of packets, and determining a feature for each type of classified packets; an estimation step of estimating the quality of the service performed by the first communication device by using a learning model trained using the feature amount and the quality of the service at the timing when the packet used to determine the feature amount is transmitted; An estimation program to be executed by a computer included in the monitoring device.

[0101] (Appendix 10) A communication system including a first communication device that executes a service, a second communication device that communicates with the first communication device via a network, and a communication processing device that monitors the network, the communication processing device estimating quality of a service executed by the first communication device, an acquisition unit that acquires packets transmitted in a first direction on the network; a feature determining unit that classifies the acquired packets into a first type of packets and a second type of packets different from the first type of packets, and determines a feature for each type of classified packets; an estimation unit that estimates a quality of a service performed by the first communication device by using a learning model that is trained using the feature amount and a quality of the service at a timing when the packet used to determine the feature amount is transmitted; A communication processing device having: [Explanation of symbols]

[0102] 10: Communication Systems 100: Monitoring device 110: CPU 120: Storage 121: Communications monitoring program 1211: Packet collection module 1212: Packet analysis module 1213: Feature Calculation Module 122: Model learning program 123: Service quality estimation model 130: Memory 150: Communication circuit 200: Client machine 300 : Network 400: Server machine 1001: Packet collection unit 1002: Application Estimation Department 1003: Packet type identification section 1004: Feature calculation unit 1005: Model Learning Department 1006: Service quality estimation unit

Claims

1. 1. An estimation method for estimating quality of a service in a communication system having a first communication device that executes a service, a second communication device that communicates with the first communication device via a network, and a monitoring device that monitors the network, comprising: acquiring packets transmitted in a first direction on the network; a feature determining step of classifying the acquired packets into a first type of packets and a second type of packets different from the first type of packets, and determining a feature for each type of classified packets; an estimation step of estimating a quality of a service performed by the first communication device by using a learning model trained using the feature amount and the quality of the service at a timing when the packet used for determining the feature amount is transmitted; The estimation method has the following structure:

2. the first type of packet is a packet containing an acknowledgment indicating successful reception of data; The estimation method according to claim 1.

3. the service is a video distribution application; The first direction is from the first communication device to the second communication device. The estimation method according to claim 2.

4. In the feature determining step, the second type packets are classified into request packets that the first communication device makes a request to the second communication device and other packets, and features of the request packets are determined. The estimation method according to claim 3.

5. The first direction is a direction in which a smaller number of packets are transmitted within a predetermined time by the execution of the service. The estimation method according to claim 1.

6. Further, the method includes a learning step of learning the learning model using the feature amount determined by executing the acquisition step and the feature amount determination step and the quality of service acquired from the first communication device. The estimation method according to claim 1.

7. An estimation program for estimating quality of a service in a communication system having a first communication device that executes a service, a second communication device that communicates with the first communication device via a network, and a monitoring device that monitors the network, the estimation program comprising: acquiring packets transmitted in a first direction on the network; a feature determining step of classifying the acquired packets into a first type of packets and a second type of packets different from the first type of packets, and determining a feature for each type of classified packets; an estimation step of estimating a quality of a service performed by the first communication device using a learning model trained using the feature amount and the quality of the service at a timing when a packet used to determine the feature amount is transmitted; An estimation program to be executed by a computer included in the monitoring device.

8. A communication system including a first communication device that executes a service, a second communication device that communicates with the first communication device via a network, and a communication processing device that monitors the network, the communication processing device estimating quality of a service executed by the first communication device, an acquisition unit that acquires packets transmitted in a first direction on the network; a feature determining unit that classifies the acquired packets into a first type of packets and a second type of packets different from the first type of packets, and determines a feature for each type of classified packets; an estimation unit that estimates a quality of a service performed by the first communication device by using a learning model that is trained using the feature amount and the quality of the service at a timing when a packet used to determine the feature amount is transmitted; A communication processing device having:

Images