Network experience quality evaluation method and device, electronic equipment and program product

Abstract

The embodiment of the invention discloses a network experience quality evaluation method and device, electronic equipment and a program product. The method comprises: obtaining network quality of service (QoS) data, the network QoS data comprising network QoS information respectively corresponding to a plurality of moments; processing the network QoS data through a first network QoE (Quality of Experience) evaluation model to obtain a first network QoE evaluation result; processing the network QoS data through a second network QoE (Quality of Experience) evaluation model to obtain a second network QoE evaluation result; the model architecture of the first network QoE evaluation model is different from that of the second network QoE evaluation model; and determining a target network QoE evaluation result according to the first network QoE evaluation result and the second network QoE evaluation result. According to the network QoE evaluation method and device, the electronic equipment and the program product, the accuracy of evaluating the network QoE of the user is improved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a network experience quality evaluation method, device, electronic device and program product. Background Art

[0002] With the rapid development of Internet technology, all electronic devices can be connected to the Internet, allowing users to obtain the information they need from the Internet, or conduct shopping, entertainment and other activities through the Internet. When users use electronic devices, the user's network QoE (Quality of Experience) is an important indicator to measure user experience and can reflect the user's overall experience of the network. How to accurately evaluate the user's network QoE has always been one of the hot issues in the industry. Summary of the invention

[0003] The embodiments of the present application provide a network experience quality evaluation method, device, electronic device, and program product, which improve the accuracy of evaluating a user's network QoE.

[0004] The present application discloses a method for evaluating network experience quality, the method comprising:

[0005] Acquire network quality of service (QoS) data, wherein the network QoS data includes network QoS information corresponding to multiple time periods;

[0006] Processing the network QoS data by using a first network quality of experience (QoE) evaluation model to obtain a first network QoE evaluation result;

[0007] Processing the network QoS data through a second network quality of experience (QoE) evaluation model to obtain a second network QoE evaluation result; the model architecture of the first network QoE evaluation model is different from the model architecture of the second network QoE evaluation model;

[0008] A target network QoE evaluation result is determined according to the first network QoE evaluation result and the second network QoE evaluation result.

[0009] The present application embodiment discloses a network experience quality evaluation device, the device comprising:

[0010] A data acquisition module, used to acquire network QoS data, wherein the network QoS data includes network QoS information corresponding to multiple time periods;

[0011] A first evaluation module, configured to process the network QoS data through a first network QoE evaluation model to obtain a first network QoE evaluation result;

[0012] A second evaluation module, configured to process the network QoS data through a second network QoE evaluation model to obtain a second network QoE evaluation result; the model architecture of the first network QoE evaluation model is different from the model architecture of the second network QoE evaluation model;

[0013] A fusion module is used to determine a target network QoE evaluation result according to the first network QoE evaluation result and the second network QoE evaluation result.

[0014] An embodiment of the present application discloses an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor implements the method described above.

[0015] An embodiment of the present application discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described above is implemented.

[0016] An embodiment of the present application discloses a computer program product, including a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0017] The network experience quality evaluation method, device, electronic device and program product disclosed in the embodiment of the present application obtain network QoS data, the QoS data includes network QoS information corresponding to multiple moments, processes the network QoS data through a first network QoE evaluation model to obtain a first network QoE evaluation result, processes the network QoS data through a second network QoE evaluation model to obtain a second network QoE evaluation result, and determines the target network QoE evaluation result according to the first network QoE evaluation result and the second network QoE evaluation result. In the embodiment of the present application, since the QoS data includes network QoS information corresponding to multiple moments, the first network QoE evaluation model and the second network QoE evaluation model can pay attention to the network changes over a period of time according to the network QoS information corresponding to multiple moments, and can more accurately perform network QoE evaluation, and respectively perform network QoE evaluation through the first network QoE evaluation model and the second network QoE evaluation model with different model architectures, and then determine the target network QoE evaluation result according to the network QoE evaluation results of the two, further improving the accuracy of evaluating the user's network QoE. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is an application scenario diagram of a network experience quality evaluation method in an embodiment;

[0020] Figure 2 is a flow chart of a method for evaluating network experience quality in one embodiment;

[0021] Figure 3 A schematic diagram of a flow chart of a method for evaluating network experience quality in one embodiment;

[0022] Figure 4 A flowchart of processing network QoS data by using a first network QoE evaluation model to obtain a first network QoE evaluation result in an embodiment;

[0023] Figure 5 A flowchart of extracting features of network QoS information corresponding to multiple time periods to obtain a first time series feature in an embodiment;

[0024] Figure 6 A flowchart of processing network QoS data by using a second network QoE evaluation model to obtain a second network QoE evaluation result in an embodiment;

[0025] Figure 7 A schematic diagram of a process of processing network QoS data by a second network QoE evaluation model in an embodiment;

[0026] Figure 8 is an architectural diagram of an FCN module in one embodiment;

[0027] Fig. 9 An architectural diagram of an LSTM module in one embodiment;

[0028] Fig.10 A flowchart of training a first network QoE evaluation model and a second network QoE evaluation model in one embodiment;

[0029] Fig.11 is a block diagram of a network experience quality evaluation device in one embodiment;

[0030] Fig.12 FIG. 4 is a structural block diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present application and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0033] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element. For example, without departing from the scope of this application, the first network QoE evaluation model may be referred to as the second network QoE evaluation model, and similarly, the second network QoE evaluation model may be referred to as the first network QoE evaluation model. Both the first network QoE evaluation model and the second network QoE evaluation model are network QoE evaluation models, but they are not the same network QoE evaluation model. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the schemes, or any combination of multiple schemes.

[0034] Network QoS (Quality of Service) data is an important basis for evaluating users' network QoE. In related technologies, it is usually based on mathematical model building and rule-based mapping of network QoS data to network QoE. In addition to rule-based network QoE evaluation methods, there are also machine learning models (such as random forests, classifiers, etc.) to evaluate users' network QoE.

[0035] Currently, when performing real-time network QoE evaluation of users, the network QoE evaluation is usually performed based on network QoS data at a single time point (such as network QoS data at the current moment), resulting in low accuracy of the user's network QoE evaluation.

[0036] The embodiments of the present application provide a network experience quality evaluation method, device, electronic device, and program product, which improve the accuracy of evaluating a user's network QoE.

[0037] Figure 1FIG. 1 is an application scenario diagram of a method for evaluating network experience quality in an embodiment. Figure 1 As shown, the network experience quality evaluation method can be applied to an electronic device 100, which may include but is not limited to mobile phones, wearable devices (such as smart watches, smart glasses, etc.), vehicle terminals, tablet computers, laptops, PCs (Personal Computers), and other devices.

[0038] In some embodiments, the electronic device 100 may store a first network QoE evaluation model and a second network QoE evaluation model, and the model architecture of the first network QoE evaluation model is different from the model architecture of the second network QoE evaluation model. The electronic device 100 may obtain network QoS data, which may include network QoS information corresponding to multiple moments, and may process the network QoS data through the model of the first network QoE evaluation model to obtain a first network QoE evaluation result, and may process the network QoS data through the model of the second network QoE evaluation model to obtain a second network QoE evaluation result. The electronic device 100 may determine a target network QoE evaluation result based on the first network QoE evaluation result output by the first network QoE evaluation model and the second network QoE evaluation result output by the second network QoE evaluation model, and the target network QoE evaluation result may be used to characterize the user's experience of the network service provided by the electronic device 100.

[0039] Optionally, the electronic device 100 may also be a server (such as a cloud) or a server cluster, and the user's terminal device (such as a mobile phone, a wearable device, etc.) may obtain network QoS data and send the obtained network QoS data to the server, and the server processes the network QoS data through the first network QoE evaluation model and the second network QoE evaluation model, respectively, and obtains the first network QoE evaluation result output by the first network QoE evaluation model and the second network QoE evaluation result output by the second network QoE evaluation model. The server may determine the target network QoE evaluation result based on the first network QoE evaluation result and the second network QoE evaluation result, and feed the target network QoE evaluation result back to the terminal device.

[0040] like Figure 2 As shown, in one embodiment, a network experience quality evaluation method is provided, which can be applied to the above electronic device. The method may include the following steps:

[0041] Step 210: Acquire network QoS data, where the network QoS data includes network QoS information corresponding to multiple time periods.

[0042] When an electronic device needs to evaluate a user's network QoE, network QoS data may be obtained. Network QoS data may refer to data that can be used to describe network conditions. When the electronic device is a user's terminal device, the network QoS data may be used to characterize the network conditions of the electronic device.

[0043] The network QoS data may include network QoS information corresponding to multiple moments. Further, the network QoS information may include parameter values ​​corresponding to multiple network QoS parameters, and the network QoS parameters may include transport layer parameters and radio frequency layer parameters. Among them, the transport layer parameters may refer to parameters used to indicate data transmission conditions. For example, the transport layer parameters may include but are not limited to one or more of the uplink transmission rate and downlink transmission rate based on the transmission protocol, the number of transmission data packets, the number of packet losses, etc. The transmission protocol may include but is not limited to UDP (User Datagram Protocol), TCP (Transmission Control Protocol), etc. The radio frequency layer parameters may refer to parameters used to indicate the quality of communication signals. For example, the radio frequency layer parameters may include but are not limited to one or more of signal strength, signal-to-noise ratio, bit error rate, etc.

[0044] Optionally, the electronic device can obtain network QoS information corresponding to multiple consecutive moments, and the time interval between two adjacent moments can be set according to actual needs. The time interval represents the period of the electronic device detecting the network QoS information. For example, the time interval between two adjacent moments can be 1 second, 2 seconds, 5 seconds, etc., but is not limited to this.

[0045] In some embodiments, the electronic device can evaluate the user's real-time network QoE. The electronic device can obtain the network QoS data corresponding to the current evaluation cycle according to the evaluation cycle, and determine the user's real-time network QoE based on the network QoS data corresponding to the current evaluation cycle.

[0046] Optionally, the network QoS data corresponding to the current evaluation period may include the network QoS information corresponding to the current moment and N historical moments before the current moment, respectively, where N is a positive integer. Further, the cycle duration of each evaluation period may be greater than or equal to the duration corresponding to the current moment and the N historical moments before the current moment. It should be noted that the cycle duration of the evaluation period can be set according to actual needs and is not limited here.

[0047] Here, N may be a preset fixed value, or N may be determined according to the current business scenario.

[0048] The current business scenario may refer to the scenario corresponding to the business currently being performed by the electronic device. For example, the current business scenario may be determined based on the application currently running on the electronic device, or the current business scenario may be determined based on the interface content currently displayed on the electronic device.

[0049] As an implementation method, N can be determined based on the sensitivity of the current business scenario to the network quality, and the sensitivity to the network quality can be used to indicate whether the impact of network fluctuations is obvious. For business scenarios that are highly sensitive to network quality, the corresponding N can be smaller. For example, business scenarios such as gaming business scenarios that have high requirements for network real-time performance have high sensitivity to network quality, so the corresponding N can be smaller; business scenarios such as web browsing scenarios that have low requirements for network real-time performance have low sensitivity to network quality, so the corresponding N can be larger.

[0050] As another implementation, N may be determined according to the business scenario type corresponding to the current business scenario, which may include a buffering business scenario, an instant business scenario, etc., wherein a buffering business scenario may refer to a business scenario that caches data through a network, such as video playback, audio playback, etc., and an instant business scenario may refer to a business scenario that accesses real-time data through a network, such as a game business scenario, a chat business scenario, etc. Among them, the buffering business scenario pays more attention to the continuous changes in the network over a period of time, so the corresponding N may be larger, while the instant business scenario pays more attention to the recent network conditions, so the corresponding N may be smaller, etc.

[0051] In an embodiment of the present application, the electronic device can determine the number N of historical moments included in the network QoS data based on the current business scenario, that is, it can flexibly adjust the time length corresponding to the network QoS data according to the current business scenario, so that the acquired network QoS data can be more in line with the current business scenario, thereby further improving the accuracy of evaluating the user's network QoE.

[0052] Step 220: Process the network QoS data using a first network QoE evaluation model to obtain a first network QoE evaluation result.

[0053] Step 230: Process the network QoS data using a second network QoE evaluation model to obtain a second network QoE evaluation result.

[0054] The first network QoE evaluation model and the second network QoE evaluation model may be pre-trained models used to evaluate the user's network QoE, and the model architecture of the first network QoE evaluation model and the model architecture of the second network QoE evaluation model may be different.

[0055] Exemplarily, the first network QoE evaluation model may be an artificial intelligence model such as a decision tree or a random forest, and the second network QoE evaluation model may be a convolutional neural network, a recurrent neural network, or the like.

[0056] In some embodiments, the first network QoE evaluation model may be an artificial intelligence model of the GBDT (Gradient Boosting Decision Tree) type. Since the network QoS data has uneven feature distribution and contains both numerical data and categorical data, the network QoS data may be regarded as structured data (similar to tabular data). The GBDT type artificial intelligence model is better at analyzing structured data and can better handle data with more complex feature dependencies. Exemplarily, the first network QoE evaluation model may include an XGBoost model, which is an optimized distributed gradient enhancement model. The prediction performance of the model is improved by optimizing the gradient boosting algorithm. The XGBoost model can process structured network QoS data to achieve user QoE evaluation.

[0057] In some embodiments, the second network QoE evaluation model may be a convolutional neural network, or a combination of a convolutional neural network and a recurrent neural network, for processing time series. Further, the second network QoE evaluation model may be a combination of FCN (Fully Convolutional Networks) and LSTM (Long Short-Term Memory) that can process time series. Exemplarily, the second network QoE evaluation model may include an MLSTM-FCN model, which may process network QoS data based on time series to achieve user QoE evaluation.

[0058] After acquiring the network QoS data, the electronic device may process the network QoS data through the first network QoE evaluation model to obtain a first network QoE evaluation result, which may refer to the user's network QoE predicted by the first network QoE evaluation model.

[0059] As an implementation method, since the network QoS data includes network QoS information corresponding to multiple moments, the time series characteristics corresponding to the network QoS data can be extracted according to the network QoS information corresponding to the multiple moments, and the time series characteristics can refer to characteristics that can reflect network changes over a period of time. The first network QoE evaluation model can analyze the time series characteristics and predict the first network QoE evaluation result.

[0060] After acquiring the network QoS data, the electronic device may also process the network QoS data through a second network QoE evaluation model to obtain a second network QoE evaluation result, which may refer to the user's network QoE predicted by the second network QoE evaluation model.

[0061] As an implementation method, the network QoS data includes network QoS information corresponding to multiple moments, and the network QoS information corresponding to the multiple moments can form a network QoS time series in chronological order, and each element of the network QoS time series corresponds to the network QoS information at one moment. For example, the network QoS time series can be expressed as {X1, X2, …, X i ,…,X N+1}, where X i The network QoS information at the i-th moment may be N+1 moments in total (the current moment and N historical moments before the current moment). The second network QoE evaluation model may process the network QoS time series and predict the second network QoE evaluation result.

[0062] It should be noted that there is no strict timing requirement between processing the network QoS data through the first network QoE evaluation model and processing the network QoS data through the second network QoE evaluation model. The two can be performed simultaneously in parallel, or the network QoS data can be processed first through the first network QoE evaluation model and then through the second network QoE evaluation model, or the network QoS data can be processed first through the second network QoE evaluation model and then through the first network QoE evaluation model. The embodiment of the present application does not limit this.

[0063] Step 240: Determine the target network QoE evaluation result according to the first network QoE evaluation result and the second network QoE evaluation result.

[0064] The electronic device may fuse the first network QoE evaluation result output by the first network QoE evaluation model with the second network QoE evaluation result output by the second network QoE evaluation model to obtain a target network QoE evaluation result, which may be used to reflect the user's network experience. Further, the target network QoE evaluation result may be used to reflect the user's real-time network experience.

[0065] For example, Figure 3 FIG. 1 is a flow chart of a method for evaluating network experience quality in one embodiment. Figure 3As shown, after the electronic device obtains the network QoS data, it can process the network QoS data through the first network QoE evaluation model and the second network QoE evaluation model respectively to obtain the first network QoE evaluation result and the second network QoE evaluation result, and then merge the first network QoE evaluation result and the second network QoE evaluation result to obtain the target network QoE evaluation result.

[0066] In some embodiments, the network QoE evaluation result may be represented by a network experience level, which may be pre-divided into multiple network experience levels. For example, the higher the network experience level, the better the user's network experience, and the lower the network experience level, the worse the user's network experience. Further, the first network QoE evaluation result may include the first network experience level, and the second network QoE evaluation result may include the second network experience level. The target network experience level (i.e., the target network QoE evaluation result) may be determined based on the first network experience level and the second network experience level. For example, the smaller of the first network experience level and the second network experience level may be used as the target network experience level; or, the first network experience level and the second network experience level may be averaged, and the average result may be rounded down to obtain the target network experience level, etc. The network experience level can accurately reflect the user's network experience.

[0067] In some embodiments, the network QoE evaluation result may be expressed by a jamming probability or by whether a jamming occurs. The jamming probability refers to the probability of jamming occurring. The greater the jamming probability, the worse the user's network experience. The smaller the jamming probability, the better the user's network experience. Further, the first network QoE evaluation result may include a first jamming probability, and the second network QoE evaluation result may include a second jamming probability. The target jamming probability may be determined based on the first jamming probability and the second jamming probability.

[0068] As an implementation method, the first jamming probability and the second jamming probability may be combined by weighted averaging to obtain a target jamming probability. The electronic device may perform weighted averaging according to the first jamming probability, the first weight corresponding to the first network QoE evaluation model, the second jamming probability, and the second weight corresponding to the second network QoE evaluation model to obtain the target jamming probability. Exemplarily, the target jamming probability may be calculated by the following formula (1):

[0069]

[0070] Wherein, P1 represents the first jamming probability, w1 represents the first weight corresponding to the first network QoE evaluation model, P2 represents the second jamming probability, w2 represents the second weight corresponding to the second network QoE evaluation model, and P t Indicates the target jam probability.

[0071] Optionally, the first weight and the second weight may be set according to actual business conditions, or may be obtained through training, which is not limited here.

[0072] After determining the target jamming probability, it can be determined whether the target jamming probability is greater than the probability threshold. If the target jamming probability is greater than the probability threshold, it can be determined that the target network QoE evaluation result is jamming, indicating that the user's network experience is poor; if the target jamming probability is not greater than the probability threshold, it can be determined that the target network QoE evaluation result is not jamming, indicating that the user's network experience is good. It should be noted that the probability threshold can be set according to actual needs or determined through a large number of experiments, and is not specifically limited here.

[0073] In the embodiment of the present application, the network QoE evaluation result is represented by the jamming probability, so whether jamming occurs can be accurately evaluated. Measuring the user's network experience by whether jamming occurs is more objective and stable, thereby improving the accuracy of the user's network QoE evaluation.

[0074] In some embodiments, after the electronic device determines the target network QoE evaluation result, it can also determine whether the network needs to be optimized based on the target network QoE evaluation result. For example, when the target network QoE evaluation result indicates that the user's network experience is poor, the network can be optimized. Optimizing the network may include but is not limited to switching the network accessed by the electronic device (such as switching access to different wireless hotspots, switching from a cellular network to a Wi-Fi network or from a Wi-Fi network to a cellular network, etc.), closing background applications that occupy a large network bandwidth, and reducing the resolution of the displayed image (such as reducing the quality of the video image to reduce the video data required to be read, etc.). One or more of the above. According to the target network QoE evaluation result, the user's network experience can be accurately measured, and the network can be appropriately optimized to improve the user's network experience.

[0075] In an embodiment of the present application, since the QoS data includes network QoS information corresponding to multiple time periods, the first network QoE evaluation model and the second network QoE evaluation model can pay attention to network changes over a period of time based on the network QoS information corresponding to multiple time periods, and can perform network QoE evaluation more accurately. The network QoE evaluation is performed using the first network QoE evaluation model and the second network QoE evaluation model with different model architectures, respectively, and the target network QoE evaluation result is determined based on the network QoE evaluation results of the two, thereby further improving the accuracy of evaluating the user's network QoE.

[0076] like Figure 4As shown, in one embodiment, the step of processing the network QoS data by using the first network QoE evaluation model to obtain the first network QoE evaluation result may include the following steps:

[0077] Step 402: extract features of network QoS information corresponding to multiple time periods to obtain first time series features.

[0078] After the electronic device obtains the network QoS data, it can extract features of the network QoS information corresponding to multiple moments contained in the network QoS data to obtain a first time series feature. The first time series feature can be a feature that can characterize the network changes in the time period corresponding to the multiple moments.

[0079] In some embodiments, the network QoS information may include parameter values ​​corresponding to multiple network QoS parameters. The network QoS information corresponding to each moment may include parameter values ​​corresponding to multiple network QoS parameters at that moment. The network QoS parameters may include one or more of the uplink transmission rate and downlink transmission rate based on the transmission protocol, the number of transmitted data packets, the number of packet losses, signal strength, signal-to-noise ratio and bit error rate.

[0080] Optionally, for each network QoS parameter, a second time series feature corresponding to the network QoS parameter may be determined according to the parameter values ​​corresponding to the network QoS parameter at multiple moments, and the second time series feature corresponding to the network QoS parameter may be used to characterize the parameter value of the network QoS parameter in the time period corresponding to the multiple moments, thereby reflecting the change of the network QoS parameter in the time period. The first time series feature may be determined according to the second time series feature corresponding to each network QoS parameter.

[0081] Optionally, some network QoS parameters may be selected from multiple network QoS parameters for feature extraction to obtain the first time series feature. Figure 5 As shown, in some embodiments, step 402 may include steps 502-506.

[0082] Step 502: Determine one or more target network QoS parameters from a plurality of network QoS parameters.

[0083] The electronic device may determine one or more target network QoS parameters from multiple network QoS parameters, and the one or more target network QoS parameters may be some of the multiple network QoS parameters. Optionally, the electronic device may determine one or more target network QoS parameters from multiple network QoS parameters according to the current business scenario, and the target network QoS parameters may be network QoS parameters with a high correlation with the network QoE evaluation result in the current business scenario.

[0084] In different business scenarios, the impact of various network QoS parameters on the network QoE evaluation results may be different. In the same business scenario, some network QoS parameters have a greater impact on the network QoE evaluation results, while some network QoS parameters have a smaller impact on the network QoE evaluation results. The same network QoS parameter has a greater impact on the network QoE evaluation results in some business scenarios, while has a smaller impact on the network QoE evaluation results in some business scenarios.

[0085] For example, in voice service scenarios, network QoS parameters such as signal quality and signal-to-noise ratio have a greater impact on network QoE evaluation results, while network QoS parameters such as uplink transmission rate, downlink transmission rate, and number of packet losses have a smaller impact on network QoE evaluation results; in video playback service scenarios, network QoS parameters such as uplink transmission rate, downlink transmission rate, and number of packet losses have a greater impact on network QoE evaluation results, while network QoS parameters such as signal quality and signal-to-noise ratio have a smaller impact on network QoE evaluation results.

[0086] Therefore, according to the current business scenario, one or more target network QoS parameters with a high correlation with the network QoE evaluation result in the current business scenario can be determined from multiple network QoS parameters, and the first timing feature can be determined based on the second timing feature corresponding to the one or more target network QoS parameters with a high correlation with the network QoE evaluation result in the current business scenario. This can make the first network QoE evaluation result obtained by the subsequent first network QoE evaluation model based on the first timing feature more accurate.

[0087] In some embodiments, correlation values ​​corresponding to multiple network QoS parameters can be determined, and the correlation values ​​can be used to characterize the degree of correlation between the network QoS parameters and the network QoE evaluation results in the current business scenario. The network QoS parameters with correlation values ​​greater than the correlation threshold can be determined as the target network QoS parameters.

[0088] Optionally, the correlation value corresponding to the network QoS parameter may include a mutual information value and / or a correlation coefficient value, wherein the mutual information value is calculated using a mutual information method, and the correlation coefficient value is calculated using a correlation coefficient.

[0089] The mutual information method is an information measure, which can be regarded as the amount of information contained in a random variable about another random variable. The mutual information method can be used to measure the dependency between each network QoS parameter and the network QoE evaluation result. For example, the information method can be used to measure the dependency between each network QoS parameter and the result of whether a jam occurs, thereby statistically analyzing the correlation between each network QoS parameter and the result of whether a jam occurs.

[0090] The correlation coefficient can be used to reflect the relationship between two variables. The correlation coefficient can be used to measure the correlation between each network QoS parameter and the network QoE evaluation result. For example, the correlation coefficient can be used to measure the correlation between each network QoS parameter and the result of whether a jam occurs (for example, the correlation can be a linear relationship), thereby statistically analyzing the degree of correlation between each network QoS parameter and the result of whether a jam occurs.

[0091] The network QoE evaluation result is expressed by whether a jam occurs. As an implementation method, a large amount of network QoS information in different business scenarios and the jam results corresponding to each network QoS information (indicating whether a jam occurs) can be collected in advance. Based on the large amount of network QoS information collected in each business scenario and the jam results corresponding to each network QoS information, the mutual information value and correlation coefficient value corresponding to each network QoS parameter can be calculated according to the mutual information method and the correlation coefficient, respectively.

[0092] Exemplarily, for each business scenario, based on a large amount of network QoS information and the jamming results corresponding to each piece of network QoS information, the probability of jamming occurring for different parameter values ​​of each network QoS parameter in the business scenario and the probability of no jamming occurring for different parameter values ​​of each network QoS parameter in the business scenario can be counted. According to the probability of jamming occurring for different parameter values ​​of each network QoS parameter in the business scenario and the probability of no jamming occurring in the business scenario, the mutual information value corresponding to each network QoS parameter is calculated, and the mutual information value can characterize the dependency of the network QoS parameter on the result of whether jamming occurs in the business scenario. Exemplarily, the larger the mutual information value, the stronger the dependency of the network QoS parameter on the result of whether jamming occurs in the business scenario, and the smaller the mutual information value, the smaller the dependency of the network QoS parameter on the result of whether jamming occurs in the business scenario.

[0093] Exemplarily, the Pearson correlation coefficient can be used to calculate the correlation coefficient value. For each business scenario, the covariance and variance corresponding to each network QoS parameter can be counted based on a large amount of network QoS information and the jamming result corresponding to each network QoS information, and the covariance and variance corresponding to the result of whether the jamming occurs in the business scenario can also be counted. The correlation coefficient value corresponding to each network QoS parameter in the business scenario can be calculated based on the covariance and variance corresponding to each network QoS parameter and the covariance and variance corresponding to the result of whether the jamming occurs in the business scenario. Exemplarily, the correlation coefficient value is usually in [-1, 1]. The larger the absolute value of the correlation coefficient value, the stronger the correlation between the network QoS parameter and the result of whether the jamming occurs in the business scenario. The smaller the absolute value of the correlation coefficient value, the smaller the correlation between the network QoS parameter and the result of whether the jamming occurs in the business scenario.

[0094] It should be noted that other correlation coefficient calculation methods may also be used to calculate the correlation coefficient value, such as Spearman's rank correlation coefficient, Kendall rank correlation coefficient, etc., which are not limited here.

[0095] Optionally, after determining the mutual information value and correlation coefficient value corresponding to each network QoS parameter in each business scenario, the scenario identifier of each business scenario, and the mutual information value and correlation coefficient value corresponding to each network QoS parameter in the business scenario can be stored in a correlation file. In the case where the electronic device needs to evaluate the user's network QoE, after determining the current business scenario, the mutual information value and correlation coefficient value corresponding to each network QoS parameter and the current business scenario can be found in the correlation file according to the scenario identifier (such as the scenario name) of the current business scenario.

[0096] Further, the associated threshold may include a mutual information threshold and a correlation threshold. For each network QoS parameter, taking the first network QoS parameter as an example, the first network QoS parameter may be any network QoS parameter, and it may be determined whether the mutual information value corresponding to the first network QoS parameter and the current service scenario is greater than the mutual information threshold, and whether the absolute value of the correlation coefficient value corresponding to the first network QoS parameter and the current service scenario is greater than the correlation threshold. If the mutual information value corresponding to the first network QoS parameter and the current service scenario is greater than the mutual information threshold, and the absolute value of the correlation coefficient value corresponding to the first network QoS parameter and the current service scenario is greater than the correlation threshold, the first network QoS parameter may be determined as the target network QoS parameter.

[0097] Optionally, the association threshold may be a value set according to actual needs, or may be determined according to the association values ​​corresponding to each network QoS parameter in the current business scenario. For example, the mutual information threshold may be the mean of the mutual information values ​​corresponding to each network QoS parameter and the current business scenario; the correlation coefficient value may be the mean of the absolute values ​​of the correlation coefficient values ​​corresponding to each network QoS parameter and the current business scenario, etc., but is not limited thereto. By dynamically adjusting the association threshold according to the association values ​​corresponding to each network QoS parameter in the current business scenario, the selected target network QoS parameters may be more closely associated with the current business scenario, further improving the accuracy of the first network QoE evaluation result obtained subsequently.

[0098] In some embodiments, the number of parameters corresponding to the target network QoS parameters may also be pre-set. After determining the correlation values ​​corresponding to the multiple network QoS parameters, the network QoS parameters with the first number of parameters may be selected in descending order of correlation values ​​as the target network QoS parameters. By fixing the number of parameters corresponding to the target network QoS parameters, the richness of the acquired features can be guaranteed, thereby improving the accuracy of the first network QoE evaluation result obtained subsequently.

[0099] In an embodiment of the present application, according to the correlation values ​​corresponding to the multiple network QoS parameters, the target network QoS parameters with a high degree of correlation with the network QoE evaluation results in the current business scenario can be accurately selected from the multiple network QoS parameters for feature extraction, thereby ensuring the accuracy of the time series feature extraction and further improving the accuracy of the first network QoE evaluation results predicted by the subsequent first network QoE evaluation model.

[0100] Step 504: extract the second time series features corresponding to each target network QoS parameter according to the parameter values ​​corresponding to each target network QoS parameter and a plurality of time moments.

[0101] For each target network QoS parameter, the time series data corresponding to the target network QoS parameter can be formed according to the parameter values ​​corresponding to the target network QoS parameter at multiple moments, and the second time series feature corresponding to the target network QoS parameter can be extracted according to the time series data. The second time series feature corresponding to each target network QoS parameter can be extracted by using the time series-based feature engineering technology, and the parameter values ​​of each target network QoS parameter at multiple moments are introduced, so that the obtained second time series feature can reflect the parameter value changes of the target network QoS parameter in the time period corresponding to the multiple moments.

[0102] In some embodiments, the second time series characteristics corresponding to the target network QoS parameters may include: one or more of: historical shift characteristics, window statistical characteristics, weighted average characteristics and differential characteristics; wherein, the historical shift characteristics refer to new characteristics obtained by shifting the time series data at a certain time interval; the window statistical characteristics refer to characteristics obtained by defining a sliding window to slide on the time series data and calculating the statistical quantities within the sliding window; the weighted average characteristics refer to characteristics obtained by performing weighted average calculations on the time series data; the differential characteristics refer to characteristics obtained by calculating the difference between each two adjacent parameter values ​​in the time series data.

[0103] Taking the first target network QoS parameter as an example, the first target network QoS parameter may be any target network QoS parameter, and one or more of the following feature engineering may be used to extract the second time series feature corresponding to the first target network QoS parameter:

[0104] (1) Historical shift characteristics: A historical shift operation may be performed based on the first target network QoS parameter and parameter values ​​corresponding to a plurality of time periods to obtain a historical shift characteristic corresponding to the first target network QoS parameter.

[0105] The electronic device may perform a translation operation on the time series data corresponding to the first target network QoS parameter at a certain time interval to obtain a historical translation feature corresponding to the first target network QoS parameter, and the certain time interval may be set according to actual needs. For example, the parameter values ​​corresponding to the first target network QoS parameter at M historical moments before the current moment may be used as the historical translation feature corresponding to the current moment, and M may be a positive integer.

[0106] (2) Window statistical characteristics: According to the parameter values ​​corresponding to the first target network QoS parameter and the multiple time periods, based on the target sliding window, the parameter values ​​at the multiple time periods contained in the target sliding window are calculated to obtain the window statistical characteristics corresponding to the first target network QoS parameter.

[0107] The window size corresponding to the target sliding window may be preset, and the electronic device may control the target sliding window to slide on the time series data corresponding to the first target network QoS parameter, and the step size of each sliding may correspond to the window size corresponding to the target sliding window. For example, if the window size corresponding to the target sliding window includes K parameter values, then the step size of each sliding may be K parameter values, and K may be a positive integer.

[0108] After each slide, the statistics corresponding to each parameter value included in the target sliding window may be calculated, and the statistics may include but are not limited to one or more of the minimum value, maximum value, average value, standard deviation, etc. corresponding to each parameter value included in the target sliding window. The statistics calculated after each slide may be used as the window statistical feature corresponding to the first target network QoS parameter.

[0109] (3) Weighted average characteristics: The first target network QoS parameter and the parameter values ​​corresponding to multiple time periods may be weighted averaged to obtain the weighted average characteristics corresponding to the first target network QoS parameter.

[0110] The electronic device may perform a weighted average calculation on the first target network QoS parameter and the parameter values ​​corresponding to the multiple moments. For example, since the closer the data is in time, the higher the correlation is usually, the greater the weight corresponding to the historical moment closer to the current moment may be, and the smaller the weight corresponding to the historical moment farther from the current moment may be. A weighted average calculation may be performed based on the weights corresponding to each moment and the corresponding parameter values ​​to obtain the weighted average feature corresponding to the first target network QoS parameter.

[0111] (4) Perform differential calculation on the first target network QoS parameter and the parameter values ​​corresponding to the multiple time periods to obtain differential features corresponding to the first target network QoS parameter.

[0112] The electronic device can perform differential calculation on the first target network QoS parameter and the parameter values ​​corresponding to multiple time periods, and can calculate the difference in parameter values ​​between each two adjacent time periods to obtain the differential characteristics corresponding to the first target network QoS parameter.

[0113] It should be noted that the above-mentioned feature engineering can be combined arbitrarily. For example, the historical translation feature corresponding to the first target network QoS parameter can be extracted first, and the historical translation feature may include the parameter values ​​corresponding to M historical moments before the current moment. Then, the target sliding window is controlled to slide on the historical translation feature, and the window statistical feature corresponding to the first target network QoS parameter is calculated, etc., but it is not limited to this.

[0114] It should be noted that, in addition to the above-mentioned feature engineering, other feature engineering can also be used to extract the second time series features corresponding to the target network QoS parameters, which is not limited here.

[0115] In an embodiment of the present application, a variety of different feature engineering methods can be used to extract the second timing features corresponding to each target network QoS parameter, which can accurately extract useful information from the parameter values ​​corresponding to each target network QoS parameter at multiple moments, thereby improving the accuracy of timing feature extraction.

[0116] Step 506: Determine the first timing characteristics according to the second timing characteristics corresponding to each target network QoS parameter.

[0117] After obtaining the second timing characteristics corresponding to each target network QoS parameter, the first timing characteristics can be determined according to the second timing characteristics corresponding to each target network QoS parameter. The first timing characteristics can be a set including the second timing characteristics corresponding to each target network QoS parameter.

[0118] In an embodiment of the present application, one or more target network QoS parameters can be determined from multiple network QoS parameters, and the second timing characteristics corresponding to the one or more target network QoS parameters can be extracted to obtain the first timing characteristics. There is no need to extract the timing characteristics of all network QoS parameters. Under the premise of ensuring the accuracy of network QoE evaluation, the amount of calculation can be reduced and the efficiency of network QoE evaluation can be improved.

[0119] Step 404: input the first timing feature into a first network QoE evaluation model, and determine a first network QoE evaluation result according to the first timing feature through the first network QoE evaluation model.

[0120] The first timing feature may be input into the first network QoE evaluation model, and the first network QoE evaluation model may predict and obtain the first network QoE evaluation result according to the first timing feature.

[0121] In some embodiments, the first network QoE evaluation model may include an XGBoost model, which may include multiple decision trees, each decision tree corresponding to a decision weight. The first time series feature may be input into the multiple decision trees respectively, and the first time series feature is predicted by the multiple decision trees respectively to obtain a third network QoE evaluation result output by each decision tree, for example, the first time series feature is predicted by the multiple decision trees respectively to obtain a third freeze probability output by each decision tree.

[0122] According to the decision weight corresponding to each decision tree, the third network QoE evaluation results output by each decision tree can be weighted and fused to obtain the first network QoE evaluation result. For example, according to the decision weight corresponding to each decision tree, the third jamming probability output by each decision tree can be weighted averaged to obtain the first jamming probability output by the XGBoost model. Exemplarily, the first jamming probability can represent the probability of jamming at the current moment. The XGBoost model can accurately evaluate the user's network QoE according to the first time series feature, thereby improving the accuracy of the user's network QoE evaluation.

[0123] In an embodiment of the present application, feature extraction can be performed on the network QoS information corresponding to multiple time moments contained in the network QoS data to obtain a first time series feature. The first network QoE evaluation model can determine the first network QoE evaluation result based on the first time series feature. By introducing the network QoS information at multiple time moments into the extracted first time series feature, the first time series feature can be integrated with the network change information of the time periods corresponding to the multiple time moments, so that the first network QoE evaluation model can more accurately evaluate the user's network QoE.

[0124] like Figure 6 As shown, in one embodiment, the step of processing the network QoS data by the second network QoE evaluation model to obtain the second network QoE evaluation result may include the following steps:

[0125] Step 602: extract local timing features and global timing features corresponding to the network QoS data through the second network QoE evaluation model, fuse the local timing features and the global timing features to obtain a third timing feature, and determine the second network QoE evaluation result according to the third timing feature.

[0126] The network QoS data includes network QoS information corresponding to multiple moments. For example, the network QoS data includes network QoS information corresponding to the current moment and N historical moments before the current moment. The network QoS information corresponding to multiple moments can be arranged in order from earliest to latest time to obtain a network QoS time series.

[0127] The network QoS time series can be input into the second network QoE evaluation model, and the second network QoE evaluation model can extract local timing features and global timing features based on the network QoS time series. Among them, the local timing features can refer to the features that can characterize the network changes corresponding to one or more time slices in the network QoS time series, and the global timing features can refer to the features that can characterize the long-term dependencies in the network QoS time series (i.e., the continuous changes of the network). The second network QoE evaluation model can fuse the local timing features and the global timing features to obtain a third timing feature, and predict the second network QoE evaluation result based on the third timing feature.

[0128] In some embodiments, the second network QoE evaluation model may include an FCN module and an LSTM module, and the local timing features corresponding to the network QoS data may be extracted by the FCN module, and the global timing features corresponding to the network QoS data may be extracted by the LSTM module.

[0129] The FCN module can perform convolution processing on the network QoS time series to extract local time series features, and the LSTM module can perform sequence modeling on the multivariate network QoS time series to extract global time series features.

[0130] For example, Figure 7 FIG. 1 is a flow chart of a second network QoE evaluation model processing network QoS data in an embodiment. Figure 7 As shown, the network QoS information corresponding to multiple moments included in the network QoS data can be arranged into a network QoS time series and input into the FCN module and the LSTM module respectively. The FCN module can extract local time series features, and the LSTM module can extract global time series features. The local time series features and the global time series features can be fused. For example, the local time series features and the global time series features can be connected (such as performing a concat operation) to obtain a third time series feature. The third time series feature can be input into the activation function layer, and predicted by the activation function layer to obtain a second network QoE evaluation result. Optionally, the activation function layer can use an activation function such as a softmax function to convert the third time series feature into a second network QoE evaluation result, such as converting the third time series feature into a second jamming probability.

[0131] In some embodiments, the FCN module may include multiple convolution blocks, which may be connected in series, and the local time series features corresponding to the network QoS data may be extracted through the multiple convolution blocks. The FCN module may treat the network QoS time series as a multivariate time series with multiple time slices, and perform feature extraction on the network QoS time series through multiple convolution blocks, thereby extracting local time series features, which may be features that can reflect the network conditions of each time slice.

[0132] The number of convolution blocks included in the FCN module is not limited here. For example, the FCN module may include 3 convolution blocks, 4 convolution blocks, etc., but is not limited to this. The number of convolution blocks should not be too many, so as to ensure the lightweight of the FCN module and avoid the FCN module being too complicated. The number of convolution blocks should not be too small. For example, there is only one convolution block, which may cause the extracted local timing features to be inaccurate, affecting the accuracy of the user's network QoE evaluation.

[0133] The network QoS time series can be used as the input data of the first convolution block, and input into the first convolution block for processing to obtain the output data of the first convolution block. The output data of the first convolution block can be used as the input data of the second convolution block. Each convolution block can process the input data and obtain the corresponding output data. The output data of the previous convolution block can be used as the input data of the next convolution module until the last convolution block obtains the output data. The local time series features can be generated according to the output data of the last convolution block.

[0134] The first convolution block may include a convolution layer, an activation function, and an attention module. The first convolution block may be one or more of the plurality of convolution blocks.

[0135] The convolution layer can be used to perform convolution processing on the time domain of the input data to obtain convolution features. The convolution layer of the first convolution block can perform convolution processing on the input data of the first convolution block in the time domain to obtain convolution features. Exemplarily, the convolution layer can be a one-dimensional convolution layer.

[0136] The activation function can be used to transform the convolution features output by the convolution layer to obtain transformed data.

[0137] In some embodiments, before the activation function converts the convolutional features output by the convolutional layer, the convolutional features output by the convolutional layer may be normalized, such as batch normalization (BN), and then the normalized convolutional features are converted by the activation function to obtain conversion data. By normalizing the convolutional features output by the convolutional layer, the generalization ability of the FCN module can be improved.

[0138] Exemplarily, the activation function may include, but is not limited to, activation functions such as ReLU (Rectified Linear Unit, linear rectification function), thereby introducing nonlinear relationships and improving the expression ability of the FCN module.

[0139] The attention module can be used to perform feature extraction on the conversion data output by the activation function based on the attention mechanism to obtain the output data of the first convolution block.

[0140] The attention module can perform further feature extraction on the transformed data obtained by the activation function based on the attention mechanism. Exemplarily, the attention module can include an SE block, which can explicitly model the dependencies between different feature channels, thereby improving the feature extraction capability of the convolution block.

[0141] Optionally, the architectures of the multiple convolution blocks included in the FCN module may be the same or different. For example, some convolution blocks include convolution layers, activation functions, and attention modules, while other convolution blocks may only include convolution layers and activation functions. Figure 8 FIG. 1 is an architecture diagram of an FCN module in an embodiment. Figure 8 As shown, the FCN module may include three convolution blocks, wherein the first convolution block and the second convolution block may include a convolution layer, an activation function and an attention module, and the third convolution block may include a convolution layer and an activation function. The FCN module may also include a pooling layer, and the conversion data obtained by the activation function of the third convolution block may be input into the pooling layer, and the pooling layer performs global average pooling processing on the conversion data to obtain local time series features.

[0142] In the above embodiment, the FCN module may treat the network QoS time series as a multivariate time series with multiple time slices, and perform feature extraction on the network QoS time series through multiple convolution blocks to obtain local time series features, thereby improving the accuracy of local time series feature extraction, thereby improving the accuracy of subsequent predictions of the second network QoE evaluation results.

[0143] In some embodiments, the LSTM module may be a neural network implemented by combining LSTM with an attention mechanism. The LSTM module may be used to extract global time series features corresponding to network QoS data based on the attention mechanism.

[0144] The network QoS time series can be input into the LSTM module. Based on the attention mechanism, the LSTM module can identify which features in the network QoS time series are important and which features are not important for the network QoE evaluation results, and obtain features that are more meaningful to the network QoE evaluation results, so as to better capture the long-term dependencies in the network QoS time series and effectively obtain the continuous change information of the network.

[0145] Exemplarily, the LSTM module can give greater weights to features in the network QoS time series that are more important to the network QoE evaluation results based on the attention mechanism, and give smaller weights to features that are unimportant or irrelevant to the network QoE evaluation results, thereby obtaining global time series features.

[0146] In some embodiments, the LSTM module may include an LSTM layer based on an attention mechanism and a Dropout layer. The LSTM layer based on the attention mechanism may be used to extract features corresponding to network QoS data based on the attention mechanism, and the Dropout layer may be used to regularize the features output by the LSTM layer to obtain global time series features. The Dropout layer may prevent overfitting of the LSTM module.

[0147] For example, Fig. 9 FIG. 1 is an architecture diagram of an LSTM module in one embodiment. Fig. 9 As shown, the network QoS time series can be input into the LSTM layer based on the attention mechanism. The LSTM layer based on the attention mechanism can be used to extract the features corresponding to the network QoS data based on the attention mechanism, and then the features are regularized through the Dropout layer to obtain the global time series features.

[0148] In the above implementation, the LSTM module can extract the global time series features of the network QoS data based on the attention mechanism. Based on the attention mechanism, the long-term dependencies in the network QoS time series can be better captured, thereby effectively obtaining the continuous change information of the network, and improving the accuracy of the extracted global time series features, thereby improving the accuracy of the subsequent prediction of the second network QoE evaluation result.

[0149] It should be noted that there is no strict timing requirement between extracting local timing features through the FCN module and extracting global timing features through the LSTM. The two can be performed simultaneously in parallel. It is also possible to first extract local timing features through the FCN module and then extract global timing features through the LSTM, or first extract global timing features through the LSTM and then extract local timing features through the FCN module. The embodiments of the present application are not limited to this.

[0150] In the present embodiment, the second network QoE evaluation model can extract local timing features and global timing features corresponding to the network QoS data, and fuse the local timing features and the global timing features to obtain a third timing feature, and determine the second network QoE evaluation result based on the third timing feature. The second network QoE evaluation model can extract features from the time dimension, and fuse the local timing features with the global timing features to obtain a high-order feature expression, thereby improving the accuracy of the predicted second network QoE evaluation result.

[0151] In addition, the second network QoE evaluation model combines the FCN module with the LSTM module, extracts local timing features through the FCN module, and extracts global timing features through the LSTM based on the attention mechanism, which improves the accuracy of feature extraction and further improves the accuracy of the second network QoE evaluation model in performing user network QoE evaluation.

[0152] In some embodiments, the above-mentioned network experience quality evaluation method may further include the step of training the first network QoE evaluation model and the second network QoE evaluation model. Fig.10 As shown, the training process of the first network QoE evaluation model and the second network QoE evaluation model may include the following steps:

[0153] Step 1010: Based on a first training data set, a first network QoE evaluation model is trained.

[0154] Step 1020: Train the second network QoE evaluation model to be trained based on the first training data set to obtain a trained second network QoE evaluation model.

[0155] The first network QoE evaluation model can be obtained by training based on the first training data set, and the second network QoE evaluation model can also be obtained by training based on the first training data set. The first training data set may include multiple groups of first sample network QoS data, and annotated network QoE evaluation results corresponding to each group of first sample network QoS data. For example, the first training data set may include multiple groups of first sample network QoS data, and annotated jamming results (indicating whether jamming occurs) corresponding to each group of first sample network QoS data. The first training data set can be obtained by collecting actual data from users in the actual use of electronic devices, or it can be virtual data (such as data generated by an artificial intelligence model or artificially fabricated, etc.).

[0156] In some embodiments, the first network QoE evaluation model may be an artificial intelligence model of the GBDT type, for example, the first network QoE evaluation model may be an XGBoost model, and the first network QoE evaluation model may include multiple decision trees. Optionally, during the training process of the first network QoE evaluation model, feature extraction may be performed on multiple groups of first sample network QoS data to obtain sample time series features corresponding to the multiple groups of first sample network QoS data, and based on the Focal Loss loss function, multiple decision trees may be constructed according to the sample time series features corresponding to the multiple groups of first sample network QoS data.

[0157] As an implementation method, feature extraction can be performed on each group of first sample network QoS data to obtain sample timing features corresponding to each group of first sample network QoS data. The sample timing features may include but are not limited to one or more of historical shift features, window statistical features, weighted average features, and differential features.

[0158] It should be noted that the method of extracting the sample timing characteristics corresponding to each group of first sample network QoS data can be similar to the method of extracting the second timing characteristics corresponding to the target network QoS parameters introduced in the above embodiment, and will not be repeated here.

[0159] Since in the actual use of electronic devices, the situations of good network quality are far more than those of poor network quality, it may result in that in the first training data set collected, the number of samples of the first sample network QoS data whose network QoE evaluation results indicate that the user's network experience is good will be far greater than the number of samples of the first sample network QoS data whose network QoE evaluation results indicate that the user's network experience is poor. Therefore, the Focal Loss loss function can be used as the loss function of the first network QoE evaluation model training process. The Focal Loss loss function adds a coefficient factor to balance the uneven number of samples of different labeled network QoE evaluation results, thereby solving the problem of imbalance in the first sample network QoS data of different labeled network QoE evaluation results.

[0160] Exemplarily, taking the first network QoE evaluation model as an XGBoost model, the initial predicted network QoE evaluation results corresponding to all first sample network QoS data can be pre-configured, and the objective function can be constructed based on the gap between the initial predicted network QoE evaluation results corresponding to each first sample network QoS data and the corresponding labeled network QoE evaluation results, the first-order derivative of the Focal Loss loss function, and the second-order derivative of the Focal Loss loss function. The objective function can be optimized and solved according to the sample time series features corresponding to multiple groups of first sample network QoS data, thereby constructing the first decision tree. The sample time series features corresponding to each group of first sample network QoS data can be predicted based on the constructed first decision tree to update the predicted network QoE evaluation results corresponding to each group of first sample network QoS data, and update the objective function. The next decision tree is constructed based on the updated objective function and the sample time series features corresponding to multiple groups of first sample network QoS data. The construction of each decision tree will take into account the prediction results of the previous decision tree, thereby continuously optimizing the prediction ability of the decision tree.

[0161] It should be noted that the model training process of the first network QoE evaluation model may also adopt other methods and is not limited to the above training method.

[0162] In some embodiments, during the training process of the second network QoE evaluation model, each group of first sample network QoS data can be input into the second network QoE evaluation model to be trained, and the input first sample network QoS data can be feature extracted by the second network QoE evaluation model to be trained, and a predicted network QoE evaluation result can be obtained based on the extracted features. The first loss can be calculated based on the predicted network QoE evaluation result corresponding to the input first sample network QoS data and the corresponding annotated network QoE evaluation result, and the model parameters of the second network QoE evaluation model to be trained are adjusted based on the first loss until the model convergence condition is met (for example, the number of times the model parameters are adjusted reaches a number threshold or the first loss is less than the first loss threshold, etc.), thereby obtaining a trained second network QoE evaluation model.

[0163] In some embodiments, the first network QoE evaluation model obtained through the training and the second network QoE evaluation model completed through the training may be tested respectively through a test data set to determine the model performance of the first network QoE evaluation model and the second network QoE evaluation model, such as model accuracy, model stability, etc. If the model performance of the first network QoE evaluation model or the second network QoE evaluation model does not meet the performance requirements, it is necessary to continue to train the first network QoE evaluation model or the second network QoE evaluation model that does not meet the performance requirements.

[0164] Step 1030: Based on the second training data set, the trained first network QoE evaluation model and the trained second network QoE evaluation model are jointly trained to obtain a first weight corresponding to the first network QoE evaluation model and a second weight corresponding to the second network QoE evaluation model.

[0165] In some embodiments, the first weight corresponding to the first network QoE evaluation model and the second weight corresponding to the second network QoE evaluation model can be obtained by jointly training the trained first network QoE evaluation model and the trained second network QoE evaluation model according to the second training data set.

[0166] The second training data set may include multiple groups of second sample network QoS data and annotated network QoE evaluation results corresponding to each group of second sample network QoS data. Each group of second sample network QoS data may be input into the first network QoE evaluation model and the second network QoE evaluation model respectively, and the first network QoE evaluation model may output the first predicted network QoE evaluation result corresponding to the input second sample network QoS data, and the second network QoE evaluation model may output the second predicted network QoE evaluation result corresponding to the input second sample network QoS data.

[0167] According to the current weight corresponding to the first network QoE evaluation model and the current weight corresponding to the second network QoE evaluation model, a weighted average calculation can be performed on the first predicted network QoE evaluation result and the second predicted network QoE evaluation result to obtain a third predicted network QoE evaluation result, and the second loss is calculated according to the third predicted network QoE evaluation result and the marked network QoE evaluation result corresponding to the input second sample network QoS data, and the current weight corresponding to the first network QoE evaluation model and the current weight corresponding to the second network QoE evaluation model are adjusted based on the second loss until the second loss is less than the second loss threshold, thereby obtaining the first weight corresponding to the first network QoE evaluation model and the second weight corresponding to the second network QoE evaluation model.

[0168] In an embodiment of the present application, the first weight corresponding to the first network QoE evaluation model and the second weight corresponding to the second network QoE evaluation model can be obtained by jointly training the first network QoE evaluation model and the second network QoE evaluation model, thereby improving the accuracy of the first weight and the second weight, thereby further improving the accuracy of the user's network QoE evaluation.

[0169] like Fig.11As shown, in one embodiment, a network experience quality evaluation device 1100 is provided, which can be applied to the above-mentioned electronic device. The network experience quality evaluation device 1100 may include a data acquisition module 1110, a first evaluation module 1120, a second evaluation module 1130 and a fusion module 1140.

[0170] The data acquisition module 1110 is used to acquire network QoS data, where the network QoS data includes network QoS information corresponding to multiple time periods.

[0171] The first evaluation module 1120 is used to process the network QoS data through a first network QoE evaluation model to obtain a first network QoE evaluation result.

[0172] The second evaluation module 1130 is used to process the network QoS data through a second network QoE evaluation model to obtain a second network QoE evaluation result; the model architecture of the first network QoE evaluation model is different from the model architecture of the second network QoE evaluation model.

[0173] The fusion module 1140 is used to determine the target network QoE evaluation result according to the first network QoE evaluation result and the second network QoE evaluation result.

[0174] In some embodiments, the network QoS data includes network QoS information corresponding to the current moment and N historical moments before the current moment; N is a positive integer;

[0175] Wherein, N is a preset fixed value, or N is determined according to the current business scenario.

[0176] In some embodiments, the first evaluation module 1120 includes a feature extraction unit and an evaluation unit.

[0177] The feature extraction unit is used to extract features from network QoS information corresponding to multiple time periods to obtain a first time series feature.

[0178] The evaluation unit is used to input the first timing feature into the first network QoE evaluation model, and determine the first network QoE evaluation result according to the first timing feature through the first network QoE evaluation model.

[0179] In some embodiments, the network QoS information includes parameter values ​​corresponding to multiple network QoS parameters respectively; the feature extraction unit is also used to determine one or more target network QoS parameters from the multiple network QoS parameters; extract the second timing characteristics corresponding to each target network QoS parameter according to the parameter values ​​corresponding to each target network QoS parameter and multiple time moments; and determine the first timing characteristics according to the second timing characteristics corresponding to each target network QoS parameter.

[0180] In some embodiments, the feature extraction unit is also used to determine the correlation values ​​corresponding to multiple network QoS parameters, the correlation values ​​are used to characterize the degree of correlation between the network QoS parameters and the network QoE evaluation results in the current business scenario; and to determine the network QoS parameters whose correlation values ​​are greater than the correlation threshold as the target network QoS parameters.

[0181] In some embodiments, the feature extraction unit is further used to perform a historical translation operation based on the parameter values ​​corresponding to the first target network QoS parameter at multiple moments, so as to obtain a historical translation feature corresponding to the first target network QoS parameter; the first target network QoS parameter is any target network QoS parameter; and / or, it is also used to calculate the parameter values ​​at multiple moments contained in the target sliding window based on the target sliding window according to the parameter values ​​corresponding to the first target network QoS parameter at multiple moments, so as to obtain a window statistical feature corresponding to the first target network QoS parameter; and / or, it is also used to perform a weighted average calculation on the parameter values ​​corresponding to the first target network QoS parameter at multiple moments, so as to obtain a weighted average feature corresponding to the first target network QoS parameter; and / or, it is also used to perform a differential calculation on the parameter values ​​corresponding to the first target network QoS parameter at multiple moments, so as to obtain a differential feature corresponding to the first target network QoS parameter.

[0182] In some embodiments, the first network QoE evaluation model includes an XGBoost model.

[0183] In some embodiments, the second evaluation module 1130 is also used to extract local timing characteristics and global timing characteristics corresponding to the network QoS data through the second network QoE evaluation model, fuse the local timing characteristics and the global timing characteristics to obtain a third timing characteristic, and determine the second network QoE evaluation result based on the third timing characteristic.

[0184] In some embodiments, the second network QoE evaluation model includes a fully convolutional neural network FCN module and a long short-term memory network LSTM module; the second evaluation module 1130 is also used to extract local timing features corresponding to network QoS data through the FCN module; and to extract global timing features corresponding to network QoS data through the LSTM module.

[0185] In some embodiments, the FCN module includes multiple convolution blocks, which are connected in series; the second evaluation module 1130 is also used to extract local timing features corresponding to the network QoS data through the multiple convolution blocks.

[0186] Among them, the first convolution block includes a convolution layer, an activation function and an attention module; the convolution layer is used to perform convolution processing in the time domain of the input data to obtain convolution features; the activation function is used to transform the convolution features output by the convolution layer to obtain transformed data; the attention module is used to extract features of the transformed data output by the activation function based on the attention mechanism to obtain the output data of the first convolution block; the first convolution block is one or more of the multiple convolution blocks.

[0187] In some embodiments, the second evaluation module 1130 is also used to extract global time series features corresponding to the network QoS data based on the attention mechanism through the LSTM module.

[0188] In some embodiments, the first network QoE evaluation result includes a first jamming probability, and the second network QoE evaluation result includes a second jamming probability.

[0189] The fusion module 1140 is also used to perform weighted averaging processing according to the first jamming probability, the first weight corresponding to the first network QoE evaluation model, the second jamming probability, and the second weight corresponding to the second network QoE evaluation model to obtain a target jamming probability; if the target jamming probability is greater than the probability threshold, the target network QoE evaluation result is determined to be that jamming occurs; if the target jamming probability is not greater than the probability threshold, the target network QoE evaluation result is determined to be that no jamming occurs.

[0190] In some embodiments, the first network QoE evaluation model is trained based on a first training data set, and the second network QoE evaluation model is trained based on the first training data set, and the first training data set includes multiple groups of first sample network QoS data, and annotated network QoE evaluation results corresponding to each group of first sample network QoS data;

[0191] The first weight corresponding to the first network QoE evaluation model and the second weight corresponding to the second network QoE evaluation model are obtained by jointly training the trained first network QoE evaluation model and the trained second network QoE evaluation model according to the second training data set.

[0192] In some embodiments, the first network QoE evaluation model includes multiple decision trees; the network experience quality evaluation device 1100 also includes a training module.

[0193] The training module is used to extract features from multiple groups of first sample network QoS data to obtain sample time series features corresponding to the multiple groups of first sample network QoS data; based on the Focal Loss loss function, multiple decision trees are constructed according to the sample time series features corresponding to the multiple groups of first sample network QoS data.

[0194] In an embodiment of the present application, since the QoS data includes network QoS information corresponding to multiple time periods, the first network QoE evaluation model and the second network QoE evaluation model can pay attention to network changes over a period of time based on the network QoS information corresponding to multiple time periods, and can perform network QoE evaluation more accurately. The network QoE evaluation is performed using the first network QoE evaluation model and the second network QoE evaluation model with different model architectures, respectively, and the target network QoE evaluation result is determined based on the network QoE evaluation results of the two, thereby further improving the accuracy of evaluating the user's network QoE.

[0195] Fig.12 FIG. 1 is a structural block diagram of an electronic device in an embodiment. Fig.12 As shown, the electronic device 1200 may include one or more of the following components: a processor 1210, and a memory 1220 coupled to the processor 1210, wherein the memory 1220 may store one or more computer programs, and the one or more computer programs may be configured to implement the methods described in the above embodiments when executed by one or more processors 1210.

[0196] The processor 1210 may include one or more processing cores. The processor 1210 uses various interfaces and lines to connect various parts of the entire electronic device 1200, and executes various functions and processes data of the electronic device 1200 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1220, and calling data stored in the memory 1220. Optionally, the processor 1210 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1210 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1210, but may be implemented separately through a communication chip.

[0197] The memory 1220 may include a random access memory (RAM) or a read-only memory (ROM). The memory 1220 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 1220 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may also store data created by the electronic device 1200 during use, etc.

[0198] It can be understood that the electronic device 1200 may include more or fewer structural elements than those in the above structural block diagram, for example, a radio frequency module, a power module, physical buttons, a WiFi (Wireless Fidelity) module, a speaker, a Bluetooth module, a sensor, etc., and no limitation is made here.

[0199] An embodiment of the present application discloses a computer-readable storage medium storing a computer program, wherein the computer program implements the methods described in the above embodiments when executed by a processor.

[0200] An embodiment of the present application discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program can be executed by a processor to implement the methods described in the above embodiments.

[0201] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a ROM, etc.

[0202] As used herein, any reference to memory, storage, database, or other medium may include nonvolatile and / or volatile memory. Suitable nonvolatile memory may include ROM, programmable ROM (Programmable ROM, PROM), erasable PROM (Erasable PROM, EPROM), electrically erasable PROM (Electrically Erasable PROM, EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which is used as an external cache memory. By way of illustration and not limitation, RAM may be in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus direct RAM (RDRAM) and direct memory bus dynamic RAM (DRDRAM).

[0203] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. Those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required for the present application.

[0204] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the above-mentioned processes does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0205] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0206] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0207] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0208] The above is a detailed introduction to a network experience quality evaluation method, device, electronic device and program product disclosed in the embodiments of the present application. The principles and implementation methods of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A method for evaluating network experience quality, characterized in that: The method comprises: Acquire network quality of service (QoS) data, wherein the network QoS data includes network QoS information corresponding to multiple time periods; Processing the network QoS data by using a first network quality of experience (QoE) evaluation model to obtain a first network QoE evaluation result; Processing the network QoS data through a second network quality of experience (QoE) evaluation model to obtain a second network QoE evaluation result; the model architecture of the first network QoE evaluation model is different from the model architecture of the second network QoE evaluation model; A target network QoE evaluation result is determined according to the first network QoE evaluation result and the second network QoE evaluation result.

2. The method according to claim 1, characterized in that: The network QoS data includes network QoS information corresponding to the current moment and N historical moments before the current moment; N is a positive integer; Wherein, the N is a preset fixed value, or the N is determined according to the current business scenario.

3. The method according to claim 1, characterized in that The processing of the network QoS data by using a first network quality of experience (QoE) evaluation model to obtain a first network QoE evaluation result includes: Extracting features of the network QoS information corresponding to the multiple moments to obtain a first time series feature; The first timing feature is input into a first network QoE evaluation model, and a first network QoE evaluation result is determined according to the first timing feature by the first network QoE evaluation model.

4. The method according to claim 3, characterized in that The network QoS information includes parameter values ​​corresponding to a plurality of network QoS parameters; and extracting features from the network QoS information corresponding to the plurality of moments to obtain a first time series feature includes: Determining one or more target network QoS parameters from the plurality of network QoS parameters; Extracting second time series features corresponding to each of the target network QoS parameters according to parameter values ​​corresponding to each of the target network QoS parameters and the multiple time points; The first timing characteristics are determined according to the second timing characteristics corresponding to each of the target network QoS parameters.

5. The method according to claim 4, characterized in that The step of determining one or more target network QoS parameters from the multiple network QoS parameters comprises: Determine the correlation values ​​corresponding to the multiple network QoS parameters respectively, where the correlation values ​​are used to characterize the correlation degree between the network QoS parameters and the network QoE evaluation result in the current service scenario; The network QoS parameter whose correlation value is greater than the correlation threshold is determined as the target network QoS parameter.

6. The method according to claim 4, characterized in that The extracting, according to the parameter values ​​of each of the target network QoS parameters corresponding to the multiple time periods, the second timing characteristics corresponding to each of the target network QoS parameters, comprises: Perform a historical shift operation according to the first target network QoS parameter and the parameter values ​​corresponding to the multiple time points to obtain a historical shift feature corresponding to the first target network QoS parameter; the first target network QoS parameter is any of the target network QoS parameters; and / or, According to the parameter values ​​corresponding to the first target network QoS parameter and the multiple moments respectively, based on the target sliding window, the parameter values ​​at multiple moments included in the target sliding window are calculated to obtain the window statistical characteristics corresponding to the first target network QoS parameter; and / or, Performing weighted average calculation on the first target network QoS parameter and the parameter values ​​corresponding to the multiple time points to obtain a weighted average feature corresponding to the first target network QoS parameter; and / or, A differential calculation is performed on the first target network QoS parameter and the parameter values ​​corresponding to the multiple time moments to obtain a differential feature corresponding to the first target network QoS parameter.

7. The method according to any one of claims 1 to 6, characterized in that: The first network QoE evaluation model includes an XGBoost model.

8. The method according to claim 1, characterized in that The processing of the network QoS data by using the second network quality of experience (QoE) evaluation model to obtain a second network QoE evaluation result includes: The local timing features and global timing features corresponding to the network QoS data are extracted through the second network QoE evaluation model, the local timing features and the global timing features are fused to obtain a third timing feature, and the second network QoE evaluation result is determined according to the third timing feature.

9. The method according to claim 8, characterized in that The second network QoE evaluation model includes a fully convolutional neural network FCN module and a long short-term memory network LSTM module; the extracting the local time series features and the global time series features corresponding to the network QoS data through the second network QoE evaluation model includes: Extracting local time series features corresponding to the network QoS data through the FCN module; The global time series features corresponding to the network QoS data are extracted through the LSTM module.

10. The method according to claim 9, characterized in that The FCN module includes a plurality of convolution blocks, and the plurality of convolution blocks are connected in series; the extracting the local time series features corresponding to the network QoS data through the FCN module includes: Extracting local time series features corresponding to the network QoS data through the multiple convolution blocks; Among them, the first convolution block includes a convolution layer, an activation function and an attention module; the convolution layer is used to perform convolution processing in the time domain of the input data to obtain convolution features; the activation function is used to transform the convolution features output by the convolution layer to obtain transformed data; the attention module is used to perform feature extraction on the transformed data output by the activation function based on the attention mechanism to obtain the output data of the first convolution block; the first convolution block is one or more of the multiple convolution blocks.

11. The method according to claim 9, characterized in that The extracting the global time series features corresponding to the network QoS data through the LSTM module includes: The LSTM module is used to extract the global time series features corresponding to the network QoS data based on the attention mechanism.

12. The method according to any one of claims 1 to 6 and 8 to 11, characterized in that: The first network QoE evaluation result includes a first jamming probability, and the second network QoE evaluation result includes a second jamming probability; and determining a target network QoE evaluation result according to the first network QoE evaluation result and the second network QoE evaluation result includes: Performing weighted averaging processing according to the first jamming probability, the first weight corresponding to the first network QoE evaluation model, the second jamming probability, and the second weight corresponding to the second network QoE evaluation model to obtain a target jamming probability; If the target jamming probability is greater than the probability threshold, determining that the target network QoE evaluation result is that jamming occurs; If the target jamming probability is not greater than the probability threshold, the target network QoE evaluation result is determined to be no jamming.

13. The method according to claim 12, characterized in that The first network QoE evaluation model is trained based on a first training data set, and the second network QoE evaluation model is trained based on the first training data set, wherein the first training data set includes multiple groups of first sample network QoS data and annotated network QoE evaluation results corresponding to each group of the first sample network QoS data; The first weight corresponding to the first network QoE evaluation model and the second weight corresponding to the second network QoE evaluation model are obtained by jointly training the trained first network QoE evaluation model and the trained second network QoE evaluation model according to the second training data set.

14. The method according to claim 13, characterized in that The first network QoE evaluation model includes a plurality of decision trees; and the training process of the first network QoE evaluation model includes: Extracting features from the multiple groups of first sample network QoS data to obtain sample time series features corresponding to the multiple groups of first sample network QoS data; Based on the FocalLoss loss function, multiple decision trees are constructed according to the sample time series features corresponding to the multiple groups of first sample network QoS data.

15. A network experience quality evaluation device, characterized in that: The device comprises: A data acquisition module, used to acquire network QoS data, wherein the network QoS data includes network QoS information corresponding to multiple time periods; A first evaluation module, configured to process the network QoS data through a first network QoE evaluation model to obtain a first network QoE evaluation result; A second evaluation module, configured to process the network QoS data through a second network QoE evaluation model to obtain a second network QoE evaluation result; the model architecture of the first network QoE evaluation model is different from the model architecture of the second network QoE evaluation model; A fusion module is used to determine a target network QoE evaluation result according to the first network QoE evaluation result and the second network QoE evaluation result.

16. An electronic device, characterized in that: The electronic device comprises a memory, a processor and a transceiver unit. The memory stores a computer program. When the computer program is executed by the processor, the electronic device implements the method according to any one of claims 1 to 14.

17. A computer program product, characterized in that The method comprises a computer program, and when the computer program is executed by a processor in an electronic device, the electronic device implements the method according to any one of claims 1 to 14.

Images