A method, apparatus and device for quality of service assessment

Abstract

The application provides a service quality evaluation method, device and equipment, and relates to the technical field of communication. The method comprises the following steps: analyzing first service information of a target service; in the case that the analysis result shows that the target service needs to be evaluated based on different scenes, dividing the first service information according to scenes; determining quality evaluation models corresponding to different scenes according to the first service information of the target service in different scenes; in the case that second service information of the target service is acquired, using the determined quality evaluation models corresponding to different scenes to evaluate the service quality of the second service information. The scheme of the application solves the problem of low service quality evaluation accuracy.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a service quality assessment method, apparatus, and equipment. Background Technology

[0002] With the continuous development of mobile network technology, audio and video services, as indispensable resources in people's daily lives, are increasing in number every day. As users' demands for audio and video services become higher, operators need to promptly monitor and continuously evaluate service quality to improve user satisfaction, such as stuttering and long buffering delays in video services, and word swallowing and intermittent voice services.

[0003] Currently, assessment schemes based on network performance statistics to obtain service quality evaluation values ​​are proposed. These schemes primarily rely on human experience to formulate assessment strategies. Specifically, they involve: collecting necessary service data through techniques such as terminal testing and DPI detection; statistically analyzing the data to obtain relevant network indicators capable of supporting services; establishing a multi-dimensional functional mapping relationship between network indicators and service quality based on human experience; and generating service quality indicators through weighted calculations. Finally, using rule-based strategies, these service quality indicators are categorized into levels to form service quality grades. However, the mapping relationship between network performance parameters and service quality depends on pre-set mathematical functions or rule-based classification strategies based on human experience. This approach cannot deeply explore the relationship between all network performance parameters and service quality, resulting in the loss of valuable information carried by the original data. Summary of the Invention

[0004] The purpose of this invention is to provide a business quality assessment method, apparatus, and equipment to improve the accuracy of business quality assessment.

[0005] To achieve the above objectives, embodiments of the present invention provide a business quality assessment method, comprising: Analyze the primary business information of the target business; If the analysis results indicate that the target service needs to be evaluated based on different scenarios, the first service information will be divided according to the scenarios. Based on the first business information of the target business in different scenarios, determine the quality assessment model corresponding to different scenarios; Upon obtaining the second business information of the target business, the second business information is evaluated for business quality using the determined quality assessment model corresponding to different scenarios.

[0006] Optionally, before analyzing the first business information of the target business, the method further includes: Obtain the initial network-side service information and initial terminal-side service information of the target service; The initial network-side service information and the initial terminal-side service information are correlated to obtain the first service information.

[0007] Optionally, the step of associating the initial network-side service information and the initial terminal-side service information to obtain the first service information includes: According to the preset association strategy, extract network-side service information and terminal-side service information with the same association identifier from the initial network-side service information and the initial terminal-side service information, and then associate them.

[0008] Optionally, the first service information has a corresponding service quality indicator label; The analysis of the first business information of the target business includes at least one of the following: Based on the same service quality indicator label, the distribution range of the associated network-side service information and terminal-side service information in the first service information is analyzed under different service characteristics to obtain the first result; Based on different service quality indicator labels, the distribution range of the associated network-side service information and terminal-side service information in the first service information is analyzed under the same service characteristics to obtain the second result.

[0009] Optionally, the target service needs to be evaluated based on different scenarios and determined by at least one of the following: The first result indicates that, under the same business quality indicator label, the distribution range of different business characteristics of the target business is greater than the first threshold; The second result indicates that the target business has overlapping distribution ranges of the same business features under different business quality indicator labels, and the overlapping part is greater than the second threshold.

[0010] Optionally, dividing the first business information according to scenarios includes: Determine the scene segmentation model; The scenario segmentation model is used to segment the first business information.

[0011] Optionally, the determination of the scene partitioning model includes: The first business information is input into multiple first candidate models to obtain the corresponding partitioning results; Based on the partitioning results of the plurality of first candidate models, the scene partitioning model is determined; wherein, the scene partitioning model satisfies at least one of the following: The scene classification accuracy is greater than the third threshold; The number of classifications is less than the number of classifications of the remaining first candidate models among the plurality of first candidate models excluding the scene segmentation model.

[0012] Optionally, determining the quality assessment model corresponding to different scenarios based on the first business information of the target business in different scenarios includes: The first business information in the target scenario is input into multiple second candidate models to obtain the corresponding quality assessment results; Based on the quality assessment results of the multiple second candidate models, a quality assessment model corresponding to the target scenario is determined.

[0013] Optionally, after performing a business quality assessment on the second business information using the quality assessment model corresponding to different scenarios, the method further includes: Obtain the evaluation feedback information, network-side service information, and terminal-side service information of the second service information; The evaluation feedback information of the second service information, network-side service information, and terminal-side service information are added to the training samples to update the relevant model for service quality evaluation.

[0014] To achieve the above objectives, embodiments of the present invention provide a business quality assessment apparatus, comprising: The analysis module is used to analyze the primary business information of the target business. The first processing module is used to divide the first service information according to the scenario when the analysis results indicate that the target service needs to be evaluated based on different scenarios. The second processing module is used to determine the quality assessment model corresponding to different scenarios based on the first business information of the target business in different scenarios. The evaluation module is used to evaluate the business quality of the second business information by using a determined quality evaluation model corresponding to different scenarios, when the second business information of the target business is obtained.

[0015] Optionally, the device further includes: The first acquisition module is used to acquire the initial network-side service information and the initial terminal-side service information of the target service; The third processing module is used to associate the initial network-side service information and the initial terminal-side service information to obtain the first service information.

[0016] Optionally, the third processing module is further configured to: According to the preset association strategy, extract network-side service information and terminal-side service information with the same association identifier from the initial network-side service information and the initial terminal-side service information, and then associate them.

[0017] Optionally, the first service information has a corresponding service quality indicator label; The analysis module is also used for at least one of the following: Based on the same service quality indicator label, the distribution range of the associated network-side service information and terminal-side service information in the first service information is analyzed under different service characteristics to obtain the first result; Based on different service quality indicator labels, the distribution range of the associated network-side service information and terminal-side service information in the first service information is analyzed under the same service characteristics to obtain the second result.

[0018] Optionally, the target service needs to be evaluated based on different scenarios and determined by at least one of the following: The first result indicates that, under the same business quality indicator label, the distribution range of different business characteristics of the target business is greater than the first threshold; The second result indicates that the target business has overlapping distribution ranges of the same business features under different business quality indicator labels, and the overlapping part is greater than the second threshold.

[0019] Optionally, the first processing module includes: The first processing submodule is used to determine the scene partitioning model; The segmentation submodule is used to segment the first business information using the scenario segmentation model.

[0020] Optionally, the first processing submodule includes: The first processing unit is used to input the first business information into multiple first candidate models to obtain the corresponding division results; A determining unit is configured to determine the scene partitioning model based on the partitioning results of the plurality of first candidate models; wherein the scene partitioning model satisfies at least one of the following: The scene classification accuracy is greater than the third threshold; The number of classifications is less than the number of classifications of the remaining first candidate models among the plurality of first candidate models excluding the scene segmentation model.

[0021] Optionally, the second processing module includes: The second processing submodule is used to input the first business information under the target scenario into multiple second candidate models to obtain the corresponding quality assessment results. The third processing submodule is used to determine the quality assessment model corresponding to the target scenario based on the quality assessment results of the multiple second candidate models.

[0022] Optionally, the device further includes: The second acquisition module is used to acquire the evaluation feedback information, network-side service information, and terminal-side service information of the second service information. The fourth processing module is used to add the evaluation feedback information of the second service information, the network-side service information, and the terminal-side service information to the training samples, and update the relevant model for service quality evaluation.

[0023] To achieve the above objectives, embodiments of the present invention provide a business quality assessment device, including a processor, the processor being used for: Analyze the primary business information of the target business; If the analysis results indicate that the target service needs to be evaluated based on different scenarios, the first service information will be divided according to the scenarios. Based on the first business information of the target business in different scenarios, determine the quality assessment model corresponding to different scenarios; Upon obtaining the second business information of the target business, the second business information is evaluated for business quality using the determined quality assessment model corresponding to different scenarios.

[0024] Optionally, the processor is further configured to: Obtain the initial network-side service information and initial terminal-side service information of the target service; The initial network-side service information and the initial terminal-side service information are correlated to obtain the first service information.

[0025] Optionally, the processor is further configured to: According to the preset association strategy, extract network-side service information and terminal-side service information with the same association identifier from the initial network-side service information and the initial terminal-side service information, and then associate them.

[0026] Optionally, the first service information has a corresponding service quality indicator label; The processor is also used for at least one of the following: Based on the same service quality indicator label, the distribution range of the associated network-side service information and terminal-side service information in the first service information is analyzed under different service characteristics to obtain the first result; Based on different service quality indicator labels, the distribution range of the associated network-side service information and terminal-side service information in the first service information is analyzed under the same service characteristics to obtain the second result.

[0027] Optionally, the target service needs to be evaluated based on different scenarios and determined by at least one of the following: The first result indicates that, under the same business quality indicator label, the distribution range of different business characteristics of the target business is greater than the first threshold; The second result indicates that the target business has overlapping distribution ranges of the same business features under different business quality indicator labels, and the overlapping part is greater than the second threshold.

[0028] Optionally, the processor is further configured to: Determine the scene segmentation model; The scenario segmentation model is used to segment the first business information.

[0029] Optionally, the processor is further configured to: The first business information is input into multiple first candidate models to obtain the corresponding partitioning results; Based on the partitioning results of the plurality of first candidate models, the scene partitioning model is determined; wherein, the scene partitioning model satisfies at least one of the following: The scene classification accuracy is greater than the third threshold; The number of classifications is less than the number of classifications of the remaining first candidate models among the plurality of first candidate models excluding the scene segmentation model.

[0030] Optionally, the processor is further configured to: The first business information in the target scenario is input into multiple second candidate models to obtain the corresponding quality assessment results; Based on the quality assessment results of the multiple second candidate models, a quality assessment model corresponding to the target scenario is determined.

[0031] Optionally, the processor is further configured to: Obtain the evaluation feedback information, network-side service information, and terminal-side service information of the second service information; The evaluation feedback information of the second service information, network-side service information, and terminal-side service information are added to the training samples to update the relevant model for service quality evaluation.

[0032] To achieve the above objectives, embodiments of the present invention provide a service quality assessment device, including a transceiver, a processor, a memory, and a program or instructions stored in the memory and executable on the processor; when the processor executes the program or instructions, it implements the service quality assessment method as described above.

[0033] To achieve the above objectives, embodiments of the present invention provide a readable storage medium having a program or instructions stored thereon, which, when executed by a processor, implement the steps in the service quality assessment method described above.

[0034] The beneficial effects of the above-described technical solution of the present invention are as follows: The method of this invention, after analyzing the first business information of a target business, can, based on the analysis results, divide the first business information that needs to be evaluated based on different scenarios according to the scenarios, and determine the applicable quality assessment model for each scenario based on the first business information of the target business in different scenarios. Finally, the determined quality assessment model is used to evaluate the business quality of the second business information of the target business. In this way, it does not rely excessively on human experience, and achieves differential modeling for sample data of the same business in different scenarios, enabling a more accurate assessment of perceived quality for the same business in different scenarios. Attached Figure Description

[0035] Figure 1 This is a flowchart of the business quality assessment method according to an embodiment of the present invention; Figure 2 This is a schematic diagram for box plot analysis; Figure 3 This is a schematic diagram of the module structure of the business quality assessment device according to an embodiment of the present invention; Figure 4 This is a structural diagram of the business quality assessment device according to an embodiment of the present invention; Figure 5 This is a structural diagram of a business quality assessment device according to another embodiment of the present invention. Detailed Implementation

[0036] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0037] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0038] In various embodiments of the present invention, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0039] In addition, the terms "system" and "network" are often used interchangeably in this article.

[0040] In the embodiments provided in this application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean that B is determined solely based on A, but can also be determined based on A and / or other information.

[0041] like Figure 1 As shown, an embodiment of the present invention provides a business quality assessment method, comprising: Step 101: Analyze the first business information of the target business.

[0042] Here, the target service is the service that requires evaluation; it can be pre-defined, such as video-on-demand or short video playback. The first service information is the historical service information of the target service, which can be understood as a sample for the quality assessment model.

[0043] Step 102: If the analysis results indicate that the target service needs to be evaluated based on different scenarios, the first service information is divided according to the scenario.

[0044] In this step, based on the analysis results of step 101, and considering the need for evaluation of the target business in different scenarios, the first business information will be divided according to the scenarios in order to proceed to the next step.

[0045] Step 103: Determine the quality assessment model corresponding to different scenarios based on the first business information of the target business in different scenarios.

[0046] In this step, after dividing the first business information according to the scenario in step 102, the first business information under different scenarios is used to determine the quality assessment model applicable to different scenarios.

[0047] Step 104: After obtaining the second business information of the target business, use the determined quality assessment model corresponding to different scenarios to assess the business quality of the second business information.

[0048] Here, the second business information refers to the new business information generated by the target business, which can also be understood as the business information to be evaluated. In this step, after determining the quality assessment model applicable to different scenarios in step 103, for the target business, the generated business information to be evaluated can be more effectively evaluated using the quality assessment model applicable to different scenarios.

[0049] Thus, following the steps described above, the method of this embodiment of the invention, after analyzing the first business information of the target business, can, based on the analysis results, divide the first business information that needs to be evaluated based on different scenarios according to the scenarios, and, based on the first business information of the target business in different scenarios, determine the applicable quality assessment model for different scenarios. Finally, the determined quality assessment model is used to evaluate the business quality of the second business information of the target business. In this way, there is no need to overly rely on human experience; differential modeling is achieved for sample data of the same business in different scenarios, enabling a more accurate assessment of perceived quality for the same business in different scenarios.

[0050] It should be noted that by applying the solution of this invention, target services can be set for geographical areas, and hourly service information can be used as secondary service information for service quality assessment. This allows for hourly-level statistics on the proportion of poor service quality (e.g., the proportion of service information with quality level y_label equal to 1 out of all service information in that hour), reflecting the overall network performance of the geographical area at different times of the day. Furthermore, work orders can be dispatched for service information with y_label predicted as 1, allowing professional network optimization personnel to analyze and locate the causes of poor service quality, optimize and adjust network parameters in certain areas, and compare the network performance before and after optimization based on the aforementioned statistics on the proportion of poor service quality.

[0051] It should be understood that in this embodiment, the first service information is obtained from the initial network-side service information and the initial terminal-side service information. Therefore, before step 101, the following steps are also included: Obtain the initial network-side service information and initial terminal-side service information of the target service; The initial network-side service information and the initial terminal-side service information are correlated to obtain the first service information.

[0052] In other words, each piece of information in the first service information consists of relevant initial network-side service information and initial terminal-side service information.

[0053] The methods for obtaining initial network-side service information and initial terminal-side service information are as follows: collecting service performance statistics data of the network side and terminal side through methods such as terminal dialing tests and deep packet inspection (DPI), including but not limited to data time, latency information, packet loss information, terminal information, etc.

[0054] Among them, the initial network-side service information is network-side statistical information, which has different dimensions from the initial terminal-side service information. The dimensions of the initial terminal-side service information are biased towards the user's experience with the service, while the dimensions of the initial network-side service information are biased towards the overall quality of the network. For example, when a user plays video A, the initial terminal-side service information will record the playback latency, number of stutters, and duration of each stutter of video A. The initial network-side service information will record the number of out-of-order service data packets, the total number of service data packets, latency information (including but not limited to the playback latency of video A), the number of stutters (including but not limited to the number of stutters of video A), and the duration of each stutter (including but not limited to the duration of each stutter of video A) during the playback time of video A.

[0055] Optionally, the step of associating the initial network-side service information and the initial terminal-side service information to obtain the first service information includes: According to the preset association strategy, extract network-side service information and terminal-side service information with the same association identifier from the initial network-side service information and the initial terminal-side service information, and then associate them.

[0056] Here, the preset association strategy includes an association identifier, which is used to determine related network-side service information and terminal-side service information. For example, if the association identifier is an Internet Protocol (IP) address, then network-side service information and terminal-side service information with the same IP address will be associated. In other words, in the initial network-side service information and the initial terminal-side service information, network-side service information and terminal-side service information corresponding to the same IP address are considered related.

[0057] Of course, the associated identifiers include, but are not limited to, IP address, service start time, video VID, etc.

[0058] It should also be noted that the initial terminal-side service information has corresponding service quality indicator tags. Therefore, after association processing, each piece of information in the first service information also has a corresponding service quality indicator tag.

[0059] Therefore, optionally, in this embodiment, the first service information has a corresponding service quality indicator label; Step 101 includes at least one of the following: Based on the same service quality indicator label, the distribution range of the associated network-side service information and terminal-side service information in the first service information is analyzed under different service characteristics to obtain the first result; Based on different service quality indicator labels, the distribution range of the associated network-side service information and terminal-side service information in the first service information is analyzed under the same service characteristics to obtain the second result.

[0060] Here, service quality indicator (SMI) tags are used to represent the quality level of a service. Since the initial terminal-side service information has corresponding SMI tags, after associating the relevant network-side and terminal-side service information from the initial network-side and initial terminal-side service information, the network-side service information can be matched with its corresponding SMI tags. That is, for service performance statistics, service information with corresponding SMI tags for the target service can be collected.

[0061] Furthermore, this business information can be further analyzed based on the same business quality indicator labels, under different business characteristics, to determine the distribution range of the associated network-side and terminal-side business information; and / or based on different business quality indicator labels, under the same business characteristics, to determine the distribution range of the associated network-side and terminal-side business information. In this way, the analysis results can be used to determine whether the target business needs to be evaluated based on different scenarios.

[0062] Taking the service quality indicator labels as "0" and "1", and the service features as A, B, and C as an example, based on the same service quality indicator labels, the distribution range of the associated network-side service information and terminal-side service information in the first service information is analyzed under different service features. Specifically, for the associated network-side service information and terminal-side service information with the corresponding service quality indicator label "0", the distribution range of their respective features A, B, and C is analyzed; and for the associated network-side service information and terminal-side service information with the corresponding service quality indicator label "1", the distribution range of their respective features A, B, and C is analyzed.

[0063] Similarly, taking the service quality indicator labels as "0" and "1", and the service features as "Feature A", "Feature B", and "Feature C" as an example, based on different service quality indicator labels, the distribution range of the associated network-side service information and terminal-side service information in the first service information is analyzed under the same service features. This analysis focuses on the associated network-side service information and terminal-side service information corresponding to Feature A, analyzing their distribution range under the service quality indicator labels "0" and "1"; the associated network-side service information and terminal-side service information corresponding to Feature B, analyzing their distribution range under the service quality indicator labels "0" and "1"; and the associated network-side service information and terminal-side service information corresponding to Feature C, analyzing their distribution range under the service quality indicator labels "0" and "1".

[0064] Specifically, the analysis process in this embodiment can employ a box plot method, for example, as follows: Figure 2As shown, each piece of information in the first business information (each piece of information includes associated network-side business information and terminal-side business information) is grouped according to the business quality indicator label. The 1 / 4 and 3 / 4 quantiles and the inner limit range of each business feature (also known as performance statistical feature) in each group are calculated. The inner limit calculation formula is: upper inner limit = 3 / 4 quantile + 1.5 * (3 / 4 quantile - 1 / 4 quantile), lower inner limit = 3 / 4 quantile - 1.5 * (3 / 4 quantile - 1 / 4 quantile). Then, the differences in data distribution of each performance statistical feature (especially key performance features) under the same business quality indicator label and the differences in data distribution under different business quality indicator labels are compared and analyzed.

[0065] The service quality indicator labels can include the percentage of buffering time, latency, etc. Taking the percentage of buffering time as an example, the formula for calculating the percentage of buffering time is: Percentage of buffering time = Buffering time / Total video playback time. The percentage of buffering time is segmented according to certain standards and discretized into different percentage intervals. A sequence of values ​​starting from 0 is used to represent each percentage interval, i.e., different quality levels, denoted as y_label. The larger the sequence value, the worse the quality.

[0066] Optionally, the target service needs to be evaluated based on different scenarios and determined by at least one of the following: The first result indicates that, under the same business quality indicator label, the distribution range of different business characteristics of the target business is greater than the first threshold; The second result indicates that the target business has overlapping distribution ranges of the same business features under different business quality indicator labels, and the overlapping part is greater than the second threshold.

[0067] Here, the first threshold and the second threshold are preset.

[0068] Optionally, in this embodiment, dividing the first business information according to scenarios includes: Determine the scene segmentation model; The scenario segmentation model is used to segment the first business information.

[0069] Thus, for target businesses that need to be evaluated based on different scenarios in business performance statistics, a scenario segmentation model will be determined first, and then the first business information of the target business will be segmented using the scenario segmentation model, so that the corresponding information can be used to train the quality evaluation model for different scenarios.

[0070] Optionally, the determination of the scene partitioning model includes: The first business information is input into multiple first candidate models to obtain the corresponding partitioning results; Based on the partitioning results of the plurality of first candidate models, the scene partitioning model is determined; wherein, the scene partitioning model satisfies at least one of the following: The scene classification accuracy is greater than the third threshold; The number of classifications is less than the number of classifications of the remaining first candidate models among the plurality of first candidate models excluding the scene segmentation model.

[0071] Here, the multiple first candidate models are pre-set candidate models for scene segmentation. By inputting the first business information into the multiple first candidate models, the corresponding segmentation results are obtained, and then the scene segmentation model is selected from the segmentation results of these multiple first candidate models. The scene segmentation model can be selected based on at least one of scene classification accuracy and the number of classifications, and then the scene segmentation model is saved.

[0072] In this embodiment, to balance the accuracy of scene classification with the complexity of subsequent differential modeling, a smaller number of categories (generally no more than 4 categories) is selected while ensuring that the scene classification accuracy meets certain requirements. Preferably, the scene partitioning model is the first candidate model with the smallest number of categories among multiple first candidate models, whose scene classification accuracy is greater than the third threshold. Here, the third threshold is preset.

[0073] The accuracy of scene classification is obtained by statistically analyzing the proportion of each business quality indicator label in samples of the same scene category; the more uneven the proportion, the better.

[0074] It should also be noted that in this embodiment, the first candidate model can be a clustering model, which can be used to classify the scenarios of the first business information through clustering methods.

[0075] Suppose that the first candidate model is the DBSCAN clustering model. By setting reasonable parameters, the first business information (i.e., samples) is divided into several categories. The sequence values ​​starting from 0 are taken to represent each scenario in order, that is, different data distributions, which are represented as class_label. The method to measure the accuracy of scenario classification is to count the proportion of each y_label in samples with the same class_label. The more uneven the proportion, the better.

[0076] Furthermore, after segmenting the first business information using the established scenario segmentation model, it is necessary to determine the applicable quality assessment model for the business information in different scenarios, i.e., differential modeling. Optionally, step 103 includes: The first business information in the target scenario is input into multiple second candidate models to obtain the corresponding quality assessment results; Based on the quality assessment results of the multiple second candidate models, a quality assessment model corresponding to the target scenario is determined.

[0077] Here, different scenarios are selected sequentially as target scenarios, and the quality assessment model for each scenario is determined through the steps described above. For each target scenario, multiple second candidate models are pre-set for selection. Thus, when determining the quality assessment model corresponding to the target scenario, the first business information under the target scenario is input into multiple second candidate models to obtain more effective quality assessment results. Subsequently, the quality assessment results from these multiple second candidate models can better select the appropriate quality assessment model for the target scenario.

[0078] These multiple second-candidate models can be models from various AI model libraries, such as XGBoost, DNN, LGBT, and SVM. During the selection of the quality assessment model based on the quality assessment results, metrics such as precision, recall, and accuracy can be used to measure these multiple second-candidate models, in order to select the model that best evaluates the primary business information in the target scenario.

[0079] It should be understood that, in this embodiment, the training samples may include the aforementioned initial network-side service information and initial terminal-side service information, used for training at least one of the scenario segmentation model and the quality assessment model. The training samples also include valid quality assessment information corresponding to the first service information. That is, in the first service information, each associated initial network-side service information and initial terminal-side service information corresponds to a valid quality assessment information, in order to complete the training of the model.

[0080] After determining the quality assessment model for each scenario, the determined scenario segmentation model and quality assessment model can be used to conduct a business quality assessment of the second business information (which can also be understood as the business information to be assessed) of the target business.

[0081] Of course, the second service information is similar to the first service information, and it needs to be obtained by associating the service information to be evaluated for the target service on the network side and the terminal side. Furthermore, during the evaluation process, a defined scenario segmentation model needs to be used for segmentation, thereby inputting the second service information of a certain scenario into the quality evaluation model of that scenario to complete the service quality evaluation.

[0082] Considering that the determined quality assessment model may be inapplicable during use, this embodiment also proposes a feedback optimization mechanism. Thus, optionally, in this embodiment, after step 104, the following step is also included: Obtain the evaluation feedback information, network-side service information, and terminal-side service information of the second service information; The evaluation feedback information of the second service information, network-side service information, and terminal-side service information are added to the training samples to update the relevant model for service quality evaluation.

[0083] Here, the evaluation feedback information can be user complaints about the quality of the second service information, or it can be obtained from real-time monitoring of user service data collected on the network side. This evaluation feedback information includes user quality evaluation information for the second service information. Therefore, after obtaining the evaluation feedback information, network-side service information, and terminal-side service information for the second service information, these three pieces of information will be added to the training samples to update the relevant model for service quality evaluation, thereby achieving model tuning and automated model optimization.

[0084] Specifically, the relevant model includes at least one of a scene segmentation model and a quality assessment model.

[0085] The following example, using video-on-demand (VOD) as the target service, illustrates the application of this invention: 1. Obtain initial network-side service information and initial terminal-side service information; Initial network-side service information (such as XDR performance statistics data) and initial terminal-side service information (such as SDK terminal data) generated in video-on-demand services are collected through technologies such as DPI detection and terminal testing.

[0086] 2. Associate the XDR performance statistics feature data and SDK terminal data according to the association identifier (such as user IP address, service start time and video VID) so that the XDR performance statistics feature data matches the service quality indicator label of the SDK terminal.

[0087] 3. Analyze XDR performance statistics and SDK terminal data based on business quality indicator tags.

[0088] Using a box plot approach, the associated XDR performance statistics data and SDK terminal data are grouped according to business quality indicator labels. The 1 / 4 and 3 / 4 quantiles and inner limits of each performance statistics feature in each group are calculated. The differences in data distribution of each performance statistics feature, especially key performance features, are compared and analyzed under the same business quality indicator label and under different business quality indicator labels.

[0089] 4. Based on multiple first candidate models (DBSCAN clustering model), classify the XDR performance statistical feature data and SDK terminal data (samples). Each first candidate model divides the samples into several classes, and takes the sequence values ​​starting from 0 to represent each scene in order, that is, different data distributions, denoted as class_label. The method to measure the scene classification accuracy is to count the proportion of each y_label in the samples with the same class_label. The more uneven the proportion, the better.

[0090] 5. While ensuring that the accuracy of scene classification is greater than the third threshold, select a smaller number of categories (generally no more than 4 categories), determine the scene classification model, and save it.

[0091] 6. Model the samples with different class_labels output by the scene classification model separately to build multiple quality assessment models, i.e., differential modeling.

[0092] For each class_label, multiple second candidate models (including models from a model library of various AI models such as XGBoost, DNN, LGBT, and SVM) are trained. The evaluation performance of each model is measured by precision, recall, and accuracy. The model that makes the evaluation of that class of samples more accurate is selected. For example, XGBoost is selected for samples with class_label=0, GBDT is selected for samples with class_label=1, XGBoost is selected for samples with class_label=2, and so on, so as to achieve the best overall evaluation performance.

[0093] 7. Deploy the scene classification model and quality assessment model online to perform quality assessment on newly generated XDR performance statistics data and SDK terminal data for video-on-demand services. Additionally, obtain user feedback information (real-time monitored network-side user service data and / or collected user complaint information). Obtain scene category labels for each complaint sample (complaint-related XDR performance statistics data and SDK terminal data) through the scene classification model (the scene category label corresponds to the applicable quality assessment model). Add the complaint samples to the training set of the quality assessment model corresponding to the scene category label, and fine-tune and update the scene classification model and / or quality assessment model to achieve automated model optimization.

[0094] In summary, the method of the embodiments of the present invention has the following advantages: Instead of relying too much on human experience, it does not pre-determine the relationship between various network performance statistical parameters and business quality indicator labels that characterize service quality. Instead, it uses a variety of technical means, such as data analysis, clustering, and AI models, to deeply explore the relationship between business information and business quality indicator labels, and to efficiently utilize the effective information carried by the raw data. Scenario classification is performed on sample sets of the same business under different scenarios, and differential modeling is implemented for data in different scenarios, so that the perceived quality of the same business can be more accurately evaluated under different scenarios. By segmenting online user complaints into different scenarios and adding them to the training set of the corresponding scenario, the evaluation model can be optimized, making it more targeted and resulting in better overall evaluation performance.

[0095] like Figure 3As shown, an embodiment of the present invention provides a business quality assessment device, comprising: Analysis module 310 is used to analyze the first business information of the target business; The first processing module 320 is used to divide the first service information according to the scenario when the analysis results indicate that the target service needs to be evaluated based on different scenarios. The second processing module 330 is used to determine the quality assessment model corresponding to different scenarios based on the first business information of the target business in different scenarios. The evaluation module 340 is used to evaluate the business quality of the second business information by using a determined quality evaluation model corresponding to different scenarios when the second business information of the target business is obtained.

[0096] Optionally, the device further includes: The first acquisition module is used to acquire the initial network-side service information and the initial terminal-side service information of the target service; The third processing module is used to associate the initial network-side service information and the initial terminal-side service information to obtain the first service information.

[0097] Optionally, the third processing module is further configured to: According to the preset association strategy, extract network-side service information and terminal-side service information with the same association identifier from the initial network-side service information and the initial terminal-side service information, and then associate them.

[0098] Optionally, the first service information has a corresponding service quality indicator label; The analysis module is also used for at least one of the following: Based on the same service quality indicator label, the distribution range of the associated network-side service information and terminal-side service information in the first service information is analyzed under different service characteristics to obtain the first result; Based on different service quality indicator labels, the distribution range of the associated network-side service information and terminal-side service information in the first service information is analyzed under the same service characteristics to obtain the second result.

[0099] Optionally, the target service needs to be evaluated based on different scenarios and determined by at least one of the following: The first result indicates that, under the same business quality indicator label, the distribution range of different business characteristics of the target business is greater than the first threshold; The second result indicates that the target business has overlapping distribution ranges of the same business features under different business quality indicator labels, and the overlapping part is greater than the second threshold.

[0100] Optionally, the first processing module includes: The first processing submodule is used to determine the scene partitioning model; The segmentation submodule is used to segment the first business information using the scenario segmentation model.

[0101] Optionally, the first processing submodule includes: The first processing unit is used to input the first business information into multiple first candidate models to obtain the corresponding division results; A determining unit is configured to determine the scene partitioning model based on the partitioning results of the plurality of first candidate models; wherein the scene partitioning model satisfies at least one of the following: The scene classification accuracy is greater than the third threshold; The number of classifications is less than the number of classifications of the remaining first candidate models among the plurality of first candidate models excluding the scene segmentation model.

[0102] Optionally, the second processing module includes: The second processing submodule is used to input the first business information under the target scenario into multiple second candidate models to obtain the corresponding quality assessment results. The third processing submodule is used to determine the quality assessment model corresponding to the target scenario based on the quality assessment results of the multiple second candidate models.

[0103] Optionally, the device further includes: The second acquisition module is used to acquire the evaluation feedback information, network-side service information, and terminal-side service information of the second service information. The fourth processing module is used to add the evaluation feedback information of the second service information, the network-side service information, and the terminal-side service information to the training samples, and update the relevant model for service quality evaluation.

[0104] After analyzing the primary business information of the target service, this device can categorize the primary business information requiring evaluation based on different scenarios according to the analysis results. Then, based on the primary business information of the target service in different scenarios, it determines the applicable quality assessment model for each scenario. Finally, it uses the determined quality assessment model to evaluate the business quality of the secondary business information of the target service. In this way, it avoids excessive reliance on human experience and achieves differentiated modeling for sample data of the same service in different scenarios, enabling more accurate assessment of perceived quality for the same service in different scenarios.

[0105] It should be noted that this device is an apparatus that applies the above-mentioned business quality assessment method, and the implementation method of the above-mentioned method embodiment is applicable to this device and can achieve the same technical effect.

[0106] like Figure 4 As shown, a service quality assessment device 400 according to an embodiment of the present invention includes a processor 410, wherein... The processor is used for: Analyze the primary business information of the target business; If the analysis results indicate that the target service needs to be evaluated based on different scenarios, the first service information will be divided according to the scenarios. Based on the first business information of the target business in different scenarios, determine the quality assessment model corresponding to different scenarios; Upon obtaining the second business information of the target business, the second business information is evaluated for business quality using the determined quality assessment model corresponding to different scenarios.

[0107] Optionally, the processor is further configured to: Obtain the initial network-side service information and initial terminal-side service information of the target service; The initial network-side service information and the initial terminal-side service information are correlated to obtain the first service information.

[0108] Optionally, the processor is further configured to: According to the preset association strategy, extract network-side service information and terminal-side service information with the same association identifier from the initial network-side service information and the initial terminal-side service information, and then associate them.

[0109] Optionally, the first service information has a corresponding service quality indicator label; The processor is also used for at least one of the following: Based on the same service quality indicator label, the distribution range of the associated network-side service information and terminal-side service information in the first service information is analyzed under different service characteristics to obtain the first result; Based on different service quality indicator labels, the distribution range of the associated network-side service information and terminal-side service information in the first service information is analyzed under the same service characteristics to obtain the second result.

[0110] Optionally, the target service needs to be evaluated based on different scenarios and determined through at least one of the following: The first result indicates that, under the same business quality indicator label, the distribution range of different business characteristics of the target business is greater than the first threshold; The second result indicates that the target business has overlapping distribution ranges of the same business features under different business quality indicator labels, and the overlapping part is greater than the second threshold.

[0111] Optionally, the processor is further configured to: Determine the scene segmentation model; The scenario segmentation model is used to segment the first business information.

[0112] Optionally, the processor is further configured to: The first business information is input into multiple first candidate models to obtain the corresponding partitioning results; Based on the partitioning results of the plurality of first candidate models, the scene partitioning model is determined; wherein, the scene partitioning model satisfies at least one of the following: The scene classification accuracy is greater than the third threshold; The number of classifications is less than the number of classifications of the remaining first candidate models among the plurality of first candidate models excluding the scene segmentation model.

[0113] Optionally, the processor is further configured to: The first business information in the target scenario is input into multiple second candidate models to obtain the corresponding quality assessment results; Based on the quality assessment results of the multiple second candidate models, a quality assessment model corresponding to the target scenario is determined.

[0114] Optionally, the processor is further configured to: Obtain the evaluation feedback information, network-side service information, and terminal-side service information of the second service information; The evaluation feedback information of the second service information, network-side service information, and terminal-side service information are added to the training samples to update the relevant model for service quality evaluation.

[0115] The service quality assessment equipment also includes a transceiver 420, which is used to receive and send data under the control of the processor 410.

[0116] After analyzing the primary business information of the target service, the device can categorize the primary business information requiring evaluation based on different scenarios according to the analysis results. Then, based on the primary business information of the target service in different scenarios, it determines the applicable quality assessment model for each scenario. Finally, it uses the determined quality assessment model to evaluate the business quality of the secondary business information of the target service. In this way, it avoids excessive reliance on human experience and achieves differentiated modeling for sample data of the same service in different scenarios, enabling more accurate assessment of perceived quality for the same service in different scenarios.

[0117] It should be noted that this device is an apparatus that applies the above-mentioned business quality assessment method, and the implementation method of the above-mentioned method embodiment is applicable to this device and can achieve the same technical effect.

[0118] Another embodiment of the business quality assessment device of the present invention, such as Figure 5 As shown, it includes a transceiver 510, a processor 500, a memory 520, and a program or instructions stored in the memory 520 and executable on the processor 500; when the processor 500 executes the program or instructions, it implements the service quality assessment method as described above.

[0119] The transceiver 510 is used to receive and send data under the control of the processor 500.

[0120] Among them, Figure 5 In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 500) and memory (memory 520). The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 510 may be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. The processor 500 is responsible for managing the bus architecture and general processing, and the memory 520 may store data used by the processor 500 during operation.

[0121] An embodiment of the present invention provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the steps in the service quality assessment method described above and achieve the same technical effect. To avoid repetition, further details are omitted here.

[0122] The processor mentioned above is the processor in the service quality assessment device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] It should be further noted that many of the functional components described in this specification are referred to as modules in order to emphasize the independence of their implementation.

[0124] In this embodiment of the invention, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.

[0125] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.

[0126] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.

[0127] The exemplary embodiments described above are with reference to the accompanying drawings. Many different forms and embodiments are feasible without departing from the spirit and teachings of the invention. Therefore, the invention should not be construed as limiting the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make the invention complete and convey the scope of the invention to those skilled in the art. In these drawings, component dimensions and relative dimensions may be exaggerated for clarity. The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. As used herein, unless clearly indicated otherwise, the singular forms “a,” “an,” and “the” are intended to include all such forms. It will be further understood that the terms “comprising” and / or “including”, when used in this specification, indicate the presence of the stated features, integers, steps, operations, components, and / or elements, but do not exclude the presence or addition of one or more other features, integers, steps, operations, components, and / or groups thereof. Unless otherwise indicated, when stated, a range of values ​​includes the upper and lower limits of the range and any subranges in between.

[0128] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A business quality assessment method, characterized in that, include: Analyze the primary business information of the target business; If the analysis results indicate that the target service needs to be evaluated based on different scenarios, the first service information will be divided according to the scenarios. Based on the first business information of the target business in different scenarios, determine the quality assessment model corresponding to different scenarios; Upon obtaining the second business information of the target business, the second business information is evaluated for business quality using the determined quality assessment model corresponding to different scenarios. The first service information has a corresponding service quality indicator label; The analysis of the first business information of the target business includes at least one of the following: Based on the same service quality indicator label, the distribution range of the associated network-side service information and terminal-side service information in the first service information is analyzed under different service characteristics to obtain the first result; Based on different service quality indicator labels, the distribution range of the associated network-side service information and terminal-side service information in the first service information is analyzed under the same service characteristics to obtain the second result; The first result and the second result are used to determine whether the target service needs to be evaluated based on different scenarios.

2. The method according to claim 1, characterized in that, Before analyzing the first business information of the target business, the following is also included: Obtain the initial network-side service information and initial terminal-side service information of the target service; The initial network-side service information and the initial terminal-side service information are correlated to obtain the first service information.

3. The method according to claim 2, characterized in that, The step of associating the initial network-side service information and the initial terminal-side service information to obtain the first service information includes: According to the preset association strategy, extract network-side service information and terminal-side service information with the same association identifier from the initial network-side service information and the initial terminal-side service information, and then associate them.

4. The method according to claim 1, characterized in that, The target business needs to be evaluated based on different scenarios and determined by at least one of the following: The first result indicates that, under the same business quality indicator label, the distribution range of different business characteristics of the target business is greater than the first threshold; The second result indicates that the target business has overlapping distribution ranges of the same business features under different business quality indicator labels, and the overlapping part is greater than the second threshold.

5. The method according to claim 1, characterized in that, The step of dividing the first business information according to scenarios includes: Determine the scene segmentation model; The scenario segmentation model is used to segment the first business information.

6. The method according to claim 5, characterized in that, The determined scene partitioning model includes: The first business information is input into multiple first candidate models to obtain the corresponding partitioning results; Based on the partitioning results of the plurality of first candidate models, the scene partitioning model is determined; wherein, the scene partitioning model satisfies at least one of the following: The scene classification accuracy is greater than the third threshold; The number of classifications is less than the number of classifications of each of the remaining first candidate models other than the scene segmentation model among the plurality of first candidate models.

7. The method according to claim 1, characterized in that, The step of determining the quality assessment model corresponding to different scenarios based on the first business information of the target business in different scenarios includes: The first business information in the target scenario is input into multiple second candidate models to obtain the corresponding quality assessment results; Based on the quality assessment results of the multiple second candidate models, a quality assessment model corresponding to the target scenario is determined.

8. The method according to claim 1, characterized in that, After performing a business quality assessment on the second business information using the quality assessment models corresponding to different scenarios, the method further includes: Obtain the evaluation feedback information, network-side service information, and terminal-side service information of the second service information; The evaluation feedback information of the second service information, the network-side service information, and the terminal-side service information are added to the training samples to update the scene segmentation model and / or the quality assessment model.

9. A business quality assessment device, characterized in that, include: The analysis module is used to analyze the primary business information of the target business. The first processing module is used to divide the first service information according to the scenario when the analysis results indicate that the target service needs to be evaluated based on different scenarios. The second processing module is used to determine the quality assessment model corresponding to different scenarios based on the first business information of the target business in different scenarios. The evaluation module is used to evaluate the business quality of the second business information using a determined quality evaluation model corresponding to different scenarios when the second business information of the target business is obtained. The analysis module is also used for at least one of the following: Based on the same service quality indicator label, the distribution range of the associated network-side service information and terminal-side service information in the first service information is analyzed under different service characteristics to obtain the first result; Based on different service quality indicator labels, the distribution range of the associated network-side service information and terminal-side service information in the first service information is analyzed under the same service characteristics to obtain the second result; The first result and the second result are used to determine whether the target service needs to be evaluated based on different scenarios.

10. A business quality assessment device, characterized in that, Includes a processor, the processor being used for: Analyze the primary business information of the target business; If the analysis results indicate that the target service needs to be evaluated based on different scenarios, the first service information will be divided according to the scenarios. Based on the first business information of the target business in different scenarios, determine the quality assessment model corresponding to different scenarios; Upon obtaining the second business information of the target business, the second business information is evaluated for business quality using the determined quality assessment model corresponding to different scenarios. The processor is also used for at least one of the following: Based on the same service quality indicator label, the distribution range of the associated network-side service information and terminal-side service information in the first service information is analyzed under different service characteristics to obtain the first result; Based on different service quality indicator labels, the distribution range of the associated network-side service information and terminal-side service information in the first service information is analyzed under the same service characteristics to obtain the second result; The first result and the second result are used to determine whether the target service needs to be evaluated based on different scenarios.

11. A business quality assessment device, comprising: A transceiver, a processor, a memory, and a program or instructions stored in the memory and executable on the processor; characterized in that, when the processor executes the program or instructions, it implements the service quality assessment method as described in any one of claims 1-8.

12. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the steps in the business quality assessment method as described in any one of claims 1-8.

Images