Application fluency evaluation method and device

Abstract

The invention provides a fluency evaluation method and device for an application in the field of artificial intelligence, and the method and device are used for analyzing the message features of a message flow through a classifier, obtaining the fluency evaluation result of the application, obtaining the confidence distribution corresponding to the fluency evaluation result, updating the classifier according to the confidence distribution and a preset expected value, and obtaining the fluency evaluation result of the application. The detection accuracy of the classifier is improved. The method comprises the steps that multiple groups of message flows and corresponding labels are acquired, the message flows are message flows input into an application, and the labels are used for indicating the fluency when the application processes the corresponding message flows; analyzing the message flow and the label through a classifier to obtain a fluency evaluation result; acquiring confidence distribution corresponding to the fluency evaluation result; parameters of the classifier are updated according to the confidence degree distribution and a preset expected value, an updated classifier is obtained, and the updated classifier is used for conducting fluency evaluation on the message flow subsequently input into the application.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method and device for evaluating the fluency of an application. Background Art

[0002] The development of broadband access has entered the all-optical era. Operators have shifted from pursuing high-speed networks to pursuing high-quality networks. Home users are not only pursuing high bandwidth, but are also pursuing the ultimate experience. Poor quality will have a great impact on user experience. Poor quality of optical access equipment (including weak light) can easily lead to slow Internet access, video freezes and other problems, which in turn lead to user complaints. In order to ensure the user's Internet experience, improve the efficiency of fault handling during weak light rectification, and reduce repeated visits to the site, operators urgently need a complete solution to realize the quality difference identification of different applications, early warning of faults and accurate demarcation of fault point locations.

[0003] Currently, in order to identify whether the home broadband network has poor quality, user perception indicators and network or performance indicators are generally collected, and then based on the user perception indicators and network or performance indicators, a home broadband network poor quality identification model is constructed to identify the poor quality of the home broadband network.

[0004] However, the quality difference detection results of existing artificial intelligence (AI) algorithm quality difference detection technology are inaccurate in some scenarios, which can easily bring misleading messages to users and affect the credibility of quality difference identification results. Summary of the invention

[0005] An embodiment of the present application provides a method and device for evaluating the fluency of an application, which is used to analyze the message features of a message flow through a classifier, obtain an application fluency evaluation result and a corresponding confidence level, and update the classifier according to the confidence level and a preset expected value to improve the detection accuracy of the classifier.

[0006] In view of this, on the first aspect, the present application provides a method for evaluating the fluency of an application, comprising: first, obtaining multiple groups of message flows and corresponding labels after preprocessing initial application messages, the initial application messages being messages during the operation of the application, and the labels being used to indicate the fluency of the application; subsequently, analyzing the message flows and the corresponding labels through a classifier, and obtaining a fluency evaluation result of the application and a confidence level corresponding to the fluency evaluation result based on an output of the classifier; after obtaining a confidence distribution of the fluency evaluation result, the parameters of the classifier can be updated based on the confidence distribution and a preset expected value to obtain an updated classifier, and the updated classifier can perform fluency evaluation on subsequent message flows.

[0007] In the implementation manner of the present application, the message flow and the corresponding labels can be analyzed by a classifier, and the fluency evaluation result of the application and the corresponding confidence can be obtained according to the output result of the classifier, and the classifier can be tuned and updated according to the confidence distribution and the preset expected value, thereby shortening the training time of the optimized update model, and the updated classifier retains the output results with high confidence, so that the accuracy of the detection results of the updated classifier is improved.

[0008] In a possible implementation, the aforementioned analysis of the message flow and the label by a classifier to obtain the fluency evaluation result may include: extracting features from the message flow to obtain message features, the message features include basic message features and augmented message features, the augmented message features are obtained by augmenting the basic message features; analyzing the message features and the label by a classifier to obtain the fluency evaluation result.

[0009] In an embodiment of the present application, the basic message features can be augmented to obtain augmented message features, and feature extraction of the message flow can obtain 85-dimensional message features. The 85-dimensional message features are used as the input of the classifier to improve the accuracy of the output results.

[0010] In a possible implementation, the aforementioned feature extraction of the message flow to obtain the message feature may include: using a sliding window algorithm to extract the feature of the message flow to obtain the message feature.

[0011] In a possible implementation, the aforementioned obtaining of the confidence distribution corresponding to the fluency evaluation result may include: analyzing message features and labels by generating a flow network to obtain the confidence distribution corresponding to the fluency evaluation result.

[0012] In an embodiment of the present application, message features may be input into a generated flow network to obtain a confidence distribution of a fluency evaluation result, so that the classifier may be subsequently updated according to the confidence distribution.

[0013] In a possible implementation, the method may further include: obtaining message features of sample data; calculating the sum of squares of message features of sample data, the Pearson correlation coefficient and mutual information between the message features of sample data and the fluency evaluation results of sample data; calculating the confidence corresponding to the fluency evaluation results of sample data based on the Pearson correlation coefficient, mutual information and sum of squares; training an untrained generative flow network based on the message features of sample data and the confidence corresponding to the fluency evaluation results of sample data to obtain a generative flow network.

[0014] In an embodiment of the present application, the confidence of the fluency assessment result can be calculated based on the Pearson correlation coefficient and mutual information between the fluency assessment result and the message features, and the sum of the squares of the message features, and the generation flow network is trained based on the confidence and the message features of the sample data, so that the input message features can obtain the corresponding confidence through the generation flow network.

[0015] In a possible implementation, the aforementioned updating of classifier parameters according to the confidence distribution and the preset expected value to obtain the updated classifier may include: if there is a fluency assessment result whose confidence is lower than the preset expected value, removing the fluency assessment result to obtain the updated classifier.

[0016] In the embodiment of the present application, the fluency evaluation results with confidence levels lower than a preset expected value are removed, the fluency evaluation results with high confidence levels are retained, and the classifier is updated, thereby improving the detection accuracy of the updated classifier.

[0017] In a possible implementation, the aforementioned acquisition of multiple groups of message flows and corresponding labels may include: acquiring a jam record table, an initial application message, and an application identification table, the jam record table being used to record the fluency of the application, and the application identification table including the application category and activity category of each message flow; preprocessing the initial application message according to the jam record table and the application identification table to obtain the message flow and the corresponding label.

[0018] In an embodiment of the present application, the initial application message is not a pure application message, but contains a large amount of invalid information, which also makes it impossible for the message flow to correspond one-to-one with the label. The initial application message can be preprocessed according to the jam record table and the application identification table to obtain a useful message flow, and make the message flow correspond one-to-one with the label, so as to facilitate the subsequent training of the model as training data.

[0019] In a possible implementation, the aforementioned preprocessing of the initial application message according to the jam record table and the application identification table to obtain the message flow and the corresponding label may include: extracting the valid feature field of the initial application message to obtain a temporary application message, the valid feature field including at least one of the message flow rate, the average message flow rate, the average message packet size, the maximum value of the message packet, or the minimum value of the message packet; grouping the temporary application messages according to the quintuple to obtain multiple groups of temporary message flows; determining the message flow from the temporary message flow according to the application identification table; obtaining the message flow and the corresponding label according to the jam record table and the message flow.

[0020] In a possible implementation, the method may further include: acquiring training data; clustering the training data to obtain cluster centers, the cluster centers including M jamming class centers and N non-jamming class centers, the jamming class centers are numbered as any value from 1 to M, and the non-jamming class centers are numbered as any value from M+1 to M+N, M is an integer greater than or equal to 1, and N is an integer greater than or equal to 1; determining a fluency evaluation result based on the distance from the message features of the calculated message flow to the cluster center; and calculating a corresponding confidence level based on the fluency evaluation result.

[0021] In an embodiment of the present application, the acquired training data can also be clustered to obtain multiple cluster centers, and the distance between the message features and the cluster centers can be calculated to determine the fluency assessment result, and the corresponding confidence can be calculated based on the fluency assessment result. This provides a variety of methods for applying fluency assessment and calculating confidence, thereby increasing the diversity of solutions.

[0022] In a possible implementation, the aforementioned determination of the fluency evaluation result based on the distance from the calculated message features of the message flow to the cluster center may include: calculating the distance from the message features of the message flow to the center of each jamming class to obtain a first distance set; calculating the distance from the message features of the message flow to the center of each non-jamming class to obtain a second distance set; combining the first distance set and the second distance set to obtain a third distance set; if the number of the category center corresponding to the minimum distance value in the third distance set is less than or equal to M, the fluency evaluation result is determined to be jamming; if the number of the category center corresponding to the minimum distance value in the third distance set is greater than M, the fluency evaluation result is determined to be non-jamming.

[0023] In a possible implementation, if the fluency evaluation result is a freeze, the aforementioned calculation of the corresponding confidence based on the fluency evaluation result may include: determining the minimum distance value in the third distance set; calculating the average value of the third distance set; determining the minimum distance value in the first distance set; calculating the average value of the first distance set; calculating the confidence of the fluency evaluation result based on the minimum distance value in the third distance set, the average value of the third distance set, the minimum distance value in the first distance set, and the average value of the first distance set.

[0024] In a possible implementation, if the fluency evaluation result is no lag, the aforementioned calculation of the corresponding confidence based on the fluency evaluation result may include: determining the minimum distance value in the third distance set; calculating the average value of the third distance set; determining the minimum distance value in the second distance set; calculating the average value of the second distance set; calculating the confidence of the fluency evaluation result based on the minimum distance value in the third distance set, the average value of the third distance set, the minimum distance value in the second distance set, and the average value of the second distance set.

[0025] In a possible implementation, the aforementioned clustering of the training data may include: clustering the training data using a K-means clustering algorithm or a fuzzy clustering method.

[0026] In a possible implementation, the classifier may be a decision tree model, which is used to analyze the message flow and the label to obtain a fluency evaluation result.

[0027] In a second aspect, the present application provides a device for evaluating the fluency of an application, comprising:

[0028] An acquisition module, used to acquire multiple groups of message flows and corresponding labels, where the message flows are message flows input to the application, and the labels are used to indicate the fluency of the application when processing the corresponding message flows;

[0029] The analysis module is used to analyze the message flow and labels through the classifier to obtain the fluency evaluation result;

[0030] The acquisition module is also used to obtain the confidence distribution corresponding to the fluency assessment result;

[0031] The updating module is used to update the parameters of the classifier according to the confidence distribution and the preset expected value to obtain an updated classifier, and the updated classifier is used to evaluate the fluency of the message flow of the subsequent input application.

[0032] In one possible implementation, the analysis module is specifically used to: extract features from the message flow to obtain message features, where the message features include basic message features and augmented message features, where the augmented message features are obtained by augmenting the basic message features; analyze the message features and labels through a classifier to obtain a fluency evaluation result.

[0033] In a possible implementation, the analysis module is specifically used to: extract features from the message flow using a sliding window algorithm to obtain message features.

[0034] In a possible implementation, the acquisition module is specifically used to: analyze message features and labels by generating a flow network to obtain a confidence distribution corresponding to a fluency evaluation result.

[0035] In a possible implementation, a fluency assessment device for an application may further include: an acquisition module, which is also used to acquire message features of sample data; a calculation module, which is specifically used to calculate the sum of squares of message features of sample data, the Pearson correlation coefficient and mutual information between the message features of sample data and the fluency assessment results of the sample data; the calculation module is also used to calculate the confidence corresponding to the fluency assessment results of the sample data based on the Pearson correlation coefficient, mutual information and sum of squares; a training module, which is specifically used to train an untrained generative flow network based on the message features of the sample data and the confidence corresponding to the fluency assessment results of the sample data to obtain a generative flow network.

[0036] In a possible implementation, the updating module is specifically configured to remove the fluency evaluation result if the confidence level of the fluency evaluation result is lower than a preset expected value, so as to obtain an updated classifier.

[0037] In one possible implementation, the acquisition module is specifically used to: obtain a jam record table, an initial application message, and an application identification table, wherein the jam record table is used to record the fluency of the application, and the application identification table includes the application category and activity category of each message flow; according to the jam record table and the application identification table, the initial application message is preprocessed to obtain the message flow and the corresponding label.

[0038] In one possible implementation, the acquisition module is specifically used to: extract the valid feature fields of the initial application message to obtain a temporary application message, the valid feature fields including at least one of a message flow rate, an average message flow rate, an average message packet size, a maximum value of a message packet, or a minimum value of a message packet; group the temporary application messages according to the quintuple to obtain multiple groups of temporary message flows; determine the message flow from the temporary message flow according to the application identification table; obtain the message flow and the corresponding label according to the jam record table and the message flow.

[0039] In a possible implementation, a fluency assessment device for an application may further include: an acquisition module, which is also used to acquire training data; a clustering module, which is used to cluster the training data to obtain cluster centers, the cluster centers including M jamming class centers and N non-jamming class centers, the jamming class centers are numbered as any value from 1 to M, and the non-jamming class centers are numbered as any value from M+1 to M+N, M is an integer greater than or equal to 1, and N is an integer greater than or equal to 1; a determination module, which is used to determine the fluency assessment result based on the distance from the message features of the calculated message flow to the cluster center; and a calculation module, which is used to calculate the corresponding confidence level based on the fluency assessment result.

[0040] In one possible implementation, the determination module is specifically used to: calculate the distance from the message features of the message flow to the center of each jamming class to obtain a first distance set; calculate the distance from the message features of the message flow to the center of each non-jamming class to obtain a second distance set; combine the first distance set and the second distance set to obtain a third distance set; if the number of the category center corresponding to the minimum distance value in the third distance set is less than or equal to M, the fluency evaluation result is determined to be jamming; if the number of the category center corresponding to the minimum distance value in the third distance set is greater than M, the fluency evaluation result is determined to be non-jamming.

[0041] In one possible implementation, the calculation module is specifically used to: determine the minimum distance value in the third distance set; calculate the average value of the third distance set; determine the minimum distance value in the first distance set; calculate the average value of the first distance set; and calculate the confidence of the fluency assessment result based on the minimum distance value in the third distance set, the average value of the third distance set, the minimum distance value in the first distance set, and the average value of the first distance set.

[0042] In one possible implementation, the calculation module is specifically used to: determine the minimum distance value in the third distance set; calculate the average value of the third distance set; determine the minimum distance value in the second distance set; calculate the average value of the second distance set; and calculate the confidence of the fluency assessment result based on the minimum distance value in the third distance set, the average value of the third distance set, the minimum distance value in the second distance set, and the average value of the second distance set.

[0043] In a possible implementation, the clustering module is specifically configured to cluster the training data using a K-means clustering algorithm or a fuzzy clustering method.

[0044] In a possible implementation, the classifier may be a decision tree model, and the decision tree model is used to analyze the message flow and the label to obtain a fluency evaluation result.

[0045] In a third aspect, the present application provides an application fluency assessment device, the fluency assessment device comprising: a processor, a memory, an input / output device, and a bus; the memory stores computer instructions; when the processor executes the computer instructions in the memory, the memory stores computer instructions; when the processor executes the computer instructions in the memory, the processor is used to implement the implementation method of the first aspect.

[0046] In a fourth aspect, an embodiment of the present application provides a chip system, which includes a processor and an input / output port, wherein the processor is used to implement the processing functions involved in the method described in the first aspect above, and the input / output port is used to implement the transceiver functions involved in the method described in the first aspect above.

[0047] In one possible design, the chip system also includes a memory, which is used to store program instructions and data for implementing the functions involved in the method described in the first aspect above.

[0048] The chip system may be composed of chips, or may include chips and other discrete devices.

[0049] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium. The computer-readable storage medium stores computer instructions; when the computer instructions are executed on a computer, the computer executes the method described in the possible implementation of the first aspect.

[0050] In a sixth aspect, an embodiment of the present application provides a computer program product. The computer program product includes a computer program or instructions, and when the computer program or instructions are executed on a computer, the computer executes the method described in the possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of a system framework provided for this application;

[0052] Figure 2 A schematic diagram of a data acquisition system architecture provided for this application;

[0053] Figure 3 A flowchart of a method for evaluating the fluency of an application provided in this application;

[0054] Figure 4 A schematic diagram of a feature extraction process provided for this application;

[0055] Figure 5 A schematic diagram of training a decision tree model based on a generative flow network provided in this application;

[0056] Figure 6 A flow chart of another method for evaluating fluency of an application provided in the present application;

[0057] Figure 7 A flowchart of a quality difference identification method provided in this application;

[0058] Figure 8 The relationship diagram between the 85-dimensional message features provided in this application and the Pearson correlation coefficient, chi-square and mutual information;

[0059] Fig. 9 A flowchart of another quality difference identification method provided in this application;

[0060] Fig.10A schematic diagram of the structure of a fluency evaluation device for an application provided in the present application;

[0061] Fig.11 A schematic diagram of the structure of a quality difference identification device provided in this application;

[0062] Fig.12 A schematic diagram of the structure of another fluency evaluation device provided by the present application;

[0063] Fig.13 A schematic diagram of the structure of another quality difference identification device provided by the present application;

[0064] Fig.14 A schematic diagram of the structure of a chip provided in this application. DETAILED DESCRIPTION

[0065] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0066] The method provided in this application can be applied to artificial intelligence (AI) scenarios. AI is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robots, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

[0067] First, the overall workflow of the artificial intelligence system is described. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

[0068] (1) Infrastructure

[0069] The infrastructure provides computing power support for the AI ​​system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0070] (2) Data

[0071] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0072] (3) Data processing

[0073] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0074] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0075] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0076] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0077] (4) General ability

[0078] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0079] (5) Smart products and industry applications

[0080] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0081] The embodiments of the present application involve related applications of neural networks. In order to better understand the solutions of the embodiments of the present application, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0082] (1) Neural Network

[0083] A neural network may be composed of neural units. A neural unit may refer to an operation unit with xs and intercept 1 as input. The output of the operation unit may be as shown in formula (1-1):

[0084]

[0085] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0086] (2) Deep Neural Networks

[0087] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple intermediate layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, intermediate layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all intermediate layers, or hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

[0088] Although DNN looks complicated, each layer can be expressed as a linear relationship expression: in, is the input vector, is the output vector, is the offset vector or bias parameter, w is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Taking the coefficient w as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0089] In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as

[0090] It should be noted that the input layer does not have a W parameter. In a deep neural network, more intermediate layers allow the network to better characterize complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (a weight matrix formed by many layers of vectors W).

[0091] (3) Convolutional Neural Network

[0092] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract features is independent of position. Convolution kernels can be formalized as matrices of random size, and convolution kernels can obtain reasonable weights through learning during the training process of convolutional neural networks. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0093] (6) Graph Convolutional Network (GCN)

[0094] Graph neural network is a deep learning model that models and processes non-Euclidean spatial data (such as graph data). Its principle is to use pairwise message passing so that graph nodes iteratively update their corresponding representations by exchanging information with their neighbors.

[0095] GCN is similar to CNN, but the difference is that the input of CNN is usually two-dimensional structured data, while the input of GCN is usually graph structured data. GCN has cleverly designed a method to extract features from graph data, so that these features can be used to perform node classification, graph classification, link prediction, and graph embedding.

[0096] (7) Loss function

[0097] In the process of training deep neural networks, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually a process of optimization before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict lower, and continue to adjust until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible. The loss function can usually include loss functions such as squared error, cross entropy, logarithm, exponential, etc. For example, the squared error can be used as a loss function, defined as The specific loss function can be selected according to the actual application scenario.

[0098] (8) Back propagation algorithm

[0099] An algorithm that calculates the gradient of model parameters based on the loss function and updates the model parameters. The neural network can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0100] (9) Supervised Learning

[0101] It can also be called supervised learning. It is a training method / learning method in machine learning. It refers to the process of training a model by letting the machine learn a large amount of labeled sample data, so that the model can get the corresponding output based on the input.

[0102] Usually, supervised learning requires a large amount of labeled data, that is, data with labels. The more labeled data there is, the better the machine learning algorithm will be, and the more capable it will be of selecting the correct answer from the input data. It can be understood that supervised learning is about finding the relationship between input and output. For example, input an image or text, and output classification features or segmentation features.

[0103] (10) Unsupervised Learning

[0104] Unsupervised learning is a training method / learning method in machine learning. It can be understood as letting the model learn by itself, with its input being samples without labels, and learning the relationship between input data based on clustering or other methods.

[0105] Unsupervised learning is often used in data mining to learn the relationship between data in a large amount of unlabeled data. The training data is unlabeled, and the training goal is to classify or distinguish the observed values. For example, unsupervised learning should be able to distinguish "cat" pictures from a large number of various pictures based on the characteristics of all "cat" pictures without any additional prompts.

[0106] Compared with supervised learning, the input data in the training process of unsupervised learning does not need to carry labels, can be applied to more complex tasks, and can be used to learn deeper relationships between data.

[0107] (11) Generate flow network

[0108] Generative Flow Network (GFlowNet) is a new probabilistic machine learning framework at the same level as reinforcement learning, which can be applied to the modeling, hypothesis generation, and experimental design stages of the experimental science cycle. It is a generative model for sampling high-diversity candidate objects from a given reward function. The theory of generative flow networks is based on thinking about unnormalized probabilities, where the flow through the path describes how to construct a problem or object x. All paths start from a root node and end at a synchronization node, but have different probabilities.

[0109] (12) Decision Tree Model

[0110] A decision tree model is a prediction model used to represent the mapping relationship between object attributes and object values. This model is usually used for classification problems, but can also be used for regression problems. The decision tree model selects discriminative variables from the data, splits the data set into two or more subsets, and finally forms a "tree" through step-by-step splitting. Each leaf node represents the final prediction result of the branch.

[0111] (13) Pearson correlation coefficient

[0112] The Pearson correlation coefficient is a statistic that measures the strength of the linear relationship between two variables, and its value is between -1 and 1. It calculates the covariance and standard deviation of the two variables to obtain the correlation coefficient, reflecting the degree of correlation between the two variables. If the correlation coefficient is close to 1, it means that there is a completely positive linear relationship; if it is close to -1, it means that there is a completely negative linear relationship; if it is close to 0, it means that there is no linear relationship. The calculation formula of the Pearson correlation coefficient is: r = cov(X,Y) / (σX*σY), where cov(X,Y) represents the covariance of variables X and Y, and σX and σY represent the standard deviations of variables X and Y, respectively.

[0113] The Pearson correlation coefficient is widely used in many fields, such as statistics, machine learning, and data mining. It can help us understand the strength of the linear relationship between variables, perform data analysis and modeling, etc.

[0114] (14) Mutual Information

[0115] Mutual information is a useful information measure in information theory, usually referring to the correlation between two sets of events. It can be seen as the amount of information contained in one random variable about another random variable, or the uncertainty of a random variable reduced by knowing another random variable.

[0116] For discrete random variables X, Y, the calculation formula of mutual information is I(X,Y)=∑ y∈Y ∑ x∈X p(x,y)log(p(x,y) / p(x)p(y)), the larger the value of I(X,Y), the closer the relationship between X and Y, so mutual information can be used for feature screening.

[0117] (15) Chi-square distribution

[0118] Chi-square distribution is a probability distribution. If there are n independent random variables x1, x2, …, x n , all obey the standard normal distribution (also called independent and identically distributed in the standard normal distribution), then the sum of the squares of these n random variables obeying the standard normal distribution is It forms a new random variable whose distribution law is called chi-square distribution.

[0119] (16) K-means clustering

[0120] K-means clustering is a dynamic clustering method. First, k original images are randomly selected from the original image set as k classes, and then the remaining images are analyzed one by one, the distance between each image and the k classes is calculated, the image is classified into the class closest to it, and the class average map of the class is recalculated, and so on until the remaining nk images are analyzed; then, the above calculation and classification are repeated for the n images in the set one by one, and this step is repeated continuously until the k class average maps converge, thereby obtaining k classifications for the original image set.

[0121] (17) Fuzzy c-means clustering

[0122] Fuzzy C-means clustering is a fuzzy clustering algorithm, also known as fuzzy c-means or FCM. The algorithm obtains the membership of each sample point to all class centers by optimizing the objective function, thereby determining the class of the sample point. Fuzzy C-means clustering algorithm has a wide range of applications in many fields, such as image processing, pattern recognition, data mining, etc.

[0123] The following introduces the system framework provided by the embodiments of the present application.

[0124] See also Figure 1 The embodiment of the present application provides a system framework, in which a database 101 is used to store data collected by a data acquisition system, a feature extraction device 102 performs feature extraction based on the data stored in the database to obtain corresponding features, and a model 103 is trained based on the features. The model 103 may be a classifier or a decision tree model mentioned in the present application.

[0125] The specific description of the training and updating process of model 103 is detailed in the training stage below. It should be noted that in actual applications, the data in database 101 may not all come from the data acquisition system, but may also be received from other devices. It should also be noted that model 103 may not necessarily be trained based on the data in database 101, but may also obtain data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0126] The system framework provided in the embodiment of the present application can be applied to model training. For the pre-trained model 103, the confidence corresponding to the output result of the model 103 can be obtained through the confidence evaluator 104, and the parameters of the model 103 are updated according to the confidence to obtain the updated model 103; the system architecture can also be applied to the reasoning stage. For the results output by the model 103 and the confidence evaluator 104 and the corresponding confidence, the quality of the reasoning result can be judged by a pre-set threshold, and the reasoning results with confidence lower than the threshold can be discarded, thereby improving the accuracy of the reasoning, and the reasoning results can be statistically analyzed based on the confidence, and finally used to analyze the root cause when the reasoning accuracy of the model 103 is reduced.

[0127] The following is an introduction to the data acquisition system in combination with the aforementioned system framework.

[0128] See also Figure 2 The data collection system mainly includes three parts: data collection and marking system, packet capture server and application identification server. The optical line terminal (OLT) is an access device, which is connected to the FTTR to provide users with an Internet environment. The uplink message data is sent to the server and mirrored to the packet capture server to provide application message capture function. The test equipment (such as mobile phones, tablets and other terminal devices) connects to the WiFi issued by the FTTR to perform application activities. The data collection and marking system connects to the test equipment to monitor the application activities of the equipment, identifies the application jams and issues packet capture commands. The jam record table will be generated through the data collection and marking system to record the start time of the application's mainstream activities (such as watching videos or holding meetings, etc.), the jams of the application per second, and the time when the packet starts to capture. After receiving the packet capture command, the packet capture server will capture the OLT mirror corresponding to the uplink port traffic to obtain the message during the application operation, and the application identification server will identify the application message and make a detailed distinction between the application category and activity category of each message flow.

[0129] The following is an introduction to the method flow provided by this application in conjunction with the aforementioned system framework.

[0130] The present application provides a method for evaluating the fluency of an application. It can be understood that the method for evaluating the fluency of an application provided by the present application can include a training phase and a reasoning phase. The training phase can be referred to as a method for evaluating the fluency of an application, and the reasoning phase can be referred to as a process of a method for identifying poor quality.

[0131] 1. Training Phase

[0132] See also Figure 3 , a flow chart of a method for evaluating the fluency of an application provided in the present application is as follows.

[0133] 301. Obtain multiple groups of message flows and corresponding labels;

[0134] Before obtaining multiple groups of message flows and corresponding labels, you can first obtain the initial application message, the jam identification table and the application identification table. The initial application message is the message of the application during its running process. The jam identification table records the fluency of the application, which can be represented by jamming or not jamming (1 / 0), and the application identification table records the application category and activity category of each message flow.

[0135] Optionally, after obtaining the initial application message, the initial application message can be preprocessed according to the jam identification table and the application identification table to obtain multiple groups of message flows and corresponding labels.

[0136] Among them, the initial application message is a complete transmission message, in which the valid and available message fields only occupy a small part of the message. Therefore, a temporary application message can be obtained by extracting the valid feature fields of the initial application message. The temporary application message only includes valid feature fields, and the valid feature fields can be the message flow rate, the average message flow rate, the average message packet size, the maximum value of the message packet, or the minimum value of the message packet.

[0137] Optionally, after obtaining the temporary application messages, the temporary application messages may be grouped according to the five-tuple to obtain multiple groups of temporary message flows.

[0138] Optionally, since the captured initial application message is not a pure message but is mixed with messages of other applications, and the application identification table can distinguish the application category and activity category of each message flow, the message flow can be determined from the temporary message flow through the application identification table to eliminate the interference of miscellaneous flows.

[0139] In addition, the jam record table not only records the fluency of the application per second, but also records the start time of the application's mainstream activities (such as watching videos and holding meetings) and the time when the packet capture starts. Therefore, the fluency and the packet flow can be matched one by one according to the jam record table, that is, the label corresponding to the packet flow can be obtained.

[0140] 302. Analyze the message flow and the label through the classifier to obtain a fluency evaluation result;

[0141] After obtaining the message flow and the corresponding label, the message flow and the label can be analyzed by a classifier to obtain a fluency evaluation result of the application, which can be either a stuck or no stuck situation.

[0142] Exemplarily, the classifier may be a decision tree model, and the decision tree model may be used to analyze the message flow and the label to obtain an application fluency evaluation result.

[0143] Optionally, before obtaining the confidence distribution, a sliding window algorithm can be used to extract features from the message flow to obtain message features, which include 13 basic message features and 72 augmented message features, that is, 85-dimensional message features are obtained by extracting features from the message flow. The basic message features may include 13 basic message statistics, including flow speed, average packet size (pkt size avg), maximum flow interval (flow interval max), L1 norm of packet size (l1 norm), maximum burst size (max burst size) i.e. the number of messages, packet speed (pkt speed), flow mean deviation (flow mean dev), maximum packet size (pkt size max), minimum packet size (pkt size min), maximum packet duration (max pkt durtime), minimum flow interval (flow interval min), average flow interval (flow interval avg) and flow mean difference (flow diff mean).

[0144] Among them, the augmented message features are obtained by augmenting the basic message features, which are mainly divided into three categories: message time dimension expansion, IP header length expansion and message packet length expansion. Then, 72 augmented message features are expanded from these three categories. The specific augmented message features are shown in Table 1, where IAT is the abbreviation of Interarrival time, Q1 and Q3 are the first and third digits of the quartile from 1 to 3, Var is the L1 norm, len is the abbreviation of length, and pkt is the abbreviation of data packet.

[0145] Table 1

[0146]

[0147] Among them, the process of using sliding window algorithm to extract features from message flow is as follows: Figure 4 As shown, the threshold of the sliding window can be set to 200ms. When the data value in the window is greater than 200ms, the message features in the window are calculated. The sliding window takes the form of closed front and open back. When the last sliding window cannot meet the window threshold, this part of the features can be discarded, or the message features in the window can be directly calculated. The specific details are not limited here.

[0148] 303. Obtain confidence distribution corresponding to the fluency evaluation result;

[0149] After obtaining the fluency evaluation result of the message flow, the confidence distribution corresponding to the fluency evaluation result can be obtained by generating a flow network.

[0150] Optionally, after extracting features from the message flow, the obtained 85-dimensional message features may be input into a generation flow network, through which the confidence distribution corresponding to the fluency evaluation result may be obtained.

[0151] 304. Update the parameters of the classifier according to the confidence distribution and the preset expected value to obtain an updated classifier.

[0152] After obtaining the confidence distribution corresponding to the fluency evaluation result, the parameters of the classifier can be updated according to the preset relationship between the confidence and the preset expected value to obtain an updated classifier.

[0153] Specifically, the fluency evaluation results with confidence lower than the preset expected value can be discarded, and the fluency evaluation results with confidence higher than the preset expected value can be retained to obtain an updated classifier. The classifier can be a decision tree model, so the pruning operation of the decision tree model can be combined to prune the leaf nodes corresponding to the confidence lower than the preset expected value to obtain an updated decision tree model.

[0154] like Figure 5 The figure shows the process of training the decision tree model based on the generative flow network. The 85-dimensional features at time t are used as the input of the decision tree model and the generative flow network. The confidence distribution of the final state, that is, the confidence of the fluency assessment result, is gradually obtained through the generative flow network.

[0155] See also Figure 6 , a flow chart of another application fluency assessment method provided in the present application is as follows.

[0156] 601. Obtain a jam record table, an initial application message, and an application identification table;

[0157] The obtained message flow data may be obtained by collecting data through a data collection system, so firstly, a jam record table, an initial application message, and an application identification table may be obtained through the data collection system.

[0158] Specifically, the jam record table records the start time of the application's mainstream activities, such as watching videos, holding meetings, the fluency of the application, that is, jamming or not (1 / 0), and the start time of the packet capture server's packet capture. The jam record table is obtained by the data acquisition and marking system of the data acquisition system; the initial application message is the message during the application operation, that is, the message from the opening of the application to the end of the mainstream activity, which can be obtained by the packet capture server capturing the upstream port traffic corresponding to the optical line terminal OLT mirror; the application identification table can be the identification result obtained by the application identification server for identifying the initial application message. The table distinguishes in detail the application category and activity category of the message flow, and the required message flow can be screened out through the application identification table.

[0159] Among them, the time recorded in the jam record table and the time of the message flow are both counted in the time stamp interval, so the label corresponding to the message flow can be obtained through the jam record table and the message flow.

[0160] 602. Preprocess the initial application message according to the jam record table and the application identification table to obtain a message flow and a corresponding label;

[0161] Since the packet capture server mixes messages of other applications when capturing messages, and the initial application message is a complete application message, its effective feature field only occupies a part of the message, so in order to improve the processing efficiency of message data, the initial application message can be preprocessed.

[0162] Optionally, effective feature fields may be extracted from the initial application message, where the effective feature fields may be message flow rate, message flow rate average, message packet size average, message packet maximum value or message packet minimum value, to obtain a temporary application message.

[0163] Optionally, after obtaining the temporary application message, the temporary application message can be grouped according to the quintuple, which includes the source IP address, source port, destination IP address, destination port and transport layer protocol. Temporary application messages with the same quintuple are grouped into the same group to obtain multiple groups of temporary message streams.

[0164] Optionally, the required message flows can be filtered out from multiple groups of temporary message flows according to the application identification table to obtain multiple groups of message flows, and then the labels corresponding to the message flows can be obtained according to the jam record table.

[0165] 603. Extract features from the message flow to obtain message features;

[0166] To extract features from message flows, a sliding window algorithm may be used to obtain 85-dimensional message features of the message flows. When extracting features, multiple groups of message flows may be used in parallel and in multiple processes to perform calculations simultaneously.

[0167] 604. Obtain a generated flow network according to the message characteristics;

[0168] After obtaining the 85-dimensional features of the message flow, a generative flow network can be generated based on the 85-dimensional features. According to the characteristics of the generative flow network, that is, the input flow and output flow of each node in the generative flow network are equal, the probability distribution of each node can be clearly known. Based on the generative flow network, the confidence distribution of each application fluency evaluation result can be more clearly understood.

[0169] 605. Analyze the message features and labels through the classifier to obtain a fluency evaluation result;

[0170] After obtaining the message features of the message flow, the message features and the labels can be analyzed by a classifier in combination with corresponding labels to obtain a fluency evaluation result of the message flow.

[0171] 606. Analyze the message features by generating a flow network to obtain a confidence distribution corresponding to the fluency evaluation result;

[0172] Among them, the generated flow network is a pre-trained network that can analyze the message features and output the confidence distribution corresponding to the fluency evaluation result, so that the classifier can be updated according to the confidence later.

[0173] 607. Update the parameters of the classifier according to the confidence distribution and the preset expected value to obtain an updated classifier.

[0174] Step 607 in the embodiment of the present application is similar to the aforementioned Figure 3 The process is similar to step 303 and will not be described in detail here.

[0175] In an implementation manner of the present application, feature extraction is performed on the acquired message flow to obtain message features, a flow network is generated based on the message features, a confidence distribution corresponding to the fluency assessment result is obtained based on the generated flow network, and a classifier is updated based on the confidence distribution and a preset expected value. The classifier can be tuned and updated based on the confidence distribution of the output result of the classifier, which shortens the training time of the optimized update model, and the updated classifier retains the output results with high confidence, so that the accuracy of the detection results of the updated classifier is improved.

[0176] 2. Reasoning Stage

[0177] In the aforementioned training phase, the classifier is trained and updated. The updated classifier can be applied to poor quality identification (also known as application fluency assessment). There are many ways to calculate the confidence of poor quality identification and the corresponding poor quality identification results. The poor quality identification results can be obtained according to the classifier, and then the corresponding confidence can be obtained according to the generated flow network. The training data can also be used to cluster the training data in different ways to obtain multiple cluster centers. By calculating the distance from the 85-dimensional message feature to the cluster center, the poor quality identification result is determined and the confidence of the poor quality identification result is calculated using different calculation methods.

[0178] (i) Analyze the quality difference recognition results based on the classifier and obtain the corresponding confidence based on the generated flow network.

[0179] See also Figure 7 , a flow chart of a quality difference identification method provided in this application is as follows.

[0180] 701. Obtain message flow;

[0181] The process of acquiring the message flow is the same as the aforementioned process of acquiring multiple groups of message flows and acquiring the message flows from the corresponding labels, and will not be repeated here.

[0182] 702. Extract features from the message flow to obtain message features.

[0183] Optionally, after acquiring the message flow, a sliding window algorithm may be used to extract features from the message flow to obtain 85-dimensional message features.

[0184] 703. Using the message features as input of the updated classifier to obtain a quality difference identification result;

[0185] Optionally, the 85-dimensional message feature is used as an input of an updated classifier to obtain a quality difference identification result, that is, whether there is a freeze or not.

[0186] 704. Using the message features as input to generate a flow network, and obtaining the confidence level corresponding to the quality difference identification result.

[0187] The generation flow network can obtain corresponding confidence according to the input 85-dimensional message features, and the generation flow network can be trained by data of the sample set.

[0188] Specifically, the message features of the sample data in the sample set can be obtained, and the sum of squares of the message features, the Pearson correlation coefficient and the mutual information between the message features and the quality difference identification results of the data samples, i.e., whether they are stuck or not, can be calculated. The confidence can be calculated according to the confidence calculation formula related to the Pearson correlation coefficient, mutual information and the sum of squares. The obtained confidence and the corresponding message features are used to train an untrained generative flow network, and finally a generative flow network is obtained. The generative flow network can obtain the corresponding confidence by inputting message features.

[0189] According to the Pearson correlation coefficient, mutual information and the confidence calculation formula of the sum of squares, it can be K is any value from 1 to 85, X topK is the first K message feature subset of sample set X, is an offline training dataset, including 85-dimensional message features, and can also be The specific information is not limited here. The Pearson correlation coefficient, chi-square and mutual information obtained based on the 85-dimensional message characteristics of the message flow are as follows: Figure 8 shown.

[0190] In addition, the accuracy of the quality difference identification result can also be used as the confidence of the quality difference identification result. Specifically, the accuracy can be determined based on the historical quality difference identification results output by the updated classifier, and the accuracy can be used as the confidence of the quality difference identification result. For example, in the historical quality difference identification results, there are a total of ten output results, eight of which are stuck and two are not stuck. The accuracy of the output being stuck is 80%, and the confidence that the output result is stuck is also 80%.

[0191] In an embodiment of the present application, the updated classifier can identify poor quality of the message flow, and can analyze the message features of the message flow according to the generated flow network to obtain the corresponding confidence, thereby providing a measurable confidence result for the poor quality identification result.

[0192] (ii) Clustering the training data, calculating the distance between the message feature and the cluster center to determine the quality difference identification result, and calculating its corresponding confidence.

[0193] See also Fig. 9 , a flow chart of another quality difference identification method provided by the present application is as follows.

[0194] 901. Obtain training data;

[0195] Offline training data is obtained. The offline training data may be stored in a data acquisition system or in a memory, which is not specifically limited here.

[0196] 902. Cluster the training data to obtain cluster centers;

[0197] Optionally, the acquired offline training data may be clustered using a K-means clustering algorithm or a fuzzy c-means clustering algorithm, which is not specifically limited here.

[0198] Specifically, by clustering the offline training data, multiple clustering centers, namely M jamming class centers and N non-jamming class centers, can be obtained and numbered. The M jamming class centers can be numbered 1, 2, ..., M, and the N non-jamming class centers can be numbered M+1, M+1, ..., M+N, where M and N are both positive integers greater than or equal to 1.

[0199] 903. Determine the quality difference identification result of the message flow according to the distance from the message feature of the message flow to the cluster center;

[0200] Specifically, calculate the distances A1, A2, ..., Am of the message features of the message flow from the M jamming class centers and the distances B1, B2, ..., Bn from the N non-jamming class centers, and determine the class center number corresponding to the smallest distance from all class centers. If the number is less than or equal to M, it is predicted to be jamming, otherwise it is predicted to be non-jamming.

[0201] 904. Calculate corresponding confidence levels according to the quality difference identification results of the message flow.

[0202] After the quality difference identification result of the message flow is determined, the confidence level is calculated according to the quality difference identification result.

[0203] When the quality identification result of the message flow is jamming, the calculation formula of its confidence can be X AB is the minimum distance between the packet feature of the packet flow and the M jamming centers and the N non-jamming centers, is the average distance between the packet feature of the packet flow and the M jam centers and the N non-jamming centers, X A is the minimum distance between the message feature of the message flow and the M jam centers, is the average distance between the message features of the message flow and the M jam centers.

[0204] When the quality identification result of the message flow is no jamming, the calculation formula of its confidence can be X B is the minimum distance between the message feature of the message flow and the M jam centers, is the average distance between the message features of the message flow and the M jam centers.

[0205] For example, offline training data is used and divided into long freeze and short freeze categories. Fig. 9 The confidence calculation method proposed in shows that the confidence of long-stuttering data is lower than that of short-stuttering data, and the detection of short-stuttering data is more accurate.

[0206] In an embodiment of the present application, offline training data can be clustered to obtain multiple cluster centers, and the quality difference identification result can be determined by calculating the distance between the message features of the message flow and the cluster centers. Different confidence calculation methods are proposed based on different quality difference identification results, and measurable confidence results are also provided when the quality difference identification results are obtained.

[0207] The above is an introduction to the method flow provided by the present application. Based on the above method flow, the following is an introduction to the device provided by the present application.

[0208] See also Fig.10 , a schematic diagram of a structure of a fluency evaluation device for an application provided in the present application, comprising:

[0209] The acquisition module 1001 is used to acquire multiple groups of message flows and corresponding labels, where the message flows are message flows input to the application, and the labels are used to indicate the fluency of the application when processing the corresponding message flows;

[0210] An analysis module 1002 is used to analyze the message flow and the label through a classifier to obtain a fluency evaluation result;

[0211] The acquisition module 1001 is also used to obtain the confidence distribution corresponding to the fluency evaluation result;

[0212] The updating module 1003 is used to update the parameters of the classifier according to the confidence distribution and the preset expected value to obtain an updated classifier, and the updated classifier is used to evaluate the fluency of the message flow subsequently input into the application.

[0213] In one possible implementation, the analysis module 1002 is specifically used to: extract features from the message flow to obtain message features, where the message features include basic message features and augmented message features, where the augmented message features are obtained by augmenting the basic message features; and analyze the message features and labels through a classifier to obtain the fluency evaluation result.

[0214] In a possible implementation, the acquisition module 1001 is specifically configured to: analyze message features and labels by generating a flow network to obtain a confidence distribution corresponding to a fluency evaluation result.

[0215] In a possible implementation, the updating module 1003 is specifically configured to remove the fluency evaluation result if the confidence level of the fluency evaluation result is lower than a preset expected value, so as to obtain an updated classifier.

[0216] In a possible implementation, the acquisition module 1001 is specifically used to: obtain a jam record table, an initial application message, and an application identification table, the jam record table is used to record the fluency of the application, and the application identification table includes the application category and activity category of each message flow; according to the jam record table and the application identification table, the initial application message is preprocessed to obtain the message flow and the corresponding label.

[0217] In one possible implementation, the acquisition module 1001 is specifically used to: extract the valid feature fields of the initial application message to obtain a temporary application message, the valid feature fields including at least one of a message flow rate, an average message flow rate, an average message packet size, a maximum value of a message packet, or a minimum value of a message packet; group the temporary application messages according to the five-tuple to obtain multiple groups of temporary message flows; determine the message flow from the temporary message flow according to the application identification table; obtain the message flow and the corresponding label according to the jam record table and the message flow.

[0218] In a possible implementation, the analysis module 1002 is specifically configured to extract features from the message flow using a sliding window algorithm to obtain message features.

[0219] In a possible implementation, the output of the updated classifier is used for poor quality identification.

[0220] In a possible implementation, the classifier is a decision tree model, and the decision tree model is used to analyze the message flow and the label to obtain a fluency evaluation result.

[0221] See also Fig.11 , a schematic diagram of the structure of a quality difference identification device provided in the present application includes:

[0222] The acquisition module 1101 is used to acquire the message features of the message flow, where the message features are obtained by extracting features from the message flow, and the message features include basic message features and augmented message features, where the augmented message features are obtained by augmenting the basic message features;

[0223] Output module 1102 is used to use message features as input to the updated classifier and generate flow network to obtain quality difference recognition results and confidence levels corresponding to the quality difference recognition results. The quality difference recognition results are obtained by the updated classifier, and the confidence levels are obtained by generating flow network.

[0224] In a possible implementation, the device also includes: an acquisition module 1101, which is also used to acquire message features of sample data; a calculation module 1103, which is used to calculate the sum of squares of message features of sample data, the Pearson correlation coefficient and mutual information between the message features of sample data and the quality difference identification results of sample data; the calculation module 1103 is also used to calculate the confidence corresponding to the quality difference identification results of sample data based on the Pearson correlation coefficient, mutual information and sum of squares; a training module 1104, which is used to train an untrained generative flow network based on the message features of sample data and the confidence corresponding to the quality difference identification results of sample data to obtain a generative flow network.

[0225] In a possible implementation, the device also includes: an acquisition module 1101, which is also used to acquire training data; a clustering module 1105, which is used to cluster the training data to obtain cluster centers, wherein the cluster centers include M jamming class centers and N non-jamming class centers, the jamming class centers are numbered as any value from 1 to M, and the non-jamming class centers are numbered as any value from M+1 to M+N, M is an integer greater than or equal to 1, and N is an integer greater than or equal to 1; a determination module 1106, which is used to determine the quality difference identification result of the message flow based on the distance from the message features of the message flow to the cluster center; a calculation module 1103, which is used to calculate the corresponding confidence level based on the quality difference identification result of the message flow.

[0226] In one possible implementation, the determination module 1106 is specifically used to: calculate the distance from the message feature of the message flow to the center of each jamming class to obtain a first distance set; calculate the distance from the message feature of the message flow to the center of each non-jamming class to obtain a second distance set; combine the first distance set and the second distance set to obtain a third distance set; if the number of the category center corresponding to the minimum distance value in the third distance set is less than or equal to M, the quality difference identification result of the message flow is determined to be jamming; if the number of the category center corresponding to the minimum distance value in the third distance set is greater than M, the quality difference identification result of the message flow is determined to be non-jamming.

[0227] In one possible implementation, the calculation module 1103 is specifically used to: determine the minimum distance value in the third distance set; calculate the average value of the third distance set; determine the minimum distance value in the first distance set; calculate the average value of the first distance set; and calculate the confidence of the quality difference identification result of the message flow based on the minimum distance value in the third distance set, the average value of the third distance set, the minimum distance value in the first distance set, and the average value of the first distance set.

[0228] In one possible implementation, the calculation module 1103 is specifically used to: determine the minimum distance value in the third distance set; calculate the average value of the third distance set; determine the minimum distance value in the second distance set; calculate the average value of the second distance set; calculate the confidence of the quality difference identification result of the message flow based on the minimum distance value in the third distance set, the average value of the third distance set, the minimum distance value in the second distance set and the average value of the second distance set.

[0229] In a possible implementation, the clustering module 1105 is specifically configured to cluster the training data using a K-means clustering algorithm or a fuzzy clustering method.

[0230] In a possible implementation, the updated classifier is obtained by training through an applied fluency assessment device.

[0231] In a possible implementation, the output of the updated classifier is used for poor quality identification.

[0232] See also Fig.12 , a structural schematic diagram of another application fluency assessment device provided by the present application is described as follows.

[0233] The device for evaluating the fluency of an application may include a processor 1201 and a memory 1202. The processor 1201 and the memory 1202 are interconnected via a line. The memory 1202 stores program instructions and data.

[0234] The memory 1202 stores the aforementioned Figure 3 and Figure 6 The program instructions and data corresponding to the steps in .

[0235] Processor 1201 is used to execute the above Figure 3 and Figure 6 The method steps are performed by the application fluency assessment device shown in any embodiment.

[0236] Optionally, the application fluency assessment device may further include a transceiver 1203 for receiving or sending data.

[0237] The present application also provides a computer-readable storage medium in which a program is stored. When the program is run on a computer, the computer executes the above-mentioned Figure 3 and Figure 6 The illustrated embodiments describe steps in a method.

[0238] Optionally, the aforementioned Fig.12 The device for evaluating the fluency of the application shown in FIG. 1 is a chip.

[0239] The embodiment of the present application also provides an application fluency evaluation device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface. The program instructions are executed by the processing unit. The processing unit is used to execute the aforementioned Figure 3 and Figure 6 The method steps are performed by the application fluency assessment device shown in any embodiment.

[0240] See also Fig.13 , a structural schematic diagram of another quality difference identification device provided in this application is described as follows.

[0241] The quality difference identification device may include a processor 1301 and a memory 1302. The processor 1301 and the memory 1302 are interconnected via a line. The memory 1302 stores program instructions and data.

[0242] The memory 1302 stores the aforementioned Figure 7 and Fig. 9 The program instructions and data corresponding to the steps in .

[0243] Processor 1301 is used to execute the above Figure 7 and Fig. 9 The method steps performed by the quality difference identification device shown.

[0244] Optionally, the quality difference identification device may further include a transceiver 1303 for receiving or sending data.

[0245] The present application also provides a computer-readable storage medium in which a program is stored. When the program is run on a computer, the computer executes the above-mentioned Figure 7 and Fig. 9 The illustrated embodiments describe steps in a method.

[0246] Optionally, the aforementioned Fig.13 The poor quality identification device shown in is a chip.

[0247] The embodiment of the present application also provides a quality difference identification device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface. The program instructions are executed by the processing unit. The processing unit is used to execute the aforementioned Figure 7 and Fig. 9 The method steps performed by the quality difference identification device shown in the embodiment.

[0248] The embodiment of the present application also provides a digital processing chip. The digital processing chip integrates a circuit and one or more interfaces for implementing the functions of the above-mentioned processor 1201, processor 1301, or processor 1201, processor 1301. When a memory is integrated in the digital processing chip, the digital processing chip can complete the method steps of any one or more of the above-mentioned embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory through a communication interface. The digital processing chip implements the actions performed by the fluency evaluation device applied in the above-mentioned embodiment according to the program code stored in the external memory.

[0249] The present application also provides a computer program product which, when executed on a computer, enables the computer to execute the above Figure 3 , Figure 6 , Figure 7 or Fig. 9 The illustrated embodiments describe method steps.

[0250] The fluency evaluation device or poor quality identification device of the application provided in the embodiment of the present application can be a chip, and the chip includes: a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit. The processing unit can execute the computer execution instructions stored in the storage unit to enable the chip in the server to execute the above Figure 3 , Figure 6 , Figure 7 or Fig. 9 Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0251] Specifically, the aforementioned processing unit or processor may be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0252] For example, see Fig.14 , Fig.14 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 1400, NPU 1400 is mounted on the host CPU (HostCPU) as a coprocessor, and the Host CPU assigns tasks. The core part of the NPU is the operation circuit 1403, which is controlled by the controller 1404 to extract matrix data from the memory and perform multiplication operations.

[0253] In some implementations, the operation circuit 1403 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 1403 is a two-dimensional systolic array. The operation circuit 1403 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1403 is a general-purpose matrix processor.

[0254] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 1402 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1401 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 1408.

[0255] The unified memory 1406 is used to store input data and output data. The weight data is directly transferred to the weight memory 1402 through the direct memory access controller (DMAC) 1405. The input data is also transferred to the unified memory 1406 through the DMAC.

[0256] The bus interface unit (BIU) 1410 is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 1409 .

[0257] The bus interface unit 1410 (BIU) is used for the instruction fetch memory 1409 to obtain instructions from the external memory, and is also used for the storage unit access controller 1405 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0258] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1406 or to transfer weight data to the weight memory 1402 or to transfer input data to the input memory 1401.

[0259] The vector calculation unit 1407 includes multiple operation processing units, and further processes the output of the operation circuit when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, upsampling of feature planes, etc.

[0260] In some implementations, the vector calculation unit 1407 can store the processed output vector to the unified memory 1406. For example, the vector calculation unit 1407 can apply a linear function and / or a nonlinear function to the output of the operation circuit 1403, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 1407 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1403, for example, for use in a subsequent layer in a neural network.

[0261] An instruction fetch buffer 1409 connected to the controller 1404 is used to store instructions used by the controller 1404;

[0262] Unified memory 1406, input memory 1401, weight memory 1402 and instruction fetch memory 1409 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0263] The processor mentioned in any of the above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the above Figure 3 , Figure 6 , Figure 7 or Fig. 9 The method of procedure.

[0264] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0265] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0266] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0267] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0268] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0269] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

[0270] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0271] Finally, it should be noted that the above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A method for evaluating the fluency of an application, characterized in that: include: Acquire multiple groups of message flows and corresponding labels, where the message flows are message flows input to an application, and the labels are used to indicate the fluency of the application in processing the corresponding message flows; Analyzing the message flow and the label by a classifier to obtain a fluency evaluation result; Obtaining a confidence distribution corresponding to the fluency assessment result; The parameters of the classifier are updated according to the confidence distribution and the preset expected value to obtain an updated classifier, and the updated classifier is used to evaluate the fluency of the message flow subsequently input into the application.

2. The method according to claim 1, characterized in that The analyzing the message flow and the label by the classifier to obtain a fluency evaluation result includes: Extracting features from the message flow to obtain message features, wherein the message features include basic message features and augmented message features, and the augmented message features are obtained by augmenting the basic message features; The message features and the labels are analyzed by the classifier to obtain the fluency evaluation result.

3. The method according to claim 2, characterized in that The extracting features from the message flow to obtain message features includes: A sliding window algorithm is used to extract features from the message flow to obtain the message features.

4. The method according to any one of claims 1 to 3, characterized in that The obtaining of the confidence distribution corresponding to the fluency evaluation result includes: The message features and the labels are analyzed by generating a flow network to obtain a confidence distribution corresponding to the fluency evaluation result.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Obtain message features of sample data; Calculating the sum of squares of the message features of the sample data, the Pearson correlation coefficient and the mutual information between the message features of the sample data and the fluency evaluation results of the sample data; Calculating the confidence level corresponding to the fluency evaluation result of the sample data according to the Pearson correlation coefficient, the mutual information and the sum of squares; An untrained generative flow network is trained according to the message features of the sample data and the confidence corresponding to the fluency evaluation result of the sample data to obtain the generative flow network.

6. The method according to any one of claims 1 to 5, characterized in that The updating of the parameters of the classifier according to the confidence distribution and the preset expected value to obtain an updated classifier includes: If there is a fluency evaluation result whose confidence is lower than the preset expected value, the fluency evaluation result is removed to obtain the updated classifier.

7. The method according to any one of claims 1 to 6, characterized in that The obtaining of multiple groups of message flows and corresponding labels includes: Obtain a jam record table, an initial application message, and an application identification table, wherein the jam record table is used to record the fluency of the application, and the application identification table includes an application category and an activity category of each message flow; The initial application message is preprocessed according to the jam record table and the application identification table to obtain the message flow and the corresponding label.

8. The method according to claim 7, characterized in that The preprocessing of the initial application message according to the jam record table and the application identification table to obtain the message flow and the corresponding label includes: Extracting a valid characteristic field of the initial application message to obtain the temporary application message, wherein the valid characteristic field includes at least one of a message flow rate, an average message flow rate, an average message packet size, a maximum value of a message packet, or a minimum value of a message packet; Grouping the temporary application messages according to the five-tuple to obtain multiple groups of temporary message flows; Determining the message flow from the temporary message flow according to the application identification table; According to the jam record table and the message flow, the message flow and the corresponding label are obtained.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: Get training data; Clustering the training data to obtain cluster centers, the cluster centers including M jamming centers and N non-jamming centers, the jamming centers are numbered as any value from 1 to M, the non-jamming centers are numbered as any value from M+1 to M+N, M is an integer greater than or equal to 1, and N is an integer greater than or equal to 1; Determine the fluency evaluation result according to calculating the distance from the message feature of the message flow to the cluster center; The corresponding confidence level is calculated according to the fluency evaluation result.

10. The method according to claim 9, characterized in that The determining the fluency evaluation result according to calculating the distance from the message feature of the message flow to the cluster center includes: Calculate the distance from the message feature of the message flow to each of the jam class centers to obtain a first distance set; Calculate the distance from the message feature of the message flow to the center of each non-stuttering class to obtain a second distance set; Combining the first distance set and the second distance set to obtain a third distance set; If the number of the category center corresponding to the minimum distance value in the third distance set is less than or equal to M, the fluency evaluation result is determined to be a jam; If the number of the category center corresponding to the minimum distance value in the third distance set is greater than M, the fluency evaluation result is determined to be no lag.

11. The method according to claim 9, characterized in that If the fluency evaluation result is a freeze, the step of calculating the corresponding confidence level according to the fluency evaluation result includes: Determining a minimum distance value in the third distance set; Calculating an average value of the third distance set; Determining a minimum distance value in the first distance set; Calculating an average value of the first distance set; The confidence of the fluency evaluation result is calculated according to the minimum distance value in the third distance set, the average value of the third distance set, the minimum distance value in the first distance set, and the average value of the first distance set.

12. The method according to claim 9, characterized in that If the fluency evaluation result is no lag, then calculating the corresponding confidence level according to the fluency evaluation result includes: Determining a minimum distance value in the third distance set; Calculating an average value of the third distance set; Determining a minimum distance value in the second distance set; Calculating an average value of the second distance set; The confidence of the fluency evaluation result is calculated according to the minimum distance value in the third distance set, the average value of the third distance set, the minimum distance value in the second distance set, and the average value of the second distance set.

13. The method according to any one of claims 9 to 12, characterized in that The clustering of the training data comprises: The training data is clustered using a K-means clustering algorithm or a fuzzy clustering method.

14. The method according to any one of claims 1 to 13, characterized in that The classifier is a decision tree model, and the decision tree model is used to analyze the message flow and the label to obtain the fluency evaluation result.

15. A device for evaluating the fluency of an application, characterized in that: include: An acquisition module, used to acquire multiple groups of message flows and corresponding labels, wherein the message flows are message flows input to an application, and the labels are used to indicate the fluency of the application when processing the corresponding message flows; An analysis module, used to analyze the message flow and the label through a classifier to obtain a fluency evaluation result; The acquisition module is further used to obtain the confidence distribution corresponding to the fluency evaluation result; An updating module is used to update the parameters of the classifier according to the confidence distribution and a preset expected value to obtain an updated classifier, and the updated classifier is used to evaluate the fluency of a message flow subsequently input into the application.

16. The device according to claim 15, characterized in that The analysis module is specifically used for: Extracting features from the message flow to obtain message features, wherein the message features include basic message features and augmented message features, and the augmented message features are obtained by augmenting the basic message features; The message features and the labels are analyzed by the classifier to obtain the fluency evaluation result.

17. The device according to any one of claims 15 or 16, characterized in that The acquisition module is specifically used for: The message features and the labels are analyzed by generating a flow network to obtain a confidence distribution corresponding to the fluency evaluation result.

18. The device according to any one of claims 15 to 17, characterized in that The update module is specifically used for: If there is a fluency evaluation result whose confidence is lower than the preset expected value, the fluency evaluation result is removed to obtain the updated classifier.

19. The device according to any one of claims 15 to 18, characterized in that The acquisition module is specifically used for: Obtain a jam record table, an initial application message, and an application identification table, wherein the jam record table is used to record the fluency of the application, and the application identification table includes an application category and an activity category of each message flow; The initial application message is preprocessed according to the jam record table and the application identification table to obtain the message flow and the corresponding label.

20. The device according to any one of claims 15 to 19, characterized in that The device also includes: The acquisition module is also used to acquire training data; A clustering module, used to cluster the training data to obtain cluster centers, wherein the cluster centers include M stuck centers and N non-stuttering centers, the stuck centers are numbered as any value from 1 to M, the non-stuttering centers are numbered as any value from M+1 to M+N, M is an integer greater than or equal to 1, and N is an integer greater than or equal to 1; A determination module, configured to determine the fluency evaluation result according to the distance between the message feature of the message flow and the cluster center; A calculation module is used to calculate the corresponding confidence level according to the fluency evaluation result.

21. The device according to claim 20, characterized in that The determining module is specifically used for: Calculate the distance from the message feature of the message flow to each of the jam class centers to obtain a first distance set; Calculate the distance from the message feature of the message flow to the center of each non-stuttering class to obtain a second distance set; Combining the first distance set and the second distance set to obtain a third distance set; If the number of the category center corresponding to the minimum distance value in the third distance set is less than or equal to M, the fluency evaluation result is determined to be a jam; If the number of the category center corresponding to the minimum distance value in the third distance set is greater than M, the fluency evaluation result is determined to be no lag.

22. The device according to claim 20, characterized in that The computing module is specifically used for: Determining a minimum distance value in the third distance set; Calculating an average value of the third distance set; Determining a minimum distance value in the first distance set; Calculating an average value of the first distance set; The confidence of the fluency evaluation result is calculated according to the minimum distance value in the third distance set, the average value of the third distance set, the minimum distance value in the first distance set, and the average value of the first distance set.

23. The device according to claim 20, characterized in that The computing module is specifically used for: Determining a minimum distance value in the third distance set; Calculating an average value of the third distance set; Determining a minimum distance value in the second distance set; Calculating an average value of the second distance set; The confidence of the fluency evaluation result is calculated according to the minimum distance value in the third distance set, the average value of the third distance set, the minimum distance value in the second distance set, and the average value of the second distance set.

24. A device for evaluating the fluency of an application, characterized in that: include: A processor and a memory, wherein the processor is coupled to the memory; The memory is used to store programs; The processor is configured to execute the program in the memory so as to perform the method according to any one of claims 1 to 14.

25. A computer-readable storage medium comprising instructions, which, when executed on a computer, enable the computer to perform the method according to any one of claims 1 to 14.

26. A computer program product comprising instructions which, when executed on a computer, cause the computer to perform the method as claimed in any one of claims 1 to 14.

Images