Application performance evaluation method, device, equipment, medium and product

Abstract

The embodiment of the invention provides an application performance evaluation method and device, equipment, a storage medium and a program product. In the method, target data generated when a target application executes a target operation is obtained, and the target data comprises multiple items of operation data related to operation of the target application and multiple items of environment information related to the multiple clients in the process that the target application executes the target operation at the multiple clients. Based on the target data, feedback information for the target application is determined, and the feedback information indicates the influence of the multiple pieces of environment information on the multiple pieces of operation data. And determining performance evaluation information of the target application based on the feedback information and the target data. In this manner, based on the plurality of items of operational data and the plurality of items of environmental information, feedback information indicative of an effect of the plurality of items of environmental information on the plurality of items of operational data is determined. Therefore, the non-linear incidence relation between the multiple weak network parameters and the multiple indexes is determined, and the performance evaluation information of the application is determined more accurately.

Description

TECHNICAL FIELD

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to a method, an apparatus, an electronic device, a computer readable storage medium and a computer program product for application performance evaluation. BACKGROUND

[0002] More and more applications are designed to provide various services to users. Users can perform various operations on the applications. For example, users can play various media content, such as images, videos, audios, etc. in a content sharing type application. However, there may be abnormalities in the working process of the application, which affects the normal use of the application by the user. SUMMARY

[0003] In a first aspect of the present disclosure, a method for application performance evaluation is provided. In the method, target data generated by a target application performing a target operation is obtained, the target data including a plurality of running data related to running of the target application in a process of the target application performing the target operation at a plurality of clients and a plurality of environment information related to the plurality of clients; feedback information for the target application is determined based on the target data, the feedback information indicating an influence of the plurality of environment information on the plurality of running data; and performance evaluation information of the target application is determined based on the feedback information and the target data.

[0004] In a second aspect of the present disclosure, an apparatus for application performance evaluation is provided, the apparatus comprising: an obtaining module configured to obtain target data generated by a target application performing a target operation, the target data including a plurality of running data related to running of the target application in a process of the target application performing the target operation at a plurality of clients and a plurality of environment information related to the plurality of clients; a first determining module configured to determine feedback information for the target application based on the target data, the feedback information indicating an influence of the plurality of environment information on the plurality of running data; and a second determining module configured to determine performance evaluation information of the target application based on the feedback information and the target data.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions when executed by the at least one processing unit cause the electronic device to perform the method of the first aspect of the present disclosure.

[0006] In a fourth aspect of the present disclosure, a computer readable storage medium is provided. The computer readable storage medium has stored thereon a computer program, which is executable by a processor to perform the method according to the first aspect of the present disclosure.

[0007] In a fifth aspect of the disclosure, a computer program product is provided, the program product comprising a computer program executable by a processor to implement the method of the first aspect.

[0008] It should be understood that all statements herein made regarding the exemplary embodiments of the disclosure are intended to encompass both the specific and the general embodiments, and that the examples provided herein are intended to be illustrative of various aspects of the disclosure and not restrictive of its scope. The following detailed description is presented to enable any person skilled in the art to make and use the disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0009] The above-described and other features, advantages, and aspects of the implementations of the present disclosure will become more apparent as various ones of the embodiments thereof are described in conjunction with the following detailed description, which should be taken in conjunction with the accompanying drawings. In the drawings, like reference numerals refer to like elements, in which: Figure 1 A schematic diagram illustrating an example environment in which embodiments of the disclosure can be implemented is shown; Figure 2 A schematic diagram illustrating an example architecture of a system for application performance evaluation in accordance with some embodiments of the disclosure is shown; Figure 3 A schematic diagram illustrating an example architecture of a system for processing target data in accordance with some embodiments of the disclosure is shown; Figure 4 A flow diagram illustrating an example process for application performance evaluation in accordance with some embodiments of the disclosure is shown; Figure 5 A block diagram illustrating an example apparatus for application performance evaluation in accordance with some embodiments of the disclosure is shown; and Figure 6 A block diagram of an electronic device in which one or more embodiments of the disclosure can be implemented is shown. DETAILED DESCRIPTION

[0010] Embodiments of the disclosure will be described below in reference to drawings. While some embodiments of the disclosure are illustrated, it is understood that the disclosure can be embodied in various forms and should not be interpreted as being limited to the embodiments set forth herein; rather, these embodiments are provided so as to facilitate a more complete and thorough understanding of the disclosure. It is understood that the drawings and embodiments of the disclosure are for exemplary purposes only and are not intended to limit the scope of the disclosure.

[0011] In the description of embodiments of the disclosure, the term "including" and its analogues are to be interpreted as open-ended, i.e., "including but not limited to". The term "based on" is to be interpreted as "based, at least in part, on". The term "one embodiment" or "the embodiment" are to be interpreted as "at least one embodiment". The term "some embodiments" is to be interpreted as "at least some embodiments". Other explicit and implicit definitions can also be included below.

[0012] As used herein, the term “model” can learn an association between a corresponding input and output from training data, such that after training is completed, the corresponding output can be generated for a given input. The generation of a model can be based on a machine learning technique. Deep learning is a machine learning algorithm that processes an input and provides a corresponding output by using multiple layers of processing units. In this document, a “model” can also be referred to as a “machine learning model”, a “machine learning network”, a “neural network”, or a “network”, which terms are used interchangeably herein.

[0013] A “neural network” is a machine learning network based on deep learning. A neural network is capable of processing an input and providing a corresponding output, which typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. A neural network used in deep learning applications typically includes many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence, such that the output of a previous layer is provided as an input to a subsequent layer, with the input layer receiving the input to the neural network and the output of the output layer as the final output of the neural network. Each layer of a neural network includes one or more nodes (also referred to as processing nodes or neurons), each of which processes an input from a previous layer.

[0014] Generally, machine learning can include three stages, namely a training stage, a testing stage, and an application stage (also referred to as an inference stage). In the training stage, a given model can be trained using a large amount of training data, iteratively updating parameter values until the model is able to obtain consistent inferences from the training data that meet an expected goal. Through training, the model can be considered to have learned an association (also referred to as a mapping) from input to output from the training data. The parameter values of the trained model are determined. In the testing stage, test inputs are applied to the trained model to test whether the model is able to provide correct outputs, thereby determining the performance of the model. In the application stage, the model can be used to process actual inputs based on the parameter values obtained through training to determine the corresponding outputs.

[0015] It should be noted that in the technical solutions of the present disclosure, the acquisition, storage and application of user personal information involved in the technical solutions comply with relevant laws and regulations and do not violate public order and good customs.

[0016] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the type of personal information involved in the present disclosure, the scope of use, the use scenario, etc. should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0017] For example, in response to receiving an active request of a user, a prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed by the user will require obtaining and using personal information of the user, so that the user can autonomously select whether to provide the personal information to the software or hardware performing the operation of the technical solution of the present disclosure according to the prompt information.

[0018] As an optional but non-limiting implementation manner, in response to receiving an active request of a user, the manner of sending a prompt information to the user can be, for example, a pop-up window manner, in which the prompt information can be presented in a text manner. In addition, the pop-up window can also carry a selection control for the user to select “agree” or “disagree” to provide the personal information to the electronic device.

[0019] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In the example environment 100, a target application 120 is installed in a client 110. A user 140 can interact with the target application 120 via the client 110 and / or an attached device of the client 110. Illustratively, the target application 120 can be a content generation application, a content sharing application, or a social application, which can provide services related to media content to the user 140, including browsing, commenting, forwarding, creating (e.g., shooting and / or editing), publishing, etc. of the content. “Media content” can include one or more types of content, such as video, image, moving picture, image set, audio, text, etc. The target application 120 can support the user 140 to create multimedia content. Such multimedia content can include image data and audio data.

[0020] In the environment 100, Figure 1 In the environment 100,

[0021] In some embodiments, client 110 communicates with server 130 to provide services to target application 120. Client 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, client 110 can also support any type of user-facing interface (such as "wearable" circuitry). Server 130 can be various types of computing systems / servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0022] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. For example, embodiments of this disclosure can be applied to any suitable one or more applications, and are not limited to office suites.

[0023] As briefly mentioned earlier, applications are designed to provide various services to users. During user operation, media content may fail to play due to application malfunctions or / or issues with the application's operating environment, preventing normal service delivery. To better serve users, application developers need to conduct weak network testing to accurately assess application performance and optimize it accordingly.

[0024] Application weak network testing primarily involves simulating different network environments (e.g., insufficient network bandwidth, high packet loss, high latency) to evaluate application performance under weak network conditions (e.g., video latency or stuttering rate), and combining this with Quality of Experience (QoE) for a comprehensive evaluation from the user's perspective. Currently, most applications require developers to manually sift through test data in weak network environments to determine application performance. However, this approach is resource-intensive and inefficient. Furthermore, application developers can only analyze the linear relationship between a single weak network parameter (e.g., packet loss rate, jitter, or network latency) and specific metrics, and cannot determine the correlation between multiple metrics.

[0025] In view of this, embodiments of this disclosure provide a scheme for application performance evaluation. In embodiments of this disclosure, target data generated by a target application performing a target operation is acquired. The target data includes multiple operational data related to the operation of the target application during the execution of the target operation at multiple clients, and multiple environmental information related to the multiple clients. Based on the target data, feedback information for the target application is determined, indicating the impact of the multiple environmental information on the multiple operational data. Based on the feedback information and the target data, performance evaluation information for the target application is determined.

[0026] In this way, on the one hand, based on multiple operational data related to the target application and multiple environmental information related to multiple clients, feedback information indicating the impact of multiple environmental information on multiple operational data is determined. This establishes the nonlinear correlation between multiple weak network parameters and multiple indicators, thereby more accurately determining application performance evaluation information. On the other hand, it improves data processing efficiency.

[0027] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0028] Figure 2 A schematic diagram of an example architecture for an application performance evaluation system according to some embodiments of this disclosure is shown. Figure 2 As shown, the application performance evaluation system includes a data acquisition layer 220, a computing engine layer 230, and a visualization layer 240.

[0029] In some embodiments, the data acquisition layer 220 may be implemented or included in the server 130. The data acquisition layer 220 is communicatively connected to the client 110, on which the target application is running. The user 140 sends operation commands to the client 110 running the target application via a human-computer interaction device (e.g., a touchscreen, keyboard, mouse, etc.) to control the target application to perform corresponding operations. For example, if the target application 120 is a content sharing application, the operation commands may be to control the target application to play or upload media content. The media content may include, but is not limited to, images, videos, and audio. In some embodiments, the client 110 may include an Android device, an iOS device, a Mac device, or a Windows device, etc.

[0030] The data acquisition layer 220 can acquire target data generated by the target application performing target operations in the client 110. In some embodiments, the data acquisition layer can acquire target data through full-tracking. For example, the data acquisition layer 220 can dynamically extract device hardware characteristics (device serial number, MAC address, CPU model, sensor parameters) and system characteristics (unique operating system identifier, system installation time, time zone setting) through the tracking SDK. In some embodiments, to ensure data security, the data acquisition layer 220 can concatenate the above characteristic information and process it using the SHA-256 hash algorithm to generate a unique and irreversible device identifier and cache it locally. In some embodiments, the data acquisition layer 220 can acquire the operating system version number and corresponding SDK version identifier of the client 110, the network type of the client 110, and the version information of the target application, etc.

[0031] In some embodiments, the data acquisition layer 220 can acquire multiple operational data related to the operation of the target application (e.g., the stuttering rate and latency of the target application when playing media content) to determine the operating status of the target application. The data acquisition layer 220 can also acquire multiple environmental information related to the client 110 where the target application resides, such as packet loss rate, network jitter, and network latency. Figure 2 As shown, the data acquisition layer 220 can detect, through the card frequency detection module 221, whether a stuttering event occurs (e.g., video frame rate is below 24 FPS), the frequency of stuttering events, and the duration of stuttering events when the target application performs a target operation (e.g., video playback), using the stuttering frequency detection module 221. In some embodiments, if the operation performed by the target application is video playback, the target data may have audio-visual synchronization deviation. The data acquisition layer 220 can determine the audio-visual synchronization deviation using a timestamp alignment algorithm.

[0032] In some embodiments, the data acquisition layer 220 can acquire target data through multiple communication links, which are established based on different communication protocols. The data acquisition layer 220 can determine the communication link used to acquire the target data based on the type of target data to be acquired. The data acquisition layer 220 can also determine the communication link used to acquire the target data based on the current network state. For example, the data acquisition layer 220 can attempt to acquire target data sent by the target application via a first communication link. If, after a predetermined number of attempts, the performance of the first communication link is detected as not meeting a first preset condition, the target data is acquired via a second communication link. In some embodiments, the first preset condition can be determined based on a predetermined packet loss rate or network latency. For example, if the data acquisition layer 220 detects that the packet loss rate of data transmitted using the first communication link exceeds a threshold (e.g., 15%), or the data transmission latency exceeds a threshold (e.g., 300ms), it determines that the performance of the first communication link does not meet the first preset condition.

[0033] In some embodiments, a first preset condition may indicate that the performance of the first communication link is lower than a first threshold and the handover success rate exceeds a second threshold. The handover success rate is determined based on multiple environmental information and historical handover data. Historical handover data includes historical environmental information and the corresponding handover success rate. For example, if the data acquisition layer 220 detects that the performance of the first communication link is lower than the first threshold, it may first acquire historical handover data matching the current environmental information, and then determine the historical handover success rate corresponding to the environmental information. If the historical environmental success rate exceeds the second threshold, the communication link is switched.

[0034] In some embodiments, the data acquisition layer 220 may determine a first preset condition based on historical handover success rates. For example, if a high historical handover success rate is detected when the packet loss rate is 20%, the packet loss rate threshold is set to 20%.

[0035] In some embodiments, the first communication link can be a communication link established based on WebRTC (UDP protocol), and the second communication link can be a communication link established based on RTMP. The first communication link can repair packet loss problems through forward error correction (FEC) technology. The second communication link can implement streaming media chunked transmission based on TCP protocol, and balance network latency and network stability through buffer adaptive algorithm.

[0036] Figure 3 A schematic diagram of an example architecture 300 for processing target data according to some embodiments of the present disclosure is shown. Figure 3 As shown, after the data acquisition layer 220 obtains the target data from the client 110 through the data acquisition module 310, it compresses the target data using the compression module 320. For example, the data acquisition layer 220 can perform protobuf serialization on the target data to compress it. Subsequently, the data acquisition layer 220 can provide the compressed target data to the computing engine layer 230 (e.g., heatmap matrix engine 231 and Bayesian network modeling engine 232) through the data aggregation interface 330 to reduce the network bandwidth occupied by transmitting the target data.

[0037] Continue to refer to Figure 2 The computing engine layer 230 can determine feedback information for the target application based on the target data obtained, which corresponds to multiple clients 110 respectively. The target data includes multiple data sets corresponding to different performance metrics, and each set of data includes relevant environmental information and operational data. The feedback information indicates the impact of the multiple environmental data sets on the multiple operational data sets.

[0038] like Figure 2 As shown, the computation engine layer 230 includes a heatmap matrix engine 231 and a Bayesian network modeling engine. The heatmap matrix engine 231 is used to determine whether the performance of the target application meets expectations based on the target data, and to generate a heatmap indicating the performance of the target application.

[0039] In some embodiments, the heatmap matrix engine 231 can perform clustering on all obtained target data based on multiple environmental information corresponding to each set of target data to obtain multiple sets of detection data. For example, the computation engine layer 230 can perform normalized clustering on the target data based on packet loss rate (0-20%), latency (0-500ms), and jitter (0-100ms) to obtain multiple sets of detection data. After performing the clustering operation, the heatmap matrix engine 231 can use Euclidean distance to generate corresponding weak network labels for each set of detection data. For example, the weak network label can be "PL20_LAT150_JITTER5", meaning that the packet loss rate of this set of detection data is 15-20%, the latency is 100-150ms, and the jitter is 0-5ms. The optimal inflection point of the SSE curve when k=10 is verified using the Elbow method. In some embodiments, the computation engine layer 230 can determine the cluster centers by plotting the Elbow curve and profile coefficient curve of the distance analysis.

[0040] In some embodiments, the heatmap matrix engine 231 can determine performance evaluation information based on the relationship between target data and a third threshold related to the performance of the target application. The third threshold can be a value specified by the application developer or a value determined based on historical data. For example, for a single data item (e.g., first data), the heatmap matrix engine 231 can first obtain first historical data related to the first data. Subsequently, the third threshold is determined based on the normal distribution of the first historical data. The heatmap matrix engine 231 can determine sub-evaluation information corresponding to the first data based on the relationship between the target application performance level indicated by the first data and the third threshold. If it is determined that the target application performance level indicated by the first data exceeds the third threshold, the sub-evaluation information corresponding to the first data is determined to be "qualified". Otherwise, the sub-evaluation information is determined to be "unqualified". For example, if the first data is "stuttering rate 10%" and the third threshold indicates "stuttering rate of the target application 15%", then the sub-evaluation information corresponding to the first data is determined to be "qualified".

[0041] In some embodiments, the Bayesian network modeling engine 232 can use machine learning models to determine causal relationships between at least one environmental data and at least one operational data corresponding to each of a plurality of sets of detection data.

[0042] In some embodiments, the Bayesian network modeling engine 232 can be used to build a dynamic Bayesian network. For example, the Bayesian network modeling engine 232 can process the reference detection data based on a greedy search algorithm, and combine this with a Bayesian information criterion scoring function to learn and generate a directed acyclic graph from the reference detection data. This determines the network structure of the Bayesian network. Furthermore, the Bayesian network modeling engine 232 can iteratively optimize the hidden nodes in the Bayesian network using an expectation-maximization algorithm. For example, the Bayesian network modeling engine 232 can use the forward error correction strategy as a hidden variable, and infer the state of the hidden variable through the version number of the target application, thereby establishing a causal chain of "version number - forward error correction strategy - packet loss repair effect". Subsequently, the Bayesian network modeling engine 232 can update the parameters of the Bayesian network based on the reference detection data.

[0043] In some embodiments, the Bayesian network modeling engine 232 can allow for a latency difference of ≤200ms between audio and video metrics, constrain the network structure by dividing the time slice (e.g., 1-second intervals), construct a dynamic Bayesian network using discrete time slices (Δt=1s), and determine the state changes of the running data in adjacent time slices by modeling the transition probability matrix. In some embodiments, the weights of multiple nodes in the Bayesian network can be updated in real time based on the target data.

[0044] In some embodiments, the Bayesian network modeling engine 232 utilizes a machine learning model to determine a causal relationship between at least one piece of environmental data and at least one piece of runtime data corresponding to each of multiple sets of detection data. For example, the Bayesian network modeling engine 232 can utilize a trained Bayesian network to perform causal inference and attribution analysis on target data, and use variable elimination to calculate the posterior probability under specific conditions, such as the probability of a stuttering rate exceeding 10% when the packet loss rate is high and forward error correction is enabled. In some embodiments, the Bayesian network modeling engine 232 can utilize a Bayesian network, combined with SHapley Additive exPlanations (SHAP) values, to quantify the contribution of environmental data to runtime data. In some embodiments, the Bayesian network modeling engine 232 can determine feedback information based on multiple causal relationships corresponding to multiple sets of detection data. For example, the Bayesian network modeling engine 232 can generate natural language conclusions (such as "Version V3.1 FEC algorithm reduces stuttering rate by 30% in high packet loss scenarios") based on the quantified contribution of environmental data to runtime data, combined with Do-Calculus.

[0045] In some embodiments, the heatmap matrix engine 231 may use data with the same application version or the same client identifier as the first data as the first historical data. In this case, the heatmap matrix engine can determine the third threshold based solely on the target application with the same version, thereby improving the reliability of the third threshold.

[0046] Visualization layer 240 can present performance evaluation information to the user. For example... Figure 2 As shown, the visualization layer 240 may include a dynamic alert module 241 and a drill-down analysis module 242. The dynamic alert module 241 is used to provide dynamic alerts to the user based on the heatmap information generated by the heatmap matrix engine 231. For example, if the sub-evaluation information is unqualified, the heatmap cell corresponding to the sub-evaluation information will display a red border warning.

[0047] In some embodiments, the drill-down analysis module 242 can provide the user with analytical information regarding performance evaluation information through the heatmap matrix engine 231. For example, if the heatmap matrix engine 231 detects a selection of an interactive control in the heatmap, it determines second data corresponding to the interactive control. For example, the interactive control in the heatmap can be each cell in the heatmap table. Based on second historical data corresponding to the second data, the trend of the second data is determined. If the heatmap matrix engine 231 detects a selection of a cell in the heatmap (e.g., a cell corresponding to the stuttering rate), it presents the trend of the stuttering rate over time. Subsequently, the heatmap matrix engine 231 can present the user with an indication representing the trend. For example, the heatmap matrix engine 231 can present the trend of the second data in the form of a line graph.

[0048] In some embodiments, the visualization layer 240 can be used to present a causal graph corresponding to environmental information and operational information to the user, so as to demonstrate the causal relationship between environmental information and operational information.

[0049] Figure 4 A flowchart of an application performance evaluation process 400 according to some embodiments of the present disclosure is shown. Process 400 can be implemented at server 130.

[0050] In box 410, server 130 obtains target data generated by the target application performing target operations. The target data includes multiple runtime data related to the operation of the target application and multiple environmental information related to multiple clients during the execution of target operations by the target application on multiple clients.

[0051] In some embodiments, acquiring target data includes: attempting to acquire target data sent by a target application via a first communication link; and in response to detecting, based on a predetermined number of attempts, that the performance of the first communication link does not meet a first preset condition, acquiring target data via a second communication link, wherein the first communication link and the second communication link are established based on different communication protocols.

[0052] In some embodiments, a first preset condition indicates that the performance of the first communication link is lower than a first threshold and the handover success rate exceeds a second threshold. The handover success rate is determined based on multiple environmental information and historical handover data.

[0053] In box 420, server 130 determines feedback information for the target application based on the target data. The feedback information indicates the impact of multiple environmental information on multiple operational data.

[0054] In some embodiments, determining feedback information based on target data includes: performing clustering on the target data based on multiple environmental information to obtain multiple sets of detection data; using a machine learning model to determine a causal relationship between at least one environmental data and at least one operational data corresponding to each set of detection data in the multiple sets of detection data; and determining feedback information based on multiple causal relationships corresponding to the multiple sets of detection data.

[0055] In some embodiments, the performance evaluation information is expressed in natural language.

[0056] In box 430, server 130 determines the performance evaluation information of the target application based on feedback information and target data.

[0057] In some embodiments, the target data includes multiple data points corresponding to different performance metrics, and determining the performance evaluation information of the target application based on feedback information and the target data includes: for a first data point among the multiple data points, determining a third threshold related to the performance of the target application based on first historical data related to the first data point; in response to determining that the performance level of the target application indicated by the first data point exceeds the third threshold, determining sub-evaluation information corresponding to the first data point; and using a machine learning model to determine the performance evaluation information based on the feedback information and the multiple sub-evaluation information determined for the multiple data points respectively.

[0058] In some embodiments, process 400 further includes, in response to determining multiple sub-evaluation information, determining a heatmap corresponding to the multiple sub-evaluation information based on the multiple sub-evaluation information and multiple data, the heatmap being used to characterize the network performance of the target application; and presenting the heatmap.

[0059] In some embodiments, process 400 further includes, in response to detecting a selection of an interactive control in a heatmap, determining second data corresponding to the interactive control; determining a trend of change of the second data based on second historical data corresponding to the second data; and presenting an indication for characterizing the trend of change.

[0060] Figure 5 A block diagram of an apparatus 500 for application performance evaluation according to some embodiments of the present disclosure is shown. The apparatus 500 may be implemented as or included in the server 130.

[0061] like Figure 5 As shown, the device 500 includes an acquisition module 510 configured to acquire target data generated by a target application performing a target operation. The target data includes multiple operational data related to the operation of the target application during the execution of the target operation at multiple clients, and multiple environmental information related to the multiple clients. The device 500 also includes a first determination module 520 configured to determine feedback information for the target application based on the target data. The feedback information indicates the impact of multiple environmental information on multiple operational data. The device 500 further includes a second determination module 530 configured to determine performance evaluation information of the target application based on the feedback information and the target data.

[0062] In some embodiments, the acquisition module 510 is further configured to attempt to acquire target data sent by the target application via a first communication link; and in response to detecting, based on a predetermined number of attempts, that the performance of the first communication link does not meet a first preset condition, acquire the target data via a second communication link, wherein the first communication link and the second communication link are established based on different communication protocols.

[0063] In some embodiments, a first preset condition indicates that the performance of the first communication link is lower than a first threshold and the handover success rate exceeds a second threshold. The handover success rate is determined based on multiple environmental information and historical handover data.

[0064] In some embodiments, the first determining module 520 is further configured to perform clustering on the target data based on multiple environmental information to obtain multiple sets of detection data; use a machine learning model to determine the causal relationship between at least one environmental data and at least one operational data corresponding to each set of detection data in the multiple sets of detection data; and determine feedback information based on the multiple causal relationships corresponding to the multiple sets of detection data.

[0065] In some embodiments, the performance evaluation information is expressed in natural language.

[0066] In some embodiments, the target data includes multiple data points corresponding to different performance metrics, and the second determining module is further configured to, for the first data point among the multiple data points, determine a third threshold related to the performance of the target application based on first historical data related to the first data point; in response to determining that the performance level of the target application indicated by the first data point exceeds the third threshold, determine sub-evaluation information corresponding to the first data point; and determine performance evaluation information using a machine learning model based on feedback information and the multiple sub-evaluation information determined for the multiple data points respectively.

[0067] In some embodiments, the apparatus 500 further includes a first presentation module configured to, in response to determining multiple sub-evaluation information, determine a heatmap corresponding to the multiple sub-evaluation information based on the multiple sub-evaluation information and multiple data, the heatmap being used to characterize the network performance of the target application; and present the heatmap.

[0068] In some embodiments, the apparatus 500 further includes a second presentation module configured to, in response to detecting a selection of an interactive control in a heatmap, determine second data corresponding to the interactive control; determine a trend of change of the second data based on second historical data corresponding to the second data; and present an indication for characterizing the trend of change.

[0069] The modules included in device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 700 can be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0070] Figure 6 A block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 6 The electronic device 600 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein.

[0071] like Figure 6 As shown, electronic device 600 is in the form of a general-purpose computing device. Components of electronic device 600 may include, but are not limited to, one or more processors or processing units 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processing unit 610 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 600.

[0072] Electronic device 600 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 630 can be a removable or non-removable medium and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within electronic device 600.

[0073] Electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 6 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 620 may include computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0074] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or another network node.

[0075] Input device 650 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 600 can also communicate with one or more external devices (not shown) via communication unit 640 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 600, or with any device that enables electronic device 600 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interfaces (not shown).

[0076] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores one or more computer instructions, wherein the one or more computer instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transient computer-readable medium and includes computer-executable instructions that are executed by a processor to implement the methods described above.

[0077] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0078] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0079] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0080] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0081] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the implementations disclosed herein.

Claims

1. An application performance evaluation method, comprising: Acquire target data generated by the target application performing the target operation. The target data includes multiple operational data related to the operation of the target application and multiple environmental information related to the multiple clients during the execution of the target operation by the target application on multiple clients. Based on the target data, feedback information is determined for the target application, wherein the feedback information indicates the impact of the multiple environmental information items on the multiple operational data items; as well as Based on the feedback information and the target data, the performance evaluation information of the target application is determined.

2. The method according to claim 1, wherein obtaining the target data comprises: An attempt is made to obtain the target data sent by the target application via the first communication link; as well as In response to the detection that the performance of the first communication link does not meet the first preset condition after a predetermined number of attempts, the target data is obtained via a second communication link, wherein the first communication link and the second communication link are established based on different communication protocols.

3. The method according to claim 2, wherein the first preset condition indicates that the performance of the first communication link is lower than a first threshold and the handover success rate exceeds a second threshold, and the handover success rate is determined based on the plurality of environmental information and historical handover data.

4. The method according to claim 1, wherein determining the feedback information based on the target data includes: Based on the aforementioned environmental information, clustering is performed on the target data to obtain multiple sets of detection data; Using machine learning models, determine the causal relationship between at least one environmental data point and at least one operational data point corresponding to each of the multiple sets of detection data; as well as The feedback information is determined based on multiple causal relationships corresponding to the multiple sets of detection data.

5. The method according to claim 1, wherein the target data includes multiple data corresponding to different performance indicators, and The performance evaluation information for the target application determined based on the feedback information and the target data includes: Regarding the first data among the multiple data sets, Based on first historical data related to the first data, a third threshold related to the performance of the target application is determined; In response to determining that the target application performance level indicated by the first data exceeds the third threshold, sub-evaluation information corresponding to the first data is determined; as well as Based on the feedback information and the multiple sub-evaluation information determined for the multiple data, the performance evaluation information is determined using a machine learning model.

6. The method of claim 1, wherein the performance evaluation information is information expressed in natural language.

7. The method of claim 5, further comprising: In response to determining the multiple sub-evaluation information, a heatmap corresponding to the multiple sub-evaluation information is determined based on the multiple sub-evaluation information and the multiple data, the heatmap being used to characterize the network performance of the target application; as well as The heatmap is presented.

8. The method according to claim 7, further comprising: In response to detecting a selection of an interactive control in the heatmap, second data corresponding to the interactive control is determined; Based on the second historical data corresponding to the second data, determine the trend of change of the second data; as well as Indicators are presented to characterize the aforementioned trend of change.

9. An apparatus for application performance evaluation, comprising: The acquisition module is configured to acquire target data generated by the target application performing the target operation. The target data includes multiple operational data related to the operation of the target application and multiple environmental information related to the multiple clients during the execution of the target operation by the target application at multiple clients. The first determining module is configured to determine feedback information for the target application based on the target data, wherein the feedback information indicates the impact of the multiple environmental information on the multiple operational data. as well as The second determining module is configured to determine the performance evaluation information of the target application based on the feedback information and the target data.

10. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 8 when executed by the at least one processing unit.

11. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 8.

12. A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images