Data processing method and related device

Abstract

The invention provides a data processing method and a related device. The method comprises the steps of performing performance prediction based on real-time performance data of a target application to obtain a prediction result; under the condition that the prediction result meets the collection condition, sending a performance collection instruction to a performance detection client; and receiving performance acquisition data acquired by the performance detection client, and storing the performance acquisition data. Through the application, the efficiency of collecting and monitoring the performance data of the target application can be improved.

Description

Technical Field

[0001] The present application relates to computer technology, and in particular to a data processing method and related devices. Background Art

[0002] Microservice architecture can decompose large software system projects into multiple smaller microservices that are easy to implement and deploy independently. Through performance collection, the performance of applications in microservices can be monitored in real time, and performance bottlenecks can be discovered and located in a timely manner; the system's operating status can also be monitored in real time, potential problems can be discovered in a timely manner, system crashes or service interruptions can be avoided, and system stability can be ensured; application performance data can also be provided to help developers understand the performance of applications and make targeted optimizations.

[0003] Performance collection is crucial to ensuring application performance, availability, and user satisfaction, and is also the basis for supporting fast, effective decision-making and continuous performance improvement. Therefore, how to conduct efficient performance collection has become one of the hot research issues. Summary of the invention

[0004] The embodiments of the present application provide a data processing method and related devices, which can improve the efficiency of collecting and monitoring performance data of a target application.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a data processing method, the method comprising:

[0007] Perform performance prediction based on the real-time performance data of the target application to obtain prediction results;

[0008] When the prediction result meets the collection condition, sending a performance collection instruction to the performance detection client;

[0009] Receive the performance collection data collected by the performance detection client, and store the performance collection data.

[0010] The present application also provides a data processing method, the method comprising:

[0011] Receiving a performance collection instruction sent by the performance detection server, wherein the performance collection instruction is sent when a prediction result of the performance prediction of the target application meets a collection condition;

[0012] Performing data collection on the target application based on the performance collection instruction to obtain performance collection data;

[0013] The performance collection data is sent to the performance detection service end.

[0014] The present application provides a data processing device, the device comprising:

[0015] A prediction module is used to perform performance prediction based on the real-time performance data of the target application and obtain prediction results;

[0016] A first sending module, configured to send a performance collection instruction to a performance detection client when the prediction result meets the collection condition;

[0017] The first receiving module is used to receive the performance collection data collected by the performance detection client and store the performance collection data.

[0018] The present application also provides a data processing device, the device comprising:

[0019] A second receiving module is used to receive a performance collection instruction sent by the performance detection server, wherein the performance collection instruction is sent when the prediction result of the performance prediction of the target application meets the collection condition;

[0020] A collection module, used for collecting data from the target application based on the performance collection instruction to obtain performance collection data;

[0021] The second sending module is used to send the performance collection data to the performance detection server.

[0022] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0023] A memory for storing computer executable instructions;

[0024] The processor is used to implement the data processing method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0025] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the data processing method provided in the embodiment of the present application when executed by a processor.

[0026] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the data processing method provided in the embodiment of the present application is implemented.

[0027] The embodiments of the present application have the following beneficial effects:

[0028] In an embodiment of the present application, a performance prediction is performed based on the real-time performance data of the target application to obtain a prediction result, and when the prediction result meets the collection condition, a performance collection instruction is sent to the performance detection client. In this way, the future performance of the target application is predicted through the real-time performance data of the target application, and the future performance data of the target application can be accurately grasped in advance. When the prediction result meets the collection condition, a performance collection instruction is sent to the performance detection client. It should be understood that the prediction result meets the collection condition, indicating that performance collection is required according to the prediction result. In this way, when it is predicted that performance collection is required, the performance data of the target application can be triggered based on the performance collection instruction, thereby achieving the purpose of on-demand, efficient and timely performance collection. By receiving the performance collection data collected by the performance detection client and storing the performance collection data, wherein the performance collection data is collected by the performance monitoring client in response to the performance collection instruction, it can be ensured that the performance data of the target application is accurately analyzed according to the stored performance collection data, thereby assisting in locating performance hotspots and improving the efficiency of collecting and monitoring the performance data of the target application. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a schematic diagram of the architecture of a data processing system 100 provided in an embodiment of the present application;

[0030] Figure 2A is a schematic diagram of the structure of the performance detection server 200 provided in an embodiment of the present application;

[0031] Figure 2B is a schematic diagram of the structure of the terminal 400 provided in an embodiment of the present application;

[0032] Figure 3A It is a schematic diagram of a process of performing data processing by a performance detection server provided in an embodiment of the present application;

[0033] Figure 3B It is a flowchart of the model training method provided in the embodiment of the present application;

[0034] Figure 3C It is a flowchart of a data storage method provided in an embodiment of the present application;

[0035] Figure 3D It is a flowchart of a data query method provided in an embodiment of the present application;

[0036] Figure 4 It is a schematic diagram of a process of performing data processing by a performance detection client provided in an embodiment of the present application;

[0037] Figure 5It is a schematic diagram of the interactive process of performing data processing by a performance detection server and a performance detection client provided in an embodiment of the present application;

[0038] Figure 6 It is a first flow chart of the practical application of the data processing method provided in the embodiment of the present application;

[0039] Figure 7 It is a second flow chart of the practical application of the data processing method provided in the embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0041] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0042] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0043] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0044] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0045] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0046] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0047] 1) Performance glitches: Performance glitches refer to short-term, abnormal performance fluctuations that occur during the operation of an application. Performance glitches can cause a sudden increase in time consumption. These fluctuations may be caused by factors such as hardware resource competition, system load changes, and network delays. Performance glitches may affect the response speed, stability, and user experience of the application.

[0048] 2) Duration: In performance analysis and optimization, duration is a key performance indicator that can help identify potential performance bottlenecks and improvement points. By monitoring and analyzing duration, you can take appropriate measures to optimize the performance of the system, application, or service.

[0049] 3) Performance hotspots: refers to the key areas or operations that cause performance bottlenecks or inefficiencies in an application. Identifying and optimizing the performance hotspots of an application is critical to improving overall performance.

[0050] 4) Microservices: An enterprise IT architecture model or architectural style proposed by Martin Fowler and James Lewis in 2014. Microservices emphasize dividing a single application into a series of small services, each of which runs in its own process and on its own server. These services are independent and communicate with each other through lightweight, network-based communication mechanisms. Each service focuses on completing a specific business task and can be deployed and expanded independently.

[0051] 5) Flame Graph: Displays the call stack and sampling count in the application code in a hierarchical manner, and arranges the application code names from bottom to top.

[0052] 6) Java Flight Recorder (JFR): A historical event data format for JAVA applications, mainly used to diagnose the historical performance of JAVA applications.

[0053] 7) Java Application Performance Analysis Tool (Async-Profiler): A set of open source tools that can perform real-time performance sampling of JAVA applications and support conversion of JFR format to Flame Graph format.

[0054] 8) Long Short-Term Memory (LSTM): A special recurrent neural network that aims to solve the gradient vanishing or gradient exploding problems that occur when traditional recurrent neural networks (RNNs) deal with long-distance dependency problems. LSTM can better capture and utilize long-term dependencies in time series data, so it performs well in processing and predicting long-term dependency problems in time series data.

[0055] 9) Deep Learning Framework Keras: A high-level neural network application programming interface (API) for building and training deep learning models.

[0056] 10) Application Performance Management (APM) system: A tool used to monitor and manage system performance. It can help users understand the system's operating status, identify potential performance issues, and provide corresponding solutions.

[0057] 11) Real-time performance data: The performance indicators and data generated by the application during operation can help developers and administrators monitor the performance of the application and provide real-time feedback. Through real-time performance data, developers and administrators can monitor the operation of the application at any time, discover and solve potential performance problems in a timely manner, optimize the performance of the application, and improve the user experience. Real-time performance data can also serve as an important basis for formulating further optimization strategies and decisions.

[0058] 12) Sequence mode: refers to the method of operating on time series data, mainly used to process data containing multiple time steps. Sequence mode can better process and mine the information in time series data and improve the performance and accuracy of the model.

[0059] In the related art, the call link tracking identifier is automatically added to the application through the APM plug-in, and the application service monitoring is provided by analyzing the operation data of each application service associated with the call link collected by the APM system. However, since APM provides the time consumption data of the interface call link between each microservice, when performance glitches (time consumption surges) occur in the online production environment, the APM system cannot collect performance data.

[0060] The present invention provides a data processing method, apparatus, device, computer-readable storage medium, and computer program product.

[0061] Based on the real-time performance data of the target application, a performance prediction is performed to obtain a prediction result. When the prediction result meets the collection condition, a performance collection instruction is sent to the performance detection client. In this way, the future performance of the target application is predicted through the real-time performance data of the target application, and the future performance data of the target application can be accurately grasped in advance. When the prediction result meets the collection condition, a performance collection instruction is sent to the performance detection client. It should be understood that the prediction result meets the collection condition, indicating that performance collection is required according to the prediction result. In this way, when it is predicted that performance collection is required, the performance data of the target application can be triggered based on the performance collection instruction, thereby achieving the purpose of on-demand, efficient and timely performance collection. By receiving the performance collection data collected by the performance detection client and storing the performance collection data, wherein the performance collection data is collected by the performance monitoring client in response to the performance collection instruction, it can be ensured that the performance data of the target application is accurately analyzed according to the stored performance collection data, thereby assisting in locating performance hotspots and improving the efficiency of collecting and monitoring the performance data of the target application.

[0062] The following describes an exemplary application of the electronic device provided by the embodiment of the present application. The electronic device provided by the embodiment of the present application can be implemented as various types of terminals such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a smart phone, a smart speaker, a smart watch, a smart TV, and a vehicle-mounted terminal, and can also be implemented as a server. The following describes an exemplary application when the electronic device is implemented as a server or a client.

[0063] See also Figure 1 , Figure 1 1 is a schematic diagram of the architecture of the data processing system 100 provided in an embodiment of the present application, wherein the performance detection server 200 is connected to the time series database 500, the terminal 400 includes a display interface 410, and a performance detection client is installed in the terminal 400, and the terminal 400 is connected to the performance detection server 200 via a network 300. The network 300 may be a wide area network or a local area network, or a combination of the two.

[0064] Among them, the performance detection server 200 is used to receive the real-time performance data of the target application sent by the application performance management system, perform performance prediction based on the real-time performance data of the target application, and obtain the prediction result; when the prediction result meets the collection condition, send a performance collection instruction including: collection sequence information, application identification information of the target application and process identification information of the target application to the performance detection client in the terminal 400. The performance detection client in the terminal 400 is used to receive the performance collection instruction sent by the performance detection server 200, perform data collection on the target application based on the performance collection instruction, and obtain performance collection data; send the performance collection data to the performance detection server 200. Then the performance detection server 200 receives the performance collection data collected by the performance detection client in response to the performance collection instruction, and stores the performance collection data in the time series database 500. When the user needs to query the collection results, the user inputs the collection result query information in the terminal 400. The performance detection server 200 will search in the time series database 500 according to the collection result query information, obtain the query results and return the query results to the terminal 400, so that the terminal 400 outputs the query results on the display interface 410, which is convenient for the user to perform performance analysis on the target application according to the output query results.

[0065] See also Figure 2A , Figure 2A is a schematic diagram of the structure of the performance detection server 200 provided in an embodiment of the present application, Figure 2A The performance detection server 200 shown includes at least one processor 210, a memory 230, and at least one network interface 220. The various components in the performance detection server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 2A Various buses are labeled as bus system 240 .

[0066] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0067] The memory 230 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 230 may optionally include one or more storage devices that are physically remote from the processor 210.

[0068] The memory 230 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 230 described in the embodiments of the present application is intended to include any suitable type of memory.

[0069] In some embodiments, memory 230 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0070] Operating system 231, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0071] The network communication module 232 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 220. Exemplary network interfaces 220 include: Bluetooth, Wireless Authentication (WiFi), and Universal Serial Bus (USB).

[0072] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2A The data processing device 233 stored in the memory 230 is shown, which can be software in the form of a program and a plug-in, etc., and includes the following software modules: a prediction module 2331, a first sending module 2332, and a first receiving module 2333. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0073] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0074] See also Figure 2B , Figure 2B is a schematic diagram of the structure of the terminal 400 provided in an embodiment of the present application, Figure 2B The terminal 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2B Various buses are labeled as bus system 440 .

[0075] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0076] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0077] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0078] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0079] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0080] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0081] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0082] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0083] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0084] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2B The data processing device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a second receiving module 4551, a collection module 4552 and a second sending module 4553. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0085] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0086] The data processing method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the performance collection server provided in the embodiment of the present application.

[0087] It should be noted that in the data processing examples below, the performance data of the application is used as an example. Based on their understanding of the following, those skilled in the art can apply the data processing method provided in the embodiments of the present application to the processing of other types of data.

[0088] The data processing method provided in the embodiment of the present application is described below, taking the electronic device implementing the data processing method in the embodiment of the present application as a performance detection server as an example. Therefore, the execution subject of each step will not be repeatedly described below.

[0089] See also Figure 3A , Figure 3A This is a schematic diagram of the process of data processing by the performance detection server provided in the embodiment of the present application, which is described in detail below:

[0090] In step 301, performance prediction is performed based on real-time performance data of the target application to obtain a prediction result.

[0091] Here, a trained performance prediction model can be obtained, and the trained performance prediction model can be used to perform performance prediction based on the real-time performance data of the target application to obtain a prediction result. Specifically, the real-time performance data is standardized, and then the standardized real-time performance data is input into the trained performance prediction model. The trained performance prediction model will predict the future performance data of the target application based on the real-time performance data, and obtain a prediction result that can characterize the performance data of the future target application. Among them, the prediction results include prediction data such as "CPU usage, disk I / O, network traffic, system load, number of concurrent requests, number of active threads, and heap memory percentage", which respectively characterize the changing trends of various performance data of the target application in the future. In this way, directly predicting the performance of the target application through a trained performance prediction model can improve the prediction accuracy and prediction efficiency, thereby improving the effect of monitoring the performance data of the target application.

[0092] In some embodiments, before executing the above step 301, real-time performance data of the target application sent by the application performance management system may be received first.

[0093] Here, the real-time performance data sent by the application performance management system is received through the performance detection server (PPROF system). The application performance management system is a tool for monitoring and managing system performance. It is used to analyze and diagnose the performance bottlenecks of applications and can help users diagnose and optimize application performance issues. The application performance management system will monitor the performance data of the target application in real time. When it is necessary to predict the performance data of the target application, the application performance management system will send the monitored real-time performance data of the target application to the performance detection server for prediction of the future performance data of the target application. The target application is a microservice application that requires performance detection. Real-time performance data includes data indicators related to the performance of the target application at the current moment, mainly "CPU usage, disk I / O, network traffic, system load, number of concurrent requests, number of active threads, heap memory percentage", etc.

[0094] In some embodiments, the prediction result is obtained by predicting the performance prediction model, see Figure 3B Before executing the above step 301, the performance prediction model may be trained through steps 401 to 404, as described below:

[0095] In step 401, a training data set and a validation data set are determined based on sample performance data.

[0096] In an embodiment of the present application, multiple sample performance data may be first acquired, and the multiple sample performance data may be sorted according to acquisition time information to obtain an initial data set; and then a training data set and a verification data set may be generated based on the initial data set.

[0097] Here, multiple sample performance data can be obtained from the APM system. The multiple sample performance data include multiple performance-related data indicators of multiple different sample applications (microservice applications). The performance data of each sample application mainly include "CPU usage, disk I / O, network traffic, system load, number of concurrent requests, number of active threads, heap memory ratio", etc. The collection time information can include a timestamp, such as "collection time: 04:44:10", and the collection time can also be expressed in other time formats such as year, month, and day. The specific representation format can be set according to actual needs. Among them, for each sample application, the multiple sample performance data of each sample application are sorted according to the collection time information of each sample performance data to obtain the initial data subset of each sample application, and then the initial data subsets of the multiple sample applications are used as the initial data set. For example, the multiple sample performance data include 4 sample performance data of sample application A and 3 sample performance data of sample application B. Among them, the collection time information of the four sample performance data of sample application A: sample performance data 1, sample performance data 2, sample performance data 3 and sample performance data 4 are respectively: collection time 12:00:01; collection time 12:44:10; collection time 13:00:00; collection time 11:10:00. Then, the initial data subset of sample application A obtained for sample application A includes: sample performance data 4, sample performance data 1, sample performance data 2, sample performance data 3. Similarly, the initial data subset of sample application B is obtained for sample application B, and then the initial data subset of sample application A and the initial data subset of sample application B are used as the initial data set.

[0098] After obtaining the initial data set, the initial data set needs to be standardized, wherein the standardization includes cleaning and normalization, so as to obtain a standardized initial data set. Among them, the purpose of cleaning is to identify and correct the erroneous, abnormal or inconsistent data in the initial data set, specifically including: missing value processing, abnormal value processing, duplicate value processing, data conversion, inconsistent data processing, etc.; the purpose of normalization is to convert the data to the same scale, including: Min-Max Normalization, Z-score normalization, decimal scaling, and maximum absolute value standardization. Then, the initial data subsets of multiple sample applications in the initial data set are divided into two parts according to a preset ratio, one of which is used as a training data set and the other is used as a verification data set. Therefore, the training data set includes multiple initial data subsets corresponding to a part of sample applications, and the verification data set includes multiple initial data subsets corresponding to another part of sample applications. Among them, the preset ratio is a preset ratio. Usually, the number of data in the training data set is larger than the verification data set. For example, the preset ratio is 4:1, then an initial data subset of 80% of the samples in the standardized initial data set is used as a training data set, and an initial data subset of 20% of the samples is used as a validation data set.

[0099] In step 402, a performance prediction model is constructed through a sequence pattern, and the performance prediction model is trained based on a training data set to obtain a candidate performance prediction model.

[0100] In an embodiment of the present application, a performance prediction model constructed by a sequence pattern may be first obtained.

[0101] Here, the sequential mode of the deep learning framework Keras can be used to build the model, and the parameters such as the number of network layers of the model can be set and the optimizer and loss function of the model training can be specified to obtain the performance prediction model. For example, a Keras sequential model is built, a multi-layer network is added (for example, 5 layers can be added), the loss function is selected as Mean Squared Error (MSE), and the optimizer is selected as the optimization algorithm (Root Mean Square Propagation, RMSPROP), so as to build a performance prediction model.

[0102] Here, the performance prediction model is trained by the initial data subset of each sample application in the training data set to obtain a candidate performance prediction model. Specifically, for each sample application's initial data subset, the sample performance data ranked first in the initial data subset is input into the performance prediction model, so that the performance prediction model performs performance prediction based on the sample performance data to obtain the predicted performance data of the sample application in the future, and then determines the predicted loss value according to the preset loss function and the sample performance data ranked second in the initial data subset (i.e., the real performance data of the sample application in the future), and back-propagates the loss value to the performance prediction model, and adjusts the parameters of the performance prediction model based on the loss value through the gradient descent algorithm. Then, the above steps are repeated in sequence for the next sample performance data in the initial data subset until the training end condition is reached to obtain a candidate performance prediction model. Among them, the training end condition can be that the loss value is lower than the loss threshold or the difference between the current loss value and the previous adjacent loss value is less than the preset difference.

[0103] In step 403, the prediction result of the candidate performance prediction model is verified based on the verification data set to obtain a prediction error value.

[0104] Here, the prediction results of the candidate performance prediction model are verified by verifying the initial data subset of each sample application in the verification data set to obtain a prediction error value. Specifically, for each sample application's initial data subset, the sample performance data ranked first in the initial data subset is input into the candidate performance prediction model, so that the candidate performance prediction model performs performance prediction based on the sample performance data to obtain the predicted performance data of the sample application in the future, and then determines the error value based on the preset error calculation function and the sample performance data ranked second in the initial data subset (i.e., the real performance data of the sample application in the future). Then, the above steps are repeated in sequence for the next sample performance data in the initial data subset until the penultimate sample performance data in the initial data subset is input into the candidate performance prediction model to obtain multiple error values. Finally, the average or mode of the multiple error values ​​is taken to obtain the prediction error value.

[0105] In step 404, when the prediction error value is less than a preset error threshold, the candidate performance prediction model is determined to be a trained performance prediction model.

[0106] Here, the preset error threshold is a prediction error tolerance value of a performance prediction model defined in advance in combination with specific problems and business requirements, and is used to determine whether the prediction performance of the candidate performance prediction model meets the expected standard. Therefore, when the prediction error value is less than the preset error threshold, it means that the prediction performance of the candidate performance prediction model meets the expected standard, and the candidate performance prediction model can be determined to be a trained performance prediction model.

[0107] In an embodiment of the present application, a plurality of sample performance data are obtained, and the plurality of sample performance data are sorted according to the acquisition time information to obtain an initial data set, and a training data set and a verification data set are generated. Then a performance prediction model is obtained, and the performance prediction model is a model constructed by a sequence pattern. The performance prediction model is then trained based on the training data set to obtain a candidate performance prediction model, and the prediction result of the candidate performance prediction model is verified based on the verification data set to obtain a prediction error value, and the candidate performance prediction model is determined to be a trained performance prediction model when the prediction error value is less than a preset error threshold. In this way, the sequence pattern construction model obtains a trained performance prediction model, which can better understand and predict the future performance data of the target application when performing performance prediction, thereby improving the accuracy of predicting the performance data of the target application, and then improving the efficiency of collecting and monitoring the performance data of the target application.

[0108] Continue to see Figure 3A , and the description continues with step 301 above.

[0109] In step 302, when the prediction result meets the collection condition, a performance collection instruction is sent to the performance detection client.

[0110] Here, the collection condition may be that the future interface time consumption exceeds the pre-collection threshold, and the pre-collection threshold may be a reference time consumption set in advance. The collection condition is used to determine whether to collect the performance data of the target application. When the prediction result meets the collection condition, it means that the target application is predicted to have a future interface time consumption that exceeds the pre-collection threshold, that is, the performance data of the target application may have performance glitches in the future. Therefore, a performance collection instruction is sent to the performance detection client to timely collect the performance data of the target application at the performance glitches stage. Among them, the performance collection instruction includes: collection sequence number information, application identification information of the target application, and process identification information of the target application. The collection sequence number information is used to assign a unique sequence number or identifier to this performance collection instruction, which is used to track and manage the performance collection instruction to ensure the uniqueness and traceability of the performance collection instruction. The collection sequence number information includes an identification part and a data part, for example, the collection sequence number (identification part): 20240326093655500001 (data part). The application identification information of the target application is used to indicate the name of the target application, and also includes an identification part and a data part, for example: application (identification part): XX (data part), where XX is the name of the target application. The process identification information of the target application is used to indicate the process ID (Process ID) of the process of the target application running in the operating system, and also includes an identification part and a data part, for example: process (identification part): YY (data part), where YY is the process ID.

[0111] In some embodiments, since the performance collection instruction includes the collection sequence number information and the process identification information of the target application, after sending the performance collection instruction, a collection instruction execution record can also be generated based on the collection sequence number information, the process identification information and the first state information, wherein the first state information indicates that data collection is being performed on the target application; and then the collection instruction execution record is stored in the collection instruction execution table. Based on this, after the performance collection data is subsequently stored, the first state information in the collection instruction execution record corresponding to the performance collection data can be updated to the second state information, wherein the second state information indicates that data collection of the target application has been completed.

[0112] Here, the first state information is used to characterize that data collection is being performed on the target application, such as "doing", and the second state information is used to characterize that data collection on the target application has been completed, such as "done". A collection instruction execution record is generated based on the collection sequence number corresponding to the collection sequence number information in the performance collection instruction, the process identifier corresponding to the process identifier information of the target application, and the first state information, and then stored in the collection instruction execution table. After the performance collection data is stored, the first state information in the collection instruction execution record corresponding to the performance collection data is updated to the second state information, for example, "doing" is updated to "done", indicating that data collection on the target application has been completed. In this way, by saving the collection instruction execution record each time data is collected on the target application through the collection instruction execution table, each collection instruction and task can be stored and tracked during the data collection process, thereby improving the efficiency, traceability, reliability and security of collecting performance data of the target application.

[0113] In step 303, the performance collection data collected by the performance detection client is received and stored.

[0114] Here, after receiving the performance collection instruction, the performance detection client (PPROF-AGENT system) will collect data from the target application according to the performance collection instruction, obtain performance collection data, and send the performance collection data to the performance detection server. The performance detection server receives the performance collection data and stores the performance collection data in the time series database.

[0115] In some embodiments, see Figure 3C , the "storing the performance collection data" in step 303 can be implemented through steps 3031 to 3033, which are described in detail below:

[0116] In step 3031, verification parameters for verification are obtained from the performance collection data, and primary key data and key-value pair data to be stored are determined based on the performance collection data.

[0117] Here, by parsing the performance collection data, the collection sequence number information, application identification information, process identification information, collection time information, host address information and collection result file information in the performance collection data are obtained. Among them, the collection time information is used to indicate the time when data is collected for the target application, including an identification part and a data part, for example, the collection time (identification part): 2024.04.04.12:00 (data part). The host address information is used to indicate the IP address (Internet Protocol Address) of the host where the performance detection client that sends the performance collection data is located, for example, the host address: 10.0.0.1. The collection result file information is used to indicate the collection result file of the target application, including an identification part (such as a collection result file) and a collection result file, wherein the file format of the collection result file is JFR.

[0118] In some embodiments, the verification parameters include collection sequence number information and process identification information. Based on this, obtaining the verification parameters from the performance collection data is to obtain the collection sequence number information and process identification information from the performance collection data.

[0119] In some embodiments, the time data of the collection time information in the performance collection data is determined as the primary key data, and corresponding key-value pair data are respectively constructed based on the application identification information, host address information and collection result file information in the performance collection data.

[0120] Here, the time data in the acquisition time information is determined as the primary key data, and the corresponding key-value pair data is constructed. Among them, the primary key data is used to uniquely identify the column or column combination of each row of data in the time series database. The role of the primary key data is to ensure the uniqueness of each row of data and allow the acquisition result records to be retrieved and operated quickly and effectively. For example, if the acquisition time information is, acquisition time: 02161200, then the time data 02161200 is determined as the primary key data. The application identification information is, application identification: application A, then the key-value pair data is constructed with "application identification" as the key and "application A" as the value; the host address information is, host address: 192.168.1.1, then the key-value pair data is constructed with "host address" as the key and "192.168.1.1" as the value; the acquisition result file information is, acquisition result file: XX.JFR, then the key-value pair data is constructed with "acquisition result file" as the key and "XX.JFR" as the value.

[0121] In step 3032, parameter verification is performed based on the verification parameters to obtain a verification result.

[0122] Here, the acquisition instruction execution table is first obtained, and then the acquisition instruction execution record corresponding to the performance acquisition data is searched in the acquisition instruction execution table based on the verification parameters to obtain the search result. Since the verification parameters include the acquisition sequence number information and the process identification information of the target application, the implementation process of step 3032 includes: based on the acquisition sequence number information and the process identification information, the acquisition instruction execution record corresponding to the performance acquisition data is searched in the acquisition instruction execution table; if the search is successful, a verification result of passing the verification is generated.

[0123] In some embodiments, by searching the acquisition instruction execution record corresponding to the performance acquisition data in the acquisition instruction execution table, a search result can be obtained. The search result is used to indicate whether the acquisition instruction execution table includes the acquisition instruction execution record corresponding to the performance acquisition data. In the case where the search result indicates that the acquisition instruction execution table includes the acquisition instruction execution record corresponding to the performance acquisition data, it means that the performance acquisition data is obtained by executing the acquisition task corresponding to the acquisition instruction execution record, so the parameter verification passes, and a verification result indicating that the verification passes is obtained, that is, a verification result indicating that the verification passes is generated. In the case where the search result indicates that the acquisition instruction execution table does not include the acquisition instruction execution record corresponding to the performance acquisition data, it means that the performance acquisition data is not obtained by executing the acquisition task corresponding to the acquisition instruction execution record, so the parameter verification fails, and a verification result indicating that the verification fails is obtained. In this way, by searching the acquisition instruction execution record corresponding to the performance acquisition data in the acquisition instruction execution table, it can be ensured that the performance acquisition data is obtained by executing the acquisition task corresponding to the acquisition instruction execution record, thereby ensuring the reliability and effectiveness of the performance acquisition data and improving the security and efficiency of collecting performance data of the target application.

[0124] In step 3033, when the verification result indicates that the verification has passed, a collection result record is generated based on the primary key data and the key-value pair data, and the collection result record is stored in the time series database.

[0125] Here, when the search result represents that the collection instruction execution table includes the collection instruction execution record corresponding to the performance collection data, that is, when the verification result represents the verification passed, the collection result record corresponding to the performance collection data is generated based on the primary key data and key-value pair data corresponding to this performance collection data, and stored in the time series database, so that the user can query the corresponding collection result record according to the time information, application name and host IP, and obtain the corresponding collection result file.

[0126] In an embodiment of the present application, verification parameters including collection sequence number information and process identification information are obtained from the performance collection data, and the time data of the collection time information in the performance collection data is determined as the primary key data. Based on the application identification information, host address information and collection result file information in the performance collection data, corresponding key-value pair data are respectively constructed; then, parameter verification is performed based on the verification parameters to obtain the verification result; when the verification result indicates that the verification has passed, a collection result record is generated based on the primary key data and the key-value pair data, and the collection result record is stored in a time series database. In this way, the reliability of the performance collection data can be guaranteed by verifying the performance collection data, and the collection result record is generated by the primary key data and the key-value pair data, and the collection result record is stored in a time series database, which can help users better manage and use the performance collection data and improve the monitoring effect of the performance data of the target application.

[0127] In an embodiment of the present application, by receiving the real-time performance data of the target application sent by the application performance management system, the performance of the target application is predicted based on the real-time performance data of the target application to obtain a prediction result; when the prediction result meets the collection condition, a performance collection instruction is sent to the performance detection client, and then the performance collection data collected by the performance detection client is received, and the performance collection data is stored. In this way, the future performance of the target application is predicted by the real-time performance data of the target application, and the future performance data of the target application can be accurately grasped in advance. When the prediction result meets the collection condition, the performance collection instruction is sent to the performance detection client, that is, when it is predicted that the future interface time will exceed the pre-collection threshold, the performance collection instruction is sent to the performance detection client, so that when the performance data of the target application may have performance glitches, the performance data of the target application can be timely triggered to collect, so as to avoid the failure to collect the performance data of the target application in the performance glitches stage in time due to the short duration of the glitches, thereby achieving the purpose of on-demand, efficient and timely performance collection. At the same time, when the performance prediction result of the target application does not meet the collection conditions, there is no need to send a performance collection instruction. That is, when it is predicted that the future interface time does not exceed the pre-collection threshold, the performance data characterizing the target application will not have performance glitches in the future. In this case, there is no need to send a performance collection instruction, that is, the collection of performance data will not be triggered, which can save device resources and storage costs.

[0128] In some embodiments, the collection result record includes target key-value pair data corresponding to the collection result file information, and the value of the target key-value pair data is the collection result file. Based on this, see Figure 3D , the query of the collection result file can be realized through steps 501 to 503, which are described in detail below:

[0129] In step 501, based on the collection result query information, a target collection result record corresponding to the collection result query information is determined in a time series database.

[0130] In the embodiment of the present application, the acquisition result query information is first obtained. The acquisition result query information includes: target time information, target application identifier and target host address. The target time information can be a specific time point or a time range, which is used to indicate the acquisition time corresponding to the acquisition result file to be queried, the target application identifier indicates the application name of the application where the acquisition result file to be queried is located, and the target host address indicates the IP address of the host where the acquisition result file to be queried is located.

[0131] Here, since the time series database may include multiple collection result records, each collection result record has corresponding primary key data determined according to the time data, the collection result records can be quickly and effectively retrieved from the time series database according to the target time information in the collection result query information to find the collection result records that meet the target time information. If there is only one collection result record that meets the target time information, this collection result record can be directly determined as the target collection result record. If there are multiple collection result records that meet the target time information, it is necessary to further determine the collection result record that matches the target application identifier and the target host address according to the application identifier or host address in each collection result record, and determine it as the target collection result record.

[0132] In step 502, it is determined that the collection result file in the target collection result record is the target collection result file.

[0133] Here, the target collection result record includes key-value pair data corresponding to the application identification information, the host address information and the collection result file information, and the value in the key-value pair data corresponding to the collection result file information is determined as the target collection result file, wherein the value in the key-value pair data corresponding to the collection result file information is the collection result file in the target collection result record, that is, the collection result file in the target collection result record is determined as the target collection result file.

[0134] In step 503, the target acquisition result file is converted into a preset format to obtain a query result, and the query result is output.

[0135] Here, the preset format is a format that can intuitively and accurately display the performance data of the application, which is convenient for users to analyze and optimize the performance data, and helps users identify and locate performance bottlenecks and optimization points in the program. For example, the preset format can be the format of a flame graph that can be displayed on a browser, the format of a performance analysis graph, and the format of a performance analysis table. Since the target acquisition result file is in JFR format and the data therein is in binary format, it is necessary to convert the format of the target acquisition result file into a flame graph format as a query result and output it, so that users can analyze and optimize the target application according to the query results.

[0136] In the embodiment of the present application, acquisition result query information is obtained, and the target acquisition result record corresponding to the acquisition result query information is determined in the time series database; the acquisition result file in the target acquisition result record is determined to be the target acquisition result file, and the target acquisition result file is converted into a preset flame graph format to obtain the query result, and the query result is output. In this way, the flame graph facilitates users to analyze and optimize performance data, helps users identify and locate performance bottlenecks and optimization points in the program, thereby improving the execution efficiency and speed of the program, and thus achieving the effect of monitoring the performance data of the target application.

[0137] The data processing method provided by the embodiment of the present application is described below, taking the electronic device implementing the data processing method of the embodiment of the present application as a performance collection client as an example. Therefore, the execution subject of each step will not be repeatedly described below.

[0138] See also Figure 4 , Figure 4 This is a schematic diagram of a process for data processing by a performance detection client provided in an embodiment of the present application, which is described in detail below:

[0139] In step 601, a performance collection instruction sent by a performance detection server is received.

[0140] Here, the performance collection instruction is sent when the prediction result of the performance prediction of the target application meets the collection condition, and the performance collection instruction is used to instruct the performance collection client to collect data of the target application.

[0141] In step 602, data is collected on the target application based on the performance collection instruction to obtain performance collection data.

[0142] Here, the acquisition sequence number information, the application identification information of the target application, the process identification information corresponding to the target application, and the process identification list of the host address where the performance detection client is located can be first obtained in the performance acquisition instruction. Then, it is determined that the process identification in the process identification information is included in the process identification list. If the process identification in the process identification information is not included in the process identification list, it means that the target application is not running on the host where the performance prediction client is located, so an error prompt information is returned. In the case where the process identification in the process identification information is included in the process identification list, data is collected for the target application based on the performance acquisition instruction, a collection result file is obtained, and collection result file information is generated. Then, the host address information is determined based on the host address where the performance detection client is located, and the collection time information is determined based on the time information when the data is collected for the target application. For example, the host address where the performance detection client is located is 192.168.1.1, and the time information when the data is collected for the target application is 12:00:02, then the host address information is "host address: 192.168.1.1", and the collection time information is "collection time: 12:00:02". Finally, the collection sequence number information, application identification information, process identification information, collection result file information, host address information and collection time information are determined as performance collection data.

[0143] In step 603, the performance collection data is sent to the performance detection server.

[0144] In an embodiment of the present application, a performance collection instruction sent by a performance detection server is received. The performance collection instruction is sent when the prediction result of the performance prediction of the target application meets the collection conditions; data is collected on the target application based on the performance collection instruction to obtain performance collection data, and sent to the performance detection server. In this way, the performance collection instruction sent by the performance detection server will be received only when the prediction result of the performance prediction of the target application meets the collection conditions, that is, when it is predicted that the future interface time of the target application will exceed the pre-collection threshold, the performance collection instruction sent by the performance detection server will be received, thereby triggering data collection, and the performance data of the target application can be collected in time when the performance data of the target application may have performance glitches, avoiding the failure to collect the performance data of the target application in time during the performance glitches stage due to the short duration of the glitches. At the same time, in the case where the performance prediction result of the target application does not meet the collection conditions, data collection is not required, which can save equipment resources and storage costs.

[0145] The following describes a data processing method provided in an embodiment of the present application, taking an example in which the electronic device implementing the data processing method in an embodiment of the present application is a performance collection server and a performance collection client.

[0146] See also Figure 5 , Figure 5 This is a schematic diagram of the interactive process of performing data processing by a performance detection server and a performance detection client provided in an embodiment of the present application, which is described in detail below:

[0147] First, in step 701, the performance collection server receives the real-time performance data of the target application sent by the application performance management system; in step 702, the performance collection server performs performance prediction based on the real-time performance data of the target application to obtain a prediction result. In step 703, the performance collection server determines whether the prediction result meets the collection condition. In step 704, if the prediction result meets the collection condition, the performance collection server sends a performance collection instruction to the performance detection client; if the prediction result does not meet the collection condition, it is not necessary to send the performance collection instruction to the performance detection client, and the process can be directly ended, that is, entering step 708. After receiving the performance collection instruction, the performance collection client performs data collection on the target application based on the performance collection instruction in step 705 to obtain performance collection data, and then in step 706, the performance collection client sends the performance collection data to the performance collection server. After receiving the performance collection data, the performance collection server stores the performance collection data in step 707. In this way, data collection will only be performed when the prediction results of the performance prediction of the target application meet the collection conditions. The performance data of the target application can be collected in time when performance glitches may occur in the performance data of the target application, avoiding the failure to collect the performance data of the target application in time during the performance glitches due to the short duration of the glitches. At the same time, data collection is not required when the performance prediction results of the target application do not meet the collection conditions, which can save equipment resources and storage costs. In addition, by storing the performance collection data, it can ensure that the performance data of the target application is accurately analyzed based on the stored performance collection data, thereby assisting in locating performance hotspots, thereby improving the efficiency of collecting and monitoring the performance data of the target application.

[0148] Below, an exemplary application of the embodiment of the present application in an actual application scenario will be described. In the related art, an open source APM system is usually introduced to automatically add a call link tracking identifier to the application through an APM plug-in. By analyzing the operation data related to each application service associated with the call link collected by the APM system, application service monitoring is provided. Figure 6 is a first flow chart of the practical application of the data processing method provided in the embodiment of the present application, such as Figure 6As shown, it includes two parts, application instance 61 and application performance management system 62. The application performance management plug-in adds corresponding call link tracking identifiers to application A instance and application B instance to obtain application C instance and application D instance. Then, the link collector 621 collects the operation data related to each application service transmitted by the call link and saves it to the database 622. The user can query the operation data related to each application service stored in the database 622.

[0149] Figure 7 This is a second flow chart of the practical application of the data processing method provided in the embodiment of the present application, see Figure 7 The host includes microservice A, microservice B and performance detection client 71 (PPROF-Agent). The performance detection client 71 is responsible for monitoring the performance collection instructions issued by the performance detection server 200 (PPROF system), and performs data collection on microservice A and microservice B according to the performance collection instructions, and uploads the collected performance collection data (PPROF file) to the performance detection server 200.

[0150] The performance detection server 200 includes a prediction part 721, a collection part 722, a query part 723 and a time series database 724. The prediction part 721 performs performance prediction on the target application through the received real-time performance data of the target application to obtain the prediction result, and issues a performance collection instruction when the prediction result meets the collection condition, that is, the performance collection instruction is sent only when it is predicted that the future interface time consumption will exceed the pre-collection threshold. The collection part 722 is responsible for receiving the performance collection data reported by the performance detection client 71, and saving it to the time series database 724 according to the time characteristics of the performance collection data. The query part 723 queries the corresponding target collection result file according to the collection result query information input by the user (including: target time information, target application identifier and target host address), and can convert the target collection result file into a flame graph 73 online, so that the user can locate the code hotspot.

[0151] First, in the performance testing client:

[0152] First, a performance collection instruction sent by the performance detection server is received. Here, the performance collection instruction format adopts JSON format, and the collection sequence number information, the application identification information of the target application and the process identification information (process ID) corresponding to the target application are obtained by parsing JSON.

[0153] Exemplary:

[0154] {

[0155] no: "20240326093655500001", / / Collection serial number information

[0156] app: "app-hello", / / application identification information

[0157] pid: "10000101" / / process identification information

[0158] }

[0159] Then, obtain the process ID list (JAVA process ID list) of the host address where the performance detection client is located, and determine whether there is a process ID corresponding to the target application in the process ID list. If not, return the error message of the command "processing failed", otherwise continue to execute the data collection task.

[0160] Afterwards, an asynchronous thread is started to use async-profiler to perform short-term data collection on the target application and generate collection result file information (wherein the file format of the collection result file is defined as JFR).

[0161] Finally, the performance collection data including the collection result file information is reported to the performance detection server through an asynchronous thread to complete the data collection. The performance collection data includes: collection sequence number information, application identification information, process identification information, collection result file information, host address information and collection time information.

[0162] Second, in the performance testing server:

[0163] The first part, prediction, includes:

[0164] First, data collection and preprocessing are performed, including obtaining real-time performance data of the target application from the APM system. The data mainly include "CPU usage, disk I / O, network traffic, system load, number of concurrent requests, number of active threads, heap memory percentage", etc., and the data are saved as the initial data set in the order of data collection time information. Then, the prediction module cleans and normalizes the initial data set, and divides the standardized initial data into "training data set" and "validation data set".

[0165] Then, a performance prediction model is constructed and trained, including: using the sequence mode of Keras to define the performance prediction model to be trained, setting parameters such as the number of network layers of the performance prediction model to be trained, and specifying the optimizer and loss function of the training of the performance prediction model to be trained, wherein the performance prediction model to be trained corresponds to the performance prediction model in other embodiments. The performance prediction model to be trained is trained by the training data set to obtain a candidate performance prediction model. After training, the prediction results of the candidate performance prediction model are verified based on the verification data set to obtain a prediction error value. If the prediction error value is greater than a preset error threshold or overfitting, the parameters of the performance prediction model to be trained are readjusted and retrained. By repeatedly adjusting the parameters, retraining, performing verification and evaluating the error, until the prediction error value is less than the preset error threshold, a trained performance prediction model is obtained.

[0166] Finally, the target application is predicted for performance, including: using the trained performance prediction model, inputting the real-time performance data of the target application for performance prediction, and obtaining the prediction results. When the prediction results meet the collection conditions, indicating that the future interface time consumption will exceed the pre-collection threshold, that is, it is believed that the target application may have slow requests, CPU increase, etc. in the future, it is necessary to immediately send a "performance collection instruction" to the performance detection client. At the same time, the collection instruction execution record is generated and saved in the "Collection Command Execution Table", as shown in Table 1:

[0167] Table 1. Collection command execution table

[0168] Collection serial number Application Identification Process ID state 20240326093655500001 app-hello 100101 doing

[0169] The second part, the Collector, includes:

[0170] First, the performance collection data sent by the performance detection client is received, and the performance collection data is parsed to obtain: collection sequence number information, application identification information, process identification information, collection time information, host address information and collection result file information. Then, based on the collection sequence number information and process identification information, the collection instruction execution record corresponding to the performance collection data is searched from the collection command execution table. If it is found, it means that the parameter verification has passed. After that, after the parameter verification passes, the time data in the collection time information is determined as the primary key, and the corresponding key-value pairs are respectively constructed based on the application identification information, the host address information and the collection result file information. The collection result record is generated based on the primary key and the key-value pair and saved in the time series database. Finally, the status of the record in the collection command execution table is updated to the second status information, wherein the primary key corresponds to the primary key data in other embodiments, and the key-value pair corresponds to the key-value pair data in other embodiments.

[0171] The third part, the query part (Query), includes:

[0172] First, quickly locate the corresponding target collection result record according to the collection result query information entered by the user. Then, use the async-profiler conversion tool to convert the target collection result file in the target collection result record into a flame graph that can be displayed on the browser in real time. Finally, users can view the data in the flame graph to help locate the time-consuming hotspot code.

[0173] In the embodiment of the present application, the application side does not need to make any modifications, nor does it need to perform continuous performance collection and timed performance collection on the application. By inputting the real-time performance data of the target application collected in the APM system into the trained performance prediction model for performance prediction, data collection is performed when it is calculated that the future interface time consumption will exceed the set threshold, thereby reducing the storage cost of the collected data and solving the problem that the timed collection and the burr time point cannot be well matched. At the same time, the timing characteristics of the time series database are used to save the performance collection data, which is convenient for the rapid positioning of the performance collection data by time period in the later stage.

[0174] The following further describes an exemplary structure of the data processing device 233 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2A As shown, the software modules stored in the data processing device 233 of the memory 230 may include:

[0175] The prediction module 2331 is used to perform performance prediction based on the real-time performance data of the target application to obtain a prediction result.

[0176] The first sending module 2332 is used to send a performance collection instruction to the performance detection client when the prediction result meets the collection condition.

[0177] The first receiving module 2333 is used to receive the performance collection data collected by the performance detection client and store the performance collection data.

[0178] In some embodiments, the first receiving module 2333 is also used to obtain verification parameters for verification from the performance collection data, and determine the primary key data and key-value pair data that need to be stored based on the performance collection data; perform verification based on the verification parameters to obtain a verification result; when the verification result indicates that the verification has passed, generate a collection result record based on the primary key data and the key-value pair data, and store the collection result record in a time series database.

[0179] In some embodiments, the verification parameters include collection sequence number information and process identification information of the target application, and the first receiving module 2333 is further used to search for the collection instruction execution record corresponding to the performance collection data in the collection instruction execution table based on the collection sequence number information and the process identification information; if the search is successful, a verification result of verification passing is generated.

[0180] In some embodiments, the performance collection instruction includes collection sequence number information and process identification information of the target application, and the software module in the data processing device 233 may also include:

[0181] A generating module, configured to generate a collection instruction execution record based on the collection sequence number information, the process identification information and first status information, wherein the first status information indicates that data collection is being performed on the target application;

[0182] A storage module, used for storing the acquisition instruction execution record in an acquisition instruction execution table;

[0183] The updating module is used to update the first state information in the collection instruction execution record corresponding to the performance collection data to the second state information, wherein the second state information indicates that the data collection of the target application has been completed.

[0184] In some embodiments, the prediction result is obtained by prediction using a performance prediction model, and the software module in the data processing device 233 may further include:

[0185] A first determination module is used to determine a training data set and a validation data set according to sample performance data;

[0186] A training module, used to construct a performance prediction model through a sequence pattern, and train the performance prediction model based on the training data set to obtain a candidate performance prediction model;

[0187] A verification module, used to verify the prediction result of the candidate performance prediction model based on the verification data set to obtain a prediction error value;

[0188] The second determination module is used to determine that the candidate performance prediction model is the trained performance prediction model when the prediction error value is less than a preset error threshold.

[0189] In some embodiments, the collection result record includes target key-value pair data corresponding to the collection result file information, and the value of the target key-value pair data is the collection result file. The software module in the data processing device 233 may also include:

[0190] A third determination module is used to determine, in the time series database, a target collection result record corresponding to the collection result query information based on the collection result query information;

[0191] A fourth determination module, configured to determine that the collection result file in the target collection result record is the target collection result file;

[0192] The conversion module is used to convert the target acquisition result file into a preset format, obtain the query result, and output the query result.

[0193] The following is a description of an exemplary structure of the data processing device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2B As shown, the software modules stored in the data processing device 455 of the memory 450 may include:

[0194] The second receiving module 4551 is used to receive a performance collection instruction sent by the performance detection server, where the performance collection instruction is sent when the prediction result of the performance prediction of the target application meets the collection condition.

[0195] The collection module 4552 is used to collect data on the target application based on the performance collection instruction to obtain performance collection data.

[0196] The second sending module 4553 is used to send the performance collection data to the performance detection server.

[0197] In some embodiments, the collection module 4552 is also used to obtain the collection sequence number information included in the performance collection instruction, the application identification information and process identification information of the target application, and the process identification list of the host address where the performance detection client is located; when the process identification information of the target application is included in the process identification list, data is collected for the target application to obtain a collection result file, and collection result file information is generated; the host address information is determined based on the host address where the performance detection client is located, and the collection time information is determined based on the time information when data is collected for the target application; the collection sequence number information, the application identification information, the process identification information, the collection result file information, the host address information and the collection time information are determined as the performance collection data.

[0198] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the data processing method described above in the embodiment of the present application.

[0199] The present application embodiment provides a computer-readable storage medium, in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will execute the data processing method provided by the present application embodiment, for example, Figure 3A or Figure 4 The data processing method is shown.

[0200] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0201] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0202] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0203] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0204] In summary, through the embodiment of the present application, the application end does not need to make any modifications, nor does it need to perform continuous performance collection and timed performance collection on the application. By inputting the real-time performance data of the target application collected in the APM system into the trained performance prediction model for performance prediction, data collection is performed when the performance prediction result of the target application meets the collection conditions, that is, when it is determined that the future interface time consumption will exceed the set threshold, data collection is performed, thereby reducing the storage cost of the collected data, and also solving the problem that the timed collection and the burr time point cannot be well matched. At the same time, if the performance prediction result of the target application does not meet the collection conditions, the performance collection instruction is not sent, which can save equipment resources and storage costs. By receiving the performance collection data collected by the performance detection client in response to the performance collection instruction and storing the performance collection data, it can ensure that the performance data of the target application is accurately analyzed according to the stored performance collection data, thereby assisting in locating performance hotspots, thereby improving the efficiency of collecting and monitoring the performance data of the target application.

[0205] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Perform performance prediction based on the real-time performance data of the target application to obtain prediction results; When the prediction result meets the collection condition, sending a performance collection instruction to the performance detection client; Receive the performance collection data collected by the performance detection client, and store the performance collection data.

2. The method according to claim 1, characterized in that The storing of the performance collection data includes: Acquire verification parameters for verification from the performance collection data, and determine primary key data and key-value pair data to be stored based on the performance collection data; Perform verification based on the verification parameters to obtain a verification result; When the verification result indicates that the verification has passed, a collection result record is generated based on the primary key data and the key-value pair data, and the collection result record is stored in a time series database.

3. The method according to claim 2, characterized in that The verification parameters include the collection sequence number information and the process identification information of the target application, and the verification based on the verification parameters to obtain the verification result includes: Based on the collection sequence number information and the process identification information, searching the collection instruction execution record corresponding to the performance collection data in the collection instruction execution table; If the search is successful, a verification result indicating that the verification has passed is generated.

4. The method according to claim 1, characterized in that: The performance collection instruction includes collection sequence number information and process identification information of the target application. After sending the performance collection instruction to the performance detection client, the method further includes: generating a collection instruction execution record based on the collection sequence number information, the process identification information and the first status information, wherein the first status information indicates that data collection is being performed on the target application; Storing the acquisition instruction execution record in an acquisition instruction execution table; After storing the performance collection data, the method further includes: The first state information in the collection instruction execution record corresponding to the performance collection data is updated to the second state information, where the second state information indicates that the data collection for the target application has been completed.

5. The method according to any one of claims 1 to 4, characterized in that: The prediction result is obtained by predicting the performance prediction model, and the performance prediction model is trained in the following way: Determine a training data set and a validation data set based on sample performance data; Constructing a performance prediction model through a sequence pattern, and training the performance prediction model based on the training data set to obtain a candidate performance prediction model; Verifying the prediction result of the candidate performance prediction model based on the verification data set to obtain a prediction error value; When the prediction error value is less than a preset error threshold, the candidate performance prediction model is determined to be a trained performance prediction model.

6. The method according to claim 2, characterized in that The collection result record includes target key-value pair data corresponding to the collection result file information, and the value of the target key-value pair data is the collection result file. The method further includes: Based on the collection result query information, determining a target collection result record corresponding to the collection result query information in the time series database; Determine that the collection result file in the target collection result record is the target collection result file; The target acquisition result file is converted into a preset format to obtain a query result, and the query result is output.

7. A data processing method, characterized in that: The method comprises: Receiving a performance collection instruction sent by the performance detection server, wherein the performance collection instruction is sent when a prediction result of the performance prediction of the target application meets a collection condition; Performing data collection on the target application based on the performance collection instruction to obtain performance collection data; The performance collection data is sent to the performance detection service end.

8. The method according to claim 7, characterized in that The performing data collection on the target application based on the performance collection instruction to obtain performance collection data includes: Acquire the acquisition sequence number information included in the performance acquisition instruction, the application identification information and process identification information of the target application, and the process identification list of the host address where the performance detection client is located; In a case where the process identification information of the target application is included in the process identification list, data collection is performed on the target application to obtain a collection result file, and collection result file information is generated; Determine host address information based on the host address where the performance detection client is located, and determine collection time information based on time information when data is collected for the target application; The collection sequence number information, the application identification information, the process identification information, the collection result file information, the host address information and the collection time information are determined as the performance collection data.

9. A data processing device, characterized in that: The device comprises: A prediction module is used to perform performance prediction based on the real-time performance data of the target application and obtain prediction results; A first sending module, configured to send a performance collection instruction to a performance detection client when the prediction result meets the collection condition; The first receiving module is used to receive the performance collection data collected by the performance detection client and store the performance collection data.

10. A data processing device, characterized in that: The device comprises: A second receiving module is used to receive a performance collection instruction sent by the performance detection server, wherein the performance collection instruction is sent when the prediction result of the performance prediction of the target application meets the collection condition; A collection module, used for collecting data from the target application based on the performance collection instruction to obtain performance collection data; The second sending module is used to send the performance collection data to the performance detection server.

11. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions; A processor, configured to implement the method of any one of claims 1 to 6 or 7 and 8 when executing the computer executable instructions stored in the memory.

12. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer programs are executed by a processor, the method according to any one of claims 1 to 6 or 7 and 8 is implemented.

13. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 6 or 7 and 8 is implemented.

Images