Intelligent evaluation method and system for software system performance data

Abstract

The invention discloses an intelligent evaluation method and system for software system performance data. The method comprises the steps of collecting system performance original data; performing data cleaning and processing on the system performance original data to obtain system performance intermediate result data; performing feature construction and analysis on the system performance intermediate result data to obtain system performance analysis data; constructing a system performance training set and a system performance test set based on the system performance analysis data; inputting the system performance training set into the large language model for training to obtain a system performance evaluation model; and inputting the system performance test set into the system performance evaluation model and outputting a system performance evaluation result. According to the method, the software performance data is efficiently collected, processed and analyzed, the training set is constructed, and the performance analysis model is constructed by using the large language model, so that the efficiency of the evaluation process is greatly improved, manual intervention is reduced, and a more objective and accurate performance evaluation result is provided.

Description

Technical Field

[0001] The present invention belongs to the technical field of software system performance evaluation, and in particular relates to an intelligent evaluation method and system for software system performance data. Background Art

[0002] In today's era of rapid development of information technology, system performance evaluation and optimization are crucial for key applications in various industries. The purpose of software system performance evaluation is to ensure that the software can meet performance requirements under the predetermined workload, including response time, throughput, stability and reliability. This is crucial for software user experience, market competitiveness and operational efficiency. Through performance evaluation, developers and operation and maintenance teams can identify and solve potential performance bottlenecks, optimize system design, and improve resource utilization, thereby ensuring that the software system can run stably and efficiently. However, traditional software performance evaluation methods usually rely on manual collection, analysis and interpretation of large amounts of performance data. This method has many problems such as inefficiency, strong subjectivity, time-consuming and labor-intensive. Summary of the invention

[0003] To solve the above problems, the present invention provides an intelligent evaluation method and system for software system performance data to solve the problems that traditional software system performance evaluation methods rely on manual collection, analysis and interpretation of a large amount of performance data, which are inefficient, highly subjective, time-consuming and labor-intensive.

[0004] To achieve the above object, the present invention provides a method and system for intelligently evaluating software system performance data, comprising:

[0005] Collect raw data on system performance;

[0006] Clean and process the original data of system performance to obtain the intermediate result data of system performance;

[0007] Construct and analyze the characteristics of the intermediate results of system performance to obtain system performance analysis data;

[0008] Build system performance training set and system performance test set based on system performance analysis data;

[0009] Input the system performance training set into the large language model for training to obtain a system performance evaluation model;

[0010] The system performance test set is input into the system performance evaluation model and the system performance evaluation results are output.

[0011] According to a specific embodiment of the present invention, collecting system performance raw data includes:

[0012] Use PromQL query language to obtain CPU data, memory data, disk data, network data, JVM data and Gatling performance process data from the monitoring system in real time, including:

[0013] CPU data includes CPU overall data, system kernel data and I / O wait data;

[0014] Memory data includes free memory, buffer memory, cache memory, and the memory usage ratio of each instance in the total memory;

[0015] Disk data includes disk read rate and disk write rate;

[0016] Network data includes network received data and network sent data;

[0017] JVM data includes the JVM old generation memory usage ratio, Metaspace memory usage ratio, the total first major GC pause time, and the total second major GC pause time. The total first major GC pause time is the total major GC pause time caused by allocation failure in the JVM, and the total second major GC pause time is the total major GC pause time caused by reaching the MetadataGC threshold in the JVM.

[0018] Gatling performance process data includes log files generated by Gatling stress testing.

[0019] According to a specific embodiment of the present invention, data cleaning and processing of the original system performance data to obtain the intermediate system performance result data further includes:

[0020] Extract necessary information data from the original data of system performance and remove redundant data;

[0021] The necessary information data of the system performance original data is formatted with timestamps and numerical processing, and the necessary information data of different data types are unified in data format to obtain the system performance intermediate result data.

[0022] According to a specific embodiment of the present invention, the system performance intermediate result data includes CPU intermediate result data, memory usage intermediate result data, disk intermediate result data, network intermediate result data, JVM intermediate result data and gatling performance process intermediate data, wherein the disk intermediate result data also includes disk read rate intermediate result data and disk write rate intermediate result data, and the network intermediate result data also includes network receiving rate intermediate result data and network sending rate intermediate result data.

[0023] According to a specific embodiment of the present invention, the system performance intermediate result data is subjected to feature construction and analysis, and the system performance analysis data obtained includes:

[0024] Extract multiple characteristic parameters of the system performance intermediate result data and perform multi-dimensional analysis and calculation to obtain multiple key performance indicators of the system performance intermediate result data;

[0025] Multiple key performance indicators are aggregated to obtain system performance analysis data.

[0026] According to a specific embodiment of the present invention, multiple characteristic parameters of the system performance intermediate result data are extracted and multi-dimensional analysis and calculation are performed to obtain multiple key performance indicators of the system performance intermediate result data, including:

[0027] Extract the overall CPU usage, system kernel usage, and I / O wait data usage from the CPU intermediate result data, and perform multi-dimensional analysis and calculation based on the overall CPU usage, system kernel usage, and I / O wait data usage to obtain the average and standard deviation of the overall CPU usage, the average system kernel usage, the average I / O wait data usage, and the change trend of the overall CPU usage;

[0028] Extract the collection time and overall memory usage rate from the intermediate memory usage result data, and perform multi-dimensional analysis and calculation based on the collection time and overall memory usage rate to obtain the average value of the overall memory usage rate, the standard deviation of the overall memory usage rate, and the change trend of the overall memory usage rate;

[0029] Extract the disk read rate and its acquisition time, disk write rate and its acquisition time, disk read rate and disk write rate from the disk intermediate result data, and perform multi-dimensional analysis and calculation based on the disk read rate and its acquisition time, disk write rate and its acquisition time, disk read rate and disk write rate to obtain the disk read rate average value and standard deviation, disk write rate average value and standard deviation, disk read rate change trend and disk write rate change trend;

[0030] Extract the network receiving rate and its collection time, the network sending rate and its collection time from the network intermediate result data, and perform multi-dimensional analysis and calculation based on the network receiving rate and its collection time, the network sending rate and its collection time to obtain the average value and standard deviation of the network receiving rate, the average value and standard deviation of the network sending rate, the change trend of the network receiving rate, and the change trend of the network sending rate;

[0031] Extract the collection time, JVM old generation memory usage ratio, Metaspace memory usage ratio, the total first major GC pause time, and the total second major GC pause time from the JVM intermediate result data, and perform multi-dimensional analysis and calculation based on the collection time, JVM old generation memory usage ratio, Metaspace memory usage ratio, the total first major GC pause time, and the total second major GC pause time to obtain the average and standard deviation of the JVM old generation memory usage ratio, the ratio of the first major GC pause time to the total duration, the ratio of the second major GC pause time to the total duration, and the change trend of the JVM old generation memory usage ratio;

[0032] Transaction name, request start and end timestamps, and request status are extracted from the intermediate data of the gatling performance process. Multi-dimensional analysis and calculation are performed based on the transaction name, request start and end timestamps, and request status to obtain the average response time and standard deviation of each transaction, the number of completed transactions per second and the response time per second, the transaction linear slope, the average response time stability, and the time series change trend.

[0033] According to a specific embodiment of the present invention, constructing a system performance training set and a system performance test set based on system performance analysis data includes:

[0034] Generate monitoring reports containing key performance indicators based on system performance analysis data;

[0035] Each key performance indicator in the monitoring report is input into the Prompt reasoning template, and potential problem areas are marked according to preset thresholds and corresponding recommended measures are given to obtain a structured system performance training set and system performance test set.

[0036] According to a specific embodiment of the present invention, the system performance evaluation result includes a system performance analysis result and a system performance prediction result.

[0037] An intelligent evaluation system for software system performance data, characterized by comprising:

[0038] Data acquisition module, used to collect raw data of system performance;

[0039] The data cleaning and processing module is used to clean and process the original data of system performance to obtain the intermediate result data of system performance;

[0040] The feature construction and analysis module is used to construct and analyze the features of the intermediate result data of system performance to obtain system performance analysis data;

[0041] A training data construction module is used to construct a system performance training set and a system performance test set based on system performance analysis data;

[0042] The LLM large language model is used to input the system performance training set into the large language model for training to obtain a system performance evaluation model, and to input the system performance test set into the system performance evaluation model and output the system performance evaluation result.

[0043] According to a specific embodiment of the present invention, the data acquisition module further includes:

[0044] CPU data collection submodule, used to collect CPU overall data, system kernel data and I / O wait data;

[0045] The memory data collection submodule is used to collect free memory, buffer memory, cache memory and total memory, and calculate the memory usage ratio of each instance;

[0046] The disk data collection submodule is used to collect disk read rate data and disk write rate data;

[0047] The network data collection submodule is used to collect network received data and network sent data;

[0048] The JVM data collection submodule is used to collect the JVM old generation memory usage percentage, Metaspace memory usage percentage, the first major GC pause time sum, and the second major GC pause time sum. The first major GC pause time sum is the sum of major GC pause times caused by allocation failures in the JVM, and the second major GC pause time sum is the sum of major GC pause times caused by reaching the Metadata GC threshold in the JVM.

[0049] The Gatling performance process data collection submodule is used to collect log files generated by Gatling stress testing.

[0050] Compared with the prior art, the intelligent evaluation method and system for software system performance provided by the present invention has the following advantages:

[0051] 1. Efficiency and automation: The present invention realizes efficient collection, cleaning and analysis of software system performance data through automated data collection and processing and analysis processes, greatly improving the efficiency of the evaluation process, saving manpower and time costs, and reducing the need for manual intervention.

[0052] 2. Objectivity and accuracy: Compared with subjective manual evaluation, the evaluation scheme of the present invention is rule-driven and data-driven, eliminating subjective bias and providing more objective and accurate performance evaluation results.

[0053] 3. Intelligence and continuous optimization: The present invention trains a large model and uses the trained artificial intelligence large model to form a highly optimized performance analysis model for a specific vertical field. The model has the ability of self-learning and continuous optimization. With the passage of time and the accumulation of data, the performance evaluation model of the present invention will be continuously optimized and improved to enhance the credibility and accuracy of the evaluation results.

[0054] 4. Comprehensiveness and multi-dimensional analysis: The present invention can comprehensively collect various data indicators of the system in the performance test, and can perform multi-dimensional analysis to provide users with more comprehensive and in-depth performance evaluation information. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 The present invention is a flowchart of a method for intelligently evaluating software system performance data according to an embodiment of the present invention.

[0057] Figure 2 It is a flow chart of a method for cleaning and processing system performance raw data provided according to an embodiment of the present invention.

[0058] Figure 3 It is a flow chart of a method for feature construction and analysis of system performance intermediate result data provided according to an embodiment of the present invention.

[0059] Figure 4 It is a flow chart of a method for constructing a system performance training set and a system performance test set according to an embodiment of the present invention.

[0060] Figure 5 The invention is a block diagram of an intelligent evaluation system for software system performance provided according to an embodiment of the invention.

[0061] Figure 6 is a structural block diagram of a data acquisition module provided according to an embodiment of the present invention.

[0062] Reference numerals:

[0063] 1-Data collection module; 2-Data cleaning and processing module; 3-Feature construction and analysis module; 4-Training data construction module; 5-LLM large language model;

[0064] 11-CPU data collection submodule; 12-memory data collection submodule; 13-disk data collection submodule; 14-network data collection submodule; 15-JVM data collection submodule; 16-Gatling performance process data collection submodule. DETAILED DESCRIPTION

[0065] In order to make those skilled in the art understand the concept and thought of the present invention more clearly, the present invention is described in detail below in conjunction with specific embodiment.It should be understood that the embodiment provided herein is only a part of all embodiments that the present invention may have.Those skilled in the art, after reading the specification of the application, have the ability to make improvements, transformations, or replacements to part or the whole of the following embodiments, and these improvements, transformations, or replacements are also included in the scope of the present invention.

[0066] In this article, the terms "first", "second" and other similar words are not intended to imply any order, quantity and importance, but are only used to distinguish different elements. In this article, the terms "one", "an" and other similar words are not intended to indicate that there is only one thing, but to indicate that the relevant description is only for one of the things, and the thing may have one or more. In this article, the terms "comprise", "include" and other similar words are intended to indicate logical mutual relations, and cannot be regarded as indicating spatial structural relations. For example, "A includes B" is intended to indicate that B belongs to A logically, but does not mean that B is located inside A in space. In addition, the meanings of the terms "comprise", "include" and other similar words should be regarded as open, rather than closed. For example, "A includes B" is intended to indicate that B belongs to A, but B does not necessarily constitute the whole of A, and A may also include other elements such as C, D, and E.

[0067] In this document, the terms "embodiment", "this embodiment", "one embodiment", and "an embodiment" do not mean that the relevant description is only applicable to a specific embodiment, but rather that the description may also be applicable to one or more other embodiments. Those skilled in the art should understand that in this document, any description of a certain embodiment can be replaced, combined, or combined in other ways with the relevant descriptions in one or more other embodiments, and the new embodiments generated by the replacement, combination, or combination in other ways are easily conceivable by those skilled in the art and fall within the scope of protection of the present invention.

[0068] Example 1

[0069] Additional aspects and advantages of embodiments of the present invention will be described in part in the following description, and will become apparent from the following description, or will be learned through practice of embodiments of the present invention. Figure 1-Figure 4, an embodiment of the present invention provides an intelligent evaluation method for software system performance data, comprising:

[0070] S1: Collect raw data of system performance.

[0071] S2: Clean and process the original data of system performance to obtain the intermediate result data of system performance.

[0072] S3: Construct and analyze the characteristics of the system performance intermediate result data to obtain system performance analysis data.

[0073] S4: Build a system performance training set and a system performance test set based on the system performance analysis data.

[0074] S5: Input the system performance training set into the large language model for training to obtain a system performance evaluation model.

[0075] S6: Input the system performance test set into the system performance evaluation model and output the system performance evaluation results, including system performance analysis results and system performance prediction results.

[0076] Specifically, step S1 of collecting raw system performance data includes:

[0077] Use PromQL query language to obtain CPU data, memory data, disk data, network data, JVM data and Gatling performance process data from the monitoring system in real time.

[0078] Among them, the PromQL query language is a built-in query language of the Prometheus monitoring system, which can be used to query and analyze the collected time series data efficiently and flexibly. The PromQL query language provides a concise and powerful syntax to support complex data retrieval and real-time monitoring.

[0079] In a specific embodiment of the present invention, the CPU data also includes CPU overall data, system kernel data and I / O wait data. The CPU overall data is the CPU utilization, including the rate of change of the average system time of each CPU instance in the last 5 minutes. The system kernel data is the rate of change of the average system state time of each CPU instance in the last 5 minutes. The I / O wait data is the rate of change of the average I / O wait time of each CPU instance in the last 5 minutes.

[0080] Before collecting the overall CPU data, first construct a PromQL query language that includes obtaining the CPU status of the specified system, which includes the node_cpu_seconds_total built-in indicator and its parameters, which are used to indicate the total amount of time consumed by the CPU, and pass in the parameter mode = "idle", indicating that only the idle time of the CPU is considered. In the process of collecting the overall CPU data, the PromQL query language is first used to calculate the idle time change rate of the CPU idle time within 5 minutes through the built-in indicators and parameters, and the different CPU instances are grouped to calculate the average value, and then the calculated CPU idle time change rate is converted to CPU utilization. In an embodiment of the present invention, the average idle time change rate of each CPU instance in the last 5 minutes is first calculated by constructing a PromQL query language, and then the percentage of the CPU idle rate is subtracted from 100 to obtain the CPU utilization.

[0081] Before collecting system kernel data, first construct a PromQL query statement that contains the CPU state of the specified system, which contains the node_cpu_seconds_total built-in indicator and its parameters, which are used to indicate the total amount of time consumed by the CPU, and by passing in the parameter mode = "system", it means that only the system state time of the CPU is considered. In the process of collecting system kernel data, the PromQL query language is first used to calculate the system state time change rate of the CPU in the last 5 minutes through the built-in indicators and parameters, and different CPU instances are grouped to calculate the average value, and then on this basis, it is converted into the percentage of the system state time of the CPU. In the embodiment of the present invention, the average system state time change rate of each CPU instance in the last 5 minutes is first calculated by constructing the PromQL query language, and on this basis, the system state time change rate is multiplied by 100 to obtain the percentage of the system state time of the CPU.

[0082] Before collecting I / O waiting data, a PromQL query statement containing the CPU status of the specified system is first constructed, which includes the node_cpu_seconds_total built-in indicator and its parameters, which are used to indicate the total amount of CPU consumption time, and the parameter mode = "iowait" is passed in, indicating that only the I / O waiting time of the CPU is considered. In the process of collecting I / O waiting data, the PromQL query language is first used to calculate the change rate of the I / O waiting time of the CPU in the last 5 minutes through the built-in indicators and parameters, and different CPU instances are grouped to calculate the average value, and on this basis, it is converted into the percentage of the I / O waiting time of the CPU. In the embodiment of the present invention, the change rate of the average I / O waiting time of each CPU instance in the last 5 minutes is first calculated by constructing the PromQL query language, and on this basis, the change rate of the I / O waiting time of the CPU is multiplied by 100 to obtain the percentage of the I / O waiting time of the CPU.

[0083] In a specific embodiment of the present invention, the memory data includes free memory, buffer memory, cache memory and the memory occupancy ratio of each instance in the total memory.

[0084] Before collecting memory data, a PromQL query statement containing the memory usage of the specified system is first constructed, which contains node_memory_MemFree_bytes, node_memory_Buffers_bytes, node_memory_Cached_bytes and node_memory_MemTotal_bytes built-in indicators, which are used to represent the free memory, buffer memory, cache memory and total memory of the system respectively. In the process of collecting memory data, the PromQL query language is first used to calculate the sum of free memory, buffer memory and cache memory through built-in indicators and parameters to obtain the sum of memory that can be reused, and then the sum of memory that can be reused is divided by the total memory by division operation to calculate the free memory ratio, and then 1 is subtracted to calculate the free memory ratio, and the memory occupancy ratio of each instance is calculated, and on this basis, it is multiplied by 100 to obtain the memory usage percentage of the system. The present invention calculates the sum of memory that can be reused by the system through mathematical operations, and further obtains the memory occupancy ratio of each instance, which is finally converted into a percentage form. This process ensures the accurate quantification of memory usage.

[0085] In a specific embodiment of the present invention, the disk data includes disk read and write rates, that is, the instantaneous read rate and instantaneous write rate of each instance within 1 minute.

[0086] Before collecting disk data, first construct a PromQL query statement that includes obtaining the specified system disk read rate, which includes the node_disk_read_bytes_total{device="vd b"} built-in indicator, which is used to indicate the total number of disk read bytes of the virtual machine disk device vdb. At the same time, construct a PromQL query statement that includes obtaining the specified system disk write rate, which includes the node_disk_written_bytes_total{device="vdb"} built-in indicator, which is used to indicate the total number of disk write bytes of the virtual machine disk device vdb. In the process of collecting disk data, we first use the PromQL query language to calculate the read byte rate and write byte rate of each instance in the last minute through built-in indicators and parameters. Then, we use the division operation to divide the read byte rate by 1024*1024 to calculate the read rate in megabytes per second (MB / s). Similarly, we divide the write byte rate by 1024*1024 to calculate the write rate in megabytes per second (MB / s). Finally, we classify the read rate and write rate by instance to get the read rate and write rate of each instance.

[0087] In a specific embodiment of the present invention, the network data includes network receiving data and network sending data, wherein the network receiving data is the receiving rate per second of each instance after excluding the network loopback interface within the last minute, and the network sending data is the sending rate per second of each instance after excluding the network loopback interface within the last minute.

[0088] Before collecting network data, first construct a PromQL query statement that contains the data related to the network reception of the specified system, which contains the node_network_receive_bytes_total built-in indicator, which is used to indicate the total number of bytes received by the network. At the same time, construct a PromQL query statement that contains the data related to the network transmission of the specified system, which contains the node_network_transmit_bytes_total built-in indicator, which is used to indicate the total number of bytes sent by the network. In the process of collecting network data, first use the PromQL query language, through built-in indicators and parameters, to calculate the receiving rate and sending rate of each instance per second after excluding the network loopback interface in the last minute, and then use the division operation to divide the read byte rate by 1024*1024 to calculate the read rate in megabytes per second (MB / s). Similarly, divide the read send byte rate by 1024*1024 to calculate the send rate in megabytes per second (MB / s). Finally, classify the read rate and send rate by instance to obtain the read rate and send rate of each instance.

[0089] In a specific embodiment of the present invention, JVM data includes the JVM old generation memory usage percentage, the Metaspace memory usage percentage, the first majorGC pause time sum and the second majorGC pause time sum, wherein the first majorGC pause time sum is the major GC pause time sum caused by allocation failure in the JVM, and the second majorGC pause time sum is the major GC pause time sum caused by reaching the Metadata GC threshold in the JVM.

[0090] Before collecting the usage percentage of the JVM old generation memory, first construct a PromQL query statement that contains the percentage of JVM old generation memory usage for each instance, which contains the jvm_memory_used_bytes and jvm_memory_max_bytes built-in indicators, which are used to represent the number of used bytes and the maximum number of bytes of the JVM old generation memory. In the process of collecting the usage percentage of the JVM old generation memory, first use the PromQL query language to calculate the number of used bytes of the old generation memory of each instance by using the built-in indicator jvm_memory_used_bytes{id="PS Old Gen"}, and then summarize each instance to obtain the total amount of old generation memory used for each instance. Use the indicator jvm_memory_max_bytes{id="PS Old Gen"} to calculate the maximum number of bytes of the old generation memory of each instance. Then summarize each instance to obtain the maximum capacity of the old generation memory of each instance. Then use the division operation to divide the used memory of each instance by the maximum memory to calculate the usage rate of the old generation memory. Multiply the above calculation result by 100 and convert it into a percentage to get the percentage of JVM old generation memory usage.

[0091] Before collecting the Metaspace memory usage percentage of the JVM, first construct a PromQL query statement that contains the Metaspace memory usage, which contains the jvm_memory_used_bytes built-in indicator and parameter, which is used to represent the number of used bytes in the Metaspace memory area of ​​the JVM. In the process of collecting the Metaspace memory usage percentage of the JVM, first use the PromQL query statement jvm_memory_used_bytes{id="Metaspace"} to obtain the number of used bytes in the Metaspace memory area of ​​each JVM instance. Then use the sum function to summarize the results to obtain the total amount of Metaspace memory used for each JVM instance. Use the jvm_memory_max_bytes{area="nonheap", id="Metaspace"} indicator to obtain the maximum number of available bytes in the Metaspace memory area of ​​each JVM instance. Also use the sum function to summarize the results to obtain the maximum capacity of Metaspace memory for each JVM instance. Then, by dividing the total used memory of each JVM instance by its maximum total memory, the Metaspace memory usage of each instance is calculated. The above calculation result is multiplied by 100 and converted into a percentage to obtain the Metaspace memory usage ratio of the JVM.

[0092] Before collecting the sum of the first majorGC pause time, first construct a PromQL query statement to obtain the sum of pause times at the end of major GC operations caused by allocation failures in JVM garbage collection (GC) operations. The query statement contains the jvm_gc_pause_seconds_sum built-in indicator to represent the sum of garbage collection pause times. In the process of collecting the sum of the first majorGC pause time, first filter specific situations through the PromQL query statement to collect the garbage collection pause time at the end of the major GC operation caused by allocation failure, then group the data, and use the sum function to summarize the data of each instance to obtain the sum of garbage collection pause times at the end of major GC caused by allocation failure for each JVM instance, that is, the sum of the first majorGC pause time.

[0093] Before collecting the sum of the second major GC pause time, first construct a PromQL query statement to obtain the sum of pause times at the end of major GC operations caused by reaching the Metadata GC threshold in the JVM garbage collection (GC) operation. The query statement contains the jvm_gc_pause_seconds_sum built-in indicator to represent the sum of garbage collection pause times. In the process of collecting the sum of the second major GC pause time, first filter specific situations through the PromQL query statement to collect the garbage collection pause time at the end of major GC operations caused by reaching the Metadata GC threshold, then group the data, and use the sum function to summarize the data of each instance to obtain the sum of garbage collection pause times at the end of major GC for each JVM instance due to reaching the Metadata GC threshold, that is, the sum of the second major GC pause time.

[0094] In a specific embodiment of the present invention, the Gatling performance process data includes a log file generated by the Gatling stress test. When collecting the Gatling performance process data, the system automatically reads the log file generated by the Gatling stress test.

[0095] The embodiments of the present invention achieve efficient collection, cleaning and analysis of performance data through automated data collection and analysis processes, greatly improving the efficiency of the evaluation process, saving manpower and time costs, and reducing the need for manual intervention.

[0096] Specifically, step S2 cleans and processes the original system performance data to obtain the intermediate system performance result data including:

[0097] S21: Extract necessary information data from the original system performance data and remove redundant data.

[0098] S22: Formatting timestamps and numerical processing are performed on the necessary information data of the original data of the system performance, and unifying the data formats of the necessary information data of different data types to obtain the intermediate result data of the system performance.

[0099] The system performance intermediate result data includes CPU intermediate result data, memory usage intermediate result data, disk intermediate result data, network intermediate result data, JVM intermediate result data and gatling performance process intermediate data, wherein the disk intermediate result data also includes disk read rate intermediate result data and disk write rate intermediate result data, and the network intermediate result data also includes network receiving rate intermediate result data and network sending rate intermediate result data. The embodiment of the present invention organizes the above-collected system performance raw data, including extracting necessary information, removing redundant data, formatting timestamps, numerical processing and integrating different types of data into a unified format, so as to facilitate subsequent analysis.

[0100] In a specific embodiment of the present invention, during the data processing and cleaning of CPU data, after collecting CPU overall data, system kernel data and I / O waiting data through PromQL query language, the output result of collecting CPU overall data is used as the initial CPU intermediate result, and the "name" key value of an element added to the result is set to "cpu", indicating that this metric is related to CPU. Only the "instance" key value in the "metric" field of the first element in the CPU intermediate result is retained, and other key-value pairs are deleted. The first element (timestamp) in the "value" field of the first element in the CPU intermediate result is converted to a readable date and time format such as yyyy-mm-dd hh:mm:ss. The second element (CPU usage) in the "value" field of the first element in the CPU intermediate result is converted to a floating point number, rounded to two decimal places, and then converted to a string format. The second element (system CPU usage) in the "value" field of the first element in the system kernel data result is converted to a floating point number, rounded to two decimal places, and converted to a string, and then added to the "value" field of the first element in the CPU intermediate result. Convert the second element (CPU IO usage) in the "value" field of the first element in the CPU's I / O wait data result to a floating point number, round it to two decimal places, convert it to a string, and then add it to the "value" field of the first element in the CPU intermediate result.

[0101] Finally, the cleaned CPU intermediate result data is constructed, in the form of ['metric':{'instance':'{IP}'}, 'value':['{datetime}', '0.48', '0.0', '0.0'], 'name':'cpu'].

[0102] In a specific embodiment of the present invention, during the process of data processing and cleaning of memory data, after collecting memory usage data through the PromQL query language, the output result is used as the initial memory occupancy intermediate result, and the "name" key value of an element added to the result is set to "mem", indicating that this metric is about mem. Only the "instance" key value in the "metric" field of the first element in the initial memory occupancy intermediate result is retained, and other key-value pairs are deleted. The first element (timestamp) in the "value" field of the first element of the initial memory occupancy intermediate result is converted to a readable date and time format such as yyyy-mm-dd hh:mm:ss. The second element (memory usage) in the "value" field of the first element in the initial memory occupancy intermediate result is converted to a floating point number, rounded to two decimal places, and then converted to a string format for storage. Finally, the cleaned memory usage intermediate result data is constructed, in the form of ['metric':{'instance':'{IP}'}, 'value':['{datetime}', '35.99'], 'name':'mem'].

[0103] In a specific embodiment of the present invention, during the process of data processing and cleaning of disk data, after collecting disk read rate data through the PromQL query language, the output result is used as the initial disk read rate intermediate result, and the "name" key value of an element added to the result is set to "io_read", indicating that this metric is about disk read. Only the "instance" key value in the "metric" field of the first element in the initial disk read rate intermediate result is retained, and other key-value pairs are deleted. The first element (timestamp) in the "value" field of the first element of the initial disk read rate intermediate result is converted to a readable date and time format such as yyyy-mm-dd hh:mm:ss. The second element (disk read rate) in the "value" field of the first element in the initial disk read rate intermediate result is converted to a floating point number, rounded to two decimal places, and then converted to a string format for storage. Finally, the cleaned disk read rate intermediate result is constructed, in the form of ['metric':{instance':'{IP}'}, 'value':['{datetime}', 0.0], 'name':'io_read'].

[0104] Similarly, after collecting disk write rate data through PromQL query language, the output result is used as the initial disk write rate intermediate result, and the "name" key value of an element added to the result is set to "io_written", indicating that this metric is about disk write. Only the "instance" key value in the "metric" field of the first element in the initial disk write rate intermediate result is retained, and other key-value pairs are deleted. The first element (timestamp) in the "value" field of the first element of the initial disk write rate intermediate result is converted to a readable date and time format such as yyyy-mm-dd hh:mm:ss. The second element (disk read rate) in the "value" field of the first element in the initial disk write rate intermediate result is converted to a floating point number, rounded to two decimal places, and then converted to a string format for storage. Finally, the cleaned disk write rate intermediate result is constructed, such as: ['metric':{instance':'{IP}'}, 'value':['{datetime}', 0.0], 'name':'io_written'].

[0105] In a specific embodiment of the present invention, in the process of data processing and cleaning of network data, after collecting network reception rate data through PromQL query language, the output result is used as the initial network reception rate intermediate result, and the "name" key value of an element added to the result is set to "network_receive", indicating that this metric is about network reception. Only the "instance" key value in the "metric" field of the first element in the initial network reception rate intermediate result is retained, and other key-value pairs are deleted. The first element (timestamp) in the "value" field of the first element of the initial network reception rate intermediate result is converted to a readable date and time format such as yyyy-mm-ddhh:mm:ss. The second element (network reception rate) in the "value" field of the first element in the initial network reception rate intermediate result is converted to a floating point number, rounded to two decimal places, and then converted to a string format for storage. Finally, the cleaned network receiving rate intermediate result is constructed, such as: ['metric':{instance':'{IP}'}, 'value':['{datetime}', 0.0], 'name':'network_receive'].

[0106] Similarly, after collecting the above network transmission rate data results through the PromQL query language, the output result is used as the initial network transmission rate intermediate result, and the "name" key value of an element added to the result is set to "network_transmit", indicating that this metric is about network transmission. Only the "instance" key value in the "metric" field of the first element in the initial network transmission rate intermediate result is retained, and other key-value pairs are deleted. The first element (timestamp) in the "value" field of the first element of the initial network transmission rate intermediate result is converted to a readable date and time format such as yyyy-mm-ddhh:mm:ss. The second element (network transmission rate) in the "value" field of the first element in the initial network transmission rate intermediate result is converted to a floating point number, rounded to two decimal places, and then converted to a string format for storage. Finally, the cleaned network transmission rate intermediate result is constructed, such as: ['metric':{instance':'{IP}'}, 'value':['{datetime}', 0.0], 'name':'network_transmit'].

[0107] In a specific embodiment of the present invention, during the process of data processing and cleaning of JVM data, after collecting the JVM old generation memory usage percentage, Metaspace memory usage percentage, the first majorGC pause time sum and the second majorGC pause time sum data through the PromQL query language, the output result of collecting the JVM old generation memory usage percentage data is used as the initial JVM intermediate result, and the "name" key value of an element added to the result is set to "jvm" to indicate that this metric is about the JVM. Only the "instance" key value in the "metric" field of the first element in the JVM intermediate result is retained, and other key-value pairs are deleted. The first element (timestamp) in the "value" field of the first element in the JVM intermediate result is converted to a readable date and time format such as yyyy-mm-dd hh:mm:ss. The second element (JVM old generation memory usage percentage) in the "value" field of the first element in the JVM intermediate result is converted to a floating point number, rounded to two decimal places, and then converted to a string format. Convert the second element in the "value" field of the first element in the JVM's Metaspace memory usage percentage data result (JVM's Metaspace memory usage percentage) to a floating point number, round it to two decimal places, convert it to a string, and then add it to the "value" field of the first element in the JVM intermediate result. Convert the second element in the "value" field of the first element in the JVM's first major GC pause time sum due to allocation failure data result (the sum of major GC pause times due to allocation failure) to a floating point number, round it to two decimal places, convert it to a string, and then add it to the "value" field of the first element in the JVM intermediate result. Convert the second element in the "value" field of the first element in the JVM's second major GC pause time sum due to reaching the Metadata GC threshold data result (the sum of major GC pause times due to reaching the Metadata GC threshold) to a floating point number, round it to two decimal places, convert it to a string, and then add it to the "value" field of the first element in the JVM intermediate result. Finally, the cleaned JVM intermediate result data is constructed, in the form of ['metric':{'instance':'{IP}'},'value':['{datetime}','3.3','3.56','0.82','0.06'],'name':'jvm'].

[0108] In a specific embodiment of the present invention, in the process of data processing and cleaning of Gatling performance process data, the system automatically reads the log file generated by Gatling stress testing, sets the index of the parsing field according to the version information, traverses the log content to extract the request information, calculates the request duration, and generates a list containing transaction information and a set of request names. For each line containing 'REQUEST' and 'OK' or 'KO', after the line content is split by a tab character, each index, with 0 as the starting index value, uses the request end timestamp to subtract the request start timestamp to calculate the request duration. For each request, a dictionary transaction details element containing transaction information is created, including transaction name, duration, start timestamp, end timestamp, and request completion status. Then all dictionary transaction details elements are added to the transaction details list, in the format of [{transaction name key: transaction name value, duration key: duration value, start timestamp key: start timestamp value, end timestamp key: end timestamp value, request completion status key: request completion status value}]. All transaction names are removed and added to the transaction name list [transaction name]. Next, a list containing detailed analysis results for each transaction name is generated. It not only calculates the average duration, standard deviation, and least squares results, but also includes the number of transactions per second (TPS) and the average response time per second (ART) for each request. By calculating the total duration and number of occurrences of each transaction name in the log file, a transaction summary information list is finally returned. The format is as follows: [transaction name, number of occurrences, total duration]. Traverse each element in the transaction summary information list. Calculate the average duration by dividing the total duration by the number of occurrences, and convert the result to an integer. Add the current transaction name and its average duration to the transaction duration average list. The format is [[transaction name, average duration]]. Compare the first index of all elements in the transaction details list with the elements in the transaction name list. If the comparison exists, classify them by transaction name, calculate the standard deviation of the transaction response time of the second index of the element after classification by transaction name, divide the standard deviation by the average duration, and obtain the standard deviation. According to the format [transaction name, standard deviation], add it to the transaction standard deviation list [[transaction name, standard deviation]]. Compare the transaction name at the first index position of all elements in the transaction details list with the elements in the transaction name list, classify them by transaction name, calculate the slope and intercept of the duration of the second index position of the elements after classification by transaction name, and add them to the transaction linear slope list [[transaction name, slope, intercept]] in the format [transaction name, slope, intercept].The first index of all elements in the transaction details list is compared with the elements in the transaction name list, and then classified by transaction name. The detailed analysis results list belonging to the transaction name is summarized by category, and finally the classified transaction detailed analysis result list is stored in the transaction details dictionary list. [[{transaction name key: transaction name value, duration key: duration value, start timestamp key: start timestamp value, end timestamp key: end timestamp value, request completion status key: request completion status value}]]. Traverse the end timestamp of the transaction detailed analysis result list in the transaction details dictionary list, divide it by 1000, round it to seconds and classify it as the end time key, count the number of requests completed per second, and obtain the list of completed transactions per second. Group the data according to the end time key column, and calculate the average execution time of the grouped transactions to obtain the response time list per second. Construct a new result list, save each transaction name, average duration, standard deviation, transaction linear slope, intercept, and put the list of completed transactions per second and the list of average response time per second obtained above into the result list. The start timestamp of the first request in each transaction list classified by transaction in the transaction details dictionary list is divided by 1000 and converted into seconds. The start timestamp and duration of the last request in the transaction list are added and divided by 1000 to obtain the end time of the transaction (in milliseconds). The start time and end time of each scenario are calculated in turn and put into the result list to construct the complete intermediate data of the gatling performance process.

[0109] By cleaning and processing the original data of system performance, the identifiability of the data is enhanced to facilitate subsequent analysis and processing.

[0110] Specifically, step S3 constructs and analyzes the characteristics of the system performance intermediate result data to obtain the system performance analysis data, which further includes:

[0111] S31: extract multiple characteristic parameters of the system performance intermediate result data and perform multi-dimensional analysis and calculation to obtain multiple key performance indicators of the system performance intermediate result data.

[0112] S32: Summarize multiple key performance indicators to obtain system performance analysis data.

[0113] Step S31 extracts multiple characteristic parameters of the system performance intermediate result data and performs multi-dimensional analysis and calculation to obtain multiple key performance indicators of the system performance intermediate result data, including:

[0114] The overall CPU utilization, system kernel utilization, and I / O wait data utilization are extracted from the CPU intermediate result data. Multi-dimensional analysis and calculation are performed based on the overall CPU utilization, system kernel utilization, and I / O wait data utilization to obtain the average value and standard deviation of the overall CPU utilization, the average value of the system kernel utilization, the average value of the I / O wait data utilization, and the change trend of the overall CPU utilization.

[0115] The collection time and overall memory usage are extracted from the intermediate result data of memory usage, and multi-dimensional analysis and calculation are performed based on the collection time and overall memory usage to obtain the average value of the overall memory usage, the standard deviation of the overall memory usage, and the change trend of the overall memory usage.

[0116] The disk read rate and its collection time, disk write rate and its collection time, disk read rate and disk write rate are extracted from the disk intermediate result data, and multi-dimensional analysis and calculation are performed based on the disk read rate and its collection time, disk write rate and its collection time, disk read rate and disk write rate to obtain the disk read rate average and standard deviation, disk write rate average and standard deviation, disk read rate change trend and disk write rate change trend.

[0117] The network receiving rate and its collection time, the network sending rate and its collection time are extracted from the network intermediate result data, and multi-dimensional analysis and calculation are performed based on the network receiving rate and its collection time, and the network sending rate and its collection time to obtain the average value and standard deviation of the network receiving rate, the average value and standard deviation of the network sending rate, the changing trend of the network receiving rate, and the changing trend of the network sending rate.

[0118] The collection time, the proportion of JVM old generation memory usage, the proportion of Metaspace memory usage, the total time of the first major GC pause, and the total time of the second major GC pause are extracted from the JVM intermediate result data. Based on the collection time, the proportion of JVM old generation memory usage, the proportion of Metaspace memory usage, the total time of the first major GC pause, and the total time of the second major GC pause, multi-dimensional analysis and calculation are performed to obtain the average and standard deviation of the proportion of JVM old generation memory usage, the proportion of the first major GC pause time to the total duration, the proportion of the second major GC pause time to the total duration, and the changing trend of the proportion of JVM old generation memory usage.

[0119] Transaction name, request start and end timestamps, and request status are extracted from the intermediate data of the gatling performance process. Multi-dimensional analysis and calculation are performed based on the transaction name, request start and end timestamps, and request status to obtain the average response time and standard deviation of each transaction, the number of completed transactions per second and the response time per second, the transaction linear slope, the average response time stability, and the time series change trend.

[0120] In a specific embodiment of the present invention, in the process of constructing and analyzing the features of the CPU intermediate result data, the obtained CPU intermediate result data is first converted into JSON format, and then various data of CPU usage are extracted from the CPU intermediate result data in JSON format, including the overall CPU usage rate, the system kernel usage rate and the I / O waiting data usage rate, and the overall CPU usage rate extracted from each CPU instance is summarized into the overall CPU usage rate list data, and the average value of the overall CPU usage rate is calculated. The system kernel usage rate extracted from each CPU instance is summarized into the system kernel usage rate list data, and the average value of the system kernel usage rate is calculated. The I / O waiting data usage rate extracted from each CPU instance is summarized into the I / O waiting data usage rate list data, and the average value of the I / O waiting data usage rate is calculated, and then the standard deviation of the overall CPU usage rate is calculated, and the polyfit function of NumPy is used to linearly fit the overall CPU usage rate to obtain the change trend of the overall CPU usage rate, so that a macro view and change dynamics of the CPU load can be provided.

[0121] In a specific embodiment of the present invention, in the process of feature construction and analysis of memory data, the acquired memory usage intermediate result data is first converted into JSON format, and then various data of memory usage are extracted from the memory usage intermediate result data in JSON format, including collection time and overall memory usage rate, and the collection time list data is summarized in turn, and the time difference between the second time and the first time in the list is calculated to obtain the collection time interval; the overall memory usage rate list data is summarized, the average value of the overall memory usage rate and the standard deviation of the overall memory usage rate are calculated, and then the overall memory usage rate is linearly fitted using NumPy's polyfit function to obtain the overall memory usage rate change trend.

[0122] In a specific embodiment of the present invention, during the feature construction and analysis of disk data, the acquired disk read rate intermediate result data is first converted into JSON format, and then various data of disk read rate usage are extracted from the disk read rate intermediate result data in JSON format, including acquisition time, disk read rate, and the acquisition time list data is summarized in turn, and the time difference between the second time and the first time in the list is calculated to obtain the acquisition time interval; the disk read rate list data is summarized, the disk read rate average value and the disk read rate standard deviation value are calculated, and the standard deviation value is divided by the disk read rate average value to obtain the disk read rate standard deviation, and then the disk read rate is linearly fitted using NumPy's polyfit function to obtain the disk read rate change trend.

[0123] Similarly, the obtained disk write rate intermediate result data is converted into JSON format, and then various data on disk write rate usage are extracted from the disk write rate intermediate result data in JSON format, including collection time and disk write rate. The collection time list data is summarized in turn, and the time difference between the second time and the first time in the list is calculated to obtain the collection time interval; the disk write rate list data is summarized, the average disk write rate and the standard deviation of the disk write rate are calculated, and the standard deviation is divided by the average disk write rate to obtain the standard deviation of the disk write rate. Then, the polyfit function of NumPy is used to perform linear fitting on the disk write rate to obtain the change trend of the disk write rate.

[0124] In a specific embodiment of the present invention, in the process of feature construction and analysis of network data, the acquired network reception rate intermediate result data is first converted into JSON format, and then various data of network reception rate usage are extracted from the network reception rate intermediate result data in JSON format, including collection time, network reception rate, and the collection time list data are summarized in turn, and the time difference between the second time and the first time in the list is calculated to obtain the collection time interval; the network reception rate list data is summarized, the network reception rate average value and the network reception rate standard deviation value are calculated, and the standard deviation value is divided by the network reception rate average value to obtain the network reception rate standard deviation, and then the network reception rate is linearly fitted using NumPy's polyfit function to obtain the network reception rate change trend.

[0125] Similarly, the obtained network sending rate intermediate result data is converted into JSON format, and then various data on network sending rate usage are extracted from the network sending rate intermediate result data in JSON format, including collection time and network sending rate. The collection time list data is summarized in turn, and the time difference between the second time and the first time in the list is calculated to obtain the collection time interval; the network sending rate list data is summarized, the average network sending rate and the standard deviation of the network sending rate are calculated, and the standard deviation is divided by the average network sending rate to obtain the standard deviation of the network sending rate. Then, the polyfit function of NumPy is used to perform linear fitting on the network sending rate to obtain the changing trend of the network sending rate.

[0126] In a specific embodiment of the present invention, during the feature construction and analysis of JVM data, the acquired JVM intermediate result data is first converted into JSON format, and then various data of JVM occupancy are extracted from the JVM intermediate result data in JSON format, including the collection time, and the collection time list data, the proportion of JVM old generation memory usage, the proportion of JVM Metaspace memory usage, the total pause time of the first major GC due to allocation failure, and the total pause time of the second major GC due to reaching the Metadata GC threshold are summarized in turn. Calculate the time difference between the second time and the first time in the list to get the collection time interval, use the collection time list data to calculate the total duration, summarize the JVM old generation memory usage percentage list data, calculate the average JVM old generation memory usage percentage and the standard deviation of the JVM old generation memory usage percentage, and then use NumPy's polyfit function to perform linear fitting on the JVM old generation memory usage percentage to get the JVM old generation memory usage percentage change trend, divide the sum of the first major GC pause time by the total duration to get the ratio of the first major GC pause time to the total duration, and divide the sum of the second major GC pause time by the total duration to get the ratio of the second major GC pause time to the total duration.

[0127] In a specific embodiment of the present invention, during the feature construction and analysis of Gatling performance process data, the intermediate data obtained during the gatling stress test process is first converted into JSON format, and then various data in the test process are extracted from the intermediate data during the gatling stress test process in JSON format, including: transaction name, average response time, standard deviation, transaction linear slope, list of completed transactions per second, and list of response time per second. The time difference between the second time and the first time in the list is calculated to obtain the collection time interval, and the execution time of each transaction is obtained by subtracting the end time from the start time of each transaction. The list of completed transactions per second is summed and divided by the execution time of each transaction to obtain the average number of transactions per second of the transaction. After sorting the list of response time per second from small to large, the response time per second at the 90th percentile, the response time per second at the 95th percentile, and the response time per second at the 99th percentile are obtained in turn.

[0128] The embodiment of the present invention can comprehensively collect various data indicators of the system in the performance test by constructing and analyzing the characteristics of the system performance data, and can perform multi-dimensional analysis to provide users with more comprehensive and in-depth performance evaluation information.

[0129] Specifically, step S4 of constructing a system performance training set and a system performance test set based on the system performance analysis data further includes:

[0130] S41: Generate a monitoring report containing key performance indicators based on the system performance analysis data.

[0131] S42: Input each key performance indicator in the monitoring report into the Prompt reasoning template, mark the potential problem areas according to the preset thresholds, and give corresponding suggested measures to obtain a structured system performance training set and system performance test set.

[0132] Among them, the Prompt inference template is mainly used to prompt the AI ​​model with the context of input information and parameter information of the input model. When training supervised learning or unsupervised learning models, Prompt can help the model better understand the intention of the input and respond accordingly.

[0133] In a specific embodiment of the present invention, in the process of constructing a CPU data training set, according to the data obtained in step S3, the data are arranged in the order of the average value of the overall CPU utilization, the average value of the system kernel utilization, the average value of the I / O waiting data utilization, the standard deviation of the overall CPU utilization, the trend of the overall CPU utilization, the overall CPU utilization list data, the average value of the overall CPU utilization, the average value of the system kernel utilization, the average value of the I / O waiting data utilization, and the standard deviation of the overall CPU utilization, and a monitoring report containing the above key performance indicators is generated, and the information of the monitoring report is filled into the Prompt reasoning template, and then the collected data are risk assessed according to the preset thresholds built into the Prompt reasoning template, potential problem areas are marked, and corresponding suggested measures are given, and finally a structured CPU data training set and test set are constructed for subsequent model training and testing using a large language model.

[0134] In a specific embodiment of the present invention, in the process of constructing a memory data training set, according to the data obtained in step S3, the data are arranged in the order of collection time interval, average value of overall memory usage, standard deviation of overall memory usage, change trend of overall memory usage, and list data of overall memory usage, and a monitoring report containing the above key performance indicators is generated, and the information of the monitoring report is filled into the Prompt reasoning template, and then a risk assessment is performed on the collected data according to the preset threshold built into the Prompt reasoning template, potential problem areas are marked, and corresponding recommended measures are given, and finally a structured memory data training set and test set are constructed for subsequent model training and testing using a large language model.

[0135] In a specific embodiment of the present invention, in the process of constructing a disk data training set, according to the data obtained in step S3, the data are arranged in the order of collection time interval, disk read rate average value, disk read rate standard deviation, disk read rate change trend, and disk read rate list data, and then the collected data are risk assessed according to the preset threshold, potential problem areas are marked, and corresponding recommended measures are given, and finally a monitoring report containing the above key performance indicators is generated. Similarly, according to the data obtained in step S3, the data are arranged in the order of collection time interval, disk write rate average value, disk write rate standard deviation, disk write rate change trend, and disk write rate list data, and then the collected data are risk assessed according to the preset threshold, potential problem areas are marked, and corresponding recommended measures are given, and a monitoring report containing the above key performance indicators is generated, and the information of the monitoring report is filled into the Prompt reasoning template, and then the collected data are risk assessed according to the preset threshold built in the Prompt reasoning template, potential problem areas are marked, and corresponding recommended measures are given, and finally a structured disk data training set and test set are constructed, so as to prepare for subsequent model training and testing using a large language model.

[0136] In a specific embodiment of the present invention, in the process of constructing a network data training set, according to the data obtained in step S3, the data are arranged in the order of collection time interval, average network sending rate, standard deviation of network sending rate, change trend of network sending rate, and list data of network sending rate, and then the collected data are risk assessed according to the preset threshold, potential problem areas are marked, and corresponding suggested measures are given. Similarly, according to the data obtained in step S3, the data are arranged in the order of collection time interval, average network receiving rate, standard deviation of network receiving rate, change trend of network receiving rate, and list data of network receiving rate, a monitoring report containing the above key performance indicators is generated, and the information of the monitoring report is filled into the Prompt reasoning template, and then the collected data are risk assessed according to the preset threshold built into the Prompt reasoning template, potential problem areas are marked, and corresponding suggested measures are given, and finally a structured network data training set and test set are constructed for subsequent model training and testing using a large language model.

[0137] In a specific embodiment of the present invention, in the process of constructing a JVM data training set, according to the data obtained in step S3, the data are arranged in the order of collection time interval, average value of JVM old generation memory usage percentage, standard deviation of JVM old generation memory usage percentage, change trend of JVM old generation memory usage percentage, ratio of first major GC pause time to total duration, ratio of second major GC pause time to total duration, and list data of JVM old generation memory usage percentage, a monitoring report containing the above key performance indicators is generated, and the information of the monitoring report is filled into the Prompt reasoning template, and then the collected data are risk assessed according to the preset threshold built into the Prompt reasoning template, potential problem areas are marked, and corresponding suggested measures are given, and finally a structured JVM data training set and test set are constructed to prepare for subsequent model training and testing using a large language model.

[0138] In a specific embodiment of the present invention, in the process of constructing a Gatling performance process data training set, according to the data obtained in step S3, the data are arranged in the order of transaction name, average number of transactions per second, average response time, standard deviation, transaction linear slope, 90th percentile response time per second, 95th percentile response time per second, and 99th percentile response time per second, to generate a monitoring report containing the above key performance indicators, and fill the information of the monitoring report into the Prompt reasoning template, and then perform risk assessment on the collected data according to the preset threshold built into the Prompt reasoning template, mark potential problem areas, and give corresponding recommended measures, and finally construct a structured Gatling performance process data training set and test set for subsequent model training and testing using a large language model.

[0139] Specifically, step S5 inputs the system performance training set into the large language model for training to obtain a system performance evaluation model.

[0140] Among them, the big language model is a natural language processing model based on deep learning, which can learn the grammar and semantics of natural language and generate human-readable text.

[0141] In the embodiment of the present invention, the system performance training set constructed in step S4 is input into the large language model for training, and the validation set is input during the training process to tune the model, and the system performance evaluation model is obtained through training. In the embodiment of the present invention, the open source model training framework Swift is used as the large language model for training. When constructing the Swift training framework, the main parameters required include: prompt word engineering template prompt_template, data to be inferred query, and training result answer. By training the large model, a highly optimized performance evaluation model for a specific vertical field of the software system is constructed. The system performance evaluation model obtained by training in the present invention has the ability of self-learning and continuous optimization. With the passage of time and the accumulation of data, the performance evaluation model will be continuously optimized and improved to improve the credibility and accuracy of the evaluation results. Compared with subjective manual evaluation, this system performance evaluation model is an evaluation model based on rule-driven and data-driven, which eliminates subjective bias and can provide more objective and accurate performance evaluation results.

[0142] Specifically, step S6 inputs the system performance test set into the system performance evaluation model and outputs the system performance evaluation result.

[0143] The system performance evaluation results include system performance analysis results and system performance prediction results. In the embodiment of the present invention, the system performance test set constructed in step S4 is input into the large language model for testing, and the system performance analysis results and system performance prediction results output by model reasoning can be obtained by inputting the data to be inferred constructed according to the system performance reasoning data Prompt template.

[0144] Example 2

[0145] Combination Figure 5 and Figure 6 The embodiment of the present invention further provides an intelligent evaluation system for software system performance, including:

[0146] The data acquisition module 1 is used to collect the original data of system performance.

[0147] The data cleaning and processing module 2 is used to clean and process the original data of system performance to obtain the intermediate result data of system performance.

[0148] The feature construction and analysis module 3 is used to construct and analyze the features of the system performance intermediate result data to obtain system performance analysis data.

[0149] The training data construction module 4 is used to construct a system performance training set and a system performance test set based on the system performance analysis data.

[0150] The LLM large language model 5 is used to input the system performance training set into the large language model for training to obtain a system performance evaluation model, and to input the system performance test set into the system performance evaluation model and output the system performance evaluation result.

[0151] Specifically, the data acquisition module 1 also includes:

[0152] The CPU data collection submodule 11 is used to collect CPU overall data, system kernel data and I / O waiting data. This module is designed to monitor and evaluate the use of the system CPU in real time, with particular attention to the overall utilization, system kernel time and I / O waiting time, coordinate resource management and build a training data set. The CPU data collection submodule 11 also includes a CPU overall data collector, a system kernel data collector and an I / O waiting data collector. The CPU overall data collector uses the PromQL query language, through the built-in indicator node_cpu_seconds_total and the parameter mode = "idle", to calculate the change rate of the CPU idle time within 5 minutes, and further convert it into CPU utilization. The system kernel data collector uses the PromQL query language, through the built-in indicator node_cpu_seconds_total and the parameter mode = "system", to calculate the change rate of the system state time. The I / O waiting data collector uses the PromQL query language, through the built-in indicator node_cpu_seconds_total and the parameter mode = "iowait" CPU I / O waiting time, to evaluate the impact of I / O on CPU efficiency.

[0153] The memory data collection submodule 12 is used to collect free memory, buffer memory, cache memory and total memory, and calculate the memory usage ratio of each instance. This module extracts memory-related indicators from the system through PromQL query statements, including free memory, buffer memory, cache memory and total memory. Through mathematical operations, the total amount of memory that can be reused by the system is calculated, and the memory usage ratio of each instance is further obtained, which is finally converted into a percentage form to ensure accurate quantification of memory usage.

[0154] The disk data acquisition submodule 13 is used to collect disk read rate data and disk write rate data. This module uses the PromQL query language to dynamically obtain disk read and write rate data from the monitoring system. For read operations, the instantaneous read rate within 1 minute is calculated and converted into MB / s units. For write operations, the instantaneous write rate within 1 minute is calculated. In this way, real-time monitoring data collection of disk I / O activities is achieved.

[0155] The network data collection submodule 14 is used to collect network reception data and network transmission data. This module uses PromQL to dynamically generate query instructions and collect network reception and transmission data in real time. The i rate function is used to calculate the rate change within the specified time window to ensure that the data reflects the instantaneous network activity status. The rate unit is further standardized to MB / s to facilitate subsequent analysis and comparison. Multiple instances are summarized at the same time during the data collection stage to enhance the comprehensiveness and representativeness of the data.

[0156] The JVM data collection submodule 15 is used to collect the JVM old generation memory usage percentage, Metaspace memory usage percentage, the first majorGC pause time sum and the second majorGC pause time sum. The first majorGC pause time sum is the major GC pause time sum caused by allocation failure in the JVM, and the second majorGC pause time sum is the major GC pause time sum caused by reaching the Metadata GC threshold in the JVM.

[0157] This module uses the PromQL language to dynamically construct query instructions, and accurately collects the proportion of JVM old generation memory usage, Metaspace memory usage, major GC pause time caused by allocation failure, and major GC pause time caused by reaching the Metadata GC threshold. This module uses built-in indicators to accurately extract the required data through aggregation functions and filtering conditions to ensure the real-time and accuracy of the data.

[0158] The Gatling performance process data collection submodule 16 is used to collect log files generated by Gatling stress testing.

[0159] Specifically, the data cleaning and processing module 2 also includes:

[0160] CPU data cleaning module, which is used to organize the collected CPU raw data, including extracting necessary information, removing redundant data, formatting timestamps, numerical processing, and integrating different types of CPU usage data into a unified format for subsequent analysis.

[0161] Memory data cleaning module, which is used to organize the collected raw memory data, including extracting key information, removing irrelevant data, formatting timestamps, and standardizing memory usage values, in order to improve data availability and analysis efficiency, and provide preparation for subsequent steps.

[0162] Disk data cleaning module: This module cleans the collected disk raw data to ensure that the data format is unified and easy to analyze. Specifically, it includes adding labels to the data to distinguish read and write operations, retaining only instance identifiers, converting timestamps to readable formats, and standardizing rate values, so as to generate well-structured intermediate result data, laying the foundation for subsequent analysis.

[0163] The network data cleaning module organizes the collected network raw data. First, it marks the network activity type (receive or send) for each record, then extracts and retains key identifiers (such as instance IP), converts the timestamp into an easy-to-read format, and calculates the rate value to two decimal places. This process removes redundant information, highlights key indicators, and improves data quality.

[0164] JVM data cleaning module, which processes the collected JVM raw data, including formatting timestamps, rounding values, type conversion, etc., to ensure that the data format is unified and suitable for further analysis. In addition, the module integrates all key indicators into a single record for subsequent analysis, and marks the data as JVM-specific metrics to enhance data identifiability.

[0165] Gatling performance process data cleaning module, which automatically reads the log files generated by Gatling stress testing, sets parsing rules by identifying version information, and accurately extracts key fields of each record, such as transaction name, request start and end timestamps, and request status, to ensure effective preprocessing of raw data and lay the foundation for subsequent analysis.

[0166] Specifically, the feature construction and analysis module 3 also includes:

[0167] CPU data analysis module: This module further processes the sorted CPU data to form a statistical summary of CPU usage, including the average and standard deviation of various CPU usage rates. It also uses NumPy's polyfit function to perform linear fitting analysis to obtain the overall usage trend, thereby providing a macro view of the CPU load and its changing dynamics.

[0168] Memory data analysis module, which further aggregates the timestamps of the cleaned memory usage data to generate collection intervals, calculates the mean and standard deviation of memory usage, and uses NumPy's polyfit function for linear fitting to predict memory usage trends.

[0169] Disk data analysis module, which first extracts time series features from the cleaned disk data, including calculating the time interval, average rate, standard deviation, and uses NumPy's linear regression function (polyfit function) to analyze the trend of read and write rates.

[0170] The network data analysis module first extracts time series information from the cleaned network data, calculates statistical indicators such as the time interval, average rate, standard deviation, etc. of data collection, and uses NumPy's polyfit function to perform linear fitting to analyze the trend of network activities.

[0171] JVM data analysis module: This module calculates a series of key performance indicators based on the cleaned JVM data, such as the collection time interval, the average value, standard deviation and change trend of memory usage, and the duration of GC events. It also uses the NumPy library for linear fitting to evaluate the change trend of JVM old generation memory usage, providing a basis for performance prediction. In addition, by calculating the ratio of each indicator to the total duration, it evaluates the impact of GC activity on system stability.

[0172] Gatling performance process data analysis module, based on the cleaned Gatling performance process data, calculates the duration of each transaction request and integrates to generate a detailed list containing transaction name, response time, status and other information. Through the aggregate analysis of these data, the core performance indicators such as the average response time, standard deviation, linear trend (slope and intercept) of each transaction are calculated. At the same time, the system calculates the number of transactions per second (TPS) and the average response time (ART) to provide intuitive measurements for performance evaluation.

[0173] Specifically, the training data construction module 4 also includes:

[0174] CPU training data construction module, which is used to generate monitoring reports containing key performance indicators, such as average CPU usage, system kernel usage, I / O wait usage, standard deviation and trend of resource usage, etc. Risk assessment is performed according to preset thresholds, potential problem areas are marked, such as high usage or abnormal trends, and corresponding recommended measures are given. Finally, the above information is used to construct corresponding training data according to the Prompt system performance inference template, in preparation for subsequent model training using the large language model.

[0175] The memory training data construction module is used to generate monitoring reports containing key performance indicators, such as the average value, standard deviation, and memory usage trend of memory usage. Then, risk assessment is performed based on preset thresholds, potential problem areas are marked, and corresponding recommended measures are given. Finally, the above information is used to construct corresponding training data according to the Prompt system performance inference template, in preparation for subsequent model training using the large language model.

[0176] Disk training data construction module, which is used to generate monitoring reports containing key performance indicators, such as time interval, average rate, standard deviation, read and write rate trends, etc. Then, risk assessment is performed based on preset thresholds, potential problem areas are marked, and corresponding recommended measures are given. Finally, the above information is used to construct corresponding training data according to the Prompt system performance inference template, in preparation for subsequent model training using the large language model.

[0177] The network training data construction module is used to generate monitoring reports containing key performance indicators, such as the time interval, average rate, standard deviation, and trend of network activities of data collection. Then, risk assessment is performed according to the preset threshold, potential problem areas are marked, and corresponding recommended measures are given. Finally, the above information is used to construct corresponding training data according to the Prompt system performance inference template, so as to prepare for subsequent model training using the large language model.

[0178] JVM training data construction module, which is used to generate monitoring reports containing key performance indicators, such as collection time interval, average value, standard deviation and change trend of memory usage, duration ratio of GC events, change trend of JVM old generation memory usage, etc. Then, risk assessment is performed according to preset thresholds, potential problem areas are marked, and corresponding recommended measures are given. Finally, the above information is used to construct corresponding training data according to the Prompt system performance inference template, in preparation for subsequent model training using the large language model.

[0179] Gatling performance process training data construction module, which is used to generate monitoring reports containing key performance indicators, such as the average response time, standard deviation, linear trend, etc. of each transaction. Then, risk assessment is performed according to the preset threshold, potential problem areas are marked, and corresponding recommended measures are given. Finally, the above information is used to construct corresponding training data according to the Prompt system performance reasoning template, in preparation for subsequent model training using the large language model.

[0180] In summary, the intelligent evaluation method and system for software system performance provided by the present invention have the following advantages:

[0181] 1. Efficiency and automation: The present invention realizes efficient collection, cleaning and analysis of software system performance data through automated data collection and processing and analysis processes, greatly improving the efficiency of the evaluation process, saving manpower and time costs, and reducing the need for manual intervention.

[0182] 2. Objectivity and accuracy: Compared with subjective manual evaluation, the evaluation scheme of the present invention is rule-driven and data-driven, eliminating subjective bias and providing more objective and accurate performance evaluation results.

[0183] 3. Intelligence and continuous optimization: The present invention trains a large model and uses the trained artificial intelligence large model to form a highly optimized performance analysis model for a specific vertical field. The model has the ability of self-learning and continuous optimization. With the passage of time and the accumulation of data, the performance evaluation model of the present invention will be continuously optimized and improved to enhance the credibility and accuracy of the evaluation results.

[0184] 4. Comprehensiveness and multi-dimensional analysis: The present invention can comprehensively collect various data indicators of the system in the performance test, and can perform multi-dimensional analysis to provide users with more comprehensive and in-depth performance evaluation information.

[0185] The above describes in detail the concepts, principles and ideas of the present invention in conjunction with specific implementation methods (including embodiments and examples). Those skilled in the art should understand that the implementation methods of the present invention are not limited to the forms given above. After reading this application document, those skilled in the art can make any possible improvements, substitutions and equivalent forms to the steps, methods, systems and components in the above implementation methods. These improvements, substitutions and equivalent forms should be deemed to fall within the scope of the present invention, and the protection scope of the present invention is subject only to the claims.

Claims

1. An intelligent evaluation method for software system performance data, characterized in that: include: Collect raw data on system performance; Cleaning and processing the original system performance data to obtain intermediate system performance result data; Constructing and analyzing the characteristics of the intermediate result data of the system performance to obtain system performance analysis data; Building a system performance training set and a system performance test set based on the system performance analysis data; Inputting the system performance training set into a large language model for training to obtain a system performance evaluation model; The system performance test set is input into the system performance evaluation model and the system performance evaluation result is output.

2. The intelligent evaluation method for software system performance data according to claim 1, characterized in that: The raw data of the acquisition system performance includes: Use PromQL query language to obtain CPU data, memory data, disk data, network data, JVM data and Gatling performance process data from the monitoring system in real time, including: The CPU data includes CPU overall data, system kernel data and I / O waiting data; The memory data includes the memory usage ratio of each instance in free memory, buffer memory, cache memory and total memory; The disk data includes disk read rate and disk write rate; The network data includes network received data and network sent data; The JVM data includes the JVM old generation memory usage ratio, the Metaspace memory usage ratio, the first majorGC pause time sum and the second majorGC pause time sum, wherein the first majorGC pause time sum is the major GC pause time sum caused by allocation failure in the JVM, and the second majorGC pause time sum is the major GC pause time sum caused by reaching the Metadata GC threshold in the JVM; The Gatling performance process data includes log files generated by Gatling stress testing.

3. The intelligent evaluation method for software system performance data according to claim 2, characterized in that: The data cleaning and processing of the system performance raw data to obtain the system performance intermediate result data further includes: Extracting necessary information data from the system performance raw data and removing redundant data; The necessary information data of the system performance original data is formatted with a timestamp and numerically processed, and the necessary information data of different data types are unified in data format to obtain system performance intermediate result data.

4. The intelligent evaluation method for software system performance data according to claim 3, characterized in that: The system performance intermediate result data includes CPU intermediate result data, memory occupancy intermediate result data, disk intermediate result data, network intermediate result data, JVM intermediate result data and gatling performance process intermediate data, wherein the disk intermediate result data also includes disk read rate intermediate result data and disk write rate intermediate result data, and the network intermediate result data also includes network reception rate intermediate result data and network sending rate intermediate result data.

5. The intelligent evaluation method for software system performance data according to claim 4, characterized in that: The feature construction and analysis of the intermediate result data of the system performance to obtain the system performance analysis data includes: Extracting multiple characteristic parameters of the system performance intermediate result data and performing multi-dimensional analysis and calculation to obtain multiple key performance indicators of the system performance intermediate result data; A plurality of the key performance indicators are aggregated to obtain system performance analysis data.

6. The intelligent evaluation method for software system performance data according to claim 4, characterized in that: The extracting of multiple characteristic parameters of the system performance intermediate result data and performing multi-dimensional analysis and calculation to obtain multiple key performance indicators of the system performance intermediate result data includes: Extract the overall CPU usage, system kernel usage and I / O waiting data usage from the CPU intermediate result data, and perform multi-dimensional analysis and calculation based on the overall CPU usage, the system kernel usage and the I / O waiting data usage to obtain the average value and standard deviation of the overall CPU usage, the average value of the system kernel usage, the average value of the I / O waiting data usage and the change trend of the overall CPU usage; Extract the collection time and the overall memory usage rate from the intermediate memory usage result data, and perform multi-dimensional analysis and calculation based on the collection time and the overall memory usage rate to obtain the average value of the overall memory usage rate, the standard deviation of the overall memory usage rate, and the change trend of the overall memory usage rate; Extract the disk read rate and its acquisition time, the disk write rate and its acquisition time, the disk read rate and the disk write rate from the disk intermediate result data, and perform multi-dimensional analysis and calculation based on the disk read rate and its acquisition time, the disk write rate and its acquisition time, the disk read rate and the disk write rate to obtain the disk read rate average value and standard deviation, the disk write rate average value and standard deviation, the disk read rate change trend and the disk write rate change trend; Extract the network receiving rate and its collection time, the network sending rate and its collection time from the network intermediate result data, and perform multi-dimensional analysis and calculation based on the network receiving rate and its collection time, the network sending rate and its collection time to obtain the average value and standard deviation of the network receiving rate, the average value and standard deviation of the network sending rate, the change trend of the network receiving rate, and the change trend of the network sending rate; Extract the collection time, the proportion of JVM old generation memory usage, the proportion of Metaspace memory usage, the sum of the first major GC pause time, and the sum of the second major GC pause time from the JVM intermediate result data, and perform multi-dimensional analysis and calculation based on the collection time, the proportion of JVM old generation memory usage, the proportion of Metaspace memory usage, the sum of the first major GC pause time, and the sum of the second major GC pause time to obtain the average value and standard deviation of the JVM old generation memory usage proportion, the proportion of the first major GC pause time to the total duration, the proportion of the second major GC pause time to the total duration, and the change trend of the JVM old generation memory usage proportion; The transaction name, request start and end timestamps, and request status are extracted from the intermediate data of the gatling performance process, and multi-dimensional analysis and calculation are performed based on the transaction name, the request start and end timestamps, and the request status to obtain the average response time and standard deviation of each transaction, the number of completed transactions per second and the response time per second, the transaction linear slope, the average response time stability, and the time series change trend.

7. The intelligent evaluation method for software system performance data according to claim 6, characterized in that: The constructing of a system performance training set and a system performance test set based on the system performance analysis data comprises: Generating a monitoring report including the key performance indicators based on the system performance analysis data; Each of the key performance indicators in the monitoring report is input into the Prompt reasoning template, and potential problem areas are marked according to preset thresholds and corresponding suggested measures are given to obtain a structured system performance training set and a system performance test set.

8. The intelligent evaluation method for software system performance data according to claim 1, characterized in that: The system performance evaluation results include system performance analysis results and system performance prediction results.

9. An intelligent evaluation system for software system performance data, characterized in that: include: Data acquisition module, used to collect raw data of system performance; A data cleaning and processing module is used to clean and process the original system performance data to obtain intermediate system performance result data; A feature construction and analysis module is used to construct and analyze the features of the intermediate result data of the system performance to obtain system performance analysis data; A training data construction module, used to construct a system performance training set and a system performance test set based on the system performance analysis data; The LLM large language model is used to input the system performance training set into the large language model for training to obtain a system performance evaluation model, and to input the system performance test set into the system performance evaluation model and output a system performance evaluation result.

10. The intelligent evaluation system for software system performance data according to claim 9, characterized in that: The data acquisition module also includes: CPU data collection submodule, used to collect CPU overall data, system kernel data and I / O wait data; The memory data collection submodule is used to collect free memory, buffer memory, cache memory and total memory, and calculate the memory usage ratio of each instance; The disk data collection submodule is used to collect disk read rate data and disk write rate data; The network data collection submodule is used to collect network received data and network sent data; The JVM data collection submodule is used to collect the JVM old generation memory usage ratio, Metaspace memory usage ratio, the first major GC pause time sum and the second major GC pause time sum. The first major GC pause time sum is the major GC pause time sum caused by allocation failure in the JVM, and the second major GC pause time sum is the major GC pause time sum caused by reaching the Metadata GC threshold in the JVM. The Gatling performance process data collection submodule is used to collect log files generated by Gatling stress testing.

Images