Domestic environment-oriented distributed performance testing method and device

Abstract

The invention relates to a distributed performance testing method and device oriented to a domestic environment, and the method comprises the following steps: defining a data model, defining a Mock generation function of each field, generating a basic Mock value for each field according to the Mock generation function, and enabling the basic Mock value to serve as a final field value or an intermediate value, performing rule calculation or splicing to obtain final field values, and combining the final field values to obtain simulation data; the main control mechanism constructs a time sequence vector of each node, forms a whole network performance matrix by using a plurality of node data, calculates a performance score and identifies a bottleneck, and completes a performance test; and setting an early warning value for the monitoring parameter, and meanwhile, analyzing a parameter value at the next moment through an intelligent analysis model, and carrying out pre-judgment. The method has the advantages that 1) the dynamic configuration capability of the structured data pool and the Mock engine is utilized to realize a data constructor close to a real scene, and 2) the main control computer synchronously integrates multiple test results to realize second-level bottleneck positioning.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer models, and in particular to a distributed performance testing method and device for a domestic environment. BACKGROUND

[0002] The core practice scene of online distributed stress testing mainly concentrates in complex business systems with high concurrency and multiple dependencies. Such stress testing needs to reproduce the peak traffic in the online environment, inject traffic through a distributed stressor cluster (such as a million concurrent users), while ensuring that the stress testing data is isolated from the production data to avoid polluting normal business.

[0003] Most current simulation data generation methods for stress testing use simple random rules or template splicing, and do not have the ability to understand context semantics, making it difficult to generate structured data close to reality, and unable to dynamically adjust the generation strategy to meet the data quality requirements in multiple scenarios, resulting in low test coverage, algorithm training distortion, and incomplete system verification.

[0004] Due to the large number of servers in a distributed system test environment, manual collection and analysis of performance indicators for each server is required, and the monitoring analysis, organization, and accuracy of index statistics are low, making it difficult to efficiently and quickly locate the performance bottlenecks of the system under test. Traditional performance testing tools rely on centralized polling of the server under test by the master node, which has the following technical defects:

[0005] (1) Lack of elastic expansion capability, each newly added server will increase the load of the master;

[0006] (2) The monitoring accuracy is limited by the polling frequency of the master and network delay;

[0007] (3) The analysis and positioning of bottlenecks is inefficient and cannot meet the performance requirements of multi-machine collaboration in a domestic heterogeneous environment;

[0008] (4) Real-time fusion of multi-source data cannot be achieved, resulting in performance analysis lag. SUMMARY

[0009] To solve the above problems, the purpose of the present application is to provide a distributed performance testing method and device for a domestic environment, which utilizes the dynamic configuration capability of structured data pool and Mock engine to realize a data constructor close to reality, and dynamically adjusts the Mock rules through system feedback and adaptive weight update, thereby continuously improving the quality of generated data.

[0010] To achieve the above purpose, the present application adopts the following technical solutions:

[0011] Technical solution one

[0012] A distributed performance testing method for a domestic environment, comprising the following steps:

[0013] Generating simulation data: defining a data model M, the data model M being composed of multiple fields; defining a Mock generation function for each field, each field generating a basic Mock value according to the Mock generation function, if the field does not depend on other fields, the basic Mock value being the final field value, if the field depends on other fields, the basic Mock value being an intermediate value participating in rule calculation or splicing to obtain the final field value; generating field values of each field according to the defined data model and the Mock generation function of each field, and then combining the field values to obtain simulation data;

[0014] Distributed performance testing: composed of a host computer, multiple agent nodes and a performance perception module, each agent node registers its capability vector to the host computer, the host computer builds a capability matrix and executes a distributed load perception scheduling algorithm according to the capability matrix, calculates the priority value of each agent node, and then dispatches tasks to the agent nodes according to the priority value, each agent node collects performance indicators through the performance perception module and performs indicator screening, and records the indicator data that does not meet the requirements for local compression or discarding; the host computer constructs a time series vector of each node and groups multiple node data into a network-wide performance matrix, calculates a performance score and identifies a bottleneck, and completes performance testing.

[0015] To solve the above technical problems, the application further provides a distributed performance testing device for a domestic environment.

[0016] Technical solution two

[0017] A distributed performance testing device for a domestic environment, comprising

[0018] The simulation data generation module performs the following steps: defining a data model M, the data model M being composed of multiple fields; defining a Mock generation function for each field, each field generating a basic Mock value according to the Mock generation function, if the field does not depend on other fields, the basic Mock value being the final field value, if the field depends on other fields, the basic Mock value being an intermediate value participating in rule calculation or splicing to obtain the final field value; generating field values of each field according to the defined data model and the Mock generation function of each field, and then combining the field values to obtain simulation data;

[0019] The distributed performance testing module consists of a master control unit, multiple agent nodes, and a performance awareness module. It executes the following steps: Each agent node registers its own capability vector with the master control unit. The master control unit constructs a capability matrix and executes a distributed load-aware scheduling algorithm based on the capability matrix to calculate the priority value of each agent node. Then, tasks are dispatched to the agent nodes according to the priority value. Each agent node collects performance metrics through the performance awareness module and filters the metrics, compressing or discarding data that does not meet the requirements locally. The master control unit constructs a time-series vector for each node and combines the data from multiple nodes into a network-wide performance matrix, calculates the performance score, identifies bottlenecks, and completes the performance test.

[0020] Performance monitoring module: performs performance and real-time monitoring and issues alerts when anomalies occur; Data acquisition: collects performance metrics from multiple dimensions and formats the data uniformly; Predicts future moments using a trained hybrid prediction model LSTM-ARIMA. Performance metrics.

[0021] The present invention has the following beneficial effects:

[0022] (1) Based on intelligent simulation data generation technology, the structured data pool and the dynamic configuration capability of the Mock engine are used to realize a data builder that closely resembles the real scene. The Mock rules are dynamically adjusted through system feedback, adaptive weight update and other methods, thereby continuously improving the quality of generated data.

[0023] (2) A collaborative model of “master control machine + multiple agent nodes” is proposed. The master control machine dynamically distributes stress test scripts and concurrent tasks to agent nodes. Each node actively reports performance index data. The addition of new nodes automatically increases the system monitoring capacity. The master control machine synchronously integrates multiple test results to achieve second-level bottleneck location, which solves the efficiency bottleneck of large-scale cluster monitoring.

[0024] (3) In response to the heterogeneous characteristics of domestic software and hardware, the high-frequency data pipeline of WebSocket ensures the millisecond-level collection of indicators such as CPU, memory, and disk, and the built-in intelligent analysis engine correlates the stress test traffic with system indicators. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the overall process and architecture of the performance testing method of the present invention;

[0026] Figure 2 This is a flowchart of the simulation data generation process of the present invention. Detailed Implementation

[0027] Please see Figure 1 This invention provides a distributed performance testing method for domestically developed environments, primarily targeting high-concurrency, heterogeneous domestically developed business systems. The overall functionality includes:

[0028] (1) Intelligent simulation data generation. Through the semantic rule engine and the feedback-driven mechanism, test data highly close to the real scene is generated to provide support for downstream stress testing and monitoring.

[0029] (2) Distributed performance testing oriented to domestic environment. The Master-Agent architecture is adopted, the node autonomously collects and pre-analyzes, the chain index is propagated, the load is perceived and scheduled, online stress testing under million-level concurrency and rapid positioning of performance bottlenecks are realized, and data aggregation, visual display and trend prediction are carried out after the test is completed.

[0030] (3) Performance monitoring oriented to domestic environment. On various domestic operating systems and hardware platforms, multi-dimensional indexes such as CPU, memory, disk I / O and network I / O are collected in real time, intelligent analysis and prediction are carried out combined with the LSTM-ARIMA hybrid model, potential bottlenecks are discovered in time and alarms are triggered.

[0031] The application will be further described in detail in combination with the drawings and specific embodiments:

[0032] A distributed performance testing method oriented to domestic environment, comprising the following steps:

[0033] Referring to Figure 2 , step 10, simulation data generation and optimization mechanism.

[0034] Step 11, abstract data model definition and analysis: the user or system designer defines the data model M according to the required simulation scene, the data model M is composed of multiple fields, each field has three attributes, including field name, field type and field constraint.

[0035] For example:

[0036] Field name: such as "name", "ID card number", "date of birth", etc.

[0037] Field type: including string (String), date (Date), Boolean, enumeration (Enum), integer (Integer / Float), etc.

[0038] Field constraint: including field length, whether to allow null, format expression (such as regular), value range, value list, whether unique, etc.

[0039] An identity information model can be defined as follows:

[0040] Table 1:

[0041] Field Type Constraint name String Length 2~4, must be Chinese id_card String Length 18, check bit verification legal gender Enum Value: male, female birthday Date Format yyyy-mm-dd region String Derived from the real area dictionary table

[0042] For some fields, there are logical or structural dependencies between them, such as the birth date and gender information in the ID number, which need to be consistent with the actual field values. A semantic structure graph G=(V,E) is introduced to model the dependencies between fields, where V represents all field nodes and E represents dependency edges. For example:

[0043] "Birth date"→"ID number" (ID number generation depends on birth date),

[0044] "Gender"→"Name" (partial semantic consistency constraint);

[0045] Step 12, MOCK semantic rule generation. The core of this step is to automatically match and generate semantic rules for each field to ensure that the simulation data not only conforms to the format, but also has realistic semantic consistency. It should be noted that "Mock" refers to a general "Mock data generation" technical concept. "Mock data generation" refers to automatically generating "fake" test data that closely approximates real data in structure and semantics, in order to perform system testing, performance testing, or model training without real data or for privacy compliance needs.

[0046] Define the Mock generation function for each field : ;

[0047] where, is the Mock generation function for the ith field, is the generation function, such as: random sampling from a dictionary, template splicing, self-incrementing encoding, timestamp generation, etc. is the parameter of the generation function, such as maximum and minimum value, regular expression, weight distribution; is the data source, such as the surname library, place name library, ID number area table, etc., and x is the field value generated above, i.e., the previously generated field value.

[0048] The generation function includes the following four types: static field rules, dynamic field rules, format verification rules, and distribution sampling control rules. The specific explanations are as follows:

[0049] (1) Static field rules: field values are randomly selected from a pre-defined data set, such as combining the surname library and name library for the name field (used for generating data such as names, company names, and place names);

[0050] (2) Dynamic field rules: field values are generated dynamically depending on other fields, such as the ID number being generated by combining the "birth date" and "gender" fields (used for generating ID numbers, depending on gender, birth date, province, city, and district);

[0051] (3) Format verification rules: field values need to meet the pre-set verification format, such as ID card needs to meet the regular rules, verification bit algorithm, etc. (used in mobile phone number, email, account number, etc. Data generation scene);

[0052] (4) Distribution sampling control rules: analog data should meet the pre-set distribution, such as the number of people aged 20-40 years old accounts for 70%, and the control is realized through weight allocation. (Data needs to meet the pre-set data distribution or probability model, used for age, gender, income, etc. Data generation).

[0053] For example:

[0054] Name field ; The rule function of generating "name" randomly selects a surname from the surname list, and then randomly selects a name from the name list, and splices the two. The surname dictionary library, such as "Zhao, Qian, Sun, Li"; The name dictionary library, such as "Xiaohong, Qiang, Siyuan"; Join means splicing.

[0055] ID card field ; The rule function of generating "ID card number" generates an 18-digit ID card number according to the given region r, birth date b, gender g and checksum. Indicates the region code, such as "440100" for Guangzhou, Indicates the birth date, such as "19950121", Indicates the sequence code, odd for male and even for female, such as "023" for male, Indicates the check code, which is calculated according to the first 17 digits.

[0056] Verification code calculation steps:

[0057] 1. Convert the previously generated 17 digits into an array, such as A'=[1,1,0,1,0,5,1,9,4,9,1,2,3,1,0,0,2];

[0058] 2. Set the weight factor list W=[7,9,10,5,8,4,2,1,6,3,7,9,10,5,8,4,2], the specific numbers can be set by yourself;

[0059] 3. Set the mapping list M'=[1,0,X,9,8,7,6,5,4,3,2], the order can be self-defined, but it must contain these 11 elements;

[0060] 4. A and W calculate the weighted sum: 1*7+1*9+0*10.......=167, take the remainder of 11, 167%11=2, then the verification bit is M[2]=X.

[0061] Step 13, Mock data generation and combination. This step executes all rule generation functions to generate structured and logically consistent simulation data.

[0062] Step 13-1, Each field is generated according to the MOCK generation function The basic Mock value is generated independently, and the system parses the generation path from the configured rules. Mock value is a semantic simulation that represents a typical value that this field may appear in the real world, consistent with the field's format, distribution, and context dependence. At the same time, it is also the raw material for generating simulation data: Mock value is the raw material for building complete records, and it may also be spliced with other field values or subjected to verification algorithms (such as calculating check digits) to become the final field. The basic Mock value is the most original generation result of the field before it is combined with other fields. For fields that do not contain dependent synthesis, the basic Mock value is the final field value; if it depends on the context, it is the final field value calculated from the existing basic Mock values in the context.

[0063] Exemplarily, according to the configuration information of the generation rule of the field "id_card" (generally written in a unified configuration file, such as a JSON file, for example, field_rules.json; or written in code: such as Python dict / Java Map, these configuration information is generally loaded in the Mock data generation task initialization stage to control the subsequent generation logic), the generation path is parsed, and the steps are as follows:

[0064] Suppose the configuration is {

[0065] "field":"id_card",

[0066] "type":"composite",

[0067] "dependencies":["region", "birthday", "gender"],

[0068] "rule":"concat(region_code,format(birthday,'yyyyMMdd'),gender_seq,checksum"}.

[0069] Parsing process:

[0070] 1. Field name: "id_card";

[0071] 2、System identifies the field type as composite, indicating that the field is not independently generated, but depends on other fields for calculation, and is a composite field;

[0072] 3、Analyze dependencies→ need to use the values of "region", "birthday", and "gender" fields;

[0073] 4、Extract function chain in rule expression: the generated rule "rule" specifies how to generate the target value (such as splicing, formatting, etc.) through these dependent fields.

[0074] region_code→ Get the administrative region code from the region field;

[0075] format(...)→ Date format for birthday field;

[0076] gender_seq→ Generate odd-even sequence code according to gender;

[0077] checksum→ Perform checksum function on the first 17 bits.

[0078] Get the generation path:

[0079] ["region_code"→"format(birthday)"→"gender_seq"→"checksum"],

[0080] This path is the specific operation chain for generating the ID number, which describes the generation trajectory from the input basic Mock value to the final field value.

[0081] Step 13-2, the specific process of generating basic Mock value for each field according to MOCK generation function: each field generates basic Mock value according to the rule, and the system parses the generation path from its configured rule: prepare context object X1; read parameters and data sources ; call the generation function ; generate basic Mock value vi; write the basic Mock value vi back to the context information for subsequent use; according to the semantic structure diagram, judge whether the field depends on other fields, if the field does not depend on other fields, the basic Mock value vi is the final field value, if the field depends on other fields, the basic Mock value is an intermediate value, which participates in rule calculation or splicing to get the final field value.

[0082] Exemplarily, the generation steps of the basic Mock value are as follows:

[0083] 1. Prepare the context object X1. For example, X1 = {"name":"Zhang San","gender":"Male","birthday":"1992-03-15"};

[0084] 2. Read parameter P i and data source D i .

[0085] Parameter P i For example, P of id_card i It may contain: 1) the administrative level (province / city / county) to be used; 2) the parity rule for generating the gender code; 3) the check code algorithm type.

[0086] Data source D i For example: 1) a regional code table (depending on province, city, and county); 2) a gender-to-sequence code mapping dictionary; 3) a check code mapping table.

[0087] 3. Call the generation function F i .

[0088] Input:

[0089] P i : Parameters that control the generation details;

[0090] D i Real datasets, mapping tables, etc.;

[0091] Context (i.e., X1): The upstream field values ​​that have already been generated;

[0092] implement:

[0093] Generating function F i Depending on your rule type (static extraction, dynamic combination, format validation, distributed sampling, etc.), access field values ​​in the Context (such as context["birthday"], context["gender"]) when necessary, from D i Read a dictionary or mapping from it;

[0094] According to P i The strategies (weights, regular expressions, templates) in the code compute a single scalar or string result;

[0095] 4. Obtain the basic Mock value vi.

[0096] For example:

[0097] vi=F_id_card(P_id_card,D_id_card,{"gender":"male","birthday":"1992-03-15"})

[0098] =="44010019920315027",

[0099] 5. Write the result back to the context object for later use.

[0100] X1[fieldname]=vi.

[0101] Based on the defined data model and the Mock generation function for each field, the field values ​​of each field are generated and then combined to obtain the simulation data.

[0102] For example, for fields with dependencies, the steps for generating the final field value are as follows:

[0103] Assuming the generated base Mock value is:

[0104] region → “Guangzhou City”, birthday → “1992-06-15”, gender → “Male”

[0105] Execute in sequence:

[0106] region_code("Guangzhou") → "440100",

[0107] format("1992-06-15","yyyyMMdd")→"19920615",

[0108] gender_seq("male")→"027" (hypothetical odd-numbered last digit).

[0109] checksum("44010019920615027")→"X",

[0110] The final result is: 44010019920615027X.

[0111] Step 13-3, the steps for generating the simulation data are as follows:

[0112] load Configuration file, parsed fields and Mock generation function ;

[0113] Construct a semantic structure graph;

[0114] The generation order of fields is determined by the dependency order: fields without dependencies are generated first, and fields with dependencies are generated later.

[0115] According to the generation order, the Mock generation function of the field is called in turn If the field does not have a dependency relationship, the basic Mock value vi of the field is obtained as the field value, and if the field has a dependency relationship, the basic Mock value vi generated by other fields is obtained as an intermediate value, and the Mock generation function is combined to obtain the field value;

[0116] Generate field values field by field. If it is a basic type, directly call the basic generation function (such as a random string, a random time, etc.); If it is a composite type, first get the value of the dependent field from the context, and execute the combination logic defined in the rule;

[0117] Finally, assemble each field value into a complete simulation data.

[0118] Exemplarily, it is assumed that the simulation data needs to be automatically generated as follows:

[0119] Table 2:

[0120] Name Gender Date of birth ID card Zhang San Male 1992 / 3 / 15 44010119920315021X

[0121] Here, the “name” can be randomly spelled, but the “ID card number” must be generated according to the “gender” and “birthday”;

[0122] We must first generate gender→ then generate birthday→ then generate ID card number using the two, that is, there is a dependency relationship between fields (who generates first, who uses the result of who), so there must be an execution order (that is, “generation path”).

[0123] 1, Load fields and rules. The system sees the defined fields as follows:

[0124] name (name): randomly spell a name from a dictionary

[0125] gender (gender): randomly select one from “male / female”

[0126] birthday (birthday): generate a date according to the age distribution

[0127] id_card (ID card number): depend on gender+birthday to spell out a legal ID card number

[0128] Each field has a rule R i (x), which tells the system how to generate the value of this field.

[0129] 2, Build a “dependency graph”. The system identifies the field dependencies:

[0130] id_card←gender,birthday,other fields have no dependency.

[0131] 3、Determine the generation order (topological sorting). First generate the field without dependency, and then generate the field with dependency: order = [name, gender, birthday, id_card]

[0132] 4、Formally start generating data.

[0133] Step1: Generate name

[0134] Call R_name({}), randomly combine from surname and name → "Zhang San"

[0135] Step2: Generate gender

[0136] Call R_gender({}), randomly → "male"

[0137] Step3: Generate birth date

[0138] Call R_birthday({}), assume → "1992-03-15"

[0139] Step4: Generate ID card number

[0140] Now you have gender = male, birthday = 1992-03-15

[0141] Pass these two values to R_id_card({gender: "male", birthday: "1992-03-15"})

[0142] System performs ID card number splicing + verification → gets "44010119920315021X"

[0143] Step5: Assemble into a complete record

[0144] d_1={

[0145] name: "Zhang San",

[0146] gender: "male",

[0147] birthday: "1992-03-15",

[0148] id_card: "44010119920315021X"

[0149] }。

[0150] Step 14, simulate data generation logic optimization steps:

[0151] Step 14-1, Collect feedback information, including: the ratio of simulation data causing test failure in system testing, the probability of simulation data being judged as low quality or fake in model training, the proportion of samples that artificial labeling system considers as not conforming to reality; user explicit feedback;

[0152] Step 14-2, Construct feedback adjustment function : where m is the number of generated simulation data, is the kth simulation data, is the reality score function for the kth data, outputting a score value, such as 0~1, is the expected score or target value for the kth data, representing the quality, authenticity or label value that should be achieved, is the gradient symbol, indicating that the parameters are updated (such as adjusting weights, adjusting generation strategies) by minimizing the loss function;

[0153] Step 14-3, through evaluate the deviation of simulation data from the expected target, and then adjust the parameters of the Mock generation function.

[0154] Exemplarily, the simulation data generation logic optimization steps are as follows:

[0155] 1. The system generates a batch of simulation data {d1, d2,..., dk,..., dn}; k m ;

[0156] 2. Execute the scoring function S(d k :) for each data to get its "reality score", such as scoring from the following dimensions:

[0157] · Is the structure legal?

[0158] · Are the values common?

[0159] · Do multiple fields match (such as gender and ID number)?

[0160] 3. Compare each data with the expected score E k ;

[0161] · E k can be a static target value (such as we hope all >0.9);

[0162] · It can also be the result of artificial labeling or expert scoring;

[0163] 4. Take the square of the deviation as the target loss function;

[0164] ​The smaller the deviation, the more reasonable the Mock rule is.

[0165] 5. Adjust the parameters in Mock generation rules using △;

[0166] For example:

[0167] · Change random weights;

[0168] · Modify rule paths (such as concatenation order, dependent fields);

[0169] · Adjust the selection ratio of external data sets (high frequency vs. low frequency);

[0170] 6. Write the optimized rules back into the configuration table and enter the next round of simulation generation;

[0171] Let's take a specific example

[0172] The "address" field in the Mock rule currently randomly extracts national place names, but a certain business wants the addresses to be concentrated in the "South China region" to simulate user concentration. The actual score S(d k ) shows that the current address distribution covers the entire country, far from the expected distribution E k . The optimization goal is to make (S(d k :)-E k :)² smaller, i.e., to make the simulation distribution closer to the target

[0173] After calling , the system may automatically adjust the sampling weights of Mock rules for place names (increase the sampling probability of Guangdong, Fujian, and Guangxi) or change the address field generation function parameters to make the generated values closer to the expected values.

[0174] The above steps provide an intelligent simulation data generation method combining semantic rule engines, data reality scoring, and feedback adjustment mechanisms, with the following innovations: (1) Multi-level semantic rule parsing engine: abstract data model is parsed into field semantics, and Mock data generation logic is dynamically controlled based on context rules; (2) Feedback-driven simulation optimization mechanism: collect historical test results, user feedback, and system adaptability indicators to dynamically adjust simulation data generation logic; (3) Configurable data generation syntax tree: define field dependency relationships, value ranges, and structure templates based on configuration description language to achieve rapid customization of generation strategies.

[0175] Step 20, distributed performance testing: composed of a master, multiple agent nodes and a performance sensing module (PerfSense), each agent node registers its capability vector to the master, the master builds a capability matrix and executes a distributed load-aware scheduling algorithm according to the capability matrix, calculates the priority value of each agent node, and then dispatches tasks to the agent nodes according to the priority value, each agent node collects performance indicators through the performance sensing module and performs indicator screening, and records the indicator data that does not meet the requirements for local compression or discarding; the master constructs a time series vector for each node and forms a network-wide performance matrix from multiple node data, calculates the performance score and identifies bottlenecks, and completes the performance test.

[0176] The distributed performance test can be specifically divided into the following steps:

[0177] Step 21, system initialization and node registration.

[0178] 1. The master is deployed on a central control node or management node, and first starts the control service components, including the node management module, the task scheduling module, the performance data fusion module, etc. The master starts the broadcast service by configuring a local listening port (such as a TCP port or a WebSocket service), periodically publishes discovery signals to the network, and supports automatic access of nodes.

[0179] 2. An Agent program is deployed on each server to be tested as an agent node. After starting, the Agent actively listens to the broadcast signal of the master, and after successfully parsing the broadcast information, sends a registration request to the master. After each agent node h is online, it automatically registers its capability vector to the master : , where is the number of CPU cores, is the available memory, is the bandwidth, is the current load.

[0180] The master dynamically updates the node list and constructs a capability matrix A:

[0181] Node heartbeat mechanism and connection maintenance. To ensure the stability of the system and the traceability of the node state, the Agent sends a heartbeat packet to the master at a period Th (such as 10 seconds), carrying the current load change information and the running state. If the master does not receive the node heartbeat for N consecutive periods, it is considered that the node is offline.

[0182] Step 22, test task generation and scheduling

[0183] Step 22-1, user configures test strategy, including the number of concurrent users , test script set Test cycle .

[0184] Step 22-2, master machine executes distributed load-aware scheduling algorithm according to capability matrix

[0185] ;

[0186] wherein is the priority value of assigning script to agent node h, is the script resource requirement weight (how to determine the weight of each script ), , , , is the weight coefficient, which is configured according to the system state, such as increasing for CPU-intensive scenarios or increasing for network-intensive scenarios. The scripts can be classified and assigned weights according to business understanding or experience :

[0187] CPU-intensive, such as a large amount of calculation → = 0.9;

[0188] Database access-intensive → = 0.8;

[0189] Network I / O-intensive, such as high-frequency HTTP request → = 0.7;

[0190] Memory-intensive, such as large object construction → = 0.6;

[0191] Lightweight business script → = 0.3~0.5.

[0192] Step 22-3, according to the priority value of the node, the master machine assigns scripts, input parameters and pre-set user quantity to each agent node in HTTP / HTTPS protocol mode, and generates a unique task ID for each agent node to facilitate subsequent result association. Specifically,

[0193] An n*r priority value matrix is constructed:

[0194]

[0195] n: number of nodes, r: number of test scripts, P hj : priority value of agent node h executing script S j , the higher the value, the more suitable. ​

[0196] Flow: 1, traverse each script S j , extract the jth column, get all node priority P 1j ,....,P nj ; 2, sort the priority to get the priority node queue [I1, I2,...I n ]; 3, traverse the queue: if a certain agent node in the priority node queue currently meets the minimum demand of the remaining resources, assign the task S j to it; if not, try the next node; if all nodes do not meet, put it into the delay queue to wait for resource release.

[0197] Example: there is a priority matrix, as shown in the following table:

[0198] Table 3:

[0199]

[00100] S1 [S2] ​ 0.82 0.75 [I2] 0.92 0.6 [I3] 0.71 0.85

[0200] The task allocation flow is: I2 executes S1, I3 executes S2 (maximum value), if I2 task is too much, then execute by the suboptimal I1.

[0201] Step 23, performance index collection and processing.

[0202] Step 23-1, each agent node collects the following indexes through the PerfSense module:

[0203] CPU usage: ;

[0204] Memory occupancy rate: ;

[0205] Disk I / O rate: ;

[0206] Network I / O throughput: ;

[0207] Among them, is the total time spent by CPU running user programs, is the total time spent by CPU in kernel mode, is the total time of CPU (the sum of all states such as user mode, kernel mode, idle, waiting for I / O, etc.), UsedMemory is the actual physical memory capacity used by the system, TotalMemory is the total physical memory capacity available to the system, is the total amount of data read from the disk in time period t, is the total amount of data written to the disk in time period t, is the sampling interval time, is the number of bytes sent to the network in time period t, The number of network bytes received in the time period t.

[0208] Step 23-2, PerfSense performs index filtering:

[0209] Local compression or discard is performed on indexes with performance changes less than a threshold value: .

[0210] represents the absolute value of the CPU usage rate change amplitude at the current time t, that is, , represents the CPU usage rate at time t, represents the CPU usage rate at time t-1, when is greater than a certain set threshold value, it indicates that the system load fluctuates sharply in a short time, and there may be performance bottlenecks or burst tasks; when it is less than a certain set threshold value, it indicates that the system load is relatively stable.

[0211] This mechanism is essentially a performance index change filtering mechanism and can be applied to multiple indexes: CPU usage rate change, memory occupancy rate change, network throughput change, and disk I / O throughput change. A separate threshold value can be set for each index for compression or discard.

[0212] If the performance change is less than a threshold value , it indicates that the index has little or no change in the current time window, i.e., the system is running stably or the load in this dimension is basically constant or the current data collection has little value for fault detection and trend judgment; such data is redundant data with low information entropy, and the cost performance of uploading and storing is low. Index data records that meet the condition of "performance change < threshold value" are compressed locally or discarded completely without uploading to the host computer; specific methods include skipping uploading (not sending the data) and merging compression (merging multiple consecutive small changes into a stable value).

[0213] The main purpose of compression or complete discard is to improve the overall efficiency and intelligence of the performance monitoring system: only useful change data is uploaded, saving network bandwidth and reducing network transmission load; a large amount of invalid data is not written into the database, saving space and reducing storage costs, avoiding the host computer from processing redundant information, and improving real-time processing performance; reducing the upload frequency of the agent node, reducing its processing pressure, and saving agent node resources.

[0214] Step 24, performance result aggregation and display.

[0215] Step 24-1, data aggregation and vector construction. All agent nodes report test indexes to the host computer after the test execution ends or reaches the reporting period, and the host computer constructs time series vectors for each node​ :

[0216] ;

[0217] Multiple agent node data constitutes a network performance matrix :

[0218] ;

[0219] Step 24-2, performance score calculation and bottleneck identification. For each time point t, the master node evaluates the performance load level of each node based on the weighted scoring model : ;

[0220] If If the duration exceeds the warning threshold θ, it is considered that the node has a bottleneck, triggering the alarm mechanism.

[0221] Each agent node in step 20 of the present application has the ability of autonomous reporting and intelligent collaboration, breaking through the centralized polling model and realizing the following technical innovations:

[0222] (1) Collaborative distributed index collection model: each node actively collects and uploads, without relying on centralized polling.

[0223] (2) Chain index propagation mechanism: supports data reuse and sharing between nodes, reducing the load of the master.

[0224] (3) Local index pre-analysis and filtering mechanism: nodes calculate in advance, filter out invalid / redundant information, and only upload high-value data.

[0225] (4) Index fusion mapping function: define a unified performance index mapping model for heterogeneous systems, with portability.

[0226] (5) Master-guided test script distribution and load-aware scheduling algorithm: enables automatic distribution and balanced execution of test tasks.

[0227] Step 30, real-time performance monitoring.

[0228] Step 31, heterogeneous environment support and multi-dimensional data collection:

[0229] 1. Environment adaptation. The monitoring system supports multiple operating systems (such as UOS used in domestic operating systems, Kylin, etc.) and hardware platforms (such as X86 architecture and domestic ARM architecture devices). For different operating systems, customized performance data collection interfaces are provided. For example, under Linux, CPU, memory, network I / O, etc. performance data can be read through the procfs file system; while in the Windows environment, performance data is obtained through WMI (Windows Management Instrumentation).

[0230] 2. Multi-dimensional data collection. Data collection content includes but is not limited to: 1) CPU usage: collect CPU load, CPU temperature, CPU core number, and load of each CPU core. 2) Memory usage: including total memory, used memory, free memory, cache memory, and swap space usage. 3) Disk I / O: disk read / write rate, disk usage, disk response time, etc. 4) Network I / O: including network bandwidth usage, network delay, packet loss rate, network traffic, etc.

[0231] 3. Standardized data format. The monitoring system needs to format the collected data uniformly, such as through JSON, XML or Protobuf for data transmission, to ensure cross-platform compatibility.

[0232] Step 32, high real-time monitoring. Use HTTP or WebSocket service for data polling and pushing. Specifically, establish a long connection with the server through WebSocket to ensure that performance data can be transmitted to the monitoring platform in real time. The key to realizing real-time monitoring is a low-latency data transmission protocol, such as the bidirectional communication mechanism provided by WebSocket, which can effectively reduce latency and transmit data in each dimension in real time.

[0233] Monitoring parameters include: data reporting average delay , abnormal detection time difference , node timeout not reported , number of lost data packets per minute .

[0234] 1, data reporting average delay , that is, the time delay from the occurrence of an event to the collection and successful transmission of data and the display on the monitoring platform; represents the time point at which the abnormal information or performance index change is finally displayed or visualized by the system; represents the time point at which a performance abnormal event (such as CPU utilization soaring, memory leakage, I / O blocking, etc.) is first identified or triggered by the system.

[0235] 2. Abnormal detection time difference , that is, the time from the real occurrence of performance anomaly to the detection and alarm of the monitored system. Indicates the time point when the system officially issues a performance alarm.

[0236] The system realizes dynamic evaluation of monitoring real-time performance and abnormal early warning by monitoring the changes of , , and

[0237] For example, the alarm rules of each monitoring parameter are as follows:

[0238] Parameter Description Alarm rule Data reporting average delay Greater than 5s, 3 times, determine data delay exception Abnormal detection time difference Greater than 10s, determine alarm reaction delay Node does not report for a certain time Greater than 30s, node drop or system exception Number of lost data packets per minute Greater than 5, network congestion or service pressure

[0239] Step 33, abnormal detection and automatic response: set threshold values for each performance indicator and monitoring parameter, and issue an abnormal early warning when the threshold values are exceeded. For example, CPU load exceeding 80%, memory usage exceeding 90%, disk usage exceeding 85%, etc., which triggers real-time alarm.

[0240] Step 34, data collection and intelligent analysis step:

[0241] Step 34-1, clean and pretreat the collected data to remove abnormal values and noise data. For example, the collected CPU usage data may have instantaneous abnormal fluctuations, which need to be smoothed based on sliding window.

[0242] Step 34-2, based on the hybrid prediction model LSTM-ARIMA, combined with time series analysis method, train the performance data of the system to predict the future performance change trend; the structure of LSTM-ARIMA fusion model is as follows:

[0243] ;

[0244] ;

[0245] ;

[0246] Among them, in the formula of LSTM layer, is the hidden state output at the current time, is the unit state at the current time, is the unit state at the last time, which is used to capture long-term dependence, is the hidden state at the last time, which contains memory information, is the input data at time t-1, that is, the observation value, which comes from historical records and is part of the input sequence; in the formula of ARIMA layer, is the predicted value of the current time t by the ARIMA layer,​ The regression coefficients for the AR component are... These are historical observations used for the regression modeling. For the current white noise term, The coefficients of the MA component are... For the historical white noise term, Let this be the order of the AR part. Let be the order of the MA part; in the fusion output formula, The final fusion output, i.e., the current prediction value, For the output from the ARIMA layer, The output from the LSTM layer, For splicing the outputs of ARIMA and LSTM, This is the weight matrix of the fusion layer. For the bias term of the fusion layer, This represents an activation function used to enhance nonlinear expressive power;

[0247] LSTM layer: Processes sequence data Output the hidden state h t and cell state c t Capture non-linear patterns and long-term dependencies in time series;

[0248] ARIMA layer: AR part Modeling autoregression, MA part Process moving averages to output linear forecasts ;

[0249] Fusion output: ARIMA predictions and LSTM state h t splicing, through the weight matrix and bias A linear transformation is performed, and the Sigmoid activation generates the final prediction. .

[0250] Intelligent analysis formula: ;in, Indicates a future moment Predictive performance metrics at the time For the trained hybrid prediction model LSTM-ARIMA, This provides real-time performance data for the current moment.

[0251] Step 35, problem diagnosis and optimization suggestion. Through the intelligent analysis system, potential performance bottlenecks can be identified. For example, if the system memory frequently reaches the threshold, it may be necessary to increase the physical memory or optimize the memory management strategy. The system can provide optimization suggestions and automatically generate optimization reports to guide the operation and maintenance personnel to upgrade the system or adjust the configuration.

[0252] The performance monitoring method for the domestic environment provided in step 30 of the application can support multi-dimensional data monitoring in a heterogeneous environment, including CPU, memory, disk I / O, network I / O, etc., and combine intelligent analysis technology to intelligently process monitoring data, quickly discover potential performance problems and predict future performance changes, and provide system optimization suggestions.

[0253] The application provides a distributed performance testing method for a domestic environment, which combines research on online distributed stress testing technology in a domestic environment to realize performance testing and monitoring functions of a business system in a domestic environment. The implementation route of the application includes three steps: firstly, intelligent simulation data generation technology is researched to automatically generate massive simulation test data with high fidelity to simulate real power business scenarios and provide test data simulating real scenarios for testing and verification of the business system. Secondly, distributed performance testing technology for a domestic environment is researched to support performance testing scenarios under high concurrency. Finally, performance monitoring technology for a domestic environment is researched to realize accurate collection and analysis of performance indexes such as CPU usage, memory usage, disk I / O, etc., to meet the collection requirements of the performance indexes. The whole system deeply integrates simulation data generation, distributed stress injection and heterogeneous environment monitoring to form an automatic closed loop of "data construction-stress testing-bottleneck diagnosis", which especially solves the industry problems of low cluster monitoring efficiency and fragmented analysis of heterogeneous resources in a domestic environment.

[0254] Based on the same inventive concept, the application also provides a device for executing the method steps in embodiment one.

[0255] Embodiment two

[0256] A distributed performance testing device for a domestic environment includes:

[0257] The simulation data generation module performs the following steps: defining a data model M, the data model M being composed of multiple fields; defining a Mock generation function for each field, each field generating a basic Mock value according to the Mock generation function, if the field does not depend on other fields, the basic Mock value being the final field value, if the field depends on other fields, the basic Mock value being an intermediate value participating in rule calculation or splicing to obtain the final field value; and generating field values of each field according to the defined data model and Mock generation function of each field, and combining the field values to obtain simulation data.

[0258] Distributed performance test module: composed of a host computer, multiple agent nodes and a performance perception module; the following steps are executed: each agent node registers its capability vector to the host computer, the host computer builds a capability matrix and executes a distributed load perception scheduling algorithm according to the capability matrix, calculates the priority value of each agent node, and then dispatches tasks to the agent nodes according to the priority value; each agent node collects performance indicators through the performance perception module and performs indicator screening, and records the indicator data that does not meet the requirements for local compression or discarding; the host computer constructs a time series vector of each node and a whole network performance matrix composed of multiple node data, calculates a performance score and identifies a bottleneck, and completes performance testing;

[0259] Performance monitoring module: performs performance monitoring and real-time monitoring and sends out a warning when an exception occurs; data collection: multi-dimensional collection of performance indicators and unified formatting of data formats; the performance indicators at future time points are predicted by using the trained hybrid prediction model LSTM-ARIMA.

[0260] Since the device introduced in the second embodiment of the present application is the device used to implement the method of the first embodiment of the present application, the specific structure and variations of the device can be understood by those skilled in the art based on the method introduced in the first embodiment of the present application, and therefore will not be described here. Any device used in the method of the first embodiment of the present application belongs to the scope of protection of the present application.

[0261] The above is only a specific embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structural transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.​

Claims

1. A distributed performance testing method for a domestic environment, characterized in that: Comprise the following steps: Generating simulation data: defining a data model M, the data model M is composed of a plurality of fields; defining the Mock generation function of each field, each field generates a basic Mock value according to the Mock generation function, if the field does not depend on other fields, the basic Mock value is the final field value, if the field depends on other fields, the basic Mock value is an intermediate value, which participates in rule calculation or splicing to obtain the final field value; according to the defined data model and the Mock generation function of each field, the field values of each field are generated to obtain the simulation data by combination; Distributed performance test: composed of a host computer, a plurality of agent nodes and a performance perception module, each agent node registers its capability vector to the host computer, the host computer builds a capability matrix and executes a distributed load perception scheduling algorithm according to the capability matrix, calculates the priority value of each agent node, and then distributes tasks to the agent nodes according to the priority value, each agent node collects performance indicators through the performance perception module and performs indicator screening, and records the indicator data that does not meet the requirements for local compression or discarding; the host computer constructs the time series vector of each node and groups the node data to form a network performance matrix, calculates the performance score and identifies the bottleneck, and completes the performance test.

2. The method of claim 1, wherein the method is a performance testing method for a domestic environment. The definition process of the data model M is that the data model M is composed of a plurality of fields, each field has three attributes, including field name, field type and field constraint; for some fields with logical or structural dependence, the dependence between fields is modeled through a semantic structure graph G=(V,E), V represents all field nodes, and E represents dependent edges.

3. The method of claim 2, wherein: The Mock generation function of each field is defined as: wherein, is the Mock generation function of the i-th field, is the generation function, is the parameter of the generation function, is the data source, and x is the field value generated above; the generation function includes the following four types: static field rules, dynamic field rules, format verification rules, and distribution sampling control rules; the static field rules refer to that the field value is randomly extracted from a preset data set, the dynamic field rules refer to that the field value is dynamically generated depending on multiple fields, the format verification rules refer to that the field value needs to satisfy a preset verification format, and the distribution sampling control rules refer to that the simulation data conforms to a preset distribution and is controlled through weight distribution.

4. The distributed performance testing method for a domestic environment according to claim 3, characterized in that: The field value generation step of the field includes: Each field independently generates a base mock value according to rules, the system parses the generation path from its configured rules: prepare context object X1 ; read parameters and data source ; call generation function ; generate base mock value vi; write base mock value vi back to context information for subsequent use; According to the semantic structure graph, it is judged whether the field depends on other fields, if there is no dependence, the basic Mock value vi is the final field value, if there is no dependence, the basic Mock value vi is an intermediate value, which participates in rule calculation or splicing to obtain the final field value.

5. The method of claim 4, wherein: The simulation data generation logic optimization step includes the following steps: Loading the configuration file, parsing the fields and the Mock generation function ; constructing a semantic structure diagram; determining a generation order, generating fields without dependencies first and then generating fields with dependencies; and then calling the Mock generation function of each field in turn according to the generation order ; if the field has no dependency relationship, obtaining the basic Mock value vi of the field as the field value, and if the field has a dependency relationship, taking the basic Mock value vi generated by other fields as an intermediate value and combining the Mock generation function to obtain the field value; and finally assembling each field value into a piece of simulation data and adding the data to the dataset.

6. The method of claim 1, wherein: It also includes simulation data generation logic optimization steps: Collecting feedback information, including: the ratio of simulation data leading to test failure in system test, the probability of simulation data being judged as low quality or false in model training, the sample proportion that the artificial labeling system considers the data to be inconsistent with the actual situation; user explicit feedback; Constructing feedback adjustment function : where m is the number of generated simulation data, is the kth simulation data, is the reality score function for the kth data, outputting a score value, is the expected score or target value for the kth data, representing the quality, authenticity or label value that should be achieved, is the gradient sign; By The deviation of the simulation data from the expected target is evaluated and the parameters of the Mock generation function are adjusted.

7. The method of claim 1, wherein the method is for a domestic environment. The distributed performance test can be divided into the following steps: System initialization and node registration: master is deployed on the central control node or management node, master starts broadcast service through configuring local listening port, periodically publishes discovery signal to network, supports node automatic access; each server to be tested is deployed with Agent program, becomes agent node, Agent program actively listens to master's broadcast signal after starting, and sends registration request to master after parsing broadcast information; each agent node h is online, automatically registers itself to master : , wherein is the number of CPU cores, is the available memory, is the bandwidth, is the current load; the master dynamically updates the node list to build the capability matrix A: ; node heartbeat mechanism and connection maintenance, the agent node sends a heartbeat packet to the master with a period Th, carrying the current load change information and running state, if the master does not receive the node heartbeat for N consecutive periods, it is determined that the node is offline; Test task generation and scheduling: Users configure test strategies, including the number of concurrent users. Test script collection Test cycle The main control unit is based on the capability matrix. Execute the distributed load-aware scheduling algorithm: ,in To make the script The priority value assigned to proxy node h. Weighting of script resource requirements, , , , This is the weighting coefficient; based on the priority value of the proxy node, the host computer distributes the script, input parameters, and preset number of users to each proxy stage via HTTP / HTTPS protocol, and generates a unique task ID for each proxy node; Performance indicator collection and processing: each agent node collects the following indicators through the performance perception module: CPU usage : ; Memory footprint : ; Disk I / O rate : ; Network I / O throughput : ; wherein, is the total time spent by the CPU running programs in user mode, is the total time spent by the CPU in kernel mode, is the total time of the CPU, UsedMemory is the actual physical memory capacity used by the system, and TotalMemory is the total physical memory capacity available to the system, is the total amount of data read from the disk in the time period t, is the total amount of data written to the disk in the time period t, is the sampling interval time, is the number of bytes sent to the network in the time period t, is the number of bytes received from the network in the time period t; The performance-aware module performs metric filtering: locally compressing or discarding metrics whose performance changes are less than a threshold ; Performance result aggregation and vector construction: each agent node reports test indicators to the host computer at the end of test execution or after reaching the reporting period, and the host computer constructs a time series vector for each agent node : ； Multiple proxy node data sets make up a global network performance matrix : ； Performance score calculation and bottleneck identification: for each time point t, the master evaluates the performance load level of each node based on a weighted scoring model : ； If If the duration exceeds the warning threshold θ, it is considered that the agent node has a bottleneck, triggering the alarm mechanism.

8. The method of claim 1, wherein: Including the following steps: also including performance real-time monitoring steps: Data collection: multi-dimensional collection of performance indicators and unified formatting of data format; Real-time monitoring: monitoring parameters include: data reporting average delay , abnormal detection time difference , node timeout not reported , the number of lost data packets per minute ; Abnormality detection and automatic response: set thresholds for each performance indicator and monitoring parameter, and issue an abnormal warning when the threshold is exceeded.

9. The method of claim 8, wherein the method is for a domestic environment. It also includes data collection intelligent analysis steps: Clean and pretreat the collected data to remove abnormal values and noise data; Based on the hybrid prediction model LSTM-ARIMA, combined with time series analysis method, the performance data of the system is trained to predict the future performance trend; The structure of the LSTM-ARIMA fusion model is as follows: ； ； ； wherein, in the LSTM layer formula, is the hidden state output at the current time, is the cell state at the current time, is the cell state at the previous time, used to capture long-term dependencies, is the hidden state at the previous time, containing memory information, is the input data at time t-1, i.e., the observation value, from the historical record, which is part of the input sequence; in the ARIMA layer formula, is the predicted value of the ARIMA layer for the current time t, is the regression coefficient of the AR part, is the historical observation value, used to model the regression part, is the current white noise term, is the coefficient of the MA part, is the historical white noise term, is the order of the AR part, is the order of the MA part; in the fusion output formula, is the final fusion output, i.e., the current prediction value, is the output from the ARIMA layer, is the output from the LSTM layer, is the concatenation of the outputs of ARIMA and LSTM, is the weight matrix of the fusion layer, is the bias term of the fusion layer, represents the activation function, used to enhance the non-linear expression ability; LSTM layer: handles sequential data , output hidden state h t and cell state c t , captures non-linear patterns and long-term dependencies in time series; ARIMA layer: AR part Modeling autoregression, MA part Processing moving average, output linear prediction ; Fusion output: ARIMA prediction and LSTM state h t Concatenated, linearly transformed by weight matrix and bias Sigmoid activation to generate final prediction ; Intelligent analysis formula: ; wherein, represents the predicted performance indicator at future time t, is the trained hybrid prediction model LSTM-ARIMA, is the real-time performance data at the current time.

10. A distributed performance testing apparatus oriented to a domestic environment, characterized by: It comprises A simulation data generation module, which performs the following steps: defining a data model M, the data model M being composed of multiple fields; defining a Mock generation function for each field, each field generating a basic Mock value according to the Mock generation function, if the field does not depend on other fields, the basic Mock value being the final field value, if the field depends on other fields, the basic Mock value being an intermediate value participating in rule calculation or splicing to obtain the final field value; generating field values of each field according to the defined data model and Mock generation function of each field, and then combining to obtain simulation data; A distributed performance test module: composed of a master machine, multiple agent nodes and a performance perception module; performing the following steps: each agent node registers its own capability vector to the master machine, the master machine builds a capability matrix and executes a distributed load perception scheduling algorithm according to the capability matrix, calculates the priority value of each agent node, and then dispatches tasks to the agent nodes according to the priority value, each agent node collects performance indicators through the performance perception module and performs indicator screening, and records the indicator data that does not meet the requirements for local compression or discarding; the master machine constructs a time series vector for each node and groups multiple node data into a network-wide performance matrix, calculates the performance score and identifies the bottleneck, and completes the performance test; A performance monitoring module: performing performance monitoring and real-time monitoring and issuing a warning when an exception occurs; Data collection: multi-dimensional performance indicators are collected and the data format is unified. The performance indicators at future time are predicted by the trained hybrid prediction model LSTM-ARIMA. Data collection: multi-dimensional performance indicators are collected and the data format is unified. The performance indicators at future time are predicted by the trained hybrid prediction model LSTM-ARIMA. Data collection: multi-dimensional performance indicators are collected and the data format is unified. The performance indicators at future time are predicted

Images