Test method, test device and electronic equipment

Abstract

The invention discloses a test method, a test device and electronic equipment. The method comprises the following steps: acquiring a test case corresponding to a target application; in the process of performing a chaos test on the target application based on the test case by using a chaos test tool, obtaining performance index parameters of target equipment to which the target application belongs according to a preset time period; the feature vectors of the performance index parameters obtained in the preset time period are input into a performance evaluation model, the performance state of the target equipment in the preset time period is obtained, and the performance evaluation model comprises a convolutional neural network sub-model, a long and short-term memory network sub-model and a converter sub-model.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and specifically relates to a testing method, a testing device and an electronic device. Background Art

[0002] In the process of evaluating device status using chaos testing methods based on automated means, traditional machine learning models cannot fully utilize the timing correlation and global dependencies of hardware performance monitoring data, and cannot capture complex performance fluctuations, resulting in inaccurate evaluation of device status during the test. Summary of the invention

[0003] The embodiments of the present application provide a testing method, a testing device and an electronic device, which can solve the problem of low accuracy in device status assessment during the test process.

[0004] In a first aspect, an embodiment of the present application provides a testing method, the method comprising: obtaining a test case corresponding to a target application; in a process of performing chaos testing on the target application based on the test case using a chaos testing tool, obtaining performance indicator parameters of a target device to which the target application belongs according to a preset time period; inputting the characteristic vector of the performance indicator parameters obtained within the preset time period into a performance evaluation model to obtain the performance status of the target device within the preset time period, wherein the performance evaluation model comprises a convolutional neural network sub-model, a long short-term memory network sub-model and a converter sub-model. In second aspect, an embodiment of the present application provides a testing device, which includes: a first acquisition module, used to acquire test cases corresponding to a target application; a second acquisition module, used to acquire performance indicator parameters of a target device to which the target application belongs according to a preset time period during a process of performing chaos testing on the target application based on the test cases using a chaos testing tool; a processing module, used to input the characteristic vector of the performance indicator parameters acquired within the preset time period into a performance evaluation model to obtain the performance status of the target device within the preset time period, wherein the performance evaluation model includes a convolutional neural network sub-model, a long short-term memory network sub-model and a converter sub-model.

[0005] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0006] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0007] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0008] In a sixth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes at least one computer program, and when the computer program is loaded and executed by a processor, the method described in the first aspect is implemented.

[0009] In an embodiment of the present application, by obtaining a test case corresponding to the target application, and then using a chaos testing tool to perform chaos testing on the target application based on the test case, the performance indicator parameters of the target device to which the target application belongs are obtained according to a preset time period, and finally the characteristic vector of the performance indicator parameters obtained within the preset time period is input into the performance evaluation model to obtain the performance status of the target device within the preset time period, wherein the performance evaluation model includes a convolutional neural network sub-model, a long short-term memory network sub-model and a converter sub-model, which can not only extract local features within the preset time period, but also capture timing correlations and global dependencies, thereby improving the model's ability to recognize complex performance fluctuations and enhancing the accuracy of the evaluation; at the same time, performing performance evaluation according to the preset time period can reduce errors caused by short-term fluctuations or instantaneous load changes, thereby making the evaluation results more stable and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flow chart of a testing method provided in an embodiment of the present application; Figure 2 It is a flow chart of a testing method provided in an embodiment of the present application; Figure 3 is a structural schematic diagram of a testing device provided in an embodiment of the present application; Figure 4 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0012] In the following, in conjunction with the accompanying drawings, a test method, a test device and an electronic device provided in an embodiment of the present application are described in detail through specific embodiments and their application scenarios.

[0013] Figure 1 A test method provided by an embodiment of the present application is shown, and the method can be executed by an electronic device, and the electronic device may include: a server and / or a terminal device. In other words, the method can be executed by software or hardware installed in the electronic device, and the method includes the following steps: S110: Acquire a test case corresponding to the target application.

[0014] In one exemplary embodiment, the obtaining of a test case corresponding to the target application may include the following steps: S112: Obtaining program semantics or control flow of the target application.

[0015] Program semantics refers to the behavior or function of a program during execution, which may include calculation rules, data flow, control flow, input and output, etc. Control flow refers to the execution order and method of the control program of the target application, which may include sequence structure, selection structure, loop structure, etc.

[0016] In this step, by obtaining the program semantics or control flow of the target application, the behavior of the program can be understood and predicted, so that the test cases generated subsequently are more compatible with the target application, which helps to improve the accuracy of the test.

[0017] S114: Generate the test case based on the program semantics; or generate the test case based on the control flow; wherein the test case includes: normal input and abnormal input.

[0018] Exemplarily, normal input may include opening the target application and waiting for startup completion, logging into the target application using the correct username and password, performing normal operations on the target application, including creating, modifying, and deleting files, and checking whether the performance and functionality of the target application meet expectations, etc. Abnormal input may include performing a large amount of data operations when the device to which the target application belongs has insufficient memory, transmitting data when the network connection is unstable, performing operations on the target application that exceed its processing capacity, running multiple high-load programs on the target application at the same time, entering an invalid username or password into the target application, entering data containing malicious code into the target application, entering a value that exceeds the allowable range into the target application, entering data that does not meet format requirements into the target application, etc.

[0019] In another exemplary embodiment, the test cases corresponding to the target application may include the test cases corresponding to the target application itself, and may also include the test cases corresponding to the device where the target application is located. Exemplarily, the test cases corresponding to the device where the target application is located may include: turning on the device and waiting for the startup to complete, logging into the device using the correct user name and password, performing normal operations on the device, including creating, modifying and deleting files, checking whether the performance and functions of the device meet expectations, performing a large amount of data operations when the device memory is insufficient, performing data transmission when the network connection is unstable, performing operations on the device that exceed its processing capacity, running multiple high-load programs on the device at the same time, inputting an invalid user name or password into the device, inputting data containing malicious code into the device, inputting a value exceeding the allowable range into the device, inputting data that does not meet the format requirements into the device, etc.

[0020] S120: In the process of performing chaos testing on the target application based on the test case using the chaos testing tool, obtaining performance indicator parameters of the target device to which the target application belongs according to a preset time period.

[0021] Among them, chaos testing is a testing method based on device status. By measuring the device status, the operating status of the device under different conditions can be tested. Over time, the device can experience a transition from uncertainty. When the device fails during use, it is necessary to re-evaluate its performance and re-determine whether it is stable and continuous in the actual environment. In other words, chaos testing is to test the device based on the known device status and record its change trend to understand the performance of the device in actual applications.

[0022] The performance indicator parameters of the target device may include central processing unit (CPU) occupancy, CPU utilization, memory occupancy, memory utilization, etc. It should be noted that the performance indicator parameter corresponds to at least one time series data set. For example, assuming that the performance indicator parameter includes CPU usage and memory usage, then the CPU usage corresponds to a time series data set, which records the change of the CPU usage at each time point in the preset time period. For example, the time series data set may contain CPU usage recorded once every minute or every second. These values ​​are represented as a vector x(t), where t is a time index, and x(t) represents the CPU usage at time t. For example, if the CPU usage is as follows: t = [0, 1, 2, 3, 4, 5, ...], x(t) = [30%, 32%, 35%, 38%, 40%, 42%, ...]; where x(0) = 30% means that the CPU usage at time t=0 is 30%; similarly, the memory usage also corresponds to a time series data set, which records the change of the memory usage at each time point in the preset time period.

[0023] In an embodiment of the present application, during the chaos test process, the performance indicator parameters of the target device are obtained according to a preset time period, which can not only realize real-time monitoring of the target device status, but also ensure the stability and representativeness of the performance indicator parameters.

[0024] In an exemplary embodiment, before S120, the method may further include: selecting a chaos test tool according to the characteristics of the target device; configuring the chaos test tool and configuring the test environment. Configuring the test environment refers to building a test environment including the target device to ensure that the chaos test tool can correctly communicate with the target device. Optionally, the chaos test tool may be Peach Fuzzer. The framework of Peach Fuzzer may include the following parts: (1) Data model: represents the data structure required for input and output.

[0025] (2) Mutator: mutates the data using different mutation strategies.

[0026] (3) Generator: It can generate simple data types such as string data and integer values. It can also generate complex hierarchical binary data, and even concatenate simple data to generate more complex data types.

[0027] (4) State model: In each test case, the user can configure the initial finite state machine according to the state model and perform maintenance.

[0028] (5) Agent: Communicates with the test engine to monitor the status of the target under test and control its execution.

[0029] (6) Test engine: parses the Pit configuration file and creates corresponding components according to the configuration file for initialization, and then enters the main loop to execute the test cases.

[0030] Among them, configuring Peach Fuzzer may include: building a data model to represent the data structure of input and output; setting a state model to manage the state machine of the test case; and configuring the communication interface between the agent and the test engine.

[0031] Exemplarily, the implementation process of Peach Fuzzer may include: obtaining test cases, sending the test cases to the target application, and then monitoring the behavior of the target application to detect whether there are vulnerabilities or defects; or, using a mutation strategy to modify some fields in the test case, and then sending the modified test case to the target application, and then monitoring the behavior of the target application to detect whether there are vulnerabilities or defects.

[0032] In another exemplary embodiment, the step of obtaining the performance indicator parameters of the target device to which the target application belongs according to a preset time period may include: obtaining the performance indicator parameters of the target device to which the target application belongs according to a preset time period by using a performance monitoring tool. The performance monitoring tool may be determined based on the characteristics of the target device. Exemplarily, assuming that the system of the target device is a Windows system, the performance monitoring tool may be a Performance Monitor or a WMI (Windows Management Instrumentation) service.

[0033] S130: Inputting the characteristic vector of the performance indicator parameter acquired within the preset time period into a performance evaluation model to obtain the performance status of the target device within the preset time period.

[0034] Among them, the performance evaluation model includes a convolutional neural network sub-model, a long short-term memory network sub-model and a converter sub-model.

[0035] It is understandable that traditional machine learning models cannot fully utilize the timing correlation and global dependencies of hardware performance monitoring data, which makes the analysis results prone to deviation and cannot capture complex performance fluctuations. In the embodiment of the present application, by combining the Convolutional Neural Network sub-model (CNN), the Long Short-Term Memory Network sub-model (LSTM) and the Transformer sub-model Transformer to build an integrated performance evaluation model, it can not only extract local features, but also capture timing correlation and global dependencies, thereby improving the model's ability to recognize complex performance fluctuations and enhancing the accuracy of the evaluation.

[0036] In an embodiment of the present application, by obtaining a test case corresponding to the target application, and then using a chaos testing tool to perform chaos testing on the target application based on the test case, the performance indicator parameters of the target device to which the target application belongs are obtained according to a preset time period, and finally the characteristic vector of the performance indicator parameters obtained within the preset time period is input into the performance evaluation model to obtain the performance status of the target device within the preset time period, wherein the performance evaluation model includes a convolutional neural network sub-model, a long short-term memory network sub-model and a converter sub-model, which can not only extract local features within the preset time period, but also capture timing correlations and global dependencies, thereby improving the model's ability to recognize complex performance fluctuations and enhancing the accuracy of the evaluation; at the same time, performing performance evaluation according to the preset time period can reduce errors caused by short-term fluctuations or instantaneous load changes, thereby making the evaluation results more stable and accurate.

[0037] In an exemplary embodiment, obtaining the program semantics or control flow of the target application includes: determining the program semantics and control flow of the target application by statically analyzing the source code of the target application; or determining the program semantics and control flow of the target application by disassembling and reverse engineering the binary file of the target application.

[0038] It can be understood that static analysis refers to identifying key data structures and program vulnerabilities by parsing the source code of the target application, thereby determining the program semantics and control flow of the target application. Binary file analysis refers to disassembling and reverse engineering the binary file of the target application when the source code cannot be obtained, thereby determining the program semantics and control flow of the target application. In other words, static analysis is implemented when the source code of the target application can be obtained, while binary file analysis can be implemented with or without source code, thereby enhancing the flexibility of determining program semantics and control flow.

[0039] In an exemplary embodiment, the test case is generated based on the control flow, including: using a symbolic execution engine to execute the control flow in a symbolic manner and search for multiple candidate execution paths; screening the multiple candidate execution paths based on a heuristic algorithm to obtain a target execution path; and generating a test case corresponding to each target execution path by solving constraints for each target execution path.

[0040] It can be understood that the symbolic execution engine is first used to execute the control process in a symbolic manner, so as to search for multiple candidate execution paths corresponding to the control process, and then the multiple candidate execution paths are screened based on the heuristic algorithm to obtain the target execution path, which covers each logical branch of the control process, and then the constraints of each target execution path are solved to obtain the test cases corresponding to each target execution path.

[0041] Optionally, constraint solving can be implemented based on a constraint solver. That is, during the execution of the symbolic execution engine, for each candidate execution path, a set of path constraints are generated, and the generated path constraints describe all the necessary conditions for the program to execute along the path. The path constraints are taken as input, fed into the constraint solver, and the actual input values ​​that satisfy these conditions are found. The actual input values ​​returned by the constraint solver can be directly used to generate test cases.

[0042] In another exemplary embodiment, after generating the test case corresponding to each target execution path, the method may further include the following steps: Step 1: Using the first fitness function, obtain the first fitness value of each target execution path and the actual path corresponding to the corresponding test case.

[0043] The first fitness value is used to indicate the degree of matching between the target execution path and the actual path of the corresponding test case.

[0044] Step 2: When there is an unmatched path in the target execution path whose first fitness value is less than a first threshold, select a first number of test cases from the test cases corresponding to each of the target execution paths for genetic operation to obtain a first population.

[0045] It can be understood that the first fitness value is used to indicate the degree of match between the target execution path and the corresponding test case. If there is an unmatched path in the target execution path whose first fitness value is less than the first threshold, the test case corresponding to the unmatched path needs to be regenerated. Among them, selecting a first number of test cases from the test cases corresponding to each of the target execution paths for genetic operation to obtain a first population may include: sorting the first fitness values ​​corresponding to each of the test cases in descending order; obtaining the first number of test cases corresponding to the first fitness value for genetic operation to obtain the first population. Among them, genetic operations may include crossover, mutation, etc.

[0046] Step 3: Based on the first population, use a genetic algorithm to re-determine the test case corresponding to the unmatched path.

[0047] It can be understood that after obtaining the first population, the test cases corresponding to the unmatched paths are re-determined using a genetic algorithm until the first fitness value of each target execution path and the corresponding test case is greater than a first threshold.

[0048] In this embodiment, since each target execution path generally represents a specific business process, using the most matching test case on each path can enhance the precision and accuracy of the test.

[0049] In an exemplary embodiment, generating the test case based on the program semantics includes: obtaining at least one program state of the target application; and generating the test case corresponding to each program state according to the program semantics and preset constraints.

[0050] It is understandable that the program state refers to the various changes and storage states of data during the operation of the target application. Exemplarily, the program state may include memory state, cache state, transaction state, network state, page state, security state, etc. Among them, the preset constraint condition is used to ensure the validity of the test case, that is, the test case must meet the preset constraint condition required by the program semantics, otherwise the test case may not pass the program verification correctly.

[0051] In this embodiment, by combining program semantics and preset constraints, it can be ensured that the test cases cover different states of the program and the compliance of the test cases.

[0052] Among them, in another exemplary embodiment, after generating the test case corresponding to each of the program states, the method may further include the following steps: Step 1: Using a second fitness function, obtain a second fitness value of an actual path corresponding to each of the program states and the corresponding test case.

[0053] The second fitness value is used to indicate the degree of matching between the program state and the actual state of the corresponding test case.

[0054] Step 2: When there is an unmatched program state in the program state whose second fitness value is less than a second threshold, select a second number of the test cases from the test cases corresponding to each of the program states for genetic operation to obtain a second population.

[0055] It is understandable that the second fitness value is used to indicate the degree of match between the program state and the corresponding test case. If there is an unmatched program state in the program state whose second fitness value is less than the second threshold, it is necessary to regenerate the test case corresponding to the unmatched program state. Among them, selecting a second number of test cases from the test cases corresponding to each of the program states for genetic operation to obtain a second population may include: sorting the second fitness values ​​corresponding to each of the test cases in descending order; obtaining the test cases corresponding to the second fitness values ​​of the first second number for genetic operation to obtain the second population. Among them, genetic operations may include crossover, mutation, etc.

[0056] Step 3: Based on the second population, use a genetic algorithm to redetermine the test case corresponding to the unmatched program state.

[0057] It can be understood that after obtaining the second population, the test cases corresponding to the unmatched program states are re-determined using a genetic algorithm until the second fitness value of each program state and the corresponding test case is greater than a second threshold.

[0058] In this embodiment, since each program state generally represents a specific business state, using the most matching test case on each program state can enhance the precision and accuracy of the test.

[0059] Optionally, the test case may also be determined by at least one of the following methods: (1) Design a parser for a specific network protocol to parse and generate data packets that comply with the protocol specification to simulate the network communication process. Generate test cases during the simulated network communication process. The generated test cases can cover different communication processes.

[0060] (2) Generate diverse test cases based on protocol specifications and communication scenarios. The generated test cases can cover different protocol states and abnormal situations.

[0061] (3) By analyzing the specifications and structure of the target file format, the data domains and boundary conditions are identified, and corresponding test cases are generated based on the data domains and boundary conditions.

[0062] In an exemplary embodiment, before inputting the feature vector of the performance indicator parameters obtained within the preset time period into the performance evaluation model, the method further includes: obtaining a first feature by performing time difference and wavelet transformation processing on the performance indicator parameters obtained within the preset time period; and obtaining the feature vector by performing feature conversion on the first feature.

[0063] Among them, the time series difference is used to capture the trend and change of the data. For example, its formula can be: ; in, is the original data at time t, It is the data after time series difference.

[0064] Wavelet transform is used to decompose the signal into components of different frequencies and extract features at different scales. For example, the formula of its discrete form can be: ; in, is the time-series differenced data, Ψ(t) is the wavelet basis function, a and b are the scale and translation parameters, respectively. Indicates the first feature.

[0065] After obtaining the first feature, the first feature is subjected to feature conversion to obtain a feature vector, wherein the feature conversion can be implemented by using the Word2vec model based on the Embedding method. The Word2vec model includes the CBOW model and the Skip-Gram model, which are neural language models used to generate word vectors. Through training, words or phrases can be converted into numerical vectors with semantic information. The vectors can capture the contextual relationship and similarity between words. In the hardware performance fluctuation detection scenario, the Word2vec model can help convert and refine feature data, that is, convert the original data into a more compact representation, which is used to improve the expressiveness and generalization ability of the performance evaluation model. The formula for feature conversion can be: ; Where W represents the first feature, It represents the representation obtained after the Embedding technology transforms the first feature, which is a vector representation of a fixed length.

[0066] In addition, since the measured data is a finite time series, there will be a "boundary effect" at both ends of the time series. In order to eliminate or reduce this effect, it is necessary to extend both ends of the data. At the same time, after the wavelet transform is completed, the extended part of the data needs to be removed. It is understandable that since the boundary effect of the time series usually refers to the result deviation caused by sample limitation when processing time series data, especially when analyzing or predicting the data points at the beginning and end. This is because the time series usually has a start time and an end time, and the observations close to these two boundaries may be fewer, or more affected by historical data, and lack sufficient historical context information. Therefore, the boundary effect can be alleviated by extending the data, that is, filling in the missing values.

[0067] In an exemplary embodiment, before inputting the characteristic vector of the performance indicator parameter acquired within the preset time period into the performance evaluation model, the method may further include the following steps: Step 1: Construct the performance evaluation model.

[0068] Among them, constructing the performance evaluation model may include: constructing the performance evaluation model based on a convolutional neural network sub-model, a long short-term memory network sub-model and a converter sub-model.

[0069] Step 2: Training the performance evaluation model based on a training set, wherein the training set includes performance indicator data of the target device within a preset time period.

[0070] Among them, in another exemplary embodiment, the training of the performance evaluation model based on the training set includes: initializing the model parameters of the convolutional neural network sub-model, the long short-term memory network sub-model and the converter sub-model; during the training process, using the target algorithm to update the model parameters of the convolutional neural network sub-model, the long short-term memory network sub-model and the converter sub-model until each sub-model converges and stops training.

[0071] It can be understood that the convolutional neural network sub-model, the long short-term memory network sub-model and the converter sub-model have independent model parameters respectively. During the training process, the three sub-models need to be trained simultaneously until a sub-model converges, then the training of the sub-model stops. If the other sub-models have not converged, the training continues until each sub-model converges to obtain a trained performance evaluation model.

[0072] The target algorithm may be a genetic algorithm or a Bayesian optimization algorithm. The genetic algorithm randomly generates an initial population, which includes multiple sets of model parameters, and selects, crosses, and mutates through multiple iterations until the optimal model parameters are found. The Bayesian optimization algorithm defines the objective function, uses the Gaussian process as a priori, selects the next hyperparameter combination to be evaluated through the acquisition function, and continuously updates the posterior distribution until convergence. Exemplarily, the Bayesian optimization algorithm can be expressed by the following formula: ; in, is the next hyperparameter combination to be evaluated, represents the expected improvement, the goal is to maximize function.

[0073] In an exemplary embodiment, after training the performance evaluation model based on the training set, the method further includes at least one of the following: (1) Based on the validation set, cross-validate the performance evaluation model.

[0074] Among them, cross-validation of the performance evaluation model refers to evaluating the stability and generalization ability of the model, and performing model selection and adjustment. Exemplarily, the evaluation indicators include but are not limited to accuracy, precision and recall.

[0075] (2) Using the stochastic gradient descent algorithm to update the performance evaluation model.

[0076] It can be understood that in order to enable the performance evaluation model to continuously adapt to new data and environmental changes over time, the performance evaluation model can be designed to be automatically updated. In the embodiment of the present application, the performance evaluation model can be updated using a stochastic gradient descent algorithm, wherein the update rule using the stochastic gradient descent algorithm can be expressed as: ;in, is the parameter at time t, is the learning rate, is the loss function.

[0077] In an exemplary embodiment, the step of inputting the characteristic vector of the performance indicator parameter acquired within the preset time period into a performance evaluation model to obtain the performance status of the target device within the preset time period may include the following steps: Step 1: Extract features from the feature vector using the convolutional neural network sub-model to obtain a second feature.

[0078] It can be understood that the second feature is obtained by taking the local feature through the convolutional neural network sub-model. Exemplarily, its formula can be: ; in, represents the i-th feature map output by the convolutional neural network sub-model, represents the weight of the i-th feature map, is the input data, i.e., the feature vector, is the bias term, and f is the activation function.

[0079] Step 2: Perform time series modeling based on the second feature through the long short-term memory network sub-model to obtain a time series state corresponding to the second feature.

[0080] It can be understood that the time series correlation is captured by the long short-term memory network sub-model to obtain the time series state corresponding to the second feature. Exemplarily, its formula may include: ; ; ; ; ; in, Represents the second feature , Indicates the timing status, Indicates the cell state, , and is the gated state of the forget gate, input gate, and output gate, W and U are weight matrices, b is the bias term, is the sigmoid function, and is the tanh function, and ⊙ represents element-wise multiplication.

[0081] Step 3: Generate a context feature vector based on the timing state through the converter sub-model.

[0082] Among them, when generating the context feature vector based on the time series state through the transformer sub-model, an attention mechanism can be introduced, and the transformer uses a self-attention mechanism to capture the global dependencies in the sequence. Exemplarily, its formula can be expressed as: ; Where Q, K, and V represent the matrices of query, key, and value, respectively. is the dimension of the key, and the softmax function is used to calculate the attention weights.

[0083] Step 4: Fusing the second feature, the temporal state and the context feature vector to obtain a fused feature.

[0084] Step 5: Obtain the performance status of the target device within the preset time period by weighted processing of the fusion features.

[0085] It can be understood that the outputs of different models are fused to obtain fused features, wherein the fusion method may include concatenation, weighted summation or multi-layer perceptron. After the feature fusion is obtained, the fused feature is weighted to obtain the performance status of the target device within the preset time period. For example, assuming that the feature value obtained by weighted processing of the fused feature is 0.7, the performance status corresponding to 0.7 can be "sharp rise".

[0086] In another exemplary embodiment, after obtaining the performance status of the target device within the preset time period, the method further includes: triggering a corresponding warning level according to the performance status; and starting a performance balancing strategy based on the warning level, wherein the performance balancing strategy may include a load balancing strategy.

[0087] In an exemplary embodiment, after obtaining the performance indicator parameters of the target device to which the target application belongs according to a preset time period, the method further includes: using an anomaly detection algorithm to mark abnormal values ​​in the performance indicator parameters; and identifying abnormal behavior of the target device based on the abnormal values.

[0088] It is understandable that by detecting abnormal values ​​in performance indicator parameters, potential problems in the hardware or software level of the target device can be quickly discovered. The potential problem may be caused by erroneous code, improper configuration or hardware failure, which may easily lead to instability or complete crash of the control program of the target device. In addition, some abnormal behaviors may be the result of malicious activities, such as denial of service attacks (DoS) may cause a sudden surge in CPU usage. Detecting anomalies helps to take preventive measures and enhance the security of the target device, thereby ensuring the stability of the test process.

[0089] In another exemplary embodiment, the anomaly detection algorithm can be implemented based on the mean and standard deviation, that is, by calculating the mean and standard deviation of each feature, and then marking the value exceeding the specified threshold as an outlier, and its formula can be expressed as: ; where x is the original data, μ is the mean, σ is the standard deviation, and z is the standardized value. Optionally, if If it is greater than μ+3σ, the data point is an outlier.

[0090] In an exemplary embodiment, in the process of performing chaos testing on the target application based on the test case using the chaos testing tool, the method further includes: monitoring the running status of the target application and recording abnormal behavior of the target application during the chaos testing process.

[0091] It is understandable that during the test process, by monitoring the running status of the target application and recording the abnormal behavior of the target application during the chaos test process, it is helpful to take preventive measures in time when anomalies are detected, thereby ensuring the stability of the test process.

[0092] Figure 2 Another flow chart of the test method provided in the embodiment of the present application is shown, which may include the following steps: S210: Determine the test objectives and test scope.

[0093] That is, determine the hardware devices that need to be tested, and then define the test scope, which can include the following three parts: (1) Normal operation: Ensure that the system can run stably under normal circumstances.

[0094] (2) Boundary conditions: verify the performance of the system under extreme conditions.

[0095] (3) Illegal input: Test the system's ability to handle erroneous or malicious input.

[0096] S220: Select a chaos testing tool based on the characteristics of the hardware device.

[0097] S230: Design test cases.

[0098] Among them, the test cases include normal operation, boundary conditions and illegal input conditions.

[0099] S240: Configure the test environment.

[0100] That is to say, build a test environment that includes the hardware device to be tested and ensure that the test tool can communicate with the hardware device correctly.

[0101] S250: Introduced hardware performance fluctuation detection.

[0102] By collecting diverse performance indicator data of hardware devices and using statistical methods for anomaly detection, potential performance anomalies can be identified. Advanced feature engineering methods such as time series difference and wavelet transform are introduced, and embedding technology is used to convert raw data into a more compact representation to improve the expressiveness of the model. In terms of deep learning model design, an integrated learning method is adopted, combining multiple deep learning models such as CNN, LSTM and Transformer, and an attention mechanism is introduced to automatically focus on the most important features. Automated hyperparameter optimization technology and cross-validation are used for model training and evaluation, and to help optimize model performance. Finally, a model online learning and multi-stage early warning mechanism are designed to achieve real-time performance fluctuation detection and early warning, so as to better cope with the dynamic changes in hardware device performance.

[0103] S260: Execute chaos test.

[0104] That is, using chaos testing tools to perform chaos testing on hardware devices according to designed test cases. During the test process, the performance indicators of the hardware devices are monitored in real time, and the data is stored for subsequent analysis.

[0105] S270: Analyze the performance status of the hardware device in the chaos test according to the collected performance data.

[0106] Figure 3 A schematic diagram of the structure of a test device provided in an embodiment of this specification is shown. Figure 3 As shown, the test device 300 may include: a first acquisition module 310, a second acquisition module 320 and a processing module 330. In this embodiment, the first acquisition module 310 is used to acquire a test case corresponding to the target application; the second acquisition module 320 is used to acquire the performance indicator parameters of the target device to which the target application belongs according to a preset time period in the process of performing a chaos test on the target application based on the test case using a chaos test tool; the processing module 330 is used to input the characteristic vector of the performance indicator parameters acquired within the preset time period into a performance evaluation model to obtain the performance status of the target device within the preset time period, and the performance evaluation model includes a convolutional neural network sub-model, a long short-term memory network sub-model and a converter sub-model.

[0107] In an exemplary embodiment, the first acquisition module 310 is specifically used to: acquire the program semantics or control flow of the target application; generate the test case based on the program semantics; or, generate the test case based on the control flow; wherein the test case includes: normal input and abnormal input.

[0108] In an exemplary embodiment, the first acquisition module 310 is specifically used to: determine the program semantics and control flow of the target application by statically analyzing the source code of the target application; or determine the program semantics and control flow of the target application by disassembling and reverse engineering the binary file of the target application.

[0109] In an exemplary embodiment, the first acquisition module 310 is specifically used to: utilize a symbolic execution engine to execute the control process in a symbolic manner and search for multiple candidate execution paths; screen the multiple candidate execution paths based on a heuristic algorithm to obtain a target execution path; and generate a test case corresponding to each target execution path by solving constraints for each target execution path.

[0110] In an exemplary embodiment, the testing device 300 may also include a first determination module, which is used to: use a first fitness function to obtain a first fitness value of an actual path corresponding to each of the target execution paths and the corresponding test cases; when there is an unmatched path in the target execution path whose first fitness value is less than a first threshold, select a first number of the test cases from the test cases corresponding to each of the target execution paths for genetic operation to obtain a first population; based on the first population, use a genetic algorithm to re-determine the test cases corresponding to the unmatched paths.

[0111] In an exemplary embodiment, the first acquisition module 310 is specifically used to: acquire at least one program state of the target application; and generate the test case corresponding to each program state according to the program semantics and preset constraints.

[0112] In an exemplary embodiment, the testing device 300 may also include a first determination module, which is used to: use a second fitness function to obtain a second fitness value of an actual path corresponding to each of the program states and the corresponding test cases; when there is an unmatched program state in the program state whose second fitness value is less than a second threshold, select a second number of the test cases from the test cases corresponding to each of the program states for genetic operation to obtain a second population; based on the second population, use a genetic algorithm to re-determine the test cases corresponding to the unmatched program states.

[0113] In an exemplary embodiment, the testing device 300 may also include a feature conversion module, which is used to: obtain a first feature by performing time difference and wavelet transformation processing on the performance indicator parameters obtained within the preset time period; and obtain the feature vector by performing feature conversion on the first feature.

[0114] In an exemplary embodiment, the testing device 300 may further include a training module for: constructing the performance evaluation model; and training the performance evaluation model based on a training set, wherein the training set includes performance indicator data of the target device within a preset time period.

[0115] In an exemplary embodiment, the training module is specifically used to: initialize the model parameters of the convolutional neural network sub-model, the long short-term memory network sub-model and the converter sub-model; during the training process, use the target algorithm to update the model parameters of the convolutional neural network sub-model, the long short-term memory network sub-model and the converter sub-model until each sub-model converges and stops training.

[0116] In an exemplary embodiment, the processing module 330 is specifically used to: extract features from the feature vector through the convolutional neural network sub-model to obtain a second feature; perform timing modeling based on the second feature through the long short-term memory network sub-model to obtain a timing state corresponding to the second feature; generate a context feature vector based on the timing state through the converter sub-model; perform feature fusion on the second feature, the timing state and the context feature vector to obtain a fused feature; and obtain the performance state of the target device within the preset time period by weighted processing of the fused feature.

[0117] In an exemplary embodiment, the testing device 300 may further include: a second processing module, used to perform cross-validation on the performance evaluation model based on a validation set; and a third processing module, used to update the performance evaluation model using a stochastic gradient descent algorithm.

[0118] In an exemplary embodiment, the testing device 300 may further include a detection module, which is used to: mark abnormal values ​​in the performance indicator parameters using an abnormality detection algorithm; and identify abnormal behavior of the target device based on the abnormal values.

[0119] In an exemplary embodiment, the testing device 300 may further include a monitoring module, which is used to monitor the running status of the target application and record abnormal behaviors of the target application during the chaos testing process.

[0120] A testing device provided in an embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented in the illustrated method embodiment will not be described again here.

[0121] A test device in the embodiment of the present application may be a device, or a component, an integrated circuit, or a chip in an electronic device. The embodiment of the present application is not specifically limited.

[0122] A test device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0123] Optional, such as Figure 4 As shown, an embodiment of the present application also provides an electronic device 400, including a processor 410, a memory 420, and a program or instruction stored in the memory 420 and executable on the processor 410. When the program or instruction is executed by the processor 410, each process of the above-mentioned test method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0124] An embodiment of the present application also provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned test method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0125] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0126] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned test method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0127] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0128] The embodiment of the present application also provides a computer program product, which includes at least one computer program. When the computer program is loaded and executed by a processor, it implements the various processes of the above-mentioned test method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0129] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0130] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM / RAM, disk, CD), including several instructions to enable a terminal (which can be a mobile phone, computer, server, air conditioner, or network equipment, etc.) to execute the methods described in each embodiment of the present application. The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods, which are merely illustrative and not restrictive. Under the enlightenment of the present application, those skilled in the art can also make many forms without departing from the scope of protection of the purpose of the present application and the claims, all of which belong to the protection of the present application.

Claims

1. A testing method, characterized in that: include: Get the test cases corresponding to the target application; In the process of performing chaos testing on the target application based on the test case using the chaos testing tool, obtaining performance indicator parameters of the target device to which the target application belongs according to a preset time period; The characteristic vector of the performance indicator parameter obtained within the preset time period is input into a performance evaluation model to obtain the performance status of the target device within the preset time period, wherein the performance evaluation model includes a convolutional neural network sub-model, a long short-term memory network sub-model and a converter sub-model.

2. The method according to claim 1, characterized in that The obtaining of the test case corresponding to the target application includes: Obtaining program semantics or control flow of the target application; Based on the program semantics, generating the test case; or, Based on the control flow, generating the test case; The test cases include normal input and abnormal input.

3. The method according to claim 2, characterized in that The obtaining of program semantics or control flow of the target application includes: Determine the program semantics and control flow of the target application by statically analyzing the original code of the target application; or, The program semantics and control flow of the target application are determined by disassembling and reverse engineering the binary file of the target application.

4. The method according to claim 2, characterized in that: The generating the test case based on the control flow includes: Using a symbolic execution engine, executing the control flow in a symbolic manner, and searching for multiple candidate execution paths; Screening the plurality of candidate execution paths based on a heuristic algorithm to obtain a target execution path; By solving constraints on each of the target execution paths, a test case corresponding to each of the target execution paths is generated.

5. The method according to claim 4, characterized in that After generating the test case corresponding to each target execution path, the method further includes: Using a first fitness function, obtaining a first fitness value of an actual path corresponding to each of the target execution paths and the corresponding test case; In the case where there is an unmatched path whose first fitness value is less than a first threshold in the target execution path, selecting a first number of test cases from the test cases corresponding to each of the target execution paths for genetic operation to obtain a first population; Based on the first population, a genetic algorithm is used to redetermine the test case corresponding to the unmatched path.

6. The method according to claim 2, characterized in that The generating the test case based on the program semantics includes: Acquire at least one program state of the target application; The test case corresponding to each program state is generated according to the program semantics and preset constraints.

7. The method according to claim 6, characterized in that After generating the test case corresponding to each of the program states, the method further includes: using a second fitness function to obtain a second fitness value of an actual path corresponding to each of the program states and the corresponding test case; In the case where there is an unmatched program state whose second fitness value is less than a second threshold value in the program state, selecting a second number of the test cases from the test cases corresponding to each of the program states for genetic operation to obtain a second population; Based on the second population, a genetic algorithm is used to redetermine the test case corresponding to the unmatched program state.

8. The method according to claim 1, characterized in that Before inputting the characteristic vector of the performance indicator parameter acquired within the preset time period into the performance evaluation model, the method further includes: Obtaining a first feature by performing time series difference and wavelet transformation processing on the performance indicator parameters obtained within the preset time period; The feature vector is obtained by performing feature transformation on the first feature.

9. The method according to claim 1 or 8, characterized in that: The step of inputting the characteristic vector of the performance indicator parameter acquired within the preset time period into a performance evaluation model to obtain the performance status of the target device within the preset time period includes: Performing feature extraction on the feature vector by using the convolutional neural network sub-model to obtain a second feature; Performing time series modeling based on the second feature by using the long short-term memory network sub-model to obtain a time series state corresponding to the second feature; generating a context feature vector based on the timing state by the converter sub-model; Performing feature fusion on the second feature, the temporal state and the context feature vector to obtain a fused feature; The performance status of the target device within the preset time period is obtained by performing weighted processing on the fusion features.

10. The method according to claim 1, characterized in that Before inputting the characteristic vector of the performance indicator parameter acquired within the preset time period into the performance evaluation model, the method further includes: constructing the performance evaluation model; The performance evaluation model is trained based on a training set, wherein the training set includes performance indicator data of the target device within a preset time period.

11. The method according to claim 10, characterized in that The step of training the performance evaluation model based on the training set includes: Initializing model parameters of the convolutional neural network sub-model, the long short-term memory network sub-model, and the converter sub-model; During the training process, the target algorithm is used to update the model parameters of the convolutional neural network sub-model, the long short-term memory network sub-model and the converter sub-model until each sub-model converges and the training is stopped.

12. The method according to claim 10, characterized in that After training the performance evaluation model based on the training set, the method further includes at least one of the following: Based on the validation set, cross-validating the performance evaluation model; The performance evaluation model is updated using a stochastic gradient descent algorithm.

13. The method according to claim 1, characterized in that After obtaining the performance indicator parameters of the target device to which the target application belongs according to the preset time period, the method further includes: Using an anomaly detection algorithm, marking outliers in the performance indicator parameters; Based on the abnormal value, abnormal behavior of the target device is identified.

14. The method according to claim 1, characterized in that In the process of performing chaos testing on the target application based on the test case using the chaos testing tool, the method further includes: Monitor the running status of the target application and record abnormal behaviors of the target application during the chaos test.

15. A testing device, characterized in that: include: A first acquisition module, used to acquire test cases corresponding to the target application; A second acquisition module is used to acquire the performance indicator parameters of the target device to which the target application belongs according to a preset time period during the process of performing chaos testing on the target application based on the test case using a chaos testing tool; A processing module is used to input the characteristic vector of the performance indicator parameter obtained within the preset time period into a performance evaluation model to obtain the performance status of the target device within the preset time period, and the performance evaluation model includes a convolutional neural network sub-model, a long short-term memory network sub-model and a converter sub-model.

16. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the test method according to any one of claims 1 to 14.

17. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the testing method according to any one of claims 1 to 14 are implemented.

18. A computer program product, characterized in that The computer program product comprises program instructions, and when the program instructions are executed by a computer, the computer is enabled to implement the steps of the testing method according to any one of claims 1 to 14.

Images