Automatic testing method and device of application, equipment, storage medium and program product

Abstract

The invention relates to an automatic testing method and device of an application, equipment, a storage medium and a program product. The method comprises the following steps: applying preset input excitation to a tested protection measurement and control device through a test execution engine, and collecting multi-source data in a test process; preprocessing the collected multi-source data to obtain time sequence data in a unified form; performing various analyses on the time series data to obtain an analysis result; based on the analysis result, a test report is automatically generated, and the test report comprises at least one of statistical data of the quantitative performance indexes, function passing rate data and a compatibility matrix. Therefore, comprehensive automation and data driving in the testing process are achieved, the functions, performance and compatibility problems applied to different device models, firmware versions and environment configurations can be found accurately and efficiently, the testing period is greatly shortened, and the product quality is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, in particular to an automatic testing method and device for an application, a computer device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] With the development of computer technology, device software architecture tends to be open and application (APP) based. Function iteration is achieved by installing and upgrading functional APPs on different hardware platforms or different versions of devices, which has become an industry trend. However, this mode puts forward very high requirements for the compatibility testing of APPs.

[0003] In the traditional technology, the testing method of an application mainly relies on manual operation of a simulation device by a tester, application of excitation, observation of device behavior and recording of results. However, this method has significant limitations: first, the testing process is highly dependent on manual operation, which is low in efficiency, time-consuming and seriously slows down the development progress, and cannot meet the needs of rapid iteration; second, the testing coverage is limited, and it is difficult to simulate a large number of working conditions and different hardware environment combinations; third, the testing results are subjective, lack of objective and quantitative data support, and the positioning ability for performance bottlenecks and occasional compatibility problems is insufficient; finally, there is a lack of automatic association between test cases, environments and results, making it difficult to accurately reproduce and locate the root cause of the problem.

[0004] Therefore, there is an urgent need for a testing method that can efficiently, comprehensively and deeply verify the performance of a general-purpose protection and control device APP. SUMMARY

[0005] Therefore, there is an urgent need for a testing method that can efficiently, comprehensively and deeply verify the performance of a general-purpose protection and control device APP.

[0006] In a first aspect, the present application provides an automatic testing method for an application, which comprises:

[0007] applying a preset input excitation to a protection and control device under test through a test execution engine, and collecting multi-source data during the testing process;

[0008] preprocessing the collected multi-source data to obtain time series data in a unified form;

[0009] performing various analyses on the time series data to obtain analysis results;

[0010] generating a test report automatically based on the analysis results, wherein the test report includes at least one of statistical data of quantitative performance indicators, function pass rate data and a compatibility matrix.

[0011] In one of the embodiments, the test execution engine applies preset input excitation to the tested protection and control device, and collects multi-source data during the test process, including:

[0012] The application simulator in the test execution engine generates the preset input excitation, which includes analog signals and / or control commands; the analog signals include analog quantities and state quantities.

[0013] In the case where the tested protection and control device performs the test process according to the preset input excitation, the multi-source data are collected by the preset probes and / or listeners; the multi-source data include at least one of the static data of the tested protection and control device, output response data, system resource data and test environment data.

[0014] In one of the embodiments, the static data of the tested protection and control device includes at least one of the device model, hardware version, firmware version, software application version, communication protocol type and configuration parameter template.

[0015] The output response data include at least one of the response message generated by the tested protection and control device, action signal, actual value and timestamp of the fault report.

[0016] The system resource data include at least one of the CPU occupancy, memory usage, network throughput and thread state.

[0017] The test environment data include at least one of the operating system image version, test tool version, network configuration and simulator model.

[0018] In one of the embodiments, the collected multi-source data are preprocessed to obtain time series data in a unified form, including:

[0019] According to a predefined protocol specification, the binary message in the multi-source data is parsed, and the parsed result is subjected to keyword extraction to obtain key fields.

[0020] Taking the global clock source as a reference, the timestamps of all data are uniformly aligned to obtain time series data in a unified time coordinate system.

[0021] The time series data are subjected to cleaning processing to obtain target time series data.

[0022] In one of the embodiments, the time series data are subjected to various analyses to obtain analysis results, including at least one of the following steps:

[0023] performing performance index analysis on the target time-series data to obtain quantitative performance indexes, the quantitative performance indexes including at least one of response time, precision error and resource consumption;

[0024] performing functional logic analysis on the target time-series data to determine whether each function is correct, the functional logic analysis including matching the standardized output response data with an expected behavior pattern in a preset rule engine to determine whether each function is correctly implemented;

[0025] performing compatibility analysis on the target time-series data to determine whether there is a compatibility problem, the compatibility analysis including horizontally comparing test results of the same application under different device models, different firmware versions and different environment configurations.

[0026] In one of the embodiments, after automatically generating the test report based on the analysis result, the method further includes at least one of the following:

[0027] displaying the test report by using a visual chart;

[0028] in response to a query operation on an abnormal result in the test report, displaying at least one of corresponding raw messages, fault recording information and log information;

[0029] after converting the test report into a preset format, automatically pushing the test report to a target platform or a project management tool.

[0030] In a second aspect, the application further provides an automatic testing device of an application, the testing device including:

[0031] a test execution engine configured to apply a preset input excitation to a tested protection and control device;

[0032] a multi-source data collection module configured to collect multi-source data during a test;

[0033] a data processing module configured to pre-process the collected multi-source data to obtain time-series data in a unified form;

[0034] a data analysis module configured to perform various analyses on the time-series data to obtain analysis results;

[0035] a test report generation module configured to automatically generate a test report based on the analysis results, the test report including at least one of statistical data of quantitative performance indexes, functional pass rate data and a compatibility matrix.

[0036] In a third aspect, the application further provides a computer device including a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0037] applying preset input excitation to the measured protection and control device through a test execution engine, and collecting multi-source data during the test process;

[0038] preprocessing the collected multi-source data to obtain time series data in a unified form;

[0039] performing various analyses on the time series data to obtain analysis results;

[0040] generating a test report automatically based on the analysis results, the test report including at least one of statistical data of quantitative performance indicators, function pass rate data and a compatibility matrix.

[0041] In a fourth aspect, the present application further provides a computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the following steps:

[0042] applying preset input excitation to the measured protection and control device through a test execution engine, and collecting multi-source data during the test process;

[0043] preprocessing the collected multi-source data to obtain time series data in a unified form;

[0044] performing various analyses on the time series data to obtain analysis results;

[0045] generating a test report automatically based on the analysis results, the test report including at least one of statistical data of quantitative performance indicators, function pass rate data and a compatibility matrix.

[0046] In a fifth aspect, the present application further provides a computer program product comprising a computer program, the computer program being executed by a processor to implement the following steps:

[0047] applying preset input excitation to the measured protection and control device through a test execution engine, and collecting multi-source data during the test process;

[0048] preprocessing the collected multi-source data to obtain time series data in a unified form;

[0049] performing various analyses on the time series data to obtain analysis results;

[0050] generating a test report automatically based on the analysis results, the test report including at least one of statistical data of quantitative performance indicators, function pass rate data and a compatibility matrix.

[0051] The aforementioned automated testing methods, apparatus, computer equipment, computer-readable storage media, and computer program products apply preset input stimuli to the protected control device under test (DDT) through a test execution engine and collect multi-source data during the testing process. This allows the test execution engine to simulate various input signals and control commands, enabling the DDT to complete various functional tests. The collected multi-source data is preprocessed to obtain time-series data in a unified format, aligning the time of the collected data for subsequent analysis. Various analyses are performed on the time-series data to obtain analysis results. Based on these results, a test report is automatically generated, including at least one of the following: statistical data on quantitative performance indicators, functional pass rate data, and a compatibility matrix. This achieves full automation and data-driven testing, accurately and efficiently identifying functional, performance, and compatibility issues in applications with different device models, firmware versions, and environmental configurations, significantly shortening the testing cycle and improving product quality. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a diagram illustrating the application environment of an automated testing method used in one embodiment.

[0054] Figure 2 This is a flowchart illustrating an automated testing method applied in one embodiment;

[0055] Figure 3 This is a flowchart illustrating the automated testing method applied in another embodiment;

[0056] Figure 4 A schematic diagram of the data flow for an automated testing method provided in an embodiment of this application;

[0057] Figure 5 This is a structural block diagram of an automated testing device used in one embodiment;

[0058] Figure 6 This is a structural block diagram of the automated testing apparatus used in another embodiment;

[0059] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0060] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0061] It should be noted that the terms "comprising" and "having" and any variations thereof used in the present application are intended to cover non-exclusive inclusion. The term "a plurality of" used in the present application refers to two or more. The term "and / or" used in the present application refers to one of the options or any combination of the options.

[0062] In the secondary field of the power system, the protection and measurement device is the core equipment to ensure the safe and stable operation of the power grid. In the prior art, for the protection and measurement device, mainly relies on the manual operation of the simulation device by the test personnel, applies the excitation, observes the device behavior and records the results. This method has significant limitations: first, the test process is highly dependent on manual operation, low in efficiency, time-consuming and long, seriously slows down the research and development progress, and cannot meet the needs of rapid iteration; second, the test coverage is limited, it is difficult to simulate a large number of working conditions and different hardware environment combinations; third, the test results are subjective, lack of objective and quantitative data support, and the positioning ability for performance bottlenecks and occasional compatibility problems is insufficient; finally, there is a lack of automatic association between test cases, environment and results, making it difficult to accurately reproduce and locate the problem root cause.

[0063] In view of the problems in the prior art, the application embodiment proposes an automatic test method of application, aiming to solve the technical pain point of long traditional research and development test cycle. By constructing an automatic data flow pipeline, by simulating the real running environment and working condition of the protection and measurement device, automatically executing the test case, and collecting, processing and analyzing the massive data generated in the test process; finally, a quantifiable test report is automatically generated. It can accurately locate the compatibility, function and performance problems, so as to change the traditional manual, serial long-period test mode into an automatic, parallel high-efficiency mode, greatly shortening the research and development test cycle. The overall automation and data-driven of the test process are realized, which can accurately and efficiently find the function, performance and compatibility problems of the APP under different device models, firmware versions and environment configurations, greatly shorten the test cycle and improve the product quality.

[0064] The automatic test method of application provided by the application embodiment can be applied to, for example Figure 1The application environment shown. Among them, the measured protection control device 102 communicates with the server 104 through the network. The data storage system can store the data required by the server 104 to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The test execution engine is pre-installed in the server 104, which applies the preset input excitation to the measured protection control device through the test execution engine, and collects multi-source data during the test process; pre-process the collected multi-source data to obtain time series data in a unified form; analyze the time series data to obtain analysis results; based on the analysis results, automatically generate a test report, which includes at least one of the statistical data of the quantitative performance indicators, the function pass rate data and the compatibility matrix. The server 104 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0065] In an exemplary embodiment, as shown in Figure 2 An automated testing method for an application is provided, which is applied to Figure 1 The server in the method is taken as an example for illustration, which includes the following steps S201 to S204. Among them:

[0066] Step S201, applying a preset input excitation to the measured protection control device through the test execution engine, and collecting multi-source data during the test process.

[0067] In this embodiment, the test execution engine is a key component in the field of software testing, mainly responsible for driving the execution of test cases, managing test processes and outputting results.

[0068] For example, the preset input excitation is generated by the application simulator in the test execution engine, which includes: analog signals and / or control commands; analog signals include: analog quantities and state quantities; under the condition that the measured protection control device executes the test process according to the preset input excitation, multi-source data is collected through pre-set probes and / or listeners; wherein the multi-source data includes at least one of the static data of the measured protection control device, the output response data, the system resource data and the test environment data.

[0069] In this embodiment, the analog quantity includes: current and / or voltage waveform; the state quantity includes the on-off state of the switch. The control command can include: theoretical value and sending timestamp T1. The theoretical value can be modified or remotely controlled as needed.

[0070] Optionally, the static data of the measured protection control device includes at least one of the device model, hardware version, firmware version, software application version, communication protocol type and configuration parameter template.

[0071] In this embodiment, the static data of the measured protection and control device is automatically read and entered into the database by the test platform (the applied automatic test device) before testing.

[0072] Optionally, the output response data includes at least one of the response message (such as GOOSE, SV, MMS message) generated by the measured protection and control device, the action signal (such as the change of the relay outlet and the indicator light), the actual value of the fault report and the time stamp T2.

[0073] Optionally, the system resource data includes at least one of the CPU occupancy rate, the memory usage, the network throughput and the thread state.

[0074] In this embodiment, the system resource data is mainly obtained by the system monitoring agent (Agent) built in the device or by the debugging interface sampling.

[0075] Optionally, the test environment data includes at least one of the operating system image version, the test tool version, the network configuration and the simulator model.

[0076] In this embodiment, the test environment data is mainly determined by the environment setting of the test platform (the applied automatic test device).

[0077] Step S202, pre-process the collected multi-source data to obtain time series data in a unified form.

[0078] In this embodiment, since the original formats of the multi-source data are different, such as binary message, analog waveform, system log and the like, in order to facilitate subsequent analysis, it is necessary to convert these multi-source data into unified and regular time series data. The time series data refers to sequence data arranged in time order.

[0079] For example, according to the pre-defined protocol specification (such as IEC 61850 SCL file), the binary message in the multi-source data is parsed, and the parsed results are keyword extracted to obtain key fields (such as protection start, protection action and fault quantity); taking the global clock source (such as PTP clock) as the reference, the time stamps (such as T1, T2) of all data are uniformly aligned to obtain time series data in a unified time coordinate system; the time series data is cleaned to obtain target time series data.

[0080] Among them, the cleaning of the time series data includes filtering out invalid data, abnormal burrs and redundant information and the like.

[0081] Step S203, various analyses are performed on the time series data to obtain analysis results.

[0082] Exemplarily, the performance indicator analysis is performed on the target time-series data to obtain a quantitative performance indicator, and the quantitative performance indicator includes at least one of a response time, an accuracy error, and a resource consumption.

[0083] In this embodiment, the response time can be determined by the following calculation formula:

[0084] T_response = T2_action - T1_stimulus;

[0085] wherein T_response is the response time, T2_action is the device action timestamp obtained from the output response data (source: output response data), and T1_stimulus is the stimulus sending timestamp obtained from the input stimulus data (source: input stimulus data). Optionally, the response time of each action is calculated, and the average value, maximum value, and minimum value are counted.

[0086] In this embodiment, the accuracy error can be determined by the following calculation formula:

[0087] Error = |(A_measured - A_expected) / A_fullscale| * 100%

[0088] wherein Error is the measurement error percentage, A_measured is the device measurement actual value obtained from the output response data (source: output response data), A_expected is the theoretical value of the input stimulus (source: input stimulus data), and A_fullscale is the range range obtained from the device basic information (source: static data).

[0089] In this embodiment, the resource consumption can be determined by the following calculation formula:

[0090] CPU_avg = (ΣCPU_sample_i) / N

[0091] wherein CPU_avg is the average CPU occupancy, CPU_sample_i is the ith sampling value (source: system resource data), and N is the sampling point number. Similarly, the memory average usage, etc. can also be calculated.

[0092] Exemplarily, the functional logic analysis is performed on the target time-series data to determine whether each function is correct, and the functional logic analysis includes: matching the standardized output response data with the expected behavior pattern in the pre-set rule engine to determine whether each function is correctly implemented.

[0093] In this embodiment, first, a rules engine is set, and the expected results of the test cases are preset (for example, after injecting a fault current, a trip signal should be sent within 25 ms, and a corresponding GOOSE message should be generated), and then the output standardized response data is matched with the expected mode in the rules engine; if the matching is successful, the corresponding function passes, otherwise it fails. Finally, the detailed context data when the failure occurs (such as the actual action sequence, fault recording data) is recorded.

[0094] For example, compatibility analysis is performed on the target timing data to determine whether there is a compatibility problem; the compatibility analysis includes: comparing the test results of the same application on different device models, different firmware versions, and different environment configurations.

[0095] In this embodiment, the analysis results in each step can be associated with the collected static data and test environment data. Assume that the same APP runs on devices with hardware versions v1.0 and v2.0. Extract all performance indicators and function results of the two groups of tests respectively, and compare them. If the response time on the v2.0 hardware is significantly higher than that on the v1.0, or a certain function fails on the v2.0 but passes on the v1.0, the platform automatically identifies it as a compatibility defect and associates it with specific environment variables (such as: the v2.0 hardware uses a different CPU model).

[0096] In step S204, a test report is automatically generated based on the analysis results.

[0097] Among them, the test report includes at least one of the statistical data of the quantitative performance indicators, the function pass rate data and the compatibility matrix.

[0098] In this embodiment, the obtained analysis results can be automatically synthesized into a structured test report and presented through a visual interface.

[0099] In the above-mentioned automatic testing method of the application, the test execution engine applies a preset input excitation to the tested protection and control device, and collects multi-source data during the test; so that the test execution engine can simulate input of various signals and control instructions to enable the tested protection and control device to complete various function tests. The collected multi-source data is preprocessed to obtain timing data in a unified form; so that the time of various collected data can be aligned, facilitating subsequent analysis. Various analyses are performed on the timing data to obtain analysis results; based on the analysis results, a test report is automatically generated, and the test report includes at least one of the statistical data of the quantitative performance indicators, the function pass rate data and the compatibility matrix. Thus, the overall automation and data-driven of the test process are realized, and the function, performance and compatibility problems of the application under different device models, firmware versions and environment configurations can be accurately and efficiently found, the test cycle is greatly shortened, and the product quality is improved.

[0100] In another exemplary embodiment, as shown in Figure 3 , an automated testing method of an application is provided, which is applied to Figure 1 a server as an example, including the following steps S301 to S307. Among them:

[0101] Step S301, a test execution engine applies a preset input excitation to a measured protection control device, and collects multi-source data during the test process.

[0102] Step S302, the collected multi-source data is preprocessed to obtain time series data in a unified form.

[0103] Step S303, various analyses are performed on the time series data to obtain analysis results.

[0104] Step S304, a test report is automatically generated based on the analysis results.

[0105] The specific implementation process and technical effects of steps S301 to S304 in this embodiment are described in the related description of steps S201 to S204 in the method embodiment shown in Figure 2 , which will not be repeated here.

[0106] Step S305, the test report is displayed using a visual chart.

[0107] In this embodiment, the test report can be displayed using a visual dashboard (Dashboard). The trend chart can display the change trend of response time, resource consumption, etc. with time or different versions. The scatter plot / matrix chart can display the test result distribution under different device models and configurations, and intuitively expose the compatibility problem cluster.

[0108] Step S306, in response to a query operation on an abnormal result in the test report, at least one of the corresponding raw message, fault recording information and log information is displayed.

[0109] In this embodiment, for any failed use case or abnormal indicator, detailed context data such as raw messages, fault recordings, system logs, etc. can be drilled down to assist developers in quickly locating the root cause.

[0110] Step S307, after converting the test report into a preset format, the test report is automatically pushed to a target platform or a project management tool.

[0111] In this embodiment, the test results (in JSON / XML format) can be automatically pushed to a CI / CD platform (a set of automation tools and services) or a project management tool to trigger subsequent automated processes (such as automatically submitting a bug ticket).

[0112] It should be understood that the above steps S305 to S307 are optional steps, which can be executed alternatively or in combination.

[0113] In the embodiment, the full process of test case execution, data collection, processing analysis and report generation is completed automatically, which completely liberates the testers from repetitive labor, greatly shortens the test period, and can quickly respond to the needs of agile development and continuous integration / continuous deployment (CI / CD). Thus, the test efficiency and automation degree are greatly improved.

[0114] In the embodiment, the multi-source data collection technology is adopted, which can not only verify the external functional behavior of the device, but also deeply monitor the internal system resource consumption, combined with performance quantization analysis, which can find performance degradation and invisible defects that are difficult to detect by artificial. Thus, the test depth and coverage are comprehensively improved.

[0115] In the embodiment, through the compatibility analysis logic, the test results are automatically associated and compared with the device basic information and environment configuration, which can accurately and automatically identify the compatibility problems caused by hardware differences, firmware versions, configuration changes, and locate the problem source, thereby realizing the breakthrough of compatibility testing capability.

[0116] In the embodiment, the entire test process is based on data driving, and all conclusions are supported by quantitative data and analysis algorithms, avoiding subjective misjudgment. At the same time, the complete test data chain supports reverse drilling query from the final report to the original message and log, which greatly facilitates the problem reproduction and debugging, and improves the R&D debugging efficiency. Thus, the test conclusion is objective and accurate, and can be traced.

[0117] The platform (device) to which the embodiment method is applied can be based on an open architecture design, with high modularity, easy integration of new test tools, adaptation of new communication protocols or addition of new analysis algorithms, good foresight and adaptability, protection of investment value, good openness and scalability.

[0118] Exemplarily, Figure 4 A data flow diagram of an application of an automated test method provided by the embodiment of the present application is shown in FIG. 1. Figure 4As shown, first, the test case and the configuration parameter are input into the test execution engine, and the control command and / or the simulation signal are generated by the APP simulator; after the control command and / or the simulation signal are received by the measured protection control device, the test of various functions is performed, and the response message and / or the action signal are fed back to the APP simulator. In the process of performing the test, the multi-source data are collected through the preset probe and / or the listener, and then the collected multi-source data are preprocessed and standardized. The data after the preprocessing and the standardization are further processed and analyzed (such as performance index analysis, functional logic analysis, and compatibility analysis), and the analysis result is obtained. Finally, based on the analysis result, the test report is automatically generated. The test report can be presented in the form of a visual chart.

[0119] In the embodiment, through the complete, automatic, and closely related data flow, the following are achieved: data-driven decision making: discarding artificial subjective judgment, and all test conclusions are based on objective and quantitative data analysis and comparison; deep compatibility test: not only answering “whether it can be used”, but also answering “whether the performance is consistent in different environments”, and accurately locating the root of the compatibility problem through data correlation; greatly improving efficiency: automatic execution and data analysis liberate the test personnel from repetitive and tedious labor, and focus on use case design and complex problem analysis, thereby significantly shortening the entire research and development test cycle, and perfectly solving the pain point of “long traditional research and development test cycle”. In addition, the embodiment also adopts an open architecture, that is, the data collection, processing, and analysis modules all adopt standardized interface design, which is easy to integrate new test tools, protocols, and algorithms, and ensures the scalability of the platform in the future.

[0120] It should be understood that, although each step in the flowchart involved in each of the above embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each of the above embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps. It can be understood that the steps in different embodiments can be freely combined as needed, and various non-contradictory schemes formed by the combination are within the scope of protection of the present application.

[0121] Based on the same inventive concept, the embodiments of the present application also provide an application automation testing device for implementing the application automation testing method.

[0122] In one exemplary embodiment, as shown in Figure 5 An application automation testing device is provided, comprising a test execution engine 501, a multi-source data acquisition module 502, a data processing module 503, a data analysis module 504 and a test report generation module 505, wherein:

[0123] The test execution engine 501 is configured to apply a preset input stimulus to the tested protection and control device.

[0124] The multi-source data acquisition module 502 is configured to acquire multi-source data during the test.

[0125] The data processing module 503 is configured to preprocess the acquired multi-source data to obtain time series data in a unified form.

[0126] The data analysis module 504 is configured to analyze the time series data to obtain analysis results.

[0127] The test report generation module 505 is configured to automatically generate a test report based on the analysis results, wherein the test report includes at least one of quantitative performance index statistical data, function pass rate data and compatibility matrix.

[0128] For example, the test execution engine 501 is specifically configured to generate a preset input stimulus through an application simulator in the test execution engine, wherein the preset input stimulus includes analog signals and / or control commands; the analog signals include analog quantities and state quantities; and the multi-source data is acquired through a preset probe and / or listener under the condition that the tested protection and control device performs a test process according to the preset input stimulus; wherein the multi-source data includes at least one of static data, output response data, system resource data and test environment data of the tested protection and control device.

[0129] Exemplarily, the static data of the tested protection control device includes at least one of a device model, a hardware version, a firmware version, a software application version, a communication protocol type and a configuration parameter template; the output response data includes at least one of a response message generated by the tested protection control device, an action signal, an actual value and a time stamp of a fault report; the system resource data includes at least one of a CPU occupancy rate, a memory usage, a network throughput and a thread state; and the test environment data includes at least one of an operating system image version, a test tool version, a network configuration and a simulator model.

[0130] Exemplarily, the data processing module 503 is specifically configured to: parse binary messages in the multi-source data according to a predefined protocol specification, and perform keyword extraction on a parsed result to obtain key fields; and perform unified alignment on time stamps of all data based on a global clock source to obtain time series data in a unified time coordinate system.

[0131] Exemplarily, the data analysis module 504 is specifically configured to: perform performance index analysis on the target time series data to obtain quantitative performance indexes, the quantitative performance indexes including at least one of a response time, an accuracy error and resource consumption; perform functional logic analysis on the target time series data to determine whether each function is correct; the functional logic analysis includes: matching standardized output response data with an expected behavior pattern in a preset rule engine to determine whether each function is correctly implemented; and performing compatibility analysis on the target time series data to determine whether there is a compatibility problem; the compatibility analysis includes: horizontally comparing test results of the same application under different device models, different firmware versions and different environment configurations.

[0132] In another exemplary embodiment, as shown in Figure 6 there is provided an automatic testing device for an application, based on the device shown in Figure 5 the device can further include:

[0133] The display module 506 is configured to display a test report by using a visual chart.

[0134] The query module 507 is configured to display at least one of corresponding original messages, fault recording information and log information in response to a query operation on an abnormal result in the test report.

[0135] The sending module 508 is configured to automatically push the test report to a target platform or a project management tool after converting the test report into a preset format.

[0136] The modules in the automatic test device of the application can be implemented by software, hardware, or a combination thereof. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form, so that the processor can call and execute the operations of the modules.

[0137] In an exemplary embodiment, a computer device, which can be a terminal, has an internal structure as shown in Figure 7 The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to perform wired or wireless communication with external terminals. The wireless communication can be achieved through WIFI, mobile cellular network, near field communication (NFC), or other technologies. The computer program is executed by the processor to implement an automatic test method of an application. The display unit of the computer device is configured to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball, or touchpad arranged on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0138] Those skilled in the art can understand that Figure 7 The structure shown in the above description is only a block diagram of part of the structure related to the scheme of the present application, and does not limit the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0139] In an exemplary embodiment, a computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the following steps:

[0140] The test execution engine applies a preset input excitation to the measured protection and control device, and collects multi-source data during the test.

[0141] Preprocessing the collected multi-source data to obtain time series data in a unified form;

[0142] Performing various analyses on the time series data to obtain analysis results;

[0143] Based on the analysis results, automatically generating a test report, the test report including at least one of the following: statistical data of quantitative performance indicators, function pass rate data, and compatibility matrix.

[0144] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0145] Generating a preset input stimulus through an application simulator in the test execution engine, the preset input stimulus including: analog signals and / or control commands; the analog signals including: analog quantities and state quantities; under the condition that the measured protection and control device executes a test process according to the preset input stimulus, collecting multi-source data through a pre-set probe and / or listener; wherein the multi-source data includes at least one of the following: static data of the measured protection and control device, output response data, system resource data, and test environment data.

[0146] In one embodiment, the static data of the measured protection and control device includes at least one of the following: device model, hardware version, firmware version, software application version, communication protocol type, and configuration parameter template; the output response data includes at least one of the following: response message generated by the measured protection and control device, action signal, actual value and timestamp of fault report; the system resource data includes at least one of the following: CPU occupancy, memory usage, network throughput, and thread state; the test environment data includes at least one of the following: operating system image version, test tool version, network configuration, and simulator model.

[0147] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0148] According to a predefined protocol specification, parsing the binary message in the multi-source data, and performing keyword extraction on the parsed results to obtain key fields; taking a global clock source as a reference, uniformly aligning the timestamps of all data to obtain time series data in a unified time coordinate system; performing cleaning processing on the time series data to obtain target time series data.

[0149] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0150] The performance index analysis is performed on the target timing data to obtain a quantitative performance index, the quantitative performance index including at least one of a response time, an accuracy error and resource consumption; the function logic analysis is performed on the target timing data to determine whether each function is correct; the function logic analysis including: matching the standardized output response data with an expected behavior mode in a preset rule engine to determine whether each function is correctly implemented; the compatibility analysis is performed on the target timing data to determine whether there is a compatibility problem; the compatibility analysis including: horizontally comparing test results of the same application under different device models, different firmware versions and different environment configurations.

[0151] In one embodiment, the processor further implements the following steps when executing the computer program:

[0152] The test report is displayed by using a visual chart;

[0153] In response to a query operation on an abnormal result in the test report, at least one of corresponding original message, fault recording information and log information is displayed;

[0154] After the test report is converted into a preset format, the test report is automatically pushed to a target platform or a project management tool.

[0155] In one embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method steps in the above embodiments.

[0156] In one embodiment, a computer program product is provided, and the computer program product includes a computer program, and the computer program is executed by a processor to implement the method steps in the above embodiments.

[0157] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the processes of the above-mentioned embodiment methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., but is not limited thereto. The processor involved in the embodiments provided in the present application can be a general processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but is not limited thereto.

[0158] The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, it should be considered as the scope of the present application.

[0159] The above embodiments only express several implementation ways of the present application, and the description is specific and detailed, but it should not be understood as a limitation to the patent scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. An automated testing method for an application, characterized in that, The method includes: The test execution engine applies preset input stimuli to the protection and control device under test and collects multi-source data during the test. The collected multi-source data is preprocessed to obtain time-series data in a unified form; Various analyses were performed on the time series data to obtain the analysis results; Based on the analysis results, a test report is automatically generated. The test report includes at least one of the following: statistical data on quantitative performance indicators, functional pass rate data, and a compatibility matrix.

2. The method according to claim 1, characterized in that, The process involves applying a preset input stimulus to the protected monitoring and control device under test through a test execution engine, and collecting multi-source data during the test, including: Preset input stimuli are generated by the application simulator in the test execution engine. The preset input stimuli include: analog signals and / or control commands; the analog signals include: analog quantities and status quantities. When the protection and control device under test performs the test process according to the preset input excitation, the multi-source data is collected by preset probes and / or listeners; wherein, the multi-source data includes at least one of the following: static data of the protection and control device under test, output response data, system resource data and test environment data.

3. The method according to claim 2, characterized in that, The static data of the tested protection and control device includes at least one of the following: device model, hardware version, firmware version, software application version, communication protocol type, and configuration parameter template; The output response data includes at least one of the following: the response message generated by the tested protection and control device, the action signal, the actual value of the fault report, and the timestamp; The system resource data includes at least one of the following: CPU utilization, memory usage, network throughput, and thread status. The test environment data includes at least one of the following: operating system image version, test tool version, network configuration, and simulator model.

4. The method according to claim 1, characterized in that, The preprocessing of the collected multi-source data to obtain time-series data in a unified form includes: According to the predefined protocol, the binary messages in the multi-source data are parsed, and the keywords of the parsing results are extracted to obtain key fields; Using a global clock source as a reference, the timestamps of all data are aligned to obtain time-series data in a unified time coordinate system; The time series data is cleaned to obtain the target time series data.

5. The method according to claim 4, characterized in that, The process of performing various analyses on the time-series data to obtain analysis results includes at least one of the following steps: The target time series data is subjected to performance index analysis to obtain quantitative performance indexes, which include at least one of response time, accuracy error and resource consumption. Perform functional logic analysis on the target time series data to determine whether each function is correct; the functional logic analysis includes: matching the standardized output response data with the expected behavior pattern in the preset rule engine to determine whether each function is correctly implemented; A compatibility analysis is performed on the target time series data to determine whether there are any compatibility issues. The compatibility analysis includes: comparing the test results of the same application under different device models, different firmware versions, and different environmental configurations.

6. The method according to any one of claims 1 to 5, characterized in that, After automatically generating a test report based on the analysis results, the method further includes at least one of the following: The test report is presented using visual charts; In response to a query operation on an abnormal result in the test report, at least one of the following is displayed: the corresponding original message, fault waveform information, and log information; After the test report is converted into a preset format, it is automatically pushed to the target platform or project management tool.

7. An automated testing device for an application, characterized in that, The device includes: The test execution engine is used to apply preset input stimuli to the protection and control device under test; The multi-source data acquisition module is used to acquire multi-source data during the testing process; The data processing module is used to preprocess the collected multi-source data to obtain time-series data in a unified form; The data analysis module is used to perform various analyses on the time series data and obtain analysis results; The test report generation module is used to automatically generate a test report based on the analysis results. The test report includes at least one of the following: statistical data of quantitative performance indicators, functional pass rate data, and compatibility matrix.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images