Release-oriented multi-dimensional integrated test quality real-time observation method and system

Abstract

The invention relates to a release-oriented multi-dimensional integrated test quality real-time observation method and system. The method comprises the following steps: creating release; constructing a to-be-tested product; deploying the product to be tested; configuring an integrated test environment, and adding a test coverage data probe in the process of the to-be-tested product; based on the integrated testing environment, testing the to-be-tested product in real time; in a test, acquiring test coverage data by using the test coverage data probe; the method comprises the following steps: performing first-stage aggregation on cross-process test coverage data of a to-be-tested product in a test, realizing second-stage aggregation of cross-version test coverage data based on the test coverage data subjected to the first-stage aggregation, and realizing third-stage aggregation of cross-test environment test coverage data based on the test coverage data subjected to the second-stage aggregation; and obtaining a full coverage report, an incremental coverage report and an interface coverage report based on the test coverage data after the three-stage aggregation.

Description

Technical Field

[0001] The present invention relates to the field of software testing technology, and in particular to a release-oriented multi-dimensional integrated test quality real-time observation method and system. Background Art

[0002] Software development is a complex engineering management process involving multiple links and positions, and is usually demand-oriented. A complex software system consists of multiple independent and mutually collaborative components. In order to meet new requirements, multiple components often need to be updated during the development process. Therefore, it is crucial to introduce the concept of release to clarify the list of requirements, cycle, quality requirements and update component list for delivery. Testing is a key link in improving software quality, including unit testing, integration testing and system testing. Unit testing is mainly aimed at testing a single module or function, while integration testing is the responsibility of testers. After multiple components run in coordination, test cases are executed to evaluate the overall quality of the software system. The quality of integration testing has a significant impact on the overall quality of the software system. The quality of integration testing is usually evaluated using test reports. In practice, there are problems such as reliance on subjective judgment, low timeliness and high labor costs. How to automatically and real-timely observe the quality of integration testing is crucial. Observing the test coverage of the system code under test is an important means of measuring test quality. Conventionally, probes can be set to collect test coverage and finally generate full code coverage. In practice, in addition to the test coverage of the full code, the test coverage of the incremental code and the interface coverage are also needed, because the incremental code coverage can more intuitively reflect the quality within the scope of the change released, and the interface coverage can evaluate the quality from the business perspective and the perspective of interaction between components. Therefore, it is crucial to evaluate the test quality from multiple dimensions.

[0003] Existing methods provide tools for observing the test coverage of a single process (such as JacocoAgent), but in order to meet the above purposes, a series of problems need to be solved, including how to aggregate the test coverage data of a single process into a release-oriented quality report, how to merge the test coverage data of cross-commit code, how to retain the test coverage data of the previous committed code version in the release, how to generate an interface coverage report based on the test coverage data, and how to distinguish test traffic to ensure that the quality report accurately reflects the effect of the integration testing activities.

[0004] In summary, there is currently a lack of a multi-dimensional integrated test quality real-time observation method and system to solve or partially solve the above-mentioned problems. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a release-oriented multi-dimensional integrated test quality real-time observation method and system to solve or partially solve one or more problems of how to aggregate the test coverage data of a single process into a release-oriented quality report, how to merge the test coverage data across commit codes, how to retain the test coverage data of the previous submitted code version in the release, how to generate an interface coverage report based on the test coverage data, and how to distinguish test traffic to ensure that the quality report accurately reflects the effect of the integrated testing activities.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] One aspect of the present invention provides a method for real-time observation of multi-dimensional integrated test quality for release, comprising the following steps:

[0008] Step S1, create a release;

[0009] Step S2, constructing the product to be tested;

[0010] Step S3, deploying the product to be tested;

[0011] Step S4, configuring an integrated test environment, and adding a test coverage data probe in the process of the product to be tested;

[0012] Step S5, performing real-time testing on the product to be tested based on the integrated testing environment;

[0013] Step S6, during the test, using the test coverage data probe to obtain test coverage data;

[0014] Step S7, performing a first-stage aggregation of the test coverage data of the product under test across processes during the test, and based on the test coverage data after the first-stage aggregation, implementing a second-stage aggregation of the test coverage data across versions, and based on the test coverage data after the second-stage aggregation, implementing a third-stage aggregation of the test coverage data across test environments;

[0015] Step S8, based on the test coverage data after three-stage aggregation, obtain a full coverage report, an incremental coverage report and an interface coverage report.

[0016] As a preferred technical solution, in step S7, the one-stage polymerization includes:

[0017] Step S701, adding write locks to the test coverage data CovData0 before aggregation and the test coverage data CovData1, CovData2 and CovData3 after aggregation at each stage;

[0018] Step S702, filtering the branch list corresponding to the target release status;

[0019] Step S703, screening the test coverage data CovData0 matching the branch list from the test coverage data CovData0 before aggregation;

[0020] Step S704, based on the screened test coverage data CovData0, according to the environment and the target product, the test coverage data corresponding to the multiple processes generated by restarting the product to be tested in the test environment are aggregated in one stage to obtain the test coverage data CovData1;

[0021] Step S705, if the existing test coverage data CovData1 is inconsistent with the test coverage data CovData1 generated by the one-stage aggregation in step S704, a list of inconsistent test coverage data is obtained;

[0022] Step S706, if the test coverage data CovData1 generated by the one-stage aggregation in step S704 does not exist in the existing test coverage data CovData1, obtain a list of missing test coverage data;

[0023] Step S707, CovData0 implements the same commit merging of test coverage data by group;

[0024] Step S708: based on the inconsistent test coverage data group list, delete the existing invalid test coverage data CovData1, and based on the inconsistent test coverage data group list, insert the newly added aggregated test coverage data CovData1.

[0025] As a preferred technical solution, in step S7, the two-stage polymerization includes:

[0026] Step S710, filtering a branch list corresponding to a target release state, and filtering test coverage data CovData1 matching the branch list from the test coverage data CovData1 after the first stage of aggregation;

[0027] Step S711, based on the screened test coverage data CovData1, perform two-stage aggregation according to environment, branch and component to obtain test coverage data CovData2;

[0028] Step S712, if the existing test coverage data CovData2 is inconsistent with the test coverage data CovData2 generated by the one-stage aggregation in step S704, obtain a list of inconsistent test coverage data;

[0029] Step S713, if the test coverage data CovData2 generated by the one-stage aggregation in step S711 does not exist in the existing test coverage data CovData2, obtain a list of missing test coverage data;

[0030] Step S714, CovData1 implements cross-commit merging of test coverage data by group;

[0031] Step S715: based on the inconsistent test coverage data group list, delete the existing invalid test coverage data CovData2, and based on the inconsistent test coverage data group list, insert the newly added aggregated test coverage data CovData2.

[0032] As a preferred technical solution, the step S714 includes:

[0033] Step S91, obtaining test coverage data of a reference commit artifact that is earlier in time, and test coverage data of a target commit artifact that is later in time, wherein the test coverage data of the reference commit artifact and the test coverage data of the target commit artifact both include test coverage data of bytecodes clustered by classes and methods;

[0034] Step S92, for each class in the test coverage data of the target commit artifact, compare and merge with the corresponding class in the test coverage data of the benchmark commit artifact;

[0035] Step S93, summarize and output the test coverage data of the merged target commit artifacts.

[0036] As a preferred technical solution, the step S92 includes:

[0037] Step S921, determine whether the current class of the test coverage data of the target commit artifact exists in the coverage data of the baseline commit, if not, execute S922, if yes, execute S923;

[0038] Step S922, using the target coverage data as the summary basis of step S93, skipping the comparison and merging of the current class, and performing the comparison and merging of the next class;

[0039] Step S923, compare the class constructor method in the class with the class constructor method of the same class in the benchmark coverage data to determine whether there is a change, if there is a change, execute S922, if there is no change, execute S924;

[0040] Step S924, iterating the test coverage data of each method in the test coverage data of the current class, and performing bytecode comparison and merging of the coverage data of each method.

[0041] As a preferred technical solution, the step S924 includes:

[0042] Step S9241, determine whether the current method exists in the coverage data of the baseline commit, if not, execute S9242, if yes, execute S9243;

[0043] Step S9242, using the target coverage data as the summary benchmark of step S93, skipping the comparison and merging of the current method, and performing the comparison and merging of the next class;

[0044] Step S9243, comparing the test coverage data of the bytecodes of the benchmark method and the target method coverage data, if there is a partial overlap, execute S9244, if there is no overlap, execute S9242;

[0045] Step S9244, determine whether the overlapping decomposition of the benchmark method and the target method is located in the if instruction block, if the boundary is located in the if instruction block, execute S9245, if not, execute S9246;

[0046] Step S9245, remove the covered data in the if instruction block in the benchmark, determine whether it is an if instruction block according to the content, if so, set all isCover fields in the instruction block to not covered, and execute S9246;

[0047] Step S9246, merge the covered data in the benchmark method into the target method. If the instruction isCover field in the benchmark is covered, modify the coverage mark in the target method to uncovered, and use the calculated coverage data of the method as the summary benchmark in step S93.

[0048] As a preferred technical solution, in step S7, the three-stage polymerization includes:

[0049] Step S720, filtering a branch list corresponding to the target release state, and filtering the test coverage data CovData2 matching the branch list from the test coverage data CovData2 after the second-stage aggregation;

[0050] Step S721, based on the screened test coverage data CovDat2, three-stage aggregation is performed according to branches and components to obtain test coverage data CovData3;

[0051] Step S722, if the existing test coverage data CovData3 is inconsistent with the test coverage data CovData3 generated by the one-stage aggregation in step S704, a list of inconsistent test coverage data is obtained;

[0052] Step S723, if the test coverage data CovData3 generated by the one-stage aggregation in step S711 does not exist in the existing test coverage data CovData3, obtain a list of missing test coverage data;

[0053] Step S724, CovData2 implements cross-commit merging of test coverage data by group, and stores the test coverage data CovData3 based on the preset branch-release relationship;

[0054] Step S725: based on the inconsistent test coverage data group list, delete the existing invalid test coverage data CovData3, and based on the inconsistent test coverage data group list, insert the newly added aggregated test coverage data CovData3.

[0055] As a preferred technical solution, the step S8 includes:

[0056] Step S81, generating a full coverage report and indicators, using a test coverage data parsing tool to generate a full report, parsing to obtain line coverage and branch coverage, and storing them in a test coverage indicator and report data structure;

[0057] Step S82, generating an incremental coverage report and indicators, parsing the full coverage report generated in S81 according to the code comparison result of the component under the release and the code in the baseline version, screening the test coverage of the changed line numbers, and summarizing them into incremental coverage reports and indicators;

[0058] Step S83, generate interface coverage report and indicators.

[0059] As a preferred technical solution, the step S83 includes:

[0060] Step S831, parse the bytecode file, obtain the method list, and parse the bytecode file of the product to generate the method list;

[0061] Step S832, filter out private methods and initialization methods;

[0062] Step S833, identifying the interface and the methods belonging to the interface according to the configuration;

[0063] Step S834, generating a coverage method list according to the test coverage data;

[0064] Step S835, generating a list of covered and uncovered interfaces based on the identified interface and the methods belonging to the interface and the list of covered methods;

[0065] Step S836, generate an interface coverage report.

[0066] Another aspect of the present invention provides a release-oriented multi-dimensional integrated test quality real-time observation system, comprising:

[0067] A test coverage data collection module is used to configure an integrated test environment, create a release, build and deploy a product to be tested, add a test coverage data probe to the process of the product to be tested, perform real-time testing on the product to be tested based on the integrated test environment, and use the test coverage data probe to obtain test coverage data during the test;

[0068] Test coverage data merging module, used to merge test coverage data in the same commit and across commits;

[0069] A test coverage data aggregation calculation module is used to perform a first-stage aggregation of the test coverage data of the product to be tested across processes during the test by calling the test coverage data merging module, and based on the test coverage data after the first-stage aggregation, realize a second-stage aggregation of the test coverage data across versions, and based on the test coverage data after the second-stage aggregation, realize a third-stage aggregation of the test coverage data across test environments;

[0070] The report and indicator module is used to obtain full coverage reports, incremental coverage reports, and interface coverage reports based on the test coverage data after three-stage aggregation.

[0071] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0072] (1) By configuring the integrated test environment, adding test coverage data probe settings in the process of the product to be tested, integrating the test environment, and distinguishing test traffic, the integrated test coverage data can more accurately reflect the coverage generated by the integrated test activities.

[0073] (2) Observe the test quality in real time from the release perspective, covering multiple components, cross-code commits, and multiple environments. The test coverage data results of previous commits within the release cycle can be retained to avoid repeated testing. Through three-stage aggregation, the problem of merging several single test coverage data files into a test coverage situation that can be observed from the release perspective is solved.

[0074] (5) Step S6 and step S704 provide a test coverage data collection logic to solve the problems of application restart, unique identification, and avoiding duplicate collection and omission.

[0075] (6) Step S714 solves the problem of merging test coverage data based on different tested code commits by providing a cross-version test coverage data merging logic. The conservative algorithm used in the merging ensures the prudence of the merging result and makes the result credible.

[0076] (7) S83 provides a method for generating interface coverage reports and indicator logic, and avoids relying on the observable system by identifying interfaces from coverage data. It can also control the interface identification range by configuring the identification mode, unifying the identification and statistics of different types of interfaces. The calculation of coverage data solves the coverage situation and provides test coverage from a business perspective.

[0077] (8) The quality of integrated testing can be observed in multiple dimensions, including full code coverage, incremental code coverage, and interface coverage. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 It is a flow chart of the real-time observation method of multi-dimensional integrated test quality for release in the embodiment;

[0079] Figure 2 This is a schematic diagram of the first stage polymerization in the embodiment;

[0080] Figure 3 It is a schematic diagram of the second stage polymerization in the embodiment;

[0081] Figure 4 This is a schematic diagram of the third stage polymerization in the embodiment;

[0082] Figure 5 A schematic diagram of the cross-version test coverage data file merging logic in the embodiment;

[0083] Figure 6 A schematic diagram of generating an interface coverage report and indicator logic in an embodiment;

[0084] Figure 7 Schematic diagram of a real-time observation system for multi-dimensional integrated test quality for release in an embodiment;

[0085] Figure 8 Schematic diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0086] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0087] The definitions of the relevant terms in the present invention are as follows:

[0088] Probes: Probes allow developers to dynamically monitor and modify bytecodes while the application is running. They are often used in scenarios such as program debugging, hot deployment, and performance diagnostic analysis. They can monitor method calls and record method names, parameters, return values, execution time, and exception information. Probes can achieve non-invasive tracking, which means that monitoring and tracking can be achieved without modifying business code, which is particularly important for service observability in microservice architectures.

[0089] Bytecode: Bytecode is the intermediate representation of a compiled programming language. It is a low-level machine code. Taking Java as an example, a bytecode file has a .class file extension and contains the definition of a Java class or interface. In bytecode, the if conditional statement is implemented through a series of conditional jump instructions, such as ifeq, which means jumping to the specified label position if the integer value at the top of the stack is equal to 0. These instructions are used to implement if, else if, else, and switch statements.

[0090] Example 1

[0091] In view of the problems existing in the aforementioned prior art, this embodiment provides a real-time observation method for multi-dimensional integrated test quality oriented to release.

[0092] This method meets the following conditions:

[0093] (1) An independent branch model is used for release management. Each release can contain multiple components, and each component corresponds to a unique branch. This structural design allows the branch of a component to be directly traced back to the specific release to which it belongs, thus ensuring the clarity of version control and the convenience of tracking.

[0094] (2) The test environment is deployed in a dedicated test resource pool, and each test environment is used for only one specific test purpose at a time, such as verification in the development phase or the integration test phase. These test environments can be generated and maintained by automated tools, and manual creation is also supported. Each test environment may contain multiple container environments (such as namespaces in Kubernetes) and several virtual machines or physical machines. In each test environment, build products of multiple components can be deployed, but in order to ensure the consistency and accuracy of the test, only one build product of the same component can be deployed in the same test environment. This design helps to isolate the impact of different test activities and ensure the accuracy and reliability of the test results.

[0095] See also Figure 1The overall process of this method covers a series of steps from creating a release to generating the final report and indicators, including creating a release, building artifacts, deploying artifacts, configuring an integrated test environment, reporting test coverage data in the integrated test environment, and aggregating coverage data in three stages, and finally forming reports and indicators that can be observed from a release perspective. Specifically, this method includes steps S1-S8, which are described as follows:

[0096] Step S1, create a release. Specify the list of planned release components and access control list included in the release. Component related information includes: component code repository, branch, version number, baseline version number and other information. In the present invention, according to the constraints, there is a one-to-one correspondence between the release and the component branch, that is, the unique branch can be determined based on the component and the release, and the release and component information can be uniquely determined based on the branch. The access control list specifies the quality requirements that the release needs to meet, and the type is List <qagate>, in QaGate, the gate indicator values ​​that the indicator items need to meet are specified, and these indicator items refer to the test coverage indicators generated in subsequent steps, such as full volume line coverage, etc. After completion, if the build artifact activity related to the release occurs, execute step S2.

[0097] Step S2, build the product. The product is derived from the code and provided by the code repository. During the construction process, each product will generate a data record of the Package type and store it in the database, including the component identification number to which the product belongs, the branch of the product, the code commit id and other fields. In order to meet the requirements of subsequent steps, the present invention saves the source code files (source_files) and the compiled binary files (target_files) in the construction environment at the same time when building the product. The product files can be saved in the product warehouse for downloading in subsequent steps. After the construction is completed, if a deployment activity related to the product occurs, execute step S3.

[0098] Step S3, deploy the product. The deployment process can be performed by an automated pipeline or manually. The deployment target environment can be a containerized environment (such as Kubernetes) or a virtual machine / physical machine environment. During deployment, the deployment environment needs to record the unique identification number of the product. After the execution is completed, execute step S4.

[0099] Step S4, set up the integrated test environment. This step is the core link. There are usually multiple sets of test environments, which are only used for one test purpose at the same time. Each set of test environments may include several container environments (such as Kubernetes namespaces) and several virtual machines / physical machines. During the software development process, multiple links may carry out testing activities. For example, in the development stage, the initial verification function will also be deployed to the test environment for testing activities. Therefore, it is necessary to distinguish different test traffic flows so that the final coverage report and indicators accurately reflect the effect of the integrated test activities. In this step, after setting a test environment as an integrated test environment, the integrated test traffic can be filtered out according to the aforementioned constraints. After receiving the instruction, the test environment management system will add a test coverage data probe (such as JacocoAgent) to the process of the product to be tested, so that it can export and report test coverage data. After completion, execute step S5.

[0100] Step S5, integration test activities. Test activities are carried out in the integrated test environment. This step can be initiated at any time or continuously. When the test coverage data changes, steps S6 to S8 are triggered. In steps S5 to S8, the test coverage data is uploaded in real time as the test activities continue and converted into test coverage reports and indicators. Users can monitor changes in the quality of integrated tests through the Web, etc. This part of the logic can realize the real-time description of this method.

[0101] Step S6, report the test coverage data. The process of each tested product during operation will be monitored by the probe added in step S4. When this step is triggered regularly, the probe will export and save the test coverage data, and generate a data record of type CovData0 at the same time, and put the environment where the process is located, the unique identification number of the product, the branch of the product, the commit id, the uuid of the coverage data, the file storage location, etc. into the data record. When the process of each tested product during operation is started, the probe will generate the uuid (process_uuid) of the process, which remains unchanged during the life of the process. The purpose of this field is to solve the problem that the coverage data is refreshed and may be lost when the process of the same product is restarted. Each test coverage file calculates an identification code (such as MD5) as data_uuid, which can ensure that the same data files are not pushed repeatedly. After execution, execute step S7.

[0102] Step S7, calculate the reported test coverage data, including three-stage aggregation. This step is the core link of this method, and is described in the logic of calculating the reported test coverage data (three-stage aggregation). After the execution is completed, execute step S8.

[0103] Step S8, generate coverage reports and indicators, including full coverage reports and indicators, incremental coverage reports and indicators, interface coverage reports and indicators, see steps S81 to S83.

[0104] Step S81, generate a full coverage report and indicators, use a test coverage data parsing tool (such as JacocoCli) to generate a full report, parse its line coverage and branch coverage, and store them in the test coverage indicator and report data structure (CovIndice).

[0105] Step S82, generate incremental coverage reports and indicators. According to the code comparison results of the component under this release and its baseline version (such as the previous release), parse the full coverage report generated by step S81, select the test coverage of the changed line numbers, and summarize them into incremental coverage reports and indicators.

[0106] Step S83, generate interface coverage report and indicators, see the following section for details on generating full coverage report and indicator logic.

[0107] Preferably, the first stage of merging groups CovData0 records according to certain rules, and merges each group into a CovData1 record. This stage of merging is the merging of test coverage data of the same commit. Figure 2 , the first phase of the merging process includes the following steps:

[0108] Step S701: Add a write lock to the database. Add a write lock to CovData0 / CovData1 / CovData2 / CovData3 to ensure that the data in these tables can only be executed by the current session during the database writing process to avoid data inconsistency caused by concurrent writing. After completion, execute step S702.

[0109] Step S702, calculate the valid branch list. Releases are stateful, such as under development, released and waiting to be launched, launched, etc. The upload of test coverage data has nothing to do with the release state, but the test coverage data is meaningful only in certain release states (such as under development). This step will filter the releases that need to calculate coverage and calculate the valid branch list. After completion, execute S703

[0110] Step S703, filter out the valid CovData0 list. Read all CovData0 records from the database, where there is a branch field, and filter out the valid CovData0 list according to the result of S702. After completion, execute S704.

[0111] Step S704: Aggregate CovData0 by env and package_id. After aggregating and grouping CovData0 by env and package_id, a Map is generated. <env,Map<package_id,Set <covdata0>>>Type data structure. This step is the core of the first phase of merging. Considering that a product is restarted multiple times in a certain environment, different process_uuid is generated after each restart. Their test coverage data should be merged and calculated. Therefore, the purpose of this step is to merge the process_uuid field according to the rules. After completion, execute step S705.

[0112] Step S705, calculate the CovData1 list to be deleted. Read the CovData1 list in the database, if their cov_data0_ids match a certain group Set in the group generated in S704 <covdata0>If the id list in is inconsistent, the record needs to be deleted. After completion, execute step S706.

[0113] Step S706, calculate the list of CovData0 groups to be added. Read the groups generated in step S704. If a group Set <covdata0>The id list in the set does not exist in the cov_data0_ids of the CovData1 list in the database, then the set <covdata0>The database needs to be newly added. After completion, step S707 is executed.

[0114] Step S707, CovData0 performs the same commit merging of test coverage data by group. <covdata0>It is easy to know that the commit_id value of each group is the same, and the data_files in each group are found to form a list, and then the test coverage file merging tool (such as Jacoco-client) is called to perform the same version merging to generate CovData1 records. After completion, step S708 is executed.

[0115] Step S708, clear invalid CovData1 and insert new CovData1. According to the deletion and addition records generated in step S705 and step S707, update the CovData1 records in the database. After completion, execute step S710.

[0116] Preferably, the second stage of merging groups CovData1 records according to certain rules, and each group is merged into a CovData2 record. The merge in this stage is a cross-commit merge. Figure 3 , the second phase of the merging process includes the following steps:

[0117] Step S710, filter out valid CovData1 lists. Read all CovData1 lists from the database, and filter out valid CovData1 lists according to the valid branch lists calculated in step S702. After completion, execute step S711.

[0118] Step S711, aggregate CovData1 by env, branch, component_id. After aggregating and grouping CovData1 by env, branch, and component_id, a Map is generated. <env,Map<branch,Map<component_id,Set <covdata1>>>>Type of data structure. This step is the core of the second phase of merging, and its purpose is to merge the test coverage data of different commit_ids according to the rules. After completion, execute step S712.

[0119] Step S712, calculate the CovData2 list to be deleted. The logic is similar to S705. After completion, execute step S713.

[0120] Step S713, calculate the list of CovData1 groups to be added. The logic is similar to S706. After completion, execute step S714.

[0121] Step S714, CovData1 performs cross-commit test coverage data merging by group. <covdata1>It is easy to know that the commit_id value of CovData1 in each group is different. Find the data_file in each group to form a list, perform the merging of cross-commit test coverage data, and generate CovData2 records. The merging of cross-commit test coverage data is the core of this method. The specific cross-version test coverage data file merging logic will be explained later. After completion, execute step S715.

[0122] Step S715, clearing invalid CovData2 and inserting newly added CovData2. After completion, execute step S720.

[0123] Preferably, the third stage merge groups CovData2 records according to certain rules, and each group is merged into a CovData3 record. The merge in this stage is a cross-commit merge. Figure 4 , the second phase of the merging process includes the following steps:

[0124] Step S720, filter out valid CovData2 lists. Read all CovData2 lists from the database, and filter out valid CovData2 lists based on the valid branch lists calculated in step S702. After completion, execute step S721.

[0125] Step S721, aggregate CovData2 by branch and component_id. After aggregating and grouping CovData2 by branch and component_id, a Map is generated. <branch,Map<component_id,Set <covdata2>>>Type of data structure. Consider that there may be multiple integrated test environments in the release, such as different test teams using their own integrated test environments, but they may test the same branch of products, and their test coverage data needs to be merged. This step is the core of the third phase of merging, and its purpose is to merge the test coverage data of different test environments (env) according to the rules. After completion, execute step S712.

[0126] Step S722, calculate the CovData3 list to be deleted. The logic is similar to step S705. After completion, execute step S723.

[0127] Step S723, calculate the list of CovData2 groups to be added. The logic is similar to step S706. After completion, execute step S724.

[0128] Step S724, CovData2 performs cross-commit test coverage data merging by group, and for the Set that needs to be added <covdata2>It is easy to know that the commit_id value of CovData2 in each group may be different. Find the data_file in each group to form a list and perform the merging of cross-commit test coverage data. According to the constraints, the corresponding release can be found from the branch of the component, and the branch is mapped to the release field and stored in the generated CovData3 record. After completion, execute step S725.

[0129] Step S725, clearing invalid CovData3 and inserting newly added CovData3.

[0130] In S726, the database write lock of step S701 is released.

[0131] Preferably, the cross-version test coverage data file merging logic, the test coverage data tools obtained from public channels (such as Jacoco-client) can only merge the test coverage data of the same commit, because the merging logic merges the test coverage data involved in the code of the baseline commit into the test coverage data of the target commit. If the two codes are inconsistent, they cannot be merged. In practice, the version of the application will be continuously iterated and updated during the release and development process. For example, some features have been developed and need to be tested first, and problems are found and fixed during the testing process and then retested. Therefore, the commitid of the test object will continue to change. Although it can avoid missed tests if the last commit is fully tested before release, manual testing in practice requires a long cycle, and it is not realistic to fully test the last committed product before release. This logic is proposed precisely to address the above problems.

[0132] Considering that only a small part of the code changes in a commit, the coverage data of the previous commit artifact can be reused for the majority of the unchanged code, and it can be merged into the coverage data of the target commit artifact to achieve continuous display of the test effect in the same release. Considering that the merging process cannot cause the code to be marked as tested even though it has changed and has not been tested, this logic follows the principle of prudence. Through logical settings, the data is not merged when it is uncertain whether it should be merged, thereby ensuring the accuracy of the results and avoiding missed tests in subsequent commits.

[0133] See also Figure 5 , the cross-version test coverage data file merging logic includes the following steps:

[0134] Step S91, this logic has two inputs: the test coverage data of the target commit artifact and the test coverage data of the benchmark commit artifact, where the benchmark commit comes first in terms of code submission timing. The goal of this logic is to merge the benchmark commit coverage data into the target commit test data. Both input data types are Map <class,Map<method,List <bytecodecovdata>>>, where class represents a class, method represents a method, a class contains several methods, ByteCodeCovData is the test coverage data of the bytecode, and a method contains several bytecodes. After completion, execute step S92.

[0135] Step S92, iterate each class in the target commit test coverage data, and execute steps S921-S924 for each class coverage data. After completion, execute step S93.

[0136] Step S921, determine whether the class exists in the coverage data of the baseline commit. If not, it means that the class is newly added in the target commit code and does not involve merging. Execute step S922. If it exists, execute step S923.

[0137] Step S922: The target coverage data is used as the standard, and no merging is required. The class coverage data collected in this step will be summarized in step S93.

[0138] Step S923, compare the <clinit>The method and the benchmark test cover the method of the same class in the data to determine whether there is any change. <clinit>is the name of the class constructor method, which includes the assignment operations of all static variables in the class and the statements in all static initialization blocks (static{}), which are executed in the order in which they appear in the source code. If this method changes, the initialization conditions of the class change. According to the principle of prudence, the benchmark coverage data should be abandoned and step S922 should be executed. If there is no change, step S924 should be executed.

[0139] Step S924, iterate the test coverage data of each method in the class test coverage data, and execute steps S9241-S9246 for the coverage data of each method.

[0140] Step S9241, determine whether the method exists in the coverage data of the baseline commit. If not, it means that the method is newly added in the target commit code and does not involve merging. Execute step S9242. If it exists, execute step S9243.

[0141] Step S9242: The target coverage data is used as the standard, and no merging is required. The method coverage data collected in this step will be summarized in step S93.

[0142] Step S9243, compare the test coverage data of the bytecodes of the baseline method and the target method coverage data. For example, if the bytecode of the baseline method is ABC and the bytecode of the target method is ABCD, then their first halves are considered to be the same, and step S9244 is executed. It is easy to know that if only part of the code is added to the tail of the target method, the coverage data in the baseline method can be considered to be merged into the target method. If not, step S9242 is executed.

[0143] Step S9244, determine whether the overlapping decomposition of the benchmark method and the target method is located in the if instruction block. Consider the example in step S9243. If the bytecode CD is located in the if instruction block, and C is covered but D is not covered, direct merging will result in part of the code being covered in the if instruction block, which does not conform to the code execution order. In fact, CD is either covered or not covered during the actual test process. Therefore, if the boundary is located in the if instruction block, execute step S9245, otherwise execute step S9246.

[0144] Step S9245, remove the covered data in the if instruction block in the benchmark. In the ByteCodeCovData data structure of the benchmark, determine whether it is an if instruction block according to the content. If so, set all isCover in the instruction block to not covered. After completion, execute step S9246.

[0145] Step S9246, merge the covered data in the benchmark method into the target method. Traverse the ByteCodeCovData data structures of both methods. If the instruction isCover in the benchmark method is covered, modify the coverage flag in the target method. The method coverage data calculated in this step will be summarized in step S93.

[0146] Step S93, summarize and output the merged target commit test coverage data, and the process ends.

[0147] Preferably, in step S83, usually, the statistical interface coverage data needs to be monitored and recorded through an observable system (such as OpenTelemetry), but this method has the following shortcomings: 1. Building an observable system requires more resources and is costly; 2. The interface access history recorded by the observable tool does not carry the code commit information of the interface, which makes it difficult to associate interface access data and code commit, posing a challenge to ensuring the accuracy of the data; 3. There are many types of interfaces (such as http, grpc), and custom classification may be required according to different technology stacks or business needs. The records in the observable tool do not directly contain the above custom information. In practice, a method is needed that can directly generate an interface coverage report from a bytecode file and a test coverage data file, and supports custom interface classification. This logic is proposed to solve the above problems.

[0148] See also Figure 6 , full coverage reporting and indicator logic includes the following steps:

[0149] Step S831, parse the bytecode file and obtain the method list. Parse the bytecode file of the product (target_files field in the Package data structure) to generate the method list. After completion, execute step S832.

[0150] Step S832, remove private methods and initialization methods. Privatization methods, initialization methods, etc. will not become interfaces, and this step will filter them out. After completion, execute step S833.

[0151] Step S833, identify the interface and the methods belonging to the interface according to the configuration. The core of this logic. Users can specify interface identification rules and classify them. For example, by specifying the method_annotations_has field in the InterfaceFeature interface feature data structure, all methods with @Mapping in the method annotation are identified as http interfaces. The subsequent steps will identify the methods matching this feature as http interfaces. Finally, the data structure of interface identification configuration and classification is formed (List<String, String> ). <interfacerecognitionconfig>Type) as one of the inputs of this step. After traversing the interface list generated in step S833 and identifying the features, a method list belonging to the interface can be generated (Map<method,api> Type). After completion, execute step S834.

[0152] Step S834: Generate a list of covered methods based on the test coverage data. As described in step S91, the test coverage data carries the methods of the covered code, which can be parsed to form a list of covered methods. <method>Type). After completion, execute step S835.

[0153] Step S835, calculate the covered methods. According to the results of steps S833 and S834, a list of covered and uncovered interfaces can be generated. After completion, execute step S836.

[0154] Step S836, generate an interface coverage report. Convert the list of covered and uncovered interfaces generated in step S835 into an easy-to-read report, and calculate the interface coverage index (apiCoverage), which is stored in the test coverage index and report data structure (CovIndice type).

[0155] The attached table shows the data structure in the embodiment.

[0156] Table 1 Release

[0157] component_release_list List of released components,List <componentrelease>type. < / componentrelease> qa_gate_list Release access control list, List <qagate>type. < / qagate>

[0158] Table 2 ComponentRelease release components

[0159] component Components repo Component code repository branch Code branches version Version Number base_version Baseline version number

[0160] Table 3 QaGate release access control

[0161] indice Indicator name, such as incremental row coverage. operator Operator, such as greater than. value Threshold value, such as 60%.

[0162] Table 4 Package products

[0163] package_id Unique product identification number component_id Component identification number branch Branches commit Code commit id source_files Source code files target_files Compiled binary file (bytecode)

[0164] Table 5 Native test coverage data reported by CovData0

[0165]

[0166] Table 6 Test coverage data generated by the first stage aggregation of CovData1

[0167]

[0168] Table 7 Test coverage data generated by the second stage aggregation of CovData2

[0169]

[0170] Table 8 Test coverage data generated by the third stage aggregation of CovData3

[0171]

[0172] Table 9 CovIndice test coverage indicators and reports

[0173] fullLineCoverage Full row coverage. Access control indicators. fullBranchCoverage Full branch coverage. Access control indicators. fullReport Full coverage report diffLineCoverage Incremental row coverage. Gatekeeper metrics diffBranchCoverage Incremental branch coverage. Gatekeeper metrics diffReport Incremental coverage report apiCoverage Interface coverage. Access control indicators. apiReport Interface coverage report.

[0174] Table 10 ByteCodeCovData bytecode test coverage data

[0175]

[0176] Table 11InterfaceRecognitionConfig interface recognition configuration

[0177]

[0178] Table 12InterfaceFeature interface features

[0179] superclass_has Whether the parent class method contains the specified class name class_annotations_has Whether the class annotation contains the specified annotation method_annotations_has Whether the method annotation contains the specified annotation param_annotations_has Whether the method parameter annotation contains specific annotations

[0180] Example 2

[0181] Based on Example 1, this example provides a real-time observation system for multi-dimensional integrated test quality for release. Figure 7 , consisting of the following five core modules:

[0182] Test coverage data collection module: This module is responsible for receiving and storing integrated test coverage data from the test environment, ensuring that all relevant test information can be effectively captured and retained.

[0183] Test coverage data aggregation calculation module: The core responsibility of this module is to calculate the test coverage data set associated with each release. It performs three-phase merging logic and calls the test coverage data merging module to generate release-based test quality reports and key indicators.

[0184] Test coverage data merging module: responsible for handling the merging of test coverage data within the same commit and across commits, ensuring that the test data between different commits can be correctly integrated to reflect the test coverage within the entire release cycle.

[0185] Report and Indicator Module: This module focuses on generating detailed test reports and key performance indicators, providing users with an intuitive display of test results, helping users understand the quality status of integration testing and make decisions accordingly.

[0186] Through the collaborative work of these five modules, the system can achieve multi-dimensional real-time monitoring and analysis of integration test quality, providing strong data support for software release.

[0187] The present invention has the following characteristics:

[0188] (1) By setting up the integrated test environment in step S4 and distinguishing the test traffic, the integrated test coverage data can more accurately reflect the coverage generated by the integrated test activities; (2) Real-time observation of test quality from the release perspective, covering multiple components, cross-code commits, and multiple environments; (3) The test coverage data results of the previous commits in the release cycle can be retained to avoid repeated testing; (4) The three-stage merging logic solves the problem of merging several single test coverage data files into test coverage that can be observed from the release perspective. (5) The CovData0 data structure, S6, and S704 together form the test coverage data collection logic, which solves the problems of application restart, unique identification, avoiding duplicate collection, and omissions (CovData0 data structure, S6, S704). (7) The cross-version test coverage data merging logic solves the problem of merging test coverage data based on different tested code commits. Among them, the conservative algorithm used during the merging ensures the prudence of the merged results and makes the results credible.

[0189] (8) By providing the generation of interface coverage reports and indicator logic, interfaces can be identified from coverage data to avoid relying on the observable system. The interface identification range can be controlled by configuring the identification mode, unifying the identification and statistics of different types of interfaces. The calculation of coverage data solves the coverage situation and provides test coverage from a business perspective.

[0190] (9) The quality of integrated testing can be observed in multiple dimensions, including full code coverage, incremental code coverage, and interface coverage.

[0191] Example 3

[0192] This embodiment provides an electronic device, characterized in that it includes: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the real-time observation method for multi-dimensional integrated test quality for release as described in Example 1.

[0193] See also Figure 8 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1-Figure 6 The real-time observation method for multi-dimensional integrated test quality for release. Of course, in addition to the software implementation, the present invention does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0194] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0195] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.< / method> < / interfacerecognitionconfig> < / clinit> < / clinit> < / bytecodecovdata> < / qagate>

Claims

1. A real-time observation method for multi-dimensional integrated test quality for release, characterized in that: The steps include: Step S1, create a release; Step S2, constructing the product to be tested; Step S3, deploying the product to be tested; Step S4, configuring an integrated test environment, and adding a test coverage data probe in the process of the product to be tested; Step S5, performing real-time testing on the product to be tested based on the integrated testing environment; Step S6, during the test, using the test coverage data probe to obtain test coverage data; Step S7, performing a first-stage aggregation of the test coverage data of the product under test across processes during the test, and based on the test coverage data after the first-stage aggregation, implementing a second-stage aggregation of the test coverage data across versions, and based on the test coverage data after the second-stage aggregation, implementing a third-stage aggregation of the test coverage data across test environments; Step S8, based on the test coverage data after three-stage aggregation, obtain a full coverage report, an incremental coverage report and an interface coverage report.

2. A method for real-time observation of multi-dimensional integrated test quality for release according to claim 1, characterized in that: In the step S7, the one-stage polymerization includes: Step S701, adding write locks to the test coverage data CovData0 before aggregation and the test coverage data CovData1, CovData2 and CovData3 after aggregation at each stage; Step S702, filtering the branch list corresponding to the target release status; Step S703, screening the test coverage data CovData0 matching the branch list from the test coverage data CovData0 before aggregation; Step S704, based on the screened test coverage data CovData0, according to the environment and the target product, the test coverage data corresponding to the multiple processes generated by restarting the product to be tested in the test environment are aggregated in one stage to obtain the test coverage data CovData1; Step S705, if the existing test coverage data CovData1 is inconsistent with the test coverage data CovData1 generated by the one-stage aggregation in step S704, a list of inconsistent test coverage data is obtained; Step S706, if the test coverage data CovData1 generated by the one-stage aggregation in step S704 does not exist in the existing test coverage data CovData1, obtain a list of missing test coverage data; Step S707, CovData0 implements the same commit merging of test coverage data by group; Step S708: based on the inconsistent test coverage data group list, delete the existing invalid test coverage data CovData1, and based on the inconsistent test coverage data group list, insert the newly added aggregated test coverage data CovData1.

3. The method for real-time observation of multi-dimensional integrated test quality for release according to claim 1, characterized in that: In the step S7, the two-stage polymerization includes: Step S710, filtering a branch list corresponding to a target release state, and filtering test coverage data CovData1 matching the branch list from the test coverage data CovData1 after the first stage of aggregation; Step S711, based on the screened test coverage data CovData1, perform two-stage aggregation according to environment, branch and component to obtain test coverage data CovData2; Step S712, if the existing test coverage data CovData2 is inconsistent with the test coverage data CovData2 generated by the one-stage aggregation in step S704, obtain a list of inconsistent test coverage data; Step S713, if the test coverage data CovData2 generated by the one-stage aggregation in step S711 does not exist in the existing test coverage data CovData2, obtain a list of missing test coverage data; Step S714, CovData1 implements cross-commit merging of test coverage data by group; Step S715: based on the inconsistent test coverage data group list, delete the existing invalid test coverage data CovData2, and based on the inconsistent test coverage data group list, insert the newly added aggregated test coverage data CovData2.

4. The method for real-time observation of multi-dimensional integrated test quality for release according to claim 3 is characterized in that: The step S714 includes: Step S91, obtaining test coverage data of a reference commit artifact that is earlier in time, and test coverage data of a target commit artifact that is later in time, wherein the test coverage data of the reference commit artifact and the test coverage data of the target commit artifact both include test coverage data of bytecodes clustered by classes and methods; Step S92, for each class in the test coverage data of the target commit artifact, compare and merge with the corresponding class in the test coverage data of the benchmark commit artifact; Step S93, summarize and output the test coverage data of the merged target commit artifacts.

5. The method for real-time observation of multi-dimensional integrated test quality for release according to claim 4 is characterized in that: The step S92 includes: Step S921, determine whether the current class of the test coverage data of the target commit artifact exists in the coverage data of the baseline commit, if not, execute S922, if yes, execute S923; Step S922, using the target coverage data as the summary basis of step S93, skipping the comparison and merging of the current class, and performing the comparison and merging of the next class; Step S923, compare the class constructor method in the class with the class constructor method of the same class in the benchmark coverage data to determine whether there is a change, if there is a change, execute S922, if there is no change, execute S924; Step S924, iterating the test coverage data of each method in the test coverage data of the current class, and performing bytecode comparison and merging of the coverage data of each method.

6. A method for real-time observation of multi-dimensional integrated test quality for release according to claim 5, characterized in that: The step S924 includes: Step S9241, determine whether the current method exists in the coverage data of the baseline commit, if not, execute S9242, if yes, execute S9243; Step S9242, using the target coverage data as the summary benchmark of step S93, skipping the comparison and merging of the current method, and performing the comparison and merging of the next class; Step S9243, comparing the test coverage data of the bytecodes of the benchmark method and the target method coverage data, if there is a partial overlap, execute S9244, if there is no overlap, execute S9242; Step S9244, determining whether the overlapping decomposition of the benchmark method and the target method is located in the if instruction block, if the boundary is located in the if instruction block, executing S9245, if not, executing S9246; Step S9245, remove the covered data in the if instruction block in the benchmark, determine whether it is an if instruction block according to the content, if so, set all isCover fields in the instruction block to not covered, and execute S9246; Step S9246, merge the covered data in the benchmark method into the target method. If the instruction isCover field in the benchmark is covered, modify the coverage mark in the target method to uncovered, and use the calculated coverage data of the method as the summary benchmark in step S93.

7. The method for real-time observation of multi-dimensional integrated test quality for release according to claim 1, characterized in that: In step S7, the three-stage polymerization includes: Step S720, filtering a branch list corresponding to the target release state, and filtering the test coverage data CovData2 matching the branch list from the test coverage data CovData2 after the second-stage aggregation; Step S721, based on the screened test coverage data CovDat2, three-stage aggregation is performed according to branches and components to obtain test coverage data CovData3; Step S722, if the existing test coverage data CovData3 is inconsistent with the test coverage data CovData3 generated by the one-stage aggregation in step S704, a list of inconsistent test coverage data is obtained; Step S723, if the test coverage data CovData3 generated by the one-stage aggregation in step S711 does not exist in the existing test coverage data CovData3, obtain a list of missing test coverage data; Step S724, CovData2 implements cross-commit merging of test coverage data by group, and stores the test coverage data CovData3 based on the preset branch-release relationship; Step S725: based on the inconsistent test coverage data group list, delete the existing invalid test coverage data CovData3, and based on the inconsistent test coverage data group list, insert the newly added aggregated test coverage data CovData3.

8. The method for real-time observation of multi-dimensional integrated test quality for release according to claim 1, characterized in that: The step S8 comprises: Step S81, generating a full coverage report and indicators, using a test coverage data parsing tool to generate a full report, parsing to obtain line coverage and branch coverage, and storing them in a test coverage indicator and report data structure; Step S82, generating an incremental coverage report and indicators, parsing the full coverage report generated in S81 according to the code comparison result of the component under the release and the code in the baseline version, screening the test coverage of the changed line numbers, and summarizing them into incremental coverage reports and indicators; Step S83, generate interface coverage report and indicators.

9. The method for real-time observation of multi-dimensional integrated test quality for release according to claim 8, characterized in that: The step S83 includes: Step S831, parse the bytecode file, obtain the method list, and parse the bytecode file of the product to generate the method list; Step S832, filter out private methods and initialization methods; Step S833, identifying the interface and the methods belonging to the interface according to the configuration; Step S834, generating a coverage method list according to the test coverage data; Step S835, generating a list of covered and uncovered interfaces based on the identified interface and the methods belonging to the interface and the list of covered methods; Step S836, generate an interface coverage report.

10. A real-time observation system for multi-dimensional integrated test quality for release, characterized in that: include: A test coverage data collection module is used to configure an integrated test environment, create a release, build and deploy a product to be tested, add a test coverage data probe to the process of the product to be tested, perform real-time testing on the product to be tested based on the integrated test environment, and use the test coverage data probe to obtain test coverage data during the test; Test coverage data merging module, used to merge test coverage data in the same commit and across commits; A test coverage data aggregation calculation module is used to perform a first-stage aggregation of the test coverage data of the product to be tested across processes during the test by calling the test coverage data merging module, and based on the test coverage data after the first-stage aggregation, realize a second-stage aggregation of the test coverage data across versions, and based on the test coverage data after the second-stage aggregation, realize a third-stage aggregation of the test coverage data across test environments; The report and indicator module is used to obtain full coverage reports, incremental coverage reports, and interface coverage reports based on the test coverage data after three-stage aggregation.

Images