A multi-dimension integrated test quality real-time observation method and system

Abstract

The application relates to a release-oriented multi-dimension integrated test quality real-time observation method and system, which comprises the following steps: creating a release; constructing a product under test; deploying the product under test; configuring an integrated test environment, adding a test coverage data probe in the process of the product under test; based on the integrated test environment, performing real-time testing on the product under test; in the testing, acquiring test coverage data by using the test coverage data probe; performing one-stage aggregation on the cross-process test coverage data of the product under test in the testing, performing two-stage aggregation on the test coverage data across versions based on the one-stage aggregated test coverage data, performing three-stage aggregation on the test coverage data across test environments based on the two-stage aggregated test coverage data; and based on the three-stage aggregated test coverage data, acquiring a full-amount coverage report, an incremental coverage report and an interface coverage report.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of software testing, in particular to a multi-dimensional integrated test quality real-time observation method and system for release. BACKGROUND

[0002] Software development is a complex engineering management process involving multiple links and post collaboration, usually demand-oriented. A complex software system is composed of multiple independent components that cooperate with each other. In order to meet new demands, multiple components often need to be updated during development, therefore, it is crucial to introduce the concept of release to clarify the demand list, cycle, quality requirements and update component list of delivery. Testing is a key link to improve software quality, including unit testing, integration testing and system testing, etc. Unit testing is mainly for testing a single module or function, while integration testing is responsible for the test personnel to execute test cases after multiple components run together to evaluate the overall quality of the software system. The quality of integration testing has a significant impact on the overall quality of the software system. The quality of integration testing is usually evaluated using test reports, and in practice there are problems such as relying on subjective judgment, timeliness, and high labor costs. How to automatically and real-time observe the quality of integration testing is crucial. Observing the test coverage of the code of the system under test is an important means to measure the quality of testing, and probes can be set to collect test coverage, and finally generate full-amount code coverage. In practice, in addition to the test coverage of the full-amount code, the test coverage of the incremental code and the interface coverage are also needed, because the incremental code coverage can more intuitively reflect the quality within the scope of this release change, and the interface coverage can evaluate the quality from the perspective of business and component interaction, therefore, it is crucial to evaluate the test quality from multiple dimensions.

[0003] Existing methods provide tools to observe the test coverage of a single process (such as JacocoAgent), but in order to meet the above purpose, a series of problems need to be solved, including how to aggregate the test coverage data of a single process into a quality report for release, how to merge the test coverage data across commit code, how to preserve the test coverage data of the previous commit code version in the release, how to generate an interface coverage report based on test coverage data, and how to distinguish test traffic to ensure that the quality report accurately reflects the effect of integration testing activities, etc.

[0004] In summary, there is currently a lack of a multi-dimensional integrated test quality real-time observation method and system to solve or partially solve the aforementioned problems. SUMMARY

[0005] The purpose of the present application is to provide a multi-dimensional integrated test quality real-time observation method and system for release to overcome the defects of the prior art, so as to solve or partially solve one or more problems of how to aggregate test coverage data of a single process into a quality report for release, how to merge test coverage data across commit code, how to retain test coverage data of a previous commit code version in a release, how to generate an interface coverage report based on test coverage data, and how to distinguish test traffic to ensure that the quality report accurately reflects the effect of integrated test activities.

[0006] The purpose of the present application can be achieved by the following technical solutions:

[0007] In one aspect of the present application, a multi-dimensional integrated test quality real-time observation method for release is provided, comprising the following steps:

[0008] Step S1, creating a release;

[0009] Step S2, building a product under test;

[0010] Step S3, deploying the product under test;

[0011] Step S4, configuring an integrated test environment and adding test coverage data probes in the processes of the product under test;

[0012] Step S5, based on the integrated test environment, performing real-time testing on the product under test;

[0013] Step S6, during testing, acquiring test coverage data using the test coverage data probes;

[0014] Step S7, aggregating test coverage data across processes of the product under test in one stage, based on the test coverage data aggregated in one stage, implementing two-stage aggregation of test coverage data across versions, and based on the test coverage data aggregated in two stages, implementing three-stage aggregation of test coverage data across test environments;

[0015] Step S8, based on the test coverage data aggregated in three stages, acquiring full-coverage reports, incremental coverage reports, and interface coverage reports.

[0016] As a preferred technical solution, in step S7, the one-stage aggregation comprises:

[0017] Step S701, acquiring write locks for the test coverage data CovData0 before aggregation and the test coverage data CovData1, CovData2, and CovData3 after each stage of aggregation;

[0018] Step S702, screening a branch list corresponding to a target release state;

[0019] Step S703, filtering the test coverage data CovData0 matching the branch list from the test coverage data CovData0 before the aggregation;

[0020] Step S704, based on the filtered test coverage data CovData0, according to the environment and the target product, performing one-stage aggregation on the test coverage data corresponding to the multiple processes generated by restarting the product under test in the test environment, to obtain the test coverage data CovData1;

[0021] Step S705, if the existing test coverage data CovData1 is inconsistent with the test coverage data CovData1 obtained by one-stage aggregation in step S704, obtaining a list of inconsistent test coverage data;

[0022] Step S706, if the test coverage data CovData1 obtained by one-stage aggregation in step S704 does not exist in the existing test coverage data CovData1, obtaining a list of missing test coverage data;

[0023] Step S707, implementing the same commit merging of the test coverage data CovData0 by group;

[0024] Step S708, based on the inconsistent test coverage data group list, deleting the existing invalid test coverage data CovData1, and based on the inconsistent test coverage data group list, inserting the newly aggregated test coverage data CovData1.

[0025] As a preferred technical solution, in step S7, the two-stage aggregation includes:

[0026] Step S710, filtering the branch list corresponding to the target release state, and filtering the test coverage data CovData1 matching the branch list from the test coverage data CovData1 after one-stage aggregation;

[0027] Step S711, based on the filtered test coverage data CovData1, performing two-stage aggregation according to the environment, branch and component, to obtain the test coverage data CovData2;

[0028] Step S712, if the existing test coverage data CovData2 is inconsistent with the test coverage data CovData2 obtained by one-stage aggregation in step S704, obtaining a list of inconsistent test coverage data;

[0029] Step S713, if the test coverage data CovData2 generated by one-stage aggregation in step S711 does not exist in the existing test coverage data CovData2, obtain the missing test coverage data list;

[0030] Step S714, implement cross-commit merging of test coverage data by groups based on CovData1;

[0031] Step S715, delete the existing invalid test coverage data CovData2 based on the inconsistent test coverage data group list, and insert the newly aggregated test coverage data CovData2 based on the inconsistent test coverage data group list.

[0032] As a preferred technical solution, the step S714 comprises:

[0033] Step S91, obtain test coverage data of a reference commit artifact in time and test coverage data of a target commit artifact in time, wherein the test coverage data of the reference commit artifact and the test coverage data of the target commit artifact both comprise test coverage data of bytecodes clustered by classes and methods;

[0034] Step S92, compare and merge, for each class in the test coverage data of the target commit artifact, the corresponding class of the test coverage data of the reference commit artifact;

[0035] Step S93, aggregate and output the merged test coverage data of the target commit artifact.

[0036] As a preferred technical solution, the step S92 comprises:

[0037] Step S921, judge whether the current class of the test coverage data of the target commit artifact exists in the test coverage data of the reference commit, if not, execute S922, and if yes, execute S923;

[0038] Step S922, take the target coverage data as the aggregation reference of step S93, skip the comparison and merging of the current class, and perform comparison and merging of the next class;

[0039] Step S923, compare the class constructor method in the class with the class constructor method of the same class in the reference test coverage data, judge whether there is a change, if yes, execute S922, and if no, execute S924;

[0040] Step S924, perform iteration operation on the test coverage data of each method in the test coverage data of the current class, and perform comparison and merging of bytecodes for each method coverage data.

[0041] As a preferred technical solution, the step S924 comprises:

[0042] Step S9241, judge whether the current method exists in the reference commit coverage data, if not, execute S9242, if yes, execute S9243;

[0043] Step S9242, take the target coverage data as the summary reference of step S93, skip the comparison and merging of the current method, and perform the comparison and merging of the next class;

[0044] Step S9243, compare the test coverage data of the bytecode of the reference method and the target method coverage data, if there is partial overlap, execute S9244, if there is no overlap, execute S9242;

[0045] Step S9244, judge whether the reference method and the target method overlap decomposition are located in the if instruction block, if the boundary is located in the if instruction block, execute S9245, if not, execute S9246;

[0046] Step S9245, remove the covered data in the if instruction block in the reference, judge whether it is an if instruction block according to the content, if yes, set all isCover fields in the instruction block to uncovered, execute S9246;

[0047] Step S9246, merge the covered data in the reference method to the target method, if the instruction isCover field in the reference is covered, modify the coverage mark in the target method to uncovered, and the calculated method coverage data is used as the summary reference in step S93.

[0048] As a preferred technical solution, in the step S7, the three-stage aggregation comprises:

[0049] Step S720, filter the branch list corresponding to the target release state, and filter the test coverage data CovData2 matched with the branch list from the test coverage data CovData2 after the two-stage aggregation;

[0050] Step S721, based on the filtered test coverage data CovDat2, perform three-stage aggregation according to the branch and component, and obtain test coverage data CovData3;

[0051] Step S722, the existing test coverage data CovData3 is inconsistent with the test coverage data CovData3 obtained by the one-stage aggregation in step S704, and a list of inconsistent test coverage data is obtained;

[0052] Step S723, if the test coverage data CovData3 generated by one-stage aggregation in step S711 does not exist in the existing test coverage data CovData3, obtain the list of missing test coverage data;

[0053] Step S724, CovData2 realizes cross-commit merging of test coverage data according to groups, and stores the test coverage data CovData3 based on the preset branch-release relationship;

[0054] Step S725, based on the inconsistent test coverage data group list, delete the existing invalid test coverage data CovData3, and based on the inconsistent test coverage data group list, insert the newly aggregated test coverage data CovData3.

[0055] As a preferred technical solution, the step S8 comprises:

[0056] Step S81, generating full-amount coverage report and index, using test coverage data analysis tool to generate full-amount report, analyzing to obtain line coverage rate and branch coverage rate, and storing in test coverage index and report data structure;

[0057] Step S82, generating incremental coverage report and index, according to the code comparison result of the component in the code of the release and the code of the baseline version, analyzing the full-amount coverage report generated in S81, screening the test coverage situation of the changed line number, and summarizing as incremental coverage report and index;

[0058] Step S83, generating interface coverage report and index.

[0059] As a preferred technical solution, the step S83 comprises:

[0060] Step S831, analyzing bytecode files to obtain a method list, and analyzing the bytecode files of the product to generate a method list;

[0061] Step S832, screening private methods and initialization methods;

[0062] Step S833, identifying interfaces and methods belonging to the interfaces according to configurations;

[0063] Step S834, generating a coverage method list according to test coverage data;

[0064] Step S835, generating a covered and uncovered interface list based on the identified interfaces and methods belonging to the interfaces and the coverage method list;

[0065] Step S836, generating an interface coverage report.

[0066] In another aspect of the present application, a multi-dimension integrated test quality real-time observation system for release is provided, comprising:

[0067] A test coverage data collection module is configured to configure an integrated test environment, create a release, build and deploy a product under test, add test coverage data probes in the process of the product under test, perform real-time testing on the product under test based on the integrated test environment, and acquire test coverage data by using the test coverage data probes in the testing.

[0068] A test coverage data merging module is configured to implement test coverage data merging within a commit and across commits.

[0069] A test coverage data aggregation calculation module is configured to perform one-stage aggregation of test coverage data of the product under test across processes in the testing by calling the test coverage data merging module, perform two-stage aggregation of test coverage data across versions based on the one-stage aggregated test coverage data, and perform three-stage aggregation of test coverage data across test environments based on the two-stage aggregated test coverage data.

[0070] A report and index module is configured to acquire full-amount coverage reports, incremental coverage reports and interface coverage reports based on the three-stage aggregated test coverage data.

[0071] Compared with the prior art, the present application has at least one of the following beneficial effects:

[0072] (1) By configuring an integrated test environment, adding test coverage data probe settings in the process of the product under test, distinguishing test traffic, and making the integrated test coverage data more accurately reflect the coverage situation generated by the integrated test activity.

[0073] (2) Real-time observation of test quality from the perspective of release, covering multiple components, across code commits and multiple environments. The test coverage data achievements of previous commits within the release period can be preserved to avoid repeated testing. Through three-stage aggregation, the problem of merging several single test coverage data files into test coverage data that can be observed from the perspective of release is solved.

[0074] (5) Steps S6 and S704 solve the problems of application restart, uniqueness identification, repeated collection avoidance and omission by providing a test coverage data collection logic.

[0075] (6) Step S714 solves the problem of test coverage data merging based on different code commits by providing a cross-version test coverage data merging logic. The conservative algorithm used in the merging ensures the prudence of the merging result, making the result reliable.

[0076] (7) S83 avoids relying on observable system by providing a generation interface coverage report and index logic to identify interface from coverage data, and can control interface identification range by identifying mode through configuration, unifies unified identification and statistics of different types of interfaces, and solves coverage situation by calculating coverage data to provide test coverage situation from the perspective of business.

[0077] (8) Multi-dimensional observation of integrated test quality, including full code coverage, incremental code coverage, and interface coverage. BRIEF DESCRIPTION OF DRAWINGS

[0078] Figure 1 Flowchart of the multi-dimensional integrated test quality real-time observation method for the release in the embodiment;

[0079] Figure 2 Schematic diagram of the first stage aggregation in the embodiment;

[0080] Figure 3 Schematic diagram of the second stage aggregation in the embodiment;

[0081] Figure 4 Schematic diagram of the third stage aggregation in the embodiment;

[0082] Figure 5 Schematic diagram of the cross-version test coverage data file merging logic in the embodiment;

[0083] Figure 6 Schematic diagram of the generation interface coverage report and index logic in the embodiment;

[0084] Figure 7 Schematic diagram of the multi-dimensional integrated test quality real-time observation system for the release in the embodiment;

[0085] Figure 8 Schematic diagram of the electronic device in the embodiment. DETAILED DESCRIPTION

[0086] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work should fall within the protection scope of the present application.

[0087] The related terms designed in the present application are defined as follows:

[0088] Probe: Probes allow developers to dynamically monitor and modify bytecode at runtime, commonly used in scenarios such as program debugging, hot deployment, performance diagnosis and analysis, etc. It can realize the monitoring of method calls, record method name, parameter, return value, execution time and exception information. Probes can achieve non-intrusive embedding, which means that without modifying the business code, monitoring and tracking can be achieved. This is particularly important for service observability in microservice architecture.

[0089] Bytecode: Bytecode is the intermediate representation form of the compiled programming language. It is a low-level machine code. For example, in Java, the bytecode file has a.class file extension and contains the definition of a Java class or interface. In bytecode, if condition statements are implemented through a series of conditional jump instructions, such as ifeq, which means if the integer value at the top of the stack is equal to 0, jump to the specified label position. These instructions are used to implement if, else if, else and switch statements.

[0090] Embodiment 1

[0091] To solve the problems of the prior art, the embodiment provides a multi-dimensional integrated test quality real-time observation method for publishing.

[0092] The method meets the following conditions:

[0093] (1) Independent branch mode is used for release management. Each release can contain multiple components, and each component corresponds to a unique branch. This structure design allows tracing the branch of a component directly to its specific release, ensuring the clarity of version control and the convenience of tracking.

[0094] (2) Test environments are deployed in a dedicated test resource pool. Each test environment is used for a specific test purpose at the same time, such as development stage verification or integration test stage. These test environments can be generated and maintained through automation tools, and manual creation is also supported. Each test environment may contain multiple container environments (such as namespaces in Kubernetes) and several virtual machines or physical machines. In each test environment, multiple component build products can be deployed, but to ensure the consistency and accuracy of testing, only one build product of the same component can be deployed in the same test environment. This design helps to isolate the impact of different test activities, ensuring the accuracy and reliability of test results.

[0095] Reference Figure 1The overall process of the method covers a series of steps from creating a release to generating final reports and indicators, including creating a release, building an artifact, deploying an artifact, configuring an integration test environment, reporting test coverage data in the integration test environment, and three-stage aggregation of coverage data to finally form reports and indicators that can be observed from the release perspective. Specifically, the method includes steps S1-S8, which are described as follows:

[0096] Step S1, creating a release. Specify the planned release component list and the access list in the release. Component-related information includes: code repository, branch, version number, baseline version number, and other information of the component. In this application, according to the constraint condition, the release and the component branch are a one-to-one correspondence, that is, according to the component and the release, a unique branch can be determined, and according to the branch, the release and the component information can be uniquely determined. The access list specifies the quality requirements that the release needs to meet, and the type is List <qagate>In the QaGate, the gate index value that the index item needs to meet is specified, and the index item references the test coverage index generated in the subsequent step, such as full-quantity line coverage. After completion, if a build artifact activity related to the release occurs, step S2 is executed.

[0097] Step S2, build artifact. The artifact is derived from the code, which is provided by the code repository. During the build process, each artifact will generate a data record of the Package type and store it in the database, including the component identification number to which the artifact belongs, the branch of the artifact, the code commit id, and the like. To meet the needs of the subsequent steps, the present application saves the source code files (source_files) and the compiled binary files (target_files) in the build environment at the artifact building link. The artifact file can be saved in the artifact repository for subsequent steps to download. After the build is completed, if a deployment activity related to the artifact occurs, step S3 is executed.

[0098] Step S3, deploy artifact. The deployment process can be performed by an automated pipeline or manually, and the deployment target environment can be a containerized environment (such as Kubernetes) or a virtual machine / physical machine environment. When deploying, the deployment environment needs to record the unique identification number of the artifact. After execution, step S4 is executed.

[0099] Step S4, set up integration test environment. This step is the core link. The test environment usually has multiple sets, which are used for only one testing purpose at the same time. Each test environment can include several container environments (such as Kubernetes namespaces) and several virtual machines / physical machines. During the software development process, multiple links can perform testing activities, such as deploying to the test environment for preliminary verification of functions during the development link. Therefore, it is necessary to distinguish different test traffic to make the final generated coverage report and index accurate and reflect the effect of integration testing activities. In this step, after a test environment is set as an integration test environment, the integration test traffic can be filtered according to the aforementioned constraint conditions. After receiving the instruction, the test environment management system will add a test coverage data probe (such as JacocoAgent) in the process of the artifact under test, so that it can export and report test coverage data. After completion, step S5 is executed.

[0100] Step S5, integration test activity. Test activities are performed in the integration test environment. This step can be initiated at any time or continuously performed. When the test coverage data changes, steps S6-S8 are triggered for execution. In steps S5-S8, test coverage data is uploaded and converted into test coverage reports and indexes in real time as the test activities continue. Users can monitor the changes in integration test quality through the Web, and this part of the logic can achieve the real-time description of the present method.

[0101] Step S6, reporting test coverage data. The process of each tested product running is monitored by the probe added in step S4. When this step is triggered, the probe exports and saves the test coverage data, and generates a data record of type CovData0, and puts the environment where the process is located, the unique identification number of the product, the branch of the product, the commit id, the uuid of the coverage data, the file storage location, etc. into the data record. When the process of each tested product running is started, the probe generates the uuid of the process (process uuid), which remains unchanged during the life of the process. The purpose of this field is to solve the problem of loss of coverage data caused by flushing when the process of the same product is restarted. The identification code (such as MD5) of each test coverage file is calculated as data uuid, which can ensure that the same data file is not pushed repeatedly. After execution, step S7 is performed.

[0102] Step S7, calculating the reported test coverage data, including three-stage aggregation. This step is the core link of the method, and is described in the logic of calculating the reported test coverage data (three-stage aggregation). After execution, step S8 is performed.

[0103] Step S8, generating coverage reports and indicators. Including full-coverage reports and indicators, incremental coverage reports and indicators, and interface coverage reports and indicators, see steps S81-S83.

[0104] Step S81, generating full-coverage reports and indicators. Using a test coverage data analysis tool (such as JacocoCli) to generate a full-coverage report, analyzing its line coverage and branch coverage, and storing them in a test coverage indicator and report data structure (CovIndice).

[0105] Step S82, generating incremental coverage reports and indicators. According to the comparison results of the code of the component in this release and the code in its baseline version (such as the last release), the full-coverage report generated in step S81 is parsed, and the test coverage of the changed line numbers is selected and summarized as an incremental coverage report and indicator.

[0106] Step S83, generating interface coverage reports and indicators. See the logic of generating full-coverage reports and indicators later.

[0107] Preferably, the first-stage merging groups the CovData0 records according to certain rules, and merges each group into a CovData1 record. The merging in this stage is the merging of test coverage data of the same commit. See Figure 2 The first-stage merging process includes the following steps:

[0108] Step S701, database write lock. Add write lock to CovData0 / CovData1 / CovData2 / CovData3, ensure that the data of these tables can only be executed by the current session during the process of writing the database, to avoid concurrent writing leading to data inconsistency. After completion, execute step S702.

[0109] Step S702, calculate the effective branch list. The release is stateful, such as development, released to be online, online, etc. The upload action of test coverage data is irrelevant to the state of release, but test coverage data should be meaningful in several release states (such as development). This step will filter the releases that need to calculate coverage, and calculate the effective branch list. After completion, execute S703

[0110] Step S703, filter the effective CovData0 list. Read all CovData0 records from the database, and the record has a branch field. According to the result of S702, filter the effective CovData0 list. After completion, execute S704.

[0111] Step S704, aggregate CovData0 by env and package_id. After grouping the aggregated CovData0 by env and package_id, a Map<env, Map<package_id, Set <covdata0>>> type data structure. This step is the core of the first stage of merging, considering that a certain product is restarted multiple times in a certain environment, and different process_uuids are generated after each restart, and their test coverage data should be merged and calculated, so the purpose of this step is to merge the process_uuid field according to the rules. After completion, step S705 is executed.

[0112] Step S705, calculate the CovData1 list to be deleted. Read the CovData1 list in the database, if their cov_data0_ids are in a certain group Set <covdata0>If the id list in the two tables is not consistent, the record needs to be deleted. After completion, step S706 is executed.

[0113] Step S706, the list of CovDataO groups to be added is calculated. The groups generated in step S704 are read, and if a group Set <covdata0>Set <covdata0>New additions need to be added to the database. When done, proceed to step S707.

[0114] Step S707, CovDataO performs a commit-merge of the test coverage data by group. For the Set <covdata0>The commit_id value of each group is the same, and the data_file in each group is found to form a list, and then a test coverage file merging tool (such as Jacoco-client) is called to perform the same version merging to generate the CovData1 record. After completion, step S708 is performed.

[0115] Step S708: clean up the invalid CovData1 and insert the new CovData1. According to the deletion and addition records generated in steps S705 and S707, the CovData1 records in the database are updated. After completion, step S710 is performed.

[0116] Preferably, the second stage merging groups the CovData1 records according to certain rules, and each group is merged into one CovData2 record. The merging in this stage is cross-commit merging. See Figure 3 The second stage merging process includes the following steps:

[0117] Step S710: screen the valid CovData1 list. All CovData1 lists are read from the database, and the valid CovData1 list is screened according to the valid branch list calculated in step S702. After completion, step S711 is performed.

[0118] Step S711: aggregate CovData1 according to env, branch, and component_id. After grouping the CovData1 according to env, branch, and component_id, a Map<env, Map<branch, Map<component_id, Set <covdata1>a data structure of the >>> type. This step is the core of the second phase of merging, and the purpose is to merge the test coverage data of different commit_ids according to the rules. After completion, step S712 is executed.

[0119] Step S712, calculate the CovData2 list to be deleted. Similar to the logic of S705. After completion, step S713 is executed.

[0120] Step S713, calculate the CovData1 group list to be added. Similar to the logic of S706. After completion, step S714 is executed.

[0121] Step S714, CovData1 performs cross-commit test coverage data merging of test coverage data by group. For the Set <covdata1>CovData1 in each group are different, find the data_file in each group respectively to form a list, perform the merge of test coverage data across commits to generate CovData2 records. The merge of test coverage data across commits is the core of the method, and the specific merge logic of test coverage data across versions will be described later. After completion, step S715 is performed.

[0122] Step S715, clean up the invalid CovData2 and insert the new CovData2. After completion, step S720 is performed.

[0123] Preferably, the third stage merge groups the CovData2 records according to certain rules, and each group is merged into a CovData3 record. The merge in this stage is a merge across commits. Referring to Figure 4 The second stage merge process includes the following steps:

[0124] Step S720, screen the valid CovData2 list. Read all the CovData2 list from the database, and screen out the valid CovData2 list according to the valid branch list calculated in step S702. After completion, step S721 is performed.

[0125] Step S721, aggregate CovData2 by branch and component_id. After grouping the aggregated CovData2 by branch and component_id, a Map<branch,Map<component_id,Set <covdata2>>> type data structure. Consider that there can be multiple integration test environments in a release, e.g. different test teams using their own integration test environments, but they can be testing the same branched artifacts and their test coverage data needs to be merged. This step is the core of the third phase merge and aims to merge the test coverage data from different test environments (env) according to the rules. When completed, step S712 is executed.

[0126] Step S722, calculate the list of CovData3 to be deleted. Similar logic as step S705. When completed, step S723 is executed.

[0127] Step S723, calculate the list of CovData2 groups to be added. Similar logic as step S706. When completed, step S724 is executed.

[0128] Step S724, CovData2 performs cross-commit test coverage data merge of test coverage data by group for the Set <covdata2>CovData2's commit_id value in each group can be different, find the data_file in each group respectively to form a list, and perform the cross-commit test coverage data merging. According to the constraints, the corresponding release can be found from the branch of the component, and after mapping the branch to the release field, it is stored in the generated CovData3 record. After completion, step S725 is performed.

[0129] In step S725, the invalid CovData3 is cleaned up, and the newly added CovData3 is inserted.

[0130] In S726, the database write lock of step S701 is released.

[0131] Preferably, the cross-version test coverage data file merging logic can only merge the test coverage data of the same commit by using the test coverage data tool (such as Jacoco-client) obtained through the public channel. The reason is that the merging logic will merge the test coverage data related to the code of the reference commit into the test coverage data of the target commit. If the codes are inconsistent, the merging cannot be performed. In practice, the version applied in the release development process will be continuously updated, such as some features have been developed and need to be tested first, and after the problems are found and repaired in the testing process, the testing is performed again. Therefore, the commit id of the testing object will change continuously. Although the full testing of the last commit before the release can avoid the missed testing, the period required for the manual testing is relatively long in practice, and it is not realistic to perform the full testing of the last commit before the release. The logic is proposed for the above problems.

[0132] Considering that only a small part of the code changes in one commit, the coverage data of the unchanged part of the code can be reused from the coverage data of the previous commit, and the coverage data of the target commit can be merged, so as to realize the continuous display of the testing effect in the same release. Considering that the merging process cannot cause the code to change and be marked as tested without testing, the logic follows the principle of prudence, and when it is not determined whether the data should be merged, the data is not merged, so as to ensure the accuracy of the result and avoid the missed testing of the subsequent commit.

[0133] Referring to Figure 5 The cross-version test coverage data file merging logic includes the following steps:

[0134] Step S91, this logic has two inputs: test coverage data of target commit artifact, test coverage data of baseline commit artifact, where the baseline commit is earlier than the target commit in terms of code commit chronology. The goal of this logic is to merge the baseline commit coverage data into the target commit test data. The data type of both inputs is Map<class,Map<method,List <bytecodecovdata>>>, wherein class represents a class, method represents a method, a class has several methods, ByteCodeCovData is test coverage data of byte code, and a method contains several byte codes. After completion, step S92 is executed.

[0135] Step S92, iteration is performed on each class in the target commit test coverage data, and steps S921-S924 are executed for each class coverage data. After completion, step S93 is executed.

[0136] Step S921, it is judged whether the class exists in the coverage data of the reference commit. If not, it is indicated that the class is newly added in the target commit code, and no merging is involved. Step S922 is executed. If yes, step S923 is executed.

[0137] Step S922, the target coverage data is used as the criterion, and no merging is needed. The class coverage data collected in this step is summarized in step S93.

[0138] Step S923, the class in the target commit is compared with the class in the reference commit. <clinit>The method and the benchmark test cover the same class of the method, determine whether the change occurs. <clinit>is the name of the class constructor method, contains the assignment operation of all static variables in the class and all the statements in the static {} block, which are executed in the order they appear in the source code. If the method changes, the initialization condition of the class changes, according to the principle of prudence, the benchmark test coverage data should be discarded, and step S922 is executed, if there is no change, step S924 is executed.

[0139] Step S924, iterate the test coverage data of each method in the class test coverage data, and execute steps S9241-S9246 for each method coverage data.

[0140] Step S9241, judge whether the method exists in the coverage data of the benchmark commit, if not, it means that the method is newly added in the target commit code, and is not involved in merging, execute step S9242, if exists, execute step S9243.

[0141] Step S9242, use the target coverage data as the standard, no merging is needed. The method coverage data collected in this step will be summarized in step S93.

[0142] Step S9243, compare the test coverage data of the bytecode of the benchmark method and the target method coverage data, for example, consider that the bytecode of the benchmark method is ABC, and the bytecode of the target method is ABCD, it is considered that their first half are the same, and step S9244 is executed. It is easy to know that if only part of the code is appended at the tail of the target method, the coverage data in the benchmark method can be considered to be merged into the target method. If not, execute step S9242.

[0143] Step S9244, judge whether the benchmark method and the target method overlap decomposition are located in the if instruction block, consider the example in step S9243, if the bytecode CD is located in the if instruction block, and C is covered and D is not covered, direct merging will cause part of the code in the if instruction block to be covered, which does not conform to the code execution order, in fact, CD will be covered or not covered in the real test process. Therefore, if the boundary is located in the if instruction block, execute step S9245, otherwise execute step S9246.

[0144] Step S9245, remove the covered data in the if instruction block in the benchmark. In the ByteCodeCovData data structure of the benchmark, judge whether it is an if instruction block according to the content, if it is, set all isCover in the instruction block to uncovered. After completion, execute step S9246.

[0145] Step S9246, merge the covered data in the baseline method into the target method. Traverse the ByteCodeCovData data structure of both, if the instruction isCover in the baseline is covered, modify the coverage flag in the target method. The method coverage data calculated in this step will be summarized in step S93.

[0146] Step S93, summarize and output the merged target commit test coverage data, and the flow ends.

[0147] Preferably, in step S83, generally, interface coverage data needs to be monitored and recorded by an observable system (such as OpenTelemetry) to record the interface access history, but this approach has the following disadvantages: 1. Building an observable system requires more resources and is costly; 2. The interface access history recorded by the observable tool does not carry the code commit information of the interface, making it difficult to associate interface access data and code commit, which poses a challenge to ensuring the accuracy of the data; 3. There are multiple types of interfaces (such as http, grpc), and according to different technology stacks or business needs, custom classification may be required, and the records in the observable tool do not directly include the above custom information. In practice, a method is needed that can directly generate an interface coverage report from bytecode files and test coverage data files, and support custom interface classification. This logic is proposed to solve the above problems.

[0148] Reference Figure 6 The full coverage report and index logic includes the following steps:

[0149] Step S831, parse the bytecode file to obtain the method list. Parse the bytecode file of the product (target_files field in the Package data structure) to generate a method list. After completion, execute step S832.

[0150] Step S832, remove private methods and initialization methods. Private methods, initialization methods, etc. will not become interfaces, and this step will filter them out. After completion, execute step S833.

[0151] Step S833, identify interfaces and methods belonging to interfaces according to the configuration. The core of this logic. Users can specify interface identification rules and classify them, such as by specifying the method_annotations_has field in the InterfaceFeature interface feature data structure, all methods with @Mapping in the method annotation are identified as http interfaces, and subsequent steps will identify the methods matching this feature as http interfaces. Finally, form the interface identification configuration and classification data structure (List <interfacerecognitionconfig>As one of the inputs of this step, the interface list generated in step S833 is traversed to identify the features and generate a list of methods belonging to the interface (Map<method, api> type). After completion, step S834 is executed.

[0152] In step S834, a list of covered methods is generated according to the test coverage data. As described in step S91, the test coverage data carries the methods covered by the coverage code, which can be parsed to form a list of covered methods (list <method>Type). When finished, step S835 is executed.

[0153] Step S835, calculate covered methods. From the results of steps S833 and S834, a list of covered and uncovered interfaces can be generated. When finished, step S836 is executed.

[0154] Step S836, generate interface coverage report. The list of covered and uncovered interfaces generated in step S835 is converted into a human readable report and the interface coverage metrics (apiCoverage) are calculated and stored in the test coverage metrics and report data structure (of type CovIndice).

[0155] The following tables are data structures used in the example.

[0156] Table 1 Release Release

[0157] component_release_list Published component list, List <componentrelease>Type. < / componentrelease> qa_gate_list Publish the access list, List <qagate>Type. < / qagate>

[0158] Table 2 Component Release Release component

[0159] component component repo component repository branch branch version version number base_version base version number

[0160] Table 3 QaGate Release Gate

[0161] indice indice name. e.g. incremental line coverage. operator operator. e.g. greater than. value value. e.g. 60%.

[0162] Table 4 Package Package

[0163] package_id package unique id component_id component id branch branch commit commit id source_files source files target_files compiled binary files (bytecode)

[0164] Table 5 CovData0 Reported native test coverage data

[0165]

[0166] Table 6 CovData1 First stage aggregation generated test coverage data

[0167]

[0168] Table 7 CovData2 Second stage aggregation generated test coverage data

[0169]

[0170] Table 8 CovData3 Third stage aggregation generated test coverage data

[0171]

[0172] Table 9 CovIndice Test coverage metrics and report

[0173] fullLineCoverage full line coverage. gate metric. fullBranchCoverage full branch coverage. gate metric. fullReport full report diffLineCoverage diff line coverage. gate metric diffBranchCoverage diff branch coverage. gate metric diffReport diff report apiCoverage api coverage. gate metric. apiReport api coverage report.

[0174] Table 10 Test coverage data for ByteCodeCovData bytecode

[0175]

[0176] Table 11 InterfaceRecognitionConfig interface recognition configuration

[0177]

[0178] Table 12 InterfaceFeature interface feature

[0179] superclass_has superclass_has class_annotations_has class_annotations_has method_annotations_has method_annotations_has param_annotations_has param_annotations_has

[0180] Embodiment 2

[0181] Based on Embodiment 1, this embodiment provides a multi-dimensional integrated test quality real-time observation system for release, which is shown in Figure 7 , and is composed of the following five core modules:

[0182] Test coverage data collection module: This module is responsible for receiving and storing integrated test coverage data from the test environment, ensuring that all relevant test information can be effectively captured and retained.

[0183] Test coverage data aggregation calculation module: The core responsibility of this module is to calculate the test coverage data set related to each release. It performs three-stage merging logic and calls the test coverage data merging module to generate release-based test quality reports and key indicators.

[0184] Test coverage data merging module: Responsible for handling the merging of test coverage data within and across commits, ensuring that test data between different commits can be correctly integrated to reflect the test coverage situation during the entire release cycle.

[0185] Report and indicator module: This module focuses on generating detailed test reports and key performance indicators to provide users with intuitive test result displays, helping users understand the quality status of integrated testing and make decisions accordingly.

[0186] Through the coordinated work of these five modules, the system can achieve multi-dimensional real-time monitoring and analysis of integrated test quality, providing strong data support for software release.

[0187] The present application has the following characteristics:

[0188] (1) By setting the integration test environment in step S4, distinguishing the test traffic, making the integration test cover data more accurately reflect the coverage situation caused by the integration test activity; (2) From the perspective of release, real-time observation of test quality, covering multiple components, cross-code commit, and multiple environments; (3) The test coverage data results of the previous commit in the release cycle can be preserved to avoid repeated testing; (4) The three-stage merging logic solves the problem of merging several single test coverage data files into a test coverage situation that can be observed from the release perspective. (5) CovData0 data structure, S6, S704 collectively form the test coverage data collection logic, which solves the problems of application restart, uniqueness identification, repeated collection avoidance, and omission (CovData0 data structure, S6, S704). (7) Cross-version test coverage data merging logic solves the problem of merging test coverage data based on different code commits. The conservative algorithm used in merging ensures the prudence of the merging result, making the result reliable.

[0189] (8) By providing interface coverage report and index generation logic, interface identification from coverage data avoids relying on the observable system, and the interface identification range can be controlled by configuring the identification mode. Unified identification and statistics of different types of interfaces are unified, and the calculation of coverage data provides test coverage from the business perspective.

[0190] (9) Multi-dimensional observation of integration test quality, including full code coverage, incremental code coverage, and interface coverage.

[0191] Embodiment 3

[0192] The embodiment provides an electronic device, characterized in that it comprises one or more processors and a memory, the memory having one or more programs stored therein, the one or more programs comprising instructions for executing the release-oriented multi-dimensional integration test quality real-time observation method as described in Embodiment 1.

[0193] Referring to Figure 8 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course, it can also include other hardware required by the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to implement the above Figure 1-6 The release-oriented multi-dimensional integration test quality real-time observation method. Of course, in addition to the software implementation, the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device.

[0194] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0195] The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.< / method> < / interfacerecognitionconfig> < / clinit> < / clinit> < / bytecodecovdata> < / qagate>

Claims

1. A method for real-time observation of multi-dimension integrated test quality oriented to release, characterized in that, The method comprises the following steps: Step S1, creating a release; Step S2, building a product to be tested; Step S3, deploying the product to be tested; Step S4, configuring an integration test environment, and adding a test coverage data probe in a process of the product to be tested; Step S5, performing real-time testing on the product to be tested based on the integration test environment; Step S6, obtaining test coverage data by using the test coverage data probe in testing; Step S7, performing one-stage aggregation on test coverage data of the product to be tested across processes in testing, performing two-stage aggregation on test coverage data across versions based on the one-stage aggregated test coverage data, and performing three-stage aggregation on test coverage data across test environments based on the two-stage aggregated test coverage data; Step S8, obtaining a full-coverage report, an incremental-coverage report and an interface-coverage report based on the three-stage aggregated test coverage data, In the step S7, the one-stage aggregation comprises: Step S701, adding a write lock to the test coverage data CovData0 before aggregation and the test coverage data CovData1, CovData2 and CovData3 after each stage of aggregation; Step S702, screening a branch list corresponding to a target release state; Step S703, screening test coverage data CovData0 matched with the branch list from the test coverage data CovData0 before aggregation; Step S704, performing one-stage aggregation on test coverage data corresponding to multiple processes restarted in a test environment according to an environment and a target product based on the screened test coverage data CovData0, to obtain test coverage data CovData1; Step S705, if the existing test coverage data CovData1 is inconsistent with the test coverage data CovData1 obtained after one-stage aggregation in the step S704, obtaining a list of inconsistent test coverage data; Step S706, if the test coverage data CovData1 obtained after one-stage aggregation in the step S704 does not exist in the existing test coverage data CovData1, obtaining a list of missing test coverage data; Step S707, performing same-commit merging of test coverage data according to groups in CovData0; Step S708, deleting the existing invalid test coverage data CovData1 based on the list of inconsistent test coverage data groups, and inserting newly aggregated test coverage data CovData1 based on the list of inconsistent test coverage data groups, In the step S7, the two-stage aggregation comprises: Step S710, screening a branch list corresponding to a target release state from the test coverage data CovData1 after one-stage aggregation, and screening test coverage data CovData1 matched with the branch list; Step S711, performing two-stage aggregation according to an environment, a branch and a component based on the screened test coverage data CovData1, to obtain test coverage data CovData2; Step S712, if the existing test coverage data CovData2 is inconsistent with the test coverage data CovData2 generated by the one-stage aggregation in step S704, obtaining a list of inconsistent test coverage data; Step S713, if the test coverage data CovData2 generated by the one-stage aggregation in step S711 does not exist in the existing test coverage data CovData2, obtaining a list of missing test coverage data; Step S714, CovData1 realizes cross-commit merging of test coverage data by group; Step S715, based on the inconsistent test coverage data group list, deleting the existing invalid test coverage data CovData2, based on the inconsistent test coverage data group list, inserting the newly aggregated test coverage data CovData2, The three-stage aggregation in step S7 includes: Step S720, filtering the branch list corresponding to the target release state, and filtering the test coverage data CovData2 matching the branch list from the test coverage data CovData2 after the two-stage aggregation; Step S721, based on the filtered test coverage data CovDat2, performing three-stage aggregation according to branches and components to obtain test coverage data CovData3; Step S722, the existing test coverage data CovData3 is inconsistent with the test coverage data CovData3 generated by the one-stage aggregation in step S704, and a list of inconsistent test coverage data is obtained; Step S723, if the test coverage data CovData3 generated by the one-stage aggregation in step S711 does not exist in the existing test coverage data CovData3, a list of missing test coverage data is obtained; Step S724, CovData2 realizes cross-commit merging of test coverage data by group, and stores the test coverage data CovData3 based on the preset branch-release relationship; Step S725, based on the inconsistent test coverage data group list, deleting the existing invalid test coverage data CovData3, based on the inconsistent test coverage data group list, inserting the newly aggregated test coverage data CovData3.

2. The method of claim 1, wherein the method is a publication-oriented multi-dimensional integrated test quality real-time observation method. The step S714 includes: Step S91, obtaining test coverage data of a reference commit artifact in time and test coverage data of a target commit artifact in time, wherein the test coverage data of the reference commit artifact and the test coverage data of the target commit artifact both include test coverage data of bytecode clustered by class and method; Step S92, comparing and merging each class in the test coverage data of the target commit artifact with the corresponding class in the test coverage data of the reference commit artifact; Step S93, aggregating and outputting the merged test coverage data of the target commit artifact.

3. The method of claim 2, wherein the method is a publication-oriented multi-dimensional integrated test quality real-time observation method, characterized by, The step S92 includes: Step S921, judging whether the current class of the test coverage data of the target commit product exists in the coverage data of the benchmark commit, if not, executing S922, if yes, executing S923; Step S922, taking the target coverage data as the summary benchmark of step S93, skipping the comparison and merging of the current class, and performing the comparison and merging of the next class; Step S923, comparing the class constructor method in the class with the class constructor method of the same class in the benchmark test coverage data, judging whether there is a change, if yes, executing S922, if not, executing S924; Step S924, performing an iteration operation on the test coverage data of each method in the test coverage data of the current class, and performing the comparison and merging of the bytecode of each method coverage data.

4. The method of claim 3, wherein the method is a publication-oriented multi-dimensional integrated test quality real-time observation method, characterized by, The step S924 includes: Step S9241, judging whether the current method exists in the coverage data of the benchmark commit, if not, executing S9242, if yes, executing S9243; Step S9242, taking the target coverage data as the summary benchmark of step S93, skipping the comparison and merging of the current method, and performing the comparison and merging of the next class; Step S9243, comparing the test coverage data of the bytecode of the benchmark method and the target method coverage data, if there is a partial overlap, executing S9244, if there is no overlap, executing S9242; Step S9244, judging whether the benchmark method and the target method overlap decomposition are located in the if instruction block, if the boundary is located in the if instruction block, executing S9245, if not, executing S9246; Step S9245, removing the covered data in the if instruction block in the benchmark, judging whether it is an if instruction block according to the content, if yes, setting all isCover fields in the instruction block as uncovered, and executing S9246; Step S9246, merging the covered data in the benchmark method to the target method, if the instruction isCover field in the benchmark is covered, modifying the coverage mark in the target method to uncovered, and taking the coverage data of the calculated method as the summary benchmark in step S93.

5. The method of claim 1, wherein the method is a publication-oriented multi-dimensional integrated test quality real-time observation method, characterized by, The step S8 includes: Step S81, generating a full-coverage report and an index, using a test coverage data analysis tool to generate a full-coverage report, and parsing to obtain line coverage and branch coverage, and storing in a test coverage index and report data structure; Step S82, generating an incremental coverage report and an index, according to the comparison result of the code of the component under the release and the code in the baseline version, parsing the full-coverage report generated in S81, screening the test coverage of the changed line number, and summarizing as an incremental coverage report and an index; Step S83, generating an interface coverage report and an index.

6. The method of claim 5, wherein the method is a publication-oriented multi-dimensional integrated test quality real-time observation method, characterized by, The step S83 includes: Step S831, parsing a bytecode file to obtain a method list, and generating a method list from the bytecode file of the product; Step S832, screening private methods and initialization methods; Step S833, identifying interfaces and methods belonging to interfaces according to configurations; Step S834, generating a coverage method list according to test coverage data; Step S835, based on the identified identification interface and the method and coverage method list belonging to the interface, a list of covered and uncovered interfaces is generated; Step S836, an interface coverage report is generated.

7. A publication-oriented multi-dimensional integrated test quality real-time observation system, characterized by, The system for realizing the release-oriented multi-dimension integrated test quality real-time observation method comprises: a test coverage data collection module, configured to configure an integrated test environment, create a release, build and deploy a to-be-tested product, add a test coverage data probe in a process of the to-be-tested product, and perform real-time testing on the to-be-tested product based on the integrated test environment, wherein the test coverage data probe is used to acquire test coverage data in the testing; a test coverage data merging module, configured to realize merging of test coverage data within a commit and across commits; a test coverage data aggregation calculation module, configured to realize one-stage aggregation of test coverage data of the to-be-tested product across processes in the testing by calling the test coverage data merging module, realize two-stage aggregation of test coverage data across versions based on the one-stage aggregated test coverage data, and realize three-stage aggregation of test coverage data across test environments based on the two-stage aggregated test coverage data; a report and index module, configured to acquire a full-amount coverage report, an incremental coverage report and an interface coverage report based on the three-stage aggregated test coverage data.

Images