Software testing method and software testing system

Abstract

The invention provides a software testing method and a software testing system. The software testing method comprises the following steps. And the processor obtains the test case. When the test case has an incremental code, the processor generates a supplementary test case according to the plurality of feature vectors in the vector database and the test case based on a coverage rate of the incremental code. The step of generating the supplementary test case further comprises the following steps. The processor extracts key information of a test case to obtain a plurality of first data vector units. The processor performs a similarity search according to the plurality of first data vector units and the plurality of feature vectors to obtain a plurality of second data vector units. The processor combines the plurality of second data vector units to generate a supplementary test case, thereby improving the performance of the test operation based on the incremental code.

Description

Technical Field

[0001] The invention relates to a testing method, in particular to a software testing method and a software testing system applied to software. Background Art

[0002] During the process of software development or application, developers can modify the code in the software to improve the software's functions. Generally speaking, the software testing system can generate test cases and test the newly added or modified code (i.e., incremental code) therein. In this way, the software testing system can evaluate the scope and effect of the incremental code on the software based on the test results of the test cases.

[0003] However, as software requirements and versions change, the code base used by the software contains a large amount of redundant code. In addition, the current software testing system relies on developers to judge the impact of incremental code based on their own experience, and lacks a unified testing standard. As a result, the current software testing system is not only unable to achieve full coverage and automated testing based on incremental code, but also unable to achieve accurate testing, resulting in low efficiency of testing operations. Summary of the invention

[0004] The present invention is directed to a software testing method capable of improving the efficiency of testing operations based on incremental codes.

[0005] According to an embodiment of the present invention, the software testing method of the present invention includes the following steps. The processor obtains a test case. When the test case has incremental code, the processor generates a supplementary test case based on the coverage of the incremental code according to multiple feature vectors in the vector database and the test case. The aforementioned steps for generating supplementary test cases also include the following steps. The processor extracts key information of the test case to obtain multiple first data vector units. The processor performs a similarity search based on the multiple first data vector units and the multiple feature vectors to obtain multiple second data vector units. The processor combines the multiple second data vector units to generate a supplementary test case.

[0006] According to an embodiment of the present invention, the software testing system of the present invention comprises a memory and a processor. The memory is used to store a vector database. The processor is coupled to the memory. The processor is used to execute the software testing method as described above.

[0007] Based on the above, the software testing method and software testing system of the present invention obtain multiple second data vector units corresponding to the test case through the processor, and can break up the vector data corresponding to the test case, so as to automatically collect all the code logic and data to be tested. By combining these second data vector units through the processor, the software testing system can automatically present the test case in a new combination. In this way, the software testing system can automatically complete the full coverage test to achieve accurate testing, thereby improving the efficiency of the test operation.

[0008] In order to make the above features and advantages of the present invention more clearly understood, embodiments are given below with reference to the accompanying drawings for detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram of a software testing system according to an embodiment of the present invention;

[0010] Figure 2 is a flowchart of a software testing method according to an embodiment of the present invention;

[0011] Figure 3 is a block diagram of a software testing system according to another embodiment of the present invention;

[0012] FIG. 4A to FIG. 4B The present invention Figure 3 An operation diagram of the software testing system of an embodiment;

[0013] Figure 5 The present invention Figure 3 A partial flow chart of a software testing system of an embodiment;

[0014] Figure 6 The present invention Figure 3 A partial flow chart of a software testing system of an embodiment;

[0015] Figure 7 is a flowchart of a software testing method according to another embodiment of the present invention.

[0016] Description of Reference Numerals

[0017] 100, 300: software testing system;

[0018] 110, 310: processor;

[0019] 120, 320: memory;

[0020] 121, 321: vector database;

[0021] 322: test case database;

[0022] 324: Quality database;

[0023] 325: configuration table;

[0024] 331: Relationship collection module;

[0025] 332: code mapping module;

[0026] 333: Difference detection module;

[0027] 334: Use case generation module;

[0028] 335: Use case execution module;

[0029] 336: result analysis module;

[0030] 340: Large model;

[0031] 400: System under test;

[0032] 410: Relationship system;

[0033] 420: Recommendation system;

[0034] 430: Monitoring system;

[0035] B340: Box;

[0036] EV: eigenvector;

[0037] S210-S220, S221-S223, S411-S414, S421-S425, A, S430, S441-S445, S451-S459, S510-550, S610-S660, S710-S790: steps;

[0038] TS: test case;

[0039] TS': Supplementary test cases;

[0040] UR1: user;

[0041] UR2: Tester. DETAILED DESCRIPTION

[0042] Reference will now be made in detail to exemplary embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Whenever possible, the same reference numerals are used in the drawings and the description to refer to the same or like parts.

[0043] Figure 1 FIG. 1 is a block diagram of a software testing system according to an embodiment of the present invention. Figure 1The software testing system 100 is suitable for software testing and is used to achieve automated full coverage testing and precision testing. A user can operate an electronic device to call the software testing system 100 through an application programming interface (API). The electronic device can be, for example, a mobile phone, a tablet computer, a laptop computer, a desktop computer, or other electronic devices.

[0044] In this embodiment, the software testing system 100 includes a processor 110 and a memory 120. The processor 110 is coupled to the memory 120. The memory 120 stores a vector database 121. The vector database 121 includes vectorized data, such as a plurality of feature vectors EV. These feature vectors EV indicate various source codes, logic codes, input parameters, and output parameters.

[0045] In this embodiment, the memory 120 also stores computing software and the like for implementing algorithms, programs and data related to the features extraction, combination, and various calculation functions of the present invention. The memory 120 may be, for example, a dynamic random access memory (DRAM), a flash memory, a non-volatile random access memory (NVRAM), or a combination of these memories.

[0046] In this embodiment, the processor 110 accesses the memory 120. The processor 110 may be, for example, a server, a signal converter, a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), or other programmable general-purpose or special-purpose microprocessors (Microprocessors), Digital Signal Processors (DSPs), Programmable Controllers, Application Specific Integrated Circuits (ASICs), Programmable Logic Devices (PLDs) or other similar devices or combinations of these devices, which can load and execute computer program-related firmware or software to implement features extraction, combination, and various calculations.

[0047] Figure 2 FIG. 1 is a flowchart of a software testing method according to an embodiment of the present invention. Figure 1 as well as Figure 2, the software testing system 100 may execute steps S210-S220 and S221-S223. The order of these steps S210-S220 and S221-S223 is only an example and is not limited thereto.

[0048] In step S210, the processor 110 obtains the test case TS. In this embodiment, the processor 110 may call the software system coupled to the software testing system 100 to obtain the test case TS. Alternatively, the processor 110 may access the memory 120 to obtain the temporarily stored test case TS.

[0049] When the acquired test case TS has incremental code, it means that compared with the code of the original software, the test case TS includes newly added, deleted or modified code. At this time, the processor 110 calculates the coverage of the incremental code. The coverage of the incremental code can, for example, indicate the completeness of the test case TS, for example, the code that has not been executed and not covered. The processor 110 continues step S220.

[0050] In step S220, the processor 110 generates a supplementary test case TS' according to the plurality of feature vectors EV and the test case TS based on the coverage of the incremental code. The supplementary test case TS' may be, for example, another test case including the code not covered by the test case TS.

[0051] That is, when the incremental code of the test case TS is not completely covered, the processor 110 accesses the vector database 121 and generates a new test case (ie, a supplementary test case TS′) according to the plurality of feature vectors EV and the test case TS.

[0052] In detail, step S220 includes multiple steps S221 to S223. In step S221, the processor 110 extracts key information of the test case TS to obtain multiple first data vector units. These first data vector units indicate the features of various codes and parameters in the test case TS and are vectorized data.

[0053] In step S222, the processor 110 performs a similarity search based on the plurality of first data vector units and the plurality of feature vectors EV to obtain a plurality of second data vector units. The second data vector units include various codes and parameters similar to the plurality of first data vector units and are vectorized data.

[0054] That is, the processor 110 performs a feature extraction operation on the test case TS, and converts the extracted key information into multiple vectors (i.e., multiple first data vector units). Then, the processor 110 calculates the similarity between these vectors and various known source codes, logic codes, input parameters, and output parameters (i.e., multiple feature vectors EV). The processor 110 selects the most similar ones of the multiple feature vectors EV as multiple second data vector units.

[0055] In step S223, the processor 110 combines the plurality of second data vector units to generate a supplementary test case TS'. That is, the processor 110 collects a plurality of logic codes and a plurality of parameters that are operable by the software testing system 100 and correspond to the test case TS, and reorganizes the logic codes and parameters to generate another test case (i.e., the supplementary test case TS').

[0056] It is worth mentioning that, based on the coverage of the incremental code, the software testing system 100 can screen some feature vectors EV (i.e., multiple second data vector units) corresponding to the test case TS by performing similarity search on the multiple vectors (i.e., multiple first data vector units) and multiple feature vectors EV by the processor 110. Since the multiple second data vector units are various codes and various parameters known and operable by the software testing system 100, the software testing system 100 can generate a full coverage supplementary test case TS' by reorganizing these second data vector units by the processor 110. In this way, the software testing system 100 can automatically complete the test with full coverage to achieve accurate testing, thereby improving the efficiency of the test operation.

[0057] Figure 3 FIG. 1 is a block diagram of a software testing system according to another embodiment of the present invention. Figure 3 The software testing system 300 includes a processor 310 and a memory 320. The memory 320 stores a vector database 321. The vector database 321 includes a plurality of feature vectors EV. The processor 310, the memory 320 and the vector database 321 can refer to the relevant description of the software testing system 100 and be deduced by analogy.

[0058] exist Figure 3 In an embodiment, the software testing system 300 is coupled to multiple systems used by the software. The software testing system 300 is used to call these systems to cooperate with these systems. The multiple systems include a relationship system 410, a recommendation system 420, and a monitoring system 430. In some embodiments, the relationship system 410, the recommendation system 420, and the monitoring system 430 are integrated into the software testing system 300.

[0059] In this embodiment, the relational system 410 is used to store multiple use case sets, multiple use case groups, multiple use cases, and multiple source codes through a relational database or a file system (not shown). The recommendation system 420 is used to execute a difference engine module (e.g., a diff engine) to find the differences between multiple source codes. The recommendation system 420 is also used to recommend regression use cases based on the mapping relationship between the source code and the use case. The monitoring system 430 is used to execute a coverage tool (e.g., jacoco) to calculate the code coverage of each use case in the relational system 410.

[0060] In this embodiment, the memory 320 also stores a test case database 322. The test case database 322 includes multiple test cases (e.g., test case TS). The memory 320 also stores multiple modules and a large model 340. The multiple modules include a relationship acquisition module 331, a code mapping module 332, a difference detection module 333, a use case generation module 334, a use case execution module 335, and a result analysis module 336. These modules 331-336 can be implemented, for example, in firmware or software. The processor 310 is used to execute these modules 331-336 to implement various functions.

[0061] Specifically, the relationship acquisition module 331 is used to collect data in the relationship system 410. The aforementioned data includes various use cases such as manual test cases, automated test cases, performance use cases, and security use cases. In addition, the relationship acquisition module 331 is also used to store the collected data in a relationship database or file system in the relationship system 410. In this way, the relationship system 410 is called by the relationship acquisition module 331, and the processor 310 further processes and analyzes the aforementioned various use cases.

[0062] In this embodiment, the code mapping module 332 is used to associate the data (i.e., use cases) collected by the relationship acquisition module 331 with the source code, and establish a mapping relationship between the two. In this embodiment, the code mapping module 332 is implemented as a reverse engineering tool. The code mapping module 332 can be, for example, an APK decompilation tool corresponding to an Android application, or a disassembly tool corresponding to a Windows program.

[0063] In this embodiment, the difference detection module 333 is used to periodically scan the changes of the source code and automatically update the mapping relationship established by the code mapping module 332. In this embodiment, the difference detection module 333 is implemented by a version control system (eg, Git system).

[0064] That is, when the source code is modified or has new code, the difference detection module 333 automatically identifies the changed part of the source code. The difference detection module 333 adjusts the mapping relationship between the use case and the source code according to the aforementioned part of the source code to update the mapping relationship.

[0065] In this embodiment, the use case generation module 334 is used to automatically generate new test cases (e.g., supplementary test cases TS') according to changes in the source code. In this embodiment, the use case generation module 334 infers possible input points and output points of the program by statically analyzing the structure, variable assignments, and function calls of the changed source code, and generates corresponding test cases accordingly.

[0066] In this embodiment, the test case execution module 335 is used to execute the test cases generated by the test case generation module 334 according to a specific strategy and sequence. The aforementioned strategy and sequence are associated with the simulation of scenarios such as concurrent testing and stress testing.

[0067] In the present embodiment, the result analysis module 336 is used to analyze and evaluate the results of the test case execution, and to determine whether the aforementioned results meet the expected behavior. In the present embodiment, the result analysis module 336 generates the judgment result by comparing the actual output of the test case execution with the expected output. Alternatively, the result analysis module 336 generates the judgment result by statistically analyzing the test coverage and error rate of the test case execution. In this way, the result analysis module 336 evaluates the quality of the test operation according to the judgment result.

[0068] In this embodiment, the large model 340 is implemented as a trained large language model (LLM). The large model 340 is used to generate another use case (e.g., supplementary test case TS') according to the use case (e.g., test case TS) and the vector database 321. The large model 340 can be, for example, a chat generation pre-trained transformer (ChatGPT).

[0069] Figure 4A The present invention Figure 3 Schematic diagram of the operation of the software testing system of the embodiment. Figure 3 as well as Figure 4A , the software testing system 300 and the tested system 400 can cooperate to perform a test operation on the software in the tested system 400. The software testing system 300 can perform steps S411-S414, S421-S425, S430 and step A.

[0070] In this embodiment, the software testing system 300 may perform the test operation and generate the supplementary test case TS' non-automatically through the operation of the user UR1 or the tester UR2. Alternatively, the software testing system 300 may periodically and automatically perform the test operation and generate the supplementary test case TS'.

[0071] In detail, user UR1 can operate the system under test 400, and call the software testing system 300 through the system under test 400 to initiate a test operation. User UR1 dynamically analyzes and adjusts the tool script of the system under test 400 by operating the software testing system 300, and generates a configuration table 325 accordingly. User UR1 manually generates a supplementary test case TS' according to the configuration table 325 by operating the software testing system 300. In addition, user UR1 operates the software testing system 300 to store the configuration table 325 in the quality database 324. The configuration table 325 can be used to store relevant values ​​such as business importance (BIM), historical failure rate (HFM), adjustment factors (WeightFactorCC, WeightFactorBI, WeightFactorHF), etc., wherein the initialization of these values ​​is manually assigned, and the data is subsequently automatically analyzed and the initial values ​​of the data are adjusted through a machine learning model.

[0072] On the other hand, in step S411 , the software testing system 300 accesses the tested system 400 and the test case database 322 therein through the processor 310 to perform automated testing operations on the test cases TS in the test case database 322 .

[0073] In step S412, the processor 310 unifies the test platform for executing the test operation. Based on the test platform, the processor 310 executes the test operation on the test case TS to continue with steps S413-S414. Alternatively, based on the test platform, the processor 310 detects whether the test case TS has been modified and has incremental code to continue with steps S421-S425, S430 and step A, and continues with steps S413-S414.

[0074] Specifically, when the test case TS has an error code and is a failed case, in step S413, the processor 310 executes the test case TS to generate a test report and analyzes the cause of the execution error. The test report also includes the level corresponding to the above cause to indicate the priority of the error.

[0075] In step S414, the software testing system 300 sends a notification message to the tester UR2 according to the level through the processor 310. The notification message can be implemented, for example, in the form of an email or an electronic bulletin. In this way, the tester UR2 obtains the test case TS indicated as failed and the corresponding test report. The tester UR2 generates a supplementary test case TS' by operating the software testing system 300 to access the configuration table 325.

[0076] In this embodiment, based on different levels, the processor 310 sends notification information to the tester UR2 at different frequencies. For example, when the test case TS has the highest priority level (for example, indicating a use case level of the extra-high level P0), the processor 310 sends an email notification to the tester UR2 in real time. When the test case TS has other levels (for example, indicating a use case level of the low level P3), the processor 310 sends an email notification to the tester UR2 once a day.

[0077] When the test case TS has incremental code, in step S421, the processor 310 detects which functions the modified code (i.e., incremental code) involves, and calculates a regression case set according to the aforementioned functions. The regression case set includes one or more regression cases. The regression case can be, for example, a case used to confirm whether the incremental code introduces new errors or causes errors in the original code generation.

[0078] In step S424, the processor 310 executes each regression case in the regression case set to generate a corresponding test report, and filters the corresponding regression case according to the test report indicating an error. The processor 310 stores the filtered regression case as a newly generated test case in the test case database 322.

[0079] In step S422, the processor 310 detects that the code in the incremental code is deleted. In step S425, the processor 310 accesses the test case database 322 to delete the test case corresponding to the code deleted in step S422.

[0080] In step S423, the processor 310 executes the test case TS and detects which codes are not executed in the executed test case TS to detect which codes are not covered by the executed test case TS. In addition, the processor 310 automatically supplements the test case according to the unexecuted code to continue step A, and then generates a supplementary test case TS' in step S430.

[0081] For reference Figure 4B , Figure 4B The present invention Figure 3Operational diagram of the software testing system of the embodiment. When the test case TS has incremental code, and when the coverage of the incremental code indicates that the test case TS has uncovered code, the software testing system 300 can execute the large model 340 through the processor 310 to perform steps S441-S445 and S451-S459 in block B340. The embodiment of FIG. 4 is used to illustrate how the software testing system 300 automatically supplements the test case based on the large model 340 to generate the supplementary test case TS'.

[0082] In steps S441 to S445 , the processor 310 periodically executes a timing task to access the test case database 322 , and establishes and updates a plurality of feature vectors EV in the vector database 321 according to the test cases (eg, test cases TS) therein.

[0083] In step S441, the processor 310 extracts key information of the test case TS, including input parameters, output parameters, call request parameters, return values, call sequence and other information of the test case TS.

[0084] In step S442, the processor 310 breaks up the key information in step S441. The processor 310 uses the broken up key information as the minimum unit, and vectorizes each minimum unit to generate a corresponding feature vector EV. The processor 310 stores these feature vectors EV in the vector database 321 to update the vector database 321. These feature vectors EV indicate various source codes, logic codes, input parameters, and output parameters.

[0085] For example, the processor 310 splits the input parameters of the test case TS into a single parameter value and a single parameter name. The processor 310 splits the output parameters of the test case TS into a single parameter value, a single parameter name, and at least one return value. The processor 310 vectorizes the aforementioned various parameter values, parameter names, and return values ​​to generate multiple feature vectors EV.

[0086] Meanwhile, in step S443, the processor 310 normalizes the key information in step S441. That is, the processor 310 performs a preprocessing operation on the extracted key information. The preprocessing operation includes removing irrelevant information in the key information, unifying the data format of the key information, and processing missing values ​​in the key information.

[0087] In step S444, the processor 310 extracts the feature vector of the key information standardized in step S443 to generate a plurality of first data vector units. The processor 310, for example, executes the word2vec model to perform a vector feature extraction operation on the metadata to convert the standardized key information into computable and structured vector data (i.e., a plurality of first data vector units). The feature vector (i.e., a plurality of first data vector units) indicates various standardized source codes, logic codes, input parameters, and output parameters. The metadata refers to structured information extracted from a data set extracted by an application programming interface (API) and used to describe the data. The metadata may include a variety of information, such as field names (i.e., labels or titles of each data field, data types (i.e., types of each field, such as strings, integers, or floating point numbers, etc.), field lengths / sizes (i.e., the maximum allowed value or number of characters of certain fields), constraints (e.g., whether it can be empty or uniqueness requirements, etc.), and / or relationship information, etc., wherein if there is an association between data sets (such as a relationship between tables), it may also include details of the relationship between foreign keys and other tables.

[0088] In step S445, the processor 310 creates multiple indexes of the feature vectors (i.e., multiple first data vector units) in step S444. The processor 310 allocates these indexes to the multiple first data vector units respectively. In this way, each first data vector unit has its own index as a unique identifier.

[0089] In addition, the processor 310 stores the plurality of first data vector units having the identifiers in the vector database 321. The processor 310 also establishes association relationships between the first data vector units and the metadata.

[0090] In steps S451-S459, the processor 310 starts operating in response to the incremental code of the test case TS and its coverage. The processor 310 executes a single task based on the incremental code of the test case TS and its coverage to generate a supplementary test case TS' according to the test case TS.

[0091] In step S451 , the processor 310 extracts key information of the test case TS.

[0092] In step S452, the processor 310 normalizes the key information in step S451.

[0093] In step S453 , the processor 310 extracts the feature vector of the key information normalized in step S452 to generate a plurality of first data vector units.

[0094] In this embodiment, steps S451 to S453 may refer to the relevant descriptions of steps S441 and S443 to S444 and be deduced by analogy.

[0095] In step S454, the processor 310 accesses the vector database 321 to obtain the latest multiple feature vectors EV. In addition, the processor 310 performs a similarity search (e.g., cosine similarity) based on the multiple first data vector units and the multiple feature vectors EV in step S453, and recommends and obtains multiple vectors similar to the multiple first data vector units from these feature vectors EV. The processor 310 uses these similar multiple vectors as multiple second data vector units.

[0096] Next, the processor 310 determines whether to proceed to step S446 based on the matching degree between the plurality of second data vector units to generate a supplementary test case TS' based on the test case template in the test case database 322 and the incremental code of the test case TS, or to proceed to step S457 to combine the plurality of second data vector units to generate a supplementary test case TS'. In this embodiment, the matching degree indicates the correlation between the plurality of second data vector units, that is, indicates the degree to which the known various source codes, logic codes, input parameters, and output parameters correspond to the shared functions.

[0097] Specifically, in step S455, the processor 310 determines whether the matching degree between the multiple similar vectors (ie, multiple second data vector units) recommended in step S454 is greater than a preset value.

[0098] When the matching degree is greater than a preset value (eg, 70%), it means that in step S454, the plurality of second data vector units indicate the same components and / or codes to implement a shared function. The processor 310 proceeds to step S446.

[0099] In step S446, the processor 310 obtains corresponding metadata from the vector database 321 according to the similar multiple vectors (i.e., multiple second data vector units) recommended in step S454. In addition, the processor 310 also generates a new use case according to the use case template of the metadata. The processor 310 uses the generated use case as a supplementary test case TS' and stores the supplementary test case TS' in the test case database 322.

[0100] On the other hand, when the matching degree is less than or equal to the preset value (eg, 70%), it means that in step S454, the plurality of second data vector units indicate different components and / or codes to implement respective functions. The processor 310 proceeds to step S457.

[0101] In step S457, the processor 310 divides the multiple similar vectors (i.e., multiple second data vector units) recommended in step S454 into at least one group to break up the data, and determines whether the matching degree between the multiple second data vector units in each group is greater than a preset value (e.g., 70%), and then determines whether the matching operation corresponding to the broken up data is successful.

[0102] When the matching of the scattered data is successful, it means that in step S447 at least one group of the second data vector units indicates the same component and / or code to implement the shared function. The processor 310 proceeds to step S458.

[0103] In step S458, the processor 310 reorganizes the test case according to the preset logic using the scattered data (i.e., part of the plurality of second data vector units) indicating successful matching to generate a new test case. The processor 310 uses the generated test case as a supplementary test case TS' and stores the supplementary test case TS' into the test case database 322. The preset logic includes logic such as the combination logic of parameters and the arrangement logic of requests.

[0104] On the other hand, when the matching of the scattered data is unsuccessful, it means that in step S457 each group of the plurality of second data vector units indicates different components and / or codes to implement respective functions. The processor 310 proceeds to step S459.

[0105] In step S459, the processor 310 executes the large model 340 to generate a new test case according to the similar multiple vectors (ie, multiple second data vector units) recommended in step S454. The processor 310 uses the generated test case as a supplementary test case TS' and stores the supplementary test case TS' in the test case database 322.

[0106] Figure 5 The present invention Figure 3 A partial flow chart of a software testing system of an embodiment. Figure 3 as well as Figure 5 The software testing system 300 can execute steps S510 to S550 to illustrate how the processor 310 breaks up the data of the test case TS to obtain corresponding vector data (for example, multiple first data vector units, or multiple feature vectors EV), and further illustrate the implementation details of steps S451 to S453 or S441 to S445.

[0107] In this embodiment, in response to the test case TS with incremental code, the processor 310 executes steps S510 - S550 to obtain a plurality of first data vector units, and updates the vector database 321 accordingly.

[0108] In step S510, the processor 310 accesses the test case database 322 to obtain the test case TS and extracts key information of the test case TS, including input parameters, output parameters, call request parameters, return values, call sequence and other information of the test case TS.

[0109] In step S520, the processor 310 performs data preprocessing operations on the key information to generate key data according to the key information. Specifically, the processor 310 removes irrelevant information in the key information, unifies the data format of the key information, and processes missing values ​​in the key information to obtain key data.

[0110] In step S530, the processor 310 performs a data scattering operation on the key data to split the key data into multiple data units according to multiple identifiers. These data units are multiple parts in the key data, and each has multiple identifiers. Each data unit can be, for example, a single parameter value, a single parameter name, or a single return value of the key data, etc. The minimum unit data. Each data unit has a unique identifier.

[0111] In step S540, the processor 310 vectorizes the plurality of data units to generate a plurality of first data vector units. That is, the processor 310 converts the scattered data (i.e., the plurality of data units) into a vector data format based on a word embedding operation. Alternatively, the processor 310 generates a plurality of first data vector units based on the plurality of data units based on other operations related to converting text data into digital data.

[0112] In step S550, the processor 310 stores the plurality of first data vector units in the vector database 321 to update the vector database 321. Since each first data vector unit has its own identifier, the processor 310 can search (eg, perform a similarity search) in the vector database 321 according to the identifier to obtain the corresponding first data vector unit.

[0113] In this embodiment, based on the current test case database 322, the processor 310 periodically executes steps S510-S550 to obtain multiple feature vectors EV, and updates the vector database 321 accordingly. The details of the aforementioned operation can simulate the multiple feature vectors EV as multiple first data vector units, and refer to the above steps S510-S550 and make analogies.

[0114] Figure 6 The present invention Figure 3 A partial flow chart of a software testing system of an embodiment. Figure 3 as well as Figure 6The software testing system 300 may execute steps S610 to S660 to illustrate how the processor 310 uses the scattered data (ie, part of the plurality of second data vector units) to reorganize the test case to generate the supplementary test case TS', and further illustrate the implementation details of steps S454 to S458.

[0115] In step S610, the processor 310 accesses the test case database 322 to obtain the test case TS with the incremental code, and vectorizes the extracted key information in the test case TS to generate a plurality of first data vector units. That is, for the test case TS, the processor 310 performs a data preprocessing operation and a vectorization operation on the key information of the test case TS, as shown in the description of steps S510 to S540.

[0116] In step S620 , the processor 310 accesses the vector database 321 to obtain a plurality of feature vectors EV, and performs a similarity search (eg, cosine similarity) according to the plurality of first data vector units and the plurality of feature vectors EV to generate a similarity search result.

[0117] In step S630, the processor 310 obtains a plurality of feature vectors EV that are most similar to the query input data (ie, a plurality of first data vector units) as second data vector units according to the similarity search results.

[0118] That is, the processor 310 uses the vectorized query input data (i.e., multiple first data vector units) to perform a similarity search operation in the vector database 321. The processor 310 obtains multiple feature vectors EV (i.e., multiple second data vector units) that are most similar to the multiple first data vector units, respectively, based on the similarity search results.

[0119] In step S640, the processor 310 performs a data reorganization operation to combine a plurality of second data vector units according to a preset logic to obtain a reorganization result. The preset logic includes logic such as a combination logic of parameters and a permutation logic of requests.

[0120] In step S650 , the processor 310 uses the reorganization result as a new test case to generate a supplementary test case TS′. In addition, the processor 310 stores the supplementary test case TS′ in the test case database 322 to update the test case database 322 .

[0121] That is, the processor 310 reassembles the scattered data (ie, the plurality of second data vector units in step S630 ) according to a specific logic, and presents the test case (ie, the supplementary test case TS′) in a new combination.

[0122] In step S660, the processor 310 verifies and adjusts the supplementary test case TS'. In this way, the processor 310 can ensure that the supplementary test case TS' meets the test requirements and functional objectives. In addition, when the verification result indicates that the target is not met, the processor 310 adjusts the supplementary test case TS' according to the multiple feature vectors EV in the vector database 321, and executes step S660 again until the verification result indicates that the target is met.

[0123] Figure 7 FIG. 1 is a flowchart of a software testing method according to another embodiment of the present invention. Figure 3 as well as Figure 7 , the software testing system 300 can execute steps S710 to S790 through the processor 310.

[0124] In step S710, the processor 310 executes the diff engine to obtain the incremental code of the test case TS. That is, the processor 310 executes the diff engine to compare the differences between the two versions of the test case TS. The diff engine executes the diff-git command to query the line number corresponding to the aforementioned difference. The diff engine performs a differential analysis operation on the two versions of the test case TS according to the line number. The differential analysis operation includes steps such as cutting files, filtering files, removing redundant data, and parsing line numbers of difference sections.

[0125] In this embodiment, the processor 310 may perform a chain call operation to sequentially call multiple modules or devices (eg, a cutting module, a filtering device, a difference analysis device, and a stubbing device) used by the diff engine. In this way, the processor 310 generates incremental code coverage according to the test case TS.

[0126] In detail, the processor 310 obtains the json file corresponding to the test case TS through the diff engine. The processor 310 cuts the json file corresponding to the test case TS according to the difference between the two versions through the diff engine. The diff engine determines whether the cut json file includes an invalid file.

[0127] When the above judgment result is yes, the diff engine filters the invalid files in the cut json file to generate the file to be analyzed. On the other hand, when the above judgment result is no, the diff engine uses the cut json file as the file to be analyzed. Then, the diff engine obtains the change details information in the file to be analyzed. The diff engine analyzes the changed line number according to the change details information to calculate the coverage of the incremental code.

[0128] In this embodiment, the json file includes one or more changed line numbers. The diff engine performs an abstract syntax tree (AST) operation according to the changed line numbers to instantiate the json file into a specific method in the java file.

[0129] For example, the diff engine obtains method-level difference information from a Java file. The diff engine customizes multiple fields of a method to form a specific method. These fields include the start line (startLine), the end line (endLine), the change marker (isChange), the request interface (requestmapping), the changed method (requestMethod), the method parameters (paramList), and the change person (changePerson).

[0130] In step S720, the processor 310 performs an instrumentation operation on the code of the test case TS. The processor 310 may, for example, perform an on-the-fly instrumentation operation.

[0131] That is, the processor 310 executes the Java Virtual Machine (JVM) to specify a specific jar file through a program (e.g., Java Agent), thereby starting the plugged agent. The processor 310 also uses the Java class loader to determine whether the code includes a conversion and modification class file before loading a class in the agent, and inserts the statistical code into the class file. In this way, the analysis operation of the incremental code coverage can be completed during the Java virtual machine's execution of the code.

[0132] In step S730, the processor 310 isolates the incremental code according to the coverage of the incremental code. The processor 310, for example, adds a label of a markup box in the incremental code through traffic, or adds a custom field or specific parameter in the HTTP request header to implement intelligent isolation operation of the incremental code. In this way, the processor 310 distinguishes the code coverage generated by other special projects such as manual test cases, automated test cases, or traffic playback cases, so that the test operations of different cases do not interfere with each other.

[0133] In step S740 , the processor 310 applies data-driven and automated techniques to the aforementioned steps S710 - S730 to improve the efficiency of the test operation and the coverage of the code.

[0134] In step S750 , the processor 310 generates an incremental code coverage report according to the operation results of steps S710 to S740 .

[0135] In step S760, the processor 310 executes the large model 340 to intelligently generate a supplementary test case TS'. The operation details of step S760 can be found in Figure 4B Description of the embodiments.

[0136] That is, based on various applications of model, data, natural language processing, reinforcement learning, and code analysis operations, the processor 310 executes the large model 340 to generate one or more strategies according to the test case TS and multiple feature vectors EV. In various strategies, the processor 310 executes the large model 340 to analyze the acquired incremental code based on the association between the test case TS and the code therein, and automatically generates one or more test cases as supplementary test case TS'.

[0137] In addition, the processor 310 automatically executes each supplementary test case TS'. When the executed supplementary test case TS' fails for multiple consecutive times (for example, 3 times), the processor 310 automatically sends a corresponding test report notification message to the tester.

[0138] In step S770, the processor 310 performs continuous integration (CI) / continuous deployment (CD) operations to achieve various use cases of automated integration, testing, and final deployment of software applications.

[0139] In step S780, the processor 310 sorts the uncovered code included in the test case TS according to multiple factors to define the use case level. Specifically, the processor 310 collects source data from other application systems such as code bases, continuous integration systems, and defect tracking systems. The processor 310 calculates a comprehensive score based on the source data to divide the use case level.

[0140] In this embodiment, the source data includes the number of lines of code used by the software, code complexity (e.g., cyclomatic complexity), business importance, and historical failure rate, and is represented by a plurality of parameters R1 to R4, respectively. Business importance can be evaluated, for example, based on business requirements and priorities. Historical failure rate can be evaluated, for example, based on the number of failures that occurred in the past.

[0141] For example, the processor 310 calculates the number of uncovered code lines in each method corresponding to the uncovered code to generate a calculation result, and represents the calculation result as a parameter R1.

[0142] In this embodiment, the processor 310 also configures corresponding weight values ​​for multiple data such as code complexity (for example, cyclomatic complexity), business importance, and historical failure rate according to the uncovered code, and represents them with multiple code weights W2 to W4 respectively.

[0143] For example, the processor 310 assigns a corresponding weight W2 to the code complexity of each method corresponding to the uncovered code. The higher the code complexity (i.e., parameter R2), the larger the weight W2. The processor 310 assigns a corresponding weight W3 to the degree of impact of each method corresponding to the uncovered code on the business. The higher the business importance (i.e., parameter R3), the larger the weight value W3. The processor 310 assigns a corresponding weight W4 to the number of failures that occurred in the past to each method corresponding to the uncovered code. The higher the failure rate (i.e., parameter R4), the larger the weight W4.

[0144] Continuing the above description, the processor 310 calculates the weighted total value of the uncovered code as a comprehensive score according to the multiple code weights W2-W4, and divides the test case TS into a use case level according to the comprehensive score. The use case level includes multiple use case levels P0-P3 to indicate that the test case TS is a very high level P0, a high level P1, a medium level P2, and a low level P3, respectively.

[0145] In this embodiment, the multiple code weights W2-W4 include a code complexity weight W2, a business importance weight W3, and a historical failure rate weight W4 corresponding to the uncovered codes respectively.

[0146] In detail, the processor 310 calculates the code complexity weight W2 of a single method corresponding to the uncovered code, and can be represented by the following formula (1). In formula (1), WCC,M represents the code complexity weight W2, CCM represents the code complexity of this method, MinCC represents the minimum code complexity of all methods, MaxCC represents the maximum code complexity of all methods, and WFCC represents the adjustment factor. The adjustment factor is used to control the proportion of the code complexity weight W2 in all weights W2 to W4.

[0147]

[0148] In this embodiment, the processor 310 calculates the business importance weight W3 of a single method corresponding to the uncovered code, and can be represented by the following formula (2). In formula (2), WBI,M represents the business importance weight W3, BIM represents the business importance of this method, MinBI represents the minimum business importance of all methods, MaxBI represents the maximum business importance of all methods, and WFBI represents the adjustment factor of business importance.

[0149]

[0150] In this embodiment, the processor 310 calculates the historical failure rate weight W4 of a single method corresponding to the uncovered code, and can be expressed by the following formula (3). In formula (3), WHF,M represents the historical failure rate weight W4, HFM represents the historical failure rate of this method, MinHF represents the minimum value of the historical failure rate among all methods, MaxHF represents the maximum value of the historical failure rate among all methods, and WFHF represents the adjustment factor of the historical failure rate.

[0151]

[0152] In this embodiment, based on formula (1) to formula (3), the minimum value and minimum value of the code complexity of all methods, the minimum value and minimum value of the business importance of all methods, the minimum value and minimum value of the historical failure rate of all methods, and multiple adjustment factors can be stored in the memory 320, and a configuration table (for example, Figure 4A In the present embodiment, the processor 310 may execute the large model 340 to automatically analyze the data applied by the test case TS according to the number of lines of code covered (i.e., parameter R1), and then generate and adjust the initial values ​​of the aforementioned multiple parameters, the business importance of a single method, and the historical failure rate of a single method.

[0153] In this embodiment, based on formula (1) to formula (3), the processor 310 calculates the weighted total value of the uncovered code, which can be expressed by the following formula (4). In formula (4), Score,M represents the weighted total value (i.e., the comprehensive score), WCC,i represents the code complexity of all methods, WBI,i represents the business importance of all methods, and WCC,i represents the historical failure rate of all methods.

[0154]

[0155] In this embodiment, when the weighted total value (i.e., the comprehensive score) is greater than or equal to the first preset value, the processor 310 defines the use case level of the test case TS as the extra-high level P0. When the weighted total value is less than the first preset value and greater than or equal to the second preset value, the processor 310 defines the use case level of the test case TS as the high level P1. When the weighted total value is less than the second preset value and greater than or equal to the third preset value, the processor 310 defines the use case level of the test case TS as the medium level P2. When the weighted total value is less than the third preset value, the processor 310 defines the use case level of the test case TS as the low level P3.

[0156] In this embodiment, the above-mentioned plurality of preset values ​​may be stored in the memory 320. The processor 310 may execute the large model 340 to automatically analyze the data used by the test case TS according to the number of lines of code covered (ie, the parameter R1), and further define the preset values.

[0157] In step S790, the processor 310 executes at least one strategy to generate a supplementary test case TS' according to the test case database 322, the vector database 321 and the uncovered code according to the use case level corresponding to the weighted total value (ie, the comprehensive score). The operation of generating the supplementary test case TS' can be referred to Figure 4A as well as Figure 4B The relevant description of step S430 in the above description can be deduced by analogy.

[0158] In addition, the processor 310 also sends notification information to the tester according to the use case level. The notification information instructs the software testing system 300 to generate a supplementary test case TS'. The notification information can be implemented in the form of an email or an electronic bulletin, for example.

[0159] In this embodiment, before generating the supplementary test case TS′, the processor 310 first executes the large model 340 to search the test case database 322 to determine whether the test case database 322 includes a test case (eg, a pre-test case and / or a post-test case) with uncovered code.

[0160] When the above judgment result is yes, the processor 310 accesses the corresponding test case as at least a part of the supplementary test case TS' to avoid generating repeated test cases. In addition, the processor 310 can execute the large model 340 to generate the supplementary test case TS' according to the vector database 321 and the uncovered code. On the other hand, when the above judgment result is no, the processor 310 executes the large model 340 to generate the supplementary test case TS' according to the vector database 321 and the uncovered code.

[0161] In this embodiment, when the use case level is an extra high level P0, the processor 310 executes the first strategy and the second strategy. When the use case level is a high level P1, the processor 310 executes the second strategy and the third strategy. When the use case level is a medium level P2, the processor 310 executes the third strategy. When the use case level is a low level P3, the processor 310 executes the fourth strategy.

[0162] In this embodiment, the first strategy indicates that the supplementary test case TS' is generated based on the function. In the first strategy, the processor 310 calculates the cyclomatic complexity of a single method corresponding to the uncovered code. The cyclomatic complexity is equal to the number of nodes determined in the flowchart of the method (for example, represented by parameter P) plus 1. That is, the cyclomatic complexity is equal to P+1.

[0163] Continuing with the above description, the determined number of nodes indicates the nodes used to change the process control flow, such as the number of nodes corresponding to the code of if statements, while loops, etc. Based on the first strategy, the processor 310 generates one or more (ie, 4) supplementary test cases TS' according to the cyclomatic complexity (eg, P+1=4).

[0164] In this embodiment, the second strategy indicates that supplementary test cases TS' are generated based on boundary tests. In the second strategy, the processor 310 supplements at least 4 supplementary test cases TS'. The supplemented test cases respectively indicate multiple boundary values ​​such as slightly less than the minimum value, the minimum value, the maximum value, and slightly greater than the maximum value.

[0165] For example, assume that the uncovered code indicates that a numerical input box requires the user to input an integer between 10 and 50. The processor 310 may generate four supplementary test cases TS'. The first supplementary test case TS' generated by the processor 310 indicates that the input is slightly less than a minimum value (e.g., 9), and indicates whether the system under test (e.g., the software testing system 300 or other application systems called by it) gives a prompt that the input is less than the minimum value.

[0166] In addition, the second supplementary test case TS' generated by the processor 310 indicates that the input is a minimum value (e.g., 10), and indicates whether the system under test accepts the minimum value as input. The third supplementary test case TS' generated by the processor 310 indicates that the input is a maximum value (e.g., 50), and indicates whether the system under test accepts the maximum value as input. The fourth supplementary test case TS' generated by the processor 310 indicates that the input is slightly greater than the maximum value (e.g., 51), and indicates whether the system under test gives a prompt greater than the maximum value.

[0167] In this embodiment, the third strategy indicates that a supplementary test case TS' is generated based on a state test. In the third strategy, the processor 310 sets an initial state, an input or event, an expected result, and a verification step. The initial state indicates the initial state required for the supplementary test case TS'. The input or event indicates the necessary input or trigger event provided by the supplementary test case TS' to perform a state transition. The expected result indicates the target state and any expected output or behavior defined by the supplementary test case TS' after executing the input or event. The verification step indicates whether the supplementary test case TS' has reached the expected state and result of the system under test.

[0168] In this embodiment, the fourth strategy indicates that a supplementary test case TS' is generated based on an abnormal test. In the fourth strategy, the processor 310 writes a supplementary test case TS' to verify the behavior of the supplementary test case TS' under abnormal or erroneous conditions. The aforementioned behavior is, for example, an erroneous behavior such as a parameter error, a format error, and a required item error.

[0169] In summary, the software testing method and software testing system of the present invention can perform similarity search on multiple vectors (i.e., multiple first data vector units) and multiple feature vectors corresponding to the test case through the processor, and can screen some feature vectors as multiple second data vector units. In this way, by reorganizing these second data vector units through the processor, the software testing system can generate full coverage supplementary test cases, thereby automatically completing full coverage testing to achieve accurate testing, thereby improving the efficiency of the test operation.

[0170] In addition, based on the above description, the software testing method and software testing system of the present invention support software regression and reduce the dependence on human resources. The software testing system can create scenarios and data that are difficult to achieve with manual testing, and can execute supplementary test cases that exceed the capabilities of manual testing, thereby significantly improving the coverage of test operations. The software testing system can execute test operations in advance so that test operations and development operations can be executed simultaneously, thereby improving the agility of development projects and reducing the time taken up by the testing phase.

[0171] Based on a full understanding of the requirements of test operations, the corresponding system architecture and implementation details, the software testing system can clarify the scope of test operations by focusing on code-level development. The software testing system can accurately determine the concepts of continuous integration, continuous deployment and continuous testing that need to be integrated into the test operations. At the same time, the software testing system can ensure the standardization, quality and security of the code by using white box and black box testing methods. The aforementioned testing methods include executing unit tests and evaluating test coverage, as well as using automated test operations to complete functional verification.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or perform equivalent replacements on some or all of the technical features therein. However, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A software testing method, characterized in that: include: Get the test case through the processor; as well as By means of the processor, when the test case has incremental code, based on the coverage of the incremental code, a supplementary test case is generated according to a plurality of feature vectors in a vector database and the test case, including: Extracting key information of the test case to obtain a plurality of first data vector units; Performing a similarity search based on the plurality of first data vector units and the plurality of feature vectors to obtain a plurality of second data vector units; and The plurality of second data vector units are combined to generate the supplemental test case.

2. The software testing method according to claim 1, characterized in that: The step of extracting the key information of the test case to obtain the plurality of first data vector units comprises: generating key data according to the key information by the processor; By means of the processor, the key data is divided into a plurality of data units according to a plurality of identifiers, wherein the plurality of data units respectively have the plurality of identifiers; and The plurality of data units are vectorized, by the processor, to generate the plurality of first data vector units.

3. The software testing method according to claim 2, characterized in that: The processor combines the plurality of second data vector units according to a preset logic to generate the supplementary test case, wherein the preset logic includes a combination logic of parameters and a permutation logic of requests.

4. The software testing method according to claim 1, characterized in that: The step of generating the supplementary test case based on the coverage of the incremental code and the plurality of feature vectors in the vector database and the test case also includes: According to the matching degree between the plurality of second data vector units, it is determined whether to combine the plurality of second data vector units to generate the supplementary test case, or to generate the supplementary test case according to the case template in the test case database and the incremental code.

5. The software testing method according to claim 1, characterized in that: Also includes: The processor stores the plurality of first data vector units into the vector database to update the vector database.

6. The software testing method according to claim 1, characterized in that: Also includes: Accessing a test case database to obtain the test case through the processor; as well as The processor stores the supplementary test case in the test case database to update the test case database.

7. The software testing method according to claim 1, characterized in that: Also includes: By means of the processor, a weighted total value of uncovered codes included in the test case is calculated according to a plurality of code weights; as well as By means of the processor, at least one strategy is executed according to the use case level corresponding to the weighted total value to generate the supplementary test case according to the test case database, the vector database and the uncovered code.

8. The software testing method according to claim 7, characterized in that: Also includes: The processor issues notification information according to the use case level to instruct generation of the supplementary test case.

9. The software testing method according to claim 7, characterized in that: The multiple code weights include code complexity weights, business importance weights, and historical failure rate weights corresponding to the uncovered codes respectively.

10. A software testing system, characterized in that: include: Memory, used to store the vector database; as well as The processor is coupled to the memory and is used to execute the software testing method as claimed in claim 1.

Images