Test tool result fusion method based on text embedding

Abstract

The invention relates to a test tool result fusion method based on text embedding, and belongs to the field of software testing. The method comprises four parts of standard rule set construction, multi-source polymorphic data analysis, rule mapping and result fusion. The method comprises the following steps: firstly, based on standards such as national standard and national military standard or independently establishing a standard rule set, vectorizing rule description of rules in the rule set through a text embedding algorithm, and storing the vectorized rule description into a vector database; manually extracting multi-source and multi-state rule sets of different test tools, and converting the rule sets into intermediate rule sets with the same format as the standard rule set; according to rule mapping, an intermediate rule set of the test tool is mapped to a standard rule set; and finally, result fusion ensures that the output result of the test tool can be accurately butted to a standard rule set according to a set rule mapping mechanism. According to the method, the software testing efficiency and accuracy can be remarkably improved, and the testing result difference caused by standard inconsistency is reduced.

Description

Technical Field

[0001] The invention belongs to the field of software testing, and in particular relates to a test tool result fusion method based on text embedding. Background Art

[0002] With the rapid development of software development, software code quality and security have become the focus of enterprises and developers. To ensure the reliability of software products, developers usually use a variety of automated testing tools, such as static code analysis tools (such as klockwork, Testbed, Checkmarx, Fortify, etc.) and dynamic testing tools (such as Appscan). However, due to the lack of unified standards, these tools generate test results in different formats. At the same time, the capabilities of each tool are different when detecting different rules and different codes. It is necessary to confirm all the different test results generated by different tools, which makes the analysis and integration of multi-source test data complicated and inefficient. The existing testing process usually relies on manual analysis of the test results of different tools, which is not only time-consuming and labor-intensive, but also prone to errors such as omissions in human confirmation of results, resulting in some potential software defects or security vulnerabilities not being discovered and repaired in time. In order to solve these problems, there is an urgent need for an intelligent method that can automatically parse and integrate heterogeneous test results to improve the efficiency and accuracy of test result processing and reduce the workload of developers.

[0003] Existing test result fusion methods often rely on manual configuration of rules and cannot flexibly respond to changes in new tools and new rules. In addition, due to the heterogeneity of test tools, the results of each tool may have large semantic differences, and traditional rule matching methods are difficult to accurately capture these differences. Therefore, there is an urgent need for an intelligent fusion method based on deep learning that can automatically parse, map, and fuse results from different test tools, reduce manual intervention, and improve the consistency and automation level of test result processing. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] The technical problem to be solved by the present invention is how to provide a test tool result fusion method based on text embedding to solve the problem of automatic parsing and integration of heterogeneous test data generated by various automated test tools.

[0006] (II) Technical solution

[0007] In order to solve the above technical problems, the present invention proposes a test tool result fusion method based on text embedding, which comprises the following steps:

[0008] S1. Build a standard rule set: Build a standard rule set R0 based on the national standard and the national military standard. The rule set contains several core rules for subsequent mapping and fusion processes.

[0009] S2. Multi-source and multi-state data analysis: Different test tools have their own rule sets, which may adopt different formats and standards. First, the rule sets of different test tools are obtained through manual extraction, and then the multi-source and multi-state rule sets are converted into intermediate rule sets with the same format as the rules in the standard rule set R0 through multi-source and multi-state data analysis.

[0010] S3, rule mapping: map the intermediate rule sets of different testing tools to the standard rule set R0 through the text embedding model;

[0011] S4. Result fusion: Through predefined rule mapping, the output results of each test tool are connected with the standard rule set to ensure accurate summary of the output results; in this process, all results mapped to the same standard rules will be processed uniformly to eliminate differences between different tools, achieve consistency and comparability of test results, and finally generate a complete and standardized test report.

[0012] (III) Beneficial effects

[0013] The present invention proposes a test tool result fusion method based on text embedding. The test tool result fusion method based on text embedding of the present invention has the following characteristics:

[0014] First, by constructing the standard rule set R0, the uniformity and standardization of the rules are ensured, so that the test tool results from different sources can be effectively integrated through a standardized framework.

[0015] Secondly, the introduction of multi-source and polymorphic data analysis can process a variety of data formats, achieve flexibility and adaptability, and reduce compatibility issues caused by differences in data formats.

[0016] In addition, rule mapping is proposed, which uses text embedding technology and cosine similarity to improve the accuracy and efficiency of rule mapping and ensure the intelligence and automation of the fusion process. This series of advantages makes this method have significant advantages in achieving efficient integration of test results and improving test quality.

[0017] Through the above characteristics, the test tool result fusion method based on text embedding of the present invention not only improves the standardization and consistency of software test results, but also significantly improves the efficiency and accuracy of test result processing through an intelligent processing mechanism, providing an innovative solution for the software testing field. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1It is a structural diagram of the method described in the present invention;

[0019] Figure 2 is a flow chart of the method of the present invention;

[0020] Figure 3 is a rule mapping diagram in the method of the present invention;

[0021] Figure 4 It is a text embedding technology based on the CLIP pre-training model in the method described in the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0023] The present invention relates to the field of software testing, and in particular to a test tool result intelligent fusion method based on text embedding, aiming to improve software testing efficiency and the accuracy of result fusion.

[0024] The present invention provides a test tool result fusion method based on text embedding, aiming to solve the problem of automatic parsing and integration of heterogeneous test data generated by multiple automated test tools.

[0025] This method constructs a benchmark rule set R0 based on the national standard and the national military standard, including several core rules R01, R02, etc., and uses the CLIP pre-training model to generate text vectors for each rule description in the rule set and store them in the vector database. Through multi-source and polymorphic data analysis, the rule sets of different test tools are converted into the same unified format as the standard rule set to obtain the rules and rule descriptions in the results. This method also proposes rule mapping, mapping the rules of the test results to the rules of the standard rule set R0, specifically implementing the conversion of the rule description of the test results into vectors, matching them with the rules of the existing standard rule set R0 through vector cosine similarity, and manually verifying the rules whose number of detected problems is too different from that of other tools, and integrating the rest into the standard set. For the unmatched rules, they are manually verified and expanded to the standard set.

[0026] The present invention significantly improves the efficiency and accuracy of test result processing, helps developers quickly locate and resolve security vulnerabilities and quality issues in software, and provides an intelligent test result fusion solution.

[0027] Embodiment 1:

[0028] The architecture diagram of the test tool result fusion method based on text embedding is as follows: Figure 1 As shown in the flow chart, Figure 2As shown, the implementation steps include the following key parts:

[0029] S1. Build a standard rule set:

[0030] Building a standard rule set is a key step in achieving the fusion of results from different testing tools. In order to ensure the accuracy and consistency of the fusion, a benchmark rule set R0 is first built based on relevant national standards and industry specifications. This rule set contains several core rules that cover various common vulnerability detection and security audit items. The definition of each rule has been strictly verified to ensure that its content is consistent with the provisions of the national standard and the national military standard, and has sufficient applicability, such as:

[0031]

[0032] During the construction process, not only will the specific content of each rule be described in detail, but its scope of application, application conditions, and test environment requirements will also be clearly defined. At the same time, when building a standard rule set, future expansion needs will be taken into consideration, and space for rule expansion will be reserved so that the rule set can support emerging testing tools and new security standards.

[0033] S2. Multi-source and multi-state data analysis:

[0034] Different testing tools may have different rule set formats, which may include XML, CSV, JSON, or log file formats, such as:

[0035]

[0036] In order to achieve seamless connection between heterogeneous data sources generated by different testing tools, an intermediate rule format was constructed. Its core function is to map source data in various formats - including XML, CSV, JSON or log files - into a unified intermediate rule format.

[0037] First, identify the format of the input data, which is the starting point of the mapping process. Whether it is XML, CSV or JSON, it can accurately identify and prepare the corresponding processing flow. The plug-in architecture can flexibly access and parse test data in various formats. For common data formats such as XML and CSV, standard parsing plug-ins are built in. For special or customized data formats, developers can develop new plug-ins to expand support for new formats. After identifying and parsing the source data, the data is converted into a standardized intermediate rule format, which ensures that data from different tools can be understood and processed under a unified framework. This format facilitates subsequent processing and fusion because it provides a common, standardized representation for all source data, while retaining the rule ID and rule description fields of the source data for more accurate mapping in subsequent steps. For example:

[0038]

[0039] This format facilitates subsequent processing and fusion because it provides a common, standardized representation for all source data. By mapping the source data to the intermediate rule format, the consistency and compatibility of data from different test tools during further processing is ensured. This not only improves the efficiency of data processing, but also enhances the quality of data fusion. It can achieve efficient integration of data from different test tools and lay a solid foundation for subsequent analysis and processing.

[0040] S3. Rule mapping: Rule mapping is the core link of the method of the present invention. Figure 3 As shown, the purpose is to map the rules generated by different test tools into the rules of a unified standard rule set R0. This process ensures that test results from different sources can be compared and analyzed under the same standard, thereby improving the consistency and comparability of test results.

[0041] After the multi-source polymorphic data is parsed and converted into a standardized format, the intelligent mapping algorithm is used to map the rule descriptions generated by different tools with the rule descriptions in the standard rule set R0 by vectorizing the rule descriptions in the rules generated by different tools. The core of the intelligent mapping algorithm lies in the application of text embedding technology. By introducing text embedding technology based on CLIP pre-trained models, such as Figure 4 As shown in Figure 2, the CLIP pre-trained model has been extensively trained and has strong text understanding and embedding capabilities. The algorithm can convert each rule description into a high-dimensional vector representation. Text embedding technology makes it possible to capture the semantic features of rule descriptions and perform vectorized processing without relying on a specific format.

[0042] In the mapping process, the cosine similarity of the vector generated by each rule is first calculated and compared with the rule vector in the standard rule set. In this way, the mapping rules can be automatically generated based on the semantic similarity scores between the rule descriptions. In order to ensure the accuracy of the mapping, the rules with the highest similarity scores are prioritized for mapping, and a similarity threshold w is set. s , to ensure the quality of the mapping results. For rules that are below the threshold but still need to be mapped, a manual verification function is provided to ensure that the mapping of these rules can accurately reflect the differences and commonalities between different tools.

[0043] Vector retrieval mechanism: In order to further improve the processing efficiency and expansion capabilities, an efficient vector retrieval mechanism is built on the basis of text embedding. Every time a new rule set is parsed and vectors are generated, these vectors are stored in the Milvus vector database. The vector database has the advantages of fast retrieval and high-dimensional space similarity calculation, and can support efficient storage and real-time retrieval of large-scale rule sets. When a new rule set needs to be mapped, the closest standard rule set vector is first found by searching the stored Milvus vector database. This vector retrieval-based mechanism not only improves the speed of rule mapping, but also ensures the efficient fusion of results from different tools. By combining text vectorization and vector retrieval, a large amount of rule data can be processed in a short time, which greatly improves the processing efficiency and adapts to the application scenarios of large-scale data sets and complex rule sets.

[0044] Similarity calculation and mapping rule generation: In the implementation of the intelligent mapping algorithm, similarity calculation plays a vital role. First, we will use the text embedding technology pre-trained by the CLIP model to convert each rule description into a vector form in a high-dimensional space. Then, the system calculates the cosine similarity of these newly generated rule vectors with the vectors stored in the standard rule set. This process allows us to quantitatively evaluate the similarity between different rules. The higher the similarity between the rules, the closer their matching degree is. The system will give priority to the rules with the highest similarity for mapping. This step is to automatically create mapping rules.

[0045] For those rules whose similarity exceeds the preset threshold, the system will automatically create mapping rules and store them in the mapping rule table for subsequent calls and applications. For those rules whose similarity does not reach the threshold but still need to be mapped, the system will mark them as requiring manual review to ensure the accuracy and consistency of the mapping results. This method of combining similarity calculation and mapping rule generation not only improves the efficiency of rule mapping, but also ensures the quality of integration of results from different testing tools. In this way, we can ensure the accuracy and efficiency of rule mapping while maintaining the consistency and reliability of the rule set.

[0046] Mapping rule verification: In order to ensure the accuracy of mapping rules, a mechanism combining automatic and manual verification is designed. After the automatic mapping rules are generated, the logical consistency and data consistency will be automatically verified to ensure that each mapping rule meets expectations. At the same time, for some mapping rules with low similarity or large data differences, they will be automatically marked and notified to the manual verifier for inspection. For example, if a rule R21 is mapped to the standard rule R01, but its number of vulnerabilities or other attributes are significantly different from similar rules from other testing tools, the manual verifier will be prompted to review and adjust the mapping. During the manual verification stage, experts can adjust the mapping rules according to actual conditions to ensure that the generated mapping results can accurately reflect the actual conditions of different tools. Through this automatic and manual verification mechanism, the accuracy of rule mapping is ensured, and data inconsistency caused by rule differences is effectively avoided.

[0047] S4. Result fusion:

[0048] Result fusion uses predefined rule mapping to connect the output results of each test tool with the standard rule set to ensure accurate summary of the output results. In this process, all results mapped to the same standard rules are processed uniformly to eliminate differences between different tools and achieve consistency and comparability of test results. The system stores the summarized results in a unified data structure to facilitate subsequent data analysis and visualization, ensure the transparency and ease of use of test results, and thus improve the overall efficiency and effectiveness of software testing.

[0049] Embodiment 2:

[0050] A test tool result fusion method based on text embedding, comprising:

[0051] S1. Build a standard rule set: Build a standard rule set R0 based on the national standard and the national military standard. The rule set contains several core rules R01, R02, etc., which are used in the subsequent mapping and fusion process;

[0052] S2. Multi-source and multi-state data analysis: Different testing tools have their own rule sets ( These rule sets may adopt different formats and standards. First, the rule sets of different testing tools are obtained by manual extraction ( etc.), through multi-source and multi-state data parsing, the multi-source and multi-state rule sets are converted into intermediate rule sets (R1, R2, etc.) with the same format as the rules in the standard rule set R0;

[0053] S3, rule mapping: Map the intermediate rule sets (R1, R2, etc.) of different testing tools to the standard rule set R0 through the text embedding model.

[0054] S4. Result fusion: Through predefined rule mapping, the output results of each test tool are connected with the standard rule set to ensure accurate summary of the output results. In this process, all results mapped to the same standard rules will be processed uniformly to eliminate the differences between different tools, achieve consistency and comparability of test results, and finally generate a complete and standardized test report.

[0055] Furthermore, the construction standard rule set includes:

[0056] Rule definition: clearly define the specific content of each rule R01, R02, etc., such as:

[0057]

[0058]

[0059] Rule verification: Manually verify each rule to ensure that it complies with national standards and national military standards.

[0060] Furthermore, the construction standard rule set also includes:

[0061] Vector generation: Generate rules R01, R02, …, R0 in the rule set using text embedding technology n The text vector corresponding to the description;

[0062] Establish search: store the rule ID and the corresponding generated text vector into the Milvus vector database to facilitate efficient retrieval of vector data;

[0063] Provide extensions: The rules of different testing tools may contain rules outside the standard rule set, so the constructed rule set R0 also includes extended rule R0 e0 ,R0 e1 ,…,R0 em , which facilitates rule expansion.

[0064] Furthermore, the multi-source polymorphic data analysis includes:

[0065] Heterogeneous data transformation: manually extracted rule sets for different testing tools ( etc.) belong to multi-source polymorphic heterogeneous data, including different formats such as CSV, XML, JSON, etc. Step S2 is responsible for converting the rule sets of different testing tools ( etc.) into the same format as the standard rule set as the intermediate rule set (R1, R2, etc.) for subsequent processing; assuming that the code scanning results generated by the test tool are in the following format:

[0066]

[0067]

[0068] The data format, internal key values ​​and rule descriptions of this format are uncertain, and the intermediate format is to convert the results generated by different test tools into data of the same format. The converted intermediate rule set is as follows:

[0069]

[0070] Furthermore, the text embedding model of step S3 is a CLIP pre-trained model, and the CLIP pre-trained model is used to generate text vectors of test data.

[0071] Further, the following steps are included:

[0072] Similarity calculation: The text vectors of the rule descriptions in the intermediate rule sets (R1, R2, etc.) of different testing tools are calculated using CLIP, and the similarity between the rule description vectors of different rules in the R0 rule set is calculated.

[0073] Rule mapping implementation: According to the similarity calculation results, the rule with the highest similarity is selected and mapped to the standard rule set R0;

[0074] Rule mapping verification: There may be different situations when the rules are mapped to the standard rule set R0. For some situations, manual verification is performed.

[0075] Furthermore, the similarity calculation preferentially uses cosine similarity to match the test result with the rule to ensure the accuracy of rule mapping.

[0076] Furthermore, the step S3 further includes:

[0077] Fusion into the standard set: If a rule such as R21 can be mapped to R01 through the algorithm, it will be mapped to R01 and merged into the standard set;

[0078] Expanded to the standard set: The rule description of a rule such as R31 is lower than the set similarity threshold w after the algorithm. s , and after manual verification will be expanded to the standard set.

[0079] Furthermore, the step S3 further includes:

[0080] If the rule description of a rule, such as R22, can be mapped to R02 through the algorithm, but the number of specific questions in the rule is too different from the rules that can be mapped to R02 by other testing tools, manual verification and mapping will be performed.

[0081] Furthermore, step S4 maps the output results of each test tool to the standard rule set through predefined rule mapping to ensure accurate summary of the results. All results mapped to the same standard rules are processed uniformly to eliminate differences between tools, achieve consistency and comparability of test results, and finally generate a standardized test report. The integrated data is stored in a unified structure to facilitate subsequent analysis and display, and improve the visualization and use value of the results.

[0082] The present invention proposes a method for intelligent fusion of test tool detection results based on text embedding, which belongs to the field of software testing. The method includes four parts: building a standard rule set, multi-source polymorphic data analysis, rule mapping, and result fusion. First, a standard rule set R0 is established based on national standards, national military standards, etc. or independently, and the rule descriptions of the rules in the rule set (R01, R02, etc.) are vectorized through a text embedding algorithm and stored in a vector database. The multi-source polymorphic rule sets ( etc.), and multi-source polymorphic data parsing is responsible for converting these rule sets into intermediate rule sets (R1, R2, etc.) in the same format as the standard rule set R0. Rule mapping maps the intermediate rule sets R1 and R2 of the test tool to the standard rule set R0. Finally, result fusion ensures that the output results of the test tool can be accurately connected to the standard rule set according to the established rule mapping mechanism, and effectively summarizes the detection results of multiple tools, thereby achieving consistency and comparability of the detection results. This method can significantly improve the efficiency and accuracy of software testing, reduce the differences in test results caused by inconsistent standards, and provide a new solution for the field of software testing.

[0083] The test tool result fusion method based on text embedding of the present invention has the following characteristics:

[0084] First, by constructing the standard rule set R0, the uniformity and standardization of the rules are ensured, so that the test tool results from different sources can be effectively integrated through a standardized framework.

[0085] Secondly, the introduction of multi-source and polymorphic data analysis can process a variety of data formats, achieve flexibility and adaptability, and reduce compatibility issues caused by differences in data formats.

[0086] In addition, rule mapping is proposed, which uses text embedding technology and cosine similarity to improve the accuracy and efficiency of rule mapping and ensure the intelligence and automation of the fusion process. This series of advantages makes this method have significant advantages in achieving efficient integration of test results and improving test quality.

[0087] Through the above characteristics, the test tool result fusion method based on text embedding of the present invention not only improves the standardization and consistency of software test results, but also significantly improves the efficiency and accuracy of test result processing through an intelligent processing mechanism, providing an innovative solution for the software testing field.

[0088] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A test tool result fusion method based on text embedding, characterized in that: The method comprises the following steps: S1. Build a standard rule set: Build a standard rule set R0 based on the national standard and the national military standard. The rule set contains several core rules for subsequent mapping and fusion processes. S2. Multi-source and multi-state data analysis: Different test tools have their own rule sets, which may adopt different formats and standards. First, the rule sets of different test tools are obtained through manual extraction, and then the multi-source and multi-state rule sets are converted into intermediate rule sets with the same format as the rules in the standard rule set R0 through multi-source and multi-state data analysis. S3, rule mapping: map the intermediate rule sets of different testing tools to the standard rule set R0 through the text embedding model; S4, Result Fusion: Through predefined rule mapping, the output results of each test tool are connected with the standard rule set to ensure accurate summary of the output results; During this process, all results mapped to the same standard rules will be processed uniformly to eliminate differences between different tools, achieve consistency and comparability of test results, and ultimately generate a complete and standardized test report.

2. The test tool result fusion method based on text embedding according to claim 1, characterized in that: The S1 includes: rule definition and rule verification. The rule definition is used to clearly define the specific content of each rule, and the rule verification is used to manually verify each rule to ensure that it complies with national standards and national military standards.

3. The test tool result fusion method based on text embedding according to claim 2, characterized in that: Said S1 also includes: vector generation, search establishment and extension provision; Vector generation: Use text embedding technology to generate text vectors corresponding to the descriptions of the rules in the rule set; Establish search: store the rule ID and the corresponding generated text vector into the Milvus vector database to facilitate efficient retrieval of vector data; Provide extensions: The rules of different testing tools may contain rules outside the standard rule set, so the constructed rule set R0 also contains extended rules to facilitate rule extension.

4. The test tool result fusion method based on text embedding according to any one of claims 1 to 3, characterized in that: The S2 includes: the rule sets of different testing tools extracted manually belong to multi-source polymorphic heterogeneous data, and contain different formats, including: CSV, XML and JSON. Step S2 is responsible for converting the rule sets of different testing tools into the same format as the standard rule set as an intermediate rule set for subsequent processing.

5. The test tool result fusion method based on text embedding according to claim 4, characterized in that: The text embedding model of S3 is a CLIP pre-trained model, and the CLIP pre-trained model is used to generate text vectors for test data.

6. The test tool result fusion method based on text embedding according to claim 5, characterized in that: The S3 includes: Similarity calculation: The text vectors of the rule descriptions in the intermediate rule sets of different testing tools are calculated using CLIP, and the similarity between the rule description vectors and the rule description vectors of different rules in the R0 rule set is calculated; Rule mapping implementation: According to the similarity calculation results, the rule with the highest similarity is selected and mapped to the standard rule set R0; Rule mapping verification: There may be different situations when the rules are mapped to the standard rule set R0. For some situations, manual verification is performed.

7. The test tool result fusion method based on text embedding according to claim 6, characterized in that: The similarity calculation uses cosine similarity to match the test result with the rule to ensure the accuracy of rule mapping.

8. The test tool result fusion method based on text embedding according to claim 6, characterized in that: The step S3 further comprises: Fusion into the standard set: If the rule description of a rule can be mapped to the standard rule set through the algorithm, it will be mapped and merged into the standard set; Expand to the standard set: The rule description of a rule is lower than the set similarity threshold w after the algorithm. s , and after manual verification will be expanded to the standard set.

9. The test tool result fusion method based on text embedding according to claim 8, characterized in that: The S3 also includes: if the rule description of a rule can be mapped to a standard rule through an algorithm, but the number of specific questions in the rule is too different from the number of questions that can be mapped to the standard rule by other testing tools, manual verification and mapping will be performed.

10. The test tool result fusion method based on text embedding according to claim 6, characterized in that: The S4 integrated data is stored in a unified structure, which is convenient for subsequent analysis and display, and improves the visualization effect and use value of the results.

Images