Intelligent software quality guarantee method and guarantee system based on knowledge graph and client source code analysis and application

Abstract

The invention discloses an intelligent software quality assurance method based on knowledge graph and customer source code analysis, and particularly relates to a test case generation and automatic repair technology. Comprising the following steps: acquiring a function call graph of a whole software system, and searching all functions calling artificial intelligence components and function call paths; performing defect detection on each function call path and reporting; generating a test case based on a code coverage rate for each function call path, and executing a test; when the passing rate of the test case is lower than a preset threshold value, a software fault is reported; for each software fault, performing cross comparison on the description of the output of the artificial intelligence component and the description of the client source code control flow, if the descriptions are inconsistent, generating a repair suggestion, and modifying software codes to enable the descriptions of the two descriptions to be aligned; and jointly generating a software quality assurance report by using the detected defects, the software faults and the repair suggestions. The invention also discloses a safeguard system for realizing the safeguard method and application.

Description

Technical Field

[0001] The present invention belongs to the field of software engineering and artificial intelligence technology, and relates to an intelligent software quality assurance method, assurance system and application based on knowledge graph and customer source code analysis. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, more and more software uses machine learning components to implement intelligent features. At the same time, the emergence of third-party artificial intelligence libraries has greatly reduced the development cost and difficulty of intelligent software (software applications using artificial intelligence components), allowing developers who lack artificial intelligence knowledge to build intelligent software.

[0003] However, due to the complexity of AI, such as the fact that behavior is affected by training data sets and there may be multiple correct answers to the same question, AI components lack documentation that clearly regulates their behavior. In addition, simple tests are difficult to cover the input space of AI. Therefore, developers, especially non-professional developers, often find it difficult to correctly call AI components, thereby reducing the quality of intelligent software. On the other hand, it is difficult to guarantee that AI algorithms are 100% correct. Even if the software itself does not contain defects, intelligent software may make wrong behaviors in the face of inevitable AI erroneous reasoning.

[0004] In recent years, optimization technologies for artificial intelligence and artificial intelligence frameworks have been widely studied. They respectively improve the reasoning accuracy of artificial intelligence algorithms in various scenarios and provide quality assurance methods for the implementation of artificial intelligence components, thereby providing technical support for the industrialization of artificial intelligence algorithms. However, these methods ignore how to correctly call artificial intelligence components in software. Developers still face many difficulties when integrating systems and find it difficult to truly embed artificial intelligence components into traditional software applications. Therefore, how to ensure the quality of intelligent software needs to be discussed in depth. Summary of the invention

[0005] In order to address the deficiencies in the prior art, the purpose of the present invention is to provide an intelligent software quality assurance method, assurance system and application based on knowledge graph and customer source code analysis. By cross-comparing the knowledge graph and customer source code control flow, external knowledge outside the software including the knowledge graph is used to understand the semantics of artificial intelligence tasks, effectively compensating for the interface semantic problems caused by the behavioral differences between traditional software and artificial intelligence components, and significantly improving the quality of intelligent software through methods such as defect detection and test case generation.

[0006] The core innovations of the present invention are as follows: (1) a defect detection module based on static program analysis is added; (2) the diversity of test case generation is improved by using association and search; and (3) a repair mode is added: preprocessing the input.

[0007] The present invention is achieved through the following technical solutions:

[0008] Step 1: Use a static program analysis algorithm to obtain the function call graph of the entire software system; use a directed graph search algorithm to automatically find all functions that call artificial intelligence components and the call paths involving the functions that call artificial intelligence components; then perform defect detection based on defect patterns on each function call path containing artificial intelligence components. Specifically, use a cross-function program analysis method to analyze all the codes on the function call path, compare them with the pre-built intelligent software defect pattern library, and report the detected defects. The defect patterns include repeatedly passing the same parameters to artificial intelligence components, calling asynchronous functions in a synchronous manner, serially calling remote function calls that can be parallelized, etc.

[0009] In step 1, a static code checker is used to check the functions in the software system to obtain information such as whether there are defects and the defect type.

[0010] The static program analysis algorithm is a method for analyzing program code without running the program, which is used to identify and report potential problems, errors or security vulnerabilities in the source code to help developers improve code quality and reliability. Static program analysis is usually implemented by checking the source code or compiled intermediate representation without actually executing the program.

[0011] The defect pattern-based defect inspection refers to developing test cases or detection techniques based on past experience and known defect patterns to identify new defects that match these patterns.

[0012] The cross-function program analysis method refers to grasping the program's running flow as a whole through the calling relationship between various modules within the program, so as to better understand the program and extract valuable content from it.

[0013] Step 2: Use program control flow analysis and constraint solver to generate test cases based on code coverage for each function call path mentioned in step 1 that calls the artificial intelligence component, so that the generated test cases can maximize the coverage of statements, branches, paths, functions, etc., and execute the test. For each running path involving an artificial intelligence component, several test cases that match a certain type of keyword will be generated. In order to increase the diversity of test cases, the keywords will be associated (up to 5 associated words), and the corresponding image / text input will be obtained from the Internet for each associated word. For example, when it is necessary to generate an image with the keyword being food, you can associate words such as pizza, snacks, and noodles, and search for corresponding images through an image search engine, so that this set of inputs can cover different types of food. When the pass rate of test cases that are successfully executed and meet the expected results is lower than a preset threshold, a software failure is reported;

[0014] Specifically, in step 2, for each function call path, a symbolic execution algorithm is used to analyze the conditional constraints of each specific code execution path, and then a constraint solver is used to solve the expected output of the artificial intelligence module. For each expected output, a pseudo-inverse function constructed by the method of the present invention is used to obtain a set of corresponding artificial intelligence module outputs as test cases through network search and generation algorithms. It should be noted that the corresponding relationship here is the human understanding of intelligent tasks, not the actual execution result of the code. After the test cases are obtained and executed, the pass rate of each type of test case is calculated. When the pass rate is lower than the preset threshold, a fault is reported. If only occasional errors occur, no report is made.

[0015] The preset threshold of the test case pass rate is set manually, and the default value is 0.75. This value depends on the trade-off between the accuracy of the artificial intelligence module itself and the false positives / misses, and is adjusted according to the accuracy of the artificial intelligence module itself and the false positives / misses; generally speaking, the higher the accuracy of the artificial intelligence module itself and the fewer the expected misses, the larger the value is set, and if fewer false positives are expected, a smaller value is set.

[0016] Step 2 uses static program analysis algorithms including symbolic execution, control flow analysis, constraint solving, etc. to check the software system.

[0017] Step 3: For each software failure, we use the open source online knowledge graph wikidata to cross-compare the output of the AI ​​component and the control flow of the customer source code to determine whether the AI ​​and the code use different keywords to describe the same type of object (e.g., they use "apple" and "fruit" to refer to Fuji apples), or the AI ​​and the code describe different objects in the input (e.g., they use "fruit" and "bamboo basket" to summarize the fruit basket photo). When the above two situations are detected, that is, the results of the intelligent module do not meet the software expectations, a repair suggestion will be generated, and then the two descriptions will be aligned by modifying the software code.

[0018] By using the open source online knowledge graph method, it is possible to understand the differences between the behavior of the artificial intelligence module and the expectations of the software system, and to adapt the software to the artificial intelligence module by modifying the software code, thereby performing low-cost automated fault repair.

[0019] In the present invention, the entire process is automated, including steps 1, 2, and 3. The user only needs to select which function to test.

[0020] The open source online knowledge graph wikidata is a ready-made knowledge graph that has been constructed. Each node (page) corresponds to an entity (for example, apple), and each edge (connection between pages) corresponds to a relationship between entities (for example, is a subclass of...). The open source online knowledge graph can be used to find the entity corresponding to the output of the artificial intelligence component and the entity corresponding to the software condition judgment, and then detect whether they are only reachable through the "parent class / child class" or "whole / part" relationship (whether they are only connected through these two types of edges); when the detection result shows that two entities (the entity corresponding to the output of the artificial intelligence component and the entity corresponding to the software condition judgment) are only reachable through the "parent class / child class" or "whole / part" relationship, the two are considered to be related; when the detection result shows that the two entities are not reachable only through the "parent class / child class" or "whole / part" relationship, further detect whether the distance between the two is less than or equal to 2 (that is, whether they can be connected to each other through at most 2 nodes); when the distance is less than or equal to 2, the two are considered to be related; otherwise, they are considered to be unrelated.

[0021] The entity corresponding to the software condition judgment in the present invention is the content in the user source code control flow in step 3, and the entity corresponding to the output of the artificial intelligence component is the content in the output of the artificial intelligence component in step 3;

[0022] The cross comparison means first checking the relationship between the entity corresponding to the output of the artificial intelligence component and the entity corresponding to the software condition judgment. (1) If the two are related, further analyze the specific relationship, including the following four types of relationships: "A is a subclass of B, and it is believed that the two describe the same object", "A is a part of B, and it is believed that the two describe the same object", "A and B are different subclasses under the same parent class, and it is believed that the two describe different objects", "Others, the input is segmented and recalculated to determine whether A and B really appear at the same time (that is, to determine whether the intelligent module has made a mistake. If A and B appear at the same time, it means that the intelligent module has made a mistake"; (2) If the two are not related, check the dimensions described by the two (such as color, shape, and purpose). If the dimensions are the same, it is judged that "A and B are irrelevant, and there is no problem with the intelligent module and code". If they are different, it is judged that "the intelligent module cannot complete the task required by the software";

[0023] In other types of relationships, for example, when AI components and software codes use different keywords to describe the same type of object, you can consider preprocessing the input. Specifically, split the input, use the split inputs as inputs for AI to process, and then merge the processed results of different inputs using ensemble technology.

[0024] The specific implementation method of generating the repair suggestion includes the following: the generation of the repair suggestion is based on rules, that is, trying to correct the software code according to the results of the cross comparison. The specific rules are as follows: "Both describe the same object" -> let AB appear in the if judgment statement; "Both describe different objects" and this situation often occurs -> let AB appear in the if judgment statement; "Both describe different objects" but this situation does not often occur -> increase the input preprocessing part; "The intelligent module cannot complete the task required by the software" -> replace the intelligent module. Then test the corrected code again with the original test case. If the pass rate is improved and no new errors are introduced, then confirm the repair and provide it to the user.

[0025] Step 4: Generate a software quality assurance report using detected defects, software failures, and repair suggestions.

[0026] The present invention also provides a guarantee system for implementing the above-mentioned software quality guarantee method, the guarantee system comprising: a software defect detection module, a test case generation module, and a software repair module;

[0027] The software defect detection module detects defects in the intelligent software code using a static program analysis method based on defect patterns;

[0028] The test case generation module uses symbolic execution, program control flow analysis and constraint solver to generate test cases based on code coverage and execute tests;

[0029] The software repair module generates software code repair suggestions for the test results by using a knowledge graph and a rule-based method.

[0030] An integrated development environment plugin (IDE plugin) is also designed in the present invention, which interacts with developers in a visual way, highlights the faulty code, and displays information such as the fault type, description, and repair suggestions; the interface and text design in this plugin have been verified through empirical research, and even non-expert users can easily use and understand the fault report.

[0031] The present invention also provides the application of the above method or system in intelligent software quality test evaluation, intelligent software automated repair, and artificial intelligence module design optimization.

[0032] The beneficial effects of the present invention include: compared with the traditional software quality assurance methods for general software, the intelligent software quality assurance method proposed by the present invention improves the test coverage. Compared with the existing techniques of coverage-oriented fuzz testing and test case mutation methods with an artificial intelligence training set as the seed set, the coverage can be increased by approximately 43.1% and 25.5% respectively. Whether the customer provides documentation or not, software defects and faults that affect correctness and running defects can be discovered. When the customer does not provide documentation, the control flow of the code can be directly read to analyze what types of inputs the software wants to distinguish. If the customer provides a document written in natural language, then natural language processing methods are needed to obtain the function of the program and then generate test cases. If the user provides a document in a formal language, then the function of the program can be read by a rule-based method. At the same time, the present invention provides a repair algorithm for solving the intelligent software integration problem, which only needs to modify the source code and does not need to retrain the artificial intelligence components, and realizes the improvement of intelligent software quality at low cost. Compared with the time-consuming of several hours to dozens of hours required for retraining neural networks to repair intelligent modules in the existing techniques, the method of the present invention only needs to modify a few lines of source code, takes about a few minutes, and the additional cost can be ignored.

[0033] Compared with the coverage-oriented fuzz testing method, the test cases generated by this method can achieve a higher code coverage (more complete testing), and the generated test cases are real data, close to the real running scenario, rather than randomly generated images / text / audio;

[0034] Compared with the test case mutation method with an artificial intelligence training set as the seed set, the test cases generated by this method can achieve a higher code coverage, and the test cases are not coupled with the intelligent module, can better cover the input space, and thus can better find software fault defects. Description of the Drawings

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 This is a diagram of the architecture of the method of the present invention. DETAILED DESCRIPTION

[0037] The invention is further described in detail with reference to the following specific embodiments and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0038] The present invention provides an intelligent software quality assurance method based on knowledge graph and customer source code analysis, specifically a test case generation and automated repair technology.

[0039] like Figure 1 As shown, the specific steps include:

[0040] Step 1. This implementation uses GitHub open source program code (using artificial intelligence modules to influence program decision / control flow) as the customer code to be tested;

[0041] Step 2: Based on the intelligent software defect pattern library, use the static program analysis method to find out whether the customer code contains similar errors and obtain a defect table (including defect type, defect description, code location, and repair suggestions);

[0042] Step 3: Combined with the open source online knowledge graph, perform program control flow analysis on the customer code, generate test cases for code coverage, and generate multiple test cases for each path; a single test case includes the input required to run the customer code and the corresponding correct execution path;

[0043] Step 4: Execute each test case and record the passing ratio of the test cases corresponding to each path;

[0044] a. If a test case triggers a program crash, report the crash failure;

[0045] b. If a path is never executed, report a dead code fault;

[0046] c. If a path involves an AI component and the corresponding test case pass rate is lower than the threshold, report a functional failure;

[0047] Step 5: For each functional failure, use the knowledge graph to cross-compare the AI ​​component output and the customer source code control flow, deduce the cause of the test case failure, and generate code repair suggestions to obtain a fault table (including fault type, fault description, code location, related test cases, and repair suggestions).

[0048] Example 1

[0049] I. If the AI ​​component and the software code use different keywords to describe the same type of object, and this path code describes the parent class (e.g., AI uses "apple" and the path code uses "fruit"), it is recommended to modify the code to check both the apple and fruit keywords;

[0050] II. If the AI ​​component and the software code use different keywords to describe the same type of object, the path code description granularity is smaller (i.e., the path code describes a subclass of the AI ​​description), and the rest of the code paths do not involve related objects (e.g., the AI ​​uses "fruit", the path code uses "apple", and the rest of the paths do not check fruit and its subclasses), then it is recommended to modify the code to check both apple and fruit keywords;

[0051] III. If the AI ​​component and the software code use different keywords to describe the same type of object, and the code description has no sub-class or parent-class relationship (e.g., AI uses "red" and the code uses "apple"), it is recommended to replace the AI ​​component;

[0052] IV. If the artificial intelligence component and the software code describe different objects in the same input (e.g., artificial intelligence uses "fruit" and the code uses "bamboo basket"), check the frequency of their simultaneous occurrence; if the detection result is that the frequency of the object described by the software code when the object described by the artificial intelligence component appears is greater than or equal to 90%, it is recommended to modify the code to check the bamboo basket and fruit keywords at the same time; if the detection result is that the frequency of the object described by the software code when the object described by the artificial intelligence component appears is less than 90%, it is recommended to preprocess the input; the preprocessing refers to splitting the input, processing it as input for the artificial intelligence component separately, and then merging their results (ensemble technology); the ensemble technology can improve the processing ability of the artificial intelligence model for complex semantic inputs (such as pictures containing many objects, texts involving many topics, etc.), and can more accurately identify elements that account for a small proportion of the original input (such as objects that only occupy a small number of pixels in a large picture, topics that only appear in independent paragraphs in a long text), thereby better meeting the application scenario requirements of intelligent software.

[0053] Step 6: Generate a software quality assurance report using the defect table and fault table, and sort them according to their importance (i.e., the corresponding symptoms are crash, function, dead code, and performance, respectively).

[0054] This embodiment uses test coverage (decision coverage) and discovered code defects / faults as indicators, and uses 82 open source Python codes as test objects to measure the advantages and disadvantages of the detection results of the present invention and the existing software quality assurance method, and the results are shown in Table 1. The present invention discovered 60 defects / faults and successfully repaired 35 of them.

[0055] Table 1 Comparison of results between the method of the present invention and existing software quality methods

[0056]

[0057] Example 2

[0058] This embodiment is described by taking an example of text classification.

[0059] For example, the software wants to check whether an article is about "art", but the artificial intelligence module returns "movies" and "paintings". Because they are parent-child relationships, this embodiment will propose to let the source code check these three categories at the same time.

[0060] Example 3

[0061] This embodiment is an example involving a string, which is as follows:

[0062] For example, the software wants to check whether a picture contains the word "ice cream", but the artificial intelligence module often gives the result of "lce cream" for this type of input. In this embodiment, the code will be modified so that this input can also pass the check.

[0063] Example 4

[0064] This embodiment is an example involving audio recognition, which is as follows:

[0065] For example, the software wants to check whether the user's voice input contains the command "light on", but the artificial intelligence module misidentifies the light tone at the beginning of the sentence and often gives the result of "right on". In this case, the present embodiment will propose to modify the code so that this input can also pass the check.

[0066] Example 5

[0067] This embodiment is an example of an application field involving artificial intelligence module design optimization, as follows:

[0068] For example, the software wants to check whether the picture input by the user contains objects such as "sockets", "switches", and "extension strips" (functional dimension), but the artificial intelligence module can only recognize "plastic" (material dimension). In this case, this embodiment will propose the need to optimize the design of the artificial intelligence module, use the generated test cases to fine-tune the artificial intelligence module, and enhance its recognition ability in the functional dimension.

[0069] Comparative Example

[0070] Coverage-guided fuzzing: The image / text / audio input of intelligent software is treated as a high-dimensional matrix, and floating-point numbers are generated for testing using standard fuzz testing methods. This method has three major disadvantages: 1. The generated image / text / audio input is similar to noise, does not contain valid semantic information, and does not meet the application scenarios of intelligent software; 2. The test coverage is low, and the software code cannot be fully tested; 3. It is impossible to judge the correctness of the software test output, and can only detect faults that will cause the program to crash.

[0071] Test case mutation method using AI training set as seed set: The training data set of AI algorithm is used as seed, and after random modification, it is used as input for testing of intelligent software. This method has two major disadvantages: 1. The semantic information contained in the training data set of AI algorithm is limited, and the data format and size are uniform, which cannot reflect the variable input in the application scenario of intelligent software. Therefore, the test coverage is low and the software code cannot be fully tested; 2. It is impossible to judge the correctness of the software test output, and can only detect faults that will cause the program to crash.

[0072] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the inventive concept, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. An intelligent software quality assurance method based on knowledge graph and customer source code analysis, It is characterized in that The protection method comprises the following steps: Step 1: Obtain the function call graph of the entire software system and find all functions and function call paths that call the AI ​​component; perform defect detection and report on each function call path; Step 2: Generate and execute a test case based on code coverage for each function call path in step 1; when the pass rate of the test case is lower than a preset threshold, report a software fault; Step 3: For each software fault reported in step 2, cross-compare the description of the AI ​​component output and the customer source code control flow. If the descriptions are inconsistent, generate a repair suggestion and align the two descriptions by modifying the software code; Step 4: Generate a software quality assurance report using detected defects, software failures, and repair suggestions.

2. The security method according to claim 1, It is characterized in that In step one, a program analysis method is used to perform static analysis on the entire software system to obtain a function call graph, and then a directed graph search algorithm is used to find all functions that call artificial intelligence components and function call paths; for each function call path that contains an artificial intelligence component, a cross-function program analysis method is used to analyze all the code on the function call path, compare it with a pre-built intelligent software defect pattern library, and report the detected defects.

3. The security method according to claim 1, It is characterized in that In step 2, for each function call path, a symbolic execution algorithm is used to analyze the conditional constraints of each specific code execution path, and then a constraint solver is used to solve the expected output of the artificial intelligence module; for each expected output, a pseudo-inverse function is constructed, and a set of corresponding artificial intelligence module outputs are obtained as test cases through network search and generation algorithms; After obtaining the test cases and executing them, the pass rate of each type of test cases is calculated, and a fault is reported when the pass rate is lower than the preset threshold.

4. The security method according to claim 3, It is characterized in that The preset threshold of the pass rate is set to 0.75 by default, and is adjusted according to the accuracy and false positive / miss negative conditions of the artificial intelligence module itself.

5. The security method according to claim 1, It is characterized in that In step three, open source online knowledge graphs are used to understand the differences between the behavior of the AI ​​module and the expectations of the software system. By modifying the software code, the software is adapted to the AI ​​module, thereby performing low-cost automated fault repair.

6. The security method according to claim 5, It is characterized in that Each node in the open source online knowledge graph corresponds to an entity, and each edge corresponds to a relationship between entities; the open source online knowledge graph can be used to find entities corresponding to the output of artificial intelligence components and entities corresponding to software conditional judgments; detect whether the entities of the output of artificial intelligence components and the entities of software conditional judgments are reachable only through "parent class / child class" or "whole / part" relationships. If they are reachable, the two are considered related; if not, further determine whether the distance between the two is less than or equal to 2. When the distance is less than or equal to 2, the two are considered related; otherwise, they are considered unrelated.

7. The security method according to claim 1, It is characterized in that The cross comparison needs to check the relationship between the entity judged by the software condition and the entity output by the artificial intelligence component. If the two are related, the specific relationship is further analyzed, including the following four types of relationships: "A is a subclass of B, and it is believed that the two describe the same object", "A is a part of B, and it is believed that the two describe the same object", "A and B are different subclasses under the same parent class, and it is believed that the two describe different objects", "Others, the input is split and recalculated to determine whether A and B really appear at the same time"; if the two are not related, check the dimensions of the two descriptions. If the dimensions are the same, it is judged as "A and B are unrelated, and there is no problem with the intelligent module and code"; if they are different, it is judged as "the intelligent module cannot complete the task required by the software." 8. The security method according to claim 1, It is characterized in that In step 4, the software quality assurance report is displayed by designing an integrated development environment plug-in. The integrated development environment plug-in interacts with the developer in a visual manner, highlights the fault code, and displays information including the fault type, description, and repair suggestions.

9. A security system for implementing the security method according to any one of claims 1 to 8, It is characterized in that The guarantee system includes: a software defect detection module, a test case generation module, and a software repair module; The software defect detection module detects defects in the intelligent software code using a static program analysis method based on defect patterns; The test case generation module uses symbolic execution, program control flow analysis and constraint solver to generate test cases based on code coverage and execute tests; The software repair module generates software code repair suggestions based on the test results using the knowledge graph and a rule-based approach.

10. Application of the assurance method according to any one of claims 1 to 8, or the assurance system according to claim 9 in intelligent software quality testing and evaluation, intelligent software automated repair, and artificial intelligence module design optimization.

Images