TypeScript compiler defect detection method based on variation and difference test
Through the method based on variation and differential testing, the defects of the TypeScript compiler are detected, which solves the problem that the existing technology cannot effectively detect TypeScript compiler defects, and realizes efficient automatic detection and type system testing coverage, reducing detection costs.
Patent Information
- Application Number
- CN202411430040.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-14
- Publication Date
- 2024-12-31
AI Technical Summary
The existing JavaScript engine test methods cannot effectively detect the defects of the TypeScript compiler, resulting in the inability to correctly compile the TypeScript code into JavaScript code, affecting the execution of the JavaScript code.
Using a method based on mutation and differential testing, a grammatical program dataset is constructed, a general and unique mutation algorithm is designed, and a random mutation and differential test is performed on the TypeScript compiler. Combined with Node.js execution results analysis, the mutation program is measured using code complexity indicators. The complexity of the cycle detects defects until there is no defect.
It realizes efficient automatic detection of TypeScript compiler defects, improves type system test coverage, reduces human intervention, reduces defect detection costs, and is highly scalable.
Smart Images
Figure CN119226172A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of software testing, and relates to a TypeScript compiler defect detection method based on mutation and differential testing, which can be used to automatically detect defects in a TypeScript compiler. Background Art
[0002] In software engineering, the compiler is an important computer program and a core part of building other software infrastructure. Compiler testing is to find and fix compiler errors and improve compiler reliability and security. Due to the core role of the compiler, when the compiler itself has defects, it will introduce unknown defects in other software, so the importance of compiler testing is self-evident.
[0003] TypeScript is a programming language first released by Microsoft in 2012. It aims to provide strong typing and object-oriented programming capabilities for JavaScript, and provide better maintainability and scalability for large projects. Compared with JavaScript, TypeScript has significant advantages in static type checking, code refactoring, modularization, etc., providing developers with a better development experience and higher code quality. Since its release, TypeScript has experienced rapid development and multiple major updates, expanding from a simple type system to a powerful tool with rich features and supporting modern JavaScript features. The TypeScript compiler is responsible for compiling TypeScript code into standard JavaScript code. In this process, the TypeScript compiler first performs syntax checking on the input TypeScript code and performs type checking through static code analysis to improve code quality. However, the TypeScript compiler will inevitably have defects during the compilation process, resulting in the inability to correctly compile TypeScript code into JavaScript code, which in turn affects the further execution of JavaScript code. Therefore, detecting TypeScript compiler defects is crucial for correctly processing TypeScript code and improving compatibility with JavaScript.
[0004] Since TypeScript is a superset of JavaScript and the two have similar syntax, the current methods for testing TypeScript compilers are directly derived from JavaScript engine testing methods. However, there are significant differences between TypeScript compiler testing and JavaScript engine testing. On the one hand, the testing purposes of the two are different. JavaScript is a dynamically typed language, and the focus of its engine testing is mainly on JavaScript parsers and JIT tools, such as the common JavaScript testing tools Dharma and CodeAlchemist; while TypeScript is a statically typed language with a complex type system, such as generics, union types, intersection types, decorators, and namespaces, and the focus of its compiler testing is type checking, type inference functions, and functionality, that is, whether TypeScript code can be translated into semantically correct JavaScript code. On the other hand, JavaScript engine testing can detect defects by observing execution results or crashes; while the output of the TypeScript compiler is JavaScript code, it is impossible to detect defects by directly observing the results. Due to the difference between TypeScript compiler testing and JavaScript engine testing, the current common methods for JavaScript engine testing cannot effectively detect TypeScript compiler defects. Therefore, it is particularly important to obtain a large number of test programs containing TypeScript static types and to design a method that can test the type checking and compilation functions of the TypeScript compiler. Summary of the invention
[0005] To solve the above problems, the present invention proposes a TypeScript compiler defect detection method based on mutation and differential testing. In order to better test the type checking, type inference and translation functions of TypeScript's type system and its compiler, the present invention first builds a grammar program data set containing general grammar features and unique grammar features with the help of Github open source community and large language model, and designs a set of algorithms including general mutation algorithms and type-specific mutation algorithms to perform random mutations on the programs in the grammar program data set. The obtained mutant program data set is input into different versions of TypeScript compilers for differential testing, and analyzed in combination with the execution results of Node.js, and whether the compiler has defects is determined according to the defined 6 situations. In addition, the present invention also proposes a method for representing code complexity, using 6 different indicators to measure the code complexity of the mutant program, and adding the mutant program with improved code complexity to the grammar program data set to continuously enrich its diversity.
[0006] The technical solution of the present invention is as follows:
[0007] A TypeScript compiler defect detection method based on mutation and differential testing, the steps are as follows:
[0008] Step (1) starts from the perspective of collecting common grammatical features and unique grammatical features, searches and screens existing projects in the Github open source community, proposes prompt words to input into the large language model to generate code, and then performs program syntax and semantic correctness checks to finally obtain the grammatical program dataset D program .
[0009] Step (2) construct the lexical parser TypeScriptLexer and the syntax parser TypeScriptParser corresponding to the TypeScript language, and obtain the syntax program dataset D based on this. program The parse tree corresponding to each program contained in , randomly selects one of the 30 mutation algorithms defined for mutation. Finally, we get the grammar program dataset D program The mutation programs corresponding to the program files in the program are summarized as the mutation program dataset D mutation .
[0010] Step (3) Using the mutation program dataset D mutation The selected TypeScript compiler version is differentially tested, and the test execution results are determined in combination with the six situations defined in this invention. The mutant program that has passed the compilation and executable test is paired with its corresponding original syntax program to obtain a feasible program dataset D feasible , used as a seed program for subsequent evaluation and iteration of new mutations.
[0011] Step (4) Since the TypeScript compiler does not actually run the program, the runtime coverage metric cannot be used. Calculate the feasible program dataset D feasible The code complexity of each mutant program and its corresponding original syntax program in are compared, and the mutant program with higher code complexity is selected to form the seed dataset D seed .
[0012] Step (5) ends here if no TypeScript compiler defect is detected. Otherwise, the seed dataset D seed Merge into the grammar program dataset D program Then enter step (2) and repeat the process until no TypeScript compiler defects are detected.
[0013] Furthermore, step (1) specifically includes the following steps:
[0014] 1-1) From the perspective of collecting common grammatical features, we select all high-quality open source projects whose development language includes Typescript and whose star count ranks in the top n from the GitHub open source community. For each project, we use crawler technology to collect all Typescript program files in the project and construct a common grammar program dataset D. common , each piece of data in the data set corresponds to all the codes of a collected Typescript program file;
[0015] 1-2) From the perspective of collecting specific grammatical features, design a prompt word template to be filled: "Please generate a TypeScript program of no less than m (m>20) lines. The syntax used in the program should contain <unique grammatical type>, and ensure that the code is complex and semantically correct." Use "interface, enumeration, tuple, union type, intersection type, type alias, type assertion, type annotation, never type, optional type and read-only type" to fill in the <unique grammatical type> field respectively, and obtain 11 natural language precise prompts for inputting into the large language model. Enter the selected code respectively. The large language model (such as GPT4-Turbo) is executed several times to construct a specific grammar program dataset D specific , each piece of data in the data set corresponds to all the codes of a generated Typescript program file;
[0016] 1-3) For the two data sets obtained in step 1-1) and step 1-2), the two data sets are merged into the initial test program set D initial , using the static syntax analysis tool tslint and the dynamic semantic checking tool ts-node to check the initial assembly D initial The syntax and semantic correctness of each Typescript program in the dataset D is filtered as long as any one of them fails. program .
[0017] Furthermore, step (2) specifically includes the following steps:
[0018] 2-1) Download the TypeScript grammar files from the antlr / grammars-v4 repository on GitHub, namely TypeScriptLexer.g4 for lexical rules and TypeScriptParser.g4 for grammatical rules, input the two grammar files into the ANTLR4 tool, and build the lexical parser TypeScriptLexer and grammatical parser TypeScriptParser corresponding to the TypeScript language;
[0019] 2-2) For the grammar program dataset Dprogram For each program file in the , use the ANTLR4 tool to parse the program file into a character stream, and input the lexical parser TypeScriptLexer to obtain the corresponding token stream, and then input the obtained token stream into the syntax parser TypeScriptParser to obtain the corresponding parse tree, and finally obtain the parse tree set list L corresponding to each program file tree ;
[0020] 2-3) Define 30 mutation algorithms. Traverse the list of parse trees L tree , a mutation algorithm is randomly selected for each parse tree to mutate, and the mutation program dataset D is obtained mutation The 30 mutation algorithms are as follows:
[0021] a) Randomly delete a subtree
[0022] b) Randomly replace a subtree
[0023] c) Randomly delete a node
[0024] d) Randomly delete an expression
[0025] e) Randomly replace the value of number type variable
[0026] f) Randomly replace the value of a string type variable
[0027] g) True / False swap
[0028] h) Randomly modify an identifier
[0029] i) Add an object destructuring assignment statement
[0030] j) Add an array destructuring assignment statement
[0031] k) Functions and arrow functions can be converted to each other
[0032] l) The subclass object is assigned to the parent class variable
[0033] m) The value of a union type variable is replaced with another optional type value
[0034] n) Randomly select an arithmetic operator and replace it with another arithmetic operator
[0035] o) Randomly select an assignment expression and replace the assignment symbol with another assignment symbol
[0036] p) Randomly select a comparison operator and replace it with another comparison operator
[0037] q) Randomly select a logical operator and replace it with another logical operator
[0038] r) Randomly select a bitwise operator and replace it with another bitwise operator
[0039] s) Randomly select a unary operator and replace it with another unary operator
[0040] t) The condition of the conditional expression is replaced with True / False or the content is left blank
[0041] u) Randomly delete one or more elements in a tuple
[0042] v) Randomly delete a block statement
[0043] w) Randomly delete the contents of the new expression
[0044] x) Randomly delete the body of a function
[0045] y) Expanded arithmetic expressions
[0046] z) Randomly remove type annotations
[0047] aa) Similar API replacement
[0048] bb) Randomly repeat a sentence
[0049] cc) randomly insert a new subtree at a certain position in the parse tree
[0050] dd) Numeric type constants are replaced with boundary values
[0051] Furthermore, step (3) specifically includes the following steps:
[0052] 3-1) Traverse the mutation program data set D one by one mutation For each mutant program, input different versions of the TypeScript compiler to try to compile and compare the output results; if it can be compiled, use Node.js to execute the js program compiled by the compiler of this version and compare the execution results. Define 6 corresponding situations of output and execution results, from which you can know the differential test results, that is, whether the compiler has defects. The specific definitions of the 6 situations are as follows, among which b)-e) four situations all require writing defect reports and submitting them to the developer:
[0053] a) All versions fail to compile with the same syntax error: program syntax or semantics error, compiler behavior is normal
[0054] b) All versions failed to compile with different error messages: the compiler had a bug in the syntax checking phase
[0055] c) Compilation failures and compile successes occur simultaneously in different versions: the compiler has defects in the translation stage
[0056] d) All versions are compiled successfully, but Node.js fails to execute: the compiler has a runtime error, and the generated js program cannot run successfully
[0057] e) All versions are compiled successfully, Node.js is executed successfully but the results are different: there is an inconsistency error in the compiler, and the semantics of the generated js program are not exactly the same
[0058] f) All versions compile successfully and Node.js executes successfully with the same result: the compiler passes the test of this program
[0059] 3-2) Pair the programs that meet the condition f) in step 3-1) with their corresponding original syntax programs to form a feasible program dataset D feasible .
[0060] Furthermore, step (4) specifically includes the following steps:
[0061] 4-1) Define code complexity indicators: number of lines of code, cyclomatic complexity, maximum depth of parse tree, maximum nesting depth, Halstead index, number of classes and methods. The calculation method is as follows:
[0062] a) Number of lines of code, that is, the number of program code lines calculated using the cloc tool
[0063] b) Cyclomatic complexity, which is the number of decision nodes + 1. The number of decision nodes is the sum of all decision nodes in if statements, while statements, case statements, ternary expressions, with statements, for statements, for...in statements, and for...of statements.
[0064] c) The maximum depth of the parse tree, that is, the maximum depth maxDepth calculated by recursively traversing the parse tree. As shown in formula (1), the depth Depth refers to the length of the longest path from the root node of the parse tree to the leaf node, child is the child node of the current parse tree node, currentDepth is the current depth, and calculateDepth is a function used to recursively calculate the depth of the parse tree. The two parameters it receives represent the current parse tree node and the depth of the current node respectively:
[0065] maxDepth=max(maxDepth,calculateDepth(child,currentDepth + 1)) (1)
[0066] d) Maximum nesting depth, that is, the maximum value of the nesting depth of all if statements, while statements, ternary expressions, switch statements, with statements, for statements, for...in statements, and for...of statements, obtained by traversing the subtrees corresponding to these statements.
[0067] e) Halstead index. Traverse the parse tree and count the following four indexes: n1 (number of different operators in the program), n2 (number of different operands in the program), N1 (total number of operators appearing in the program), N2 (total number of operands appearing in the program). The Halstead index is shown in formula (2), where n represents the number of operands and operators, which is the sum of n1 and n2, N represents the number of operands and operators, which is the sum of N1 and N2, and E represents the programming workload, which is calculated as shown in formula (3):
[0068] Halstead_score=0.3*n+0.4*N+0.5*E (2)
[0069]
[0070] f) The number of classes and methods, that is, the sum of the number of all classes and functions on the parse tree.
[0071] 4-2) Traverse the feasible program data set D feasible , input each mutation program and the corresponding original grammar program into the lexical parser TypeScriptLexer in turn to obtain the corresponding token stream, and then input the obtained token stream into the grammar parser TypeScriptParser to obtain the corresponding parse tree. Finally, on this basis, the six code complexity indicators of each mutation program and the original grammar program are calculated respectively;
[0072] 4-3) Compare each pair of mutant programs and original syntax programs. If any of the six code complexity indicators of the mutant program is higher than the corresponding indicator value of the original syntax program, the mutated program code complexity is considered to be higher. The mutant program is added to the seed data set, and the seed data set D is finally obtained. seed .
[0073] Compared with the prior art, the present invention has the following advantages and effects:
[0074] The method of the present invention can automatically detect defects in the TypeScript compiler efficiently. On the basis of collecting high-quality program data of open source projects, the present invention generates a large number of programs containing TypeScript-specific grammatical features by combining a large language model, improves the type complexity of the grammatical program data set, and can more comprehensively cover the type system of the TypeScript compiler; by proposing a series of type-specific mutation algorithms, targeted mutations and tests are performed on TypeScript-specific types, ensuring the test coverage of TypeScript compiler type-related defects; by designing code complexity indicators to guide the mutation process and provide loop end conditions, the test efficiency is improved while ensuring that mutation programs with higher code complexity can be retained as new seed programs, solving the problem that TypeScript cannot obtain guidance information through traditional coverage, and further improving the diversity of test programs; by combining with a large language model, human intervention is reduced, and the cost of defect detection is greatly reduced; in addition, the present invention is also highly scalable, and can meet different user needs by switching different large language models and adding different mutation algorithms. BRIEF DESCRIPTION OF THE DRAWINGS
[0075] Figure 1 It is a flow chart of the TypeScript compiler defect detection method based on mutation and differential testing of the present invention.
[0076] Figure 2 It is a flow subgraph of the phase of obtaining a grammar program data set in the TypeScript compiler defect detection method based on mutation and differential testing of the present invention.
[0077] Figure 3 It is a flow subgraph of the syntax-based program mutation phase in the TypeScript compiler defect detection method based on mutation and differential testing of the present invention.
[0078] Figure 4 It is a flow subgraph of the differential testing phase in the TypeScript compiler defect detection method based on mutation and differential testing of the present invention. DETAILED DESCRIPTION
[0079] The method of the present invention is described in detail below in conjunction with the accompanying drawings, technical solutions and embodiments.
[0080] like Figure 1As shown, the TypeScript compiler defect location method based on mutation and differential testing of the present invention is performed according to the following process: first, a general grammar program data set and a specific grammar program data set are obtained from the GitHub open source community and the large language model respectively, and are summarized into a preliminary test program set, and a grammar program data set is obtained by screening; secondly, the TypeScript grammar file is input into the ANTLR4 tool to obtain a lexical analyzer and a grammar analyzer; the grammar program in the grammar program data set is parsed by the lexical analyzer and the grammar analyzer respectively to generate a parse tree corresponding to the program, and a mutation algorithm is randomly selected to mutate the parse tree to obtain a mutant program; then, a differential test is performed on different versions of the TypeScript compiler, and the js program obtained by successfully compiling is executed with Node.js, and the error information during the compilation and execution process is analyzed to generate a defect report; finally, for the program that passes the test, the code complexity of the program before and after the mutation is calculated, and the mutant program with increased code complexity is added to the grammar program data set to guide the next round of testing.
[0081] The following takes the detection of TypeScript compiler defect #59877 as an example to explain the implementation details of each process in detail. The specific implementation methods are as follows:
[0082] (1) Figure 2 As shown in Figure 1, we obtain a grammatical program dataset. We start from the perspective of collecting common grammatical features and unique grammatical features, search and screen existing projects in the Github open source community, propose prompt words to input into the large language model to generate code, and then perform program syntax and semantic correctness checks to finally obtain the grammatical program dataset D program .
[0083] 1.1. From the perspective of collecting common grammatical features, we select all high-quality open source projects whose development language includes Typescript and whose star count ranks in the top 2000 from the GitHub open source community. For each project, we use crawler technology to collect all Typescript program files in the project and construct a common grammar program dataset D. common , each piece of data in the data set corresponds to all the codes of a collected Typescript program file;
[0084] 1.2. From the perspective of collecting specific grammatical features, design a prompt word template to be filled: "Please generate a TypeScript program of no less than 50 lines. The syntax used in the program should contain <unique grammatical type>, and ensure that the code is complex and semantically correct." Use "interface, enumeration, tuple, union type, intersection type, type alias, type assertion, type annotation, never type, optional type and read-only type" to fill in the <unique grammatical type> field respectively, and obtain 11 natural language precise prompts for inputting into the large language model. Enter the selected code respectively. The large language model (such as GPT4-Turbo) is executed several times to construct a specific grammar program dataset D specific , each piece of data in the data set corresponds to all the codes of a generated Typescript program file;
[0085] 1.3. For the two data sets obtained in step 1.1 and step 1.2, merge the two data sets into the initial test program set D initial , we use the static syntax analysis tool tslint and the dynamic semantic checking tool ts-node to check the syntax and semantic correctness of each Typescript program contained in it. If any of them fails, we will filter them out and finally get the syntax program dataset D program .
[0086] Specifically:
[0087] For step 1.1, we first crawled the top 2000 TypeScript program files ranked by the number of stars from the Github open source community, and collected a total of 498,151 TypeScript programs to form the general syntax program dataset D common .
[0088] Next, for step 1.2, we use 11 target-specific grammar types to fill in the prompt word templates, obtain 11 precise prompts, input them into the large language model, generate 20,000 TypeScript programs, and constitute the specific grammar program dataset D specific Among them, 11 precise tips are as follows:
[0089] a) Please generate a TypeScript program of no less than 50 lines. The syntax used in the program should include interface types and ensure that the code is complex and semantically correct.
[0090] b) Please generate a TypeScript program of no less than 50 lines. The syntax used in the program should include enumeration types and ensure that the code is complex and semantically correct.
[0091] c) Please generate a TypeScript program of no less than 50 lines. The syntax used in the program should include tuple types and ensure that the code is complex and semantically correct.
[0092] d) Please generate a TypeScript program of no less than 50 lines. The syntax used in the program should include union types and ensure that the code is complex and semantically correct.
[0093] e) Please generate a TypeScript program of no less than 50 lines. The syntax used in the program should include cross-types and ensure that the code is complex and semantically correct.
[0094] f) Please generate a TypeScript program of no less than 50 lines. The syntax used in the program should include type aliases and ensure that the code is complex and semantically correct.
[0095] g) Please generate a TypeScript program of no less than 50 lines. The syntax used in the program should include type assertions and ensure that the code is complex and semantically correct.
[0096] h) Please generate a TypeScript program of no less than 50 lines. The syntax used in the program should include type annotations and ensure that the code is complex and semantically correct.
[0097] i) Please generate a TypeScript program of no less than 50 lines. The syntax used in the program should contain the never type and ensure that the code is complex and semantically correct.
[0098] j) Please generate a TypeScript program of no less than 50 lines. The syntax used in the program should include optional types and ensure that the code is complex and semantically correct.
[0099] k) Please generate a TypeScript program of no less than 50 lines. The syntax used in the program should include read-only types and ensure that the code is complex and semantically correct.
[0100] Finally, for step 1.3, firstly, the universal grammar program dataset D common and specific grammar program dataset D specific Merge to get the initial test program set D initial , initial test program data set D initial A total of 518,151 programs are included. initial For each program in , we first use the static syntax analysis tool tslint to check the grammatical validity of the program, and after screening, we get 349453 programs. Then, we use the dynamic semantic checking tool ts-node to further check the semantic correctness, and after screening, we get 38826 grammatically valid programs, which constitute the grammatical program dataset D program .
[0101] (2) Figure 3 As shown, perform grammar-based program mutation. Build the lexical parser TypeScriptLexer and grammar parser TypeScriptParser corresponding to the TypeScript language, and obtain the grammar program dataset D on this basis program The parse tree corresponding to each program contained in , randomly selects one of the 30 mutation algorithms defined for mutation. Finally, we get the grammar program dataset D program The mutation programs corresponding to the program files in the program are summarized as the mutation program dataset D mutation .
[0102] 2.1. Download the TypeScript grammar files from the antlr / grammars-v4 repository on GitHub, namely TypeScriptLexer.g4 for lexical rules and TypeScriptParser.g4 for grammatical rules, input the two grammar files into the ANTLR4 tool, and build the lexical parser TypeScriptLexer and grammatical parser TypeScriptParser corresponding to the TypeScript language;
[0103] 2.2. For the grammar program dataset D program For each program file in the , use the ANTLR4 tool to parse the program file into a character stream, and input the lexical parser TypeScriptLexer to obtain the corresponding token stream, and then input the obtained token stream into the syntax parser TypeScriptParser to obtain the corresponding parse tree, and finally obtain the parse tree set list L corresponding to each program file tree ;
[0104] 2.3. Define 30 mutation algorithms. Traverse the list of parsing trees L tree , a mutation algorithm is randomly selected for each parse tree to mutate, and the mutation program dataset D is obtained mutation The 30 mutation algorithms are as follows:
[0105] a) Randomly delete a subtree
[0106] b) Randomly replace a subtree
[0107] c) Randomly delete a node
[0108] d) Randomly delete an expression
[0109] e) Randomly replace the value of number type variable
[0110] f) Randomly replace the value of a string type variable
[0111] g) True / False swap
[0112] h) Randomly modify an identifier
[0113] i) Add an object destructuring assignment statement
[0114] j) Add an array destructuring assignment statement
[0115] k) Functions and arrow functions can be converted to each other
[0116] l) The subclass object is assigned to the parent class variable
[0117] m) The value of a union type variable is replaced with another optional type value
[0118] n) Randomly select an arithmetic operator and replace it with another arithmetic operator
[0119] o) Randomly select an assignment expression and replace the assignment symbol with another assignment symbol
[0120] p) Randomly select a comparison operator and replace it with another comparison operator
[0121] q) Randomly select a logical operator and replace it with another logical operator
[0122] r) Randomly select a bitwise operator and replace it with another bitwise operator
[0123] s) Randomly select a unary operator and replace it with another unary operator
[0124] t) The condition of the conditional expression is replaced with True / False or the content is left blank
[0125] u) Randomly delete one or more elements in a tuple
[0126] v) Randomly delete a block statement
[0127] w) Randomly delete the contents of the new expression
[0128] x) Randomly delete the body of a function
[0129] y) Expanded arithmetic expressions
[0130] z) Randomly remove type annotations
[0131] aa) Similar API replacement
[0132] bb) Randomly repeat a sentence
[0133] cc) randomly insert a new subtree at a certain position in the parse tree
[0134] dd) Numeric type constants are replaced with boundary values
[0135] Specifically:
[0136] For TypeScript compiler bug #59877, the selected grammar program dataset D program The code related to this defect in each program file is as follows:
[0137] export const cilBlurLinear:string[]=[,];
[0138] Randomly select a mutation algorithm. In this example, the mutation algorithm actually selected is j) in 2.3, that is, add an array deconstruction assignment statement. The code obtained after the mutation of the above code snippet is as follows:
[0139] export const cilBlurLinear:string[]=[,];
[0140] const[,] = cilBlurLinear;
[0141] (3) Figure 4 As shown, differential testing is performed. Using the mutation program dataset D mutation Perform differential testing on the selected TypeScript compiler version, and determine the test execution results based on the six defined cases. Pair the mutant programs that pass the compilation and executable tests with their corresponding original syntax programs to obtain a feasible program dataset D feasible , used as a seed program for subsequent evaluation and iteration of new mutations.
[0142] 3.1. Traverse the mutation program dataset D one by one mutation For each mutant program, three different versions of TypeScript compilers are input to try to compile and compare the output results; if the program can be compiled, the js program compiled by the compiler of this version is executed by Node.js and the execution results are compared. Six corresponding situations of output and execution results are defined, from which the differential test results can be obtained, that is, whether the compiler has defects. The specific definitions of the six situations are as follows, among which four situations b)-e) require writing defect reports and submitting them to the developer:
[0143] a) All three versions fail to compile and report the same syntax error: the program syntax or semantics are wrong, and the compiler behaves normally
[0144] b) All three versions failed to compile and reported different error messages: the compiler had a defect in the syntax checking phase
[0145] c) Compilation failed and succeeded in all three versions: the compiler had a defect in the translation phase
[0146] d) All three versions are compiled successfully, but Node.js fails to execute: the compiler has a runtime error, and the generated js program cannot run successfully
[0147] e) All three versions are compiled successfully, and Node.js is executed successfully, but the results are different: there is an inconsistency error in the compiler, and the semantics of the generated js program are not exactly the same
[0148] f) All three versions are compiled successfully, and Node.js executes successfully with the same results: the compiler passes the test of the program
[0149] 3.2. Pair the programs that meet the condition f) in step 3-1) with their corresponding original syntax programs to form a feasible program dataset D feasible .
[0150] Specifically:
[0151] For the mutant program generated by (2), three different versions of TypeScript (v5.4.5, v5.5.4, and v5.6-Beta) were input for differential testing. The mutant program can be successfully compiled in the three versions of the compiler and the same js program is generated, as shown below:
[0152] "use strict";
[0153] Object.defineProperty(exports,"__esModule",{value:true});
[0154] exports.cilBlurLinear=void 0;
[0155] exports.cilBlurLinear=[,];
[0156] var;
[0157] The above js code was executed with Node.js, but failed. The error message "Variable declaration list cannot be empty" was used to locate the last line "var;" of the js program. By analyzing the TypeScript syntax and compiler behavior, it was found that during the translation process of the TypeScript compiler, the "const" keyword in the mutant program was mistakenly translated into the "let" keyword and no specific variable was declared. This phenomenon is consistent with
[0158] Case d) in 3.1 indicates that there is a runtime error in the TypeScript compiler and the generated js program cannot run successfully. Based on this problem, a defect report is compiled and reported to the developer.
[0159] (4) Since the TypeScript compiler does not actually run the program, the runtime coverage metric cannot be used. feasible The code complexity of each mutant program and its corresponding original syntax program in are compared, and the mutant program with higher code complexity is selected to form the seed dataset D seed .
[0160] 4.1. Define 6 code complexity indicators: number of lines of code, cyclomatic complexity, maximum depth of parse tree, maximum nesting depth, Halstead index, number of classes and methods. The calculation method is as follows:
[0161] g) Number of lines of code, that is, the number of program code lines calculated using the cloc tool
[0162] h) Cyclomatic complexity, which is the number of decision nodes + 1. The number of decision nodes is the sum of all decision nodes in if statements, while statements, case statements, ternary expressions, with statements, for statements, for...in statements, and for...of statements.
[0163] i) Maximum depth of the parse tree, that is, the maximum depth calculated by recursively traversing the parse tree, as shown in formula (4.1)
[0164] As shown, child is the subtree of the current node, and currentDepth is the current depth: maxDepth = max(maxDepth, calculateDepth(child, currentDepth+1))(4.1)
[0165] j) Maximum nesting depth, that is, the maximum value of the nesting depth of all if statements, while statements, ternary expressions, switch statements, with statements, for statements, for...in statements, and for...of statements, obtained by traversing the subtrees corresponding to these statements.
[0166] k) Halstead index. Traverse the parse tree and count the following four indexes: n1 (number of different operators in the program), n2 (number of different operands in the program), N1 (total number of operators appearing in the program), N2 (total number of operands appearing in the program). The Halstead index is shown in formula (4.2): , where n represents the number of operands and operators, which is the sum of n1 and n2, N represents the number of operands and operators, which is the sum of N1 and N2, and E represents the programming workload, which is calculated as shown in formula (4.3):
[0167] Halstead_score=0.3*n+0.4*N+0.5*E(4.2)
[0168]
[0169] l) The number of classes and methods, that is, the sum of the number of all classes and functions on the parse tree.
[0170] 4.2. Traversing the feasible program dataset D feasible , input each mutation program and the corresponding original grammar program into the lexical parser TypeScriptLexer in turn to obtain the corresponding token stream, and then input the obtained token stream into the grammar parser TypeScriptParser to obtain the corresponding parse tree. Finally, on this basis, the six code complexity indicators of each mutation program and the original grammar program are calculated respectively;
[0171] 4.3. Compare each pair of mutant programs and original syntax programs. If any of the six code complexity indicators of the mutant program is higher than the corresponding indicator value of the original syntax program, the mutated program code complexity is considered to be higher. The mutant program is added to the seed data set, and finally the seed data set D is obtained. seed .
[0172] Specifically, when the mutant program generated in step (2) passes both the compilation and execution phases in step (3), the code complexity of the original syntax program and the mutant program is calculated respectively, and the calculation results of the six indicators in the code complexity defined in 4.1 are compared. If one indicator of the mutant program is higher than that of the original syntax program, the mutant program is added to a seed dataset D seed , which can be subsequently added to the grammar program dataset to guide the next round of program mutation.
[0173] The execution process of step (4) is illustrated by taking the code snippet in step (2) and its mutated code snippet as examples. By calculating the code complexity, the six indices of the code complexity of the two programs are (1, 1, 13, 0, 0, 0) and (2, 1, 13, 0, 0, 0). Since the first indicator of the code complexity of the mutated program, the number of lines of code, is greater than the corresponding value of the original code, the code complexity of the mutated program has increased. If the mutated code can simultaneously meet the condition f) in 3.1, it will be added to the seed dataset D. seed Since this example meets the condition d) in 3.1, it is not added to the seed dataset D seed middle.
[0174] (5) If no TypeScript compiler defects are detected, the process ends here. Otherwise, the seed dataset D seed Merge into the grammar program dataset D program Then enter step (2) and repeat the process until no TypeScript compiler defects are detected.
[0175] Specifically:
[0176] For the seed dataset D in step (4) seed , if it is empty, then the grammar program data set D program unchanged, otherwise, the seed dataset D seed Merge into the grammar program dataset D program In the new grammar program data set D program And enter step (2), repeating the process until no TypeScript compiler defects are detected.
Claims
1. A TypeScript compiler defect detection method based on mutation and differential testing, characterized in that: Here are the steps: Step (1) starts from the perspective of collecting common grammatical features and unique grammatical features, searches and screens existing projects in the Github open source community, proposes prompt words to input into the large language model to generate code, and then performs program syntax and semantic correctness checks to finally obtain the grammatical program dataset D program ; Step (2) construct the lexical parser TypeScriptLexer and the syntax parser TypeScriptParser corresponding to the TypeScript language, and obtain the syntax program dataset D based on this. program The parse tree corresponding to each program contained in the grammar program dataset D is obtained by randomly selecting one of the 30 mutation algorithms defined. program The mutation programs corresponding to the program files in the program are summarized as the mutation program dataset D mutation ; Step (3) Using the mutation program dataset D mutation Perform differential testing on the selected TypeScript compiler version, and determine the test execution results based on the six defined cases; pair the mutant programs that pass the compilation and executable tests with their corresponding original syntax programs to obtain a feasible program dataset D feasible , used for subsequent evaluation and iteration of new variants with seed programs; Step (4) Calculate the feasible program data set D respectively feasible The code complexity of each mutant program and its corresponding original syntax program in are compared, and the mutant program with higher code complexity is selected to form the seed dataset D seed ; Step (5) ends here if no TypeScript compiler defect is detected. Otherwise, the seed dataset D seed Merge into the grammar program dataset D program Then enter step (2) and repeat the process until no TypeScript compiler defects are detected. The 30 mutation algorithms in step (2) are specifically as follows: a) Randomly delete a subtree b) Randomly replace a subtree c) Randomly delete a node d) Randomly delete an expression e) Randomly replace the value of number type variable f) Randomly replace the value of a string type variable g) True / False swap h) Randomly modify an identifier i) Add an object destructuring assignment statement j) Add an array destructuring assignment statement k) Functions and arrow functions can be converted to each other l) The subclass object is assigned to the parent class variable m) The value of a union type variable is replaced with another optional type value n) Randomly select an arithmetic operator and replace it with another arithmetic operator o) Randomly select an assignment expression and replace the assignment symbol with another assignment symbol p) Randomly select a comparison operator and replace it with another comparison operator q) Randomly select a logical operator and replace it with another logical operator r) Randomly select a bitwise operator and replace it with another bitwise operator s) Randomly select a unary operator and replace it with another unary operator t) The condition of the conditional expression is replaced with True / False or the content is left blank u) Randomly delete one or more elements in a tuple v) Randomly delete a block statement w) Randomly delete the contents of the new expression x) Randomly delete the body of a function y) Expanded arithmetic expressions z) Randomly remove type annotations aa) Similar API replacement bb) Randomly repeat a sentence cc) randomly insert a new subtree at a certain position in the parse tree dd) Numeric type constants are replaced with boundary values The six situations in step (3) are specifically defined as follows, where b)-e) all require a defect report to be submitted to the developer: a) All versions fail to compile with the same syntax error: program syntax or semantics error, compiler behavior is normal b) All versions failed to compile with different error messages: the compiler had a bug in the syntax checking phase c) Compilation failures and compile successes occur simultaneously in different versions: the compiler has defects in the translation stage d) All versions are compiled successfully, but Node.js fails to execute: the compiler has a runtime error, and the generated js program cannot run successfully e) All versions are compiled successfully, Node.js is executed successfully but the results are different: there is an inconsistency error in the compiler, and the semantics of the generated js program are not exactly the same f) All versions compile successfully and Node.js executes successfully with the same result: the compiler passes the test for this program.
2. According to claim 1, a TypeScript compiler defect detection method based on mutation and differential testing is characterized in that: Step (1) specifically includes the following steps: 1-1) From the perspective of collecting common grammatical features, we select all high-quality open source projects whose development language includes Typescript and whose star count ranks in the top n from the GitHub open source community. For each project, we use crawler technology to collect all Typescript program files in the project and construct a common grammar program dataset D. common , each piece of data in the data set corresponds to all the codes of a collected Typescript program file; 1-2) From the perspective of collecting specific grammatical features, a prompt word template to be filled is designed: "Please generate a TypeScript program of no less than m lines, m>20, the syntax used in the program contains <unique grammatical type>, and ensure that the code is complex and semantically correct; "Use "interface, enumeration, tuple, union type, intersection type, type alias, type assertion, type annotation, never type, optional type and read-only type" to fill in the <unique grammatical type> field respectively, and obtain 11 natural language precise prompts for inputting into the large language model, and input the selected code respectively. The large language model is executed several times to construct a specific grammar program dataset D specific , each piece of data in the data set corresponds to all the codes of a generated Typescript program file; 1-3) For the two data sets obtained in step 1-1) and step 1-2), the two data sets are merged into the initial test program set D initial , using the static syntax analysis tool tslint and the dynamic semantic checking tool ts-node to check the initial assembly D initial The syntax and semantic correctness of each Typescript program in the dataset D is filtered as long as any one of them fails. program .
3. According to claim 1, a TypeScript compiler defect detection method based on mutation and differential testing is characterized in that: The step (2) specifically comprises the following steps: 2-1) Download the TypeScript grammar files from the antlr / grammars-v4 repository on GitHub, namely TypeScriptLexer.g4 for lexical rules and TypeScriptParser.g4 for grammatical rules, input the two grammar files into the ANTLR4 tool, and build the lexical parser TypeScriptLexer and grammatical parser TypeScriptParser corresponding to the TypeScript language; 2-2) For the grammar program dataset D program For each program file in the , use the ANTLR4 tool to parse the program file into a character stream, and input the lexical parser TypeScriptLexer to obtain the corresponding token stream, and then input the obtained token stream into the syntax parser TypeScriptParser to obtain the corresponding parse tree, and finally obtain the parse tree set list L corresponding to each program file tree ; 2-3) Define 30 mutation algorithms; traverse the parse tree set list L tree , a mutation algorithm is randomly selected for each parse tree to mutate, and the mutation program dataset D is obtained mutation .
4. According to claim 1, a TypeScript compiler defect detection method based on mutation and differential testing is characterized in that: The step (3) specifically comprises the following steps: 3-1) Traverse the mutation program data set D one by one mutation For each mutant program, different versions of TypeScript compilers are input to try to compile and compare the output results. If the program can be compiled, the js program compiled by the compiler of this version is executed by Node.js and the execution results are compared. According to the corresponding situations of the defined 6 outputs and execution results, the differential test results can be obtained, that is, whether the compiler has defects. 3-2) Pair the programs that meet the condition f) in step 3-1) with their corresponding original syntax programs to form a feasible program dataset D feasible .
5. According to claim 1, a TypeScript compiler defect detection method based on mutation and differential testing is characterized in that: The step (4) specifically comprises the following steps: 4-1) Define code complexity indicators: number of lines of code, cyclomatic complexity, maximum depth of parse tree, maximum nesting depth, Halstead index, number of classes and methods; the calculation method is as follows: a) Number of lines of code, that is, the number of program code lines calculated using the cloc tool b) Cyclomatic complexity, which is the number of decision nodes + 1; the number of decision nodes is the sum of all decision nodes in if statements, while statements, case statements, ternary expressions, with statements, for statements, for...in statements, and for...of statements c) The maximum depth of the parse tree, that is, the maximum depth maxDepth calculated by recursively traversing the parse tree; as shown in formula (1), where depth Depth refers to the length of the longest path from the root node of the parse tree to the leaf node, child is the child node of the current parse tree node, currentDepth is the current depth, and calculateDepth is a function used to recursively calculate the depth of the parse tree. The two parameters it receives represent the current parse tree node and the depth of the current node respectively: maxDepth=max(maxDepth,calculateDepth(child,currentDepth + 1)) (1) d) Maximum nesting depth, that is, the maximum value of the nesting depth of all if statements, while statements, ternary expressions, switch statements, with statements, for statements, for...in statements, and for...of statements, obtained by traversing the subtrees corresponding to these statements; e) Halstead index: traverse the parse tree and count the following four indexes: the number of different operators in the program n1, the number of different operands in the program n2, the total number of operators appearing in the program N1, and the total number of operands appearing in the program N2; the Halstead index is shown in formula (2), where n represents the number of operands and operators, which is the sum of n1 and n2, N represents the number of operands and operators, which is the sum of N1 and N2, and E represents the programming workload, which is calculated as shown in formula (3): Halstead_score=0.3*n+0.4*N+0.5*E (2) f) The number of classes and methods, that is, the sum of the number of all classes and functions on the parse tree; 4-2) Traverse the feasible program data set D feasible , input each mutation program and the corresponding original grammar program into the lexical parser TypeScriptLexer in turn to obtain the corresponding token stream, and then input the obtained token stream into the grammar parser TypeScriptParser to obtain the corresponding parse tree. Finally, on this basis, the six code complexity indicators of each mutation program and the original grammar program are calculated respectively; 4-3) Compare each pair of mutant programs and original syntax programs. If any of the six code complexity indicators of the mutant program is higher than the corresponding indicator value of the original syntax program, the mutated program code complexity is considered to be higher. The mutant program is added to the seed data set, and the seed data set D is finally obtained. seed .