A natural language processing-based JavaScript engine directed fuzz testing method and system

Abstract

The application discloses a JavaScript engine oriented fuzz testing method and system based on natural language processing, which extracts JavaScript language syntax and semantic information through a BERT language model by using a natural language processing technology, performs fine-tuning by combining a residual network and a mask language model, and performs oriented fuzz testing on a JavaScript engine in combination with an AFLGO oriented fuzz testing technology. The bidirectional sequence processing capability of the BERT language model improves the precision of feature extraction, and the residual network further improves the accuracy of the language model. The oriented fuzz testing utilizes coverage information, and improves the speed of vulnerability discovery and the code coverage rate of the fuzz testing. The application takes into account the test case generation efficiency and the utilization of coverage information, greatly improves the effective test case generation rate, and also improves the code coverage rate of the fuzz testing.

Description

Technical Field

[0001] This invention relates to the field of software security testing technology, and in particular to a method and system for targeted fuzz testing of JavaScript engines based on natural language processing. Background Technology

[0002] The massive code space of the JavaScript engine determines the primary role of fuzzing technology in JavaScript engine vulnerability discovery. The following will summarize the research ideas and results of previous JavaScript engine fuzzing work.

[0003] JSfunfuzz, released by Mozilla's security team in 2008, is a fuzzing tool that generates JavaScript engine test cases using fixed JavaScript syntax templates, improving the effectiveness of test cases. LangFuzz, proposed by Holler C et al., uses syntax templates to decompose the AST of the seed file into AST fragments and generates syntactically correct test cases by randomly combining these fragments. IFuzzer, proposed by Veggalam et al., uses a genetic algorithm to exchange subtrees between different ASTs, ensuring the speed and diversity of test case generation. CodeAlchemist, proposed by Han et al., decomposes the seed file into several blocks according to a parallel structure to form a code block library, and randomly combines code blocks that match the input and output, thus ensuring the syntactic and semantic validity of the generated test cases. Superion, proposed by Wang et al., uses a syntax-aware method based on coverage for fuzzing. For fuzz testing, Superion uses a syntax-aware strategy based on dictionary-based mutation and tree-based mutation to prune the AST, generating test input at the subtree level. This syntax awareness brings higher code coverage to Superion. Montage, proposed by Lee et al., trains an LSTM language model using AST subtree sequences as input, prunes a subtree of the seed, and regenerates the subtree using the LSTM language model, ensuring the diversity and legitimacy of the generated test cases. DIE, proposed by Park et al., can preserve special structures such as branches and loops of the seed during mutation, making it easier to trigger vulnerabilities in JIT during fuzz testing, representing a new breakthrough in ensuring code coverage.

[0004] However, the aforementioned fuzzing methods also have certain limitations. JavaScript syntax templates have poor extensibility, requiring manual code modification for checking and updating; the method of randomly combining AST fragments results in most of the generated test cases not conforming to semantic logic, affecting the efficiency of fuzzing; genetic algorithms cannot guarantee that the generated test cases conform to grammatical standards and semantic logic; code block combination techniques can efficiently generate legal test cases, but code block-level mutation cannot achieve excellent code coverage, limiting the comprehensiveness of fuzzing; LSTM language models can only be trained unidirectionally and serially, limiting the accuracy and speed of language feature extraction; syntax awareness combined with coverage information can significantly improve code coverage, but the generated test cases do not consider semantic logic, affecting the efficiency of fuzzing. Summary of the Invention

[0005] To address the shortcomings of existing JavaScript engine fuzzing tools in terms of test case generation quality and coverage information utilization, this invention proposes a JavaScript engine-specific fuzzing technique based on natural language processing (NLP). It implements a JavaScript sub-Bert Fuzz (JSBFuzz) tool, which parses JavaScript seed files from a corpus into abstract syntax trees (ASTs) subtrees as pre-training inputs (Vocab) for the BERT language model. In the downstream fine-tuning stage, a residual network combined with a masked language model (MLM) is used to train a language model capable of generating efficient test cases that conform to JavaScript syntax and semantics. These test cases then serve as input for AFLGo-based targeted fuzzing. Ultimately, JSBFuzz ensures both efficient initial test case input and comprehensive and rapid testing of the JavaScript engine using coverage information, thereby uncovering more vulnerabilities. Experiments show that JSBFuzz significantly improves coverage and vulnerability triggering rates compared to Montage, Superion, and CodeAlchemist.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] This invention proposes a method for targeted fuzz testing of a JavaScript engine based on natural language processing, comprising:

[0008] Step 1: Parse the JavaScript seed files in the corpus into an Abstract Syntax Tree (AST), and delete seed files with syntax errors;

[0009] Step 2: Traverse the AST tree, construct a sequence of AST subtrees of height 2, and construct the BERT language model vocabulary and BERT language model sentence segments based on the AST subtree sequences;

[0010] Step 3: Use vocab and sentence as input for the BERT language model pre-training to obtain the BERT pre-trained model;

[0011] Step 4: Obtain the parent node sequence of the AST subtree based on the AST subtree sequence. Use the parent node sequence of the AST subtree as the input of the residual network using the BERT pre-trained model. Fine-tune it by combining it with the masked language model MLM to obtain the JavaScript test case generation model.

[0012] Step 5: Compare the source file information of different versions of the JavaScript engine, identify the patch location as a sensitive location, and instrument and compile the JavaScript engine source file.

[0013] Step 6: Use the JavaScript test case generation model to generate JavaScript test cases as input files for targeted fuzz testing, and perform targeted fuzz testing on the JavaScript engine.

[0014] Further, step 2 includes:

[0015] Define an AST subtree sequence `astlist` to initialize the seed, with empty lists for sentences and paragraphs. Use the `travel` function to recursively traverse the AST tree, constructing the AST subtree sequence, sentences, and paragraphs, and saving the relevant information. The `travel` function first recursively traverses the AST tree, checking each node sequentially. If the parent node of a node is of type "Program", it means that the node is the starting node of a sentence, so the previous sentence is added to the paragraph, and the sentence is reinitialized. If the target node `node` has child nodes, add the pruned subtree, parent node information, and parent node type information of the node to `astlist`, add the pruned subtree to the sentence, and traverse the node. If the node has no child nodes, it means that the node is already a terminal symbol, so update `vocab` and exit the current recursion.

[0016] Further, step 3 includes:

[0017] Vocabulary and sentence are used as pre-training inputs for the BERT language model. The BERT model is simplified to prevent overfitting during training. Adjacent and non-adjacent sentence pairs are selected proportionally from the sentence set. The embedding of the BERT input sequence is the sum of word embedding, sentence embedding, and position embedding. Each word, sentence tag, and position tag is transformed into a fixed-dimensional vector. The input data is processed through the simplified BERT model. Feature extraction is performed through the two pre-training tasks of MLM and next sentence prediction in the BERT model. The grammatical and semantic information of the JavaScript language is learned, and finally, the BERT pre-trained model is obtained.

[0018] Further, step 4 includes:

[0019] Based on the BERT pre-trained model, a fine-tuning structure is added. The fine-tuning structure includes two linear neural network layers and a ReLU function between the two neural network layers. The sizes of the two neural network layers are {input length * BERT model hidden dimension} and {BERT model hidden dimension * 1024}, respectively. The output of this fine-tuning structure is superimposed on the output of the MLM output segment in the BERT pre-trained model. The MLM output segment has the same composition as the fine-tuning structure.

[0020] The AST subtree sequence and the parent node sequence of the AST subtree are respectively input into the simplified BERT model and the fine-tuning mechanism for processing. Finally, the output of the MLM output segment in the BERT pre-trained model and the output of the fine-tuning mechanism are superimposed and then subjected to ReLU activation and layer normalization. The normalization result is uniformly output using a linear layer of size {1024*vocab length}. The output result is the probability distribution of the predicted next word on the vocab.

[0021] Further, step 6 includes:

[0022] A random AST subtree sequence is selected, and a certain proportion of points are mutated and replaced. The replacement points are processed sequentially. For each replacement point, the parent and child nodes are first obtained, and all replacement points and child nodes are deleted. Starting from the parent node, a JavaScript test case generation model is used to generate child nodes. The child nodes are selected from the top-K pre-selected nodes. If the child node is a non-terminal symbol, the next node is generated from that child node. A subtree height threshold is limited. Nodes exceeding the subtree height threshold are forced to generate a terminal symbol to end the generation process, thereby ensuring the legality of the generated subtree. The mutated AST tree is restored to JavaScript test cases as input for targeted fuzz testing, and targeted fuzz testing is performed on the JavaScript engine.

[0023] Another aspect of this invention proposes a targeted fuzz testing system for a JavaScript engine based on natural language processing, comprising:

[0024] The parsing module is used to parse JavaScript seed files in the corpus into an abstract syntax tree (AST), and delete seed files with syntax errors.

[0025] The building module is used to traverse the AST tree, construct a sequence of AST subtrees of height 2, and build the BERT language model vocabulary and BERT language model sentence based on the AST subtree sequence;

[0026] The pre-training module is used to take vocab and sentence as input for the pre-training of the BERT language model to obtain the BERT pre-trained model.

[0027] The fine-tuning module is used to obtain the parent node sequence of the AST subtree based on the AST subtree sequence. It uses the parent node sequence of the AST subtree as the input of the residual network using the BERT pre-trained model and combines it with the masked language model MLM for fine-tuning to obtain the JavaScript test case generation model.

[0028] The instrumentation compilation module is used to compare the source file information of different versions of the JavaScript engine, use the patch location as a sensitive location, and instrument the JavaScript engine source file for compilation.

[0029] The fuzzing module is used to generate JavaScript test cases using a JavaScript test case generation model, which serve as input files for targeted fuzzing, and to perform targeted fuzzing on the JavaScript engine.

[0030] Furthermore, the building module is specifically used for:

[0031] Define an AST subtree sequence `astlist` to initialize the seed, with empty lists for sentences and paragraphs. Use the `travel` function to recursively traverse the AST tree, constructing the AST subtree sequence, sentences, and paragraphs, and saving the relevant information. The `travel` function first recursively traverses the AST tree, checking each node sequentially. If the parent node of a node is of type "Program", it means that the node is the starting node of a sentence, so the previous sentence is added to the paragraph, and the sentence is reinitialized. If the target node `node` has child nodes, add the pruned subtree, parent node information, and parent node type information of the node to `astlist`, add the pruned subtree to the sentence, and traverse the node. If the node has no child nodes, it means that the node is already a terminal symbol, so update `vocab` and exit the current recursion.

[0032] Furthermore, the pre-training module is specifically used for:

[0033] Vocabulary and sentence are used as pre-training inputs for the BERT language model. The BERT model is simplified to prevent overfitting during training. Adjacent and non-adjacent sentence pairs are selected proportionally from the sentence set. The embedding of the BERT input sequence is the sum of word embedding, sentence embedding, and position embedding. Each word, sentence tag, and position tag is transformed into a fixed-dimensional vector. The input data is processed through the simplified BERT model. Feature extraction is performed through the two pre-training tasks of MLM and next sentence prediction in the BERT model. The grammatical and semantic information of the JavaScript language is learned, and finally, the BERT pre-trained model is obtained.

[0034] Furthermore, the fine-tuning module is specifically used for:

[0035] Based on the BERT pre-trained model, a fine-tuning structure is added. The fine-tuning structure includes two linear neural network layers and a ReLU function between the two neural network layers. The sizes of the two neural network layers are {input length * BERT model hidden dimension} and {BERT model hidden dimension * 1024}, respectively. The output of this fine-tuning structure is superimposed on the output of the MLM output segment in the BERT pre-trained model. The MLM output segment has the same composition as the fine-tuning structure.

[0036] The AST subtree sequence and the parent node sequence of the AST subtree are respectively input into the simplified BERT model and the fine-tuning mechanism for processing. Finally, the output of the MLM output segment in the BERT pre-trained model and the output of the fine-tuning mechanism are superimposed and then subjected to ReLU activation and layer normalization. The normalization result is uniformly output using a linear layer of size {1024*vocab length}. The output result is the probability distribution of the predicted next word on the vocab.

[0037] Furthermore, the fuzz testing module is specifically used for:

[0038] A random AST subtree sequence is selected, and a certain proportion of points are mutated and replaced. The replacement points are processed sequentially. For each replacement point, the parent and child nodes are first obtained, and all replacement points and child nodes are deleted. Starting from the parent node, a JavaScript test case generation model is used to generate child nodes. The child nodes are selected from the top-K pre-selected nodes. If the child node is a non-terminal symbol, the next node is generated from that child node. A subtree height threshold is limited. Nodes exceeding the subtree height threshold are forced to generate a terminal symbol to end the generation process, thereby ensuring the legality of the generated subtree. The mutated AST tree is restored to JavaScript test cases as input for targeted fuzz testing, and targeted fuzz testing is performed on the JavaScript engine.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] This invention utilizes natural language processing technology to extract syntactic and semantic information of JavaScript through the BERT language model. It then fine-tunes the model using a combination of residual networks and masked language models, and incorporates targeted fuzzing technology to perform targeted fuzzing tests on the JavaScript engine. The bidirectional sequence processing capability of the BERT language model improves the accuracy of feature extraction, the residual network further enhances the accuracy of the language model, and the targeted fuzzing utilizes coverage information to improve the speed of vulnerability discovery and the code coverage of fuzzing tests.

[0041] This invention balances test case generation efficiency and coverage information utilization, significantly improving the effective test case generation rate while also increasing the code coverage of fuzz testing. Attached Figure Description

[0042] Figure 1 This is a basic flowchart of a JavaScript engine-based targeted fuzz testing method based on natural language processing, according to an embodiment of the present invention.

[0043] Figure 2This is a schematic diagram of a JavaScript engine-based targeted fuzz testing system architecture based on natural language processing, according to an embodiment of the present invention.

[0044] Figure 3 This is a flowchart of the JSBFuzz process according to an embodiment of the present invention;

[0045] Figure 4 This is pseudocode for constructing AST subtree sequences, BERT input vocabulary, and sentence segments in an embodiment of the present invention;

[0046] Figure 5 This is a flowchart of the JSBFuzz pre-training stage processing according to an embodiment of the present invention;

[0047] Figure 6 This is a flowchart illustrating the working process of the JSBFuzz fine-tuning stage in an embodiment of the present invention.

[0048] Figure 7 Pseudocode for generating test cases in embodiments of the present invention;

[0049] Figure 8 The JSBFuzz pre-training loss variation is shown in this embodiment of the invention.

[0050] Figure 9 The training loss variation for JSBFuzz fine-tuning in this embodiment of the invention;

[0051] Figure 10 This refers to the coverage information of the JavaScript engine fuzzing tool running on JavaScriptCore in an embodiment of the present invention;

[0052] Figure 11 This refers to the test case generation capability of each fuzzing tool in the embodiments of the present invention. Detailed Implementation

[0053] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:

[0054] like Figure 1 As shown, a method for targeted fuzz testing using a JavaScript engine based on natural language processing includes:

[0055] Step S101: Parse the JavaScript seed files in the corpus into an Abstract Syntax Tree (AST), and delete seed files with syntax errors;

[0056] Step S102: Traverse the AST tree, construct an AST subtree sequence of height 2, and construct the BERT language model vocabulary and BERT language model sentence based on the AST subtree sequence;

[0057] Step S103: Use vocab and sentence as input for the BERT language model pre-training to obtain the BERT pre-trained model;

[0058] Step S104: Obtain the parent node sequence of the AST subtree based on the AST subtree sequence, use the parent node sequence of the AST subtree as the input of the residual network using the BERT pre-trained model, and fine-tune it in combination with the masked language model MLM to obtain the JavaScript test case generation model.

[0059] Step S105: Compare the source file information of each version of the JavaScript engine, take the patch location as a sensitive location, and instrument and compile the JavaScript engine source file;

[0060] Step S106: Use the JavaScript test case generation model to generate JavaScript test cases as input files for targeted fuzz testing, and perform targeted fuzz testing on the JavaScript engine.

[0061] Further, step S102 includes:

[0062] Define an AST subtree sequence `astlist` to initialize the seed, with empty lists for sentences and paragraphs. Use the `travel` function (a recursive function designed to traverse nodes and their children) to recursively traverse the AST tree, constructing the AST subtree sequence, sentences, and paragraphs, and saving the relevant information. The `travel` function first recursively traverses the AST tree, checking each node sequentially. If the node's parent node type is "Program", it indicates that the node is the starting node of a sentence, so the previous sentence is added to the paragraph, and the sentence is reinitialized. If the target node has children, the pruned subtree, parent node information, and parent node type information are added to `astlist`, the pruned subtree is added to the sentence, and the node is traversed. If the node has no children, it indicates that the node is already a terminal symbol, so `vocab` is updated and the current recursion is exited.

[0063] Further, step S103 includes:

[0064] Vocabulary and sentence are used as pre-training inputs for the BERT language model. The BERT model is simplified to prevent overfitting during training. Adjacent and non-adjacent sentence pairs are selected proportionally from the sentence set. The embedding of the BERT input sequence is the sum of word embedding, sentence embedding, and position embedding. Each word, sentence tag, and position tag is transformed into a fixed-dimensional vector. The input data is processed through the simplified BERT model. Feature extraction is performed through the two pre-training tasks of MLM and next sentence prediction in the BERT model. The grammatical and semantic information of the JavaScript language is learned, and finally, the BERT pre-trained model is obtained.

[0065] Further, step S104 includes:

[0066] Based on the BERT pre-trained model, a fine-tuning structure is added. The fine-tuning structure includes two linear neural network layers and a ReLU function between the two neural network layers. The sizes of the two neural network layers are {input length * BERT model hidden dimension} and {BERT model hidden dimension * 1024}, respectively. The output of this fine-tuning structure is superimposed on the output of the MLM output segment in the BERT pre-trained model. The MLM output segment has the same composition as the fine-tuning structure.

[0067] The AST subtree sequence and the parent node sequence of the AST subtree are respectively input into the simplified BERT model and the fine-tuning mechanism for processing. Finally, the output of the MLM output segment in the BERT pre-trained model and the output of the fine-tuning mechanism are superimposed and then subjected to ReLU activation and layer normalization. The normalization result is uniformly output using a linear layer of size {1024*vocab length}. The output result is the probability distribution of the predicted next word on the vocab.

[0068] Further, step S106 includes:

[0069] A random AST subtree sequence is selected, and a certain proportion of points are mutated and replaced. The replacement points are processed sequentially. For each replacement point, the parent and child nodes are first obtained, and all replacement points and child nodes are deleted. Starting from the parent node, a JavaScript test case generation model is used to generate child nodes. The child nodes are selected from the top-K pre-selected nodes. If the child node is a non-terminal symbol, the next node is generated from that child node. A subtree height threshold is limited. Nodes exceeding the subtree height threshold are forced to generate a terminal symbol to end the generation process, thereby ensuring the legality of the generated subtree. The mutated AST tree is restored to JavaScript test cases as input for targeted fuzz testing, and targeted fuzz testing is performed on the JavaScript engine.

[0070] Based on the above embodiments, such as Figure 2 As shown, another aspect of the present invention proposes a JavaScript engine-based targeted fuzz testing system (JSBFuzz for short) based on natural language processing, comprising:

[0071] The parsing module is used to parse JavaScript seed files in the corpus into an abstract syntax tree (AST), and delete seed files with syntax errors.

[0072] The building module is used to traverse the AST tree, construct a sequence of AST subtrees of height 2, and build the BERT language model vocabulary and BERT language model sentence based on the AST subtree sequence;

[0073] The pre-training module is used to take vocab and sentence as input for the pre-training of the BERT language model to obtain the BERT pre-trained model.

[0074] The fine-tuning module is used to obtain the parent node sequence of the AST subtree based on the AST subtree sequence. It uses the parent node sequence of the AST subtree as the input of the residual network using the BERT pre-trained model and combines it with the masked language model MLM for fine-tuning to obtain the JavaScript test case generation model.

[0075] The instrumentation compilation module is used to compare the source file information of different versions of the JavaScript engine, use the patch location as a sensitive location, and instrument the JavaScript engine source file for compilation.

[0076] The fuzzing module is used to generate JavaScript test cases using a JavaScript test case generation model, which serve as input files for targeted fuzzing, and to perform targeted fuzzing on the JavaScript engine.

[0077] Furthermore, the building module is specifically used for:

[0078] Define an AST subtree sequence `astlist` to initialize the seed, with empty lists for sentences and paragraphs. Use the `travel` function to recursively traverse the AST tree, constructing the AST subtree sequence, sentences, and paragraphs, and saving the relevant information. The `travel` function first recursively traverses the AST tree, checking each node sequentially. If the parent node of a node is of type "Program", it means that the node is the starting node of a sentence, so the previous sentence is added to the paragraph, and the sentence is reinitialized. If the target node `node` has child nodes, add the pruned subtree, parent node information, and parent node type information of the node to `astlist`, add the pruned subtree to the sentence, and traverse the node. If the node has no child nodes, it means that the node is already a terminal symbol, so update `vocab` and exit the current recursion.

[0079] Furthermore, the pre-training module is specifically used for:

[0080] Vocabulary and sentence are used as pre-training inputs for the BERT language model. The BERT model is simplified to prevent overfitting during training. Adjacent and non-adjacent sentence pairs are selected proportionally from the sentence set. The embedding of the BERT input sequence is the sum of word embedding, sentence embedding, and position embedding. Each word, sentence tag, and position tag is transformed into a fixed-dimensional vector. The input data is processed through the simplified BERT model. Feature extraction is performed through the two pre-training tasks of MLM and next sentence prediction in the BERT model. The grammatical and semantic information of the JavaScript language is learned, and finally, the BERT pre-trained model is obtained.

[0081] Furthermore, the fine-tuning module is specifically used for:

[0082] Based on the BERT pre-trained model, a fine-tuning structure is added. The fine-tuning structure includes two linear neural network layers and a ReLU function between the two neural network layers. The sizes of the two neural network layers are {input length * BERT model hidden dimension} and {BERT model hidden dimension * 1024}, respectively. The output of this fine-tuning structure is superimposed on the output of the MLM output segment in the BERT pre-trained model. The MLM output segment has the same composition as the fine-tuning structure.

[0083] The AST subtree sequence and the parent node sequence of the AST subtree are respectively input into the simplified BERT model and the fine-tuning mechanism for processing. Finally, the output of the MLM output segment in the BERT pre-trained model and the output of the fine-tuning mechanism are superimposed and then subjected to ReLU activation and layer normalization. The normalization result is uniformly output using a linear layer of size {1024*vocab length}. The output result is the probability distribution of the predicted next word on the vocab.

[0084] Furthermore, the fuzz testing module is specifically used for:

[0085] A random AST subtree sequence is selected, and a certain proportion of points are mutated and replaced. The replacement points are processed sequentially. For each replacement point, the parent and child nodes are first obtained, and all replacement points and child nodes are deleted. Starting from the parent node, a JavaScript test case generation model is used to generate child nodes. The child nodes are selected from the top-K pre-selected nodes. If the child node is a non-terminal symbol, the next node is generated from that child node. A subtree height threshold is limited. Nodes exceeding the subtree height threshold are forced to generate a terminal symbol to end the generation process, thereby ensuring the legality of the generated subtree. The mutated AST tree is restored to JavaScript test cases as input for targeted fuzz testing, and targeted fuzz testing is performed on the JavaScript engine.

[0086] Specifically, JSBFuzz pre-trains the AST subtree sequence of JavaScript test cases using the BERT language model, and then fine-tunes it using a combination of residual networks and MLM. The resulting language model is used to generate JavaScript language files as seed files for an AFLGO-based targeted fuzzing system. The BERT language model, built on a neural network, is a bidirectional transformer-based language model consisting of pre-training and downstream task fine-tuning. The BERT language model primarily utilizes bidirectional and multi-head attention mechanisms to address the issues of incomplete information attention and parallel processing in natural language processing. The JavaScript language files generated by the BERT language model conform to JavaScript's syntax rules and semantic logic, enabling them to be mutated as high-quality test cases during the targeted fuzzing phase. This results in higher code coverage and improved vulnerability discovery capabilities for JSBFuzz. Figure 3 The complete system architecture and workflow of JSBFuzz are demonstrated.

[0087] Specifically, the JSBFuzz processing flow mainly includes the following four stages:

[0088] 1. JSBFuzz Preprocessing Stage

[0089] The main task of the JSBFuzz preprocessing stage is to decompose the seed files in the corpus into a sequence of AST subtrees of height 2. Since the AST tree obtained by parsing JavaScript language program files has a rigorous structure and logical features, and the AST subtree sequence obtained by arranging its subtrees in preorder has structural features similar to natural language, each AST subtree can serve as the input vocabulary for the BERT language model, and the entire AST subtree sequence can serve as the input paragraph for the BERT language model. Algorithm 1 demonstrates the construction of the AST subtree sequence and the BERT input vocabulary and sentence segments, such as... Figure 4 As shown.

[0090] Algorithm 1 takes as input all seed files (seeds) and JavaScript syntax files from the corpus. The algorithm first parses the input seeds, obtaining an AST tree and storing the corresponding variable mappings. These mappings ensure that seed mutations do not lead to variable confusion. If a seed cannot be parsed, the erroneous seed file is discarded (lines 1-6). Next, we define an initial subtree sequence `astlist` for this seed, with empty lists for sentences and paragraphs. The `travel` function recursively traverses the AST tree, constructing the AST subtree sequence, sentence segments, and paragraphs, and saving the relevant information (lines 7-9).

[0091] Lines 10-28 detail the implementation of the `travel` function. The `travel` function first recursively traverses the AST tree, checking each node sequentially. If the node's parent node is of type "Program", it indicates that the node is the starting node of a statement. We add the previous sentence to the paragraph and reinitialize the sentence (lines 11-15). If the node has child nodes, we add the pruned subtree, parent node information, and parent node type information to the ASTlist, add the pruned subtree to the sentence, and then perform the `travel` operation on this node (lines 16-24). If the node has no child nodes, it means that the node is already a terminal symbol representation. In this case, we update `vocab` and exit the current recursion.

[0092] 2. JSBFuzz Training Phase

[0093] The task of the training phase is to train a BERT language model through machine learning that can automatically generate test cases that conform to JavaScript syntax rules and semantic logic. The training phase is divided into two parts: pre-training and downstream task fine-tuning.

[0094] 2.1. Pre-training in the JSBFuzz Training Phase

[0095] In the pre-training phase, we first need to simplify the BERT model. The BERTbase model has 12 layers and 12 attention heads, with a hidden layer output dimension of 768 and a total of 110M parameters. BERTLARGE has 24 layers and 16 attention heads, with a hidden layer dimension of 1024 and a total of 340M parameters. The total number of BERT training vocabularies is 3300M, but JSBFuzz's vocabulary only has 2.17M words and 123,200 sentences. We need to simplify the BERT model to ensure that overfitting does not occur during training. Through a large number of experiments, the model selected in this invention uses 6 bidirectional transformer encoding layers, 6 attention heads, and sets the input dimension and hidden layer output dimension to 128 and 256, respectively, with a total of 1.17M parameters. Figure 5 The pre-training phase processing is demonstrated.

[0096] This invention selects adjacent and non-adjacent sentence pairs from the sentence set with a 50% probability. The embedding of the BERT input sequence is the sum of word embedding, sentence embedding, and position embedding. Each word, sentence tag, and position tag is transformed into a 128-dimensional vector. The input data is processed through a simplified BERT model, and features are extracted through the two pre-training tasks of MLM and Next Sentence Prediction (NSP) in the BERT model. The grammatical and semantic information of the JavaScript language is learned.

[0097] 2.2. JSBFuzz Training Phase: Fine-tuning

[0098] Pre-training yielded an initial BERT model with JavaScript syntactic and semantic features. However, the MLM in this BERT model lacked parent node information for matching AST subtrees, resulting in most predicted TOP-K segments failing to match the original sequence despite conforming to syntactic rules and semantic logic. This reduced the efficiency of test case generation. To address this issue, we used a combination of Residual Networks and MLM for fine-tuning. Extensive experiments and research have shown that adding more layers to a model of appropriate depth leads to higher training errors, which are not due to overfitting. Residual Networks (ResNet) can easily gain accuracy from increasing neural network depth, solving the problem of higher training errors caused by increased neural network depth. Using Residual Networks ensures that the features extracted by the pre-trained model are not destroyed during fine-tuning, preserving the model's syntactic and semantic capabilities, while also utilizing parent node information of subtrees, thereby improving the accuracy of segment prediction. Figure 6It demonstrates the working process of the fine-tuning stage.

[0099] like Figure 6 As shown on the left, we first input the AST subtree sequence into the pre-trained BERT model. The output of the BERT model is connected to two linear neural network layers. This structure is the output segment of the MLM in the pre-training. The size of the neural network layers is (input length * BERT model hidden dimension) and (BERT model hidden dimension * 1024), respectively. ReLU is used as the activation function between the neural network layers, and the output is superimposed with the output of the neural network on the right. Figure 6 The neural network on the right processes the parent node information of the AST subtree sequence input from the left. It takes the parent node sequence of the AST subtree as input and processes it... Figure 6 The neural network on the left has the same structure. The outputs of the two neural networks are then superimposed, followed by ReLU activation and layer normalization. The normalized result is then uniformly output using a linear layer of size (1024 * vocab length). The output represents the probability distribution of the predicted next word on the vocab. Because the superposition of the left and right neural networks uses the principle of residual networks, the processing of parent node information on the right does not reduce the accuracy of the pre-trained model. The incorporation of parent node information during downstream task training improves the overall predictive ability of the model.

[0100] 3. JSBFuzz instrumentation compilation stage

[0101] Instrumentation and compilation of the program under test is the basis for obtaining coverage information, while sensitive location marking is a necessary process for targeted fuzz testing.

[0102] Research has shown that each patch increases the probability of bugs appearing. We used a diff tool to compare the current version of the JavaScript engine source code with previous versions, marking the patch locations as sensitive locations. We then used AFLGO's compilation tools to instrument and compile a version of the JavaScript engine program containing this sensitive location information. This invention uses this JavaScript engine to run test cases and obtain coverage information.

[0103] 4. JSBFuzz Targeted Fuzz Testing Phase

[0104] The preprocessing, training, and instrumentation / compilation stages constitute the preparation phase for fuzz testing. The main task of the targeted fuzzing stage is to generate a large number of seed files using the trained BERT language model as input for targeted fuzzing. According to... Figure 6 The workflow involves randomly selecting an AST subtree sequence from the AST subtree sequence library and performing mutation replacements on 15% of the points. Algorithm 2 demonstrates the specific test case generation algorithm, such as... Figure 7As shown. We process the replacement points sequentially. For each replacement point, we first obtain its parent and child nodes, then delete both the replacement point and its child nodes (lines 1-8). Starting from the parent node, we use the JavaScript test case generation model to generate child nodes. Child nodes are selected from the top-K pre-selected nodes. If a child node is a non-terminal symbol, we continue to generate the next node from that child node (lines 9-20). To avoid infinite recursion, we limit the height of the generated subtree to 50. For nodes exceeding the subtree height, we use syntax rules to force the generation of a terminal symbol to end the generation process, thus ensuring the legality of the generated subtree (line 22). The mutated AST tree is then restored to a JavaScript test case (line 24) as input for targeted fuzzing. For targeted fuzzing, we use the same simulated annealing algorithm as AFLGO. AFLGO, as a mature targeted fuzzing tool, uses simulated annealing as its core algorithm, ensuring the stability and effectiveness of JSBFuzz targeted fuzzing.

[0105] To verify the feasibility and effectiveness of JSBFuzz, this embodiment compares JSBFuzz with open-source JavaScript engine testing tools Montage, CodeAlchemist, and Superion on JerryScript, ChakraCore, and JavaScriptCore in four aspects: vulnerability discovery capability, code coverage, test case generation speed, and effective test case generation rate. Model training and fuzzing in this embodiment were performed on a server with an AMD 5950X-16 core 32-thread CPU, DDR-4-3600MHz-64GB physical memory, and an NVIDIA-RTX3080TI-12G graphics card. The operating system was Ubuntu n 20.04 Server.

[0106] (1) Model Training

[0107] JSBFuzz training requires a large amount of input text. We collected more than 40,000 JavaScript files from websites such as GitHub, ECMA projects, and TEST262 projects. These files contain all the syntax rules that the current version of the JavaScript engine can parse, ensuring the comprehensiveness of the fuzz test.

[0108] In the preprocessing stage, this embodiment uses esprima 4.0.1 with 32 threads to parse the JavaScript seed file. After 18 minutes of parsing and processing, a total of 2,172,843 words and 123,207 sentences were obtained. Through word cleaning and deduplication, the resulting vocab contains 13,252 unique words.

[0109] In the pre-training phase, this embodiment uses PyTorch 1.10.1 + CUDA 11.3 as the foundation for building the language model. The selected model uses 6 bidirectional transformer encoder layers, 6 attention heads, and the input dimension and hidden layer output dimension are set to 128 and 256, respectively, with a total of 1.17M parameters. To balance performance and insufficient GPU memory, this embodiment sets the batch size to 256 and the total training steps to 2M. To improve training quality and accelerate convergence while preventing gradient vanishing, this embodiment uses the Adam training optimizer with a learning rate of 10. -3 The cross-entropy function was used to calculate the training loss of MLM and NSP respectively. The losses of the two models were added together as the total training loss, and parameters were updated through backpropagation. JSBFuzz pre-training took 30 hours. Figure 8 The changes in loss for MLM and NSP during JSBFuzz pre-training are shown. After 2 million training steps, the loss of MLM eventually decreased to 0.43, and the loss of NSP decreased to 0.21.

[0110] In the fine-tuning phase, this embodiment uses the same training method as pre-training. To ensure better compatibility between pre-training and downstream tasks, all training parameters remain unchanged. We performed 100,000 steps of downstream task training. Figure 9 This demonstrates the change in loss during downstream task training. After 2 hours of fine-tuning on downstream tasks, the loss of the prediction model decreased from 0.43 to 0.28.

[0111] (2) Vulnerability discovery capability

[0112] This embodiment uses JSBFuzz, Montage, CodeAlchemist, and Superion to conduct 72 hours of testing on JerryScript 2.4.0, ChakraCore 1.11.0, and JavaScriptCore 2.27.4, respectively. Table 1 lists the results of JSBFuzz compared with several other JavaScript engine fuzzing tools. The results show that JSBFuzz found a total of 9 bugs within 72 hours, which is 80% and 28.5% higher than Montage and Superion, respectively. Table 2 lists the bugs found by JSBFuzz. The experimental results show that compared with Montage and CodeAlchemist, which do not use coverage information, JSBFuzz and Superion have significantly improved bug finding rates, and coverage information plays an important role in JavaScript engine fuzzing.

[0113] Table 1 Comparison of JSBFuzz results with other fuzzing tools

[0114]

[0115] Table 2. List of bugs discovered by JSBFuzz

[0116]

[0117] (3) Code coverage capability

[0118] This embodiment compares the 24-hour code coverage of JSBFuzz, Montage, CodeAlchemist, and Superion on JavaScriptCore. Coverage information was collected using afl-cov. Figure 10 This provides specific data on the coverage information of the above four JavaScript engine fuzzing tools running on JavaScriptCore. According to... Figure 10 As shown, JSBFuzz improves line coverage by 30.02% and 39.7% compared to Montage and CodeAlchemist, respectively; and improves function coverage by 17.97% and 25.63% compared to Montage and CodeAlchemist, respectively. However, both line and function coverage of JSBFuzz are lower than those of Superion. This is because AFLGO's directional fuzzing technique focuses more on how to quickly approach the target area and ignores the overall coverage. Therefore, the running coverage of JSBFuzz is lower than that of Superion.

[0119] (4) Test case generation capability

[0120] Test case generation capability includes two parts: test case generation speed and test case effectiveness. Figure 11 This section showcases the test case generation capabilities of JSBFuzz, Montage, CodeAlchemist, and Superion. Regarding test case generation speed, JSBFuzz and Montage are significantly slower than CodeAlchemist and Superion. This is because JSBFuzz and Montage use natural language models to generate test cases, and the additional overhead of natural language processing affects test case generation. However, in terms of test case effectiveness, JSBFuzz generates 17.14%, 73.22%, and 299% more effective test cases than Montage, CodeAlchemist, and Superion, respectively. The BERT language model makes a positive contribution to JavaScript test case generation.

[0121] This invention proposes a targeted fuzzing method and system for JavaScript engines based on natural language processing (JSBFuzz). JSBFuzz trains the AST subtree sequence of a JavaScript seed file using a simplified BERT language model, automatically extracting syntactic and semantic features from the JavaScript file. The input to downstream tasks includes the parent node sequence corresponding to the AST subtree sequence. A combination of residual networks and MLM is used to further improve the accuracy of the model in extracting JavaScript syntactic and semantic features. Finally, targeted fuzzing techniques are combined to improve the speed of vulnerability discovery and the comprehensiveness of fuzzing. Experimental results show that JSBFuzz improves the efficiency of test case generation, achieves higher code coverage, and discovers 9 bugs.

[0122] In summary, this invention utilizes natural language processing technology to extract syntactic and semantic information of JavaScript through the BERT language model. It then employs a combination of residual networks and masked language models for fine-tuning, and combines AFLGO's targeted fuzzing technique to perform targeted fuzzing on the JavaScript engine. The bidirectional sequence processing capability of the BERT language model improves the accuracy of feature extraction, the residual network further enhances the accuracy of the language model, and the targeted fuzzing utilizes coverage information to improve the speed of vulnerability discovery and the code coverage of fuzzing. This invention balances test case generation efficiency and coverage information utilization, significantly improving the effective test case generation rate while also increasing the code coverage of fuzzing.

[0123] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for targeted fuzz testing using a JavaScript engine based on natural language processing, characterized in that, include: Step 1: Parse the JavaScript seed files in the corpus into an Abstract Syntax Tree (AST), and delete seed files with syntax errors; Step 2: Traverse the AST tree, construct a sequence of AST subtrees of height 2, and construct the BERT language model vocabulary and BERT language model sentence segments based on the AST subtree sequences; Step 3: Use vocab and sentence as input for the BERT language model pre-training to obtain the BERT pre-trained model; Step 4: Obtain the parent node sequence of the AST subtree based on the AST subtree sequence. Use the parent node sequence of the AST subtree as the input of the residual network using the BERT pre-trained model. Fine-tune it by combining it with the masked language model MLM to obtain the JavaScript test case generation model. Step 5: Compare the source file information of different versions of the JavaScript engine, identify the patch location as a sensitive location, and instrument and compile the JavaScript engine source file. Step 6: Use the JavaScript test case generation model to generate JavaScript test cases as input files for targeted fuzz testing, and perform targeted fuzz testing on the JavaScript engine; Step 4 includes: Based on the BERT pre-trained model, a fine-tuning structure is added. The fine-tuning structure includes two linear neural network layers and a ReLU function between the two neural network layers. The sizes of the two neural network layers are {input length * BERT model hidden dimension} and {BERT model hidden dimension * 1024}, respectively. The output of this fine-tuning structure is superimposed on the output of the MLM output segment in the BERT pre-trained model. The MLM output segment has the same composition as the fine-tuning structure. The AST subtree sequence and the parent node sequence of the AST subtree are respectively input into the simplified BERT model and the fine-tuning mechanism for processing. Finally, the output of the MLM output segment in the BERT pre-trained model and the output of the fine-tuning mechanism are superimposed and then subjected to ReLU activation and layer normalization. The normalization result is uniformly output using a linear layer of size {1024*vocab length}. The output result is the probability distribution of the predicted next word on the vocab. Step 6 includes: A random AST subtree sequence is selected, and a certain proportion of points are mutated and replaced. The replacement points are processed sequentially. For each replacement point, the parent and child nodes are first obtained, and all replacement points and child nodes are deleted. Starting from the parent node, a JavaScript test case generation model is used to generate child nodes. The child nodes are selected from the top-K pre-selected nodes. If the child node is a non-terminal symbol, the next node is generated from that child node. A subtree height threshold is limited. Nodes exceeding the subtree height threshold are forced to generate a terminal symbol to end the generation process, thereby ensuring the legality of the generated subtree. The mutated AST tree is restored to JavaScript test cases as input for targeted fuzz testing, and targeted fuzz testing is performed on the JavaScript engine.

2. The method for targeted fuzz testing of a JavaScript engine based on natural language processing according to claim 1, characterized in that, Step 2 includes: Define an AST subtree sequence `astlist` to initialize the seed, with empty lists for sentences and paragraphs. Use the `travel` function to recursively traverse the AST tree, constructing the AST subtree sequence, sentences, and paragraphs, and saving the relevant information. The `travel` function first recursively traverses the AST tree, checking each node sequentially. If the parent node of a node is of type "Program", it means that the node is the starting node of a sentence, so the previous sentence is added to the paragraph, and the sentence is reinitialized. If the target node `node` has child nodes, add the pruned subtree, parent node information, and parent node type information of the node to `astlist`, add the pruned subtree to the sentence, and traverse the node. If the node has no child nodes, it means that the node is already a terminal symbol, so update `vocab` and exit the current recursion.

3. The method for targeted fuzz testing of a JavaScript engine based on natural language processing according to claim 1, characterized in that, Step 3 includes: Vocabulary and sentence are used as pre-training inputs for the BERT language model. The BERT model is simplified to prevent overfitting during training. Adjacent and non-adjacent sentence pairs are selected proportionally from the sentence set. The embedding of the BERT input sequence is the sum of word embedding, sentence embedding, and position embedding. Each word, sentence tag, and position tag is transformed into a fixed-dimensional vector. The input data is processed through the simplified BERT model. Feature extraction is performed through the two pre-training tasks of MLM and next sentence prediction in the BERT model. The grammatical and semantic information of the JavaScript language is learned, and finally, the BERT pre-trained model is obtained.

4. A targeted fuzz testing system based on a JavaScript engine using natural language processing, characterized in that, include: The parsing module is used to parse JavaScript seed files in the corpus into an abstract syntax tree (AST), and delete seed files with syntax errors. The building module is used to traverse the AST tree, construct a sequence of AST subtrees of height 2, and build the BERT language model vocabulary and BERT language model sentence based on the AST subtree sequence; The pre-training module is used to take vocab and sentence as input for the pre-training of the BERT language model to obtain the BERT pre-trained model. The fine-tuning module is used to obtain the parent node sequence of the AST subtree based on the AST subtree sequence. It uses the parent node sequence of the AST subtree as the input of the residual network using the BERT pre-trained model and combines it with the masked language model MLM for fine-tuning to obtain the JavaScript test case generation model. The instrumentation compilation module is used to compare the source file information of different versions of the JavaScript engine, use the patch location as a sensitive location, and instrument the JavaScript engine source file for compilation. The fuzzing module is used to generate JavaScript test cases using the JavaScript test case generation model as input files for targeted fuzzing, and to perform targeted fuzzing on the JavaScript engine. The fine-tuning module is specifically used for: Based on the BERT pre-trained model, a fine-tuning structure is added. The fine-tuning structure includes two linear neural network layers and a ReLU function between the two neural network layers. The sizes of the two neural network layers are {input length * BERT model hidden dimension} and {BERT model hidden dimension * 1024}, respectively. The output of this fine-tuning structure is superimposed on the output of the MLM output segment in the BERT pre-trained model. The MLM output segment has the same composition as the fine-tuning structure. The AST subtree sequence and the parent node sequence of the AST subtree are respectively input into the simplified BERT model and the fine-tuning mechanism for processing. Finally, the output of the MLM output segment in the BERT pre-trained model and the output of the fine-tuning mechanism are superimposed and then subjected to ReLU activation and layer normalization. The normalization result is uniformly output using a linear layer of size {1024*vocab length}. The output result is the probability distribution of the predicted next word on the vocab. The fuzz testing module is specifically used for: A random AST subtree sequence is selected, and a certain proportion of points are mutated and replaced. The replacement points are processed sequentially. For each replacement point, the parent and child nodes are first obtained, and all replacement points and child nodes are deleted. Starting from the parent node, a JavaScript test case generation model is used to generate child nodes. The child nodes are selected from the top-K pre-selected nodes. If the child node is a non-terminal symbol, the next node is generated from that child node. A subtree height threshold is limited. Nodes exceeding the subtree height threshold are forced to generate a terminal symbol to end the generation process, thereby ensuring the legality of the generated subtree. The mutated AST tree is restored to JavaScript test cases as input for targeted fuzz testing, and targeted fuzz testing is performed on the JavaScript engine.

5. A JavaScript engine-based targeted fuzz testing system according to claim 4, characterized in that, The building module is specifically used for: Define an AST subtree sequence `astlist` to initialize the seed, with empty lists for sentences and paragraphs. Use the `travel` function to recursively traverse the AST tree, constructing the AST subtree sequence, sentences, and paragraphs, and saving the relevant information. The `travel` function first recursively traverses the AST tree, checking each node sequentially. If the parent node of a node is of type "Program", it means that the node is the starting node of a sentence, so the previous sentence is added to the paragraph, and the sentence is reinitialized. If the target node `node` has child nodes, add the pruned subtree, parent node information, and parent node type information of the node to `astlist`, add the pruned subtree to the sentence, and traverse the node. If the node has no child nodes, it means that the node is already a terminal symbol, so update `vocab` and exit the current recursion.

6. A JavaScript engine-based targeted fuzz testing system according to claim 4, characterized in that, The pre-training module is specifically used for: Vocabulary and sentence are used as pre-training inputs for the BERT language model. The BERT model is simplified to prevent overfitting during training. Adjacent and non-adjacent sentence pairs are selected proportionally from the sentence set. The embedding of the BERT input sequence is the sum of word embedding, sentence embedding, and position embedding. Each word, sentence tag, and position tag is transformed into a fixed-dimensional vector. The input data is processed through the simplified BERT model. Feature extraction is performed through the two pre-training tasks of MLM and next sentence prediction in the BERT model. The grammatical and semantic information of the JavaScript language is learned, and finally, the BERT pre-trained model is obtained.

Images